Difference between revisions of "Contrib/PyFoam/benchFoam"

From OpenFOAMWiki
m
m (Bgschaid moved page Contrib benchFoam to Contrib/PyFoam/benchFoam)
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
''There are three kinds of lies: lies, damn lies and benchmarks''
 
''There are three kinds of lies: lies, damn lies and benchmarks''
  
This script can be found in the examples-directory of the [[contrib_PyFoam|PyFoam]]-distribution.
+
This script can be found in the examples-directory of the [[contrib_PyFoam|PyFoam]]-distribution. It used to be called <tt>benchFoam.py</tt> and now is a command <tt>pyFoamBench.py</tt>
 +
 
 +
{{VersionInfo}}{{Version1.2}}{{Version1.3}}{{Version1.4}}
  
 
== Usage ==
 
== Usage ==
  
From the example-directory of the [[contrib_PyFoam|PyFoam]] distribution copy the benchFoam.py-script and a configuration File (for instance <tt>data/default.cfg</tt> to a local directory. The benchmark can then be run with the command
+
From the example-directory of the [[contrib_PyFoam|PyFoam]] distribution copy a configuration File (for instance <tt>data/default.cfg</tt> to a local directory. The benchmark can then be run with the command
  
  ./benchFoam.py default.cfg
+
  pyFoamBench.py default.cfg
  
An arbitrary number of configuration files can be specified.
+
An arbitrary number of configuration files can be specified. Configuration files can be found in the <tt>examples</tt>-directory of the [[contrib_PyFoam|PyFoam]]-source-distribution
  
 
The script then copies the specified cases from the <tt>$FOAM_TUTORIAL</tt>-directory to the local directory, modifies them and then runs the solver on them. It records the wallClock- and the CPU-time and writes the information to a file. The speedup is calculated by comparing the wallClock-time to a reference time.
 
The script then copies the specified cases from the <tt>$FOAM_TUTORIAL</tt>-directory to the local directory, modifies them and then runs the solver on them. It records the wallClock- and the CPU-time and writes the information to a file. The speedup is calculated by comparing the wallClock-time to a reference time.
Line 26: Line 28:
 
  nProcs: 2
 
  nProcs: 2
 
  machines: benchMachines
 
  machines: benchMachines
 +
casesDirectory: ~/myBenchmarks
 
   
 
   
  [Incompressible driven cavity]
+
  [AachenBomb with dieselFoam]
  nr: 1
+
  solver: dieselFoam
  solver: icoFoam
+
  case: aachenBomb
  case: cavity
+
  additional: ["chemkin"]
 
  prepare: [("blockMesh","")]
 
  prepare: [("blockMesh","")]
  controlDict: [("endTime",2),("writeInterval",50)]
+
  controlDict: [("endTime",1e-4),("writeInterval",0.5e-4),("writeCompression","compressed")]
blockSplit: (2,2,1)
+
  baseline: 2894
  baseline: 4.013
+
 
  weight: 1
 
  weight: 1
 +
filesToRemove: ["0/ft","0/fu"]
 +
parallelOK: no
 
   
 
   
 
  [Dam-break tutorial case]
 
  [Dam-break tutorial case]
Line 45: Line 49:
 
  weight: 3
 
  weight: 3
 
  blockSplit: (2,2,1)
 
  blockSplit: (2,2,1)
 +
parallelOK: yes
 
   
 
   
 
  [HotRoom with buoyantFoam]
 
  [HotRoom with buoyantFoam]
Line 52: Line 57:
 
  prepare: [("blockMesh",""),("setHotRoom","")]
 
  prepare: [("blockMesh",""),("setHotRoom","")]
 
  controlDict: [("endTime",10),("deltaT",0.05),("writeInterval",100),("writeCompression","compressed")]
 
  controlDict: [("endTime",10),("deltaT",0.05),("writeInterval",100),("writeCompression","compressed")]
 +
setInitial: [("p","floor","1e5"),("p","ceiling","1e5"),("p","fixedWalls","1e5")]
 
  baseline: 826.375
 
  baseline: 826.375
 
  weight: 1
 
  weight: 1
 
  blockSplit: 2
 
  blockSplit: 2
 +
parallelOK: yes
 +
decomposition: simple 1
  
 
The file is split into sections. Each section starts with a section name given in square brackets. The <tt>General</tt>-section has to be present. It specifies some general information about the benchmark:
 
The file is split into sections. Each section starts with a section name given in square brackets. The <tt>General</tt>-section has to be present. It specifies some general information about the benchmark:
  
* name: the name of the benchmark. This is used in the names of the output-files and directories
+
; name: the name of the benchmark. This is used in the names of the output-files and directories
* parallel: whether or not it is a parallel benchmark
+
; parallel: whether or not it is a parallel benchmark
* nProcs: the number of CPUs used for the parallel benchmark
+
; nProcs: the number of CPUs used for the parallel benchmark
* machines: name of the file used to boot the LAM-machine
+
; machines: name of the file used to boot the LAM-machine
 +
; casesDirectory: Directory where the benchmark cases reside. If not set the value of the <tt>$FOAM_TUTORIALS</tt> (where the standard tutorial cases reside) variable is used. In that directory the cases must be organized in the same way they are in <tt>$FOAM_TUTORIALS</tt>: a separate directory for every solver in which the cases for that solver reside.
  
 
Each of the other sections specifies a different benchmark case. The name of the Section is the name under which the benchmark will be known for screen-output. The options in the section are:
 
Each of the other sections specifies a different benchmark case. The name of the Section is the name under which the benchmark will be known for screen-output. The options in the section are:
  
* nr: the order in which the benchmarks will be executed. If unspecified or two numbers are the same the order will be unspecified
+
; nr: the order in which the benchmarks will be executed. If unspecified or two numbers are the same the order will be unspecified
* solver: name of the solver to execute
+
; solver: name of the solver to execute
* case: name of the tutorial case for that solver
+
; case: name of the tutorial case for that solver
* prepare: a list of commands to execute in order to prepare the case for running the benchmark. Each command is given by a pair: the first value is the name of the command, the second value are the additional options that are inserted after the working directory and the casename (the usual calling convention for OpenFOAM-utilities)
+
; prepare: a list of commands to execute in order to prepare the case for running the benchmark. Each command is given by a pair: the first value is the name of the command, the second value are the additional options that are inserted after the working directory and the casename (the usual calling convention for OpenFOAM-utilities). If the string <tt>%case%</tt> appears in the second value, it is replaced with the name of the case-directory.
* utilities: an optional list of utilities that have to be compiled in order to run the case. It is assumed that the sources of the utilities reside in the directory of the case (usually the case for the tutorial cases)
+
; utilities: an optional list of utilities that have to be compiled in order to run the case. It is assumed that the sources of the utilities reside in the directory of the case (usually the case for the tutorial cases)
* controlDict: values that are to be changed in the standard <tt>controlDict</tt> of the tutorial case in order to change the running time of the case
+
; controlDict: values that are to be changed in the standard <tt>controlDict</tt> of the tutorial case in order to change the running time of the case
* baseline: time it takes for the case to run on a reference machine
+
; baseline: time it takes for the case to run on a reference machine
* weight: weight with which this case contributes to the overall-speedup of the benchmark suite
+
; weight: weight with which this case contributes to the overall-speedup of the benchmark suite
* blockSplit: optional value that is used to resize the mesh size in a <tt>blockMesh</tt>. If a scalar each number of cells is multiplied with that value. If a triple, then each direction is multiplied with the corresponding value.
+
; blockSplit: optional value that is used to resize the mesh size in a <tt>blockMesh</tt>. If a scalar each number of cells is multiplied with that value. If a triple, then each direction is multiplied with the corresponding value.
 +
; parallelOK: optional value that says whether or not this case can be run in parallel. If no value is set it is assumed that the case '''can not'' be run in parallel
 +
; filesToRemove: optional value. A list of files that should be removed from the case before it is prepared
 +
; decomposition: optional value. In which way a case should be decomposed for parallel runs. Default is ''metis''. The other valid value is ''simple'' plus a number (0,1 or 2) that says which is the primary direction of decomposition
 +
; setInitial: optional value. A list of triples that set initial values. The elements of the triple are:
 +
# the name of the field
 +
# the name of the boundary
 +
# the value
  
 
=== Warning ===
 
=== Warning ===
  
 
Because the script measures the wall-clock-time no other activity (users, server tasks) should take place on that machine.
 
Because the script measures the wall-clock-time no other activity (users, server tasks) should take place on that machine.
 +
 +
==== Compatibility ====
 +
 +
The script should work on all systems compatible with [[contrib_PyFoam|PyFoam]]. Some systems have shortcomings:
 +
 +
;Linux: Because <tt>getrusage</tt> is not correctly implemented the Maximum-Memory-Usage can only be approximated by a separate thread that monitors the memory usage. Because this thread only monitors ever 10 seconds (to keep the performance impact low) it might miss peaks in the usage. Debian-based Linuxes (Ubuntu for instance) seem to have similar problems as Mac OS X (no CPU-Time).
 +
;Mac OS X: Threading seems to be strangely implemented. Therefor the Wall-Clock-Time may be some split-seconds off. Also no CPU-Time is available.
  
 
== Discussion ==
 
== Discussion ==
Line 101: Line 124:
 
This machine was choosen because it is the slowest machine I have currently available.
 
This machine was choosen because it is the slowest machine I have currently available.
  
== A possible benchmark suite ==
+
=== A possible benchmark suite ===
  
 
The simulations in this suite were chosen to fit with the above requirements and give a cross-section of the available solvers in OpenFOAM:
 
The simulations in this suite were chosen to fit with the above requirements and give a cross-section of the available solvers in OpenFOAM:
Line 108: Line 131:
 
|+Benchmark suite
 
|+Benchmark suite
 
|-
 
|-
! solver !! case-name !! Modification to original case !! Features  !! Remarks/Problems
+
! solver !! case-name !! Modification to original case !! Memory !! Features  !! Remarks/Problems
 
|-
 
|-
 
! dieselFoam
 
! dieselFoam
 
| aachenBomb
 
| aachenBomb
|  
+
| files are removed before running
 +
| 275 MB
 
| Lagrangian particles, chemical reactions with ChemKin
 
| Lagrangian particles, chemical reactions with ChemKin
| Decompose fails for parallel
+
| Parallel run fails for v1.2: [http://openfoam.cfd-online.com/forum/messages/126/1634.html the patch published here fixes this]
 
|-
 
|-
 
! dnsFoam
 
! dnsFoam
 
| boxTurb16
 
| boxTurb16
 
| Splitting the grid
 
| Splitting the grid
 +
| 45 MB
 
| DNS
 
| DNS
| Parallel run fails
+
| Solver is not parallel
 
|-
 
|-
 
! bubbleFoam
 
! bubbleFoam
 
| bubbleColumn
 
| bubbleColumn
 
|  
 
|  
 +
| 11 MB
 
| Two-phase solver
 
| Two-phase solver
 
|
 
|
Line 131: Line 157:
 
| damBreak
 
| damBreak
 
| Splitting the grid
 
| Splitting the grid
 +
| 18 MB
 
| Two-phase solver  
 
| Two-phase solver  
 
|
 
|
Line 137: Line 164:
 
| forwardStep
 
| forwardStep
 
| Splitting the grid
 
| Splitting the grid
 +
| 25 MB
 
| Super-sonic solver
 
| Super-sonic solver
 
|
 
|
 
|-
 
|-
 
! buoyantFoam
 
! buoyantFoam
| hotRoom
+
| hotRoom, pseudo-BCs are set
 
| Splitting the grid
 
| Splitting the grid
 +
| 48 MB
 
| Heat transfer
 
| Heat transfer
| Decompose fails for parallel
+
|  
 
|-
 
|-
 
! engineFoam
 
! engineFoam
 
| kivaTest
 
| kivaTest
 
|  
 
|  
 +
| 49 MB
 
| Mesh motion, combustion
 
| Mesh motion, combustion
 
|
 
|
Line 154: Line 184:
 
! Xoodles
 
! Xoodles
 
| pitzDaily3D
 
| pitzDaily3D
|  
+
| pseudo-BCs are set
 +
| 468 MB
 
| Combustion, LES
 
| Combustion, LES
| Decompose fails for parallel
+
|  
 
|-
 
|-
 
! oodles
 
! oodles
 
| pitzDaily
 
| pitzDaily
 
|  
 
|  
 +
| 25 MB
 
| LES
 
| LES
 
|
 
|
Line 167: Line 199:
 
| pitzDaily
 
| pitzDaily
 
|  
 
|  
 +
| 54 MB
 
| Steady-state solver
 
| Steady-state solver
 
|
 
|
Line 172: Line 205:
 
! sonicTurbFoam
 
! sonicTurbFoam
 
| prism
 
| prism
|  
+
| pseudo-BCs are set, grid is split
 +
| 37 MB
 
| Super-sonic, turbulent
 
| Super-sonic, turbulent
| Decompose fails for parallel
+
|  
 
|}
 
|}
 +
 +
=== Specification/Publication of benchmarks ===
 +
 +
Three main categories should be specified when talking about a benchmark (if one of them changes you're benchmarking a different system):
 +
 +
;Hardware/Operating system: Most benchmarks only specify this complex. Important info is:
 +
** CPU-type and clock-frequency
 +
** Memory
 +
** Operating system + Version
 +
;Compiler: Modern CPUs are nothing without a good compiler. If you use a recompiled version of OpenFOAM specify the compiler (+version) you're using and the compiler-switches that are used for optimization
 +
;OpenFOAM version: Algorithms get better. So benchmarking version 1.3 may get you significantly different results than version 1.2 on the same machine
 +
 +
In addition to this information specify who did this benchmark and when.
 +
 +
== Benchmark results ==
 +
 +
Since version 0.2.4 of [[contrib_PyFoam|PyFoam]] a benchmark suite named <tt>standard_v1.cfg</tt> is included in the distribution. Some example results using this suite can be found at [[benchmarks_standard_v1|Benchmarks V1]].

Latest revision as of 22:23, 18 November 2013

There are three kinds of lies: lies, damn lies and benchmarks

This script can be found in the examples-directory of the PyFoam-distribution. It used to be called benchFoam.py and now is a command pyFoamBench.py

Valid versions: OF version 12.png OF version 13.png OF version 14.png

1 Usage

From the example-directory of the PyFoam distribution copy a configuration File (for instance data/default.cfg to a local directory. The benchmark can then be run with the command

pyFoamBench.py default.cfg

An arbitrary number of configuration files can be specified. Configuration files can be found in the examples-directory of the PyFoam-source-distribution

The script then copies the specified cases from the $FOAM_TUTORIAL-directory to the local directory, modifies them and then runs the solver on them. It records the wallClock- and the CPU-time and writes the information to a file. The speedup is calculated by comparing the wallClock-time to a reference time.

If the benchmark is specified to be parallel, the specified LAM-machine is automatically booted and shutdown after running the benchmark.

The script tries to determine the maximum memory used. Because the getrusage-system call is not correctly implemented on Linux-machines (and on Mac OS X, too) this feature is untested.

1.1 Format of the config File

An example for a config-file is given below:

[General]
name: default
parallel: no
nProcs: 2
machines: benchMachines
casesDirectory: ~/myBenchmarks

[AachenBomb with dieselFoam]
solver: dieselFoam
case: aachenBomb
additional: ["chemkin"]
prepare: [("blockMesh","")]
controlDict: [("endTime",1e-4),("writeInterval",0.5e-4),("writeCompression","compressed")]
baseline: 2894
weight: 1
filesToRemove: ["0/ft","0/fu"]
parallelOK: no

[Dam-break tutorial case]
solver: interFoam
case: damBreak
prepare: [("blockMesh",""),("setFields","")]
controlDict: [("endTime",0.5),("writeInterval",0.1)]
baseline: 106.38
weight: 3
blockSplit: (2,2,1)
parallelOK: yes

[HotRoom with buoyantFoam]
solver: buoyantFoam
case: hotRoom
utilities: ["setHotRoom"]
prepare: [("blockMesh",""),("setHotRoom","")]
controlDict: [("endTime",10),("deltaT",0.05),("writeInterval",100),("writeCompression","compressed")]
setInitial: [("p","floor","1e5"),("p","ceiling","1e5"),("p","fixedWalls","1e5")]
baseline: 826.375
weight: 1
blockSplit: 2
parallelOK: yes
decomposition: simple 1

The file is split into sections. Each section starts with a section name given in square brackets. The General-section has to be present. It specifies some general information about the benchmark:

name
the name of the benchmark. This is used in the names of the output-files and directories
parallel
whether or not it is a parallel benchmark
nProcs
the number of CPUs used for the parallel benchmark
machines
name of the file used to boot the LAM-machine
casesDirectory
Directory where the benchmark cases reside. If not set the value of the $FOAM_TUTORIALS (where the standard tutorial cases reside) variable is used. In that directory the cases must be organized in the same way they are in $FOAM_TUTORIALS: a separate directory for every solver in which the cases for that solver reside.

Each of the other sections specifies a different benchmark case. The name of the Section is the name under which the benchmark will be known for screen-output. The options in the section are:

nr
the order in which the benchmarks will be executed. If unspecified or two numbers are the same the order will be unspecified
solver
name of the solver to execute
case
name of the tutorial case for that solver
prepare
a list of commands to execute in order to prepare the case for running the benchmark. Each command is given by a pair: the first value is the name of the command, the second value are the additional options that are inserted after the working directory and the casename (the usual calling convention for OpenFOAM-utilities). If the string %case% appears in the second value, it is replaced with the name of the case-directory.
utilities
an optional list of utilities that have to be compiled in order to run the case. It is assumed that the sources of the utilities reside in the directory of the case (usually the case for the tutorial cases)
controlDict
values that are to be changed in the standard controlDict of the tutorial case in order to change the running time of the case
baseline
time it takes for the case to run on a reference machine
weight
weight with which this case contributes to the overall-speedup of the benchmark suite
blockSplit
optional value that is used to resize the mesh size in a blockMesh. If a scalar each number of cells is multiplied with that value. If a triple, then each direction is multiplied with the corresponding value.
parallelOK
optional value that says whether or not this case can be run in parallel. If no value is set it is assumed that the case 'can not be run in parallel
filesToRemove
optional value. A list of files that should be removed from the case before it is prepared
decomposition
optional value. In which way a case should be decomposed for parallel runs. Default is metis. The other valid value is simple plus a number (0,1 or 2) that says which is the primary direction of decomposition
setInitial
optional value. A list of triples that set initial values. The elements of the triple are:
  1. the name of the field
  2. the name of the boundary
  3. the value

1.2 Warning

Because the script measures the wall-clock-time no other activity (users, server tasks) should take place on that machine.

1.2.1 Compatibility

The script should work on all systems compatible with PyFoam. Some systems have shortcomings:

Linux
Because getrusage is not correctly implemented the Maximum-Memory-Usage can only be approximated by a separate thread that monitors the memory usage. Because this thread only monitors ever 10 seconds (to keep the performance impact low) it might miss peaks in the usage. Debian-based Linuxes (Ubuntu for instance) seem to have similar problems as Mac OS X (no CPU-Time).
Mac OS X
Threading seems to be strangely implemented. Therefor the Wall-Clock-Time may be some split-seconds off. Also no CPU-Time is available.

2 Discussion

This describes the benchmark suite in the file standard.cfg distributed with PyFoam.

2.1 Choice of benchmark cases

The first criteria for the selection of the cases was that only standard tutorial cases are selected.

The next criterium was that the complete suite should run on the reference machine in one night. Because of this criterium most cases are not calculated for their whole duration. In addition to this the cases should comfortably fit into a machine with 0.5 Gigabytes of memory.

The running time of the individual cases was adjusted to be

  • more than a quarter of an hour (in order to keep the influence of the startup-procedure low)
  • less than an hour (to meet the overall-time requirement)

For small cases the blockMesh is refined for the case to have at least 10k cells. (this is still to small to produce reasonable results for parallel benchmarks but should assure that not the whole simulation fits into the cache of the processor)

2.2 The reference machine

Currently the reference machine in the distributed config-files is a Fedora 4 machine with a 1.8 GHz Pentium 4 and 1 Gigabyte of RAM. The installed OpenFOAM is version 1.2.

This machine was choosen because it is the slowest machine I have currently available.

2.3 A possible benchmark suite

The simulations in this suite were chosen to fit with the above requirements and give a cross-section of the available solvers in OpenFOAM:

Benchmark suite
solver case-name Modification to original case Memory Features Remarks/Problems
dieselFoam aachenBomb files are removed before running 275 MB Lagrangian particles, chemical reactions with ChemKin Parallel run fails for v1.2: the patch published here fixes this
dnsFoam boxTurb16 Splitting the grid 45 MB DNS Solver is not parallel
bubbleFoam bubbleColumn 11 MB Two-phase solver
interFoam damBreak Splitting the grid 18 MB Two-phase solver
rhoSonicFoam forwardStep Splitting the grid 25 MB Super-sonic solver
buoyantFoam hotRoom, pseudo-BCs are set Splitting the grid 48 MB Heat transfer
engineFoam kivaTest 49 MB Mesh motion, combustion
Xoodles pitzDaily3D pseudo-BCs are set 468 MB Combustion, LES
oodles pitzDaily 25 MB LES
simpleFoam pitzDaily 54 MB Steady-state solver
sonicTurbFoam prism pseudo-BCs are set, grid is split 37 MB Super-sonic, turbulent

2.4 Specification/Publication of benchmarks

Three main categories should be specified when talking about a benchmark (if one of them changes you're benchmarking a different system):

Hardware/Operating system
Most benchmarks only specify this complex. Important info is:
    • CPU-type and clock-frequency
    • Memory
    • Operating system + Version
Compiler
Modern CPUs are nothing without a good compiler. If you use a recompiled version of OpenFOAM specify the compiler (+version) you're using and the compiler-switches that are used for optimization
OpenFOAM version
Algorithms get better. So benchmarking version 1.3 may get you significantly different results than version 1.2 on the same machine

In addition to this information specify who did this benchmark and when.

3 Benchmark results

Since version 0.2.4 of PyFoam a benchmark suite named standard_v1.cfg is included in the distribution. Some example results using this suite can be found at Benchmarks V1.