Difference between revisions of "Contrib/PyFoam"

Revision as of 19:56, 29 March 2007

Valid versions: works with older versions (but I don't support that)

1 Short description

This Python-library can be used to

analyze the logs produced by OpenFoam-solvers
execute OpenFoam-solvers and utilities and analyze their output simultaneously
manipulate the parameter files and the initial-conditions of a run in a non-destructive manner
plots the residuals of OpenFOAM solvers

1.1 Motivation

This library was developed to control OpenFOAM-simulations with a decent (sorry Perl) scripting language to do parameter-variations and results analysis.

It is an ongoing effort. I add features on an As-Needed basis but am open to suggestions.

1.2 Compatibility

This Library tries to be as conservative as possible in the libraries it uses. This means that it tries to use only libraries that are in the standard Python-distribution. Currently it needs at least version 2.3 of Python because of the threading-support.

For plotting support (which is pretty rudimentary anyway) it needs the Gnuplot-library, but installs without.

It should run on any system with OpenFOAM and python2.3. Currently it has been tested on

Linux (various flavours)
MacOS X

2 Examples

These examples should demonstrate the possible applications of the library.

2.1 Compact output

This example is a wrapper for solvers. For each time-step only one line is output: the time and the initial residuals of the linear solvers (with the name of the solved variable). The complete output of the solver is written to a log-file. The residuals are written to separate files in a special directory in the simulation-directory (for plotting).

 
import re,sys
 
from PyFoam.LogAnalysis.LogLineAnalyzer import LogLineAnalyzer
from PyFoam.LogAnalysis.BoundingLogAnalyzer import BoundingLogAnalyzer
from PyFoam.Execution.AnalyzedRunner import AnalyzedRunner
 
class CompactLineAnalyzer(LogLineAnalyzer):
    def __init__(self):
        LogLineAnalyzer.__init__(self)
 
        self.told=""
        self.exp=re.compile("^(.+):  Solving for (.+), Initial residual = (.+), Final residual = (.+), No Iterations (.+)$")
 
    def doAnalysis(self,line):
        m=self.exp.match(line)
        if m!=None:
            name=m.groups()[1]
            resid=m.groups()[2]
            time=self.getTime()
            if time!=self.told:
                self.told=time
                print "\n t = %6g : " % ( float(time) ),
            print " %5s: %6e " % (name,float(resid)),
            sys.stdout.flush()
 
class CompactAnalyzer(BoundingLogAnalyzer):
    def __init__(self):
        BoundingLogAnalyzer.__init__(self)
        self.addAnalyzer("Compact",CompactLineAnalyzer())
 
run=AnalyzedRunner(CompactAnalyzer(),silent=True)
run.start()

2.2 Parameter Variation

This example does 10 runs of the same case. For each case a different velocity at the inlet is set. After the simulation has ended the mass flow (with this utility) and the pressure difference (with this utility) are calculated and written to the file InflowVariationResults. In addition to this the data of the last time-step is saved to a separate directory (for further analysis):

 
from PyFoam.Execution.ConvergenceRunner import ConvergenceRunner
from PyFoam.Execution.UtilityRunner import UtilityRunner
from PyFoam.LogAnalysis.BoundingLogAnalyzer import BoundingLogAnalyzer
from PyFoam.RunDictionary.SolutionFile import SolutionFile
from PyFoam.RunDictionary.SolutionDirectory import SolutionDirectory
 
solver="simpleFoam"
case="pseudoPoro"
pCmd="calcPressureDifference"
mCmd="calcMassFlow"
 
dire=SolutionDirectory(case,archive="InletVariation")
dire.clearResults()
dire.addBackup("PyFoamSolve.logfile")
dire.addBackup("PyFoamSolve.analyzed")
dire.addBackup("Pressure.analyzed")
dire.addBackup("MassFlow.analyzed")
 
sol=SolutionFile(dire.initialDir(),"U")
 
maximum=1.
nr=10
 
f=dire.makeFile("InflowVariationResults")
 
for i in range(nr+1):
    val=(maximum*i)/nr
    print "Inlet velocity:",val
    sol.replaceBoundary("rein","(%f 0 0)" %(val))
    run=ConvergenceRunner(BoundingLogAnalyzer(),argv=[solver,".",case],silent=True)
    run.start()
 
    print "Last Time = ",dire.getLast()
 
    pUtil=UtilityRunner(argv=[pCmd,".",case],silent=True,logname="Pressure")
    pUtil.add("deltaP","Pressure at .* Difference .*\] (.+)")
    pUtil.start()
 
    deltaP=pUtil.get("deltaP")[0]
 
    mUtil=UtilityRunner(argv=[mCmd,".",case,"-latestTime"],silent=True,logname="MassFlow")
    mUtil.add("mass","Flux at (.+) = .*\] (.+)",idNr=1)
    mUtil.start()
 
    massFlow=mUtil.get("mass",ID="raus")[0]
 
    dire.lastToArchive("vel=%g" % (val))
 
    dire.clearResults()
 
    print "Vel: ",val,"DeltaP: ",deltaP,"Mass Flow:",massFlow
    f.writeLine( (val,deltaP,massFlow) )
 
sol.purgeFile()

3 Installation

The easies way to install PyFoam is as root, but it is also possible to do it as a regular user.

3.1 Installation as root

Download the tar file with the sources
Untar it
Go to the directory PyFoam-0.4.0
Install it with the command

 
python setup.py install

Should step 4 fail one of the most common causes is that some Linux-distributions (usually Debian-based, don't know about Mandrake) don't install the packages that are necessary for setup.py. Usually they are in the Developer-package which is installed with

 
apt-get install python-dev

(on Debian) or similar.

3.2 Installation as a regular user

If you don't want to (or can't) install it as root things are a bit more complicated. Let's assume that you have directory /home/nonroot/my_python to which you have write access:

Download PyFoam-0.x.x (0.x.x being the current version)
Untar it
Go to the directory PyFoam-0.x.x
Install it with the command

 
python setup.py install --prefix=/home/nonroot/my_python

In /home/nonroot/my_python there should be two directories: bin and lib

Add the path to the library to the PYTHONPATH environment-variable

 
export PYTHONPATH=/home/nonroot/my_python/lib/python-2.3/site-packages:$PYTHONPATH

(the lib/python-2.3/site-packages portion of the path may depend on your Python-version; in that directory there must be a directory named PyFoam)

If you want to use the scripts distributed with PyFoam extend your path with

 
export PATH=/home/nonroot/my_python/bin:$PATH

3.3 Installation via RPMs

Starting with version 0.4.0 a source rpm and a regular rpm are provided for those who prefer to install it that way (The rpms have not been extensivly tested. If there is a problem: please contact me)

3.4 Testing the installation

To test whether the installation of PyFoam is working start the interactive Python shell with

 
python

(The shell can be left with Control-D). In that shell type

 
import PyFoam
import PyFoam.FoamInformation
print PyFoam.FoamInformation.foamTutorials()

If the last command produces an output similar to

 
/home/nonroot/OpenFOAM/OpenFOAM-1.2-devel/tutorials

everything is well.

If things don't work check in the Python-shell with the commands

 
import sys
print sys.path

whether the directory in which PyFoam resides is on the path searched by Python.

4 Installed Utilities

All the utilies described in this section are installed to /usr/bin (if installed as root) and should be in the standard path for execution. They can all be called with the --help-option which gives all the available options and the latest, up-to-date information about the utility.

One recommended usage of these utilities would be to start a simulaiton with foamJob and display the residuals with pyFoamPlotWatcher.py without interfering with the simulation.

4.1 Runner-Utilites

These are utilities that start and control OpenFOAM-Runs. Apart from the features described below they start a little XMLRPC-server through which these runs can be controlled (see Network-Utilities).

Most of the utilities can be used to start parallel runs (the number of processors and a host-file have to be provided, depending on the MPI-implementation. The parallel environment is set up automatically).

4.1.1 `pyFoamRunner.py`

Runs an OpenFoam solver. Needs the usual 3 arguments (<solver> <directory> <case>) and passes them on (plus additional arguments). Output is sent to stdout and a logfile inside the case directory (PyFoamSolver.logfile). The Directory PyFoamSolver.analyzed contains this information:

Residuals and other information of the linear solvers
Execution time
continuity information
bounding of variables

4.1.2 `pyFoamUtilityRunner.py`

Runs a OpenFoam Utility an analyzes the output. Needs a regular expression to look for. The next 3 arguments are the usual OpenFoam argumens (<solver> <directory> <case>) and passes them on (plus additional arguments). Output is sent to stdout and a logfile inside the case directory (PyFoamUtility.logfile). The Directory PyFoamUtility.analyzed contains a file with the information of the regexp (the pattern groups) at every time-step.

Doesn't start an XMLRPC-Server

4.1.3 `pyFoamSteadyRunner.py`

Runs an OpenFoam steady solver. Needs the usual 3 arguments (<solver> <directory> <case>) and passes them on (plus additional arguments) Output is sent to stdout and a logfile inside the case directory (PyFoamSolver.logfile). The Directory PyFoamSolver.analyzed contains this information

Residuals and other information of the linear solvers
Execution time
continuity information
bounding of variables

If the solver has converged (all linear solvers below threshold) it is stopped and the last simulation state is written to disk

4.1.4 `pyFoamPlotRunner.py`

Runs an OpenFoam solver. Needs the usual 3 arguments (<solver> <directory> <case>) and passes them on (plus additional arguments). Output is sent to stdout and a logfile inside the case directory (PyFoamSolver.logfile). Information that is output as graphs are

residuals of the linear solver
continuity
bounding of variables (if it occurs)
Courant-number (optional)
Execution time of a time-step (optional)
number of iterations of the linear solver (optional)
arbitrary regular expressions can used to look for solver-specific data
- The groups in the regular expressions (patterns that are in brackets) are plotted
- in the regular expressions %f% can be used as a shortcut for floating-point numbers
- Any number of expressions can be specified and will be plotted in a separate window
For steady runs it behave like pyFoamSteadyRunner.py

4.2 Utilities for Logfiles

These utilities can be used to analyze the output of an OpenFOAM-solver that has been written to a file

4.2.1 `pyFoamPlotWatcher.py`

Gets the name of a logfile which is assumed to be the output of a OpenFOAM- solver. Parses the logfile for information about the convergence of the solver and generates gnuplot-graphs. Watches the file until interrupted.

4.2.2 `pyFoamStandardLogAnalyzer.py`

Analyzes a Log written by foamJob. Needs the name of the Logfile to be analyzed the data is being written to a directory that has the same name with _analyzed appended

4.3 Networking Utilities

Most Runner-Utilities start a Server-thread that makes it possible to contact control an OpenFOAM-run and get information about it. These utilities help to access these servers.

Notes:

These utilities are beta-qualitiy. They work for my network but have not been tested in other environments
- The address of the pyFoamMetaServer.py is hardcoded into the sources, but can be overwritten using system-wide configuration-files
pyFoamNetShell.py doesn't need any additional infrastructure. Only the address and the port have to be known

4.3.1 `pyFoamNetList.py`

This command lists all the OpenFOAM-simulations in your network environment and gives you information about them. Amongst the information is:

Hostname and port (useful for pyFoamNetShell.py)
Details about the run (solver, working directory ...)
Estimates about the time at which the run will end (using the time since the start, current timestep and the endTime)

This utilitiy only works if a pyFoamMetaServer.py can be contacted at the configured address

4.3.1.1 `pyFoamMetaServer.py`

This program spawns a daemon process that runs forever. It stores information about the running processes. There are two mechanisms for it to get that information

scanning the network (at the start and on demand)
Runner-utilities try to register with it, when they start an OpenFOAM-run
- The utilities also dregister their process when they exit gracefully

4.3.2 `pyFoamNetShell.py`

Connects to an OpenFOAM-run started by a runner-Utility using

the hostname of the machine it runs on
the port number (usually 18000, if more than one run is running one of the following ports)

and starts an interactive shell there. On the shells the XMLRPC-commands provided by the server can be used to

get information about the run
- environment
- start-times
manipulate the run
- for instance stop() tells the run to write at the next time-step and then end
- kill() brutally terminates the run

Help about the available commands is provided.

Can also be used to control the pyFoamMetaServer.py (which is usually contacted on port 17999)

4.4 Utilities for Manipulating case data

These utilities are used for manipulating case date. They are especially useful in scripts that automatically set up simulation runs.

4.4.1 `pyFoamAddEmptyBoundary.py`

Adds an empty patch to the boundary file. Such a patch is needed for some mesh-maipulation utilities.

4.4.2 `pyFoamChangeBoundaryType.py`

Changes the type of a boundary patch.

4.4.3 `pyFoamClearCase.py`

Removes all the timesteps except for the first from a case directory.

4.4.4 `pyFoamReadDictionary.py`

Reads data from the root of a OpenFOAM-dictionary and prints it. To access subexpressions of a dictionary entry Python-expressions can be used (note: the expression is evaluated by the Python-interpreter, so quoting has to be used for strings, otherwise Python confuses them with variables)

4.4.5 `pyFoamWriteDictionary.py`

Writes data to the root of a OpenFOAM-dictionary and writes it to disk. Subexpressions can be selected (see note: pyFoamReadDictionary.py)

4.5 Other

Utilities that don't fit any of the other categories.

4.5.1 `pyFoamDecompose.py`

Generates a decomposeParDict according to the user specifications and
runs dcomposePar

This utility is quite useful when writing scripts where the number of available processors is not known beforehand (for instance if you are using a queueing-system)

4.5.2 `pyFoamComparator.py`

A utilitiy that automatically does a series of variations on a run (for benchmarking, testing and verification purposes). The behaviour is specified by an XML-file.

Work in progress. Therefor no example is present and the DTD that assists editing the XML-file is not included. Contact me, if you want to give it a spin (and want the DTD)

4.5.3 `pyFoamVersion.py`

Prints the Version of OpenFOAM and pyFoam. Used for information purposes (when reporting bugs)

5 Library Documentation

The tar-file contains the documentation of the Library as HTML (in the folder doc)

6 Technical notes/known problems

This is a list of the currently known problems. If you find any other bugs (also known as "surprising features") don't hesitate to contact the author

the parsing mechanism for dictionaries seems to have problems with long turbulenceDicts. Currently there is no other way than remove all unneeded turulence-models from that file
if a run-fails the error message might be "eaten" by the utility controlling it. I'm working on this
the only MPI-environment I currently have available is OpenMPI. The pyFoam-Libraries used to work with LAM (still should, but I can not test it). If you have problems, please contact me (also "success-stories": "it works with MPIch" are most welcome)

7 Additional stuff

Since Version 0.2 the distribution contains the benchFoam.py-script (Since 0.2.4 the benchmark suite is stable enough to be used by the public). From version 0.3.0 on it can be used as pyFoamBench.py

8 History

2005-08-15: Version 0.1

2006-01-13: Version 0.2

2006-01-16: Version 0.2.1

2006-01-16: Version 0.2.2

2006-01-23: Version 0.2.3

2006-01-23: Version 0.2.4

2006-02-03: Version 0.2.5
- Improves the stability of the benchmark-script for parallel runs
- adds benchmakr suite for ancient versions of OpenFOAM

2006-02-03: Version 0.2.6
- Minor changes to the benchmark-scripts (other cases than the tutorial cases can now be used for benching)

2006-02-23: Version 0.3.0
- live plotting of residuals of solvers
- benchmark script moved to bin-directory

2006-03-22: Version 0.3.1
- BugFix: Plotting residuals in interFoam did not work
- Extending the Resiudals-Plotting (Courant-Numbers etc)
- Parsing of Dictionary-files into Python-Dictionaries (not yet fully tested)

2006-03-22: Version 0.3.2
- Plotting resiudals now works with OpenFOAM 1.3

2006-10-18: Version 0.3.3
- pyFoam distinguishes between the used MPI-implementations when starting parallel runs (currently LAM and OpenMPI)
- BugFixes
- added Utility pyFoamAddEmptyBoundary

2006-11-29: Version 0.3.4
- An error pointed out by Andrea Rimondi: the last version was incomplete. Due to a code reorganisation and missing files in the distribution half of the scripts in bin stopped working, but apart from Andrea everyone was too polite to mention it
- added Utility pyFoamClearCase.py

2007-03-29: Version 0.4.0
- Reimplementation of the parsing of OpenFOAM-Dictionaries
  - Can be manipulated like regular Python-Data-Structures
- Extension of the plotting-Utilities: arbitrary expressions are possible
  - The data can also be written to the disk
- Runs that are started by pyFoam can be accessed and controlled over the network (using XMLRPC)
- The runner-Utilities can start parallel-runs
  - A utility for generating a dictionary and decomposing a case is added
- Bug removed that didn't let pyFoamSteadyRunner.py work properly with OpenFOAM 1.3 (due to changes in the solver output)

--Bgschaid 21:50, 15 Aug 2005 (CEST)

@@ Line 369: / Line 369: @@
 The tar-file contains the documentation of the Library as HTML (in the folder <tt>doc</tt>)
+== Technical notes/known problems ==
+This is a list of the currently known problems. If you find any other bugs (also known as "surprising features") don't hesitate to contact the author
+* the parsing mechanism for dictionaries seems to have problems with long turbulenceDicts. Currently there is no other way than remove all unneeded turulence-models from that file
+* if a run-fails the error message might be "eaten" by the utility controlling it. I'm working on this
+* the only MPI-environment I currently have available is OpenMPI. The pyFoam-Libraries used to work with LAM (still should, but I can not test it). If you have problems, please contact me (also "success-stories": "it works with MPIch" are most welcome)
 == Additional stuff ==

Difference between revisions of "Contrib/PyFoam"

Revision as of 19:56, 29 March 2007

Contents

1 Short description

1.1 Motivation

1.2 Compatibility

2 Examples

2.1 Compact output

2.2 Parameter Variation

3 Installation

3.1 Installation as root

3.2 Installation as a regular user

3.3 Installation via RPMs

3.4 Testing the installation

4 Installed Utilities

4.1 Runner-Utilites

4.1.1 pyFoamRunner.py

4.1.2 pyFoamUtilityRunner.py

4.1.3 pyFoamSteadyRunner.py

4.1.4 pyFoamPlotRunner.py

4.2 Utilities for Logfiles

4.2.1 pyFoamPlotWatcher.py

4.2.2 pyFoamStandardLogAnalyzer.py

4.3 Networking Utilities

4.3.1 pyFoamNetList.py

4.3.1.1 pyFoamMetaServer.py

4.3.2 pyFoamNetShell.py