Difference between revisions of "Contrib/PyFoam"

Revision as of 23:45, 23 November 2007

Valid versions: works with older versions (but I don't support that)

1 Short description

This Python-library can be used to

• analyze the logs produced by OpenFoam-solvers
• execute OpenFoam-solvers and utilities and analyze their output simultaneously
• manipulate the parameter files and the initial-conditions of a run in a non-destructive manner
• plots the residuals of OpenFOAM solvers

1.1 Motivation

This library was developed to control OpenFOAM-simulations with a decent (sorry Perl) scripting language to do parameter-variations and results analysis.

It is an ongoing effort. I add features on an As-Needed basis but am open to suggestions.

1.2 Compatibility

This Library tries to be as conservative as possible in the libraries it uses. This means that it tries to use only libraries that are in the standard Python-distribution. Currently it needs at least version 2.3 of Python because of the threading-support.

For plotting support (which is pretty rudimentary anyway) it needs the Gnuplot-library, but installs without.

It should run on any system with OpenFOAM and python2.3. Currently it has been tested on

• Linux (various flavours)
• MacOS X

2 Examples

These examples should demonstrate the possible applications of the library. They are enclosed in the source distribution.

2.1 Compact output

This example is a wrapper for solvers. For each time-step only one line is output: the time and the initial residuals of the linear solvers (with the name of the solved variable). The complete output of the solver is written to a log-file. The residuals are written to separate files in a special directory in the simulation-directory (for plotting).

 
import re,sys
 
from PyFoam.LogAnalysis.LogLineAnalyzer import LogLineAnalyzer
from PyFoam.LogAnalysis.BoundingLogAnalyzer import BoundingLogAnalyzer
from PyFoam.Execution.AnalyzedRunner import AnalyzedRunner
 
class CompactLineAnalyzer(LogLineAnalyzer):
    def __init__(self):
        LogLineAnalyzer.__init__(self)
 
        self.told=""
        self.exp=re.compile("^(.+):  Solving for (.+), Initial residual = (.+), Final residual = (.+), No Iterations (.+)$")
 
    def doAnalysis(self,line):
        m=self.exp.match(line)
        if m!=None:
            name=m.groups()[1]
            resid=m.groups()[2]
            time=self.getTime()
            if time!=self.told:
                self.told=time
                print "\n t = %6g : " % ( float(time) ),
            print " %5s: %6e " % (name,float(resid)),
            sys.stdout.flush()
 
class CompactAnalyzer(BoundingLogAnalyzer):
    def __init__(self):
        BoundingLogAnalyzer.__init__(self)
        self.addAnalyzer("Compact",CompactLineAnalyzer())
 
run=AnalyzedRunner(CompactAnalyzer(),silent=True)
run.start()

2.2 Parameter Variation

This example does 10 runs of the same case. For each case a different velocity at the inlet is set. After the simulation has ended the mass flow (with this utility) and the pressure difference (with this utility) are calculated and written to the file InflowVariationResults. In addition to this the data of the last time-step is saved to a separate directory (for further analysis):

 
from PyFoam.Execution.ConvergenceRunner import ConvergenceRunner
from PyFoam.Execution.UtilityRunner import UtilityRunner
from PyFoam.LogAnalysis.BoundingLogAnalyzer import BoundingLogAnalyzer
from PyFoam.RunDictionary.SolutionFile import SolutionFile
from PyFoam.RunDictionary.SolutionDirectory import SolutionDirectory
 
solver="simpleFoam"
case="pitzDaily"
pCmd="calcPressureDifference"
mCmd="calcMassFlow"
 
dire=SolutionDirectory(case,archive="InletVariation")
dire.clearResults()
dire.addBackup("PyFoamSolve.logfile")
dire.addBackup("PyFoamSolve.analyzed")
dire.addBackup("Pressure.analyzed")
dire.addBackup("MassFlow.analyzed")
 
sol=SolutionFile(dire.initialDir(),"U")
 
maximum=1.
nr=10
 
f=dire.makeFile("InflowVariationResults")
 
for i in range(nr+1):
    # Set the boundary condition at the inlet
    val=(maximum*i)/nr
    print "Inlet velocity:",val
    sol.replaceBoundary("inlet","(%f 0 0)" %(val))
 
    # Run the solver
    run=ConvergenceRunner(BoundingLogAnalyzer(),argv=[solver,".",case],silent=True)
    run.start()
    
    print "Last Time = ",dire.getLast()
 
    # Get the pressure difference (Using an external utility)
    pUtil=UtilityRunner(argv=[pCmd,".",case],silent=True,logname="Pressure")
    pUtil.add("deltaP","Pressure at .* Difference .*\] (.+)")
    pUtil.start()
 
    deltaP=pUtil.get("deltaP")[0]
 
    # Get the mass flow
    mUtil=UtilityRunner(argv=[mCmd,".",case,"-latestTime"],silent=True,logname="MassFlow")
    mUtil.add("mass","Flux at (.+?) .*\] (.+)",idNr=1)
    mUtil.start()
 
    massFlow=mUtil.get("mass",ID="outlet")[0]
 
    # Archive the results
    dire.lastToArchive("vel=%g" % (val))
 
    # Clear results
    dire.clearResults()
 
    # Output current stuff
    print "Vel: ",val,"DeltaP: ",deltaP,"Mass Flow:",massFlow
    f.writeLine( (val,deltaP,massFlow) )
    
sol.purgeFile()

2.3 Manipulating dictionaries

Between Version 1.3 and 1.4 the format for the specification of the linear solvers was changed. The following script takes a fvSolution-file in the old format and rewrites it in the equivalent new format (only works for some solvers, but should work for all the tutorials):

 
#! /usr/bin/python
 
"""Given a pre-OpenFOAM-1.4 fvSolution file, this script transforms the solvers
section into the equivalen 1.4-format
Incomplete, because it doesn't support (B)DCG and GaussSeidl but should work for
instance for all the tutorials"""
 
import sys
 
from PyFoam.RunDictionary.ParsedParameterFile import ParsedParameterFile
 
if len(sys.argv)<2:
    print "I need at least one fvSolution-file to work with"
    sys.exit(1)
    
for fName in sys.argv[1:]:
    print "Working on",fName
    try:
        f=ParsedParameterFile(fName)
        sol=f["solvers"]
        for name,val in sol.iteritems():
            if type(name)!=str or type(val)!=tuple or len(val)<3:
                # this is not an old-school entry
                continue
            solver=val[0]
            tol=val[1]
            rTol=val[2]
            if solver=="ICCG":
                new=( "PCG" , { "preconditioner":"DIC" } )
            elif solver=="BICCG":
                new=( "PBiCG" , { "preconditioner":"DILU" } )
            elif solver=="AMG":
                new=( "GAMG" , { "agglomerator":"faceAreaPair",
                                 "nCellsInCoarsestLevel":val[3],
                                 "mergeLevels":1,
                                 "smoother":"GaussSeidel"} )
            else:
                print "Unsupported solver",solver
                continue
            new[1]["tolerance"]=tol
            new[1]["relTol"]=rTol
            sol[name]=new
    except IOError:
        print "File",fName,"does not exist"
    except KeyError:
        print "File",fName,"is not a fvSolution-file"
 
    try:
        f.writeFile()
    except IOError:
        print "Can't write file. Content would have been:"
        print f

Of course 1.4 can read the old format, so this script is not needed. It's only an example.

3 Installation

The easies way to install PyFoam is as root, but it is also possible to do it as a regular user.

3.1 Installation as root

1. Download the tar file with the sources
2. Untar it
3. Go to the directory PyFoam-0.4.2
4. Install it with the command

 
python setup.py install

Should step 4 fail one of the most common causes is that some Linux-distributions (usually Debian-based, don't know about Mandrake) don't install the packages that are necessary for setup.py. Usually they are in the Developer-package which is installed with

 
apt-get install python-dev

(on Debian) or similar.

3.2 Installation as a regular user

If you don't want to (or can't) install it as root things are a bit more complicated. Let's assume that you have directory /home/nonroot/my_python to which you have write access:

• Download PyFoam-0.x.x (0.x.x being the current version)
• Untar it
• Go to the directory PyFoam-0.x.x
• Install it with the command

 
python setup.py install --prefix=/home/nonroot/my_python

• Add the path to the library to the PYTHONPATH environment-variable

 
export PYTHONPATH=/home/nonroot/my_python/lib/python-2.3/site-packages:$PYTHONPATH

(the lib/python-2.3/site-packages portion of the path may depend on your Python-version; in that directory there must be a directory named PyFoam)

• If you want to use the scripts distributed with PyFoam extend your path with

 
export PATH=/home/nonroot/my_python/bin:$PATH

3.3 Installation via RPMs

Starting with version 0.4.0 a source rpm and a regular rpm are provided for those who prefer to install it that way (The rpms have not been extensivly tested. If there is a problem: please contact me)

3.4 Testing the installation

To test whether the installation of PyFoam is working start the interactive Python shell with

 
python

(The shell can be left with Control-D). In that shell type

 
import PyFoam
import PyFoam.FoamInformation
print PyFoam.FoamInformation.foamTutorials()

If the last command produces an output similar to

 
/home/nonroot/OpenFOAM/OpenFOAM-1.2-devel/tutorials

everything is well.

If things don't work check in the Python-shell with the commands

 
import sys
print sys.path

whether the directory in which PyFoam resides is on the path searched by Python.

4 Installed Utilities

All the utilies described in this section are installed to /usr/bin (if installed as root) and should be in the standard path for execution. They can all be called with the --help-option which gives all the available options and the latest, up-to-date information about the utility.

One recommended usage of these utilities would be to start a simulaiton with foamJob and display the residuals with pyFoamPlotWatcher.py without interfering with the simulation.

The following descriptions are not necessarily up to date. The text generated with the --help-option is

4.1 Runner-Utilites

These are utilities that start and control OpenFOAM-Runs. Apart from the features described below they start a little XMLRPC-server through which these runs can be controlled (see Network-Utilities).

Most of the utilities can be used to start parallel runs (the number of processors and a host-file have to be provided, depending on the MPI-implementation. The parallel environment is set up automatically).

4.1.1 pyFoamRunner.py

Runs an OpenFoam solver. Needs the usual 3 arguments (<solver> <directory> <case>) and passes them on (plus additional arguments). Output is sent to stdout and a logfile inside the case directory (PyFoamSolver.logfile). The Directory PyFoamSolver.analyzed contains this information:

• Residuals and other information of the linear solvers
• Execution time
• continuity information
• bounding of variables

4.1.2 pyFoamUtilityRunner.py

Runs a OpenFoam Utility an analyzes the output. Needs a regular expression to look for. The next 3 arguments are the usual OpenFoam argumens (<solver> <directory> <case>) and passes them on (plus additional arguments). Output is sent to stdout and a logfile inside the case directory (PyFoamUtility.logfile). The Directory PyFoamUtility.analyzed contains a file with the information of the regexp (the pattern groups) at every time-step.

Doesn't start an XMLRPC-Server

4.1.3 pyFoamSteadyRunner.py

Runs an OpenFoam steady solver. Needs the usual 3 arguments (<solver> <directory> <case>) and passes them on (plus additional arguments) Output is sent to stdout and a logfile inside the case directory (PyFoamSolver.logfile). The Directory PyFoamSolver.analyzed contains this information

• Residuals and other information of the linear solvers
• Execution time
• continuity information
• bounding of variables

If the solver has converged (all linear solvers below threshold) it is stopped and the last simulation state is written to disk

4.1.4 pyFoamPlotRunner.py

Runs an OpenFoam solver. Needs the usual 3 arguments (<solver> <directory> <case>) and passes them on (plus additional arguments). Output is sent to stdout and a logfile inside the case directory (PyFoamSolver.logfile). Information that is output as graphs are

• residuals of the linear solver
• continuity
• bounding of variables (if it occurs)
• Courant-number (optional)
• Execution time of a time-step (optional)
• number of iterations of the linear solver (optional)
• arbitrary regular expressions can used to look for solver-specific data
  • The groups in the regular expressions (patterns that are in brackets) are plotted
  • in the regular expressions %f% can be used as a shortcut for floating-point numbers
  • Any number of expressions can be specified and will be plotted in a separate window
• For steady runs it behave like pyFoamSteadyRunner.py

4.1.5 pyFoamMeshUtilityRunner.py

Some utilities that manipulate meshes store the new mesh into the first timestep (so that they don't destroy the original mesh). This can be inconvenient because you have to copy the mesh back to constant for all further steps. This utility

1. Runs a mesh-utility
2. Moves the resulting mesh from the first time-step to the constant-directory

Be careful: This utility deletes the old mesh. If you don't know what you're doing, it will be gone forever

4.2 Utilities for Logfiles

These utilities can be used to analyze the output of an OpenFOAM-solver that has been written to a file

4.2.1 pyFoamPlotWatcher.py

Gets the name of a logfile which is assumed to be the output of a OpenFOAM- solver. Parses the logfile for information about the convergence of the solver and generates gnuplot-graphs. Watches the file until interrupted.

4.2.2 pyFoamStandardLogAnalyzer.py

Analyzes a Log written by foamJob. Needs the name of the Logfile to be analyzed the data is being written to a directory that has the same name with _analyzed appended

4.3 Networking Utilities

Most Runner-Utilities start a Server-thread that makes it possible to contact control an OpenFOAM-run and get information about it. These utilities help to access these servers.

Notes:

• These utilities are beta-qualitiy. They work for my network but have not been tested in other environments
  • The address of the pyFoamMetaServer.py is hardcoded into the sources, but can be overwritten using system-wide configuration-files
• pyFoamNetShell.py doesn't need any additional infrastructure. Only the address and the port have to be known

4.3.1 pyFoamNetList.py

This command lists all the OpenFOAM-simulations in your network environment and gives you information about them. Amongst the information is:

• Hostname and port (useful for pyFoamNetShell.py)
• Details about the run (solver, working directory ...)
• Estimates about the time at which the run will end (using the time since the start, current timestep and the endTime)

This utilitiy only works if a pyFoamMetaServer.py can be contacted at the configured address

4.3.1.1 pyFoamMetaServer.py

This program spawns a daemon process that runs forever. It stores information about the running processes. There are two mechanisms for it to get that information

• scanning the network (at the start and on demand)
• Runner-utilities try to register with it, when they start an OpenFOAM-run
  • The utilities also dregister their process when they exit gracefully

4.3.2 pyFoamNetShell.py

Connects to an OpenFOAM-run started by a runner-Utility using

• the hostname of the machine it runs on
• the port number (usually 18000, if more than one run is running one of the following ports)

and starts an interactive shell there. On the shells the XMLRPC-commands provided by the server can be used to

• get information about the run
  • environment
  • start-times
• manipulate the run
  • for instance stop() tells the run to write at the next time-step and then end
  • kill() brutally terminates the run

Help about the available commands is provided.

Can also be used to control the pyFoamMetaServer.py (which is usually contacted on port 17999)

4.4 Utilities for Manipulating case data

These utilities are used for manipulating case date. They are especially useful in scripts that automatically set up simulation runs.

4.4.1 pyFoamAddEmptyBoundary.py

Adds an empty patch to the boundary file. Such a patch is needed for some mesh-maipulation utilities.

4.4.2 pyFoamChangeBoundaryType.py

Changes the type of a boundary patch.

4.4.3 pyFoamClearCase.py

Removes all the timesteps except for the first from a case directory.

4.4.4 pyFoamPackCase.py

Packs the essential files (the ones that are needed to run it) of a case into a tar-file for archiving/mailing-purposes

4.4.5 pyFoamCloneCase.py

Creates a copy of a case with only the most essential files

4.5 Manipulating dictionaries (from scripts)

For more complex cases these utilities require an understanding of the syntax/semantics of Python-lists and dictionaries

4.5.1 pyFoamReadDictionary.py

Reads data from the root of a OpenFOAM-dictionary and prints it. To access subexpressions of a dictionary entry Python-expressions can be used (note: the expression is evaluated by the Python-interpreter, so quoting has to be used for strings, otherwise Python confuses them with variables)

4.5.2 pyFoamWriteDictionary.py

Writes data to the root of a OpenFOAM-dictionary and writes it to disk. Subexpressions can be selected (see note: pyFoamReadDictionary.py)

4.5.3 pyFoamFromTemplate.py

Generate a dictionary from a template file

4.5.4 pyFoamCompareDictionary.py

Compares two dictionaries structurally (not textually). Useful when the order of the entries or the formating have changed and diff won't give useful results

4.5.5 pyFoamUpdateDictionary.py

Updates a dictionary from another by adding/removing entries that are not in both.

4.6 Other

Utilities that don't fit any of the other categories.

4.6.1 pyFoamDecompose.py

• Generates a decomposeParDict according to the user specifications and
• runs dcomposePar

This utility is quite useful when writing scripts where the number of available processors is not known beforehand (for instance if you are using a queueing-system)

4.6.2 pyFoamComparator.py

A utilitiy that automatically does a series of variations on a run (for benchmarking, testing and verification purposes). The behaviour is specified by an XML-file.

Work in progress. Therefor no example is present and the DTD that assists editing the XML-file is not included. Contact me, if you want to give it a spin (and want the DTD)

4.6.3 pyFoamVersion.py

Prints the Version of OpenFOAM and pyFoam. Used for information purposes (when reporting bugs). Also lists the Foam-versions installed in the OpenFOAM-directory

4.6.4 pyFoamExecute.py

Execute a command specifying a OpenFOAM-Version. Useful for compiling/executing a debug-version without changing the shell

4.6.5 pyFoamDumpConfiguration.py

Dumps the values that can be changed via configuration files. The location of these configuration values are (lower overrides higer):

1. Hardcoded in the Library
2. Systemwide configuration file: /etc/pyfoamrc
3. User configuration file: ~/pyFoam/pyfoamrc

Parts of the output of the utilitiy can be pasted into the configuration files and modified to change the behaviour of PyFoam

4.7 GUI-Tools

Stuff that is not purly command-line oriented

4.7.1 pyFoamDisplayBlockMesh.py

Graphically displays a blockMesh and helps highlights selected blocks and patches. It is not a blockMesh-editor, but it helps to find errors in the order of vertices etc (editing of the blockMesh is still done in a text editor).

This needs an installation of the VTK with Python-bindings. The ways to get this are very different on various systems. But usually you have to compile it from the sources. Sorry.

5 Library Documentation

The tar-file contains the documentation of the Library as HTML (in the folder doc)

6 Rocks-Cluster support

There is library support to run OpenFOAM on a link Rocks cluster.

TODO

7 Technical notes/known problems

This is a list of the currently known problems. If you find any other bugs (also known as "surprising features") don't hesitate to contact the author

• the parsing mechanism for dictionaries seems to have problems with long turbulenceDicts. Currently there is no other way than remove all unneeded turulence-models from that file
• if a run-fails the error message might be "eaten" by the utility controlling it. I'm working on this
• the only MPI-environment I currently have available is OpenMPI. The pyFoam-Libraries used to work with LAM (still should, but I can not test it). If you have problems, please contact me (also "success-stories": "it works with MPIch" are most welcome)

8 Additional stuff

Since Version 0.2 the distribution contains the benchFoam.py-script (Since 0.2.4 the benchmark suite is stable enough to be used by the public). From version 0.3.0 on it can be used as pyFoamBench.py

9 Downloads

9.1 Current version (0.4.2)

• the tar file with the sources
• source rpm
• regular rpm

9.2 Ancient history

9.3 Previous version (0.4.1)

• the tar file with the sources
• source rpm
• regular rpm

9.3.1 Previous version (0.4.0)

There were no complaints about this version

• the tar file with the sources
• source rpm
• regular rpm

9.3.2 Very old versions

• Media:PyFoam-0.3.4.tar.gz

10 History

• 2005-08-15: Version 0.1

• 2006-01-13: Version 0.2

• 2006-01-16: Version 0.2.1

• 2006-01-16: Version 0.2.2

• 2006-01-23: Version 0.2.3

--help-option--help-option

• 2006-01-23: Version 0.2.4

• 2006-02-03: Version 0.2.5
  • Improves the stability of the benchmark-script for parallel runs
  • adds benchmakr suite for ancient versions of OpenFOAM

• 2006-02-03: Version 0.2.6
  • Minor changes to the benchmark-scripts (other cases than the tutorial cases can now be used for benching)

• 2006-02-23: Version 0.3.0
  • live plotting of residuals of solvers
  • benchmark script moved to bin-directory

• 2006-03-22: Version 0.3.1
  • BugFix: Plotting residuals in interFoam did not work
  • Extending the Resiudals-Plotting (Courant-Numbers etc)
  • Parsing of Dictionary-files into Python-Dictionaries (not yet fully tested)

• 2006-03-22: Version 0.3.2
  • Plotting resiudals now works with OpenFOAM 1.3

• 2006-10-18: Version 0.3.3
  • pyFoam distinguishes between the used MPI-implementations when starting parallel runs (currently LAM and OpenMPI)
  • BugFixes
  • added Utility pyFoamAddEmptyBoundary

• 2006-11-29: Version 0.3.4
  • An error pointed out by Andrea Rimondi: the last version was incomplete. Due to a code reorganisation and missing files in the distribution half of the scripts in bin stopped working, but apart from Andrea everyone was too polite to mention it
  • added Utility pyFoamClearCase.py

• 2007-03-29: Version 0.4.0
  • Reimplementation of the parsing of OpenFOAM-Dictionaries
    • Can be manipulated like regular Python-Data-Structures
  • Extension of the plotting-Utilities: arbitrary expressions are possible
    • The data can also be written to the disk
  • Runs that are started by pyFoam can be accessed and controlled over the network (using XMLRPC)
  • The runner-Utilities can start parallel-runs
    • A utility for generating a dictionary and decomposing a case is added
  • Bug removed that didn't let pyFoamSteadyRunner.py work properly with OpenFOAM 1.3 (due to changes in the solver output)

• 2007-08-29: Version 0.4.1
  • Debugging and stabilizing of the parsed OpenFOAM-Dictionaries
    • Changes in semantics: numbers are converted to numbers (not strings as before)
    • Can read and write gnuzipped files
    • More elegant access of the data
    • Order of entries in dictionaries is preserved, if possible
  • Gnuplot-library enclosed in distribution (to ease installation)
  • Enhancement of utilities
    • Runner-utilities can choose the Foam-version that should be used
    • Plot-Runner automaticaly loads regular expressions for a case
    • pyFoamBench now writes CSV-files which can be loaded into a spreadsheet-program
    • other stuff
  • New utilities
    • Creation of boundary conditions
    • Cloning and packing cases
    • VTK-based utility to display blockMeshes
    • Specialized runner for mesh utilities
  • Implementation of Unit-Tests for base classes to ensure stability
  • Other changes I can't remember

• 2007-11-24: Version 0.4.2
  • A framework to run jobs on a Rocks-cluster was implemented
  • Support for cases that have more than one mesh-region (in the library and the utilities)
  • Changes in the Library
    • the Application-classes have been changed, so that they can be called from other scripts
    • the OpenFOAM-version can be changed for the called applications
    • Extension of the Parser (Tensors etc)
    • Updated Parser Ply to newest version 2.3
    • A symbolic link controlDict.foam is automaticaly created (for the Paraview3.x-reader)
    • Gnuplot.py-library is adapted to use either numpy or Numeric (whatever is available)
    • mpirun can now have configurable parameters
  • Changes in Utilities
    • the Runner-Utilities now have a --restart-option
    • All Utilities that start an OpenFOAM-application now can select the OF-Version with a --foam-version-option
    • Possibility to temporarily lower the relaxation factors for steady runs
    • Runner-Utilities can report memory usage
    • Clone-Case is now securer
  • New Utilities
    • pyFoamDumpConfiguration.py: Dump the current configuration file for the PyFoam-installation
    • pyFoamExecute.py: Execute a command specifying a OpenFOAM-Version
    • pyFoamFromTemplate.py: Generate a dictionary from a template file
    • pyFoamUpdateDictionary.py: Updates a dictionary from another by adding/removing entries that are not in both
    • pyFoamCompareDictionary.py: Compares two dictionieries structurally (not textually)
    • pyFoamClusterTester.py: Test pyFoam-Scripts before submitting them to the cluster
  • Semantic changes
    • Counting of linear iterations now accumulates over various solutions during the same timestep (different numbers in Plots!!)
  • Removed Bugs: many

--Bgschaid 21:50, 15 Aug 2005 (CEST)