All OpenFOAM applications do parallel communication through the Pstream library only so using any underlying message passing library (MPI, GAMMA) is just a matter of replacing Pstream. One can have multiple versions and just change the LD_LIBRARY_PATH to pick up the wanted one.
- in your ~/.OpenFOAM-1.XXX/bashrc define (or uncomment) WM_MPLIB to be the name of your message passing library. E.g.
export WM_MPLIB=MPIGAMMA
- in the central OpenFOAM/OpenFOAM-1.XXX/.bashrc set all the additional settings for your WM_MPLIB. The only non-optional setting is
FOAM_MPI_LIBBIN
which is the place where the Pstream library will be put.
- in your rules directory ($WM_DIR/rules/$WM_ARCH) make a mplib${WM_MPLIB} file. In my example: mplibMPIGAMMA. Usually easiest to start from e.g. the OpenMPI one.
- go to $FOAM_SRC/Pstream
./Allwmake
- this will have made a $FOAM_MPI_LIBBIN/libPstream.so
That is all. Now all of OpenFOAM will use your communication library.
If you are investigating higher performance replacements for e.g. OpenMPI,LAM, have a look at Open-MX http://open-mx.gforge.inria.fr/) or MPI-GAMMA (http://www.disi.unige.it/project/gamma/index.html).
Pstream for HPMPI
Linking the proprietary HP-MPI is slightly more tricky than the standard method described above.
In addition to the steps outlined, a symbol needs to be pre-defined to prevent the inclusion of C++ bindings.
After modifying the OpenFOAM login files, copy mplibOPENMPI to mplibHPMPI in $WM_DIR/rules/$WM_ARCH$WM_COMPILER and edit it as follows:
PFLAGS = -D_MPICC_H PINC = -I$(HPMPI_PATH)/include PLIBS = -L$(HPMPI_PATH)/lib/linux_amd64 -lmpi
Pre-defining the "_MPICC_H" symbol causes the linker to skip out the section of mpiCC.h that normally causes problems. Why HP have not deigned to include an explicit switch to disable C++ bindings is unclear.
Follow the rest of the steps normally.
