Extend-testHarness

1 A guide to the foam-extend test harness

This file is a guide to the foam-extend test harness.

The test harness from foam-extend version 3.2 was used as a reference implementation for this documentation.

The original version of this file will be located at $WM_PROJECT_DIR/doc/testHarness/testHarness.org.

A plain text version of this file will be located at $WM_PROJECT_DIR/doc/testHarness/testHarness.txt.

A MediaWiki version of this file is located at https://openfoamwiki.net/index.php/Extend-testHarness

In case any converted versions (text, MediaWiki, etc.) of this file differ from the original testHarness.org file, the latter should always be considered the reference version. But we do appreciate your feedback and comments for improving this text.

For instance, if you ever decide to improve the Mediawiki version only, please consider contributing your changes back to the testHarness.org file as well. Otherwise, your Mediawiki changes might get overwritten by an updated version of the .org reference file.

Note: The original testHarness.org file will be published shortly. This is a work in progress

1.1 Introduction

The foam-extend test harness scripts are based mainly on Kitware CMake/CTest utilities http://www.cmake.org/.

In its current version, the test harness is designed for running the full suite of foam-extend tutorials, and report the fail/success results to a CDash Web service one can consult using a simple browser.

The CDash Web service for foam-extend 3.2 is available here: http://foam-extend.sourceforge.net/CDash/index.php

For a quick introduction of CMake/CTest, see here: http://www.cmake.org/Wiki/CMake/Testing_With_CTest

For a more complete introduction to CMake and CTest, one can also refer to the Kitware book `Mastering CMake: A Cross-Platform Build System` http://www.kitware.com/products/books.php

For more information on Kitware CDash, see here: http://www.cdash.org/

The rest of this document will provide suficient information so you can run the foam-extend test harness on your own.

1.2 Files location and a short description

The foam-extend test harness is currently built from the following files:

$WM_PROJECT_DIR/CMakeLists.txt

This is the main CMake file driving the configuration of the test harness. When processed though the cmake utility, all the necessary Makefiles and CTest scripts will get generated.

$WM_PROJECT_DIR/CTestConfig.cmake

This file contains the various variables and definitions necessary to publish your test harness results over the CDash Web service for foam-extend 3.2.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/CMakeLists.txt

This file is a copy of the main $WM_PROJECT_DIR/CMakeLists.txt file.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/CTestConfig.cmake.foam-extend

This file is a copy of the main $WM_PROJECT_DIR/CTestConfig.cmake file.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/FOAM_Tutorials.cmake

This file will configure a copy the foam-extend tutorials for running under the test harness. The copy will be made under $WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allclean

Cleanup script. Will remove all the automatically generated files and directories for the test harness. Should be called prior to running the Allrun_ scripts.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_CMakeOnly

This script will generate all the necessary files for running the test harness on the foam-extend tutorials. One can then invoke `make help` to see all the available make options.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_Experimental

Same script as `Allrun_CMakeOnly`, but will also call `make Experimental`.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_Nightly

Same script as `Allrun_CMakeOnly`, but will also call `make Nightly`.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/AdditionalRunFunctions

Additional `bash` macros for the tutorial Allrun files.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/Allrun.default

Default Allrun script for the tutorial when none are provided.

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/addMissingAllrunFileToTutorial.sh

Bash script for adding a default Allrun file to the tutorials that do not have one. The test harness only run tutorials with an existing Allrun file

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/prepareCasesForOneTimeStep.sh

This script will modify the test cases system/controlDict in order for the case to run for only 1 time step

$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/prepareCasesForTestHarness.sh

This script will modify the test cases Allrun file so it can run properly under the test harness.

1.3 Configuring your $WM_PROJECT_DIR/etc/prefs.sh file for the test harness

The following environment variables are used for configuring the test harness. You should use your $WM_PROJECT_DIR/etc/prefs.sh file to initialize those variables.

CDASH_SUBMIT_LOCAL_HOST_ID

System identifier for the FOAM CDash test harness on foam-extend. By default, your system FQN/hostname will be used as the system identifier when publishing your test harness results on the FOAM CDash server on foam-extend. You can override your identifier using this environment variable.

CDASH_SCM_INFO

Buildname suffix for the FOAM CDash test harness on foam-extend. By default, the git branch name and git revision number will be appended to the CDash build name. Otherwise, for users not using git, or wanting to provide additionnal information, simply initialize the CDASH_SCM_INFO with the proper information.

WM_NCOMPPROCS

Specify the number of cores to use for the parallel execution of the test harness.

FOAM_TUTORIALS

Directory where the original test cases are located. For foam-extend, this would be by default $WM_PROJECT_DIR/tutorials.

1.4 The main dashboards : Experimental, Nightly and Continuous

The result of a test run, reformatted for easy review, is called a `dashboard`. A dashboard can be submitted to a central server, like CDash. Once properly configured, the test harness will offer 3 main dashboards.

Experimental

Will test the current state of the project. An experimental submission can be performed at any time, usually interactively from the current working copy of a developer.

Nightly

Is similar to Experimental, except that the source tree will be set to the state it was in at a specific nightly time. This ensures that all `nightly` submissions correspond to the state of the project at the same point in time. `Nightly` builds are usually done automatically at a preset time of day. Nightly build will also update your source code to the latest available revision. So it is best not to run a Nightly dashboard on source code that is not yet committed.

Continuous

Means that the source tree is updated to the latest revision, and a build / test cycle is performed only if any files were actually updated. Like `Nightly` builds, `Continuous` ones are usually done automatically and repeatedly in intervals.

There are also `intermediary dashboards` that allow you to select a specific test harness intermediary step. The command `make help` will show you that list:

... ContinuousBuild
... ContinuousConfigure
... ContinuousCoverage
... ContinuousMemCheck
... ContinuousStart
... ContinuousSubmit
... ContinuousTest
... ContinuousUpdate
... ExperimentalBuild
... ExperimentalConfigure
... ExperimentalCoverage
... ExperimentalMemCheck
... ExperimentalStart
... ExperimentalSubmit
... ExperimentalTest
... ExperimentalUpdate
... NightlyBuild
... NightlyConfigure
... NightlyCoverage
... NightlyMemCheck
... NightlyMemoryCheck
... NightlyStart
... NightlySubmit
... NightlyTest
... NightlyUpdate

Note

For foam-extend, the MemCheck and Coverage dashboards are not supported.

1.5 Running the test harness

Running the test harness is pretty simple:

These commands will configure and run the `Experimental` version of the test harness:

   cd $WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir
   ./Allclean
   ./Allrun_Experimental

To run the `Nightly` version of the test harness:

   cd $WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir
   ./Allclean
   ./Allrun_Nightly

To see the full range of available options, run the command:

make help

1.6 Selecting a subset of test cases to run

Instead of using the `make` command to run the test harness, one can also use command `ctest`. The commmand `ctest` offers additionnal options to select or limit the number of tests to run.

The following command will provide all the available options for ctest:

ctest -N

Here is a list of useful ctest options:

ctest -N

this command will list the available tests by their name and number. By default, all the tests are run in succession following the numerical order shown.

ctest -R <regex>

run all the tests whose name are matching the supplied regular expression. For instance, to run all the tests related to cfMesh, one can use the following command: `ctest -R cfMesh`

ctest -E <regex>

run all the tests, but exclude the ones whose name is matching the supplied regular expression. For instance, to run all the tests except those for cfMesh, one can use the following command: `ctest -E cfMesh`

ctest -I [Start,End,Stride,test#,test#]

run individual tests by number. `ctest -I 3,5` will run test 3, 4 and 5. `ctest -I 4,4,,4,7,13` will run tests 4, 7 and 13.

ctest -D dashboard

run a specific dashboard test. For example, the command `make Experimental` can be replaced by the following suite of ctest commands:

ctest -D ExperimentalStart
ctest -D ExperimentalConfigure
ctest -D ExperimentalBuild
ctest -D ExperimentalTest
ctest -D ExperimentalSubmit

Here is a more complete example where we will configure, build, test and submit the test harness results, but only for the incompressible tutorials:

ctest -D Experimental -R incompressible

1.7 Browsing the CDash service

The results of the test harness run will be published on the CDash dashboard on foam-extend.

To see your results: URL : http://foam-extend.sourceforge.net/CDash/index.php?project=foam-extend-3.2

On this interactive Web site, one can then point and click various buttons and menus to explore the various reports uploaded from your test harness runs.

1.8 New features for the test harness (foam-extend 3.2):

Running the test harness in parallel

It is now possible to run the test harness in parallel over a single node or computer. The environment variable WM_NCOMPPROCS will specify the number of cores to use for running the test harness. For `n` cores specified, `n` tutorials will be running in parallel on your computer. Since all the tests will still run on the same computer, make sure you have enough ressources to run `n` tutorials in parallel. Depending on the number of cores available, one might have to tweak some of the shell `limits` values. The command `ulimit -a` will show you the actual limits values imposed on your shell. Some limit values like `open files` (ulimit -n) or `max user processes` (ulimit -u) might need to be adjusted to some higher values. In doubt, consult with your local sysadmin.

1.9 Notes

The MediaWiki version of this file was generated using the following command:

   pandoc testHarness.org -s -S -f org -t mediawiki -o testHarness.mediawiki

The ASCII version of this file was generated using the following command:

   pandoc testHarness.org -N -s -S -f org -t asciidoc -o testHarness.txt