Chapter B2

B2.1 Configuration Options

There are four ways to pass options to the configuration process.

  1. Pass options individually in shell environment variables:

    export <option name>=<chosen value> # for bash
    setenv <option name> <chosen value> # for (t)csh
    gmake <configuration name>-config

  2. Either: create a default configuration file ${HOME}/.cactus/config.

    All available configuration options may be set in a default options file ${HOME}/.cactus/config, any option which are not set will take a default value. The file should contain lines of the form:

    <option> [=] ...

    The equals sign is optional. Spaces are allowed everywhere. Text starting wit a ’#’ character will be ignored as a comment.

  3. Or: list your Cactus configuration files in an environment variable CACTUS_CONFIG_FILES:

    gmake <config name>-config CACTUS_CONFIG_FILES=<list of config files>

    Multiple configuration files, with their file names separated by a ’:’ character, will be processed in order. Each file should be given by its full path. The options file has the same format as ${HOME}/.cactus/config.

  4. Add the options to a configuration file and use,

    gmake <config name>-config options=<filename>

    The options file has the same format as ${HOME}/.cactus/config. (Note that these options are added to those from the ${HOME}/.cactus/config file.)

  5. Pass the options individually on the command line,

    gmake <config name>-config <option name>=<chosen value>, ...

    Not all configuration options can be set on the command line. Those that can be set are indicated in the table below.

The options are listed here in order of increasing precedence, e.g. options set on the command line will take priority over (potentially conflicting) options set in ${HOME}/.cactus/config or other Cactus configuration files. Default options from ${HOME}/.cactus/config will only be read if the environment variable CACTUS_CONFIG_FILES is not set.

It is important to note that these methods cannot be used to, for example, add options to the default values for CFLAGS. Setting any variable in the configuration file or the command line will overwrite completely the default values.

B2.1.1 Available Options

There is a plethora of available options.

B2.2 Compiling with Extra Packages

B2.2.1 MPI: Message Passing Interface

The Message Passing Interface (MPI) provides inter-processor communication. It can either be implemented natively on a machine (this is usual on most supercomputers), or through a standard package such as MPICH, LAM, WMPI, or PACX.

To compile with MPI, the configure option is


where <MPI_TYPE> can take the values (entries followed by * may be specified on the configuration command line):


For a custom MPI configuration set the variables


* libraries.


* library directories.


* include file directories.


Use the native MPI for this machine, as indicated in the known-architectures directory (lib/make/known-architectures).


Use MPICH ( This is controlled by the options


* machine architecture.


* directory in which MPICH is installed. If this option is not defined, it will be searched for.


* the device used by MPICH. If not defined, the configuration process will search for this in a few defined places. Supported devices are currently ch_p4, ch_shmem, globus and myrinet. For versions of MPICH prior to 1.2.0, the devices are searched for in this order, for 1.2.0 you may need to specify MPICH_DEVICE, depending on the installation.

If MPICH_DEVICE is chosen to be globus (, an additional variable must be set


* directory in which Globus is installed.

The Globus flavor may be chosen optionally


* Globus flavor to build Cactus with.

If it is not set, the first Globus flavor found will be used.

If MPICH_DEVICE is chosen to be ch_gm, (, an additional variable must be set


* directory in which Myrinet libraries are installed.


Use LAM (Local Area Multicomputer, This is controlled by the variables


* directory in which LAM is installed. This will be searched for in a few provided places if not given.

If the LAM installation splits libraries and include files into different directories, instead of setting LAM_DIR set the two variables


* directory in which LAM libraries are installed.


* directory in which LAM include files are installed.


Use WMPI (Win32 Message Passing Interface, This is controlled by the variable


* directory in which WMPI is installed.


Use HPVM (High Performance Virtual Machine, ( This is controlled by the variable


* directory in which HPVM is installed.


Use MPIPro (


Use the PACX Metacomputing package (PArallel Computer eXtension, This is controlled by the variables


* directory in which PACX is installed. If this option is not defined, it will be searched for.


* the MPI package PACX uses node-local communication. This can be any of the above MPI packages.

Note that the searches for libraries, etc. mentioned above use the locations given in the files in lib/make/extras/MPI.

B2.2.2 HDF5: Hierarchical Data Format version 5

To compile with HDF5 (, include thorn ExternalLibraries/HDF5 in your thornlist, and use the configure option

HDF5_DIR = BUILD/<dir> [LIBZ_DIR = <dir>] [LIBSZ_DIR = <dir>]

If HDF5_DIR is not given, the configuration process will search for an installed HDF5 package in some standard places. If HDF5_DIR is set to BUILD an HDF5 installation will be build. If the found HDF5 library was built with the external deflate I/O filter, the configuration process also searches for the libz library and adds it to the linker flags. You may also point directly to the location of libz.a by setting LIBZ_DIR. If the found HDF5 library was built with the external szlib I/O filter, the configuration process also searches for the szlib library and adds it to the linker flags. You may also point directly to the location of libsz.a by setting LIBSZ_DIR. Note that the option HDF5 = yes/no is depreciated and does not work with thorn ExternalLibraries/HDF5.

B2.2.3 LAPACK: Linear Algebra PACKage

To compile with LAPACK (, include thorn ExternalLibraries/LAPACK in your thornlist, and use the configure option

[ LAPACK_DIR = BUILD | <dir> ]
[ LAPACK_LIBS = <libs> ]
[ LAPACK_EXTRA_LIBS = <libs> ]

If LAPACK_DIR is not given, the configuration process will search for a LAPACK library liblapack.[{a,so}] in some standard places. If LAPACK_DIR is set to BUILD, a Lapack installation will be build.

B2.2.4 PETSc: Portable, Extensible Toolkit for Scientific Computation

To compile with PETSc (, the configure options are

PETSC = yes | no | <blank>
[ PETSC_DIR = <dir> ]
[ PETSC_ARCH = <architecture> ]
[ PETSC_ARCH_LIBS = <architecture-specific libraries> ]

If PETSC_DIR is not given, the configuration process will search for an installed PETSc package in some standard places (defined in lib/make/extras/PETSC). If PETSC_ARCH is not given, the configuration process will choose the first PETSc configuration found in $PETSC_DIR/lib/libO/. If PETSC_ARCH_LIBS is not given, the configuration process will choose architecture-specific libraries, as required by a PETSc configuration (usually PETSc needs the LAPACK library).

B2.2.5 Pthreads: POSIX threads

To enable multithreading support within Cactus using POSIX threads, the configure option is


The configuration process will check if a reentrant C library is available, and adds it to the linker flags. It will also search for the system’s Pthreads library (either libpthread or libpthreads), and set preprocessor defines necessary for compiling multithreaded code.

B2.3 File Layout

The configuration process sets up various subdirectories and files in the configs directory to contain the configuration specific files; these are placed in a directory with the name of the configuration.


contains the files created by the configure script:

The most important ones are


contains compilers and compilation flags for a configuration.


contains details about extra packages used in the configuration.


The main configuration header file, containing architecture specific definitions.


An architecture specific header file containing things which cannot be automatically detected, and have thus been hand-coded for this architecture.

These are the first files which should be checked or modified to suit any peculiarities of this configuration.

In addition, the following files may be informative:

A Perl script used to determine how the Fortran compiler names subroutines. This is used to make some C routines callable from Fortran, and Fortran routines callable from C.


Initially empty. It can be edited to add extra architecture specific dependencies needed to generate the executable.


The make rules for generating object files from source files.

Finally, autoconf generates the following files.


A log of the autoconf process.


A script which may be used to regenerate the configuration.


An internal file used by autoconf.


An empty directory which will contain the libraries created for each thorn.


An empty directory which will contain the object files generated for this configuration, and preprocessed source files.


A file containing information about the configuration (including the options used to configure the configuration).


A directory which contains all the files generated by the CST from the .ccl files.


A scratch directory which is used to accommodate Fortran 90 modules.

B2.4 Building and Administering a Configuration

Once you have created a new configuration, the command

gmake <configuration name>

will build an executable, prompting you along the way for the thorns which should be included. There is a range of gmake targets and options which are detailed in the following sections.

B2.4.1 gmake Targets for Building and Administering Configurations

A target for gmake can be naively thought of as an argument that tells it which of several things listed in the Makefile it is to do. The command gmake help lists all gmake targets:

gmake <config>

builds a configuration. If the configuration doesn’t exist, it will create it.

gmake <config>-clean

removes all object and dependency files from a configuration.

gmake <config>-cleandeps

removes all dependency files from a configuration.

gmake <config>-cleanobjs

removes all object files from a configuration.

gmake <config>-config

creates a new configuration or reconfigures an existing one overwriting any previous configuration options.
The configuration options are stored in a file configs/<config>/config-info.

gmake <config>-configinfo

displays the options of the configuration (cat configs/<config>/config-info).

gmake <config>-delete

deletes a configuration (rm -r configs/<config>).

gmake <config>-editthorns

edits the ThornList.

gmake <config>-examples

copies all the example parameter files relevant for this configuration to the directory examples in the Cactus home directory. If a file of the same name is already there, it will not overwrite it.

gmake <config>-realclean

removes from a configuration all object and dependency files, as well as files generated from the CST (stands for Cactus Specification Tool, which is the set of Perl scripts which parse the thorn configuration files). Only the files generated by configure and the ThornList file remain.

gmake <config>-rebuild

rebuilds a configuration (reruns the CST).

gmake <config>-reconfig

reconfigures an existing configuration using its previous configuration options from the file configs/<config>/config-info.

gmake <config>--testsuite

runs the test programs associated with each thorn in the configuration. See section B2.6 for information about the test suite mechanism.

gmake <config>-ThornGuide

builds documentation for the thorns in this configuration (see section B2.5, page B32, for other targets to build documentation for thorns).

gmake <config>-thornlist

regenerates the ThornList for a configuration.

gmake <config>-utils [UTILS=<list>]

builds all utility programs provided by the thorns of a configuration. Individual utilities can be selected by giving their names (ie. name of the source file without extension) in the UTILS variable.

B2.4.2 Compiling in Thorns

Cactus will try to compile all thorns listed in configs/<config>/ThornList. The ThornList file is simply a list of the form <arrangement>/<thorn>. All text after a pound sign ‘#’ or exclamation mark ‘!’ on a line is treated as a comment and ignored. If you did not specify a ThornList already, the first time that you compile a configuration you will be shown a list of all the thorns in your arrangement directory, and asked if you with to edit them. You can regenerate this list at anytime by typing

gmake <config>-thornlist

or you can edit it using

gmake <config>-editthorns

Instead of using the editor to specify the thorns you want to have compiled, you can edit the ThornList outside the make process. It is located in configs/<config>/ThornList, where <config> refers to the name of your configuration. The directory ./configs exists after the very first make phase for the first configuration.

B2.4.3 Notes and Caveats

B2.4.4 gmake Options for building configurations

An option for gmake can be thought of as an argument which tells it how it should make a target. Note that the final result is always the same.

gmake <target> PROMPT=no

turns off all prompts from the make system.

gmake <target> SILENT=no

prints the commands that gmake is executing.

gmake <target> WARN=yes

shows compiler warnings during compilation.

gmake <target> FJOBS=<number>

compiles in parallel, across files within each thorn.

gmake <target> TJOBS=<number>

compiles in parallel, across thorns.

Note that with more modern versions of gmake, it is sufficient to pass the normal -j <number> flag to gmake to get parallel compilation.

B2.5 Other gmake Targets

gmake help

lists all make options.

gmake checkout

allows you to easily checkout Cactus arrangements and thorns. For example, it can checkout all the thorns in any thornlist file found in the thornlists subdirectory of the Cactus root directory.

gmake configinfo

prints configuration options for every configuration found in user’s configs subdirectory.

gmake default

creates a new configuration with a default name.

gmake distclean

deletes your configs directory, and hence all your configurations.

gmake downsize

removes non-essential files as documents and test suites to allow for minimal installation size.

gmake newthorn

creates a new thorn, prompting for the necessary information and creating template files.

gmake TAGS

creates an Emacs style TAGS file. See section D8 for using tags within Cactus.

gmake tags

creates a vi style tags file. See section D8 for using tags within Cactus.

Targets to generate Cactus documentation:

gmake <arrangement>-ArrangementDoc

builds the documentation for the arrangement.

gmake ArrangementDoc

builds the documentation for all arrangements.

gmake MaintGuide

runs LaTeX to produce a copy of the Maintainers’ Guide.

gmake ReferenceManual

runs LaTeX to produce a copy of the Reference Manual.

gmake <thorn>-ThornDoc

builds the documentation for the thorn.

gmake ThornDoc

builds the documentation for all thorns.

gmake ThornGuide

runs LaTeX to produce a copy of the Thorn Guide, for all the thorns in the arrangements directory.

gmake UsersGuide

runs LaTeX to produce a copy of the Users’ Guide.

gmake AllDoc

creates all of the above documentations.

B2.6 Testing

Some thorns come with a test suite, consisting of example parameter files and the output files generated by running these. To run the test suite for the thorns you have compiled use

gmake <configuration>--testsuite

These test suite serve the dual purpose of

Regression testing

i.e. making sure that changes to the thorn or the flesh don’t affect the output from a known parameter file.

Portability testing

i.e. checking that the results are independent of the architecture—this is also of use when trying to get Cactus to work on a new architecture.