TODO for this page: Format code snippets, Clarify VisIt version compatibility

Alpaca: Remote Visualisation

Motivations

The ability to access simulation data in a fast and agile way is quintessential to debugging, but also a major challenge for large parallel codes. Quick-inspection strategies such as printing values to screen at runtime are not feasible when large volumes of data are involved, and more sophisticated visualization tools usually require writing to (at runtime) and reading from (at postprocessing) disk, both time-consuming operations that slow down the debugging process.

VisIt's Simulation Library

Introduction

In order to ease data inspection during debugging, the VisIt visualization package comes with the Simulation Library libsim; when libsim is linked against a simulation code, the corresponding executable acts as a VisIt engine, and as such is able to communicate with a VisIt viewer, exposing all of its data to VisIt's high-end visualization methods. In this way, the data can be examined on the fly, bypassing costly I/O steps; the Simulation Library also provides some basic execution control in order to pause, resume and single-step a simulation through its individual function calls.

In order to use the Simulation Library with Cactus, two thorns are needed:

  1. RuntimeVisIt: enables users to connect a VisIt viewer to a running Cactus simulation, which effectively acts as a VisIt engine, exposing its own data structures to the visualization package, which can then process them using its standard toolkit, just like it would process data from a file.
  2. VisIt: links the Simulation Library with the Cactus executable, providing the interface to the functions and structures needed by RuntimeVisIt.

Installation

Both RuntimeVisIt and VisIt can be checked out from the Alpaca SVN repository:

svn co --username cvs_anon https://svn.cct.lsu.edu/repos/cactus/Alpaca/RuntimeVisIt
svn co --username cvs_anon https://svn.cct.lsu.edu/repos/cactus/Alpaca/VisIt

There are two requirements for compiling these thorns with a Cactus code: first, an installation of VisIt is required (see VisIt's Download page for instructions -- only version 1.12.1 is tested to work correctly so far); second, the two configuration options VISIT_LIB_DIRS (pointing to the path of libsim.a and libsimf.a under the VisIt tree) and VISIT_INC_DIRS (pointing to the location of libsim's include files such as VisItControlInterface_V1.h and VisItDataInterface_V1.h) must be passed to Cactus during the configuration stage.

After compiling with these additional two thorns, the corresponding code will, during the initialization, write a file with extension *.sim1 under the .visit/simulations directory in the user's home directory. When this file is opened with VisIt, the simulation code pauses, and the simulation variables appear under VisIt's "Plot" menu, ready for processing (currently, a different variable will appear for each Cactus grid function on each map and refinement level; as libsim's AMR capabilities progress, higher-level handling of meshes will be included). Furthermore, information on the simulation, along with a few execution control tools, are available in the "Simulations..." window accessible from the "File" menu.

Notice that, just like for traditional data files, VisIt has the option of having a local viewer connect to a remote engine (in this case, the simulation), tunnelling the connection via SSH if the user so requests. It is therefore not necessary to visualize the simulation data on the same machine where the simulation is running (for information on setting up a remote host profile with VisIt, see the Remote Visualization section on Cactus' VisIt page).

Example

TODO: this should be specific to some machine. ranger?

Add snapshots

  1. An example thornlist, optionlist and parameter file for instrumenting a Cactus simulation with the VisIt's Simulation Library can be downloaded from *** (for what system?)

    simfactory?

  2. Compile a Cactus executable using: make libsim-config options=wave.cfg THORNLIST=wave.th make libsim
  3. Run the simulation with the wave.par parameter file;
  4. Open the *.sim1 file under ~/.visit/simulations; the simulation will output a "VisIt connected." message and pause, and the Plot menu will become active and show a list of the simulation variables;
  5. Go to File -> Simulations... to see the simulation control window;
Troubleshooting

VTK-based runtime visualization

Introduction

A functionality similar to that provided by VisIt's Simulation Library is provided by the Cactus thorn RuntimeVTK, which makes direct use of the VTK library to process and render arrays of data, writing the corresponding images into the simulation's output directory. The main advantage of this approach is its independence from external applications and corresponding ease of use.

RuntimeVTK uses the vtkHierarchicalBoxDataSet class, introduced with VTK 5.0, to organize the grid components of a typical AMR parallel run, and pass them through a vtkCompositeDataPipeline which will apply the requested visualization methods and render the resulting scene into an image written to the output directory. RuntimeVTK does not make decisions regarding the visualization details -- rather, it provides an API with a number of arguments definiting the plotting options.

Another technical point needs to be made when RuntimeVTK is run on remote systems, possibly through a batch submission mechanism. In this case, VTK has to be configured so that the rendering process and subsequent image creation do not go through the system's window manager and graphical infrastructure, but occur via software and off-screen. The Mesa library offers precisely this functionality, through its libOSMesa.so library and corresponding libGL.so.

Installation

RuntimeVTK can be checked out from the Alpaca SVN repository as well:

  svn co --username cvs_anon https://svn.cct.lsu.edu/repos/cactus/Alpaca/RuntimeVTK
  

Provide a VTK thorn in ExternalLibraries? Or describe where we have installed this.

In order to build Cactus with RuntimeVTK, the option VTK_DIR, pointing to a VTK installation, must be provided. The options VTK_LIBS, VTK_INC_DIRS and VTK_LIB_DIRS are also included for more flexibility.

When RuntimeVTK is include in a Cactus build, the aliased function:

  void FUNCTION VTKRender(CCTK_POINTER_TO_CONST IN cctkGH, CCTK_INT IN varIndex, \
                                           CCTK_REAL IN xmin, CCTK_REAL IN xmax, \
                                           CCTK_REAL IN ymin, CCTK_REAL IN ymax, \
                                           CCTK_REAL IN zmin, CCTK_REAL IN zmax)
  
becomes available to other thorns, which can call it as needed.

Example
Troubleshooting