1. VisIt Libsim Update DOE Computer Graphics Forum 2012 Brad Whitlock
April 24, 2012
Brad Whitlock

2. Coupling of Simulations and VisIt
Libsim is a library that simulations may use to let VisIt connect and access their data
Simulation
Libsim
Front End
Libsim Front End
Runtime
VisIt
Source
Filter
Libsim is divided into front end and runtime library pieces to avoid large increases in the size of the simulation executable
The front end library is a small, static, C library with mechanisms that let VisIt connect to the simulation
Does not introduce C++ dependencies into simulation
Simplifies automatic generation of other language bindings
The runtime library loads when VisIt connects to the simulation
Provides access to all of VisIt’s visualization and data analysis algorithms
Lets one simulation executable benefit from newer VisIt server runtimes
Adaptor
Data Access Code
Data

3. Libsim Update Custom UI for simulations
Set cell/node origin and spatial extents in metadata
Better support for ghost zones (I-blanking)
Some support for static linking with VisIt runtime
Libsim and VisIt can use simulation’s MPI communicator
Integrating with codes from AWE and NASA
Integrating with codes from CSCS
CFD code on a Cray XE6 at 16K cores
poof
Blanked-out cells

4. Scaling the number of cores on Cray XE6
Time per iteration
All MPI task had a 64x64x64 block to isocontour at 10 different thresholds
Parallel I/O with MPI-IO
Rendering without Ice-T
# of cores
File size (GB)
GB/sec
2048
2.1
5.0
4096
4.2
4.8
8192
8.6
3.0
16384
19.3
2.56
The parallel IO part does not do any contouring. It simply does one iteration of the solver and dumps the data straight to disk.
The file system is Lustre, 8Gb/s controllers
IO timings are averages over iterations. The machine was quite full.
In-situ timing were done disabling Ice-T which did not scale well. This is why at 2048 cores, we don’t match the data of the next slide (also at 2048 cores), but with Ice-T
The in-situ part was connecting the Cray to my desktop.
Slide courtesy of Jean Favre

5. Benchmarking 10 isosurfaces at 2048 cores
Time per iteration
Each MPI task does 10 iso-contours + remote rendering
Parallel I/O with MPI-IO
# of cells / proc
File size (GB)
GB/sec
64x64x64
2.1
5.0
128x128x128
17.1
7.0
256x256x256
137
5.5
The parallel I/O part does not do any contouring. It simply does one iteration of the solver and dumps the data straight to disk.
The file system is Lustre.
All timings are averages over iterations. The machine was quite full.
The in-situ part was connecting the Cray to my desktop.
Parallel I/O figures are the best I could do with the current hardware. The new scratch file is being installed at CSCS on May 9, 2012.
Slide courtesy of Jean Favre

6. Work in Progress
Changes to VisIt/Libsim allow in situ without VisIt client
Additional functions in Libsim API to set up plots
VisIt Client
VisIt runtime
Simulation
output
Libsim Front End
Adaptor

7. VisIt Nek5000 Integration
Initial linking of VisIt and the Nek5000 code
Works in serial and parallel
Formal adoption of changes into Nek planned soon
Image courtesy of Hank Childs and David Camp, Lawrence Berkeley National Laboratory

8. Sequoia Will Drive Libsim Improvements
Sequoia is being built right now at LLNL
ASC codes are seeking to run at 100K’s of cores
Increased role for in situ
Alternate libsim implementation
Reuse libsim front end and adaptor
lightweight plotting back end, maybe custom to the simulation
Static linking
VisIt runtime
Simulation
Libsim Front End
Adaptor
Statically linked
Dynamically loaded
output
Sequoia is driving our code teams to reduce communication and make structures more distributed.
This has some consequences for VisIt since VisIt must then use the distributed structures (e.g. domain connectivity)
At highest scales, I/O will dog performance so code teams more interested in using in situ. Our in situ approach will likely be too heavyweight at highest scales due to memory use, etc. Fortunately, libsim is small and we could conceivably replace back end with custom plotting routines geared towards the sim’s specific needs. Then, the custom plotting and libsim could be statically linked to the simulation. Since libsim’s API has no VTK, the custom plotting back end could be very lightweight and possibly not even based on VTK.
Simulation
Libsim Front End
Adaptor
Lightweight Plotting
output
Statically linked