VisIt Libsim Update DOE Computer Graphics Forum 2012 Brad Whitlock April 24, 2012 Brad Whitlock

Coupling of Simulations and VisIt Libsim is a library that simulations may use to let VisIt connect and access their data Simulation Libsim Front End Libsim Front End Runtime VisIt Source Filter Libsim is divided into front end and runtime library pieces to avoid large increases in the size of the simulation executable The front end library is a small, static, C library with mechanisms that let VisIt connect to the simulation Does not introduce C++ dependencies into simulation Simplifies automatic generation of other language bindings The runtime library loads when VisIt connects to the simulation Provides access to all of VisIt’s visualization and data analysis algorithms Lets one simulation executable benefit from newer VisIt server runtimes Adaptor Data Access Code Data

Libsim Update Custom UI for simulations Set cell/node origin and spatial extents in metadata Better support for ghost zones (I-blanking) Some support for static linking with VisIt runtime Libsim and VisIt can use simulation’s MPI communicator Integrating with codes from AWE and NASA Integrating with codes from CSCS CFD code on a Cray XE6 at 16K cores poof Blanked-out cells

Scaling the number of cores on Cray XE6 Time per iteration All MPI task had a 64x64x64 block to isocontour at 10 different thresholds Parallel I/O with MPI-IO Rendering without Ice-T # of cores File size (GB) GB/sec 2048 2.1 5.0 4096 4.2 4.8 8192 8.6 3.0 16384 19.3 2.56 The parallel IO part does not do any contouring. It simply does one iteration of the solver and dumps the data straight to disk. The file system is Lustre, 8Gb/s controllers IO timings are averages over 70-100 iterations. The machine was quite full. In-situ timing were done disabling Ice-T which did not scale well. This is why at 2048 cores, we don’t match the data of the next slide (also at 2048 cores), but with Ice-T The in-situ part was connecting the Cray to my desktop. Slide courtesy of Jean Favre

Benchmarking 10 isosurfaces at 2048 cores Time per iteration Each MPI task does 10 iso-contours + remote rendering Parallel I/O with MPI-IO # of cells / proc File size (GB) GB/sec 64x64x64 2.1 5.0 128x128x128 17.1 7.0 256x256x256 137 5.5 The parallel I/O part does not do any contouring. It simply does one iteration of the solver and dumps the data straight to disk. The file system is Lustre. All timings are averages over 70-100 iterations. The machine was quite full. The in-situ part was connecting the Cray to my desktop. Parallel I/O figures are the best I could do with the current hardware. The new scratch file is being installed at CSCS on May 9, 2012. Slide courtesy of Jean Favre

Work in Progress Changes to VisIt/Libsim allow in situ without VisIt client Additional functions in Libsim API to set up plots VisIt Client VisIt runtime Simulation output Libsim Front End Adaptor

VisIt Nek5000 Integration Initial linking of VisIt and the Nek5000 code Works in serial and parallel Formal adoption of changes into Nek5000 planned soon Image courtesy of Hank Childs and David Camp, Lawrence Berkeley National Laboratory

Sequoia Will Drive Libsim Improvements Sequoia is being built right now at LLNL ASC codes are seeking to run at 100K’s of cores Increased role for in situ Alternate libsim implementation Reuse libsim front end and adaptor lightweight plotting back end, maybe custom to the simulation Static linking VisIt runtime Simulation Libsim Front End Adaptor Statically linked Dynamically loaded output Sequoia is driving our code teams to reduce communication and make structures more distributed. This has some consequences for VisIt since VisIt must then use the distributed structures (e.g. domain connectivity) At highest scales, I/O will dog performance so code teams more interested in using in situ. Our in situ approach will likely be too heavyweight at highest scales due to memory use, etc. Fortunately, libsim is small and we could conceivably replace back end with custom plotting routines geared towards the sim’s specific needs. Then, the custom plotting and libsim could be statically linked to the simulation. Since libsim’s API has no VTK, the custom plotting back end could be very lightweight and possibly not even based on VTK. Simulation Libsim Front End Adaptor Lightweight Plotting output Statically linked