Spheral++ An Open Source Project to Simulate Hydrodynamics in Astrophysical Systems. J. Michael Owen mikeowen@llnl.gov Lawrence Livermore National Laboratory O’Reilly Open Source Convention Python Track July 25, 2001

What is Spheral++? This project is developing an open source code for two purposes: To model the important physical processes required to study the formation and evolution of galaxies in a cosmological scenario (initially gravitation, hydrodynamics, and simple radiative processes). To provide a useful framework for developing novel new “particle” based hydrodynamic techniques.

Why Open Source? Typically scientific software is custom built, and not shared! This leads to disposable software, which is used some number of times by the author(s), and then discarded. For complex numerical simulation codes, this approach leads to massive duplication of effort. Some redundant effort is useful for comparison. “If I gave away my [numerical modeling] code, then someone would just abuse it.”

Why Open Source? There are many potential benefits to a shared resource for hydrodynamic modeling: These are complex, expensive programs, so we can reduce wasted effort by sharing. Peer review becomes possible. Once such a tool is well characterized, everyone can understand the results (and their shortcomings) that much more clearly. If properly designed, such a tool can easily be extended by other researchers. Not everyone wants to write their own!

Requirements In order to produce scientifically useful results, we minimally require: Model gravitation: N-body particle techniques are well characterized. Model hydrodynamics: Particle based hydro’s (such as smoothed particle hydrodynamics -- SPH) robustly model adiabatic and shock physics in a lagrangian manner, allowing the resolution scales to follow the mass. Adequate resolution: 2563 or greater numbers of particles, must be massively parallel!

Software Requirements Don’t assume what the user wants to do! Provide a flexible set of tools that can be used to build and analyze simulations. It must be possible to implement new algorithms and ideas in the framework without impacting (or even requiring an in depth understanding of) the rest of the code. Spheral++ should be as modular as possible, with components interacting through a simple set of interfaces.

Scripting Interface! We can meet the user flexibility requirements by using a scripting interface for the code. The code consists of building blocks adhering to a rigorous set of interfaces. The user is allowed to dynamically set up the problem, choose what set of physics modules (and their options) should be used, run it however they wish, and then analyze the results from the scripting language. This is like a Matlab for fluid simulations.

C++ is the implementation language. We use C++ to write the bulk of the code. Object oriented design (simple to specify and enforce interfaces). Templates and generic programming allow flexibility without sacrificing computational efficiency. Astrophysical hydrodynamic models are generally computationally bound, even on the fastest machines in the world!

Python Is the Interface We choose to use python as the front-end scripting interface for Spheral++. Python uses a relatively simple syntax that is easily learned. Python provides an object oriented programming model that fits well with C++ (our underlying implementation language). There are already many useful extensions to python (graphics and analysis packages) which we can directly leverage for use with Spheral++ simulations.

Exposing C++ to Python: Pyffle Until recently Spheral++ used pyffle (developed by Pat Miller at LLNL) to semi-automate “wrapping” C++ objects with a python interface. Released under open source license. Contained in the Spheral++ CVS distribution. Big plus, it works! Some shortcomings to pyffle Tiresome wrapping templated objects. No support for mathematical operations. No support for overriding virtual functions from python.

Boost Python Library (BPL) The Boost project (www.boost.org) provides a simple mechanism for wrapping and accessing C++ objects from python via the Boost Python Library (BPL). Easy to wrap templated objects. Trivial to expose mathematical operations with wrapped types (+, -, *, ==, etc.) Provides methods to override virtual functions from python. Makes it possible to rapidly prototype new ideas in python, before committing to compiled code.

Problems with BPL A few issues have surfaced trying to use BPL with Spheral++. Lack of cross module support for abstract interface classes Cross module inheritance (doesn’t work!) Accessing pointers to abstract classes (particularly STL vectors of them) These issues have been reported to the BPL developers.

Example: Hydrodynamic Test The LeBlanc shock tube. These results were generated using a Spheral python script to set up, run, and then plot the results against the analytic solution.

Example: Prototyping a New Idea We want to experiment with a new artificial viscosity. BPL lets us override C++ virtual functions from python. from ArtificialViscosity import ArtificialViscosity class myQ(ArtificialViscosity): def __init__(self, Cl, Cq): ArtificialViscosity.__init__(Cl, Cq) … def viscousPressure(self, …): myPressure = … return myPressure

Current Status The major modules for running hydrodynamic problems are in place. Several standard 1-D hydro test problems have already been constructed and run. Results are available in Spheral++ distribution. Transitioning to BPL from pyffle. A few shortcomings in BPL have been exposed during this process.

Future Plans Profiling and optimization Implement distributed parallelization (underway) Implement more physics modules Self-gravitation Radiative cooling Improve build system (autoconf?) Explore more visualization and analysis tools MayaVI VTK

Resources www.Spheral.sf.net (Spheral++ home page) www.boost.org (BPL and other useful stuff) www.gnuplot.org (Plotting package) Developers: Pat Miller Jeff Johnson Martin Casado Noel Keen Mike Fisher Acknowledgement: This work was supported under the auspices of the US DOE by LLNL under contract W-7405-Eng-48.