Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Immersed Boundary Method Simulation in Titanium

Objectives Provide easy-to-use, high-performance tool for simulation of fluid flow in biological systems. Demonstrate the Titanium compiler and language. Allow heart simulation on large-scale parallel machines.

Outline Immersed Boundary Method Titanium Immersed Boundary Method in Titanium

Immersed Boundary Method Developed at New York University by Peskin & McQueen to model biological systems where elastic fibers are immersed in an incompressible fluid. –Mammalian heart, blood platelets, sea urchin embryos Fibers (e.g., heart muscles) modeled by list of fiber points Fluid space modeled by a regular lattice

Immersed Boundary Method Structure Fiber activation & force calculation Interpolate Velocity Navier-Stokes Solver Spread Force 4 steps in each timestep Fiber Points Interaction Fluid Lattice

IB Method: Steps 1 and 2 Fiber Activation & Force calculation –Application-specific –For the heart, use an elastic spring law Spread Force –Spread forces from fiber points list to fluid lattice via the Dirac Delta function.

IB Method: Steps 3 and 4 Navier-Stokes Solver –Calculate fluid velocities –Uses a 3D FFT Interpolate velocity: –Gather velocities of fiber points from fluid lattice via Dirac delta function –Move the fiber points

Challenges to Parallelization Irregular fiber lists need to interact with regular fluid lattice. –Trade-off between load balancing of fibers and minimizing communication Efficient “scatter-gather” across processors Need a scalable elliptic solver –Plan to uses multigrid –Eventually add Adaptive Mesh Refinement New algorithms under development at LBNL

Heart Application of the IB Method Heart simulation used to design artificial heart valves

Outline Immersed Boundary Method Titanium Immersed Boundary Method on Titanium

Titanium Motivation Applications are increasingly complex –Want classes, overloading, linked data structures –C++ is hard to read, modify and tune Machines are increasingly complex –Want compiler help for optimizations –Want clear performance model and programmer control Java is a better C++ +Safe: strongly typed, garbage collected –Performance is poor due to

Titanium for Scientific Computing Java dialect for high performance Added constructs for performance & expressiveness –Immutable, value classes –SPMD parallelism with a global address space –Multidimensional arrays –Templates –Region-based memory management Compiled to C (no JVM) with lightweight messaging (Active Messages, LAPI, shmem)

SPMD Parallelism in Titanium Explicitly parallel model: –Fixed number of threads at program startup –Usually one thread per processor Global address space: –Processors can access remote data by reading and writing through “global” references (pointers) –Bulk communication happens when copying arrays –Compiler automatically converts global pointers local ones (up to 2x speedup for “LQI”) –Compiler detects synchronizations bugs in barriers

Value Classes in Titanium Java has two distinct kinds of values –Primitive scalar types: boolean, double, int, etc. –Objects: user-defined and library types implicit level of indirection (pointer to) Titanium adds support for small objects –Look like classes with “immutable” keyword –Stored in place and passed by coping –Examples: Complex type Points used to index Titanium arrays

Titanium Arrays Java arrays are 1-dimensional –Arrays of arrays are inefficient Titanium adds multidimensional arrays –Indexed by Points (tuples of ints) –Algebra over Domains (sets of points) Helps with hierarchical algorithms, e.g., multigrid –One array may be a subarray of another e.g., a is interior of b, or a is all even elements of b –Foreach loops help compiler optimize arrays Within 2x of C for multigrid kernels –Bulk I/O provided on arrays (2x-40x speedup!)

Titanium with Other Languages Native methods are sometimes useful: –Performance: allows for comparisons with other compilers and additional control –Libraries: have interfaced to other systems like PetSC and ParMetis Requires understanding of underlying Titanium implementation in C. Lower entry cost than Java: the native method is simply #included into the generated code

Titanium Implementation Run time system and compiler for: –Uniprocessors –SMP running POSIX threads –Clusters with: Shared memory - SGI Origin cluster (ANL), Tera MTA Global Address Space - T3E (NERSC) Active Messages - NOW & Millennium (UCB) LAPI - IBM SP2, SP3 (SDSC)

Outline Immersed Boundary Method Titanium Immersed Boundary Method on Titanium

Immersed Boundary Generic Software Written by Cowen at NYU Implements subset of the IB method adequate for the heart Runs on vector machines with shared memory

Immersed Boundary on Titanium IBGS rewritten in Titanium. Running since October Contractile torus –runs on Berkeley NOW and SGI Origin Needed for heart: –Input file format –Performance tuning Uniprocessor (C code used temporarily in 2 kernels) Communication

Visualization

Immersed Boundary in Titanium Performance Breakdown (torus simulation):

Immersed Boundary in Titanium

Future work Improve performance –Especially on SP machines (Blue Horizon) Add functionality –Bending angles, anchorage points, source & sinks) to the software package. Add adaptability to NS solver (AMR) –Needed for scaling and more accurate modeling of fluid features in heart