1. Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley

2. Project Summary Provide easy-to-use, high performance tool for simulation of fluid flow in biological systems. –Using the Immersed Boundary Method Enable simulations on large-scale parallel machines. –Distributed memory machine including SMP clusters Using Titanium, ADR, and KeLP with AMR Specific demonstration problem: Simulation of the heart model on Blue Horizon.

3. Outline Short term goals and plans Technical status of project –Immersed Boundary Method –Software Tools –Solvers Next Steps

4. Short Term Goals for October 2001 IB Method written in Titanium (IBT) IBT Simulation on distributed memory Heart model input and visualization support in IBT Titanium running on Blue Horizon IBT users on BH and other SPs ?Performance tuning of code to exceed T90 performance ?Replace solver with (adaptive) multigrid

5. IB Method Users Peskin and McQueen at NYU –Heart model, including valve design At Washington –Insect flight Fauchy et al at Tulane –Small animal swimming Peter Kramer at RPI –Brownian motion in the IBM John Stocky at Simon Fraser –Paper making Others –parachutes, flags, flagellates, robot insects

6. Building a User Community Many users of the IB Method Lots of concern over lack of distributed memory implementation Once IBT is more robust and efficient (May ’01), advertise to users Identify 1 or 2 early adopters Longer term: workshop or short course

7. Long Term Software Release Model Titanium –Working with UPC and possibly others on common runtime layer –Compiler is relatively stable but needs ongoing support IB Method –Release Titanium source code –Parameterized “black box” for IB Method with possible cross-language support Visualization software is tied to SGI

8. Immersed Boundary Method Developed at NYU by Peskin & McQueen to model biological systems where elastic fibers are immersed in an incompressible fluid. –Fibers (e.g., heart muscles) modeled by list of fiber points –Fluid space modeled by a regular lattice

9. Immersed Boundary Method Structure Fiber activation & force calculation Interpolate Velocity Navier-Stokes Solver Spread Force 4 steps in each timestep Fiber Points Interaction Fluid Lattice

10. Challenges to Parallelization Irregular fiber lists need to interact with regular fluid lattice. –Trade-off between load balancing of fibers and minimizing communication Efficient “scatter-gather” across processors Need a scalable elliptic solver –Plan to uses multigrid –Eventually add Adaptive Mesh Refinement New algorithms under development by Colella’s group

11. Tools used for Implementation Titanium supports –Classes, linked data structures, overloading –Distributed data structures (global address space) –Useful for planned adaptive hierarchical structures ADR provides –Help with analysis and organization of output –Especially for hierarchical data KeLP provides –Alternative programming model for solvers ADR and KeLP are not critical for first-year

12. Titanium Status Titanium runs on uniprocessors, SMPs, and distributed memory with a single programming model It has run on Blue Horizon –Issues related to communication balance –Revamped backends are more organized, but BH backend not working right now Need to replace personnel

13. Solver Status Current solver is based on 3D FFT Multigrid might be more scalable Multigrid with adaptive meshes might be more so Balls and Colella algorithm could also be used KeLP implementations of solvers included Note: McQueen is looking into solver issues for numerical reasons unrelated to scaling Not critical for first year goals

14. IB Titanium Status IB (Generic) rewritten in Titanium. Running since October Contractile torus –runs on Berkeley NOW and SGI Origin Needed for heart: –Input file format –Performance tuning Uniprocessor (C code used temporarily in 2 kernels) Communication

15. Immersed Boundary on Titanium Performance Breakdown (torus simulation):

16. Immersed Boundary on Titanium

17. Next Steps Improve performance of IBT Generate heart input for IBT Recover Titanium on BH Identify early user(s) of IBT Improve NS solver Add functionality –Bending angles, anchorage points, source & sinks) to the software package.

18. Yelick(UCB), Peskin (NYU), Colella (LBNL), Baden (UCSD), Saltz (Maryland) Adaptive Computations for Fluids in Biological Systems Immersed Boundary Method Applications Human Heart (NYU) Embryo Growth (UCB) Blood Clotting (Utah) Robot Insect Flight (NYU) Pulp Fibers (Waterloo) Generic Immersed Boundary Method (Titanium) Heart (Titanium) Insect Wings Flagellate Swimming … Spectral (Titanium) Multigrid (KeLP) AMR Application Models Extensible Simulation Solvers

19. General Questions - How has your project addressed the goals of the PACI program (providing access to tradition HPC, providing early access to experimental systems, fostering interdisciplinary research, contributing to intellectual development, broadening the base)? - What infrastructure products (e.g., software, algorithms, etc.) have you produced? - Where have you deployed them (on NPACI systems, other systems)? - What have you done to communicate the availability of this infrastructure? - What training have you done? - What kind/size of community is using your infrastructure? - How have you integrated your work with EOT activities? - What scientific accomplishments - or other measurable impacts not covered by answers to previous questions - have resulted from its use? - What are the emerging trends/technologies that NPACI should build on/leverage? - How can we increase the impact of NPACI development to date? - How can we increase the community that uses the infrastructure you've developed?