1. Advanced Simulation and Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes The FLASH Code Parallel AMR Made Easy Tomek Plewa LACSI Symposium Santa Fe, 10/12/2004

2. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Outline FLASH Center Astrophysics as primary application Verification and Validation activities External contributors and users FLASH Code View from the outside Source tree and functionality User level Problem configuration Intra-module modifications Developer level Extending the source tree Inter-module communications Summary

3. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Application: Astrophysics Target Applications Compact accreting stars (white dwarf, neutron star) Reactive hydrodynamics (DNS or subgrid model) Initial conditions close to hydrostatic equilibrium (self-gravity) Complex EOS (dense nuclear matter) Example: Type Ia Supernova Massive white dwarf Subgrid model for nuclear flame Self-gravity Degenerate EOS

4. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Length scales in White Dwarf Deflagration Before 2000 Now BG/L

5. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Verification & Validation Program Verification: solving equations right Validation: solving the right equations Verification ranging from simple analytic problems to code-code comparison. Validation: no direct access to experiments, use scaling laws Absolutely NO culture of validation in astrophysics!

6. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Verification

7. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH V&V Complete Example Experimental time series, water/glycol fog visualization of SF 6 mole fraction. Images correspond to 50, 190, 330, 470, 610, and 750  s after shock impact Composite image does not preserve time-distance relationship

8. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago AMR-generated Timestep-dependent Signal CFL=0.8 CFL=0.2 CFL=0.4

9. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago AMR-generated Timestep-dependent Signal

10. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH As An Open Source Research Code FLASH is community code, freely (with minimal restrictions) available for research External contribution: shock-cylinder experiment (Jacobs, LANL)

11. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Center and AMR Community

12. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code Primary objective To create robust, reliable, efficient and expandable code, that stands the test of time and users. Primary characteristics parallel adaptive mesh (possibly different mesh packages) block-structured (currently) multi-physics (hydro, MHD, SRHD, gravity, particles…) Fortran 9x portable freely accessible documented & actively maintained

13. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

14. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

15. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

16. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

17. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

18. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

19. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

20. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

21. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

22. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code As Seen From The Outside

23. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Portability: BG/L Successful early runs up to 2048 processors Communication is scaling well. Test size limited by small number of available processors. Large runs scheduled Several days of 16K-32K processors. Focusing on FLOPS Make up for slower CPUs, beta-compilers

24. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Code Access

25. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Source Tree And Functionality

26. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Source Tree And Functionality

27. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Driver

28. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Driver

29. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Hydrodynamics Unit

30. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH EOS Unit

31. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Unit Hierarchy API/ Stubs Common API Impl API impl Wrapper kernel API impl Wrapper kernel Wrapper kernel

32. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Solvers: Poisson

33. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Source Terms

34. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Common Fabric

35. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Inter-Module Connectivity

36. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Inter-Module Connectivity

37. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Putting It All Together

38. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Config Files

39. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Setups

40. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Setup Example

41. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Initial Conditions

42. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Configuring FLASH Application

43. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Application Build Tree

44. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Documentation

45. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH I/O

46. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Visualization

47. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Results

48. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Summary Overall architecture Unit types Unit architecture Near future

49. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Summary: Architecture Setup tool assemble application Config files define components Driver module organize interactions Unit Architecture API Inheritance Data management

50. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Summary: Units Types Physics hydro gravity Infrastructure driver grid I/O Utility profiler runtime visualization Data analysis Fidlr flashview

51. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Wrapper Summary: Unit Architecture Driver Top level: API Unit Test Grid Data Module patch data time step etc Kernel

52. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Near Future: FLASH3 FLASH original design was very Paramesh-like block structured fixed size blocks specified at compile time every section of the code assumed knowledge of block size Relaxing the constrain opens up possibilities of using variable block size mesh patch based mesh non-uniform non-adaptive mesh uniform grid without unnecessary overheads