1. November 1, 2007Scott Dodelson1 Computational Cosmology Initiative Report of the Task Force

2. November 1, 2007Scott Dodelson2 Task Force  Scott Dodelson (Center for Particle Astrophysics), co-Chair  Don Petravick (Computing Division), co-Chair  James Amundson (Computing Division)  Jim Annis (Experimental Astrophysics)  Nick Gnedin (Theoretical Astrophysics)  Don Holmgren (Computing Division)  Jim Kowalkowski (Computing Division)  Andrey Kravtsov (University of Chicago)  Andreas Kronfeld (Particle Theory)

3. November 1, 2007Scott Dodelson3 Task Force Develop, as necessary, and articulate the case for a strategic initiative for a computational cosmology computing project as part of the Fermilab computing capability. It will be important to:  Articulate an outstanding scientific case.  Discuss the existing and potential scientific collaboration across the US. Could this evolve into a national program?  put the proposed capability in the context of other computing capabilities such as that available from large national computing centers such as NERSC, leadership DOE, Terascale NSF centers.  put the proposed capability in the context of worldwide developments  discuss the technical aspects; this may include an R&D component.  discuss the infrastructure needs.  discuss the resource needs including the effort at different levels addressing not only the hardware but also the algorithmic aspects of a possible project.  discuss the connection to or synergies with any relevant other high performance computing or SciDACC efforts.  develop a business plan considering multiple funding sources. Prepare a written report considering two 3-year periods with the first period not thoroughly developed. This should take into account the funding cycle for opportunities such as SciDACC. The report should be limited to 20 pages maximum including all tables and figures using a reasonable font size. It would be interesting to receive a draft report by mid-June with the final report by July 31. Charge from Hugh Montgomery, April 10, 2007

4. November 1, 2007Scott Dodelson4 Task Force  Weekly Meetings  Briefings on: Lattice QCD (AK), Legal/Business Issues (DH), Computing Resources (Space, power, etc.) (DP), Science (AK,NG), Software Issues (AK,NG), Refining Charge (Mont)  Delivered First Draft in June 2007  Sent for review to ~10 people; received responses from ~5; incorporated these in final draft  Final draft delivered August, 2007

5. November 1, 2007Scott Dodelson5 Science Case We have learned a lot from the cosmic microwave background because deviations from uniformity are small Measurements can be compared to theory which depends on parameters

6. November 1, 2007Scott Dodelson6 Science Case  Gravitational Instability is nonlinear  Baryons governed by hydrodynamics  Radiation Field affects and is affected by structure  Stars form, Supernovae explode, … Extracting information from observations of galaxies is much more difficult

7. November 1, 2007Scott Dodelson7 Science Case QED QCD Akin to the difference between

8. November 1, 2007Scott Dodelson8 Science Case  Shape of the matter power spectrum depends on neutrino mass and inflation  Evolution of structure depends on dark energy  Gravitational wells determined by dark matter  Rich astrophysical questions: galaxy formation, reionization, black holes, quasars,… If we can obtain accurate predictions for a given set of cosmological parameters, the payoff will be enormous

9. November 1, 2007Scott Dodelson9 Science Case Computing power – software and hardware – has improved so that the goal of obtaining accurate predictions for the distribution of matter in the universe is within reach Note: Focus on cosmological simulations from smooth initial conditions

10. November 1, 2007Scott Dodelson10 Science Case Example: Weak Lensing. Leaving out physics of star formation leads to incorrect predictions in the “dark energy” region.

11. November 1, 2007Scott Dodelson11 Science Case: Finding #1 “Cosmological simulations are powerful tools to understand the growth of structure in the universe. This understanding is interesting and important in its own right, but also is essential for extracting precision information about fundamental physics - such as dark energy and dark matter - from upcoming astronomical surveys.”

12. November 1, 2007Scott Dodelson12 Local Expertise Nick Gnedin (FNAL) and Andrey Kravtsov (UChicago) are collaborators and two of the strongest numerical astrophysicists in the world. They have assembled excellent teams of students and postdocs. The two groups are extremely productive (25 papers/183 citations since 1/06).

13. November 1, 2007Scott Dodelson13 Gnedin and Kravtsov developed Adaptive Refinement Tree (ART) code  Implementation of Adaptive Mesh Refinement method  Refines on a cell-by-cell basis  Full 4D adaptive  Includes - dark matter dynamics - gas dynamics and chemistry - radiative transfer - star formation, etc Local Expertise

14. November 1, 2007Scott Dodelson14 Local Expertise

15. November 1, 2007Scott Dodelson15 Local Expertise: Finding #2 “Groups at the University of Chicago and Fermilab have the tools in the form of state- of-the-art codes to contribute to this important area.”

16. November 1, 2007Scott Dodelson16  As of today, Computing Division maintains an 8 core cluster for cosmology  Other U.S. groups have many more resources: ITC, Harvard: 316 CPU Opteron, 264 CPU Athlon LANL: 294 CPU Pentium4 (just for cosmology) SLAC: 72 CPU SGI Altrix, 128 CPU Xeon cluster Princeton: 188 CPU Xeon cluster UWash: 64 CPU Pentium cluster  International Groups are far ahead Virgo Consortium: 670 CPU SparcIII, 816 CPU Power4 CITA: 270 CPU Xeon cluster, 2000 CPU dual Pentium (shared) Resources

17. November 1, 2007Scott Dodelson17 Resources In the two years since the inception of Distributed European Infrastructure for Supercomputing Applications, 8 large-scale allocations were made for computational cosmology projects at European supercomputer centers with an average allocation size of 1 million hours or more. For comparison, only 1 computational cosmology project of similar size was approved in the last three years through the DOE INCITE program.

18. November 1, 2007Scott Dodelson18 Resources: Finding #3 “The local resources for these groups are currently inadequate to run the largest, highest resolution simulations. This inadequacy is representative of a problem on a national scale. In contrast, European groups have pooled resources and currently run the most powerful cosmological simulations.”

19. November 1, 2007Scott Dodelson19 Infrastructure @ FNAL  Space/Power for Hardware not a problem  CD currently supports many machines (>20,000 cores)  Software development for LQCD/Physics for Accelerators  Build/manage releases for software for D0,CDF, CMS …  Experience benchmarking code; optimizing hardware  Data Handling  Project Management

20. November 1, 2007Scott Dodelson20 Infrastructure @ FNAL: Finding #4 “Fermilab has the infrastructure to host a cosmological computing initiative on a national scale.”

21. November 1, 2007Scott Dodelson21 Proposal What have we done so far?  Received FRA grant ($75k)  Received funds for hardware from Kavli Institute for Cosmological Physics at UChicago ($100k)  Contributed funds from Theoretical Astrophysics (Rocky money) ($60k)  Successfully obtained LDRD at Argonne for software development and ultimately joint postdocs  Expect to have ~500 cores @ FNAL by December 2007

22. November 1, 2007Scott Dodelson22 Proposal What are we asking for from FNAL in FY08?  Funds to continue building local cluster ($200k)  Support for cluster from CD (0.2 FTE)  Software Developer from CD (0.5 FTE)

23. November 1, 2007Scott Dodelson23 Proposal

24. November 1, 2007Scott Dodelson24 Proposal Long Term Goal Develop a national program in computational cosmology powerful enough to exploit fully Stage IV dark energy experiments

25. November 1, 2007Scott Dodelson25 Back-up Slide 1: PAC Advice “Should opportunities for expanding the scope of Fermilab R&D activities emerge in the future (noble liquid detectors for Dark Matter searches and water Cerenkov detectors for long-baseline neutrino experiments are among the various possibilities), they should be evaluated in the context of the overall national scientific strategy and funding constraints, taking into account the following criteria: 1. Scientific and technical excellence; 2. The uniqueness and impact of the Fermilab contribution, given its existing infrastructure and technical expertise; and 3. The interest among Laboratory personnel in collaborating on the project. Ideally, Laboratory management would help external proponents to identify relevant personnel who might be available to participate.“

26. November 1, 2007Scott Dodelson26 Back-up Slide 2: Resources