1. Meet the Catalysts Yuri Alexeev, James Osborn, Katherine Riley

2. Diverse ALCF User Community  ALCF is targeted to a few very large science projects  Minimal award in 2014 expected to be at least 50M core-hours  Diverse base of users with diverse needs  Scientific support helps transition, plan and develop for unique HPC resources  Teams have broad expertise –Scientists/engineers –Scalable algorithms –Performance engineering –Scientific code development 2 2013 INCITE Allocations at ALCF (Mira & Intrepid) 736M Intrepid, 2.1B Mira Astro Particle Nuclear Plasma INCITE 60%, ALCC 60%, Discretionary 10%

3. Scientific Support Teams  Two primary scientific support teams –Catalysts – 10 –Performance Engineers - 6  Diversity needed for Engagement, Planning, & Communication –Adapt to new user profiles –Cross pollinate good solutions form different communities –Understand the growth in science and algorithmic requirements 3

4. Scientific Support is Collaboration The ALCF is staffed with a team of computational scientists, expert in their domain, scalable algorithms and performance engineering.  Provide a "jump-start" in the use of ALCF resources  Align the availability of ALCF resources with the needs of the project team  Collaborate to maximize the value that ALCF can bring to our project partners  Connect the needs of the scientific community with future and current hardware 4 Short term, fast solutions Compiling, Debugging, System Use Targeted problem resolution Resolve a specific hard problem like restructuring I/O Long term collaborations In depth work developing significant new capabilities Training Postdocs, students, community Understand HPC needs for different communities Plan for future needs Help planning new facilities Advise/Participate in long term code development paths TacticalStrategic Two categories of collaboration and contribution to teams using the ALCF:

5. Biology ESP PI Mark Gordon 2013 INCITE PI George Schatz One of the largest QM/FMO DFT calculations of biological systems (18k+ atoms) 4096 water molecules with QM/FMO MP2 JCTC and SC12 papers 5 Catalyst team: Yuri Alexeev, Graham Fletcher, Wei Jiang, Ray Loy Running BG/Q codes: CHARMM, GAMESS, LAMMPS, NAMD, Rosetta BG/P codes: BG/Q codes + Amber, GROMACS, …. ALCF and MCS contributions to this project include: ALCF staff Yuri Alexeev formulated and implemented heuristic load balancing algorithm needed to achieve scalability on 40 racks Former MCS staff Ashutosh Mahajan and MCS staff Sven Leyffer solved MINLP problem by using MINOTAUR ALCF staff Graham Fletcher and collaborator Dmitri Fedorov ported GAMESS to BG/P/Q and optimized its performance The ALCF and MCS teams combined with the research team achieved an immense improvement of GAMESS speedup

6. Earth science 2013 INCITE PI Michael Wehner 2013 INCITE PI Warren Washington One of the highest resolution CESM simulations (1/8 degree) Two SC13 paper submissions 6 Catalyst team: Yuri Alexeev, Ray Loy, Katherine Riley, Timothy Williams Running BG/Q codes: CESM, HIRAM, SCEC BG/P codes: BG/Q codes + WRF, … ALCF and MCS contributions to these projects include: ALCF staff Yuri Alexeev formulated and implemented heuristic load balancing algorithm for CESM CI – University of Chicago staff Sheri Mickleson ported, optimized and run CESM simulations MCS staff Sven Leyffer formulated and solved MINLP problem MCS post-dock Prasanna Balaprakash developed and applied machine learning algorithm to improve load balance in CICE The ALCF and MCS teams combined with the research team achieved an immense speedup over the course of a few months

7. Chemistry 2012 ALCC James Chelikowsky Importance of many-body van der Waals forces with self-consistent screening and with density functional theory for biological systems PNAS 2012 paper 7 Catalyst team: Yuri Alexeev, Ramesh Balakrishnan, Jeff Hammond, Graham Fletcher, Anatole Lillienfeld, Nichols Romero Running BG/Q codes: BigDFT, CASINO, GAMESS, QMCPACK, NWChem, MADNESS, MPQC ALCF contributions to this project include: ALCF staff Anatole Lillienfeld proposed and designed the systems where many- body effects are significant ALCF staff Anatole Lillienfeld and Jeff Hammond wrote scripts to perform ensemble simulations to screen lattice parameters for the molecular crystals The ALCF team combined with the research team demonstrated the scientific relevance of vdW forces for drug-DNA intercalation energies, for B versus A DNA conformational energies, and for molecular crystal polymorphs

8. Material science ESP PIs: Larry Curtiss & Jeff Greeley Quantum Monte Carlo (QMC) codes, CASINO and QMCPACK, highly efficient and scalable on Mira First QMC calculations of solid Ar, Xe and Kr (vdW-bonded materials) In progress, study of platinum nanoparticles to elucidate key properties relevant to catalysis 8 Catalyst team: Ramesh Balakrishnan, Anatole Lillienfeld, Nichols Romero Running BG/Q codes: ABINIT, BigDFT, CASINO, CP2K, CPMD, GPAW, QBOX, QMCPACK, Quantum Espresso, MADNESS ALCF contributions to this project include: ALCF post-doc Anouar Benali designed, implemented and tuned new spline algorithms in QMCPACK ALCF staff Vitali Morozov designed a new efficient algorithm for one of the main spline kernels of QMCPACK ALCF team combined with QMCPACK developers designed and implemented a single precision version of the code allowing to double the size of the systems to study. The ALCF team combined with the research team achieved a speedup of 2.65 of the total application time over the course of a few months

9. Astrophysics ESP and 2013 INCITE, PI Salman Habib Largest cosmology simulation ever currently running on Mira with HACC 69% of peak on Mira and SC12 Gordon Bell Finalist 9 1000 Mpc 100 Mpc 20 Mpc 2 Mpc Catalyst team: Katherine Riley, Timothy Williams Running BG/Q codes: CHIMERA, FLASH, GenASIS, HACC BG/P codes: BG/Q codes + ENZO, …. ALCF contributions to this project include: ALCF post-doc Hal Finkel designed, implemented and tuned key tree algorithm needed for a balanced load and data locality ALCF staff Vitali Morozov has redesigned the algorithm and data structures for short force evaluation kernel and tuned it to perform at 82% of the peak of the BG/Q machine ALCF staff Venkatram Vishwanath designed a new parallel I/O file format visualization-specific outputs, and work on the in-situ analysis The ALCF team combined with the research team achieved an immense speedup over the course of a few months

10. Nuclear & Particle physics ESP and 2013 INCITE PI’s: Paul Mackenzie (Lattice QCD): Most accurate simulations of strong nuclear force (QCD) to date Steve Pieper, James Vary (Nuclear Structure): Largest GFMC simulation of Carbon-12 10 Catalyst team: James Osborn Running BG/Q codes: MILC, CPS, Chroma, AGFMC ALCF contributions to these project include: ALCF Staff James Osborn wrote an improved version of MILC “fermion force” routine giving 7x reduction in flop count and run time ALCF post-doc Heechang Na, with James Osborn, optimized MILC on BG/Q by replacing MPI communications with SPI and adding QPX routines ALCF Staff Vitali Morozov helped optimize several key routines in GFMC The ALCF team combined with the research team achieved over a factor of 2 total speedup in the MILC code

11. Plasma physics 2013 INCITE, PI Jean C. Perez Studying Alfven wave turbulence as mechanism for coronal heating and solar wind Simulation from solar surface out to 11 solar radii—Alfven critical point 11 Catalyst team: Timothy Williams Running BG/Q codes: GTC, IRMHD, MHD, pF3d,GYRO ALCF contributions to this project include: With Sameer Shende, analyzed communication phase in code Helped with use of optimized Blue Gene math libraries The ALCF team combined with the research team and Shende achieved a significant speedup—saving millions of CPU hours and allowing more complete study with 2012 INCITE award.

12. Engineering 2013 INCITE George Karniadakis Performed the first multiscale simulation of blood flow in a brain of a patient with an aneurysm SC11 Gordon Bell Finalist Honorable Mention 12 Catalyst team: Ramesh Balakrishnan, Marta Garcia, Vitali Morozov, Scott Parker Running BG/Q codes: Code_Saturne, DNS3D, DPD-LAMMPS+NekTar, Nek5000, OpenFoam, PHASTA BG/P codes: BG/Q codes + Charles, Converge, PSDNS ALCF contributions to this project include: ALCF staff Vitali Morozov did detailed performance analysis and tuning of main kernels for NekTar and DPD-LAMMPS, by getting improved scalability of communications and higher flop rate from vector operations Collaborator Leopold Grinberg has lead the effort for implementing topology aware communication strategy and MPI mapping The ALCF team combined with the research team created highly detailed models of blood flow within the complex blood vessel networks in the brain