1. Excited State Spectroscopy using GPUs Robert Edwards Jefferson Lab TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A A A

2. Hadronic & Nuclear Physics with LQCD Hadronic spectroscopy –Hadron resonance determinations –Exotic meson spectrum (JLab 12GeV ) Hadronic structure –3-D picture of hadrons from gluon & quark spin+flavor distributions –Ground & excited E&M transition form-factors (JLab 6GeV+12GeV+Mainz) –E&M polarizabilities of hadrons (Duke+CERN+Lund) Nuclear interactions –Nuclear processes relevant for stellar evolution –Hyperon-hyperon scattering –3 & 4 nucleon interaction properties [Collab. w/LLNL] (JLab+LLNL) Beyond the Standard Model –Neutron decay constraints on BSM from Ultra Cold Neutron source (LANL) 2

3. Spectroscopy Spectroscopy reveals fundamental aspects of hadronic physics –Essential degrees of freedom? –Gluonic excitations in mesons - exotic states of matter? Status –Can extract excited hadron energies & identify spins, –Pursuing full QCD calculations with realistic quark masses. New spectroscopy programs world-wide –E.g., BES III, GSI/Panda –Crucial complement to 12 GeV program at JLab. Excited nucleon spectroscopy (JLab) JLab GlueX: search for gluonic excitations. 3

4. Excited states: anisotropy+operators+variational Anisotropic lattices with N f =2+1 dynamical fermions –Temporal lattice spacing a t < a s (spatial lattice spacing) –High temporal resolution ! Resolve noisy & excited states –Major project within USQCD – Hadron Spectrum Collab. Extended operators –Subduction: sufficient derivatives ! nonzero overlap at origin Variational method: –Distillation: matrix of correlators ! project onto excited states PRD 78 (2008) & PRD 79 (2009) PRD 76 (2007), PRD 77 (2008), PRD 80 (2009), arxiv:1002.0818 PRD 72 (2005), PRD 72 (2005), PRL 103 (2009) 4

5. Gauge Generation: Cost Scaling Cost: reasonable statistics, box size and “physical” pion mass Extrapolate in lattice spacings: 10 ~ 100 PF-yr PF-years State-of-Art Today, 10TF-yr 2011 (100TF-yr) 5

6. Computational Requirements Gauge generation : Analysis Current calculations Weak matrix elements: 1 : 1 Baryon spectroscopy: 1 : 10 Nuclear structure: 1 : 4 Computational Requirements: Gauge Generation : Analysis 10 : 1 (2005) 1 : 3 (2010) Core work: Dirac inverters - use GPU-s 6

7. SciDAC Software Stack QCD friendly API’s/libs http://www.usqcd.org Data parallel C/C++ Architectural level High-level (linpack-like) GPU-s Application level 7

8. SciDAC Impact Software development –QCD friendly API’s and libraries: enables high user productivity –Allows rapid prototyping & optimization –Significant software effort for GPU-s Algorithm improvements –Operators & contractions: clusters (Distillation: PRL (2009)) –Mixed-precision Dirac-solvers: INCITE+clusters+GPU-s, 2-3X –Adaptive multi-grid solvers: clusters, ~8X (?) Hardware development via USQCD Facilities –Adding support for new hardware –GPU-s 8

9. Inverter Strong Scaling: V=32 3 x256 Local volume on GPU too small (I/O bottleneck) 3 Tflops 9

10. New Science Reach in 2010-2011 QCD Spectrum Gauge generation: (next dataset) –INCITE: Crays&BG/P-s, ~ 16K – 24K cores –Double precision Analysis (existing dataset): two-classes –Propagators (Dirac matrix inversions) Few GPU level Single + half precision No memory error-correction –Contractions: Clusters: few cores Double precision + large memory footprint Cost (TF-yr) New: 10 TF-yr Old: 1 TF-yr 10 TF-yr 1 TF-yr 10

11. Isovector Meson Spectrum 11

12. Isovector Meson Spectrum 12

13. Exotic matter? Can we observe exotic matter? Excited string QED QCD 13

14. Exotic matter Exotics: world summary 14

15. Exotic matter Suggests (many) exotics within range of JLab Hall D Previous work: photo- production rates high Current GPU work: (strong) decays - important experimental input Exotics: first GPU results 15

16. Baryon Spectrum “Missing resonance problem” What are collective modes? What is the structure of the states? –Major focus of (and motivation for) JLab Hall B –Not resolved experimentally @ 6GeV 16

17. Nucleon & Delta Spectrum First results from GPU-s < 2% error bars 17

18. Nucleon & Delta Spectrum First results from GPU-s < 2% error bars [ 56,2 + ] D-wave [ 70,1 - ] P-wave [ 70,1 - ] P-wave [ 56,2 + ] D-wave Discern structure: wave-function overlaps Change at light quark mass? Decays! Suggests spectrum at least as dense as quark model 18

19. Towards resonance determinations Augment with multi-particle operators –Needs “annihilation diagrams” – provided by Distillation Ideally suited for (GPU-s) Resonance determination –Scattering in a finite box – discrete energy levels –Lüscher finite volume techniques –Phase shifts ! Width First results (partially from GPU-s) –Seems practical arxiv:0905.2160 19

20. Phase Shifts: demonstration 20

21. Prospects Anisotropic gauge production: –Useful for hadronic & nuclear physics Spectrum determination –Looks promising! Significant progress in last year –Possible with new correlator and operator constructions: Distillation + Subduction –Framework for multi-particle decays: on-going work –Not discussed: photon decays -> internal probe of structure GPU-s –Powerful resource for inversions –New ECC+double precision -> handle contractions 21

22. Extending science reach USQCD: –Next calculations: physical quark masses: 100 TF – 1 PF-yr –New INCITE+Early Science application (ANL+ORNL+NERSC) –NSF Blue Waters Petascale (PRAC) Need SciDAC-3 –Significant software effort for next generation GPU-s & heterogeneous environments –Participate in emerging ASCR Exascale initiatives INCITE + LQCD synergy: –ARRA GPU system well matched to current leadership facilities 22

23. Summary Capability + Capacity + SciDAC –Deliver science & HEP + NP milestones Petascale (leadership) + Petascale (capacity)+SciDAC-3 Spectrum + decays First contact with experimental resolution Exascale (leadership) + Exascale (capacity)+SciDAC-3 Full resolution Spectrum + transitions Nuclear structure Collaborative efforts: USQCD + JLab user communities 23

24. Backup slides The end 24

25. Dirac Inverter with Parallel GPU-s Divide problem among nodes: Trade-offs –On-node vs off-node bandwidths –Locality vs memory bandwidth Efficient at large problem size per node 25

26. Interpretation of Meson Spectrum Future: incorporate in bound-state model phenomenology Future: probe with photon decays

27. Distillation: annihilation diagrams Two-meson creation op Correlator arxiv:0905.2160 27

28. Operators and contractions New operator technique: Subduction –Derivative-based continuum ops -> lattice irreps –Operators at rest or in-flight, mesons & baryons Large basis of operators -> lots of contractions –E.g., nucleon H g 49 ops up through 2 derivs –Order 10000 two-point correlators Feed all this to variational method –Diagonalization: handles near degeneracies PRL 103 (2009) 28