1. Petascale astronomy and the SKA Athol Kemball Department of Astronomy & Center for Extreme-scale Computing (IACAT/NCSA) University of Illinois, USA akemball@illinois.edu

2. SKA SA 2008 Contemporary scientific methods Theory: –Develop abstract or mathematical models of the physical system or problem. Experimental and observational methods: –Take observational or experimental data to disprove or refine models Computational methods: –Simulate complex multi- scale systems that are beyond the reach of analytic methods –Process vast amounts of observed or experiment data Euclid, 3 rd century mathematician, teaching (Raphael) Very Large Array (VLA); New Mexico, USA Molecular dynamics simulation: water permeation in aquaporins (Schulten Group, UIUC

3. SKA SA 2008 Computational Cosmology: Structure Formation Nonlinear Evolution of the Universe: from 20 million to 14 billion years old The cosmological simulation computes the nonlinear evolution of the universe in the context of the standard cosmological model determined by the Wilkinson Microwave Background Anisotropy experiment. (Cen & Ostriker 2006; Advanced Vizualization Laboratory NCSA)

4. SKA SA 2008 Computational Science: Ensuring … President’s Information Technology Advisory Committee “Together with theory and experiment, computational science now constitutes the “third pillar” of scientific inquiry, enabling researchers to build and test models of complex phenomena – such as multi-century climate shifts, multidimensional flight stress on aircraft, and stellar explosions – that cannot be replicated in the laboratory, and to manage huge volumes of data rapidly and economically.” While it is itself a discipline, computational science serves to advance all of science. The most scientifically important and economically promising research frontiers in the 21st century will be conquered by those most skilled with advanced computing technologies and computational science applications.”

5. SKA SA 2008 Revolutionizing Science and Engineering … Advisory Panel on Cyberinfrastructure (D. Atkins), National Science Foundation “… a new age has dawned in scientific and engineering research, pushed by continuing progress in computing, information, and communication technology, and pulled by the expanding complexity, scope, and scale of today ʼ s challenges.”

6. SKA SA 2008 Computational Science and Engineering Molecular ScienceWeather & Climate Forecasting Earth ScienceAstronomy Health

7. SKA SA 2008 Open Challenges in Modern Astrophysics

8. SKA SA 2008 “What are the basic properties of the fundamental particles and forces?” Neutrinos, Magnetic Fields, Gravity, Gravitational Waves, Dark Energy “What constitutes the missing mass of the Universe?” Cold Dark Matter (e.g. via lensing), Dark Energy, Hot Dark Matter (neutrinos) “What is the origin of the Universe and the observed structure and how did it evolve?” Atomic hydrogen, epoch of reionization, magnetic fields, star-formation history…… “How do planetary systems form and evolve?” Movies of Planet Formation, Astrobiology, Radio flares from exo-planets…… “Has life existed elsewhere in the Universe, and does it exist elsewhere now?” SETI Fundamental questions in physics and astronomy

9. SKA SA 2008 How does SKA answer these questions ? Detect and image neutral hydrogen in the very early phases of the universe when the first stars and galaxies appeared “epoch of re-ionization” Locate 1 billion galaxies via their neutral hydrogen signature and measure their distribution in space – “dark energy”

10. SKA SA 2008 How does SKA answer these questions ? Time pulsars to test description of gravity in the strong field case (pulsar-Black Hole binaries), and to detect gravitational waves; explore the unknown transient universe Origin and evolution of cosmic magnetic fields – “the magnetic universe” Planet formation – image Earth-sized gaps in proto-planetary disks BLACK HOLE

11. SKA SA 2008 The Large Synoptic Survey Telescope (2014) (LSST; 8.4m; 3.2 Gpixel camera) LSST science goals Cosmology: probing dark energy and dark matter Exploring the transient sky Mapping the Milky Way Inventory of Solar System objects (Cerro Panchon (Iveziv et al. 2008)) (LSST deep lensing survey (Ivezic et al. 2008))

12. SKA SA 2008 The Great Survey Era SKA-era telescopes & science require: Surveys over large cosmic volumes (Ω,z), fine synoptic time-sampling Δt, and/or high completeness High receptor count and data acquisition rates Software/hardware boundary far closer to receptors than at present Efficient, high-throughput survey operations modes Processing implications High sensitivity, A e /T sys ~10 4 m 2 K -1, wide-field imaging; Demanding (t,ω,P) non-imaging analysis Large O(10 9 ) survey catalogs High associated data rates (TBps), compute processing rates (PF), and PB/EB archives (HI galaxy surveys, e.g. ALFALFA HI (Giovanelli et al. 2007); SKA requires a billion galaxy survey.) (SKA schematic: tiled aperture arrays plus parabolic dishes)

13. SKA SA 2008 Petascale Computing Challenges in the Great Survey Era

14. SKA SA 2008 14 LSST computing and data storage scale Reference science requires: Telescope data output of 15 TB per night Archive size ~ O(10 2 ) PB Processing ~ O(1) PF (LSST data flow (Ivezic et al. 2008)) (LSST focal plan: each square 4k x 4k pixels; (Ivezic et al. 2008))

15. SKA SA 2008 SKA wide-field image formation Algorithm technologies 3-D transform (Perley 1999), facet-based tesselation / polyhedral imaging (Cornwell & Perley 1992), and w-projection (Cornwell et al. 2003). (Cornwell et al. 2003; facet-based vs w-projection algorithms)

16. SKA SA 2008 LNSD data rates (Perley & Cornwell 2003): where D = dish diameter, B = max. baseline, Δν = bandwidth, and ν = frequency Wide-field imaging cost ~ O(D -4 to -8 ) (Perley & Clark 2003; Cornwell 2004; Lonsdale et al 2004). Full-field continuum imaging cost (derived from Cornwell 2004): Strong dependence on 1/D and B. Data rates of Tbps and computational costs in PF are readily obtained from underlying geometric terms. Spectral line imaging costs exceed continuum imaging costs. Possible mitigation through FOV tailoring (Lonsdale et al 2004), beam-forming, and antenna aggregation approaches (Wright et al.) –550 GBps/n a 2 (Lonsdale et al 2004) Runaway petascale costs for SKA tightly coupled to design choices SKA computing and data scale

17. SKA SA 2008 Directions in Computing Technology

18. SKA SA 2008 The declining cost of high- performance computing hardware Computing hardware system costs vary over key primary axes: –Time evolution (Moore’s Law) –Level of commoditization Commoditization effects in computing hardware costs models for general- purpose CPU and GPU accelerators at a fixed epoch (2007). Estimated from public data. Moore’s Law for general-purpose Intel CPUs. Trend-line for Top 500 leading-edge performance.

19. SKA SA 2008 Predicted leading-edge LINPACK R max performance from Top 500 trend-line (from data t yr = [1993, 2007]): Cost per unit teraflop c TF (t), for a commiditzation factor η, Moore’s Law doubling time Δt, and construction lead time Δc: [with c TF (t 0 ) = $300k/TF, t 0 = 2007, η = [0.3-1.0], Δt ~ 1.5 yr, Δc ~ 1-4 yr] Computing hardware performance and cost models

20. SKA SA 2008 Directions in Computing Technology Increasing Clock Frequency & Performance Frequency (MHz) “In the past, performance scaling in conventional single-core processors has been accomplished largely through increases in clock frequency (accounting for roughly 80 percent of the performance gains to date).” Platform 2015 S. Y. Borkar et al., 2006 Intel Corporation Intel Pentium

21. SKA SA 2008 Directions in Computing Technology Problem with Uni-core Microprocessors Decreasing Feature Size Increasing Chip Frequency Watts/cm 2 1 10 100 1000 1.5  1.0  0.7  0.5  0.35  0.25  0.18  0.13  0.1  0.07  i386 i486 Pentium Pentium Pro Pentium II Pentium III Hot Plate Nuclear Reactor Rocket Nozzle Pentium 4 (Prescott) Pentium 4 (Willamette)

22. SKA SA 2008 Directions in Computing Technology From Uni-core to Multi-core Processors AMD Uni-, Dual-, Quad-core, Processors Intel Multi-core Performance Intel Teraflops Chip

23. SKA SA 2008 Directions in Computing Technology Switch to Multicore Chips Frequency (MHz) dual core quad core “For the next several years the only way to obtain significant increases in performance will be through increasing use of parallelism: – 4× now – 8× in 2009 – 16× in 2011 – …

24. SKA SA 2008 Trends at extreme scale Inconvenient truths Moore’s Law holds, but high- performance architectures are evolving rapidly: –Breakpoint in clock speed evolution (2004) –Lateral expansion to multi-core processors and processor augmentation with accelerators Theoretical performance ≠ actual performance Sustained petascale calibration and imaging performance for SKA requires: –Demonstrated mapping of SKA calibration and imaging algorithms to modern HPC architectures, and proof of feasible scalability to petascale: [O(10 5 ) processor cores]. –Remains a considerable design unknown in both feasibility and cost. (Golap, Kemball et al. 2001, Coma cluster, VLA 74 MHz, parallelized facet-based wide-field imaging)

25. SKA SA 2008 Scalability Fastest current NCSA system (abe.ncsa.uiuc. edu * ) Generic petascale system Peak performance 0.090 PF10-20 PF Number of processors 9,600 300,000- 750,000 Amount of memory 0.0096 PB0.5-1.0 PB Disk storage 0.10 PB25-50 PB Archival storage 0.005 EB0.5-1 EB (Dunning 2007) * Abe: Dell 1955 blade cluster – 2.33 GHz Intel Cloverton Quad-Core 1,200 blades/9,600 cores 89.5 TF; 9.6 TB RAM; 170 TB disk – Power/Cooling 500 KW / 140 tons

26. SKA SA 2008 US NSF vision for open petascale computing

27. SKA SA 2008 Challenges and Solutions in Petascale Computing Petascale Computing Facility www.ncsa.uiuc.edu/BlueWaters Modern Data Center –90,000+ ft 2 total –20,000 ft 2 machine room Energy Efficiency –LEED certified (goal: silver) –Efficient cooling system Partners EYP MCF/ Gensler IBM Yahoo!

28. SKA SA 2008 Innovative Computing Technologies On to Many-core Chips Intel Teraflops Chip (80 cores) NVIDIA GeForce 8800 GTX (128 cores) IBM Cell (1+8 cores)

29. SKA SA 2008 Innovative Computing Technologies New Technologies for Petascale Computing Gflops Courtesy of John Owens (UCSD) & Ian Buck (NVIDIA) 200220032004200520062007 3.4 GHz Dual- core 2.66 GHz Quad-core 1.35 GHz G80 1.50 GHz G80 NVIDIA (GPU) INTEL (CPU) 0 50 100 150 200 250 300 350 400

30. SKA SA 2008 Innovative Computing Technologies NVIDIA: GeForce 8800 GTX GPU 128 Cores, 346 GFLOPS (SP), 768 MB DRAM, 86.4 GB/s memory bandwidth; CUDA* * Compute Unified Device Architecture

31. SKA SA 2008 Innovative Computing Technologies NVIDIA: Selected Benchmarks ApplicationDescriptionKernel XApp X H.264 SPEC ’06 version, change in guess vector 20.21.5 LBM SPEC ’06 version, change to single precision and print fewer reports 12.512.3 FEM Finite element modeling, simulation of 3D graded materials 11.010.1 RPES Rys polynomial equation solver, 2-electron repulsion integrals 210.079.4 PNS Petri net simulation of a distributed system 24.023.7 LINPACK Single-precision implementation of saxpy, used in Gaussian elimination routine 19.411.8 TRACF Two Point Angular Correlation Function 60.221.6 FDTD Finite-difference time domain analysis of 2D electromagnetic wave propagation 10.51.2 MRI-Q Computing a matrix Q, a scanner’s configuration in MRI reconstruction 457.0431.0 * W-m. Hwu et al., 2007

32. SKA SA 2008 Computational and Algorithmic Challenges for the SKA

33. SKA SA 2008 Feasibility: imaging dynamic range Richards 2000HDFVLA 1.4 GHz7.5 μJy Norris et al 2005HDF-SATCA 1.4 GHz10 μJy Middelberg et al 2008 ELAIS IATCA 1.4 GHz< 30 μJy Miller et al 2008E-CDF-S{E}VLA 1.4 GHz6.4 μJy Reference specifications (Schillizzi et al 2007) Targeted λ20cm continuum field: 10 7 :1. Routine λ20cm continuum: 10 6 :1. Driven by need to achieve thermal noise limit (nJy) over plausible field integrations. Spectral dynamic range: 10 5 :1. Current typical state of practice near λ ~ 20 cm given below. (de Bruyn and Brentjens, 2005) High-sensitivity deep fields Noordarm et al 1982 3C84WSRT 1.4 GHz10,000:1 Geller et al 20001935-692ATCA 1.4 GHz77,000:1 de Bruyn & Brentjens 2005 PerseusWSRT 92 cm400,000:1 de Bruyn et al, 2007 3C147WSRT 1.4 GHz1,000,000:1 Dynamic range

34. SKA SA 2008 Feasibility: imaging dynamic range Visibility on baseline m-n Visibility-plane calibration effect Image-plane calibration effect Source brightness (I,Q,U,V) Direction on sky: ρ Basic imaging and calibration equation for radio interferometry (e.g. Hamaker, Bregman, & Sault et al.): Key challenges Robust, high-fidelity image-plane (ρ) calibration: –Non-isoplanatism. –Antenna pointing errors. –Polarized beam response in (t,ω), … Non-linearities, non-closing errors Deconvolution and sky model limits Dynamic range budget will be set by system design elements. (Bhatnagar et al. 2004; antenna pointing self- cal: 12µJy => 1µJy rms)

35. SKA SA 2008 Feasibility: imaging dynamic range Visibility on baseline m-n Visibility-plane calibration effect Image-plane calibration effect Source brightness (I,Q,U,V) Direction on sky: ρ Basic imaging and calibration equation for radio interferometry (e.g. Hamaker, Bregman, & Sault et al.): Calibration challenges Number of free parameters in image-plane terms far greater than visibility- plane terms: –Requires large-parameter solvers for multiple calibration terms –Stability, robustness, and convergence an open research topic. Large-N arrays will almost certainly operated with reference Global Sky Models (GSM) –As well-calibrated as possible in routine observing. –A new paradigm, however … –Pathfinders will inject reality here

36. SKA SA 2008 SKA dynamic range assessment – beyond the central pixel Current achieved dynamic ranges degrade significantly with radial projected distance from field center, for reasons understood qualitatively (e.g. direction-dependent gains, sidelobe confusion etc.) An SKA design with routine uniform, ultra-high dynamic range requires a quantitative dynamic range budget. Strategies: –Real data from similar pathfinders (e.g. MeerKAT) are key. –Simulations are useful if relative dynamic range contributions or absolute fidelity are being assessed with simple models. –New statistical methods: Assume convergent, regularized imaging estimator for brightness distribution within imaging equation; need to know sampling distribution of imaging estimator per pixel, but unknown PDF a priori: Statistical resampling (Kemball & Martinsek 2005ff) and Bayesian methods (Sutton & Wandeldt 2005) offer new approaches. Feasibility: dynamic range assessment

37. SKA SA 2008 Direction-dependent variance estimation methods M1: Np=1; Δt = 60 s M2: Np=1; Δt = 150 s M3: Np=1; Δt = 300 s M4: Np=2; Δt = 900 s S1: delete frac. 12.5% S2: delete frac. 25% S3: delete frac. 50% S4: delete frac. 75% MCM1M2 M3M4S1 S2S3 (Kemball et al. (2008), AJ) Truth from MC simulationOther estimates from statistical methods

38. SKA SA 2008 Software cost models Computer operations costs: ~ 10% of system construction costs p.a. Software development costs (Boehm et al. 1981): where β ~ ratio of academic to commerical software construction costs. LSST computing costs approximately one quarter of project; order of magnitude smaller data rates than SKA (~ tens of TB per night); total construction costs perhaps a third of SKA. (LSST)

39. SKA SA 2008 Solutions for the Petascale Era

40. SKA SA 2008 Approaching the SKA petascale challenges Form interdisciplinary institutes and teams: –Computer scientists, computer engineers, and applications scientists –Invest in people not hardware Develop international projects and collaborations Focus on the (multi-wavelength) science goals Revisit current imaging algorithms for extreme scalability Learn from other disciplines in the physical sciences preparing for the petascale era New sociology needed concerning observing and data practices

41. SKA SA 2008 Great Lakes Consortium for Petascale Computation The Ohio State University* Shiloh Community Unit School District #1 Shodor Education Foundation, Inc. SURA – 60 plus universities University of Chicago* University of Illinois at Chicago* University of Illinois at Urbana-Champaign* University of Iowa* University of Michigan* University of Minnesota* University of North Carolina–Chapel Hill University of Wisconsin–Madison* Wayne City High School * CIC universities* Argonne National Laboratory Fermi National Accelerator Laboratory Illinois Math and Science Academy Illinois Wesleyan University Indiana University* Iowa State University Illinois Mathematics and Science Academy Krell Institute, Inc. Louisiana State University Michigan State University* Northwestern University* Parkland Community College Pennsylvania State University* Purdue University* Goal: Facilitate the widespread and effective use of petascale computing to address frontier research questions in science, technology and engineering at research, educational and industrial organizations across the region and nation. Charter Members

42. SKA SA 2008 US SKA calibration & processing working group (TDP) Athol Kemball (Illinois) (Chair) Sanjay Bhatnagar (NRAO) Geoff Bower (UCB) Jim Cordes (Cornell; TDP PI) Shep Doeleman (Haystack/MIT) Joe Lazio (NRL) Colin Lonsdale (Haystack/MIT) Lynn Matthews (Haystack/MIT) Steve Myers (NRAO) Jeroen Stil (Calgary) Greg Taylor (UNM) David Whysong (UCB) Calgary....... Cornell NRL UIUC MIT NRAO UCB UNM

43. SKA SA 2008 Approaching the SKA petascale challenges Form interdisciplinary institutes and teams: –Computer scientists, computer engineers, and applications scientists –Invest in people not hardware Develop international projects and collaborations Focus on the (multi-wavelength) science goals Revisit current imaging algorithms for extreme scalability Learn from other disciplines in the physical sciences preparing for the petascale era New sociology needed concerning observing and data practices