The Science Data Processor and Regional Centre Overview Paul Alexander UK Science Director the SKA Organisation Leader the Science Data Processor Consortium

SKA: A Leading Big Data Challenge for 2020 decade Antennas Digital Signal Processing (DSP) High Performance Computing Facility (HPC) Transfer antennas to DSP 2020: 5,000 PBytes/day 2030: 100,000 PBytes/day Over 10’s to 1000’s kms To Process in HPC 2020: 50 PBytes/day 2030: 10,000 PBytes/day Over 10’s to 1000’s kms HPC Processing 2020: 300 PFlop 2028: 30 EFlop

XXXXXX SKY Image Detect & amplify Digitise & delay Correlate Process Calibrate, grid, FFT Integrate s B B. s 12 Astronomical signal (EM wave) Visibility: V(B)= E 1 E 2 * = I(s) exp( i  B.s/c ) Resolution determined by maximum baseline  max ~ / B max Field of View (FoV) determined by the size of each dish  dish ~ / D Standard interferometer

Image formation computationally intensive Images formed by an inverse algorithm similar to that used in MRI Typical image sizes up to: 30k x 30k x 64k voxels

Must also solve for calibration errors

Challenge Very Dependent on Experiment

One SDP Two Telescopes Ingest (GB/s) SKA1_Low500 SKA1_Mid1000 In total need to deploy eventually a system which is close to 0.5 EFlop of processing

… and regional centres

Scope of the SDP

The SDP System

Imaging Processing Model Correlator RFI excision and phase rotation

Performance Requirements Imaging and calibration determines system sizing Data is Buffered after Ingest Double buffered Buffer size >100 PByte Imaging must account for very large field of view o Algorithms complex and computationally expensive o Will need to evolve algorithms during life of the telescope Data products will be calibrated multi- dimensional images and time-series data Volume of potential data products very large May only be able archive data specific to observation requested Commensal fast-imaging mode for slow transients Processing to be achieved (Pflop) Ingest (GB/s) SKA1_LOW25500 SKA1_Mid Detailed analysis is complex

Illustrative Computing Requirements ~100 PetaFLOPS total achieved ~200 PetaByte/s aggregate BW to fast working memory ~50 PetaByte Storage ~1 TeraByte/s sustained write to storage ~10 TeraByte/s sustained read from storage –~ FLOPS/byte read from storage 13

SDP High-Level Architecture

How do we get performance and manage data volume? Graph-based approach Hadoop

Data Driven Architecture

o Smaller FFT size at cost of data duplication

SDP Data Flow Approach: Next Generation Data Science Engine?

Hardware Platform

Wilkes Built November 2013, HPCS/SDP + Dell, NVIDIA and Mellanox UK’s fastest academic cluster 128 Dell T620 servers 256 NVIDIA K20 GPUs 256 Mellanox Connect InfiniBand cards + switch Performance of 240TFlop with a Top500 position of 166 Funded: STFC, SKA, and industrial sponsorship from Rolls Royce and Mitsubishi Heavy Industries. most energy efficient air cooled supercomputer in the world second in Green500 with 3631 MFLOP/w most energy efficient air cooled supercomputer in the world second in Green500 with 3631 MFLOP/w

3-30 EBytes / year of fully processed data for SKA2 Data rates and processing increase by FACTOR ~100 for SKA2

END