Ian Foster Computation Institute Argonne National Lab & University of Chicago Towards an Open Analytics Environment (A “Data Cauldron”)

2 An Open Analytics Environment Results out Data in Programs & rules in “No limits”  Storage  Computing  Format  Program Allowing for  Versioning  Provenance  Collaboration  Annotation

3 Towards an Open Analysis Environment: (1) Applications l Astrophysics l Cognitive science l East Asian studies l Economics l Environmental science l Epidemiology l Genomic medicine l Neuroscience l Political science l Sociology l Solid state physics

4 Towards an Open Analysis Environment: (2) Hardware SiCortex 6K cores, 6 Top/s IBM BG/P 160K cores, 500 Top/s PADS PADS Gbit/s

5 PADS: Petascale Active Data Store 500 TB reliable storage (data & metadata) 180 TB, 180 GB/s 17 Top/s analysis Data ingest Dynamic provisioning Parallel analysis Remote access Offload to remote data centers P A D S Diverse users Diverse data sources 1000 TB tape backup

6 Towards an Open Analysis Environment : (3) Methods l HPC systems software (MPICH, PVFS, ZeptOS) l Collaborative data tagging (GLOSS) l Data integration (XDTM) l HPC data analytics and visualization l Loosely coupled parallelism (Swift, Hadoop) l Dynamic provisioning, data diffusion (Falkon) l Service authoring (Introduce, caGrid, gRAVI) l Provenance recording and query (Swift) l Service composition and workflow (Taverna) l Virtualization management (Nimbus) l Distributed data management (GridFTP, etc.)

7 Towards an Open Analysis Environment : (3) Methods l HPC systems software (MPICH, PVFS, ZeptOS) l Collaborative data tagging (GLOSS) l Data integration (XDTM) l HPC data analytics and visualization l Loosely coupled parallelism (Swift, Hadoop) l Dynamic provisioning, data diffusion (Falkon) l Service authoring (Introduce, caGrid, gRAVI) l Provenance recording and query (Swift) l Service composition and workflow (Taverna) l Virtualization management (Nimbus) l Distributed data management (GridFTP, etc.)

8 High-Performance Data Analytics Functional MRI Ben Clifford, Mihael Hatigan, Mike Wilde, Yong Zhao

9./group23 drwxr-xr-x 4 yongzh users 2048 Nov 12 14:15 AA drwxr-xr-x 4 yongzh users 2048 Nov 11 21:13 CH drwxr-xr-x 4 yongzh users 2048 Nov 11 16:32 EC./group23/AA: drwxr-xr-x 5 yongzh users 2048 Nov 5 12:41 04nov06aa drwxr-xr-x 4 yongzh users 2048 Dec 6 12:24 11nov06aa. /group23/AA/04nov06aa: drwxr-xr-x 2 yongzh users 2048 Nov 5 12:52 ANATOMY drwxr-xr-x 2 yongzh users Dec 5 11:40 FUNCTIONAL. /group23/AA/04nov06aa/ANATOMY: -rw-r--r-- 1 yongzh users 348 Nov 5 12:29 coplanar.hdr -rw-r--r-- 1 yongzh users Nov 5 12:29 coplanar.img. /group23/AA/04nov06aa/FUNCTIONAL: -rw-r--r-- 1 yongzh users 348 Nov 5 12:32 bold1_0001.hdr -rw-r--r-- 1 yongzh users Nov 5 12:32 bold1_0001.img -rw-r--r-- 1 yongzh users 348 Nov 5 12:32 bold1_0002.hdr -rw-r--r-- 1 yongzh users Nov 5 12:32 bold1_0002.img -rw-r--r-- 1 yongzh users 496 Nov 15 20:44 bold1_0002.mat -rw-r--r-- 1 yongzh users 348 Nov 5 12:32 bold1_0003.hdr -rw-r--r-- 1 yongzh users Nov 5 12:32 bold1_0003.img XDTM: XML Data Typing & Mapping Logical Physical

10 Tagging & Social Networking GLOSS: Generalized Labels Over Scientific data Sources Svetlozar Nestorov and others

11 start report DOCK6 Receptor (1 per protein: defines pocket to bind to) ZINC 3-D structures ligandscomplexes NAB script parameters (defines flexible residues, #MDsteps) Amber Score: 1. AmberizeLigand 3. AmberizeComplex 5. RunNABScript end BuildNABScript NAB Script NAB Script Template Amber prep: 2. AmberizeReceptor 4. perl: gen nabscript FRED Receptor (1 per protein: defines pocket to bind to) Manually prep DOCK6 rec file Manually prep FRED rec file 1 protein (1MB) 6 GB 2M structures (6 GB) DOCK6 FRED ~4M x 60s x 1 cpu ~60K cpu-hrs Amber ~10K x 20m x 1 cpu ~3K cpu-hrs Select best ~500 ~500 x 10hr x 100 cpu ~500K cpu-hrs GCMC PDB protein descriptions Select best ~5K For 1 target: 4 million tasks 500,000 cpu-hrs (50 cpu-years) (Mike Kubal, Benoit Roux, and others)

12 DOCK on BG/P: ~1M Tasks on 118,000 CPUs l CPU cores: l Tasks: l Elapsed time: 7257 sec l Compute time: CPU years l Average task time: 667 sec l Relative Efficiency: 99.7% l (from 16 to 32 racks) l Utilization: u Sustained: 99.6% u Overall: 78.3% GPFS 1 script (~5KB) 2 file read (~10KB) 1 file write (~10KB) RAM (cached from GPFS on first task per node) 1 binary (~7MB) Static input data (~45MB) Ioan Raicu Zhao Zhang Mike Wilde Time (secs)

13 Managing 160,000 Cores Slower shared storage High-speed local “disk” Falkon

14 Efficiency (relative to no-IO case) for 4 second tasks and data sizes 1KB to 1MB for CIO and GPFS up to 32K processors

15 “MI” workload, 250K tasks, 10MB:10ms ratio, up to 64 nodes using DRP, GCC policy, 2GB caches/node

16 “Sine” workload, 2M tasks, 10MB:10ms ratio, 100 nodes, GCC policy, 50GB caches/node

17 SW workload, 2M tasks, 10MB:10ms ratio, up to 100 nodes with DRP, GCC policy, 50GB caches/node

18