UC3: A Framework for Cooperative Computing at the University of Chicago Lincoln Bryant Computation and Enrico Fermi Institutes US ATLAS and UC3

2 Outline UC3 background Implementation Published results What’s next? UChicago Computing Cooperative

3 UC Computing Cooperative Kick off January 2012 Running in beta mode since April 2012 A shared Campus distributed high throughput computing infrastructure (DHTC) Inspired by other campus grids in OSG Integrate with national-scale resources such as the Open Science Grid UChicago Computing Cooperative

4 IMPLEMENTATION UChicago Computing Cooperative

5 Basic UC3 infrastructure Identity management and automatic account creation Submit host + Condor infrastructure Open “application seeder” cluster 544 job slots BOSCO multi-user service – for accessing non-Condor schedulers (PBS, SGE) – (see Marco Mambelli’s talk tomorrow) Data server backed by 50 TB of HDFS storage – Globus Online endpoint integrated with CILogon – Data access managed by Chirp, Parrot, and Skeleton Key o (see Suchandra Thapa’s talk tomorrow) Dedicated CVMFS repository for UC3 applications Various monitoring tools UChicago Computing Cooperative

6 Identity and Accounts & sign-up Signup portal integrated with UC systems UChicago Computing Cooperative David Champion (IT Services UChicago) – pre-production LDAP GROUPER

7 UC3 Connected Resources HDFS UC3 Seeder Condor ITB PBS Midwest Tier 2 Condor SIRAF SGE UC3 Cloud Condor ITS Condor UC3 BOSCO UC3 Submit UC3 Data UC3 CVMFS NFS Mounts

8 UC3 Job Routing UC3 Seeder Condor ITB PBS Midwest Tier 2 Condor SIRAF SGE UC3 Cloud Condor ITS Condor UC3 BOSCO NFS Mounts UC3 Submit HDFS UC3 Data UC3 CVMFS

9 UC3 Data Access HDFS UC3 Seeder Condor ITB PBS Midwest Tier 2 Condor SIRAF SGE UC3 Cloud Condor ITS Condor UC3 Submit UC3 Data UC3 CVMFS UC3 BOSCO CVMFS Repo NFS Mounts

Monitoring & Console UChicago Computing Cooperative Condor viewer, Cycler Server, Sysview, Ganglia

UC3 in production UChicago Computing Cooperative Glow & Engage users via OSG Atlas via OSG UC3 users Unused cycles

SCIENCE RESULTS UChicago Computing Cooperative

Cosmic Microwave Background analysis A MEASUREMENT OF THE COSMIC MICROWAVE BACKGROUND DAMPING TAIL FROM THE 2500-SQUARE-DEGREE SPT-SZ SURVEY K. T. Story (Kavli Institute) et al. “UC3 computing resources were used to simulate 100 full-sky realizations. These simulated skies include gravitationally lensed CMB anisotropy, a Poisson distribution of radio galaxies, and Gaussian realizations of the thermal and kinetic Sunyaev-Zel'dovich (SZ) effects and cosmic infrared background (CIB). The results were used to calculate the transfer function of the SPT SZ analysis pipeline, which is an essential step in the production of the SZ power spectrum from the full 2500 square degree SPT survey” Study power spectrum from South Pole Telescope (SPT) UChicago Computing Cooperative

Cosmic Microwave Background analysis UChicago Computing Cooperative Reprocessing raw data: two weeks on SPT systems  12 hours on UC3 Submitted to ApJ (The Astrophysical Journal)

Glassy systems and supercooled liquids Modeling glass formation in supercooled liquids Probing the structure of heterogeneity in supercooled liquids is computationally intense. Glen Hocky, David Reichman (Columbia) Just submitted to Journal Chemical Physics: UChicago Computing Cooperative

Global Gridded Biophysical Modeling Simulate crop yields and climate change impact at high- resolution (global extents, multi-scale models, multiple crops – corn, soy, wheat, rice) Preliminary results on yields versus CO 2 fertilizer Analysis on UC3 – workflows managed by Swift Joshua W. Elliott w/ Michael Glotter, Neil Best, David Kelly, Cheryl Porter, Alex Ruane, Ian Foster, Cynthia Rosenzweig, Elizabeth Moyer, Jim Jones, Ken Boote, Senthold Asseng, Mike Wilde, and other Chicago and AgMIP partners RDCEP collaboration – robust decision making on climate and energy policy

and of course.. ATLAS at LHC UC3 partnering with USATLAS Tier 2 and UC Tier 3 centers Provide flocking to unused ATLAS resources Allow flocking of ATLAS to spare UC3 cycles Facilitated with CERN Virtual File System for release directories, and federated Xrootd for storage access (  minimal UC3 system modifications for a large class of jobs) UChicago Computing Cooperative 100 k Fabiola Gianotti (CERN), ATLAS Collaboration Running ATLAS jobs

ATLAS production flocking into Campus UChicago Computing Cooperative ATLAS Only UC3 seeder shown Will transition some UC Tier3 users to UC3 using Parrot

LOOKING AHEAD UChicago Computing Cooperative

Future connectivity Flocking to the new Research Computing Center’s “Midway” cluster at UC. – About 4500 cores we could flock jobs to. – Looking forward to SLURM support in BOSCO. Flocking out to OSG. – UC3 collective VO established in OSG – Submission to remote sites on OSG via GlideinWMS – First step: just use OSG VO and flocking host to VO front-end managed by Mats Rynge UChicago Computing Cooperative

In summary… UC3 has been up and running since April – 544 dedicated job slots for UC3 open (seeder) use – Theoretical maximum is over 6k slots (flockable max) – 50 TB dedicated (Hadoop) storage for staging job datasets or temporary storage – uc3 cvmfs repo + Parrot for software access Login host integrated with UC ID system UC3 Globus Online endpoint – Integrating with CILogon Chirp access to storage Hadoop storage Three groups using UChicago Computing Cooperative

Thank you!

Collaboration and Acknowledgements Enrico Fermi Institute in the Physical Sciences Division – ATLAS Collaboration (HEP) – South Pole Telescope Collaboration Computation Institute at UC (OSG, Swift) Departments of Radiology and Radiation Oncology (SIRAF project) Center for Robust Decision Making on Climate and Energy Policy group at UC (CI, Economics) UC Information Technology Services UC Research Computing Center SWIFT team UChicago Computing Cooperative

24UChicago Computing Cooperative