1. www.ci.anl.gov www.ci.uchicago.edu UC3: A Framework for Cooperative Computing at the University of Chicago Lincoln Bryant Computation and Enrico Fermi Institutes US ATLAS and UC3

2. www.ci.anl.gov www.ci.uchicago.edu 2 Outline UC3 background Implementation Published results What’s next? UChicago Computing Cooperative

3. www.ci.anl.gov www.ci.uchicago.edu 3 UC Computing Cooperative Kick off January 2012 Running in beta mode since April 2012 A shared Campus distributed high throughput computing infrastructure (DHTC) Inspired by other campus grids in OSG Integrate with national-scale resources such as the Open Science Grid UChicago Computing Cooperative

4. www.ci.anl.gov www.ci.uchicago.edu 4 IMPLEMENTATION UChicago Computing Cooperative

5. www.ci.anl.gov www.ci.uchicago.edu 5 Basic UC3 infrastructure Identity management and automatic account creation Submit host + Condor infrastructure Open “application seeder” cluster 544 job slots BOSCO multi-user service – for accessing non-Condor schedulers (PBS, SGE) – (see Marco Mambelli’s talk tomorrow) Data server backed by 50 TB of HDFS storage – Globus Online endpoint integrated with CILogon – Data access managed by Chirp, Parrot, and Skeleton Key o (see Suchandra Thapa’s talk tomorrow) Dedicated CVMFS repository for UC3 applications Various monitoring tools UChicago Computing Cooperative

6. www.ci.anl.gov www.ci.uchicago.edu 6 Identity and Accounts & sign-up Signup portal integrated with UC systems UChicago Computing Cooperative David Champion (IT Services UChicago) – pre-production LDAP GROUPER

7. www.ci.anl.gov www.ci.uchicago.edu 7 UC3 Connected Resources HDFS UC3 Seeder Condor ITB PBS Midwest Tier 2 Condor SIRAF SGE UC3 Cloud Condor ITS Condor UC3 BOSCO UC3 Submit UC3 Data UC3 CVMFS NFS Mounts

8. www.ci.anl.gov www.ci.uchicago.edu 8 UC3 Job Routing UC3 Seeder Condor ITB PBS Midwest Tier 2 Condor SIRAF SGE UC3 Cloud Condor ITS Condor UC3 BOSCO NFS Mounts UC3 Submit HDFS UC3 Data UC3 CVMFS

9. www.ci.anl.gov www.ci.uchicago.edu 9 UC3 Data Access HDFS UC3 Seeder Condor ITB PBS Midwest Tier 2 Condor SIRAF SGE UC3 Cloud Condor ITS Condor UC3 Submit UC3 Data UC3 CVMFS UC3 BOSCO CVMFS Repo NFS Mounts

10. www.ci.anl.gov www.ci.uchicago.edu 10 Monitoring & Console UChicago Computing Cooperative Condor viewer, Cycler Server, Sysview, Ganglia

11. www.ci.anl.gov www.ci.uchicago.edu 11 UC3 in production UChicago Computing Cooperative Glow & Engage users via OSG Atlas via OSG UC3 users Unused cycles

12. www.ci.anl.gov www.ci.uchicago.edu 12 SCIENCE RESULTS UChicago Computing Cooperative

13. www.ci.anl.gov www.ci.uchicago.edu 13 Cosmic Microwave Background analysis A MEASUREMENT OF THE COSMIC MICROWAVE BACKGROUND DAMPING TAIL FROM THE 2500-SQUARE-DEGREE SPT-SZ SURVEY K. T. Story (Kavli Institute) et al. “UC3 computing resources were used to simulate 100 full-sky realizations. These simulated skies include gravitationally lensed CMB anisotropy, a Poisson distribution of radio galaxies, and Gaussian realizations of the thermal and kinetic Sunyaev-Zel'dovich (SZ) effects and cosmic infrared background (CIB). The results were used to calculate the transfer function of the SPT SZ analysis pipeline, which is an essential step in the production of the SZ power spectrum from the full 2500 square degree SPT survey” Study power spectrum from South Pole Telescope (SPT) UChicago Computing Cooperative

14. www.ci.anl.gov www.ci.uchicago.edu 14 Cosmic Microwave Background analysis UChicago Computing Cooperative Reprocessing raw data: two weeks on SPT systems  12 hours on UC3 Submitted to ApJ (The Astrophysical Journal) http://arxiv.org/abs/1210.7231

15. www.ci.anl.gov www.ci.uchicago.edu 15 Glassy systems and supercooled liquids Modeling glass formation in supercooled liquids Probing the structure of heterogeneity in supercooled liquids is computationally intense. Glen Hocky, David Reichman (Columbia) Just submitted to Journal Chemical Physics: http://arxiv.org/abs/1211.0033 http://arxiv.org/abs/1211.0033 UChicago Computing Cooperative

16. www.ci.anl.gov www.ci.uchicago.edu 16 Global Gridded Biophysical Modeling Simulate crop yields and climate change impact at high- resolution (global extents, multi-scale models, multiple crops – corn, soy, wheat, rice) Preliminary results on yields versus CO 2 fertilizer Analysis on UC3 – workflows managed by Swift Joshua W. Elliott w/ Michael Glotter, Neil Best, David Kelly, Cheryl Porter, Alex Ruane, Ian Foster, Cynthia Rosenzweig, Elizabeth Moyer, Jim Jones, Ken Boote, Senthold Asseng, Mike Wilde, and other Chicago and AgMIP partners RDCEP collaboration – robust decision making on climate and energy policy

17. www.ci.anl.gov www.ci.uchicago.edu 17 and of course.. ATLAS at LHC UC3 partnering with USATLAS Tier 2 and UC Tier 3 centers Provide flocking to unused ATLAS resources Allow flocking of ATLAS to spare UC3 cycles Facilitated with CERN Virtual File System for release directories, and federated Xrootd for storage access (  minimal UC3 system modifications for a large class of jobs) UChicago Computing Cooperative 100 k Fabiola Gianotti (CERN), ATLAS Collaboration Running ATLAS jobs

18. www.ci.anl.gov www.ci.uchicago.edu 18 ATLAS production flocking into Campus UChicago Computing Cooperative ATLAS Only UC3 seeder shown Will transition some UC Tier3 users to UC3 using Parrot

19. www.ci.anl.gov www.ci.uchicago.edu 19 LOOKING AHEAD UChicago Computing Cooperative

20. www.ci.anl.gov www.ci.uchicago.edu 20 Future connectivity Flocking to the new Research Computing Center’s “Midway” cluster at UC. – About 4500 cores we could flock jobs to. – Looking forward to SLURM support in BOSCO. Flocking out to OSG. – UC3 collective VO established in OSG – Submission to remote sites on OSG via GlideinWMS – First step: just use OSG VO and flocking host to VO front-end managed by Mats Rynge UChicago Computing Cooperative

21. www.ci.anl.gov www.ci.uchicago.edu 21 In summary… UC3 has been up and running since April – 544 dedicated job slots for UC3 open (seeder) use – Theoretical maximum is over 6k slots (flockable max) – 50 TB dedicated (Hadoop) storage for staging job datasets or temporary storage – uc3 cvmfs repo + Parrot for software access Login host integrated with UC ID system UC3 Globus Online endpoint – Integrating with CILogon Chirp access to storage Hadoop storage Three groups using UChicago Computing Cooperative

22. www.ci.anl.gov www.ci.uchicago.edu Thank you!

23. www.ci.anl.gov www.ci.uchicago.edu 23 Collaboration and Acknowledgements Enrico Fermi Institute in the Physical Sciences Division – ATLAS Collaboration (HEP) – South Pole Telescope Collaboration Computation Institute at UC (OSG, Swift) Departments of Radiology and Radiation Oncology (SIRAF project) Center for Robust Decision Making on Climate and Energy Policy group at UC (CI, Economics) UC Information Technology Services UC Research Computing Center SWIFT team UChicago Computing Cooperative

24. www.ci.anl.gov www.ci.uchicago.edu 24UChicago Computing Cooperative