1. OSG All Hands, LIGO Livinston Lab, 2009 The State of the Open Science Grid Ruth Pordes Executive Director

2. OSG All Hands, LIGO Livinston Lab, 2009 Summary of the Collaboratory OSG Consortium in existence since 2005. OSG Project in 3 rd year of 5 year funding cycle. Core collaboration remains, strengthened by new participants and openness to new approaches.  US ATLAS, US CMS, & LIGO with accomplishments on the ground, open collaboration to face realities and changing needs.  Involvement of other physics increasing (CDF, D0, Alice)  Processes of engaging new non-physics users and communities being “shaken out”.  Leadership remains, strengthened by an excellent team and energetic contributors Challenges and issues from detail to the national level continue - no shortage of work! 2

3. OSG All Hands, LIGO Livinston Lab, 2009 Summary of the Operations We are successfully continuing to operate a large, shared, distributed system. We are making more than promised contributions to the World Wide LHC Computing Grid (WLCG). 3 >2500 people used resources through the OSG in 2008. We deployed 1 st software version labelled Production Level OSG 1.0 We have improved monitoring, administrative services and security response processes.

4. OSG All Hands, LIGO Livinston Lab, 2009 4 NERSC BU UNM SDSC UTA OU FNAL ANL WISC BNL VANDERBILT PSU UVA CALTECH PURDUE IU BUFFALO TTU CORNELL ALBANY UMICH IUPUI STANFORD CU UWM UNL UFL KU UNI WSU MSU LTU LSU CLEMSON UMISS UIUC UCR UCLA LEHIGH NSF ORNL HARVARD UIC SMU UCHICAGO MIT RENCI LBL GEORGETOWN UIOWA UDAVIS ND UCSB FIT UMD T RIT The Map of OSG Sites (in the US)

5. OSG All Hands, LIGO Livinston Lab, 2009 Summary of the Foundations We are paying attention to and augmenting our principles and blueprint  Enabling automony  Community grids  Grid bridges & federations  End-to-end security  Accommodating clouds Our contributions are bring broadened through partnerships. 5

6. OSG All Hands, LIGO Livinston Lab, 2009 The Principles at Work 6 Hardware Science Applications VO Services OSG software & services Shared Grid VO Specific Middleware … … … Site fabric: OS, batch, storage … Sites have autonomy, and are responsible for their security. We are open to new sites. Sites report to OSG’s middleman services. VOs have autonomy, and are responsible for their security. We are open to new VOs. Sites use OSG’s middleman services. Site Specific Middleware

7. OSG All Hands, LIGO Livinston Lab, 2009 Improvements in Technologies in 2008 Integrated into the Software Stack the Virtual Data Toolkit (now at ~70 components) Initial use of opportunistic storage for science output (D0) Early adoption in physics and generalization of “overlay” job scheduling or “pilot” technologies Resource service validation framework and (security, functionality) probes for monitoring site configurations and services Site Resource selection and OSG matchmaking services in more general use 7

8. OSG All Hands, LIGO Livinston Lab, 2009 LIGO Exemplar of Community Grid providing internal community-based security and resource management while maintaining interfaces and interoperability with the common cyber-infrastructure. All LIGO science depends on OSG software services through the VDT. Leaders in testing of Web Services at scale on OSG. Einstein@Home providing science from OSG both through direct submission and BOINC backfill mechanisms. 8

9. OSG All Hands, LIGO Livinston Lab, 2009 VO throughput over past year >40 sites contributing 9 20,000 CPUdays/day ATLAS & CMS CDF & D0 The Rest

10. OSG All Hands, LIGO Livinston Lab, 2009 US ATLAS, US CMS, (US ALICE) and the Worldwide LHC Computing Grid The US delivered >30% of throughput, robustly, to the global experiments. Together OSG & US LHC influence the directions of the WLCG  Pilot/Pull mode job scheduling  Attention to federated infrastructures  Parterning with EU (EGEE, emerging EGI) on security, operations, technologies etc. Evaluating with US Alice (Nuclear Physics) use of OSG while DOE evaluating their computing support. 10

11. OSG All Hands, LIGO Livinston Lab, 2009 11 Science Output CDF + D0 >100 physics publications in 2008; STAR >10 publications in 2008.

12. OSG All Hands, LIGO Livinston Lab, 2009 Engagement and Support Beyond Physics Engagement team provides common point of entry for new communities  Facilitating to build Campus-wide infrastructures.  Helping modify user legacy codes,  Communicating to adapt community cultural processes,  Planning with leadership to identify realities for a win or a no-win.  Managing group production until threshold of entry is passed. Usage naturally very cyclic – achievements in computational throughput need to be balanced by  Analysing results  Developing algorithms  Socializing the change in model in the organization 12

13. OSG All Hands, LIGO Livinston Lab, 2009 Non-Physics VOs & Usage – 2008 Applications smoothly come in, use resources and disappear (e.g. Glow, protein mapping) At max <~4,000 cpudays/day queue times increased. Total shows significant “cycles of use”. 13 1,000 CPUdays/day

14. OSG All Hands, LIGO Livinston Lab, 2009 Science Output ~5 non-physics publications. + “raptor” protein structure prediction ranked 2 in the world. (publication in review) 14

15. OSG All Hands, LIGO Livinston Lab, 2009 OSG & Satellite Partnerships Follow the model we discussed in “Trillium” days.... encourges efforts to adiabatically extend participating groups and research communities through collaborative proposals that benefit and make leveraged contributions... Embedded Immersive Engagement for Cyberinfrastructure – John McGee, Sebastien Goasguen (OCI) Structural Biology Grid – Piotr Sliz, Ian Stokes-Rees(MCB) Delegating Organizational Work to Virtual Organization Technologies: Beyond the Communicational Paradigm Office of CyberInfrastructure – David Ribes, Tom Finholt (OCI) 15

16. OSG All Hands, LIGO Livinston Lab, 2009 16 VOSS: Cross Organizational Analysis from David Ribes and Thomas Finholt

17. OSG All Hands, LIGO Livinston Lab, 2009 Collaborations cont: SciDAC, ESNET/Internet2, HPC/TeraGrid SciDAC collaborations ESNET/Internet2  OSG users rely on the production and research networks of ESNET and Internet2 (and LHCNET).  Broader deployment of monitoring tools (perfsonar in VDT) DOE HPC & MPI  NERSC OSG allocation has given us some limited experience.  Goal in FY09 to make smooth use across multiple sites supporting MPI.  Working on initial request to get data in/out of ANL Leadership Class Machine with GridFTP for Lattice QCD. TeraGrid  Security incident response coordination/communication  OSG training traditionally includes User access to TG resources  Nanohub Gateway runs across OSG and TG sites.  Other work items in discussion 17

18. OSG All Hands, LIGO Livinston Lab, 2009 DOE SciDAC/NSF Review Jan 2009 Encouragement to Proceed: “Continue to engage in the larger strategic questions of national cyberinfrastructure(s) among relevant parties. Continue to iterate on the best rules and framework for federation – evolutionary and strategic for all large cyberinfrastructures – be focused and need-driven (i.e. archival storage). Create conditions at the right levels for discussion of frontier issues of cyberinfrastructures.” 18

19. OSG All Hands, LIGO Livinston Lab, 2009 2008 Adjustments in Our Organization for Increased Production Focus: Inclusion of Communities in production activities; addition of production coordinator (Dan) Increased importance of software: the Software Tools Group (Alain, Mine). Importance of Computer Science: Addition of Computer Science Student Fellowship. Targetted International Outreach (Alina): Americas Initiative (Jose); International Grid Schools (Miron); (Finally) the site in South Africa. 19

20. OSG All Hands, LIGO Livinston Lab, 2009 Improve and Sustain the Software Stack Through the (new) software tools group we will be able to provide more focus and effort on interactions with software providers, enabling new software capabilities, and transition new techologies to the production infrastructure. Based on our unique environment for testing and use of technologies (at scale, under fire, and by multiple communities) we will continue to look for new software providers who can best advance the common technologies our stakeholders need. 20

21. OSG All Hands, LIGO Livinston Lab, 2009 Evolving to the Needs of the Future? Expanding the communities and partnerships especially with Campus. Managing change and increased heterogeneity Open Science GridS Security and trust (attention, defense, protection) Really (really) provide for ad-hoc group collaborations Extended computing paradigms – Workspaces, Clouds, VMs 21

22. OSG All Hands, LIGO Livinston Lab, 2009 OSG and the National Cyberinfrastructure ? How can OSG’s experience on the Campuses be of most value? What is OSG’s role and place in partnering with TeraGrid? How can OSG contribute to the usability of the Leadership Class Facilities? How does OSG contribute to software sustainability? Is OSG an exemplar from which Europe in its next phase of National Grids can benefit? 22