1. 1 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium The Open-Science-Grid: Building a US based Grid infrastructure for Open Science http://www.opensciencegrid.org/ http://www.opensciencegrid.org/

2. 2 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium History, from the trillium to Grid3 to OSG OSG –Governance and principles –Operation, release cycles Proposed program of work Status of OSG activities Summary

3. 3 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium History, from the trillium to Grid3 to OSG OSG –Governance and principles –Operation, release cycles Proposed program of work Status of OSG activities Summary

4. 4 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Once upon a time in America … Trillium Meeting, May 20 th 2004 In the U.S., three big Grid collaborationIn the U.S., three big Grid collaboration –iVDGL (International Virtual data Grid Laboratory) –GriPhyn (Grid Physics Network); VDT –PPDG (Particle Physics Data Grid) Met with two primary goals –Identify and Plan for Joint Projects for 2004 i.e. “Grid2003(s) for 2004” –Bring into initial existence the Open Science Grid Consortium

5. 5 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Open Science Grid Build on and evolve Grid3 towards a sustained production common grid infrastructure through “grass-roots” collaboration. Goals –Continue collaboration of US-ATLAS & US-CMS. Provide base for US Grid contributions to LHC –Expand Grid3 participants and users to all PPDG and to all HENP experiments. –Support existing VO in the Trillium (CDF, D0, STAR, –Invite and encourage other sciences to participate, contribute and benefit: molecular biology, genetics, protein chemistry, nanotechnology, climate, geophysics of the earth and the hydrodynamics of the ocean, human reasoning, economics, natural language processing, behavioral psychology, geographic information science and many more. –Invite partners across the world to “plug” to a common, scalable and shared infrastructure

6. 6 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Our vision http://www.opensciencegrid.org http://www.opensciencegrid.org

7. 7 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium What is it? It is a Consortium, a collaboration of many institutions. –Universities and Labs, projects, experiments, middleware, providers, campus Grids Who want to leverage their efforts by joining together to build a sustainable infrastructure for physics and other sciences Agreement by architects on some fundamental goals and scope that we reference often: Blueprint

8. 8 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium History, from the trillium to Grid3 to OSG Organization –Governance and principles –Operation, release cycles Proposed program of work Status of OSG activities Summary

9. 9 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Consortium Organization 2004-2005 Interim providers

10. 10 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium ~ 1 year after the trillium meeting OSG Integration Testbed had 20 sites and ITB infrastructure has had 3 releases. –US-ATLAS and US-CMS have committed to provide resources to ATLAS and CMS experimenters through the Open Science Grid as part of the global “LHC Computing Grid”. –STAR, CDF, D0, STAR, BaBar ready-ing their applications & infrastructure for the common grid: –CDF run simulation on an OSG ITB site –D0 running re-reconstruction on US CMS Tier-1 Grid3. –STAR running on expanded to more sites,.. –…

11. 11 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium And even more success … Approached –TeraGrid Grid Integration Group (GIG) –LCG & EGEE contacts added to OSG activities Security, Operations, Storage, Interoperability, Accounting Seeing partners and interest in joining this Grid outside of Physics –DES –DOSAR –GLOW –nanoHUB –SURA … Official Opening of the OSG facility: October 2005 … and lots of activities, … Have liaison on the council

12. 12 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Organization reshape … X Expanding => Difficult to have sustained practical effort. Went from 8 to 15 Interim

13. 13 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Projects Project Mangers Resource Managers … Projects Project Mangers Resource Managers … OSG Council Council Chair OSG Council Council Chair Contributors Universities Laboratories Sites Service Providers VOs Researchers Computer Science Grid Projects … & Partners Campus Grid EGEE TeraGrid Scientific Advisory Group OSG Users Group Applications Coordinators Program Oversight Finance Board Resource Manager OSG Facility Facility coordinator Engagement Coordinator Operations Coordinator Middleware Coordinator Security Officer OSG Consortium Projects Project managers Resource Managers Executive Board Executive Director Applications Coordinator Education Coordinator Facility Coordinator Resources Manager Council Chair Engagement Coordinator Middleware Coordinator Operations Coordinator Security Officer Liaison to EU Grid Projects Liaison to TG/USGrid Projects Project Technical Managers Contributor Technical Managers Executive Team Line reporting Advisory Contributing & Interfacing, in some cases through MOUs

14. 14 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium New structure Open Science Grid Council –governs the Consortium ensuring effective benefit to the scientific mission of the stakeholders. Scientific Advisory Group -– a set of wise people to provide advise to the Council and Executive Board. Executive Director and Executive Board –manages the OSG program of work. Resources Manager and Finance Board –manages all matters related to the costs and resources of the OSG program of work. Facility Coordinator and Facility Coordination Group - Operations Coordinator, Security Officer, Middleware Coordinator, Engagement Coordinator –manages all aspects of the OSG infrastructure and distributed facility. Applications Coordinator and OSG Users Group –coordinates and oversees end to end functionality required by applications from requirements, developments through collaboration with external projects, and testing, until inclusion of such functionality in the facility. Education Coordinator – responsible for education and outreach activities.

15. 15 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Further references More documents defines the consortium and its operation –Blueprint –Bylaws –Voting procedures –User agreement –Service agreements http://www.opensceincegrid.org/

16. 16 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium History, from the trillium to Grid3 to OSG Organization –Governance and principles –Operation, release cycles Proposed program of work Status of OSG activities Summary

17. 17 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Its principles OSG Infrastructure - a Grid of Grids –Infrastructure defined by Interfaces, Services and Policies. –Campus, Experiment, (Commercial?), “Other” Grids present Resources and Services to the Common Infrastructure. –Users and Organization environments interface to a consistent set of resources - across Grid boundaries Operation’s principles –Common Interfaces not necessarily Heterogeneous Infrastructure - Service and Infrastructure release. –Autonomy of Sites and Resources Overriding Policy Enforcement Move to “pull” model for Work –Experiment (VO) based environment, and management Common services with VO-specific –Mix of Assured and Opportunistic Latter has minimal expectations performance and support.

18. 18 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium OSG Scales In 2003, Grid3 – Now OSG –3500 CPUs across 30 sites -> Pool has doubled –Number of running VO doubled Complexity of how to manage the infrastructure and software seems to follow a cyclic expansion …

19. 19 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Follows a hierarchical structure –GOC VO support center site supported by VO support center end users / customer Two main release packages –Integration test bed (ITB) –Production release Bootstrap mechanism –Each ITB release is an evolutional step adding, removing, and updating the functional set of services –The test bed is used to identify issues of scalability, usability and incompatibility of services and service versions. Two stage release provides increasing stability OSG Operations To makes this work …

20. 20 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium

21. 21 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Stable != static Path for new services into OSG (RWG, 2005) OSG Deployment Activity Metrics & Certification Application validation VO Application Software Installation OSG Integration Activity Release Description Middleware Interoperability Software & packaging Functionality & Scalability Tests Readiness plan adopted Service deployment OSG Operations-Provisioning Activity Release Candidate Readiness plan Effort Resources feedback

22. 22 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium OSG Facility Operation: Operations, Maintenance Engagement Support OSG Facility Operation: Operations, Maintenance Engagement Support OSG Extensions Requirements development, & testing on parochial grids Resources Providers (Sites) Applications OSG Facility Provisioning: VDT, Integration, Validation, System Integration Testbed OSG Facility Provisioning: VDT, Integration, Validation, System Integration Testbed Release External Projects: Development & Research Ready? Bringing new capabilities into the Facility

23. 23 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium History, from the trillium to Grid3 to OSG Organization –Governance and principles –Operation, release cycles Proposed program of work Status of OSG activities Summary

24. 24 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Roadmap for OSG The consortium –brings together a unique ensemble of domain scientists, software developers and providers of computing resources –share a common goal to stimulate new discoveries by providing scientists with effective and dependable access to a national distributed computational facility The OSG is maturing –Governance and operation procedure –Number of VO and components represents a wide(r) range of scientific needs and application –Trillium ends this coming year … How do we keep going??

25. 25 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium OSG proposal The OSG Council is proposing a joint 5 year program of work to DOE and NSF. This program of work consists of –The Distributed Facility –Extensions to the Facility –Education and Training

26. 26 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium OSG program of work

27. 27 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium History, from the trillium to Grid3 to OSG Organization –Governance and principles –Operation, release cycles Proposed program of work Status of OSG activities Summary

28. 28 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Objectives met

29. 29 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Documentation up to date Easy to follow instructions are available on the web – Similar document exists for SE validation

30. 30 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium

31. 31 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Registered VO New VO

32. 32 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Service challenge Exercise by CMS service challenge –Submission and completion efficiency still need to be improved –Many of the failures were uniquely attributable to CMS First large scale analysis attempt for CMS on OSG –Increasing user participation on OSG analysis to the whole collaboration and improving the experience are part of the 2006 program of work 78% Lower limit !!

33. 33 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium http://grid02.uits.indiana.edu:8080/reports/daily/ Sustained # jobs > 1000 / day in Atlas

34. 34 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Include OSG configured EGEE/LCG Generic Information Providers (based on Glue Schema) for interoperability with LCG. Supplied configuration scripts back to LCG WLCG monitoring OSG sites that have these installed ( jobs submitted through RB run on OSG sites that accept them) - Work in progress. OSG Release 0.4

35. 35 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium History, from the trillium to Grid3 to OSG OSG –Governance and principles –Operation, release cycles Proposed program of work Status of OSG activities Summary

36. 36 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium A US-based infrastructure, for world-wide collaborators and grids While the backbone is is US based (Labs, universities, …), several sites are from partners abroad: South Africa, Taiwan, South Korea, Brazil, Argentina, … Perhaps India?

37. 37 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Adding to the OSG Several ways, described by site and user agreements, bylaws, VO registration, … –Addition as part of an existing VO: only requires VO head approval, SA, UA, … –An experiment may join the OSG with an activity MOU, resources, … –A group / sub-group, defined by a project (external funding) can come forward and “plug” to the OSG. The OSG offers a superb opportunity for CS and other science research. New additions godfathered by an existing VO A case is presented to the OSF council. Vote is by consensus. OSG relies on several external projects –Storage Management CET –Distributed Science CET –Object-on-demand SAP –Condor Edge service –VDS, EGEE workflow –gLite and security middleware –TG-GIG –Ulltralight, PLaNetS PIF –…

38. 38 IWLSC, Kolkata India 2006 Jérôme Lauret for the Open Science Grid consortium Summary The Open Science Grid is a mature consortium –Well structure, brings together a broad, talented and committed team –New VO and science, span across all ranks of the scientific community –Naturally matched to the needs of large-scale parameter sweep tasks of experimental science (MC, large scale, …) Leveraging on the VDT’s, Grid3, PPDG experience –OSG releases provides multi-science, multi-grid support –New IT technologies are developed in response to explicit needs and are evaluated in a real-life setting –Counts projects Becomes a home for an increasingly diverse set of sciences in the US and across the world