Open Science Grid at Condor Week Ruth Pordes Fermilab April 25th 2006

Outline OSG goals and organization Drivers and use today Middleware Focus and roadmap 2/21/2019

What, Who is Open Science Grid? High Throughput Distributed Facility Shared opportunistic access to existing clusters, storage and networks. Owner controlled resources and usage policies. Supports Science Funded by NSF and DOE projects. Common technologies & cyber-infrastructure. Open & Inclusive Collaboration of users, developers, grid technologists & facility administrators. Training & help for existing & new administrators and users Heterogeneous 2/21/2019

OSG Organization 2/21/2019

OSG Organization Executive Director: Ruth Pordes Facility Coordinator: Miron Livny Application Coordinators: Torre Wenaus & fkw Resource Managers: P. Avery & A. Lazzarini Education Coordinator: Mike Wilde Engagement Coord.: Alan Blatecky Middleware Coord.: Alain Roy Ops Coordinator: Leigh Grundhoefer Security Officer: Don Petravick Liaison to EGEE: John Huth Liaison to Teragrid: Mark Green Council Chair: Bill Kramer Condor project 2/21/2019

OSG Drivers: LIGO- gravitational wave physics; STAR - nuclear physics, CDF, D0, - high energy physics, SDSS - astrophysics GADU - bioinformatics Nanohub etc. Research groups transitioning from & extending (legacy) systems to Grids: Evolution of iVDGL, GriPhyN, PPDG US LHC Collaborations Contribute to & depend on milestones, functionality, capacity of OSG. Commitment to general solutions, sharing resources & technologies; Application Computer Scientists Contribute to technology, integration, operation. Excited to see technology put to production use! Federations with Campus Grids Bridge & interface Local & Wide Area Grids. Interoperation & partnerships with national/ international infrastructures Ensure transparent and ubiquitous access. Work towards standards. NMI, Condor, Globus, SRM GLOW, FermiGrid, GROW, Crimson, TIGRE EGEE, TeraGrid, INFNGrid 2/21/2019

LHC Physics gives schedule and performance stress: Beam starts in 2008: Distributed System must serve 20PB of data in served across 30PB disk distributed across 100 sites worldwide to be analyzed by 100MSpecInt2000 of CPU. Service Challenges give steps to full system 1 GigaByte/sec 2/21/2019

Priority to many other stakeholders New science enabled by opportunistic use of resources E.g. From OSG Proposal: LIGO : With an annual science run of data collected at roughly a terabyte of raw data per day, this will be critical to the goal of transparently carrying out LIGO data analysis on the opportunistic cycles available on other VOs hardware Opportunity to share use of “standing army” of resources E.g. From OSG news: Genome Analysis and Database Update system, uses OSG resources to run BLAST, Blocks and Chisel for all publicly available genome sequence data used by over 2,400 researchers worldwide GADU processes 3.1 million protein sequences, about 500 batch jobs per day for several weeks, to be repeated every monthly. Interface existing computing and storage facilities and Campus Grids to a common infrastructure. E.g. FermiGrid Strategy: To allow opportunistic use of otherwise dedicated resources. To save effort by implementing shared services. To work coherently to move all of our applications and services to run on the Grid. 2/21/2019

OSG Use - the Production Grid 2/21/2019

OSG Sites Today Not all registered slots are available to grid users. 23 VOs More than 18,000 batch slots registered. … but only 15% of it used via grid interfaces that are monitored. Large fraction of local use rather than grid use. Not all registered slots are available to grid users. Not all available slots are available to every grid user. Not all slots used are monitored. 2/21/2019

Use - Daily Monitoring 04/23/2006 “bridged” GLOW jobs Genome analyis (GADU) 2000 running jobs 500 waiting jobs 2/21/2019

Use - 1 Year View Deploy OSG Release 0.4.0 LHC OSG “launch” 2/21/2019

Common Middleware provided through Virtual Data Toolkit Domain science requirements. Globus, Condor, EGEE etc OSG stakeholders and middleware developer (joint) projects. Test on “VO specific grid” Integrate into VDT Release. Deploy on OSG integration grid Include in OSG release & deploy to OSG production. Condor project 2/21/2019

Middleware Services Grid Site Interfaces Grid Access: X509 User certificate extended using EGEE-VOMS. Condor-G job submission. Jobs may place initial data. Middleware Services VO Management: EGEE-VOMS extends certificate with attributes based on Role. Grid Site Interfaces Monitoring & Information: Processing Service: Job Submission: GRAM (pre-WS or WS) Overhead Mitigations: Condor Gridmonitor; Condor Managed Fork Queue Site and VO based Authorization and Account Mapping Data Movement: GridFTP Identity & Authorization: GUMS Certificate to Account Mapping. Data Storage and Management: Storage Resource Management (SRM) or Local Environment Variables ($APP, $GRID_WRITE etc) Shared File Systems Site and VO based Authorization and Access mapping 2/21/2019

S U Secure R F Usable Reliable Flexible OSG allows you to Providing a Distributed Facility S U R F Usable Reliable Flexible The Grid 2/21/2019

Security - Management, Operational and Technical Controls Management: Risk assessment and security planning, Service auditing and checking. Operational: Incident response, Awareness and Training, Configuration management. Technical: Authentication and Revocation, Auditing and analysis. End to end trust in quality of code executed on remote CPU. 2/21/2019

Usability - Throughput, Scaling, Fault Diagnosis OSG users need high throughput. Focus on effective utilization. VO control of use: examples: Batch system priority in based on VO activity. Control Write-authorization and quotas based on VO role. Prioritize data transfer based on role of the user. Usability Goals include: Minimize entry threshold for resource owners Minimize software stack. Minimize support load. Minimize entry threshold for users Feature rich software stack. Excellent user support. 2/21/2019

Reliability - Central Operations Activities Daily Grid Exerciser: Automated validation of basic services and site configuration Configuration of HeadNode and Storage to reduce errors: Remove dependence on Shared File System Condor-managed GRAM fork queue Scaling tests of WS-GRAM and GridFTP. 2/21/2019

Flexibility Multiple user interfaces. Support for multiple versions of the Middleware Releases. Heterogeneous resources. Sites have local control of use of the resources - including storage and space (no global file system). Distributed support model through VO and Facility Support Centers which contribute to global operations. 2/21/2019

Finally - the OSG Roadmap Sustain the Users, the Facility, and the Experts. Continually improve effectiveness and throughput. Capabilities and schedule driven by Science Stakeholders - maintain production system while integrating new features. Enable new Sites, Campus and Grids to contribute while maintaining self-control. Release Planned Actual OSG 0.2 Spring 2005 July 2005 OSG 0.4.0 Dec 2005 January 2006 OSG 0.4.1 April 2005 May 2006: WS-Gram; condor based fork queue; accounting V0; Inteoperability with LCG using RB. OSG 0.6.0 July 2006 Edge services; Condor-C; test gLITE wms; OSG 0.8.0 End of 2006 OSG 1.0 2007 2/21/2019