Leigh Grundhoefer Indiana University OSG Grid Operations Leigh Grundhoefer Indiana University

leighg at indiana dot edu Agenda Grid Operations Development OSG Functional Service Cycle Deployment Integration Provisioning Production OSG Operations Activities 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu “Big Picture” Goals The OSG Operations activity and Support Centers Group has been tasked with the role of preparing, provisioning and running the infrastructure used for the OSG production environment. The operations activities duties to the OSG are to ensure that the production environment is usable for the current application base and to continue to evolve as a common service environment which is able to support multiple sciences with a application-friendly grid infrastructure. 11/21/2018 leighg at indiana dot edu

Grid Operations Security, Policy and Authentication Other Grids Integration Test Bed Provisioning Production Releases And Updates Documentation of software Registration, Verification and Monitoring Knowledge base of Commonly Answered questions Trouble reporting and ticketing Support Center coordinator 11/21/2018 leighg at indiana dot edu

Operations Environment Organization and Definition of core elements Grids Support Centers Virtual Organizations Resources Registration by Owners and Providers Common software services Verification and ongoing evaluation Resources and services Support Center ticketing, ticketing response VO services 11/21/2018 leighg at indiana dot edu

Operations Environment (cont.) Information from monitoring Job slots and file transfers Published policies Accounting Coordination of OSG Help Desk Coordinator of support centers Definition and execution of Standard Operational Procedures (SOPs). Definition and execution of Standard Operational Procedures (SOPs) which add to the availability, stability and enhancement of production infrastructure. 11/21/2018 leighg at indiana dot edu

Grid to Grid relationships Understand and create specialized help desk and trouble reporting schemas Understand and create monitoring and accounting interfaces Create and deploy identity and authorization interfaces 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu Agenda Grid Operations Development OSG Functional Service Cycle Deployment Integration Provisioning Production OSG Operations Activities 11/21/2018 leighg at indiana dot edu

Where do you get this stuff? (Architecture and Requirements) Release Candidate Blueprint (ARCH) ITB 0.3 Integration Test Bed Operations OSG 0.4 Provisioning VO’s Deployment Activity Release Description Service Development (Sponsored Activities) Technical Groups 11/21/2018 leighg at indiana dot edu

OSG Integration Activity Readiness plan Effort Resources Readiness plan adopted VO Application Software Installation Software & packaging OSG Deployment Activity Service deployment OSG Operations-Provisioning Activity Application validation Release Candidate Middleware Interoperability Functionality & Scalability Tests feedback Metrics & Certification Release Description 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu Provisioning Finalize all installation software and procedures OSG based software packages Pre-compiled binaries Source code (compiled during installation) Post configuration scripts ( configure_osg.sh ) Create production “Grid Support” documentation for all software and defined procedures. 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu Provisioning Translate OSG operations model into production operations activities Provide timelines for Resources and Support Centers Setup versioning using release procedures Install or upgrade grid wide services 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu Production Reports Usage reports from OSG monitoring Operations reports to the OSG community Daily Sites Status reports to Support Centers Meetings Weekly Operations Activity Release-based provisioning Activity Weekly Support Centers Technical Group Weekly Documentation Activity Monitors for verification ACDC Operations Dashboard Issue handling 11/21/2018 leighg at indiana dot edu

Verification of Resouces ACDC Operation Dashboard provides detailed cyclic testing of all resources Tests based upon “site-verify” tool distributed with OSG common software, around 30 tests Tests results output available per test per site Five possible results No Information Pass Fail Error Not Tested Resources are grouped in to three areas: Production, Pending or Offline 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu Agenda Grid Operations Development OSG Functional Service Cycle Deployment Integration Provisioning Production OSG Operations Activities 11/21/2018 leighg at indiana dot edu

GOC: A Communications Hub Grid Operations Center Leveraged Coverage from GRNOC Abilene, NLR, TransPAC OSG Trouble Ticket System Trouble Ticket Exchange with Support Centers / Grids Weekly Issue tracking report Web Page Development and Documentation OSG Production web site www.opensciencegrid.org Commonly answered questions Knowledge Base Collaborative development information - osg.ivdgl.org OSG Registration Database 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu GOC: A Service Center Grid wide Information Services Registration Database Catalog Production Software Caches OSG Knowledge Base OpenScienceGrid web site Grid wide Monitoring Services GLUE information providers ACDC Dashboards GridCat MonaLisa archive Multiple OSG ITB resources Small but demonstrative OSG production resource 11/21/2018 leighg at indiana dot edu

GOC: Incident Response Response has been defined by the Security Technical Group All incidents should be reported mail aliases which are monitored by the GOC. The GOC maintains a list of local site security contacts, derived from the registrations. The GOC designed and implemented a specialized mail service for secure correspondence A response team leader forms a group to access, contain, and report on each incident. 11/21/2018 leighg at indiana dot edu

GOC: Policy Procedures Grid Service Change or Upgrade Registrations Support Centers Virtual Organizations Resources Critical OSG Release Update 11/21/2018 leighg at indiana dot edu

Grid Service Change/Update 11/21/2018 leighg at indiana dot edu

Support Center Registration 11/21/2018 leighg at indiana dot edu

Virtual Organization Registration 11/21/2018 leighg at indiana dot edu

Resource/Service Registration 11/21/2018 leighg at indiana dot edu

Critical Release Update 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu OSG Community Support How do we support issues that fall outside the Support Model? Open Support Model Mailing Lists (OSG-General) Knowledge Base and Release Documentation Jabber chat room Weekly meetings 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu GOC: Conclusions Enables Users and Usage Creates a known and usable service environment Allows status, monitoring and accounting Helps users and applications “bridge the gap” between single use and grid based resource utilization 11/21/2018 leighg at indiana dot edu

Grid Operations Effort Five FTEs at Indiana University Two FTEs at FermiLab One FTE at LBNL One FTE at University of Chicago Help and direction encourage from everyone. 11/21/2018 leighg at indiana dot edu

leighg at indiana dot edu Thank you! 11/21/2018 leighg at indiana dot edu

Operations activities for 0.4 A distributed support organization with a central operations organization, with Indiana GOC as the central point of contact. Create and organize Support Centers so that they coordinate with each other. Metrics publishing. Create and develop policy and process interfaces to the EGEE and LCG operations organization. Deploy and maintain production instances of the Grid Catalogs. Validation and Testing Services Create and validate verification and performance testing mechanisms for the common services. 11/21/2018 leighg at indiana dot edu