1. Leigh Grundhoefer Indiana University
OSG Grid Operations
Leigh Grundhoefer
Indiana University

2. leighg at indiana dot edu
Agenda
Grid Operations Development
OSG Functional Service Cycle
Deployment
Integration
Provisioning
Production
OSG Operations Activities
11/21/2018
leighg at indiana dot edu

3. leighg at indiana dot edu
“Big Picture” Goals
The OSG Operations activity and Support Centers Group has been tasked with the role of preparing, provisioning and running the infrastructure used for the OSG production environment.
The operations activities duties to the OSG are to ensure that the production environment is usable for the current application base and to continue to evolve as a common service environment which is able to support multiple sciences with a application-friendly grid infrastructure.
11/21/2018
leighg at indiana dot edu

4. Grid Operations Security, Policy and Authentication Other Grids
Integration
Test Bed
Provisioning
Production
Releases
And Updates
Documentation of software
Registration, Verification and Monitoring
Knowledge base
of Commonly Answered
questions
Trouble reporting
and ticketing
Support Center
coordinator
11/21/2018
leighg at indiana dot edu

5. Operations Environment
Organization and Definition of core elements
Grids
Support Centers
Virtual Organizations
Resources
Registration by Owners and Providers
Common software services
Verification and ongoing evaluation
Resources and services
Support Center ticketing, ticketing response
VO services
11/21/2018
leighg at indiana dot edu

6. Operations Environment (cont.)
Information from monitoring
Job slots and file transfers
Published policies
Accounting
Coordination of OSG Help Desk
Coordinator of support centers
Definition and execution of Standard Operational Procedures (SOPs).
Definition and execution of Standard Operational Procedures (SOPs) which add to the availability, stability and enhancement of production infrastructure.
11/21/2018
leighg at indiana dot edu

7. Grid to Grid relationships
Understand and create specialized help desk and trouble reporting schemas
Understand and create monitoring and accounting interfaces
Create and deploy identity and authorization interfaces
11/21/2018
leighg at indiana dot edu

8. leighg at indiana dot edu
Agenda
Grid Operations Development
OSG Functional Service Cycle
Deployment
Integration
Provisioning
Production
OSG Operations Activities
11/21/2018
leighg at indiana dot edu

9. Where do you get this stuff? (Architecture and Requirements)
Release Candidate
Blueprint
(ARCH)
ITB 0.3
Integration Test Bed
Operations
OSG 0.4
Provisioning
VO’s
Deployment Activity
Release Description
Service Development (Sponsored Activities)
Technical Groups
11/21/2018
leighg at indiana dot edu

10. OSG Integration Activity
Readiness
plan
Effort
Resources
Readiness
plan adopted VO
Application
Software
Installation
Software &
packaging
OSG Deployment Activity
Service
deployment
OSG Operations-Provisioning
Activity
Application
validation
Release
Candidate
Middleware
Interoperability
Functionality &
Scalability
Tests
feedback
Metrics &
Certification
Release
Description
11/21/2018
leighg at indiana dot edu

11. leighg at indiana dot edu
Provisioning
Finalize all installation software and procedures
OSG based software packages
Pre-compiled binaries
Source code (compiled during installation)
Post configuration scripts ( configure_osg.sh )
Create production “Grid Support” documentation for all software and defined procedures.
11/21/2018
leighg at indiana dot edu

12. leighg at indiana dot edu
Provisioning
Translate OSG operations model into production operations activities
Provide timelines for Resources and Support Centers
Setup versioning using release procedures
Install or upgrade grid wide services
11/21/2018
leighg at indiana dot edu

13. leighg at indiana dot edu
11/21/2018
leighg at indiana dot edu

14. leighg at indiana dot edu
Production
Reports
Usage reports from OSG monitoring
Operations reports to the OSG community
Daily Sites Status reports to Support Centers
Meetings
Weekly Operations Activity
Release-based provisioning Activity
Weekly Support Centers Technical Group
Weekly Documentation Activity
Monitors for verification
ACDC Operations Dashboard
Issue handling
11/21/2018
leighg at indiana dot edu

15. Verification of Resouces
ACDC Operation Dashboard provides detailed cyclic testing of all resources
Tests based upon “site-verify” tool distributed with OSG common software, around 30 tests
Tests results output available per test per site
Five possible results
No Information
Pass
Fail
Error
Not Tested
Resources are grouped in to three areas: Production, Pending or Offline
11/21/2018
leighg at indiana dot edu

16. leighg at indiana dot edu
Agenda
Grid Operations Development
OSG Functional Service Cycle
Deployment
Integration
Provisioning
Production
OSG Operations Activities
11/21/2018
leighg at indiana dot edu

17. GOC: A Communications Hub
Grid Operations Center
Leveraged Coverage from GRNOC
Abilene, NLR, TransPAC
OSG Trouble Ticket System
Trouble Ticket Exchange with Support Centers / Grids
Weekly Issue tracking report
Web Page Development and Documentation
OSG Production web site
Commonly answered questions Knowledge Base
Collaborative development information - osg.ivdgl.org
OSG Registration Database
11/21/2018
leighg at indiana dot edu

18. leighg at indiana dot edu
GOC: A Service Center
Grid wide Information Services
Registration Database Catalog
Production Software Caches
OSG Knowledge Base
OpenScienceGrid web site
Grid wide Monitoring Services
GLUE information providers
ACDC Dashboards
GridCat
MonaLisa archive
Multiple OSG ITB resources
Small but demonstrative OSG production resource
11/21/2018
leighg at indiana dot edu

19. GOC: Incident Response
Response has been defined by the Security Technical Group
All incidents should be reported mail aliases which are monitored by the GOC.
The GOC maintains a list of local site security contacts, derived from the registrations.
The GOC designed and implemented a specialized mail service for secure correspondence
A response team leader forms a group to access, contain, and report on each incident.
11/21/2018
leighg at indiana dot edu

20. GOC: Policy Procedures
Grid Service Change or Upgrade
Registrations
Support Centers
Virtual Organizations
Resources
Critical OSG Release Update
11/21/2018
leighg at indiana dot edu

21. Grid Service Change/Update
11/21/2018
leighg at indiana dot edu

22. Support Center Registration
11/21/2018
leighg at indiana dot edu

23. Virtual Organization Registration
11/21/2018
leighg at indiana dot edu

24. Resource/Service Registration
11/21/2018
leighg at indiana dot edu

25. Critical Release Update
11/21/2018
leighg at indiana dot edu

26. leighg at indiana dot edu
11/21/2018
leighg at indiana dot edu

27. leighg at indiana dot edu
OSG Community Support
How do we support issues that fall outside the Support Model?
Open Support Model
Mailing Lists (OSG-General)
Knowledge Base and Release Documentation
Jabber chat room
Weekly meetings
11/21/2018
leighg at indiana dot edu

28. leighg at indiana dot edu
GOC: Conclusions
Enables Users and Usage
Creates a known and usable service environment
Allows status, monitoring and accounting
Helps users and applications “bridge the gap” between single use and grid based resource utilization
11/21/2018
leighg at indiana dot edu

29. Grid Operations Effort
Five FTEs at Indiana University
Two FTEs at FermiLab
One FTE at LBNL
One FTE at University of Chicago
Help and direction encourage from everyone.
11/21/2018
leighg at indiana dot edu

30. leighg at indiana dot edu
Thank you!
11/21/2018
leighg at indiana dot edu

31. Operations activities for 0.4
A distributed support organization with a central operations organization, with Indiana GOC as the central point of contact.
Create and organize Support Centers so that they coordinate with each other.
Metrics publishing.
Create and develop policy and process interfaces to the EGEE and LCG operations organization.
Deploy and maintain production instances of the Grid Catalogs.
Validation and Testing Services
Create and validate verification and performance testing mechanisms for the common services.
11/21/2018
leighg at indiana dot edu