1. Monitoring and Information Services
Mark L. Green
CCR, University at Buffalo
Open Science Grid Consortium Meeting
July 21, 2005

2. Monitoring and Information Services
The Monitoring and Information Services Technical group is responsible for coordinating the OSG activities that relate to dynamic resource and information services within the OSG infrastructure. These services include:
Information services describing resources and other entities within the OSG.
Dynamic resource monitoring to provide, for example, information for scheduling decisions of grid applications.
Tools for troubleshooting services and end-to-end debugging of applications and services.
Services for usage and statistics, and infrastructure for higher level services such as auditing/accounting.
The chairs are Mark L. Green and Conrad Steenberg.

3. Monitoring and Information Services
October - December 2004
Discussion about MIS core infrastructure with other Technical Groups.
Monitoring applications should present information about how they are getting information from the resources, what they provide as a query analysis interface and have a common core infrastructure to build around.
Develop Schemas from all monitoring services so we can eliminate duplicates and start from a common subset (GLUE Schema, GRID3, etc.).
Have our own MIS Schema and add key/value pairs as you please.
The discovery service is a name service, providing urls and names of all services.
UCSD Technical Meeting Preparation
Plan to present the MIS core infrastructure architecture.
Meet with TeraGrid INCA developers at meeting.

4. Monitoring and Information Services
October - December 2004 cont’d
Preliminary Monitoring and Information Services - Core Infrastructure (MIS-CI) jobmanager-mis arguments:
Profile
custom user profiles
All
Jobs
running
queued
historical
GridFTP
Diskspace
Accounting
Policy
Environment
Software VO
Statistics
Security
Privileges based on authenticated user information.

5. Monitoring and Information Services
October - December 2004 cont’d
Investigate if Clarens could also be used for hierarchical repository.
Looking into potential schema issues and how to move forward with MIS-CI schema.
Test plan needed for OSG integration.
MIS-CI system architecture presented at UCSD Technical Meeting.

6. MIS-CI System Architecture
MIS Compute Element and Storage Element Discovery Service
Lightweight information gathering
Self monitoring
Scalable
Expandable
Robust and Fault Tolerant
Timestamps
API

7. MIS-CI System Architecture
MIS Compute Element and Storage Element Consumer Interface
Modular and customizable query infrastructure
Self monitoring
Scalable
Consumer profiles
Consumer throttling
API

8. MIS-CI System Architecture
MIS Compute Element and Storage Element Resource Repository
Lightweight with minimal resource loading and requirements
Self monitoring
Expandable
Common schema with unique name space
Virtual organization based aggregation
Timestamps

9. MIS-CI System Architecture
MIS Compute Element and Storage Element Information Gathering
Lightweight information gathering
Self monitoring
Expandable
Modular design
Cron based implementation

10. MIS-CI System Architecture
SQ-Lite
Database
Remote VO
Historical
Repository
Local
Backup
User
Resource
Gatekeeper
jobmanager-mis
-profile (default)
jobs
gridftp
diskspace
-accounting
policy
environment
software VO
statistics
security
Discovery
Service
MIS-CI Profiles
Schema Eval
Cron
Self-
Monitor
accounting ?
MIS-CI
Consumer
Throttling
MIS-CI Resource Throttling
Custom Remote Repository
Grid Scheduler
OSG Auditing/Accounting

11. MIS-CI Deployment Plans
Proof of Concept MIS-CI Deployment
Pacman Cache
Grid3Dev
Mama Cluster
Grid3
Acdc Cluster
Document Management System
Module management
Version control
MIS-CI API
MIS-CI Online Documentation

12. MIS-CI Deployment Timeline
Proof of Concept MIS-CI Deployment (GridCat Providers) ~ 12/2004
Pacman Cache
Grid3Dev ~ 1/2005
Mama Cluster
Grid3 ~ 2/2005
ACDC Cluster
Document Management System ~ 12/2004
Module management
Version control
MIS-CI API ~ 1/2005
MIS-CI Online Documentation ~ 1/2005
Proof of Concept Web Repository (ACDC) ~ 1/2005
Only use MIS-CI for Information Gathering
Load/Usage Statistics and Scalability reporting

13. Monitoring and Information Services
January - April 2005
OSG Monitoring and Information Services Requirement solicitation sent out.
The document management system is setup at url:
A readiness plan for the OSG integration activity needs to be developed.
MIS-CI CVS repository setup at UF: Mark, Craig, Bockjoo are the current read/writers.
MIS-CI wrapped in with OSG-INT cache and Pacman 3 compliant.

14. Monitoring and Information Services
January - April 2005 cont’d
VO specific instances of the discovery service are provided at
Look at making a pacman cache for the GIP and creating an authenticated BDII.
The discovery service can integrate information about the MIS-CI grid installations and serve as the top level MIS-CI discovery service.
Currently testing MIS-CI on UBuffalo-CCR-ITB site MIS-CI release candidate for GridCat.

15. Monitoring and Information Services
January - April 2005 cont’d
MDS GIP LDAP Integration Format
MDS calls GIP and information is returned in LDIF format for integration into MDS.
MIS-CI needs to return information in LDIF format and a call added into MDS information gathering scripts to incorporate MIS-CI information.
Identify information deficiencies that the Resource Broker requires and resolve them.
Accounting Activity and MIS-CI discussions are scheduled for the end of April.
TeraGrid Science Gateways discussions start.

16. Monitoring and Information Services
May - July 2005
General discussion about OSG Monitoring and Information Services minimum requirements.
MDS developers will helping GIP folks work on XML mapping of GLUE Schema 1.2. MDS has providers for ldif, XML, and GLUE Schema.
We need a DBII to collect the gris information.
Create services VO and Monitoring Services Integration (MIS VO)
Move the burden of monitoring configuration, update, and functionality from the novice to the experts.
Current monitoring services should join this VO (GridCat, ACDC, MonaLisa, MDS, etc.)
MIS-CI will fill in the gaps in the GIPs for OSG.

17. Monitoring and Information Services
May - July 2005 cont’d
MIS Current Proposal:
GIP 1.1 ldif
GLUE Schema 1.1
MDS 2.4 gris
BDII Index Service 3.25
MIS Roadmap Proposal:
GIP 1.2 xml
GLUE Schema 1.2
MDS 4
BDII Index Service 3.25 and/or MDS Index Service
ACDC Grid Dashboard
Functionality and tutorial discussion.
Batch job, fork job, and GridFTP accounting online.

18. Monitoring and Information Services
May - July 2005 cont’d
Pre-release VDT cache available for testing of the GIP/MDS/GRIS system.
Decided MIS-CI should form an activity and recruit volunteers for development work.
POLICY_URL variable for resource utilization policy added.
OSG Monitoring and Information Services Minimum Requirements Discussion
The first OSG release will define the minimum monitoring and information services as:
The globus top level location is the same as the VDT location.
Top level globus directory/monitoring/grid3-info.conf.
Forward map for certs to accounts.
Reverse map for accounts to VOs.
Interface requirement for minimum requirements is ACSII file.

19. Monitoring and Information Services
May - July 2005 cont’d
GRASE VO has determined that six applications are reporting correctly to the ACDC Grid Dashboards and are generating accounting information.
Verification of gridftp and batch jobs fine, VO/user mapping being verified.
Kit validation, from the VOs that wish to participate in the MIS-OPs service.
Benefit is that CEs can be validated based on the actual VO mapped user, participation is completely voluntary.

20. MIS-CI Level of Effort MIS-CI Activity Formed
Center for Computational Research SUNY Buffalo Stakeholder
1.5 FTE
Indiana GOC
0.5 FTE
Binghamton
1.0 FTE
LCG/EGEE Grid Deployment
0.25 FTE (pending)
Others
Estimated requirement 2.0 FTE
OSG GIP Configuration Recommendations Defined.
TG may need to form sub groups to focus discussions on contributing into the Blueprint activity.

21. Monitoring and Information Services
May - July 2005 cont’d
The scope of GIP work is three-fold:
The first is to address how GIP can be integrated with other OSG monitoring and information services.
The second is to ease the configuration of GIP components.
The third is to support the interoperability between the LCG and OSG.
Specific goals for this work are to:
Test GIP and provide input to VDT and GIP teams.
Provide GIP configuration guidance and tools to support its interaction/integration with other OSG monitoring and information services.
Define use cases for GIP and standard practices
Facilitate GIP use in interoperability (LCG and OSG) work.

22. Monitoring and Information Services
May - July 2005 cont’d
Action items / effort & time line:
1) Test GIP, and document testing results (5%, ongoing)
2) Evaluate GIP roles with respect to other OSG monitoring and information services (10%, a few months)
3) Provide tools to make GIP configuration easier, more accurate and useful (10%, several weeks)
4) Provide "typical" GIP use scenarios in which OSG resources are involved (5%, a few weeks)
5) Investigate how GIP is applied to address the interoperability between the LCG and OSG (10%, ongoing)
6) Customize GIP to use standards that conform to OSG fashion (for example, change LCG_2_4_0 to OSG_0_1_6) (5%, several days)
7) Interact with Condor team about publishing Condor policy and dynamic queue status to GIP (10%, few weeks)

23. Monitoring and Information Services
May - July 2005 cont’d
There is an existing interoperability activity where much of the discussion has been made. There should be two paths:
1) interoperability with LCG and
2) provide the required information for MIS group. Glue schema 2 will potentially merge these paths.
GOC should run a discovery service for the OSG, discussion what services will be discovered there?
Try to stay away from the never ending MIS schema revision cycle.
Started work to make GIP configuration easier with Perl based automatic configuration of the GIP underway.

24. MIS-TG Looking Forward
Refine MIS-CI System Architecture
Key components are
MDS, BDII, R-GMA, GIP, SQL Compliant Database, WSRF, MIS Schema,
hierarchical information transport mechanisms,
present standardized interface for higher level services.
Extend resource policy definition specification and discovery.
Log file analysis and archival procedures.
Enforcement and interpretation infrastructure and tools.
Usage, accounting, statistics analysis, and metrics reporting infrastructure.
Interoperate and interoperable with peer grids.
Coordinated effort with other OSG Technical Groups.