Monitoring and Information Services Mark L. Green CCR, University at Buffalo Open Science Grid Consortium Meeting July 21, 2005

Monitoring and Information Services The Monitoring and Information Services Technical group is responsible for coordinating the OSG activities that relate to dynamic resource and information services within the OSG infrastructure. These services include: Information services describing resources and other entities within the OSG. Dynamic resource monitoring to provide, for example, information for scheduling decisions of grid applications. Tools for troubleshooting services and end-to-end debugging of applications and services. Services for usage and statistics, and infrastructure for higher level services such as auditing/accounting. The chairs are Mark L. Green and Conrad Steenberg.

Monitoring and Information Services October - December 2004 Discussion about MIS core infrastructure with other Technical Groups. Monitoring applications should present information about how they are getting information from the resources, what they provide as a query analysis interface and have a common core infrastructure to build around. Develop Schemas from all monitoring services so we can eliminate duplicates and start from a common subset (GLUE Schema, GRID3, etc.). Have our own MIS Schema and add key/value pairs as you please. The discovery service is a name service, providing urls and names of all services. UCSD Technical Meeting Preparation Plan to present the MIS core infrastructure architecture. Meet with TeraGrid INCA developers at meeting.

Monitoring and Information Services October - December 2004 cont’d Preliminary Monitoring and Information Services - Core Infrastructure (MIS-CI) jobmanager-mis arguments: Profile custom user profiles All Jobs running queued historical GridFTP Diskspace Accounting Policy Environment Software VO Statistics Security Privileges based on authenticated user information.

Monitoring and Information Services October - December 2004 cont’d Investigate if Clarens could also be used for hierarchical repository. Looking into potential schema issues and how to move forward with MIS-CI schema. Test plan needed for OSG integration. MIS-CI system architecture presented at UCSD Technical Meeting.

MIS-CI System Architecture MIS Compute Element and Storage Element Discovery Service Lightweight information gathering Self monitoring Scalable Expandable Robust and Fault Tolerant Timestamps API

MIS-CI System Architecture MIS Compute Element and Storage Element Consumer Interface Modular and customizable query infrastructure Self monitoring Scalable Consumer profiles Consumer throttling API

MIS-CI System Architecture MIS Compute Element and Storage Element Resource Repository Lightweight with minimal resource loading and requirements Self monitoring Expandable Common schema with unique name space Virtual organization based aggregation Timestamps

MIS-CI System Architecture MIS Compute Element and Storage Element Information Gathering Lightweight information gathering Self monitoring Expandable Modular design Cron based implementation

MIS-CI System Architecture SQ-Lite Database Remote VO Historical Repository Local Backup User Resource Gatekeeper jobmanager-mis -profile (default) jobs gridftp diskspace -accounting policy environment software VO statistics security Discovery Service MIS-CI Profiles Schema Eval Cron Self- Monitor accounting ? MIS-CI Consumer Throttling MIS-CI Resource Throttling Custom Remote Repository Grid Scheduler OSG Auditing/Accounting

MIS-CI Deployment Plans Proof of Concept MIS-CI Deployment Pacman Cache Grid3Dev Mama Cluster Grid3 Acdc Cluster Document Management System Module management Version control MIS-CI API MIS-CI Online Documentation

MIS-CI Deployment Timeline Proof of Concept MIS-CI Deployment (GridCat Providers) ~ 12/2004 Pacman Cache Grid3Dev ~ 1/2005 Mama Cluster Grid3 ~ 2/2005 ACDC Cluster Document Management System ~ 12/2004 Module management Version control MIS-CI API ~ 1/2005 MIS-CI Online Documentation ~ 1/2005 Proof of Concept Web Repository (ACDC) ~ 1/2005 Only use MIS-CI for Information Gathering Load/Usage Statistics and Scalability reporting

Monitoring and Information Services January - April 2005 OSG Monitoring and Information Services Requirement solicitation sent out. The document management system is setup at url: https://grid.ccr.buffalo.edu/docmgr_osg_mis/ A readiness plan for the OSG integration activity needs to be developed. MIS-CI CVS repository setup at UF: Mark, Craig, Bockjoo are the current read/writers. MIS-CI wrapped in with OSG-INT cache and Pacman 3 compliant.

Monitoring and Information Services January - April 2005 cont’d VO specific instances of the discovery service are provided at http://discover.gridservice.info. Look at making a pacman cache for the GIP and creating an authenticated BDII. The discovery service can integrate information about the MIS-CI grid installations and serve as the top level MIS-CI discovery service. Currently testing MIS-CI on UBuffalo-CCR-ITB site MIS-CI 0.2.6 release candidate for GridCat.

Monitoring and Information Services January - April 2005 cont’d MDS GIP LDAP Integration Format MDS calls GIP and information is returned in LDIF format for integration into MDS. MIS-CI needs to return information in LDIF format and a call added into MDS information gathering scripts to incorporate MIS-CI information. Identify information deficiencies that the Resource Broker requires and resolve them. Accounting Activity and MIS-CI discussions are scheduled for the end of April. TeraGrid Science Gateways discussions start.

Monitoring and Information Services May - July 2005 General discussion about OSG Monitoring and Information Services minimum requirements. MDS developers will helping GIP folks work on XML mapping of GLUE Schema 1.2. MDS has providers for ldif, XML, and GLUE Schema. We need a DBII to collect the gris information. Create services VO and Monitoring Services Integration (MIS VO) Move the burden of monitoring configuration, update, and functionality from the novice to the experts. Current monitoring services should join this VO (GridCat, ACDC, MonaLisa, MDS, etc.) MIS-CI will fill in the gaps in the GIPs for OSG.

Monitoring and Information Services May - July 2005 cont’d MIS Current Proposal: GIP 1.1 ldif GLUE Schema 1.1 MDS 2.4 gris BDII Index Service 3.25 MIS Roadmap Proposal: GIP 1.2 xml GLUE Schema 1.2 MDS 4 BDII Index Service 3.25 and/or MDS Index Service ACDC Grid Dashboard Functionality and tutorial discussion. Batch job, fork job, and GridFTP accounting online.

Monitoring and Information Services May - July 2005 cont’d Pre-release VDT cache available for testing of the GIP/MDS/GRIS system. Decided MIS-CI should form an activity and recruit volunteers for development work. POLICY_URL variable for resource utilization policy added. OSG Monitoring and Information Services Minimum Requirements Discussion The first OSG release will define the minimum monitoring and information services as: The globus top level location is the same as the VDT location. Top level globus directory/monitoring/grid3-info.conf. Forward map for certs to accounts. Reverse map for accounts to VOs. Interface requirement for minimum requirements is ACSII file.

Monitoring and Information Services May - July 2005 cont’d GRASE VO has determined that six applications are reporting correctly to the ACDC Grid Dashboards and are generating accounting information. Verification of gridftp and batch jobs fine, VO/user mapping being verified. Kit validation, from the VOs that wish to participate in the MIS-OPs service. Benefit is that CEs can be validated based on the actual VO mapped user, participation is completely voluntary.

MIS-CI Level of Effort MIS-CI Activity Formed Center for Computational Research SUNY Buffalo Stakeholder 1.5 FTE Indiana GOC 0.5 FTE Binghamton 1.0 FTE LCG/EGEE Grid Deployment 0.25 FTE (pending) Others Estimated requirement 2.0 FTE OSG GIP Configuration Recommendations Defined. TG may need to form sub groups to focus discussions on contributing into the Blueprint activity.

Monitoring and Information Services May - July 2005 cont’d The scope of GIP work is three-fold: The first is to address how GIP can be integrated with other OSG monitoring and information services. The second is to ease the configuration of GIP components. The third is to support the interoperability between the LCG and OSG. Specific goals for this work are to: Test GIP and provide input to VDT and GIP teams. Provide GIP configuration guidance and tools to support its interaction/integration with other OSG monitoring and information services. Define use cases for GIP and standard practices Facilitate GIP use in interoperability (LCG and OSG) work.

Monitoring and Information Services May - July 2005 cont’d Action items / effort & time line: 1) Test GIP, and document testing results (5%, ongoing) 2) Evaluate GIP roles with respect to other OSG monitoring and information services (10%, a few months) 3) Provide tools to make GIP configuration easier, more accurate and useful (10%, several weeks) 4) Provide "typical" GIP use scenarios in which OSG resources are involved (5%, a few weeks) 5) Investigate how GIP is applied to address the interoperability between the LCG and OSG (10%, ongoing) 6) Customize GIP to use standards that conform to OSG fashion (for example, change LCG_2_4_0 to OSG_0_1_6) (5%, several days) 7) Interact with Condor team about publishing Condor policy and dynamic queue status to GIP (10%, few weeks)

Monitoring and Information Services May - July 2005 cont’d There is an existing interoperability activity where much of the discussion has been made. There should be two paths: 1) interoperability with LCG and 2) provide the required information for MIS group. Glue schema 2 will potentially merge these paths. GOC should run a discovery service for the OSG, discussion what services will be discovered there? Try to stay away from the never ending MIS schema revision cycle. Started work to make GIP configuration easier with Perl based automatic configuration of the GIP underway.

MIS-TG Looking Forward Refine MIS-CI System Architecture Key components are MDS, BDII, R-GMA, GIP, SQL Compliant Database, WSRF, MIS Schema, hierarchical information transport mechanisms, present standardized interface for higher level services. Extend resource policy definition specification and discovery. Log file analysis and archival procedures. Enforcement and interpretation infrastructure and tools. Usage, accounting, statistics analysis, and metrics reporting infrastructure. Interoperate and interoperable with peer grids. Coordinated effort with other OSG Technical Groups.