AstroGrid-D Host Monitoring in AstroGrid-D with GRAM-Audit or SGAS based on Usage Records Format S. Braune, F. Breitling, H. Enke AIP

xml2rdf Overview of monitoring components for grid resources in Usage Record format SGAS cron script SPARQL-Queries Stellaris globus container GRAM storage, computer, or telescope grid resource monitoring serverclient Browser GUI SPARQL Interface Grid Map... SGAS WS GRAM audit logging trigger URs in local log dir URs in central server db (PostgreSQL)

SGAS – SWE Accounting System extra software has to be installed to add in $GLOBUS_LOCATION/etc/gram-service/service-config.wsdd:... output only after a globus job is finished successfully output (Usage Record format) has to be send to stellaris via cron job

GRAM Audit Logging part of globus container software to be declared in $GLOBUS_LOCATION/container-log4j.properties:... # GRAM AUDIT log4j.category.org.globus.exec.service.exec.StateMachine.audit=DEBUG, AUDIT log4j.appender.AUDIT=org.globus.exec.utils.audit.AuditDatabaseAppender log4j.appender.AUDIT.layout=org.apache.log4j.PatternLayout log4j.additivity.org.globus.exec.service.exec.StateMachine.audit=false output into a rel. database (PostgreSQL or MySQL), Database Connection has to be declared in $GLOBUS_LOCATION/etc/gram-service/jndi-config.xml:... url jdbc:mysql:// [:port]/auditDatabase user globus password foo... table update whenever a job ist started or changed it's status, database content is converted into Globus Usage Record format and sent to stellaris via trigger (perl)

GRAM Audit Database Schema ob_grid_id: String representation of the resource EPR local_job_id: Job/process id generated by the scheduler subject_name: Distinguished name (DN) of the user username: Local username idempotence_id: Job id generated on the client-side creation_time: Date when the job resource is created queued_time: Date when the job is submitted to the scheduler stage_in_grid_id: String representation of the stageIn-EPR (RFT) stage_out_grid_id: String representation of the stageOut-EPR (RFT) clean_up_grid_id: String representation of the cleanUp-EPR (RFT) globus_toolkit_version: Version of the server-side GT resource_manager_type: Type of the resource manager (Fork, Condor,...) job_description: Complete job description document success_flag: Flag that shows whether the job failed or finished successfully finished_flag: Flag that shows whether the job is already fully processed or still in progress converting db record into UR format via db trigger (e.g. perl)

Grid Architecture: Information Service Stellaris development of AstroGrid-D by ZIB (M. Högqvist, ZIB) available at (Apache License) metadata are exchanged and stored in RDF is used by AstroGrid-D for grid resource monitoring (e.g. compute resources, telescopes),and job execution monitoring tested and runs stable Resource Description Framework (RDF) standard for storing information based on graph theory, and presents information in triples different formats exist: RDF/XML (W3C recommendation), and Notation 3 (N3) (W3C specification) the SPARQL query language exists for retrieval of RDF information for Stellaris a conversion from RTML documents and UR XML into RDF required developed an XSLT (Extensible Stylesheet Language Transformation) at AIP can transform arbitrary XML files into RDF/XML format (e.g. RTML) usage very simple, only two command line programs „xsltproc“, and „cURL“ for uploading to Stellaris available at (Apache License)

SPARQL Queries and User Interface Web browser user interface Developed in AstroGrid-D by ZIB (M. Högqvist) & AIP (F. Breitling), Available under Apache License at currently used to display job-state metadata Based on „Simile“ Timeline like Google Maps for time-based information a DHTML-based AJAXy widget for visualizing time-based events SPARQL Query Language for RDF (W3C working draft) is the counterpart for retrieving information from RDF simple syntax for selected and sorted retrieval Perl modules exist for the integration into Perl scripts SPARQL web browser interface is implemented into Stellaris for a quick lookup

Outlook / Conclusion Installation in AstroGrid-D is done Tested, stable run on resources Information about job execution on each grid host Database interface, graphical representation with web interface Based on GRAM audit and Stellaris software monitoring of job execution for each grid resource is now available