AstroGrid-D Host Monitoring in AstroGrid-D with GRAM-Audit or SGAS based on Usage Records Format S. Braune, F. Breitling, H. Enke AIP.

Slides:



Advertisements
Similar presentations
The Grid Job Monitoring Service Luděk Matyska et al. CESNET, z.s.p.o. Prague Czech Republic.
Advertisements

Building Portals to access Grid Middleware National Technical University of Athens Konstantinos Dolkas, On behalf of Andreas Menychtas.
LEAD Portal: a TeraGrid Gateway and Application Service Architecture Marcus Christie and Suresh Marru Indiana University LEAD Project (
TeraGrid Deployment Test of Grid Software JP Navarro TeraGrid Software Integration University of Chicago OGF 21 October 19, 2007.
Bookshelf.EXE - BX A dynamic version of Bookshelf –Automatic submission of algorithm implementations, data and benchmarks into database Distributed computing.
Design of Web-based Systems IS Development: lecture 10.
Introduction to Web Interface Technology (CSE2030)
Crawler-Based Search Engine By: Bryan Chapman, Ryan Caplet, Morris Wright.
Web 2.0 with AJAX Students : LASC Ioana KELEMEN Csilla POP Dan Adrian CIOBANU Dumitru Daniel Project leader : Ahmed RHIAT.
Talend 5.4 Architecture Adam Pemble Talend Professional Services.
Chapter 9 Collecting Data with Forms. A form on a web page consists of form objects such as text boxes or radio buttons into which users type information.
INTRODUCTION TO WEB DATABASE PROGRAMMING
Semantic Sensor/Device Description System EEEM042-Mobile Applications and Web Services Assignment- Spring Semester 2015 Prof. Klaus Moessner, Dr Payam.
M. Taimoor Khan * Java Server Pages (JSP) is a server-side programming technology that enables the creation of dynamic,
GRID job tracking and monitoring Dmitry Rogozin Laboratory of Particle Physics, JINR 07/08/ /09/2006.
OCLC Online Computer Library Center CONTENTdm ® Digital Collection Management Software Ron Gardner, OCLC Digital Services Consultant ICOLC Meeting April.
EXtensible Neuroimaging Archive Toolkit (XNAT) Washington University Neuroinformatics Group.
MAHI Research Database Data Validation System Software Prototype Demonstration September 18, 2001
JavaScript, Fourth Edition Chapter 12 Updating Web Pages with AJAX.
GRAM: Software Provider Forum Stuart Martin Computational Institute, University of Chicago & Argonne National Lab TeraGrid 2007 Madison, WI.
OracleAS Reports Services. Problem Statement To simplify the process of managing, creating and execution of Oracle Reports.
Grid Resource Allocation and Management (GRAM) Execution management Execution management –Deployment, scheduling and monitoring Community Scheduler Framework.
Relational Database CISC/QCSE 810 some materials from Software Carpentry.
Creating Dynamic Web Pages Using PHP and MySQL CS 320.
NMED 3850 A Advanced Online Design January 12, 2010 V. Mahadevan.
1 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre The Data Portal Glen Drinkwater.
Ontologies and Lexical Semantic Networks, Their Editing and Browsing Pavel Smrž and Martin Povolný Faculty of Informatics,
1 Welcome to CSC 301 Web Programming Charles Frank.
Grid Computing at Yahoo! Sameer Paranjpye Mahadev Konar Yahoo!
INFSO-RI Enabling Grids for E-sciencE OGSA DAI Data Access and Integration Marek Ciglan Institute of Informatics, Slovac Academy.
© 2006 Altova GmbH. All Rights Reserved. Altova ® Product Line Overview.
MyGrid/Taverna Provenance Daniele Turi University of Manchester OMII f2f Meeting, London, 19-20/4/06.
Module: Software Engineering of Web Applications Chapter 2: Technologies 1.
ClearQuest XML Server with ClearCase Integration Northwest Rational User’s Group February 22, 2007 Frank Scholz Casey Stewart
ATLAS-specific functionality in Ganga - Requirements for distributed analysis - ATLAS considerations - DIAL submission from Ganga - Graphical interfaces.
Copyright 2007, Information Builders. Slide 1 iWay Web Services and WebFOCUS Consumption Michael Florkowski Information Builders.
The AstroGrid-D Information Service Stellaris A central grid component to store, manage and transform metadata - and connect to the VO!
 Project Team: Suzana Vaserman David Fleish Moran Zafir Tzvika Stein  Academic adviser: Dr. Mayer Goldberg  Technical adviser: Mr. Guy Wiener.
Generating XML Data from a Database Eugenia Fernandez IUPUI.
XML and Distributed Applications By Quddus Chong Presentation for CS551 – Fall 2001.
Metadata V1 By Dick M.A. Schaap – technical coordinator Oostende, June 08.
MESA A Simple Microarray Data Management Server. General MESA is a prototype web-based database solution for the massive amounts of initial data generated.
Open Science Grid Configuring RSV OSG Resource & Service Validation Thomas Wang Grid Operations Center (OSG-GOC) Indiana University.
GES 2007 Baden-Baden, Providing Remote Access to Robotic Telescopes by adopting Grid Technology T. Granzer & F. Breitling 1 Providing Remote Access.
Integraion und Steuerung von verteilten Sensorsystemen Grid-Package for a Robotic Telescope Network (RTN) Geoforschungszentrum Potsdam 2010 October 20.
AstroGrid -D Job-Monitoring in AstroGrid-D with GRAM-audit-logging S. Braune, F. Breitling, H. Enke, AIP.
Virtual Organisation Management in the Level 2 Grid Steven Newhouse Technical Director London e-Science Centre Department of Computing, Imperial College.
The Abdus Salam ICTP Trieste, Workshop on Scientific Instruments and Sensors on the Grid Integration of Robotic Telescopes Frank Breitling 1.
GES 2007 Baden-Baden, Providing Remote Access to Robotic Telescopes by adopting Grid Technology T. Granzer & F. Breitling 1 Providing Remote Access.
1 Chapter 1 INTRODUCTION TO WEB. 2 Objectives In this chapter, you will: Become familiar with the architecture of the World Wide Web Learn about communication.
The Holmes Platform and Applications
Fundamental of Databases
GPIR GridPort Information Repository
Introduction to Dynamic Web Programming
Introduction to the Application Hosting Environment
Database System Concepts and Architecture
Grid-Integration of Robotic Telescopes
VI-SEEM Data Discovery Service
Middleware independent Information Service
Server Concepts Dr. Charles W. Kann.
PHP / MySQL Introduction
Chapter 2: Database System Concepts and Architecture
Web Browser server client 3-Tier Architecture Apache web server PHP
A Match Made In (Ethereal) Heaven
Module 01 ETICS Overview ETICS Online Tutorials
TN19-TCI: Integration and API management using TIBCO Cloud™ Integration
Production Manager Tools (New Architecture)
Best Practices in Higher Education Student Data Warehousing Forum
DIBBs Brown Dog BDFiddle
Presentation transcript:

AstroGrid-D Host Monitoring in AstroGrid-D with GRAM-Audit or SGAS based on Usage Records Format S. Braune, F. Breitling, H. Enke AIP

xml2rdf Overview of monitoring components for grid resources in Usage Record format SGAS cron script SPARQL-Queries Stellaris globus container GRAM storage, computer, or telescope grid resource monitoring serverclient Browser GUI SPARQL Interface Grid Map... SGAS WS GRAM audit logging trigger URs in local log dir URs in central server db (PostgreSQL)

SGAS – SWE Accounting System extra software has to be installed to add in $GLOBUS_LOCATION/etc/gram-service/service-config.wsdd:... output only after a globus job is finished successfully output (Usage Record format) has to be send to stellaris via cron job

GRAM Audit Logging part of globus container software to be declared in $GLOBUS_LOCATION/container-log4j.properties:... # GRAM AUDIT log4j.category.org.globus.exec.service.exec.StateMachine.audit=DEBUG, AUDIT log4j.appender.AUDIT=org.globus.exec.utils.audit.AuditDatabaseAppender log4j.appender.AUDIT.layout=org.apache.log4j.PatternLayout log4j.additivity.org.globus.exec.service.exec.StateMachine.audit=false output into a rel. database (PostgreSQL or MySQL), Database Connection has to be declared in $GLOBUS_LOCATION/etc/gram-service/jndi-config.xml:... url jdbc:mysql:// [:port]/auditDatabase user globus password foo... table update whenever a job ist started or changed it's status, database content is converted into Globus Usage Record format and sent to stellaris via trigger (perl)

GRAM Audit Database Schema ob_grid_id: String representation of the resource EPR local_job_id: Job/process id generated by the scheduler subject_name: Distinguished name (DN) of the user username: Local username idempotence_id: Job id generated on the client-side creation_time: Date when the job resource is created queued_time: Date when the job is submitted to the scheduler stage_in_grid_id: String representation of the stageIn-EPR (RFT) stage_out_grid_id: String representation of the stageOut-EPR (RFT) clean_up_grid_id: String representation of the cleanUp-EPR (RFT) globus_toolkit_version: Version of the server-side GT resource_manager_type: Type of the resource manager (Fork, Condor,...) job_description: Complete job description document success_flag: Flag that shows whether the job failed or finished successfully finished_flag: Flag that shows whether the job is already fully processed or still in progress converting db record into UR format via db trigger (e.g. perl)

Grid Architecture: Information Service Stellaris development of AstroGrid-D by ZIB (M. Högqvist, ZIB) available at (Apache License) metadata are exchanged and stored in RDF is used by AstroGrid-D for grid resource monitoring (e.g. compute resources, telescopes),and job execution monitoring tested and runs stable Resource Description Framework (RDF) standard for storing information based on graph theory, and presents information in triples different formats exist: RDF/XML (W3C recommendation), and Notation 3 (N3) (W3C specification) the SPARQL query language exists for retrieval of RDF information for Stellaris a conversion from RTML documents and UR XML into RDF required developed an XSLT (Extensible Stylesheet Language Transformation) at AIP can transform arbitrary XML files into RDF/XML format (e.g. RTML) usage very simple, only two command line programs „xsltproc“, and „cURL“ for uploading to Stellaris available at (Apache License)

SPARQL Queries and User Interface Web browser user interface Developed in AstroGrid-D by ZIB (M. Högqvist) & AIP (F. Breitling), Available under Apache License at currently used to display job-state metadata Based on „Simile“ Timeline like Google Maps for time-based information a DHTML-based AJAXy widget for visualizing time-based events SPARQL Query Language for RDF (W3C working draft) is the counterpart for retrieving information from RDF simple syntax for selected and sorted retrieval Perl modules exist for the integration into Perl scripts SPARQL web browser interface is implemented into Stellaris for a quick lookup

Outlook / Conclusion Installation in AstroGrid-D is done Tested, stable run on resources Information about job execution on each grid host Database interface, graphical representation with web interface Based on GRAM audit and Stellaris software monitoring of job execution for each grid resource is now available