Cracow Grid Workshop 2003 Institute of Computer Science AGH A Concept of a Monitoring Infrastructure for Workflow-Based Grid Applications Bartosz Baliś, Marian Bubak, Włodzimierz Funika, Tomasz Szepieniec, Marcin Radecki, Roland Wismüller, Tomasz Arodź, Marcin Kurdziel

Cracow Grid Workshop 2003 Institute of Computer Science AGH Plan  Workflow-based applications  Application Monitoring  Architecture of the monitoring infrastructure  Performance analysis  Monitoring of multilingual applications  Summary

Cracow Grid Workshop 2003 Institute of Computer Science AGH Workflow-based Grid Applications  Applications composed of a workflow of components  Components are independent Grid services

Cracow Grid Workshop 2003 Institute of Computer Science AGH Application Monitoring  Goal: provide monitoring services to Obtain debug / performance-related information Possibly enable manipulations on the target application Detect events and program actions to be executed when they occur  Consumers Tools for performance analysis / debugging Other tools and systems, e.g., for fault-tolerance, load balancing, etc.

Cracow Grid Workshop 2003 Institute of Computer Science AGH Monitoring infrastructure for Wf-based apps  Additional monitoring interface for each grid service (component) to be monitored Obtain monitoring info for that component  Global monitoring service Separate grid service One per application Obtain collective monitoring info

Cracow Grid Workshop 2003 Institute of Computer Science AGH Monitoring Wf-based Applications Client Monitoring service Additional monitoring interface (monitoring info related to the component) Global monitoring (grid) service. Collective monitoring information.

Cracow Grid Workshop 2003 Institute of Computer Science AGH Startup  User submits a workflow application and requests monitoring, possibly subsequently  Monitoring service is created It must discover the workflow components Workflow subsystem must provide a mechanism for this Workflow registry?

Cracow Grid Workshop 2003 Institute of Computer Science AGH Component monitoring interface – how?  Monitoring functionality inherent part of each component  How to provide the additional monitoring interface? Take monitoring interface into account at the design stage Component developers involved?  This approach (hopefully) enables instrumentation of grid services

Cracow Grid Workshop 2003 Institute of Computer Science AGH Performance analysis service  Combined with the monitoring service  Supports performance analysis of Grid infrastructure Application Intra-component Inter-component

Cracow Grid Workshop 2003 Institute of Computer Science AGH Infrastructure and intra-component analysis  Grid infrastructure Uses existing tools to monitor the status of grid environment of the components  Intra-component operations Measure some quantities describing the status and performance of a service used in the application workflow Depend on the monitoring interface defined by the component developers Reuse existing approaches to monitoring and analysis

Cracow Grid Workshop 2003 Institute of Computer Science AGH Inter-component performance monitoring  Adds a new level to the performance analysis of grid applications  Accounts for the concept of application as a workflow of services  Monitors the state and performance of the cooperation of the services to capture the state and performance of the whole application  Two modes: Structural, semantic monitoring mode Activity monitoring mode

Cracow Grid Workshop 2003 Institute of Computer Science AGH Structural, semantic monitoring  Captures information on workflow components: Component status Component usage Custom component properties  Uses the intra-component monitoring data on: Functions Code regions Synchronisation objects  Combines the information on various components to describe the status of the whole workflow based- application  Proposed techniques: Source/byte code instrumentation Dynamic instrumentation

Cracow Grid Workshop 2003 Institute of Computer Science AGH Activity monitoring  Captures information on the behaviour of the workflow  Analysed activity properties: Caller/callee relationships End-to-end response time for activity invocation Success rate of invocation Synchronisation time in activity Volume of data exchanged in activity  Proposed techniques: Distributed instrumentation of workflow components Insertion of probes

Cracow Grid Workshop 2003 Institute of Computer Science AGH Event Service  Automatically sends notifications to other grid services when some performance conditions are met.  Allows for a reaction on undesired changes of grid environment status as well as in the activity of the application.  Will be integrated with performance data provider in a uniform framework.  This integration will enable the notified service to inspect a wide range of performance information before deciding on the actions that should be taken.

Cracow Grid Workshop 2003 Institute of Computer Science AGH Two levels of performance information  A proven concept form the G-PM tool.  Used in both application performance data provider and event service.  Lower level performance information: a consistent, well-defined set of metrics, used, by system administrators, application developers, users and other parties.  Higher level performance information: more abstract, numerical, quantitative metrics, defined on the basis of the lower-level metrics with a special definition language, allowing for creation of various metrics suitable for a particular purpose.

Cracow Grid Workshop 2003 Institute of Computer Science AGH Monitoring of multi-lingual applications  Services can be implemented using multi-lingual approach e.g. as a computational kernels written in Fortran and wrapped by Java code.  A higher level abstraction of metrics must be worked out which could handle different programming approaches and languages.  The low-level monitoring system must be designed in a way that enables a simultaneous monitoring of Java, C, C++ and other language-based service parts using an uniform approach: the specification of monitoring services must be the same for all languages, the implementations of the monitoring services may vary.

Cracow Grid Workshop 2003 Institute of Computer Science AGH Summary  Workflow-based approach to applications still immature  Architecture for a monitoring infrastructure proposed Integration of monitoring services into components Global grid monitoring service  Performance analysis Resuse of existing approaches for infrastructure and intra-component analysis New level of inter-component analysis