Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter Kacsuk Laboratory of Parallel and Distributed Systems MTA SZTAKI Hungary

Computer and Automation Research Institute Hungarian Academy of Sciences Contents ● Relational Grid Information and Monitoring System (R-GMA) ● R-GMA Browser – Browsing status information ● Pulse – Analysis and presentation of performance information of grid services and components ● GRM/PROVE – Performance monitoring of applications

Computer and Automation Research Institute Hungarian Academy of Sciences R-GMA ● R-GMA is the Relational Monitoring and Information System within the EU-DataGrid project ● Gives the illusion of a virtual relational database of the information available about grid services, resources and jobs ● SQL is the information publishing and query language

Computer and Automation Research Institute Hungarian Academy of Sciences R-GMA Sensor Code Producer API Application Code Consumer API Producer Servlet Registry API Registry Schema API Schema “Definition of tables” Consumer Servlet Registry API

Computer and Automation Research Institute Hungarian Academy of Sciences R-GMA

Computer and Automation Research Institute Hungarian Academy of Sciences R-GMA

Computer and Automation Research Institute Hungarian Academy of Sciences Pulse ● On-line analysis and visualisation tool for ● monitoring data of resources and services and applications.

Computer and Automation Research Institute Hungarian Academy of Sciences Pulse ● Data source components  preprocessing components  view components their connections are defined in XML ● Separates data model from visualisation modules ● based on Model-View-Controller paradigm

Computer and Automation Research Institute Hungarian Academy of Sciences Pulse ● Data model: meta-data (name, type, unit, range) and interface to the data source. ● Controller: preprocessing, analysis components ● View: displaying components

Computer and Automation Research Institute Hungarian Academy of Sciences Example: CPU Load

Computer and Automation Research Institute Hungarian Academy of Sciences Example: CPU Load

Computer and Automation Research Institute Hungarian Academy of Sciences Example: CPU Load

Computer and Automation Research Institute Hungarian Academy of Sciences Data model ● Data channel – Name – Data type – Unit – Range of valid data LoadChannelMetaData( name=”load1”, type=float, unit=””, range=[ ] )

Computer and Automation Research Institute Hungarian Academy of Sciences Input components ● Input component (sensor): an interface to some kind of external data source (file, service, an executable, etc.) ● A sensor – Reads data from the external source – Translates data into a channel based representation – Publishes complex information in several channels – Provides one piece of information at a time ● Buffered sensor – Buffering capability to a sensor

Computer and Automation Research Institute Hungarian Academy of Sciences Presentation components ● To show the results to the user ● Tabular view, Histogram plotter ● Presentation components subscribe to the sensor's notification service -> they are notified about the status changes in the sensor -> display updates are controlled by the sensors (incoming data)

Computer and Automation Research Institute Hungarian Academy of Sciences Analysis components ● Essentially: – Take data from other components (sensors) – Do something with the data – Forward it to other components (plotter) ● E.g. – Channel selector (for histogram plotter) – Merge data from several sensors – Compute statistical values on data

Computer and Automation Research Institute Hungarian Academy of Sciences Configuration of Pulse ● XML configuration to define – Components – Connections between the components

Computer and Automation Research Institute Hungarian Academy of Sciences Pulse: conclusion ● Pulse can be used to build an analysis chain to process dynamic information and present results in a graphical form ● Flexible data representation ● Already it can be used as a tool for the R-GMA grid monitoring and information system

Computer and Automation Research Institute Hungarian Academy of Sciences Application monitoring ● Performance monitoring (not job status!) of ● Parallel applications (MPI, PVM, P-GRADE) ● Large volume of trace information ● On-line transfer of trace to the user

Computer and Automation Research Institute Hungarian Academy of Sciences GRM at EuroPar’2001 Main Monitor MM Application Process Site 1 Local Host Host 1Host 2Host 1 Local Monitor LM Site 2 Local Monitor LM Site Monitor SM Appl. Process shm

Computer and Automation Research Institute Hungarian Academy of Sciences Tools ● GRM – Instrumentation library and trace collector ● Mercury monitor – Transfer trace data from the application processes to GRM ● R-GMA – Information system for finding the application and the monitor components

Computer and Automation Research Institute Hungarian Academy of Sciences Mercury monitor in the GridLab project

Computer and Automation Research Institute Hungarian Academy of Sciences GRM+Mercury+R-GMA

Computer and Automation Research Institute Hungarian Academy of Sciences Mercury monitor ● To deliver trace data from the application to the user efficiently. – Uses TCP Socket communication – Data in XDR format and could be optimised for TCP transmission – Two sw. hops between application and GRM: local and main monitors – One hw. hop: host of main monitor

Computer and Automation Research Institute Hungarian Academy of Sciences R-GMA ● To find the application by the user/GRM – Where is it running? -> machineX.siteY – What is its global job id? -> GID ● To find the monitor to be connected – What is the address of Mercury monitor running at siteY? ● To find the application by the monitor – What processes (PIDs) belong to application GID?

Computer and Automation Research Institute Hungarian Academy of Sciences GRM ● Instrumentation library – Connect and publish trace to Mercury monitor ● GRM and PROVE – Look for application asking R-GMA – Connect to Mercury monitor and query for trace – Visualise trace information with PROVE

Computer and Automation Research Institute Hungarian Academy of Sciences Current status

Computer and Automation Research Institute Hungarian Academy of Sciences Future work ● Intrusion: replace socket communication between LM and application process for shared-memory buffering (as in original GRM) ● Scalability: data reduction at LMs and MM.

Computer and Automation Research Institute Hungarian Academy of Sciences Conclusion ● Different monitoring information can be presented with different techniques – Web browser is used to search among available information and static (or slowly changing) behaviour of the grid – Pulse can be used to build specific analysis chains to process dynamic information about the services and resources and to create graphical plots – GRM is a tool for performance monitoring of parallel applications, connected to Mercury and R-GMA to work for grid.

Computer and Automation Research Institute Hungarian Academy of Sciences Thank you

Computer and Automation Research Institute Hungarian Academy of Sciences R-GMA Schema Browser

Computer and Automation Research Institute Hungarian Academy of Sciences R-GMA Schema Browser