Computer and Automation Research Institute Hungarian Academy of Sciences SZTAKI’s work in DataGrid WP September Norbert Podhorszki Laboratory of Parallel and Distributed Systems MTA SZTAKI

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 2 Contents GRM and PROVE Pulse

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 3 GRM and PROVE Monitoring and visualisation of parallel programs Goal: Connect GRM to R-GMA –Modify instrumentation library to act as a Producer –Modify Main Monitor of GRM to act as a Consumer –Prove can be untouched –C API to R-GMA is needed

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 4 GRM Main Site User’s Host Host 1Host 2 Application Process Appl. Process R-GMA PROVE Connection to R-GMA

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 5 R-GMA Sensor Code Producer API Application Code Consumer API ProducerServlet Registry API Registry Servlet Schema API Schema Servlet “Event Dictionary” Consumer Servlet Registry API Main Monitor Instrumented code

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 6 GRM – RGMA How to use Instrument and compile your application with GRM. grm_instr.hinstrumentation header file libgrmon.alibrary to be linked to the application grm_Start should give a name (e.g. MyApp) unique for the whole grid Submit/execute the application Start GRM main monitor giving the unique application name grm –o trace_file MyApp Main monitor prints its listening socket port address (e.g. 6100) After execution start PROVE and read trace file

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 7 GRM Instrumentation library RGMA C API Two CircularBufferProducer to publish trace data Two schemas (relational tables) GRMHeader(String appName, String event) appName: unique identifier for this application event: GRM event string as is contains important header events (START) for P-GRADE also: GROUP, CHANNEL, TEMPLATE etc. small buffer in servlet (10 elements) GRMTrace(String appName, String event) all other trace events (BEGIN-END, SEND-RECEIVE etc.) local buffer (100) + large remote buffer (1000)

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 8 GRM Main monitor RGMA C API First, look for application: create a Consumer with select statement “SELECT * FROM GRMHeader WHERE appName = “...” events flushed into trace file R-GMA gives back all producers that produces such data. If found then create a streaming Consumer for trace data “SELECT * FROM GRMTrace WHERE appName = “...” remote buffer in ConsumerServlet for streaming (100)

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 9 Buffers in trace delivery Application process Producer API Local Buffer Application process Producer API Local Buffer ProducerServlet PServlet API Remote Buffer ConsumerServlet CServlet API Remote Buffer GRM – Consumer Consumer API ProducerServlet PServlet API Remote Buffer Trace file

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 10 Small PVM program worked

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 11 GRM – RGMA Problems: GRM CircularBufferProducer can loose data (overwrite buffer before it is read) PROVE should be ready for this! Buffer sizes can help a bit. In theory, no limit on buffer sizes in R- GMA Could RGMA report to Consumer if there was an overwrite? GRM main monitor exit and restart later? Tried only with a single, simple application. Parallel programs? Tried to use with P-GRADE but could not. Apps aborted always. Simple PVM program worked. Nested blocks are not allowed in PROVE (from P-GRADE) but might be a natural requirement for general monitoring.

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 12 GRM – RGMA Problems: RGMA Application exits  Producer exits  ProducerServlet exits, too  all data not delivered to ConsumerServlet are destroyed what about GRRP connection between servlets to keep all data in stream. Timeout application process uses local buffer to send events in bucks. no “near on-line” behaviour if events are generated rarely. timeout is defined but not implemented in RGMA C api Throughput performance? is the “quick-fixed bug in libwww” in the C api correctly solved? measurements needed

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 13 GRM – RGMA improvements Streaming single event per pop operation, i.e. communication between Consumer and ConsumerServlet One pop for the whole buffer in ConsumerServlet could improve performance Idea 1: when streaming, cons.execute() could give back the content of the buffer. This does not hurt the api. Idea 2: introduce new function: popAll() What about the streaming between the servlets? One-by-one or in blocks.

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 14 Improvement? Application process Producer API Local Buffer Application process Producer API Local Buffer ProducerServlet PServlet API Remote Buffer ConsumerServlet CServlet API Remote Buffer GRM – Consumer Consumer API ProducerServlet PServlet API Remote Buffer Trace file ConsumerServlet CServlet API Remote Buffer Create a ProducerServletConnection for each process and do not use Mediator?

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 15 Improvement? Application process Producer API Local Buffer Application process Producer API Local Buffer ProducerServlet PServlet API Remote Buffer GRM – Consumer Consumer API + something ProducerServlet PServlet API Remote Buffer Trace file Connect to ProducerServlets directly? Needs to write a consumer which implements features of ConsumerServlet. This is not clean RGMA anymore!!!

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 16 Pulse

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 17 Pulse Analysis and visualisation tool for monitoring data of resources and services and applications. Java code data source components  preprocessing components  view components their connections are defined in XML

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 18 Pulse Separates data model from visualisation modules based on Model-View-Controller paradigm Data model: meta-data (name, type, unit, range) and interface to the data source. Controller: preprocessing, analysis components View: displaying components

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 19 Pulse

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 20 The Pulse

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 21 The Pulse

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 22

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 23 The Pulse

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 24 Pulse

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 25 R-GMA Schema Browser Basic connection to R-GMA –RGMASimpleConsumer sensor component in Pulse which maps R-GMA data model to the data model of Pulse –Both single query and streaming supported Browser –get list of schemas (tables) from SchemaServlet –simple streaming query on a selected table (select * from table) –compose query based on selected table(s) Only tabular view currently.

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 26 R-GMA Schema Browser

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 27 R-GMA Simple Query

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 28 R-GMA Simple Query

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 29 Pulse: Future More visualisation modules for different kinds of monitoring data Analysis: modules for data analysis can be included in Pulse Interactivity in Pulse

Computer and Automation Research Institute Hungarian Academy of Sciences 2 September 2002, SZTAKI reportDataGrid WP3 30 Automatic Search for Bottlenecks APART project outcome: Apart Specification Language ASL in Java: JavaPSL from T. Fahringer, TU Vienna Talked about to use JavaPSL library and performance bottleneck search engine in Pulse. Need –to define performance bottlenecks for Grid resources Theoretical work, see Zs. Németh’s presentation –to collect necessary data through RGMA, –to implement in Pulse what is needed to connect all of it.