Scripting based architecture for Management of Streams and Services in Real-time Grid Applications Authors Harshawardhan Gadgil, Geoffrey Fox, Shrideep Pallickara, Marlon Pierce Community Grids Lab, Indiana University, Bloomington Robert Granat NASA, Jet Propulsion Lab, Pasadena Presented By Harshawardhan Gadgil

May 10, 2005 Community Grids Lab Presentation for CCGrid Talk Outline Introduction to HPSearch Architecture Introduction to HPSearch Architecture A quick view of NaradaBrokering middleware A quick view of NaradaBrokering middleware System Goals System Goals –RDAHMM Example Related and Future Work Related and Future Work

May 10, 2005 Community Grids Lab Presentation for CCGrid HPSearch A JavaScript based scripting runtime serving as an administration console A JavaScript based scripting runtime serving as an administration console –Currently uses Rhino ( implementation of JavaScript. Management viewed as Management viewed as –Setting up the distributed application (Involves setting up the broker network, initializing system components) –Querying Run-time System Metadata  For logging purposes  Monitor metadata to help dynamically rewire system to improve performance

May 10, 2005 Community Grids Lab Presentation for CCGrid HPSearch Architecture Component Summary Binds URIs to a scripting language Binds URIs to a scripting language –Allows us to manage (manipulate) the Resource identified by the URI  E.g. Read / Write to files, sockets, topics  Read from data base, data from FTP/HTTP resource  This data can then be streamed to distributed application OR data can be read from a stream and processed / stored “Host-objects” allow us to dynamically access the host system. “Host-objects” allow us to dynamically access the host system. –Useful for constructing objects that monitor system meta-data, perform management tasks –PerfMetrics gathers system performance data and allows us to query it at run-time.

May 10, 2005 Community Grids Lab Presentation for CCGrid HPSearch Architecture Diagram Request Handler JavaScript Shell Task Scheduler Flow Handler Web Service EP Other Objects HPSearch Kernel URIHandler DBHandler WSDLHandler WSProxyHandler Request Handler HPSearch Kernel HPSearch Kernel Broker Network SOAP/HTTP... DataBase Web Service Files Sockets Topics Network Protocol JDBC SOAP/HTTP WSProxy Service WSProxy Service WSProxy Service HPSearch Control Events using PUB/SUB on predefined topic Data buffers sent / received as Narada Events

May 10, 2005 Community Grids Lab Presentation for CCGrid NaradaBrokering NaradaBrokering NaradaBrokering –Messaging infrastructure for collaboration, peer-to- peer and Grid applications –Implements high-performance protocols (message transit time of 1 to 2 ms per hop) –Order-preserving, optimized message transport with QoS and security profiles for sent and received messages –Support for different underlying protocols such as TCP, UDP, Multicast, RTP –Discovery Service to locate nearest brokers

May 10, 2005 Community Grids Lab Presentation for CCGrid HPSearch + NaradaBrokering Managing Streams HPSearch uses NaradaBrokering to route data streams HPSearch uses NaradaBrokering to route data streams –Each stream is represented by a topic name –Components subscribe / publish to specified topic  The WSProxy component automatically maps topics to Input / Output streams  Each write (byte[] buffer) and byte[] read() call is mapped to a NaradaBrokering event

May 10, 2005 Community Grids Lab Presentation for CCGrid System Goals Investigate the use of HPSearch as a management console to deploy system and application components using scripting Investigate the use of HPSearch as a management console to deploy system and application components using scripting Use HPSearch in scientific / grid applications which can use streaming data Use HPSearch in scientific / grid applications which can use streaming data –Data filtering is essential in most cases and we do not want to transfer entire data set when only a small percentage of data would be used,  E.g. Choosing data satisfying certain input constraints. –Data re-ordering might be required for formatting input data to match the requirements of executable,  E.g. Converting GML to remove XML elements Use publish-subscribe methodologies to connect components. Use publish-subscribe methodologies to connect components.

May 10, 2005 Community Grids Lab Presentation for CCGrid GPS time series have modes caused by unknown underlying physical processes GPS time series have modes caused by unknown underlying physical processes RDAHMM allows us to identify these modes and time periods where physical processes dominated the sequence without any a-priori knowledge of these processes RDAHMM allows us to identify these modes and time periods where physical processes dominated the sequence without any a-priori knowledge of these processes Help to determine Help to determine –The actual physical causes –When is the system entering a new mode (perhaps a hint of some important seismic event) Example RDAHMM

May 10, 2005 Community Grids Lab Presentation for CCGrid Example Application Components GPS database (Surface displacement time series collected by SCIGN, GPS database (Surface displacement time series collected by SCIGN, Data filter (filters data, removes unwanted components, reorders data as required by the actual RDAHMM executable) Data filter (filters data, removes unwanted components, reorders data as required by the actual RDAHMM executable) RDAHMM executable, performs the time series analysis. This data is transferred to the Graph plotting application. RDAHMM executable, performs the time series analysis. This data is transferred to the Graph plotting application. Matlab based Graph plotting application Matlab based Graph plotting application

May 10, 2005 Community Grids Lab Presentation for CCGrid HPSearch Engine Applications Streaming Data Filtering Data Filter Filters the input data to get only the estimate and error values RDAHMM Analyze the data Matlab Plotting Script Output Graph HPSearch Handler GPS Data Sensor Source Sensor Source Handler Each Handler controls operation of 1 service

May 10, 2005 Community Grids Lab Presentation for CCGrid Sample Output Claremont, CA

May 10, 2005 Community Grids Lab Presentation for CCGrid Related Work Scripting languages have been very popular and successful for Rapid Application Deployment (RAD) Scripting languages have been very popular and successful for Rapid Application Deployment (RAD) –Sash by IBM, allow RAD and handlers for various tasks such as reading from databases, LDAP registries, invoking Web Services and providing GUI –Jython and Matlab are popular in many scientific communities. E.g. GeoDISE –XCAT project at IU, Extreme Lab uses Jython for scripting to deploy distributed components –Karajan (Ant Like scripting) is used for deploying applications over grid –WSRF::Lite uses perl to implement Web Services Resource Framework and host grid services

May 10, 2005 Community Grids Lab Presentation for CCGrid Future Work Investigate system and management scaling with increasing number of components. Investigate system and management scaling with increasing number of components. Use of security for streams Use of security for streams Negotiation of optimal (high performance) transport Negotiation of optimal (high performance) transport More handlers for different aspects of NaradaBrokering’s management (broker / topic discovery, security, replay etc…) More handlers for different aspects of NaradaBrokering’s management (broker / topic discovery, security, replay etc…) Investigate interaction with WS Management and WS - Distributed Management based systems Investigate interaction with WS Management and WS - Distributed Management based systems

May 10, 2005 Community Grids Lab Presentation for CCGrid To conclude… Presented a scripting based architecture for management of data streams and distributed services Presented a scripting based architecture for management of data streams and distributed services Shown how we can use publish subscribe methodologies to connect components that process data in a stream Shown how we can use publish subscribe methodologies to connect components that process data in a stream Create and deploy quick data filtering applications Create and deploy quick data filtering applications

May 10, 2005 Community Grids Lab Presentation for CCGrid More Information HPSearch Project Website HPSearch Project Websitehttp:// NaradaBrokering IU NaradaBrokering IUhttp:// CGL Publications CGL Publicationshttp://grids.ucs.indiana.edu/ptliupages/publications/

May 10, 2005 Community Grids Lab Presentation for CCGrid Questions / Comments Any Questions / Comments ? Any Questions / Comments ?THANKS for attending the presentation