1 Grids for Real-time and Streaming Applications GCC2005 Beijing China December Geoffrey Fox Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN

2 Four Data Streaming Application Areas Data Assimilation applied to link the data deluge (satellites, sensors, seismometers) in real time to small and large scale parallel simulations Use in Earthquake Science Department of Defense (and Homeland Security) have built the Global Information Grid with a target architecture NCOW (Network Centric Operations and warfare) They submit no jobs; rather stream data to brokers from which they are filtered and distributed Includes their rather dated distributed simulation HLA Audio-Video Conferencing implemented with services and Grid messaging Hand-held Grid linking PDA/cell-phones to Grids

3 Database SS SSSSSSSSS FS FSFS Portal FSFS OSOS OSOS OSOS OSOS OSOS OSOS OSOS OSOS OSOS OSOS OSOS OSOS MD MetaData Filter Service Sensor Service Other Service Another Grid Raw Data  Data  Information  Knowledge  Wisdom Decisions S S Another Service S Another Grid S SS FS SOAP Messages

4 Key Concepts Grid of Grids (System of Systems) allows “library” approach to composing Grids Service Oriented architectures (Web or Grid services) are attractive for many/most distributed systems There are many applications that are NOT best considered as jobs and files (classic Grid) but rather as streams and filters (services) Services exchanging messages becomes Services exchanging streams (sets of messages) Publish-Subscribe messaging gives better QoS and management than point to point messaging with negligible performance loss Always use standards including those for GIS

5 Typical use of Grid Messaging in NASA Datamining Grid Sensor Grid Grid Eventing GIS Grid

6 GIS and Sensor Grids OGC has defined a suite of data structures and services to support Geographical Information Systems and Sensors GML Geography Markup language defines specification of geo-referenced data SensorML and O&M (Observation and Measurements) define meta-data and data structure for sensors Services like Web Map Service, Web Feature Service, Sensor Collection Service define services interfaces to access GIS and sensor information Grid workflow links services that are designed to support streaming input and output messages We are building Grid (Web) service implementations of these specifications for NASA’s SERVOGrid

7 Grid Eventing via NaradaBrokering The Scripps Orbit and Permanent Array Center (SOPAC) GPS station network data published in RYO format is converted to ASCII and GML

8 Grid Messaging with Grid of Grids Architecture HPSearch Manages Narada Brokering Sensor Grid WS-Context Stores dynamic data WFS (GIS data) Post before Processing Post after Processing Notify Subscribe Database Archives Web Feature Service GIS Grid Geographical Information System DoD Principles Filter or Datamining Grid

9 Earthquake Faults from the WMS Client

10 WMS uses WFS that uses data sources Northridge2 Wald D. J , ,34.176

11 Google Map Client Google Central Google Map Client UDDI WFS2 Databases with SERVOGrid Faults WFS1 SOAP Sensor Grid HTTP Helper Services ArchivedReal Time DoD and Homeland Security can in a crisis combine custom geo-referenced data with that available from hundreds of thousands of computers from Microsoft, Yahoo and Google Just build simple services using Interoperability standards!

12 Real Time GPS and Google Maps Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients. Select and zoom to GPS station location, click icons for more information.

13 Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records. Integrating Archived Web Feature Services and Google Maps

14 Google Maps as Service accessed from our WMS Client

15 Google plus GIS Grid Integrated with Los Alamos Critical Infrastructure Simulations for DHS Natural Gas Layer Energy Power Layer

16 NaradaBrokering Messaging infrastructure for collaboration, peer-to-peer and Grids Implements JMS and native high-performance protocols (message transit time of 1 to 2 ms per hop) Order-preserving message transport with QoS and security profiles Support for different underlying transport such as TCP, UDP, Multicast, RTP SOAP message support and WS-Eventing, WS-RM and WS-Reliability. WS-Notification when specification agreed Active replay support: Pause and Replay live streams. Stream Linkage: can link permanently multiple streams – using in annotation of real-time video streams Replicated storage support for fault tolerance and resiliency to storage failures. Management: HPSearch Scripting Interface to streams and brokers (uses WS-Management) Broker Topics and Message Discovery: Locate appropriate Integration with Axis2 Web Service Container (?) High Performance Transport supporting SOAP Infoset

Transit Delay (Milliseconds) Message Payload Size (Bytes) Mean transit delay for message samples in NaradaBrokering: Different communication hops hop-2 hop-5 hop-7 hop-3 Pentium-3, 1GHz, 256 MB RAM 100 Mbps LAN JRE 1.3 Linux

18 Average Video Delays for one broker – Performance scales proportional to number of brokers Latency ms # Receivers One session Multiple sessions 30 frames/sec

19 Collaboration Grid Services UDDI Narada Broker HPSearch WS-Context Gateway WS-Security Narada Broker Gateway XGSP Media Service Video Mixer Transcoder Audio Mixer Replay Record Annotate Thumbnail WhiteBoard SharedDisplay SharedWS

20 GlobalMMCS SWT Client Chat TV WebcamVideo Mixer GIS

21 e - Annotation Player Archived stream player Annotation / WB player Archieved stream list Real time stream list e - Annotation Whiteboard Real time stream player Archived Real Time Real Time Stream List Stream List Player e-Annotation Archived Stream Annotated e-Annotation Player Player Stream Player Whiteboard

22 Location of software for Grid Projects in Community Grids Laboratory htpp:// provides Web service (and JMS) compliant distributed publish-subscribe messaging (software overlay network) htpp:// htpp:// is a service oriented (Grid) collaboration environment (audio-video conferencing) htpp:// is an OGC (open geospatial consortium) Geographical Information System (GIS) compliant GIS and Sensor Grid (with POLIS center) has WS-Context, Extended UDDI etc. The work is still in progress but NaradaBrokering is quite mature All software is open source and freely available