Implementing Geographical Information System Services for SERVOGrid Marlon Pierce Community Grids Lab Indiana University

SERVOGrid Components Component (“portlet”)-based portals. Component (“portlet”)-based portals. OGCE mentioned by Chris HillOGCE mentioned by Chris Hill Web Services for “execution grid” services Web Services for “execution grid” services Ant-based job specificationAnt-based job specification File transferFile transfer Distributed session management (“context”).Distributed session management (“context”). Geographic Information System (GIS) services for “data grid” services. Geographic Information System (GIS) services for “data grid” services. Web Map ServiceWeb Map Service Web Feature ServiceWeb Feature Service GIS-compatible information services.GIS-compatible information services. Support for streaming, real-time data. Support for streaming, real-time data. Distributed service management/orchestration Distributed service management/orchestration Using events and data streams.Using events and data streams.

Guiding Principles Grids are composed of families of services Grids are composed of families of services Data, execution, information, …Data, execution, information, … Use “WS-I+” approach to building service families. Use “WS-I+” approach to building service families. Build Grids out of Web Service standards conservatively.Build Grids out of Web Service standards conservatively. WS-Interoperability is the starting point. WS-Interoperability is the starting point. See position paper ServiceGrids.pdfSee position paper ServiceGrids.pdf ServiceGrids.pdf ServiceGrids.pdf SOAP and WSDL provide universal messaging framework and service definition language. SOAP and WSDL provide universal messaging framework and service definition language. All services should communicate with the same message format.All services should communicate with the same message format. Message delivery is left as an exercise.Message delivery is left as an exercise. Implementations are interesting. Implementations are interesting.

Pattern Informatics (PI) PI is a technique developed at University of California, Davis for analyzing earthquake seismic records to forecast regions with high future seismic activity. PI is a technique developed at University of California, Davis for analyzing earthquake seismic records to forecast regions with high future seismic activity. They have correctly forecasted the locations of 15 of last 16 earthquakes with magnitude > 5.0 in California.They have correctly forecasted the locations of 15 of last 16 earthquakes with magnitude > 5.0 in California. See Tiampo, K. F., Rundle, J. B., McGinnis, S. A., & Klein, W. Pattern dynamics and forecast methods in seismically active regions. Pure Ap. Geophys. 159, (2002). See Tiampo, K. F., Rundle, J. B., McGinnis, S. A., & Klein, W. Pattern dynamics and forecast methods in seismically active regions. Pure Ap. Geophys. 159, (2002). bin/fulltext?format=application/pdf&identifier=oai%3Aar Xiv.org%3Acond-mat%2F http://citebase.eprints.org/cgi- bin/fulltext?format=application/pdf&identifier=oai%3Aar Xiv.org%3Acond-mat%2F http://citebase.eprints.org/cgi- bin/fulltext?format=application/pdf&identifier=oai%3Aar Xiv.org%3Acond-mat%2F http://citebase.eprints.org/cgi- bin/fulltext?format=application/pdf&identifier=oai%3Aar Xiv.org%3Acond-mat%2F PI is being applied other regions of the world, and John has gotten a lot of press. PI is being applied other regions of the world, and John has gotten a lot of press. Google “John Rundle UC Davis Pattern Informatics”Google “John Rundle UC Davis Pattern Informatics”

Pattern Informatics in a Grid Environment PI in a Grid environment: PI in a Grid environment: Hotspot forecasts are made using publicly available seismic records.Hotspot forecasts are made using publicly available seismic records. Southern California Earthquake Data Center Southern California Earthquake Data Center Advanced National Seismic System (ANSS) catalogs Advanced National Seismic System (ANSS) catalogs Code location is unimportant, can be a service through remote executionCode location is unimportant, can be a service through remote execution Results need to be stored, shared, modifiedResults need to be stored, shared, modified Grid/Web Services can provide these capabilitiesGrid/Web Services can provide these capabilities Problems: Problems: How do we provide programming interfaces (not just user interfaces) to the above catalogs?How do we provide programming interfaces (not just user interfaces) to the above catalogs? How do we connect remote data sources directly to the PI code.How do we connect remote data sources directly to the PI code. How do we automate this for the entire planet?How do we automate this for the entire planet? Solutions: Solutions: Use GIS services to provide the input data, plot the output dataUse GIS services to provide the input data, plot the output data Web Feature Service for data archives Web Feature Service for data archives Web Map Service for generating maps Web Map Service for generating maps Use HPSearch tool to tie together and manage the distributed data sources and code.Use HPSearch tool to tie together and manage the distributed data sources and code.

WFS + Seismic Rec. WSDL WFS + State Bounds WSDL WMS + OnEarth “REST” … Aggregating WMS Stubs WFS + Seismic Rec. Stubs WSDL SOAP HTTP

GIS Behind the Scenes The web features are served up by a Web Feature Service. The web features are served up by a Web Feature Service. Web Map Service aggregates maps Web Map Service aggregates maps NASA OnEarth + our own renderings.NASA OnEarth + our own renderings. We re-implement Open Geospatial Consortium standards using Web Service Standards. We re-implement Open Geospatial Consortium standards using Web Service Standards. SOAP messages, WSDL service definitions.SOAP messages, WSDL service definitions. Will allow us to separate messages from HTTP transport layer in future.Will allow us to separate messages from HTTP transport layer in future. More WMS Info: More WMS Info: sayar.pdf. sayar.pdf. sayar.pdfhttp://grids.ucs.indiana.edu/ptliupages/publications/acm-gis- sayar.pdf atics05_asayar.pdf. atics05_asayar.pdf. atics05_asayar.pdfhttp://grids.ucs.indiana.edu/ptliupages/publications/Geoinform atics05_asayar.pdf More WFS Info: More WFS Info: More general info, software, demos: More general info, software, demos:

Tying It All Together: HPSearch HPSearch is an engine for orchestrating distributed Web Service interactions HPSearch is an engine for orchestrating distributed Web Service interactions It uses an event system and supports both file transfers and data streams.It uses an event system and supports both file transfers and data streams. Legacy nameLegacy name HPSearch flows can be scripted with JavaScript HPSearch flows can be scripted with JavaScript HPSearch engine binds the flow to a particular set of remote services and executes the script.HPSearch engine binds the flow to a particular set of remote services and executes the script. HPSearch engines are Web Services, can be distributed interoperate for load balancing. HPSearch engines are Web Services, can be distributed interoperate for load balancing. Boss/Worker modelBoss/Worker model ProxyWebService: a wrapper class that adds notification and streaming support to a Web Service. ProxyWebService: a wrapper class that adds notification and streaming support to a Web Service. More info: More info:

Data Filter (Danube) PI Code Runner (Danube)  Accumulate Data  Run PI Code  Create Graph  Convert RAW -> GML WFS (Gridfarm001) WMS HPSearch (TRex) HPSearch (Danube) HPSearch hosts an AXIS service for remote deployment of scripts GML (Danube) WS Context (Tambora) NaradaBroker network: Used by HPSearch engines as well as for data transfer Actual Data flow HPSearch controls the Web services Final Output pulled by the WMS HPSearch Engines communicate using NB Messaging infrastructure Virtual Data flow Data can be stored and retrieved from the 3 rd part repository (Context Service) WMS submits script execution request (URI of script, parameters)

Support for Real Time Applications

RDAHMM: GPS Time Series Segmentation Slide Courtesy of Robert Granat, JPL Complex data with subtle signals is difficult for humans to analyze, leading to gaps in analysis Complex data with subtle signals is difficult for humans to analyze, leading to gaps in analysis HMM segmentation provides an automatic way to focus attention on the most interesting parts of the time series HMM segmentation provides an automatic way to focus attention on the most interesting parts of the time series GPS displacement (3D) length two years. Divided automatically by HMM into 7 classes. Features: Dip due to aquifer drainage (days ) Hector Mine earthquake (day 626) Noisy period at end of time series

Towards Real-Time RDAHMM A real-time version of RDHAMM could potentially be used to detect state change events in live data from a GPS station. A real-time version of RDHAMM could potentially be used to detect state change events in live data from a GPS station. SCIGN maintains 125+ GPS stations, so trivially parallel RDAHHM clones can monitor state changes in the entire network. SCIGN maintains 125+ GPS stations, so trivially parallel RDAHHM clones can monitor state changes in the entire network. But first we must get the data to RDAHMM. But first we must get the data to RDAHMM.

NaradaBrokering: Message Transport for Distributed Services NB is a distributed messaging software system. NB is a distributed messaging software system. ng.orghttp:// ng.org NB system virtualizes transport links between components. NB system virtualizes transport links between components. Supports TCP/IP, parallel TCP/IP, UDP, SSL.Supports TCP/IP, parallel TCP/IP, UDP, SSL. See e.g. u/ptliupages/publications/A llHands2005NB-Paper.pdf for trans-Atlantic parallel tcp/ip timings. See e.g. u/ptliupages/publications/A llHands2005NB-Paper.pdf for trans-Atlantic parallel tcp/ip timings. u/ptliupages/publications/A llHands2005NB-Paper.pdf u/ptliupages/publications/A llHands2005NB-Paper.pdf

SOPAC GPS Services

NaradaBrokering topics

More Information Contact: Contact: GIS Work at CGL: GIS Work at CGL: Software, demos, publicationsSoftware, demos, publications Several recent manuscript submissions are/will be posted soon.Several recent manuscript submissions are/will be posted soon. HPSearch at CGL: HPSearch at CGL: SERVOGrid Web Sites SERVOGrid Web Sites Our fine parent projectOur fine parent project

Acknowledgements Geoffrey Fox, Community Grids Lab director. Geoffrey Fox, Community Grids Lab director. Shrideep Pallickara: NaradaBrokering design/development lead Shrideep Pallickara: NaradaBrokering design/development lead Grad Students: Ahmet Sayar, Galip Aydin, Mehmet Aktas, Harshawadhan Gadgil Grad Students: Ahmet Sayar, Galip Aydin, Mehmet Aktas, Harshawadhan Gadgil

Backup Slides

SERVO Apps and Their Data GeoFEST: Three-dimensional viscoelastic finite element model for calculating nodal displacements and tractions. Allows for realistic fault geometry and characteristics, material properties, and body forces. GeoFEST: Three-dimensional viscoelastic finite element model for calculating nodal displacements and tractions. Allows for realistic fault geometry and characteristics, material properties, and body forces. Relies upon fault models with geometric and material properties.Relies upon fault models with geometric and material properties. Virtual California: Program to simulate interactions between vertical strike-slip faults using an elastic layer over a viscoelastic half-space. Virtual California: Program to simulate interactions between vertical strike-slip faults using an elastic layer over a viscoelastic half-space. Relies upon fault and fault friction models.Relies upon fault and fault friction models. Pattern Informatics: Calculates regions of enhanced probability for future seismic activity based on the seismic record of the region Pattern Informatics: Calculates regions of enhanced probability for future seismic activity based on the seismic record of the region Uses seismic data archivesUses seismic data archives RDAHMM: Time series analysis program based on Hidden Markov Modeling. Produces feature vectors and probabilities for transitioning from one class to another. RDAHMM: Time series analysis program based on Hidden Markov Modeling. Produces feature vectors and probabilities for transitioning from one class to another. Used to analyze GPS and seismic catalog archives.Used to analyze GPS and seismic catalog archives. Can be adapted to detect state change events in real time.Can be adapted to detect state change events in real time. We will focus on the latter two. We will focus on the latter two.

Some SERVOGrid Research Challenges

Problems with Conventional Web Services Transport: HTTP Request/Response is a poor choice for non-trivial data transport. Transport: HTTP Request/Response is a poor choice for non-trivial data transport. Much better to stream out data without knowing the content-length.Much better to stream out data without knowing the content-length. Representation: ASCII XML is inefficient in obvious and not so obvious ways. Representation: ASCII XML is inefficient in obvious and not so obvious ways. For example, WS security depends upon canonicalization to make reproducible message digests.For example, WS security depends upon canonicalization to make reproducible message digests. Efficiency and performance is not just a high performance computing problem. Efficiency and performance is not just a high performance computing problem. Needed to support PDAs and other devicesNeeded to support PDAs and other devices

NaradaBrokering and Web Services SOAP 1.2 defines a message routing across distributed SOAP Nodes. SOAP 1.2 defines a message routing across distributed SOAP Nodes. Naturally maps to an NB implementation.Naturally maps to an NB implementation. This has just been released from has just been released from NB also has support for WS-Eventing and WS- ReliableMessaging. NB also has support for WS-Eventing and WS- ReliableMessaging. More generally, we argue for the use of software messaging substrates to provide/implement desirable “quality of service” features More generally, we argue for the use of software messaging substrates to provide/implement desirable “quality of service” features Transport, routing/addressing, reliability, security, discovery, etc.Transport, routing/addressing, reliability, security, discovery, etc. Specific service capabilities (like “run job”, “move file”, “query data”) are decoupled from the substrate capabilities.Specific service capabilities (like “run job”, “move file”, “query data”) are decoupled from the substrate capabilities.

Efficient XML Representation The XML Infoset provides an abstract data model. The XML Infoset provides an abstract data model. SOAP 1.2 is defined using the Infoset.SOAP 1.2 is defined using the Infoset. This separates XML from “angle bracket notation” restrictions. This separates XML from “angle bracket notation” restrictions. Infoset-compliant binary representations are possible.Infoset-compliant binary representations are possible. No loss of data, so you can translate between binary and ascii representations.No loss of data, so you can translate between binary and ascii representations. Current lab research investigates hand-held applications. Current lab research investigates hand-held applications. See P_CTS05.pdfSee P_CTS05.pdf P_CTS05.pdf P_CTS05.pdf But easily extensible to high performance transport problems. But easily extensible to high performance transport problems.

More Information Contact: Contact: GIS Work at CGL: GIS Work at CGL: Software, demos, publicationsSoftware, demos, publications Several recent manuscript submissions are/will be posted soon.Several recent manuscript submissions are/will be posted soon. HPSearch at CGL: HPSearch at CGL: SERVOGrid Web Sites SERVOGrid Web Sites Our fine parent projectOur fine parent project

A Big Picture for SERVOGrid

RDAHMM: SCIGN GPS Network Analysis Slide Courtesy of Robert Granat, JPL Have found a way to detect regional aseismic signals Have found a way to detect regional aseismic signals This software is being integrated with the Quakesim web portal This software is being integrated with the Quakesim web portal Scenarios for use with real time streaming data through the web portal are currently being investigated Scenarios for use with real time streaming data through the web portal are currently being investigated Now segment all 127 GPS stations In blue: Number of stations that change state on a given day In red: Seismic activity Days with many state changes often do not correlate with large earthquakes.

Support for Streaming Data We use NaradaBrokering messaging software to manage data streams and filters. We use NaradaBrokering messaging software to manage data streams and filters. Open source, Java-based software from the Community Grids LabOpen source, Java-based software from the Community Grids Lab Based on topic-based publication/subscription for delivery of messages from/to multiple endpoints.Based on topic-based publication/subscription for delivery of messages from/to multiple endpoints. “Message” can be anything, including SOAP and binary data streams.“Message” can be anything, including SOAP and binary data streams. We use this for audio/video collaboration.We use this for audio/video collaboration. More recently using it to build Web Service messaging substratesMore recently using it to build Web Service messaging substrates SOAP 1.2 routing model, WS-Reliability, WS-Eventing SOAP 1.2 routing model, WS-Reliability, WS-Eventing NB ensures reliable delivery of events in the case of broker or client failures and prolonged entity disconnects. NB ensures reliable delivery of events in the case of broker or client failures and prolonged entity disconnects. Also supports replay.Also supports replay. Implements high-performance protocols (message transit time of 1 to 2 ms per hop) Implements high-performance protocols (message transit time of 1 to 2 ms per hop)

GPS Stations Current implementation provides real-time access to GP messages to following stations in RYO, ASCII and GML formats: Current implementation provides real-time access to GP messages to following stations in RYO, ASCII and GML formats:

SOPAC GPS Services As a case study we implemented services to provide real- time access to GPS position messages collected from several SOPAC networks. As a case study we implemented services to provide real- time access to GPS position messages collected from several SOPAC networks. Next step is to couple data assimilation tools (such as RDAHMM) to real-time streaming GPS data. Next step is to couple data assimilation tools (such as RDAHMM) to real-time streaming GPS data. Next steps Next steps Programming APIs: currently we assume the subscriber speaks NaradaBrokering Java APIs (either NB’s native API or Java Messaging Service).Programming APIs: currently we assume the subscriber speaks NaradaBrokering Java APIs (either NB’s native API or Java Messaging Service). Need to investigate appropriate Web Service standards and C/C++ bindings. Need to investigate appropriate Web Service standards and C/C++ bindings. SOAP enveloping of the GML message stream.SOAP enveloping of the GML message stream. A Sensor Collection Service will be implemented to provide metadata about GPS sensors in SensorML.A Sensor Collection Service will be implemented to provide metadata about GPS sensors in SensorML.

Position Messages SOPAC provides 1-2Hz real-time position messages from various GPS networks in a binary format called RYO. SOPAC provides 1-2Hz real-time position messages from various GPS networks in a binary format called RYO. Position messages are broadcasted through RTD server ports. Position messages are broadcasted through RTD server ports. We have implemented tools to convert RYO messages into ASCII text and another that converts ASCII messages into GML. We have implemented tools to convert RYO messages into ASCII text and another that converts ASCII messages into GML.

Real-Time Access to Position Messages We have a Forwarder tool that connects to RTD server port to forward RYO messages to a NB topic. We have a Forwarder tool that connects to RTD server port to forward RYO messages to a NB topic. RYO to ASCII converter tool subscribes this topic to collect binary messages and converts them to ASCII. Then it publishes ASCII messages to another NB topic. RYO to ASCII converter tool subscribes this topic to collect binary messages and converts them to ASCII. Then it publishes ASCII messages to another NB topic. ASCII to GML converter subscribes this topic and publishes GML messages to another topic. ASCII to GML converter subscribes this topic and publishes GML messages to another topic.