Page 1Session 4: State of the art technology and standards, 12/10/2012 Web Processing Service Models for scientific exploitation of EO Data, Session 4: State of the art technology and standards Yves Coene, Spacebel s.a. Patrick Jacques, Spacebel s.a. Peter Baumann, Jacobs University Frascati - 12 October 2012

Page 2Session 4: State of the art technology and standards, 12/10/2012 Overview Introduction Web Processing Service Case Studies –Decision Support (DREAM) –Lunar Landing (LandSAfe) –Processing On Demand (G-POD, SSEGrid, ngEO) Open-source implementations Issues and future work Standardisation work –Ongoing –Planned Conclusion

Page 3Session 4: State of the art technology and standards, 12/10/2012 Introduction HMA Standardisation –Service Oriented Architectures –For GSCDA and Science Service Taxonomy (HMA) –Identity Management Services –Discovery Services –Invoke Services Feasibility Analysis Ordering Geospatial Processing: §3.4.7, §5.5.5 –Online Data Access Services

Page 4Session 4: State of the art technology and standards, 12/10/2012 Web Processing Service WPS is an OGC standard for: –describing and executing a geoprocessing operation via a service, –standardizing the requests and responses for a particular geospatial processing service. WPS is a generic model of a Web service supporting WSDL and REST, for which profiles need to be developed and standardised to support interoperability: –Syntax and semantics of input parameters –Input and output data format

Page 5Session 4: State of the art technology and standards, 12/10/2012 Web Processing Service Operations: –GetCapabilities Discover which processes are offered by the service* –DescribeProcess Discover input/output of a specific process Response can be used to build the user interface for capturing the input parameters**. –Execute Execute a process Storing response or returning response document (*) e.g. ProcessCoverages is a process that accepts WCPS request language as per OGC ( ** ) similar to DescribeTasking in OGC Earth Observation Satellite Tasking Extension of SPS.

Page 6Session 4: State of the art technology and standards, 12/10/2012 Web Coverage Processing Service The Web Coverage Processing Service (WCPS) specifies an expression language allowing for server-side filtering and processing of multi-dimensional raster coverages. It defines syntax and semantics of expressions for specifying extraction and analysis of coverages containing sensor, image, and statistics data. Not tied to a particular transmission protocol, the query paradigm can be embedded into any service framework, such as: –OGC Web Coverage Service (WCS) –OGC Web Processing Service (WPS).

Page 7Session 4: State of the art technology and standards, 12/10/2012 Web Coverage Processing Service Use Cases: –On-demand retrieval “thresholded NDVI where threshold value is obtained from user input” Ad-hoc queries –Processing “Fourier transform of scene X” –Filtering “Scenes with less than x% cloud coverage” –Summary data –Sensor fusion and pattern mining Slide 7

Page 8Session 4: State of the art technology and standards, 12/10/2012 WCPS By Example "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS" for $c in ( M1, M2, M3 ) return encode( abs( $c.red - $c.nir ), "hdf“ ) (hdf A, hdf B, hdf C ) 8

Page 9Session 4: State of the art technology and standards, 12/10/2012 WCPS By Example "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS" –…but only those where nir exceeds 127 somewhere for $c in ( M1, M2, M3 ) where some( $c.nir > 127 ) return encode( abs( $c.red - $c.nir ), "hdf“ ) (hdf A, hdf C ) 9

Page 10Session 4: State of the art technology and standards, 12/10/2012 WCPS By Example "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS" –…but only those where nir exceeds 127 somewhere –…inside region R for $c in ( M1, M2, M3 ), $r in ( R ) where some( $c.nir > 127 and $r ) return encode( abs( $c.red - $c.nir ), "hdf“ ) (hdf A ) 10

Page 11Session 4: State of the art technology and standards, 12/10/2012 Web Processing Service OGC r7, Web Processing Service, OGC Implementation standard. OGC r6, Corrigendum for... Web Processing Service. OGC r2, Web Coverage Processing Service (WCPS) Language Interface Standard, version OGC , Web Processing Service Best Practices, Discussion Paper, OGC Implementation standard. OGC : WPS-T Discussion Paper OGC , Web Processing Service – Application Profile for Coverage Processing. Version (status: discussed in WPS.SWG). * (*) OGC Standards.

Page 12Session 4: State of the art technology and standards, 12/10/2012 Case Study 1: DREAM DREAM Project –Dedicated to exploitation of EO data for Decision Support –Based on use cases of European Institutions: EUSC and EMSA –WPS used for Quality Assessment of ortho-images and Orthorectification of raw images: Horizontal Accuracy Maps Vertical Accuracy Maps Slide courtesy DREAM Project (ASTRIUM-GEO)

Page 13Session 4: State of the art technology and standards, 12/10/2012 O1 – Online Data Access Client O2 – Online Data Access Server Q1 - Data Quality Proxy Data Quality Service Photogrammetry Library Compute AOI WPS Processing Reference3D Database Reference Orthoimages DEM Compute RPC WPS Processing OGC WPS Service Interface Web Application Server Compute Quality WPS Processing OGC WPS Client OrthoRectify WPS Process AssessQuality WPS Process Ortho-Image Producer OGC WPS Service Interface Web JS Plug-in Viewer WMS …. File System Raw Acquisition Ortho-image Shift Shapefile QualityMask Coverage Q1 – Data Quality Client WMS Server WCS Server IF-DREAM- Q-DataQualityProxy IF-DREAM-Q-AssessQuality, IF-DREAM-Q-OrthoRectify IF-DREAM- Q-FileIO IF-DREAM-O-RegisterData [IF-HMA-Download] IF-OGC-View Slide courtesy DREAM Project (ASTRIUM-GEO) Raw Image Handler

Page 14Session 4: State of the art technology and standards, 12/10/2012 Case Study 1: DREAM DREAM, involving end users with massive data sets, allows to establish relevant demo scenarios. Sample WCPS use cases: –„all coverages with timestamp younger than T within bbox B where cloud percentage is below P“ User input: T, B, P Web client to allow input of T and P via sliders (or similar), B graphically –„Bit mask of fires detected in coverage C“ –„band ratio between B1 and B2 for given coverage C“ User input: B1, B2, C –Scenario cases to be refined with users.

Page 15Session 4: State of the art technology and standards, 12/10/2012 Case Study 2: LandSAfe Landing Site risk Analysis software framework (ESTEC) –ESA Lunar Lander Mission –Vision-based Navigation technologies for safe precision landing. –Hazard detection and avoidance.

Page 16Session 4: State of the art technology and standards, 12/10/2012 Case Study 2: LandSAfe Software modules for mission preparation stage and use during descent stage (live footage)

Page 17Session 4: State of the art technology and standards, 12/10/2012 Case Study 2: LandSAfe Processing requests sent over HTTP/SOAP to the WPS service which executes the request by running Python scripts from the LandSAfe Core. pyWPS See modules on previous slide

Page 18Session 4: State of the art technology and standards, 12/10/2012 Case Study 3: G-POD ESA Grid - Processing On Demand WPS access point of G-POD installation is http(s):// /wps/ WPS process –Refers to G-POD service –Has identifier, title and abstract –Output is RDF document referring to actual results G-POD service parameters are LiteralData (no ComplexData).

Page 19Session 4: State of the art technology and standards, 12/10/2012 Case Study 3: G-POD WPS as interface for ngEO Hosted Processing Facilities WPS e.g. G-POD

Page 20Session 4: State of the art technology and standards, 12/10/2012 Case Study 4: SSEGrid SSEGrid Project –Introduce a Grid Processing on Demand model at VITO to be integrated with Vegetation processing chains –Portal to integrate multi-Grid processing services to be used by the Principal Investigator (PI) for the systematic processing or re ‑ processing of EO data in a single or multi- mission environment –On-demand Processing of EO data Large EO data size  processing must be done where data is located Re-processing by PI  processing chains must be deployable by PI

Page 21Session 4: State of the art technology and standards, 12/10/2012 Use Cases EO Archive A: Upload Data / Deploy Processes / Design Workflows / Deploy Workflows / Submit Workflows B: Discover Data, Processes, & Workflows / Monitor Workflows / Download Results Portal & Workflow Engine SSEGrid PI UC01 - Manage SSEGrid Infrastructure UC03 - Provide Access to EO Data & Products Catalogue UC10 - Upload Data / Deploy Process UC11 - Design Single Mission Workflows UC12 - Design Multi Mission Workflows UC21 - Execute Multi Mission Workflow / Monitor & Influences the Execution UC02 - Manage User Administration Grid Administrator UC20 - Execute Single Mission Workflow / Monitor & Influences the Execution Portal Administrator UC30 - Delegate Data Upload, Process Deployment & Workflow Design

Page 22Session 4: State of the art technology and standards, 12/10/2012 Demonstration Scenario (FAPAR) Fraction of Absorbed Photosynthetically Active Radiation for each = parallel execution data transfer

Page 23Session 4: State of the art technology and standards, 12/10/2012 SSEGrid WPS Hierarchy Portal Federating Workflow Engine WPS-G Grid Workflow Engine WPS-G Grid Workflow Engine WPS-G Grid WPS-G Grid WPS-G User Processing Infrastructure (UPI) Federating Processing Infrastructure (FPI) Mission Processing Infrastructure (MPI) Grid Access Infrastructure (GAI) Grid Java SAGA Java SAGA PI One per mission Ground Segment Hierarchy of WPS-G services

Page 24Session 4: State of the art technology and standards, 12/10/2012 SSEGrid WPS-G Interface Encapsulates all forms of processing –Grid Processes –Workflows WPS-G Workflow Engine Workflow Deployment Monitoring Execute WPS-G Grid Meta - scheduler Deployment Monitoring GRAM MDS RFT Scheduler Nodes job queue Compute Elements Execute

Page 25Session 4: State of the art technology and standards, 12/10/2012 Highlighting of thresholded data Date Threshold AOI Target PNG (as URL) Image thresholding Display results AOI & threshold selection Image highlighting Highlight colour, fixed to (RGB=200,0,0) URL of thresholded PNG, from previous step Highlighted PNG, from previous step Case Study 5: SSE

Page 26Session 4: State of the art technology and standards, 12/10/2012

Page 27Session 4: State of the art technology and standards, 12/10/2012 Open-source Implementations FP7 GENESIS Toolbox –Evolution of SSE Toolbox –Supports WPS and corrigendum –Multiple bindings + asynchr. – PyWPS –Implementation in Python 52°North –Supports WPS –Supports WPS-T –Extended with WPS-G On-demand Processing in SSEGrid

Page 28Session 4: State of the art technology and standards, 12/10/2012 Open-source Implementations SSEGrid contribution: –Asynchronous Execute operation with callback (no polling) –New WPS-G specific operations GetStatus, Cancel GetAudit, DeployData, UndeployData, DescribeData –Two new WPS-T backends Apache ODE Process Manager (workflows) JavaSAGA Process Manager (Grid processes) –Backend callback operation Available as Open source at 52°North repository Spacebel

Page 29Session 4: State of the art technology and standards, 12/10/2012 Issues and future work Lack of support for: –Asynchronous operations (not polling) –Status reporting (Audit) –Cancel, GetAudit, … –Quota Management, –Discovery of SLA, QoS, Performance, availability Integrating Authentication/Authorisation and security –Restrict visibility/access to deployed Processing resources –Discover access restrictions Harmonising data model across services: –DescribeTasking (SWECommon) versus DescribeProcess

Page 30Session 4: State of the art technology and standards, 12/10/2012 Issues and future work Consistency with INSPIRE Spatial Data Services and Invoke Service (Draft) Implementing rules –May or may not retain OGC WPS. –Oriented towards WSDL and SOAP.

Page 31Session 4: State of the art technology and standards, 12/10/2012 Issues and future work Completion of standardisation of various interfaces defined in ESA TM-21 « HMA Cookbook » No yet OGC standards. Partly underway in WPS 2.0 SWG

Page 32Session 4: State of the art technology and standards, 12/10/2012 Ongoing Standardisation Activities SSEGrid Project: –Two Change Requests submitted for inclusion into WPS 2.0 –Processing On-Demand Extension (WPS-G) to be proposed when WPS 2.0 available WPS-G Processing On-Demand Extension Binding WPS-T Asynchronous

Page 33Session 4: State of the art technology and standards, 12/10/2012 WPS-G

Page 34Session 4: State of the art technology and standards, 12/10/2012 Ongoing Standardisation Activities SSEGrid Project: –“Processing On-Demand WPS Extension” –Covers changes and additions to WPS 1.0 –Available at

Page 35Session 4: State of the art technology and standards, 12/10/2012 WPS 2.0 Extensions WPS-T –Apache ODE (BPEL 2.0 Engine) deployment profile Archive (package) –Grid deployment profile Archive (scripts, libraries, executables) JSDL template WPS-G –New operations DeployData UndeployData DescribeData GetAudit –Changes Capabilities document (Data offerings) Execute Request WPS inputs JSDL template JSDL Extract Substitute Job Id Task Id …

Page 36Session 4: State of the art technology and standards, 12/10/2012 WPS with On-Demand Extension Feature/operations –Discovery of service metadata (WPS GetCapabilities operation) –Discovery of process metadata (WPS DescribeProcess operation) –Execution of processes (WPS Execute operation) –Deployment of processes (WPS-T DeployProcess operation) –Undeployment of processes (WPS-T UndeployProcess operation) –Deployment of (auxiliary) data (WPS-G DeployData operation) –Undeployment of data (WPS-G UndeployData operation) –Discovery of data metadata (WPS-G DescribeData operation) –Query of process status (WPS-G GetStatus operation) –Cancellation of process (WPS-G Cancel operation) –Query of process execution trace (WPS-G GetAudit operation) –Process completion/failure notification (callback operation) WPS-G Workflow 1: Execute 2: Callback Addressed by WPS 2.0 SWG

Page 37Session 4: State of the art technology and standards, 12/10/2012 Ongoing Standardisation Activities DREAM Project: –“Identify the requirements and scenarios, extend the WCPS to cover the needed multidimensional queries in particular for what concerns space/time phenomena related to the joint exploitation of S-2, optical high resolution and other data including geo-statistical data as the one routinely provided by EUROSTAT and other international organisations. “ –WCPS work items: WCPS 1.0 based on WCS Standardisation effort required to align with GMLCOV 1.0 and WCS 2.0 (Core) ATS to be updated ETS to be created

Page 38Session 4: State of the art technology and standards, 12/10/2012 Ongoing Standardisation Activities DREAM Project (cont’d) –space/time phenomena related to the joint exploitation of Sentinel-2, optical high resolution and other data Space-time coordinate reference systems (CRSs): addressed by OGC (CRS Name Type Specification) –geo-statistical data: ensure support for Data: allow „coverages“ with non-spatiotemporal axes, (age, population density,...) Service: support OLAP operations (slice, dice, aggregation,...)

Page 39Session 4: State of the art technology and standards, 12/10/2012 Planned Standardisation Activities HMA-S Project: –Revise WPS Specification taking into account: ESA project’s technical report “SSEGrid Processing On-Demand Extension (WPS-G)” ngEO needs in terms of process description and access to the processing outputs Relevant third party initiatives, about cloud computing standardisation, such as IEEE P2301. WPS 2.0 SWG ongoing work ESE Project: –Create new WPS-G deployment profile for Map-Reduce processing environment on dynamic resources (cloud).

Page 40Session 4: State of the art technology and standards, 12/10/2012 Conclusion Two main « Processing » specifications: WPS and WCPS WPS: –Widely adopted in ESA projects –Several open-source implementations available –Many subjects to be resolved, improved or not yet a formal standard E.g. asynchronous, WPS-T, … –Definition of OGC WPS 2.0 underway but slow process WCPS: –Less widely adopted –To be used in DREAM and ESE projects Various ongoing or planned ESA projects contribute to standardisation effort of both specifications.

Page 41Session 4: State of the art technology and standards, 12/10/2012 Conclusion Further Reading: –HMA Cookbook: publications/TM-21/TM-21.pdfhttp://esamultimedia.esa.int/multimedia/ – – – position_positionpaper_ pd__EN_1.pdf