1. HMA-FO ODA, June 2011 HMA Follow-On Task 3 Online Data Access (ODA) in the frame of ESA's Heterogeneous Missions Accessibility (HMA) initiative Final Presentation ESRIN 20 – 21. June 2011

2. HMA-FO ODA, June 2011 Agenda Slide 2 Targets & Achievements Requirements & Use Cases New OGC Standards Reference Implementations Alliance with FOSS/MapServer Deliverables ECSS Documents OGC Software EO-WCS Software EOxServer Architecture EOxServer Functionality WebMapViewer Client EO-WCS Concepts Overview Definitions Dataset Series - Past Dataset Series - Now Stitched Mosaics – Now Improvements due to WCS and EO-WCS Future Developments Summary Task-3 Demonstration Server – Adding Data Client

3. HMA-FO ODA, June 2011 Targets & Achievements (1)  Collect Requirements for Online Data Access  Include GSC-DA issues on how to handle datasets and collections  Identify requirements and scenarios for transactional tasks (adding & updating of mosaics, while keeping track of the history)  Requirement Baseline documents including an extensive set of Use Cases have been delivered  Review, analyze and update standards addressing coverages  At Kick-Off, a change to the, at that time, premature WCS 2.0 specification has been requested and was accepted  WCS 2.0 had been finalized, submitted and is now an OGC Implementation Standard Slide 3

4. HMA-FO ODA, June 2011 Targets & Achievements (2)  Define a "WCS Application Profile for EO Sensors" (EO-WCS) Document  Submit to OGC and guide its evolution  EO-WCS has been defined and Document is submitted as OGC 10-140 “WCS Earth Observation Application Profile” to OGC  OGC OAB Motion (2011-05-06): Approve OGC 10-140 for release for public comment (after applying some editorial changes). Slide 4

5. HMA-FO ODA, June 2011 Targets & Achievements (3)  Demonstrate the proposed specification by extending an OpenSource Software  MapServer has been chosen as platform and has been extended for WCS 2.0. This extension is included in MapServer 6.0 (released: 2011-05-12)  The EO-WCS has been implemented, based on a modular approach (Unix philosophy: "Write programs that do one thing and do it well.").  EO-WCS Software publicly released as: (http://eoxserver.org)  Developments are entirely based on OpenSource software  Released under the GNU GPL 3 Open Source license  Newest release is: 0.1.2, dated: 2011-05-04  Sustainably maintained through: Applying policies and procedures as laid out by OSGeo e.g. forming of a PSC (http://eoxserver.org/doc/en/rfc/rfc0.html) Subversion code repository (trunk, releases, etc.) Trac installation providing issue tracker, wiki, etc. Mailman mailing lists Slide 5

6. HMA-FO ODA, June 2011 Targets & Achievements (4)  Demonstrate the specification using an updated WCS client  WebMapViewer Client has been updated to handle WCS 2.0 and EO-WCS  Set up a demonstration and test server  The Demonstration Server (WCS 2.0 / EO-WCS) at ESA (within a virtual machine) is available at:  http://ssews1.esrin.esa.int/wcseo/ows?SERVICE=WCS &VERSION=2.0.0  http://ssews1.esrin.esa.int/wcseo-test/ows?SERVICE=WCS &VERSION=2.0.0  Perform ATS, ETS and TEAM engine testing  WCS 2.0 Test Suite is available, based on the OGC 09-147r1 Abstract Test Suite (https://svn.opengeo- spatial.org/ogc-projects/cite/scripts/wcs/2.0.0/) Slide 6

7. HMA-FO ODA, June 2011 Slide 7 Alliance with FOSS/MapServer  Results from participation at MapServer Code- Sprint MapServer community valued the chosen approach of EOxServer for providing the EO functionality by extending MapServer (separating imaging engine and metadata handling) Full integration of EO-WCS functionality into MapServer was discussed, but is not seen as adequate approach (EO metadata handling too specific) MapServer community proposed to host/distribute EOxServer on MapServer platform to form a strong alliance. Further discussion at FOSS4G (Sept., Denver) Targets & Achievements (5)

8. HMA-FO ODA, June 2011 Deliverables - ECSS Slide 8 Documents – HMA WikiDateStatus DJF-Software Implementation – Rationale v.1.0 (DJF-TN)2001-02-24Final DDF-Software User Manual (Client) v.1.0 (DDF-SUM)2011-01-25Final DDF-Software User Manual (Server) v.1.0 (DDF-SUM)2011-01-21Final DJF-Software Verification Report v.1.1 (DJF-SVR)2011-01-21Final DJF-Software Validation Specification v.1.2 (DJF-SVS)2011-01-21Final DDF - Software Design Document v.1.0 (DDF-SDD)2010-11-27Final DJF - Software Reuse File v.1.0 (DJF-SRF)2010-11-27Final TS - Software Requirements Specification v.1.2 (TS-SRS)2010-11-27Final RB - Software System Specification v1.4 (RB-SSS)2010-11-04Final Use Cases applied to WCS 2.0 EO AP TN2010-10-27Final RB - Technical Note v1.12010-07-06Final HMA_Specification_Dependencies_(Cros_linking)2010-04-28Final Metadata_requirements_1.0_(RB) Spot Image2010-03-01Final

9. HMA-FO ODA, June 2011 Deliverables - OGC Slide 9 Documents – OGCDateStatus WCS_2.0 Bundle (http://www.opengeospatial.org/standards/wcs)  WCS 2.0 Core (OGC 09-110r3)  GML 3.2.1 Application Schema - Coverages (OGC 09-146r1)  WCS 2.0 Extension: KVP protocol (OGC 09-147r1)  WCS 2.0 Extension: XML/POST protocol (OGC 09-148r1)  WCS 2.0 Extension: SOAP protocol (OGC 09-149r1)  …plus 20 XML schema & example files (submitted 2010-11-04) passed 2010-08-18) Adopted as Implementation Specification OGC 10-140 OGC Web Coverage Service 2.0 Interface Standard - Earth Observation Application Profile v.0.2.0 (2011-01-19) (https://portal.opengeospatial.org/files/?artifact_id=42579) 2011-05-06 Approval by OGC OAB WCS 2.0 Test Suite is available, based on the OGC 09-147r1 Abstract Test Suite (https://svn.opengeospatial.org/ogc- projects/cite/scripts/wcs/2.0.0/) 2011-05-10First version OGC 11-010 WCS 2.0 Extension for CF-netCDF & OPeNDAP2011-03-02Draft OGC 10-147 WCS 2.0 Format Extension - GeoTIFF v.0.0.12010-07-06Draft WCS 2.0 Overview: Core & Extensions (Best Practice) (OGC 09-153) 2010-11-04Draft

10. HMA-FO ODA, June 2011 Developments - Software Slide 10 SoftwareStatus WCS 2.0 extension to MapServer 6.0 (official release)2011-05-12 EOxServer v.0.1.2 (EO-WCS) Software Bundle (download – http://eoxserver.org) 2011-05-04 WebMapViewer WCS 2.0/EO-WCS enabled Client (GIM)2011-03-17 Demonstration Server (WCS 2.0 / EOxServer) v.0.1.2 at ESA (within a virtual machine) is online 2011-05-17

11. HMA-FO ODA, June 2011 Slide 11 Architecture 1/3  Written in Python  Based on entirely on Open Source Software (MapServer, Django, GDAL, SpatialLite or PostGIS, PROJ.4, etc.)  MapServer has been extended to support WCS 2.0 (see ticket at http://trac.osgeo.org/mapserver/ticket/3595)  Extension is included in MapServer 6.0  Implements Model – View – Controller concept

12. HMA-FO ODA, June 2011 Slide 12 Architecture 2/3

13. HMA-FO ODA, June 2011 Slide 13 Architecture 3/3

14. HMA-FO ODA, June 2011 Slide 14 Functionality 1/3  Implements EO-WCS and EO-WMS on top of MapServer's 6.0 WCS and WMS implementations  Support of GML AP – Coverages for RectifiedGridCoverages  Support of adopted WCS 2.0 specification  Anticipated support of envisaged extensions: Coverage format, GeoTIFF encoding, predefined (or EPSG) CRSs, scaling & interpolation, and non-referenced access  Protocol bindings supported  KVP (tested)  XML/POST (used together with future SOAP Proxy (currently developed in the O3S project) to support XML/SOAP protocol binding)

15. HMA-FO ODA, June 2011 Slide 15 Functionality 2/3  Support of 2-D EO Coverages derived from gmlcov:RectifiedGridCoverage  Support of Dataset Series as a collection of EO Coverages e.g. in a time series  Support of new DescribeEOCoverageSet operation on Dataser Series and EO Coverages  Support of Stitched Mosaic of Rectified EO Coverages including concept of contributingFootprint  Support of EO Metadata (retrieval and evaluation in DescribeEOCoverageSet operation)

16. HMA-FO ODA, June 2011 Slide 16 Functionality 3/3  Support of EO-WMS for EO Coverages  Coverage formats supported  GeoTIFF  Formats supported by the GDAL library (support needs to be verified for each required format separately)  Extensive unit tests based on MERIS sample data documenting available functionality  Documentation available based on Sphinx

17. HMA-FO ODA, June 2011 Slide 17  Based on evolution of the Service Support Environment (SSE) WebMapViewer WMS,WFS,WCS, SOS Client  Implemented with a new technology baseline (JSR286 Portlets, Java Server Faces)  Combining thin browser based Javascript client with portrayal server and Content Management System  Work performed in HMA-FO was to  Support the WCS 2.0 Core Specification  Support the EO extension of WCS 2.0 WCS / EO-WCS Client

18. HMA-FO ODA, June 2011 Slide 18  Allows visualization of coverages, dataset series and stitched mosaics with functionality to  Influence portrayal  RGB/Grey scales  Band mapping  Image enhancement  Transparency  Discover coverages inside dataset series  Show coverage metadata  Download coverages WCS / EO-WCS Client

19. HMA-FO ODA, June 2011 EO-WCS Concepts - Overview Slide 19  Data structures:  EO Coverage = Coverage + EO Metadata + Lineage RectifiedGridCoverage or ReferenceableGridCoverage  Dataset = 2D “horizontal” EO Coverage  Stitched Mosaic = homogeneous grouping Can be seen as a coverage itself  Dataset Series = heterogeneous grouping  Operations:  Capabilities document: more flexible  DescribeCoverage, GetCoverage: as known  DescribeEOCoverageSet: spatio-temporal search possibility

20. HMA-FO ODA, June 2011 Definitions EO-WCS EO vocabularyEO-WCS scene / image / product / raster data (image) / raw data (image) / orthoimage dataset mosaic / seamless mosaic dataset google-like mosaic / composite mosaic stitched mosaic series / time seriesdataset series collectiondataset series Slide 20

21. HMA-FO ODA, June 2011 Definitions EO-WCS TermDefinition grid refers to data represented by values at each point in a "raster" coordinate system (as opposed to "vector" geodata) quadrilateral a polygon with 4 sides, usually a rectangle or square, but could also be a rhombus, kite, etc. rectified grid grid having regular spacing in a projected or geographic CRS referenceable grid not rectified, but associated with (one or more) coordinate transformations which relate the image or engineering CRS to a projected or geographic CRS Slide 21

22. HMA-FO ODA, June 2011 Definitions EO-WCS EO vocabularyEO-WCS scene / image / product / raster data (image) / raw data (image) / orthoimage dataset mosaic / seamless mosaicdataset long lat long lat j i rectified grid coverage refereceable grid coverage Slide 22

23. HMA-FO ODA, June 2011 Definitions EO-WCS EO vocabularyEO-WCS series / time seriesdataset series extraction from a 2-D dataset series by repeated GetCoverage Requests t long lat DescribeEOCoverageSet provides a set of dataset descriptions Slide 23

24. HMA-FO ODA, June 2011 Definitions EO-WCS EO vocabularyEO-WCS google-like mosaic / composite mosaic stitched mosaic lat long lat long t dataset series 2-D stitched mosaic Slide 24 DescribeEOCoverageSet (mosaic ID) provides a set of descriptions of the dataset used in the mosaic DescribeCoverage (mosic ID) provides a descriptions of the mosaic

25. HMA-FO ODA, June 2011 Definitions EO-WCS EO vocabularyEO-WCS collectiondataset series long lat t A B D E C A ….. dataset (e.g. scene) B ….. dataset (e.g. seamless mosaic) C ….. datasets of dataset series (e.g. time series) D ….. datasets of dataset series E ….. dataset series (e.g. collection) Slide 25 A dataset may be a member in different dataset series

26. HMA-FO ODA, June 2011 Dataset Series - Past  Discover the required data using a Catalog  Go to FTP-site  Discover respective files  Download 1 st big file  Wait until download is finished  Download 2 nd big file  Wait …  Download 3 rd big file  Wait …  Download other big files  Wait …  Extract the AOI from each file and throw away the rest What needs to be done if you just need a small AOI ?

27. HMA-FO ODA, June 2011 Slide 27 With all that waiting, one may end up like this …. Credit: Ambro Dataset Series - Past

28. HMA-FO ODA, June 2011 Slide 28  You may discover the required data using a Catalog  You may Order using direct links from the Catalog for each item OR …  You may use a EO-WCS enabled Client  Request information about the dataset series  Supply the AOI  Supply your time-frame (TOI)  You may request details for each Dataset  Request the datasets of your AOI/TOI and  Wait until download of the AOIs requested is finished What can EO- WCS do for you if you just need a small AOI ? Dataset Series - Now

29. HMA-FO ODA, June 2011 Slide 29 That's much more rewarding … Dataset Series - Now Credit: Ambro

30. HMA-FO ODA, June 2011 StitchedMosaic - Now Slide 30  But each pixel will keep its history  Each coverage has its metadata record  Overlaying coverages replace existing ones EO Metadata Record-1EO Metadata Record-2 EO-WCS allows to keep track of the metadata history of stitched mosaics?

31. HMA-FO ODA, June 2011 Improvements with WCS / EO-WCS Slide 31  Users  No more endless file-listings on FTP-Servers without knowing file-content  Less data download necessary  you get only what you really need (AOIs/TOIs)  Access to original data  Providers  Offer original data or rectified data  No data duplication required  Possibility of full integration into complex service chains (e.g. pass a coverage to a Web Processing Service, WPS)  Better quality of service provisioning

32. HMA-FO ODA, June 2011 Slide 32 WCS & EOxServer Future (1)  Aim: EOxServer to become official Reference Implementation of EO-WCS  Full integration of results from OWS-8  Support of the envisaged band subsetting extension  Coverage format support: NetCDF (including the option for CF-netCDF) (via GDAL) JPEG2000 (via some available Open Source implementation (TBD)) HDF-EOS (via GDAL)  Ongoing development in parallel ESA funded project O3S (Online Open-standard Observation Service)  Demonstrate scalability with large number of coverages (  FP7 EarthServer Project)  Coupling of rasdaman and EOxServer (combine WCPS and EO-WCS)

33. HMA-FO ODA, June 2011 Slide 33 Future (2)  Support FTP as Back-end access protocol (e.g. towards legacy Rolling Archives, Virtual Archives)  Demonstrate suitability for complex Use Cases  WCS-T, WCPS, ReferenceableGridCoverages  WCS extensions (Standards & Implementation): CRS, Scaling & Interpolation, RangeSubsetting  Tools for automation of Administration tasks (e.g. hourly updating of offered coverages)  Provision of TEAM 2.0 compatible conformance test suites  Generalize EOxServer to support other Map-Servers  Update WCS Client to support further extensions WCS & EOxServer

34. HMA-FO ODA, June 2011 Slide 34 Task-3 Achievements  WCS 2.0 released as OGC Implementation Standard  EO-WCS as OGC Implementation Standard (to be released soon)  Implementation of WCS 2.0 Server (  MapServer)  Implementation of EO-WCS Server (  EOxServer)  Implementation of WCS 2.0/EO-WCS enabled Client (  WebMapViewer ) Summary

35. HMA-FO ODA, June 2011 Slide 35 Demonstration – Server  Administration Interface – show easiness of adding a Dataset Series to EOxServer (3min) Demonstration – Client  Integrated in overall Demo-Scenario  Discover EO Product within a Dataset Series through AOI and time period (TOI)  Visualize Coverage  Influence Portrayal  Download Coverage  Show coverage metadata Demonstration

36. HMA-FO ODA, June 2011 Contacts  EOX IT Services GmbH, Austria (Prime) stephan.meissl@eox.at christian.schiller@eox.at  G.I.M. Geographic Information Management nv/sa, Belgium steven.smolders@gim.be  Jacobs University Bremen gGmbH, Germany p.baumann@jacobs-university.de  Spot Image SA, France philippe.merigot@spotimage.fr