ECMWF 24 th November 2008 Deploying secure OGC services in front of a heterogeneous data archive. Bryan Lawrence, Phil Kershaw, Dominic Lowe, and Stephen Pascoe. STFC Rutherford Appleton Laboratory

ECMWF 24 th November 2008 Outline  Introduction to NERC DataGrid  Introduction to the Centre for Environmental Data Archival.  OGC Web Services  CEDA OGC Web Services (COWS) Software Stack  CSML  Software Architecture  COWS in the wild  NERC Portals Project  UKCIP (UK Climate Impacts Project) Data Delivery Portal  The Future

ECMWF 24 th November 2008 NERC DataGrid Aim: We need to integrate and compare lots of different types of data … held in multiple locations and formats under various access control regimes. Project Runtime: 6 years. Credits: We ’ d like to think in some way we ’ ve encouraged Meteorology folk to take OGC technology seriously enough that we ’ re here today!

ECMWF 24 th November 2008 NDG Products: GML compliant application schema: Climate Sciences Modelling Language (CSML). –Information Model and –Python Software Stack NDG Security –Python Software Stack MOLES (Metadata objects for linking environmental sciences) –Information Model COWS (CEDA OGC Web Services) –Python server software stack MILK (Metadata & Information Lead to Knowledge) –Python Client Software Stack NDG Discovery –Java SOAP discovery service NDG Vocabulary Server

ECMWF 24 th November 2008 NDG Deployment Multiple services (going beyond OGC, arguably unnecessarily!) –Concentrate here on the OGC Server Stack Multiple Portals Multiple Data Providers –Concentrate here on just one data provider: CEDA

ECMWF 24 th November 2008 CEDA Centre for Environmental Data Archival British Atmospheric Data Centre NERC Earth Observation Data Centre IPCC Data Distribution Centre UK Projects –NDG (NERC Data Grid) –C-SEKT (New CSML and CSML-WFS) –OJMS (Data Publication) –UKCIP08 (UK-Scale Climate Predictions) EU Projects –Metafor (Climate Model Documentation) –EUFAR (Distributed Archive for European Aircraft) –IS-ENES (Distribute Archive for European Climate Model Data) Approximately 25 staff, and half a PB of data … connected internally to SuperJanet (UK Academic network, 10 Gbit/s) at 1 Gbit/s (soon to be 10 Gbit/s all the way).

ECMWF 24 th November 2008 OGC Web Services Open Geospatial Consortium Specifications – Web Map Service (WMS) – Web Coverage Service (WCS) – KML ! – Web Processing Service (WPS) – Web Feature Service (WFS)

ECMWF 24 th November 2008 COWS: CEDA OGC Web Services CEDA OGC Web Services Lightweight OGC Service stack in Python Designed to support: –Multiple OGC services –Integration with bespoke Web Applications –Data described by GML application schema: CSML. –Data stored according to a CSML storage descriptor in a variety of formats

ECMWF 24 th November 2008 Key Aspects of CSML CSML: Climate Sciences Modelling Language Application schema of Geography Markup Language (GML) Describes datasets made up of feature collections. –Each feature instance is associated with data described in a storage descriptor.

ECMWF 24 th November 2008 Selected CSML Feature Types PointSeriesFeature (timeseries at a point) ProfileFeature (vertical profile at a point) GridSeriesFeature (series of multidimensional grids) SwathFeature (single satellite sweep) SectionFeature (vertical section) Feature Types are classified by their geometry (Slide courtesy of Jon Blower)

ECMWF 24 th November 2008 Heterogeneous Storage CSML Storage Descriptor: Designed to support Separation of concerns between storage and semantics Legacy high volume storage. Inline data etc

ECMWF 24 th November 2008 HTTP Server Paste Pylons Middleware Web Application (Portal, MILK, PyDAP etc) COWS WMSWCSW*S CSML (& Storage Descriptor) WSGI Apache + mod_python, FastCGI, Python HTTPD e.g. Authentication NetCDFGRIB Oracle Idealised COWS Application Library Standard Interface Application-specific Code Server or Server + Client Pylons: Python Web Application Framework WSGI: Python Web Server Gateway Interface Paste: Python WSGI Deployment Framework

ECMWF 24 th November 2008 WSGI Middleware Add security to a COWS Application by adding security middleware Python Paste package enables inclusion of middleware via a config file – no change to COWS code required: CowsApplication AnotherFilter AuthenticationFilter [pipeline:CowsApplication] pipeline = AuthenticationFilter AnotherFilter CowsApplication Example middleware chain: HTTP Request HTTP Response

ECMWF 24 th November 2008 NERC Portals: HiGEM WMS Portion of HiGEM dataset over WMS Uses COWS CSML Plugin OpenLayers as WMS Client Climatological time selectable as a dimension on the WMS KML generator links to WMS endpoint Enables display on GoogleEarth

ECMWF 24 th November 2008 NERC Portals Project 2 Different Station datasets Integrate with CSML Pylons Application plots CSML Features 2 visualisation interfaces KML references plot service COWS WMS provides in-browser support.

ECMWF 24 th November 2008 Station time series via COWS WMS WMS Browser ClientGoogleEarth Client Times Series displayed via plot service

ECMWF 24 th November 2008 UKCIP08 DDP Highly Dimensional Dataset – Time Period – Emission Scenario – Percentile (CDF)‏ – Meaning Period UKMO UM Rotated Grid but we need to plot it on GoogleMaps (Transverse Mercator) Custom WMS operation to serve Grid-box values as JSON

ECMWF 24 th November 2008 The Future Technical Refactoring: separate existing codebase More easily deployable code More easily configured access control (authentication and authorisation) More OGC services (WFS, WPS) Extend underlying CSML libraries for all feature types. CSML V3: better support for cell methods and cell bounds –i.e. propagating information from storage into CSML and out via services into the clients … Deployment Connect wider range of CEDA datasets Support services for CMIP5/AR5 Exploit UK National Academic Grid via WPS Use of OGC services as standard middleware in NERC interoperability. Complete INSPIRE conformance Better interoperability with ESA Connection to GEOSS