GO-ESSP LLNL, June 2006 CSML – Stocktake and Forward Look Andrew Woolf Dominic CCLRC Rutherford Appleton Laboratory with contributions from NDG Team (Bryan Lawrence, Roy Lowry, Kerstin Kleese van Dam, Ray Cramer, Siva Kondapalli, Sue Latham, Kevin O’Neill, Ag Stephens)

GO-ESSP LLNL, June 2006 Outline CSML: History CSML: Present CSML: Future

GO-ESSP LLNL, June 2006 CSML: History NERC DataGrid: the integration problem –multiple organisations, formats, storage mechanisms (file, relational) –only commonality is data semantics

GO-ESSP LLNL, June 2006 CSML: History The standards-based approach –“Conceptual modelling is the process of creating an abstract description of some portion of the real world and/or a set of related concepts.” (ISO 19101) Universe of discourse Feature types ISO Application schema ISO ISO The taxon name taxon 'ANTHOZOA',63.1,missing 'Scoloplos armiger',66.1,missing 'Spio filicornis',10,missing 'Spiophanes bombyx',60.3,missing 'Capitellidae',131.8,missing 'Pholoe',10,missing 'Owenia fusiformis',23.4,missing 'Hypereteone lactea',6.8,missing 'Anaitides groenlandica',13.2,missing 'Anaitides mucosa',6.8,missing ISO 19118

GO-ESSP LLNL, June 2006 CSML: History Is this standards stuff relevant?! One example: –EU INSPIRE Directive Establishes European Spatial Data Infrastructure – mandating interoperability of environmental data across EEA “[ISO/OGC] standards and specifications... will be considered as reference material for the Consolidation Team and relevant Drafting Teams.” (INSPIRE Work Programme Preparatory Phase 2005 – 2006) Annex III Data Themes, incl.: –Atmospheric Conditions –Meteorological geographical features –Oceanographic geographical features –Sea regions

GO-ESSP LLNL, June 2006 CSML: History grep ‘ISO|OGC|WFS|WCS|WMS|GML’ {AGU,AMS,GO-ESSP}/* | wc ?

GO-ESSP LLNL, June 2006 CSML: History British Oceanographic Data Centre British Atmospheric Data Centre ISO standards ‘Governance Principle’ Climate Science Modelling Language CSML application schema Conceptual model Schemas Tooling

GO-ESSP LLNL, June 2006 CSML: Present Conceptual model numerical weather prediction model, ocean general circulation model Timeseries of gridded parameter fields.GridSeriesFeature vertical or scanning radar, shipborne ADCP, thermistor chain timeseries Series of profile-type measurements.ProfileSeriesFeature tidegauge, rainfall timeseriesSeries of single datum measurements.PointSeriesFeature gridded analysis fieldSingle time-snapshot of a gridded field.GridFeature wind sounding, XBT, CTD, radiosonde Single ‘profile’ of some parameter along a directed line in space.ProfileFeature raingauge measurementSingle point measurement.PointFeature ship’s cruise track, aircraft’s flight path Discrete path in time and space of a platform or instrument.TrajectoryFeature ExamplesDescriptionCSML feature type ‘Coverage’ feature types Pure geometry feature type ProfileSeriesFeature ProfileFeature GridFeature

GO-ESSP LLNL, June 2006 CSML: Present GML Application schema –all sections optional CSML Dataset csml:UnitDefinitions csml:ReferenceSystemDefinitions csml:PhenomenonDefinitions Storage descriptors gml:FeatureCollection csml:NetCDFExtract, csml:AggregatedArray, csml:GRIBExtract, csml:NASAAmesExtract, etc (cf. UDUNITS.xml) GML dictionaries (cf. CFStandardNames.xml)

GO-ESSP LLNL, June 2006 CSML: Present Tooling –Scanner uses ‘feature file map’ to characterise relationship between feature type and directory/files being scanned individual Python classes for GML classes, feature types, dataset, etc  all have toXML() method for generating GML/CSML instances populated by Scanner $ python csmlscan.py -d /badc/ecmwf-e40/data/gp/gp/ap -o era40gridseries.xml -c GridSeriesFeature -t 'time' CDAT/cdunif + csmlscan CSMLFile.xml

GO-ESSP LLNL, June 2006 CSML: Present Tooling –Parser all GML/CSML Python classes have fromXML() method called recursively by Parser on ElementTree representation tree = ElementTree(file=CSMLFile) dataset = Dataset() dataset.fromXML(tree.getroot())... feature = dataset.featureCollection.members[0] variablePhenomenon = feature.parameter units = feature.rangeSet.aggregatedArray.uom... csml = dataset.toXML() ElementTree + parser CSMLFile.xml

GO-ESSP LLNL, June 2006 CSML: Present Tooling –API experimental at present defined against CSML data model e.g.: Dataset.getFeatureList(), GridSeriesFeature.getDomain(), GridSeriesFeature.subsetToGridSeries() –Different feature types may support different operations (part of UML model!) –‘Subsetting’ regarded as ‘extracting a feature’

GO-ESSP LLNL, June 2006 CSML: Present Higher-level services –BADC DataExtractor –OGC Web Coverage Service extension of GALEON server SERVICE=WCS&VERSION=1.0.0&REQUEST=GetCoverage& COVERAGE=solar_3&CRS=EPSG:4326& BBOX=0,-90,356,90&TIME= / & FORMAT=CF-netCDF&RESX=2&RESY=2

GO-ESSP LLNL, June 2006 CSML: Present CSML (and friends): software stack

GO-ESSP LLNL, June 2006 CSML Present Other User Example: Norwegian Met Office

GO-ESSP LLNL, June 2006 CSML: Future Immediate issues (v2) –Feature types: Swath ProfileSeries  {Radar, Section, ProfileSeries}? Lose ‘composite domain pattern’? –GML 3.2 (  ISO 19136) –remove ‘storage descriptors’ from core CSML schema –deployment, testing

GO-ESSP LLNL, June 2006 CSML: Future Longer-term issues –GML irregular grids – CSML extends gml:Grid (evolving standards is slooow!) –ISO ‘Data Product Specifications’ (e.g. CRS should be defined in DPS, not AS) –GML & file-based data (will be discussed next week at OGC TC in Edinburgh)

GO-ESSP LLNL, June 2006 CSML: Future Other activities: Universe of discourse Feature types ISO Application schema ISO ISO The taxon name taxon 'Anaitides mucosa',6.8,missing ISO EU project ‘MOTIIVE’ EU project ‘RISE’ GALEON MetOffice operational meteorology netCDF data standards RISE and MOTIIVE are both pilots for INSPIRE

GO-ESSP LLNL, June 2006 CSML: Future The CSML ‘niche’ –set of base feature types for specialising or using as-is

GO-ESSP LLNL, June 2006 CSML: Future CSML ‘out-of-the-box’ –conceptual model –GML application schema –‘geometric affordances’ (i.e. operations for generic Grid, Profile, Point coverages + series)

GO-ESSP LLNL, June 2006 Links CSML: CSML validator: bin/csml/parseTest.pyhttp://glue.badc.rl.ac.uk/cgi- bin/csml/parseTest.py MetOffice ‘Exeter Communiqué: NDG CSML subversion: TI02-CSML/ TI02-CSML/