1. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Finding and utilising atmospheric/oceanic data in a distributed world: the UK NERC DataGrid. Bryan Lawrence (Kerstin Kleese, Roy Lowry, Kevin O’Neill, Andrew Woolf & others) NCAS/British Atmospheric Data Centre Rutherford Appleton Laboratory, CCLRC

2. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk NDG Partners As funded a partnership between –British Atmospheric Data Centre (BADC, PI: Bryan Lawrence) –British Oceanographic Data Centre (BODC, Co-I: Roy Lowry) –CLRC E-science Centre (Co-I: Kerstin Kleese) –PCMDI at LNL in the US (Dean Williams, Bob Drach, Mike Fiorino) Project has caught the imagination, extra funding now supports: –A number of groups at the NERC Centre for Ecology and Hydrology (CEH: Ecology DataGrid) –NERC Earth Observation Data Centre & Plymouth Marine Lab Remote Sensing Not directly funded major collaborators will include: –ClimatePrediction.net, GODIVA (NERC e-science projects) –NCAS/CGAM: The Centre for Global Atmospheric Modelling at the University of Reading (via Lois Stenman-Clark and Katherine Bouton) Project will support HIGEM

3. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Outline Motivation: The NDG Goals Working in a standards based world – ISO and OGC … NDG Metadata Summary

4. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk The British Oceanographic Data Centre (not for much longer, moving to a site on Liverpool University campus imminently)

5. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk British Atmospheric Data Centre The Role: Key words: Curation and Facilitation!

6. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Easily catalogued, but successful preservation? One could argue that the writers of these documents did a brilliant job of preserving the bits-and-bytes of their time … And yes they’ve both been translated … many times, it’s a shame the meanings are different … Phaistos Disk, 1700BC

7. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk NERC Metadata Gateway - SST No clean handover from discovery to browse and use! Geospatial coordinates forgotten. Time reference forgotten. Need to get entire field(s), and find correct time! And if I want to compare data from different locations? - multiple logins - multiple formats - discovery?

8. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk A priori would any user know to look in the COAPEC data set? Earth system-science means we have to remove these boundaries! detailed file level metadata isn’t visible, and so data mining applications impossible. NB: Dynamic catalogues! How good is our metadata?

9. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Finding Data The Goal: Very simple interface, hide the complex software!

10. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk A newer “dataset” The extreme relevance of this example from Amazon was pointed out by Jon Callahan (LAS project, PMEL)!

11. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk PCMDI – Best practice! (if you know where to look) Final references are papers! Is the information coupled to the datasets? What if I take a dataset home, and another, and another … and then forget which is which? Can I ask the question: what datasets used the Semtner sea ice parameterisation?

12. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Huge variety of Data Sets

13. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Querying datasets Complex Metadata, held in Ingres database: export DIF and Z39.50 No possibility of automatic data usage …

14. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Different types of data returned: Wallingford Supporting very diverse user community: NetCDF is not enough …

15. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Modelling advances: Baseline Numbers T42 CCSM (current, 280km) –7.5GB/yr, 100 years ->.75TB T85 CCSM (140km) –29GB/yr, 100 years -> 2.9TB T170 CCSM (70km) –110GB/yr, 100 years -> 11TB NCAR Don Middleton

16. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Capacity-related Improvements Increased turnaround, model development, ensemble of runs Increase by a factor of 10, linear data Current T42 CCSM –7.5GB/yr, 100 years ->.75TB * 10 = 7.5TB

17. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Capability-related Improvements Spatial Resolution: T42 -> T85 -> T170 Increase by factor of ~ 10-20, linear data Temporal Resolution: Study diurnal cycle, 3 hour data Increase by factor of ~ 4, linear data CCM3 at T170 (70km)

18. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Capability-related Improvements Quality: Improved boundary layer, clouds, convection, ocean physics, land model, river runoff, sea ice Increase by another factor of 2-3, data flat Scope: Atmospheric chemistry (sulfates, ozone…), biogeochemistry (carbon cycle, ecosystem dynamics), middle Atmosphere Model… Increase by another factor of 10+, linear data

19. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Model Improvement Wishlist Grand Total: Increase compute by a Factor O(1000- 10000) NCAR Don Middleton

20. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Climate in 20010 – A graphic Illustration Figures from Gary Strand, NCAR, ESG website

21. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Summary thus far Contentions: The average atmospheric scientific project involves about 1/3 of the time data handling! (Getting, reformatting etc). The problem for earth system model projects is about to get worse – for everyone, from the initiator, to the archiver, to the analyst, to the contributor, to the improver. (Remember the documentation problem is growing exponentially too: new sub-components etc)

22. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk The NERC DataGrid

23. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk

24. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk The Data Use Chain

25. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Requirements: Information (1) Amazon Discovery gives good examples: Browse Similar datasets Details Content examples Learn from the library and book handling community! Our domain Issues require: Dealing with Volume Formats Providing Tools All require documentation (aka metadata); We need to improve our information handling

26. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk What is metadata? The answer depends on who you are! Firstly: information to help one use one’s own data: e.g. calibration data (A) Metadata can help one find other people’s data … and then help one obtain and use it. (C) Metadata can be used to enable the preservation of data for posterity (all of ABCD) It is information passed with the data to enable someone else to use it. It describes the data. (B) Metadata can be used to enable automatic software to manipulate data. (D)

27. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk NDG Metadata Taxonomy

28. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk ISO 19101: Geographic information – Reference model ISO 19103: Geographic information – Conceptual schema language ISO 19107: Geographic information – Spatial schema ISO 19108: Geographic information – Temporal schema ISO 19109: Geographic information – Rules for application schema ISO 19111: Geographic information – Spatial referencing by coordinates ISO 19115: Geographic information – Metadata ISO 19118: Geographic information – Encoding ISO 19121: Geographic information – Imagery and gridded data ISO TC211

29. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Dataset title Dataset reference date Dataset responsible party Metadata point of contact Dataset language Dataset character set Dataset topic categoryAbstract describing dataset Spatial resolution of dataset Spatial representation type Geographic location of dataset Vertical/temporal extent for dataset Reference system Lineage Distribution format On-line resource Metadata character set Metadata date stamp Metadata standard name Metadata standard version Metadata file identifier Metadata language ISO19115

30. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Metadata extensions and profiles ISO Direct relationship between ISO19115 and our (B) Intermediate schema.

31. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Profiling of ISO 191xx “The comprehensiveness and large number of options available in various base standards make it difficult to combine them for practical applications. … A profile integrates a set of base standards and/or modules (predefined subsets) of base standards to meet a specific implementation requirement.” Registration of profiles “A profile that is registered through an ISO registration procedure becomes an International Standardized Profile (ISP). National standards that are expressed as profiles of ISO base standards may be registered at a national level.” ISO19101

32. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk NDG A and B metadata in practice Clear separation of function between use and discovery. Standards Compliant Avoid tie-in to details of particular fields or data formats or even components Metadata model (B) “Intermediate” schema, supports multiple discovery formats NDG Data Model (A). provides an abstract semantic model for the structure of data within NDG, enables the specification of concrete instances for use by NDG Data Services

33. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk (B) Metadata Model

34. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk (B) Metadata Model: an NDG Intermediate Schema, Conceptual Overview

35. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Dataset Variables Multidimensional array... of other arrays... or from aggregated storage Rich spatiotemporal referencing (standards- compliant: ISO19108, ISO19111) NDG Data Model

36. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk This is the root element of the document. It is a limited (and incomplete!) implementation of the root element described in ISO 19118 (clause 5.4). The latter describes the format of a compliant XML exchange file, intended for encoding a single dataset. The application of ISO 19118 for the current NDG Data Model which may contain multiple datasets needs to be resolved. A dataset contains one or more elements that encode objects, grouped in a choice group that shall be used to restrict the legal objects in a dataset (ISO 19118, clause A.5.4.2). UML conceptual model: ISO 19103 (conceptual schema language) ISO 19109 (rules for application schema) XML schema ISO 19118 (encoding) NDG Data Model Schema

37. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk SAX demarshalling extraction serialisation writeData( selectedComponents) Instantiating the NDG Data Model

38. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Further Application in NERC DataGrid eg Data model “Coordinates” ISO 19111 ISO 19108

39. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk NDG Semantic Data Model

40. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk NDG Prototype Layout not important (yet!) It’s what’s under the hood that counts … ( … the data is NOT in NetCDF. The original data is available … … the search covered data that could have been harvested … … the architecture works!)

41. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk

42. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk NDG Discovery Service Element Traditional and Grid Service (GT3) Interfaces

43. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk NDG Metadata Status We have built a SIMPLE prototype based primarily on our data model and used our structures to find, locate, reformat and deliver data typical of BODC and BADC observational data. (This is a first) We are about to re-engineer. Key issues to address will be –Vocabularies, and –Ontologies –Developing a Model Attribute Language (with CGAM, PRISM, PCMDI and others). Populating our metadata; a boring and laborious job!

44. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Wider Internet Research Group Satellite SuperComputer Shared Resources DB Research Group Metadata Origins Consider a hierarchy of data users beginning with an individual scientist, who may herself be part of a research group, itself part of a community sharing resources, lying in the wider internet … To be well integrated the metadata should have a role at each level! (The data portal client and server interface may be different at each level). At each level “extra” metadata will be required, probably produced by dedicated staff at the research group, or data centre.

45. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Vocabularies BODC has a parameter dictionary with o(10K) entries CF standard name vocabulary, o(100) entries NASA Global Change Master Directory o(1000) entries … there are more. Need methods of mapping namespaces, communities will not sacrifice their existing taxonomies …

46. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk An ontology defines the terms used to describe and represent an area of knowledge by specifying the following kinds of concepts: Classes (general things) in the many domains of interest The relationships that can exist among things The properties (or attributes) those things may have Ontologies are usually expressed in a logic-based language, so that detailed, accurate, consistent, sound, and meaningful distinctions can be made among the classes, properties, and relations.. What is an Ontology?

47. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk RDF: Resource Description Framework W3C language which builds on the hierarchical attribute/entity structures of XML. Used to create a collection of assertions – specified as triples – Now we can build tools which use these concepts: –Aslan has a mane! –Aslan will also have animal properties. RDF Schema vocabulary builds on this to allow namespaces and more … (ranges etc).

48. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Real Ontologies … immature … SWEET: Semantic Web for Earth and Environmental Technology (NASA, JPL) Earth realms, Numerics Physical Properties Units Phenomena I believe they are Attempting a CF mapping…

49. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Requirements (2) We need to think about our networks and our tools for moving and keeping track of data! We can’t rely on the “leave it at the supercomputer site” –How do we do joint analysis? –How do we process the data at all? Malcolm Atkinson quoting Jim Gray pointed out that it takes: ~ o(minute) to grep or ftp a GB ~ o(2 days) to grep or ftp a TB ~ o(3 years) to grep or ftp a PB Requires –sophisticated “fire and forget” file transfer (that has to out perform “sneaker net”). –Disk and compute resources for processing.

50. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk ESG1 Results (Supercomputing, 2001) Allcock et al. 2001 Dallas to Chicago:

51. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Starting with the LAS Deployment for UK users within a few weeks (constraint is primarily access control)

52. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk LAS – Simple Box fill Output Work for us to do: Labelling is inadequate as yet.. ERA40

53. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Cache management in LAS/CDAT Calls cdms.open to open data file. CDAT BADC/CDAT intercepts command and checks cache BADC/CDAT YES Spectral file is converted on-the-fly and placed in cache. NO Cache unlocked. New cdms.open command sent to CDAT and cache file opened. Cache also checks if enough room, deletes oldest files if necessary and checks against disk space limit. Locks access to cache. Checks if regular gridded file is in cache list. localCache.py 18 TB virtual dataset LAS ERA-40 4 TB Spectral Archive ERA-40 < 1TB Grid Cache Internet User NetCDF file, plot or animations delivered to user. Data object delivered to LAS.

54. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk Summary Earth System Modelling extends the data handling challenge. We need better information management We need better tools for moving things around We need better tools for using remote data … and we need data manipulation hardware! The NDG is attempting (with help) to address: Information management Data movement Tools to manipulate large volumes of data. … and doing this all in as standards compliant a manner as possible.

55. ESM Meeting, Cambridge 2003 BADC: badc.nerc.ac.uk, NERC DataGrid: www.ndg.badc.rl.ac.uk You’ve gone TOO FAR!