IPY Data management A brief summary Øystein Godøy

IPY Organisation ICSU/WMO Joint Committee Data Policy & Management Subcommittee Observations Subcommittee Education, Outreach & Communication Subcommittee Programme Office Space Task Group

 Twelfth WMO Congress, Resolution 40 (1995)  Thirteenth WMO Congress, Resolution 25 (1999)  ICSU General Assembly Resolution (1996)  ICSU Assessment on Scientific Data and Information (2004)  Article III-1c from the Antarctic Treaty (1959)  the Intergovernmental Oceanographic Commission Data Exchange Policy (2003 ) International consensus on free and open access

 The primary purpose should be to serve the objectives of IPY  International exchange (>60 countries)  Interdisciplinary science (>50,000 participants representing physical, life, and social scientists)  Building a legacy  Increase the value of observations Overarching purpose

Data used by IPY Data generated by IPY Special Cases: Human subjects Intellectual property of LTK Where data release may cause harm “…the IPY Joint Committee requires that IPY data, including operational data delivered in real time, are made available fully, freely and on the shortest feasible timescale.” IPY Data Policy—Defining IPY Data

Implications  Publish and share your data  Increased data value  Promote interdisciplinary use  Have someone measured something you need?  Preserve and archive your data  Generate a legacy  The basis for the next IPY

What has been done?  The Data committee has defined a data policy  Workshop on data management   IPY metadata profile   Data coordinators in US, Canada, UK and Norway  Data released for IPY use by ECMWF, ESA and others  Some data management projects have been funded  But not enough...  IPY Layer in Google Earth available at

IPY data policy - metadata  “All IPY data must be accompanied by a full set of metadata that completely document and describe the data.”  “Regardless of any data access restrictions or delays in delivery of data itself, all IPY projects must promptly provide basic descriptive metadata in an internationally recognized, standard format to an appropriate catalog or registry.”  ISO19115/ISO19139

 “All data registries and repositories collecting data and metadata from IPY projects are required to collect and share sufficient information to adhere to the IPY Metadata Profile”  Basic who, what, where, when in either FGDC, DIF, THREDDS (ISO coming, but could use some help), plus some information on metadata provenance.  Controlled vocabulary from GCMD for some fields.  The “bare minimum of information necessary to allow simple discovery across disciplines and to ensure we can track the heritage of the metadata in a broadly distributed data management environment.”  Details are available at IPY Metadata Profile (and crosswalk)

Entry ID (controlled) Data set title Data set progress Data set summary Data set citation information including Online Resource Parameters Locations ISO topic categories Temporal coverage Spatial coverage Data center contact information Access restrictions Use constraints Data Set Language Metadata contact information Metadata authority Metadata version Last revision IPY flag IPY Project ID IPY Metadata profile

Formats: Negotiate common formats and conventions ASCII is useful but not really a precise format avoid proprietary formats some suggestions: netCDF is popular for some, OGC (WMS/WCS/WFS) compatibility is nice Archives and users may have different needs Access: Integrate with many systems to allow increased user discovery (register with the IPYDIS) Use open source software when possible, use open standards everywhere. Preservation Open Archive Information System Reference Model Attribute and provide info for attribution readily through all gateways The Data

What is out there?

ELOKA The Exchange for Local Observations and Knowledge of the Arctic works to provide data management and user support to facilitate the collection, preservation, exchange, and use of local observations and knowledge of the Arctic. PI: Shari Gearheard

18 May 2006

Assist on compliance with standards, identification of archives, development of the union catalogue, and other data management requirements for IPY. Visibly track the data flow for IPY. In collaboration with the IPO, develop a data registry that will continue throughout the IPY. Survey the planned projects and the data they intend to collect and identify existing archives, portals, experts, and significant gaps in the IPY data infrastructure.  Mark Parsons—Overall, US  Øystein Godøy— Operational Data, Norway  Canadian Coordinator— Overall, Canada  National coordinators in Netherlands, China, UK Data coordinators

International Operational Data Coordinator (1)  Fist version based upon DAMOCLES system  PostGRES metadatabase  OpeNDAP ⇒ THREDDS (OpeNDAP/WCS support) or early 2008  Adding WMS support for selected fields/datasets  Expected late 2007  Second version based upon SIMDAT and first version  Expected 2008

International Operational Data Coordinator (2)  Final version should be WIS DCPC  Expected 2009  Have focused on ECMWF data so far  Direct access at ECMWF disks if using WIS DCPC software  Interface for metadata submission  As well as notifications on interesting datasets  Import/export of metadata  Method and format open issue

International Operational Data Coordinator (3)  How can this service help promoting remote sensing data during IPY?  Access to metadata?  Provide metadata information on available products (??)  Users get forwarded to data catalogues at the space centres(??)  OGC access to selected datasets(??)

National recommendations (1) Norway  National IPY-projects should relate to the relevant standards for  Data collection  Processing  Archival  Description  Transfer  Metadata (ISO19115, ISO19139)  OpenGIS

National recommendations (2) Norway  Data and metadata shall be freely available with minimal time delay  Metadata shall be transmitted to relevant national and international data centres  Data shall be transmitted to relevant national and international data centres for long term archival  Delivery of data and metadata is the responsibility of the project coordinator

National recommendations (3) Norway  A minimum of standardised file formats should be used  Existing national and international infrastructure shall be utilised for long term archival  Data used in publications shall be correctly cited  The use of licences will be evaluated if this can improve data access

National requirements Norway  IPY projects with NFR-contracts  shall submit a data management plan within 3 months of contract signing  Describing who is responsible for overall data management  Who is responsible for datasets  Where datasets will be stored and finally archived  shall submit yearly updates of the data management plan  What have been collected  Where is it stored

National IPY-data service Norway  DokIPY  Funded by NFR  Shall connect existing metadata- and databases in Norway  Extract and exchange metadata  Utilise existing infrastructure and national mandates at  Institute of Marine Research  Norwegian Polar Institute  Norwegian Meteorological Institute  Robust/simple exchange of metadata

METNO version  Relies on the same software as DAMOCLES and International Operational Data Coordination  New feature – metadata exchange  Examining international requirements/specifications, not much found yet  Funding situation require simple and robust solutions

Special issue  Remote Sensing of Environment  Special issue on Remote Sensing of the Cryosphere  Remote sensing of Environment, Vol. 111, Issues 2- 3, 30 November  Available online

How can we interface?