1. 12.09.2002M. Lautenschlager (M&D/MPIM)1 WDC on Climate as Part of the CERA 1 Database System Michael Lautenschlager Modelle und Daten Max-Planck-Institut für Meteorologie WDC Review Hamburg, 13.09.2002 1) Climate and Environmental data Retrieval and Archiving

2. 12.09.2002M. Lautenschlager (M&D/MPIM)2 Content CERA Concept User Interface Data Content WDC Integration

3. 12.09.2002M. Lautenschlager (M&D/MPIM)3 CERA Accessing Countries

4. 12.09.2002M. Lautenschlager (M&D/MPIM)4 File Access Problems Missing Data Catalogue Directory structure of the Unix file system is not sufficient to organise millions of files Data are not stored application-oriented Raw data contain time series of 4D data blocks Access pattern is time series of 2D fields Lack of experience with climate model data Problems in extracting relevant information Year20012002200320042005 Moderate Increase210 TB650 TB1620 TB2670 TB3720 TB Linear Increase210 TB1270 TB4260 TB7580 TB10910 TB

5. 12.09.2002M. Lautenschlager (M&D/MPIM)5 CERA Concept: Semantic Data Management (I) Data catalogue and pointer to Unix files Enable search and identification of data Allow for data access as they are (II) Application-oriented data storage –Time series of individual variables are stored as BLOB entries in DB Tables Allow for fast and selective data access –Storage in standard file-format (GRIB) Allow for application of standard data processing routines (PINGOs)

6. 12.09.2002M. Lautenschlager (M&D/MPIM)6 CERA Database: 7.1 TB (12.2001) * Data Catalogue * Processed Climate Data * Pointer to Raw Data files Mass Storage Archive: 210 TB neglecting Security Copies (12.2001) CERA Database System Web-Based User Interface Catalogue Inspection Climate Data Retrieval DKRZ Mass Storage Archive InternetAccess Current database size is 10.1377 Terabyte Number of experiments: 279 Number of datasets: 17284 Number of blob within CERA at 06-SEP-02: 594086180 Typical BLOB sizes: 17 kB and 100 kB Number of data retrievals: 1500 – 5500 / month

7. 12.09.2002M. Lautenschlager (M&D/MPIM)7 CERA-2 Data Model Complete with respect to IEEE’s Reference Model for Metadata (Bretherton, 1994) –Browse, Search and Retrieval –Ingest, Quality Assurance, Reprocessing –Application to Application Transfer –Storage and Archive Supports interoperability due to inclusion of international standards –Directory Interchange Format (NASA, 1998) –FGDC Metadata Content Standard (FGDC, 1996) –ISO Metadata Standard for Geographic Information (ISO 19115) Reference –“The CERA-2 Data Model” (DKRZ-Report No. 15, 1998) –URL: http://www.pik-potsdam.de/dept/dc/e/sdm/cera/http://www.pik-potsdam.de/dept/dc/e/sdm/cera/

8. 12.09.2002M. Lautenschlager (M&D/MPIM)8 CERA-2 Data Model Blocks Metadata Entry This is the central CERA Block, providing information on the entry's title type and relation to other entries the project the data belong to a summary of the entry a list of general keywords related to data creation and review dates of the metadata Additionally: Modules and Local Extensions Module DATA_ORGANIZATION (grid structure) Module DATA_ACCESS (physical storage) Local extension for specific information on (e.g.) data usage data access and data administration Coverage Information on the volume of space-time covered by the data Reference Any publication related to the data togehter with the publication form Status Status information like data quality, processing steps, etc. Distribution Distribution information including access restrictions, data format and fees if necessary Contact Data related to contact persons and institutes like distributor, investigator, and owner of copyright Parameter Block describes data topic, variable and unit Spatial Reference Information on the coordinate system used

9. 12.09.2002M. Lautenschlager (M&D/MPIM)9 Data Model Functions The CERA2 data model … –allows for data search according to discipline, keyword, variable, project, author, geographical region and time interval and data retrieval. –allows for specification of data processing (aggregation and selection) without attaching the primary data. –is flexible with respect to local adaptations and storage of different types of geo-referenced data. –is open for cooperation and interchange with other database systems.

10. 12.09.2002M. Lautenschlager (M&D/MPIM)10 Data Structure in CERA Level 1 Level 2 Experiment Description Pointer to Unix-Files Dataset 1 Description Dataset n Description BLOB Data Table BLOB Data Table

11. 12.09.2002M. Lautenschlager (M&D/MPIM)11 User Interface Structure

12. 12.09.2002M. Lautenschlager (M&D/MPIM)12 User Interface Signed Java Applet: Catalogue Inspection Climate Data Retrieval http://mad.dkrz.de/java/CeraStart.html IPCC DDC

13. 12.09.2002M. Lautenschlager (M&D/MPIM)13 CERA Data Content Climate Model Data (Continuous stream of new data) –Local climate model production experiments for present and future but also for past climates IPCC DDC (Data Distribution Centre) –Archive and dissemination of selected data from international climate scenario calculations (IS92a and SRES) –Will be continued for the Forth Assessment Report Project Support (encourage Good Scientific Practice) –Archive and dissemination of project data HOAPS (Hamburg Ocean Atmosphere Parameters and Fluxes from Satellite Data) CARIBIC (Civil Aircraft for Regular Investigation of the Atmosphere Based on an Instrumentation Container), MPI Mainz

14. 12.09.2002M. Lautenschlager (M&D/MPIM)14

15. 12.09.2002M. Lautenschlager (M&D/MPIM)15 CERA Data Content Observational Data –Model related observations ERA15 (ECMWF) NCEP/NCAR 40 Year Reanalysis ERA40  in preparation –Instrumental data WOCE (World Ocean Circulation Experiment): field measurements and products are transferred from BSH –Earth observations Access to SST's from NOAA AVHRR in cooperation with DFD/DLR (distributed archive)  preparation for WDC cooperation

16. 12.09.2002M. Lautenschlager (M&D/MPIM)16 CERA Data: Jan. Temp.

17. 12.09.2002M. Lautenschlager (M&D/MPIM)17 CERA Data: Jan. Wind (2 x 250 MB)

18. 12.09.2002M. Lautenschlager (M&D/MPIM)18 Integration of WDC on Climate WDC will be part of the operational CERA DB system –Requires only little additional work –Consumes only little hardware resources –All freely available data will be part of the WDC on Climate CERA DB / WDC on Climate fit requirements from interdisciplinary data access, non experienced users and small network bandwidth –Assistance and training, visitor programme –Small data units, media copies on request Data import in cooperation with producers Data dissemination and long-term storage is maintained by the CERA DB system

19. 12.09.2002M. Lautenschlager (M&D/MPIM)19 Future Distributed data archive –Cooperation with WDC for Earth Observation (Oberpfaffenhofen) and WDC for Marine Environmental Sciences (Bremen) WDC for Paleoclimatolology (Boulder) –Sharing data holdings and responsibilities for climate data International cooperation and opening data archives –Federation of WDC's (related to Climate Research) –Web-based WDC data portal