Research Data at NCAR 1 August, 2002 Steven Worley Scientific Computing Division Data Support Section

Outline Archive building; Basic ingredients Proactive content development Support for NCAR/UCAR/Universities Access to the data

Archive Building Collect, maintain, and archive data –Data types Observations Analyses Reanalyses Model output data Staff – data stewards – SE’s educated in meteorology and oceanography –Activities Quality check the information Write discovery metadata Provide human based consulting

Proactive Content Development NCAR has a privileged position –NOT restricted by “Agency” mission demands e.g. “You must archive Level 1 MODIS data!” –Remain flexible to meet research demands –Choose efficient and cost effective methods

Proactive Content Development Example of being proactive –NCAR’s in situ observations collections are the most complete in the world Long term dedication – 35+ years Involved many special data exchanges (national and international) –Add key data not available in the standard archives Achievement widely recognized –US Academy of Science – NCEP and ECMWF

Challenges detect and repair, or identify erroneous data rectify historical format differences to create a user friendly “best” composite collection To meet the challenges have the knowledgeable staff need staff time to develop the graphics and analysis software

Proactive Content Development Many other examples – Atmospheric upper air data – Marine surface data –Etc.

Proactive Content Development Importance of these collections –Basis for research data products, e.g. NCEP/NCAR Reanalysis NCEP DOE Reanalysis II ERA40 (processing now at ECMWF) NCEP Regional Reanalysis (new)

Countries that have received NCEP/NCAR Reanalysis from SCD CD-ROMS network downloads tape media copies

Proactive Content Development Challenges of Reanalyses Products –Product Size Current sets, O( 2 TB) New sets, O( 20 TB) –Many users, O( 100’s/year) Not a problem in NCAR computing environment External access is the greatest challenge (more later)

Support for Users The Users are; – NCAR/UCAR scientists – UCAR universities scientists and students – International scientists and students – Gov. agencies and private corporations Access through; – SCD MSS – Customized requests – Online data servers (ftp, and browser) – Community Data Portal (CDP)

–CDP Features – Real time data access – 4-D dataset subsets – multiple output formats – data analysis Challenges –Develop access to large (TB) datasets »Sequential / parallel data processing »Linkages to SANS –Integration with other extant data systems To meet the challenge –Continue to improve our servers, storage networks, and network connections –Staff with expertise and dedicated time

Support for the Users Search and discover data How?  Web based Information Server Features –5K+ html pages (metadata) –All datasets are described –Access options –Higher level information Catalogs Project specific descriptions –All information is current Based on text files and change control system Automatic re-build of affected.html pages

Support for the Users Search and discover data Next Step for Data Discovery –Need a metadata descriptions in a ‘standard format’ –Enable organization wide discovery (eventually nation wide or global) –UCAR has DMWG on metadata Likely scenario – based meta standard –Well poised to based on.html experience

Take Home Key Points Long term involvement in archive building National and International collaborations Focus on scientific data content, documentation, and access stay in tune with scientific needs remain flexible so we can accommodate the needs promote data exchanges that enhance the archive distribute data at minimal or no cost provide data to large projects, e.g. NN Reanalysis have good data and metadata systems now more technology and software to enhance the access Proactive development stewardship (collecting, maintaining, improving)