October 16-18, 20011 Research Data Set Archives Steven Worley Scientific Computing Division Data Support Section.

Slides:

Advertisements

Similar presentations

BADC Workshop 1: Data & Services from the BADC Royal Met. Soc. Conference – 12 September 2005 Kevin Marsh et al.

Advertisements

Data management in SCD Steven Worley General Categories –The Mass Storage System –NCAR user file services (home directories) –Computer attached storage.

BADC Workshop 2: BADC Services to Data Suppliers Royal Met. Soc. Conference – 14 September 2005 Ag Stephens et al.

SCD Research Data For UCAR Data Management Working Group January 10, 2001 Steven Worley Scientific Computing Division Data Support Section.

ICOADS Archive Practices at NCAR JCOMM ETMC-III 9-12 February 2010 Steven Worley.

ERA-Interim and ASR Data Management at NCAR

The Research Data Archive at NCAR Doug Schuster and Steve Worley NCAR.

Integrating SeaDOG into RODIN T. Le Bas, F. Venuti, H. Snaith, and B. Soall.

The International Surface Pressure Databank (ISPD) and Twentieth Century Reanalysis at NCAR Thomas Cram - NCAR, Boulder, CO Gilbert Compo & Chesley McColl.

Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System 1 Zaihua Ji Doug Schuster Steven Worley Computational.

TPAC Digital Library Talk Overview Presenter:Glenn Hyland Tasmanian Partnership for Advanced Computing & Australian Antarctic Division Outline: TPAC Overview.

Introduction Downloading and sifting through large volumes of data stored in differing formats can be a time-consuming and sometimes frustrating process.

Scientific Investigations; Support from Research Data Archives for Computing in Atmospheric Sciences October, 2001 Steven Worley National Center.

U.S. Surface Archives Sent to China ( ) 7 th PRC-U.S. Joint Coordination Panel for Data and Information Cooperation 29 Nov. – 1 Dec, 2000 Steven.

EGU 2011 TIGGE, TIGGE LAM and the GIFS T. Paccagnella (1), D. Richardson (2), D. Schuster(3), R. Swinbank (4), Z. Toth (3), S.

Research Data at NCAR 1 August, 2002 Steven Worley Scientific Computing Division Data Support Section.

Presented by The Earth System Grid: Turning Climate Datasets into Community Resources David E. Bernholdt, ORNL on behalf of the Earth System Grid team.

Growing and Future Datasets in the SCD Research Data Archives for NSF SCD Review Panel 16 October 2001 Steven Worley Scientific Computing Division Data.

Data for Climate and Energy Studies Steven Worley Computational and Information Systems Laboratory NCAR.

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

Virtual Ice Charting System Archive Browser Interface Distribution IngestProduction Ice Analyst Application Database Click on the boxes for more information.

Scientific Investigations; Support from Research Data Archives for Joint Office for Science Support 26 February, 2002 Steven Worley SCD/DSS.

Colour of Ocean Data, Brussels, November 2002 Colour of Ocean Data: Discussion Panel Lesley Rickards British Oceanographic Data Centre.

1 Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory.

Archive and Access Practices that Support Data Reuse and Transparency Steven Worley Doug Schuster Bob Dattore National Center for Atmospheric Research.

Data Access to Marine Surface Observations and Products from COADS 29 January, 2002 Steven Worley National Center for Atmospheric Research.

CISL/DSS & MMM Data Discussion 19 March Who CISL/DSS - maintain NCEP operational analyses and observation datasets – Gregg Walters, Doug Schuster,

Improved Access to RDA from the MSS OSD Executive Meeting April 28, 2009.

ICOADS: Update Status and Data Distribution Steven J. Worley Scott D. Woodruff Sandra J. Lubker Ziahua Ji J. Eric Freeman NCAR, NOAA/ESRL, NOAA/NCDC CLIMAR-III,

Analyzed Data Products Available from NCAR that Support Marine Climate Research JCOMM ETMC-III 9-12 February 2010 Steven Worley Doug Schuster.

START-08/pre-HIPPO FIELD CATALOG AND DATA MANAGEMENT Steve Williams NCAR Earth Observing Laboratory (EOL) Boulder, Colorado START-08/pre-HIPPO Planning.

Data Discovery and Access to The International Surface Pressure Databank (ISPD) 1 Thomas Cram Gilbert P. Compo* Doug Schuster Chesley McColl* Steven Worley.

IODE Ocean Data Portal - ODP  The objective of the IODE Ocean Data Portal (ODP) is to facilitate and promote the exchange and dissemination of marine.

Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive Doug Schuster and Steve Worley NCAR.

RAINEX Data Management UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks UCAR Joint Office for Science Support José Meitín Jim Moore.

RDA Data Support Section. Topics 1.What is it? 2.Who cares? 3.Why does the RDA need CISL? 4.What is on the horizon?

NOAA/NESDIS/National Oceanographic Data Center Following the Flow of Two Underway Data Streams Within the U. S. National Oceanographic Data Center Steven.

Near Real-Time Verification At The Forecast Systems Laboratory: An Operational Perspective Michael P. Kay (CIRES/FSL/NOAA) Jennifer L. Mahoney (FSL/NOAA)

Electronic Tools for TIA Standards Development David Thompson Member of the Technical Staff GCS/EWG Meeting Sapporo, Hokkaido, JAPAN August 2000 TC

TIGGE Data Archive at NCAR 8th GIFS-TIGGE Working Group World Meteorological Organization Geneva February, 2010 Doug Schuster Steven Worley Dave.

29 March 2004 Steven Worley, NSF/NCAR/SCD 1 Research Data Stewardship and Access Steven Worley, CISL/SCD Cyberinfrastructure meeting with Priscilla Nelson.

TIGGE Archive Status at NCAR THORPEX Workshop and 6th GIFS-TIGGE Working Group Meetings WMO Headquarters Geneva September 2008 Steven Worley Doug.

SCD Research Data Archives; Availability Through the CDP About 500 distinct datasets, 12 TB Diverse in type, size, and format Serving 900 different investigators.

The Research Data Archive at NCAR: A System Designed to Handle Diverse Datasets Bob Dattore and Steven Worley National Center for Atmospheric Research.

TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox.

Research Data Archive (RDA) Access and Services from Yellowstone Grace Peng and Doug Schuster 1.

Data Discovery and Access to The International Surface Pressure Databank (ISPD) 1 Thomas Cram Gilbert P. Compo* Doug Schuster Chesley McColl* Steven Worley.

Distributed Data Servers and Web Interface in the Climate Data Portal Willa H. Zhu Joint Institute for the Study of Ocean and Atmosphere University of.

RDA Data Support Section. Topics 1.What is it? 2.Who cares? 3.Why does the RDA need CISL? 4.What is on the horizon?

SCD Research Data for Ocean Observatories Steering Committee June 18, 2001 Steven Worley Scientific Computing Division Data Support Section.

Scripps Visit Al Kellie Director SCD February 1, 2001.

5-7 May 2003 SCD Exec_Retr 1 Research Data, May Archive Content New Archive Developments Archive Access and Provision.

The TIGGE Model Validation Portal: An Improvement in Data Interoperability 1 Thomas Cram Doug Schuster Hannah Wilcox Michael Burek Eric Nienhouse Steven.

1. Gridded Data Sub-setting Services through the RDA at NCAR Doug Schuster, Steve Worley, Bob Dattore, Dave Stepaniak.

Introduction What purpose does a data archive center serve if users can’t find or access the holdings they might need to facilitate their research discoveries?

TIGGE Archives and Access

Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System Zaihua Ji Doug Schuster Steven Worley Computational.

Development and Futures of Research Data Archives

Research Data Archives at NCAR

Steven Worley, NSF/NCAR/SCD

Steven Worley, Douglas Schuster,

CISL’s Research Data Archive (RDA) : Description and Methods

Comeaux and Worley, NSF/NCAR/SCD

Long-Lived Data Collections

Data Management Components for a Research Data Archive

Objectives Explain the role of computers in client-server and peer-to-peer networks Explain the advantages and disadvantages of client- server and peer-to-peer.

Robert Dattore and Steven Worley

Data Curation in Climate and Weather

Comeaux and Worley, NSF/NCAR/SCD

Presentation transcript:

October 16-18, Research Data Set Archives Steven Worley Scientific Computing Division Data Support Section

October 16-18, Research Data sets 35 year old collection & stewardship Ocean, Atmosphere, operational analysis and re-analysis - all secured on MSS 530 distinct data sets, 17 TB, 211K files Information Interface - WEB/professional consultation (match data with research requirements) User guides to different categories of data Staff specialist for each dataset Web search tools

October 16-18, Data Management Stewardship of data is done by a meteorological & oceanographic staff Each dataset is maintained in a standardized system Metadata, identification of files, software, etc are secure under a change control system Each dataset has a web page that is automatically generated, and is always current.

October 16-18, Data Access Remote users Data discovery is through a web interface Data packages provided upon request Staged to disk for network transfer, e.g. FTP A variety of magnetic tapes and CD-ROMS NCAR facility users Directly accessible to all computing platforms File location, metadata, documentation, software, and some data are available online through a web interface All users Receive consulting from staff specialist as needed

October 16-18, TBytes of Data Provided 5.6 TB in 1995 to about 20.5 TB in 2001

October 16-18, 20016Holdings Global Observations dating from 1942 ( and a few earlier) ECMWF Global Operational Analysis from NCEP Operational Analysis from Reanalysis NCEP/NCAR, ECMWF, NCEP Reanalysis II.

October 16-18, Outstanding Features Three different coordinate surfaces Very long analysis (1948 – 2001) CD-ROMS are very popular

October 16-18, Countries Receiving Reanalysis CD-ROMs Highlights Over 8900 CDROMs /2001 Percent received; U.S. 46%, Japan 11%, (Canada, 4%, etc

October 16-18, UCAR Research Collaboration Support real-time data services Unidata’s (UCAR) role –Runs full capacity IDD/LDM application –Serves ± 20 universities downstream SCD’s role –24 hr x 7 day operation monitoring –Routine system back up –Data archive backup and maintenance Covers one year – done daily 2.3 GB per day to the MSS Observations, NCEP model data, wind profilers