Introduction Downloading and sifting through large volumes of data stored in differing formats can be a time-consuming and sometimes frustrating process.

Slides:



Advertisements
Similar presentations
1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.
Advertisements

New Resources in the Research Data Archive Doug Schuster.
The International Surface Pressure Databank (ISPD) and Twentieth Century Reanalysis at NCAR Thomas Cram - NCAR, Boulder, CO Gilbert Compo & Chesley McColl.
Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System 1 Zaihua Ji Doug Schuster Steven Worley Computational.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Unidata TDS Workshop THREDDS Data Server Overview October 2014.
October 16-18, Research Data Set Archives Steven Worley Scientific Computing Division Data Support Section.
Coordinated Energy and water-cycle Observations Peroject A Well Organized Data Archive System Data Integrating/Archiving Center at University of Tokyo.
TIGGE Archive Highlights. First Service Date ECMWF – October 2006 NCAR – October 2006 CMA – June 2007.
Research Data at NCAR 1 August, 2002 Steven Worley Scientific Computing Division Data Support Section.
GADS: A Web Service for accessing large environmental data sets Jon Blower, Keith Haines, Adit Santokhee Reading e-Science Centre University of Reading.
Bringing it All Together: NODC’s Geoportal Server as an Integration Tool for Interoperable Data Services Kenneth S. Casey, Ph.D. YuanJie Li NOAA National.
Unidata’s TDS Workshop TDS Overview – Part II October 2012.
Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.
THREDDS Data Server Ethan Davis GEOSS Climate Workshop 23 September 2011.
Weathertop Consulting, LLC Wednesday, January 14, 2009 IIPS 11A.2 1 A General Purpose System for Server-side Analysis of Earth Science Data Roland Schweitzer.
ATMOSPHERIC SCIENCE DATA CENTER ‘Best’ Practices for Aggregating Subset Results from Archived Datasets Walter E. Baskin 1, Jennifer Perez 2 (1) Science.
U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center.
N-Wave Stakeholder Users Conference Wednesday, May 11, Marine St, Rm 123 Boulder, CO Linda Miller and Mike Schmidt Unidata Program Center (UPC)-Boulder,
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.
Discovery and Web Services in Support of SST Datasets at the PO.DAAC Edward Armstrong, Jorge Vazquez Toshio M. Chin, Charles Thompson Jet Propulsion Laboratory/California.
Describe workflows used to maintain and provide the RDA to users – Both are 24x7 operations Transition to the NWSC with zero downtime NWSC is new environment.
Unidata TDS Workshop THREDDS Data Server Overview
ICOADS: Update Status and Data Distribution Steven J. Worley Scott D. Woodruff Sandra J. Lubker Ziahua Ji J. Eric Freeman NCAR, NOAA/ESRL, NOAA/NCDC CLIMAR-III,
1 NASA CEOP Status & Demo CEOS WGISS-24 Oberpfaffenhofen, Germany October 15, 2007 Yonsook Enloe.
ESIP Federation 2004 : L.B.Pham S. Berrick, L. Pham, G. Leptoukh, Z. Liu, H. Rui, S. Shen, W. Teng, T. Zhu NASA Goddard Earth Sciences (GES) Data & Information.
Recent developments with the THREDDS Data Server (TDS) and related Tools: covering TDS, NCML, WCS, forecast aggregation and not including stuff covered.
Data Discovery and Access to The International Surface Pressure Databank (ISPD) 1 Thomas Cram Gilbert P. Compo* Doug Schuster Chesley McColl* Steven Worley.
Unidata’s TDS Workshop TDS Overview – Part I July 2011.
GEON2 and OpenEarth Framework (OEF) Bradley Wallet School of Geology and Geophysics, University of Oklahoma
RDA Data Support Section. Topics 1.What is it? 2.Who cares? 3.Why does the RDA need CISL? 4.What is on the horizon?
Mercury – A Service Oriented Web-based system for finding and retrieving Biogeochemical, Ecological and other land- based data National Aeronautics and.
1 NASA CEOP Final Summary CEOS WGISS-26 Boulder, Colorado September 23, 2008 Yonsook Enloe
The TIGGE Model Validation Portal: An Improvement in Data Interoperability 1 Thomas Cram Doug Schuster Hannah Wilcox Steven Worley National Center for.
29 March 2004 Steven Worley, NSF/NCAR/SCD 1 Research Data Stewardship and Access Steven Worley, CISL/SCD Cyberinfrastructure meeting with Priscilla Nelson.
1 Adventures in Web Services for Large Geophysical Datasets Joe Sirott PMEL/NOAA.
TIGGE Archive Status at NCAR THORPEX Workshop and 6th GIFS-TIGGE Working Group Meetings WMO Headquarters Geneva September 2008 Steven Worley Doug.
SCD Research Data Archives; Availability Through the CDP About 500 distinct datasets, 12 TB Diverse in type, size, and format Serving 900 different investigators.
Weathertop Consulting, LLC Server-side OPeNDAP Analysis – Concrete steps toward a generalized framework via a reference implementation using F-TDS Roland.
LAS and THREDDS: Partners for Education Roland Schweitzer Steve Hankin Jonathan Callahan Joe Mclean Kevin O’Brien Ansley Manke Yonghua Wei.
SCD User Briefing The Community Data Portal and the Earth System Grid Don Middleton with presentation material developed by Luca Cinquini, Mary Haley,
The Research Data Archive at NCAR: A System Designed to Handle Diverse Datasets Bob Dattore and Steven Worley National Center for Atmospheric Research.
TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox.
Research Data Archive (RDA) Access and Services from Yellowstone Grace Peng and Doug Schuster 1.
ORNL DAAC SPATIAL DATA ACCESS TOOL Open Geospatial Consortium (OGC) Services Bruce E. Wilson Suresh K. Santhana Vannan Yaxing Wei Tammy W. Beaty National.
Data Discovery and Access to The International Surface Pressure Databank (ISPD) 1 Thomas Cram Gilbert P. Compo* Doug Schuster Chesley McColl* Steven Worley.
RDA Data Support Section. Topics 1.What is it? 2.Who cares? 3.Why does the RDA need CISL? 4.What is on the horizon?
A Use Case for GEON 1 A user request of the form: “For a given region (i.e. lat/long extent, plus depth), return a 3D structural model with accompanying.
1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)
5-7 May 2003 SCD Exec_Retr 1 Research Data, May Archive Content New Archive Developments Archive Access and Provision.
Interoperability Day Introduction Standards-based Web Services Interfaces to Existing Atmospheric/Oceanographic Data Systems Ben Domenico Unidata Program.
Update on Unidata Technologies for Data Access Russ Rew
LP DAAC Overview – Land Processes Distributed Active Archive Center Chris Doescher LP DAAC Project Manager (605) Chris Torbert.
The TIGGE Model Validation Portal: An Improvement in Data Interoperability 1 Thomas Cram Doug Schuster Hannah Wilcox Michael Burek Eric Nienhouse Steven.
1. Gridded Data Sub-setting Services through the RDA at NCAR Doug Schuster, Steve Worley, Bob Dattore, Dave Stepaniak.
A41I-0105 Supporting Decadal and Regional Climate Prediction through NCAR’s EaSM Data Portal Doug Schuster and Steve Worley National Center for Atmospheric.
Introduction What purpose does a data archive center serve if users can’t find or access the holdings they might need to facilitate their research discoveries?
Data Browsing/Mining/Metadata
Global Precipitation Data Access, Value-added Services and Scientific Exploration Tools at NASA GES DISC Zhong Liu1,4, D. Ostrenga1,2, G. Leptoukh4, S.
Flanders Marine Institute (VLIZ)
TIGGE Data Archive and Access System at NCAR
Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System Zaihua Ji Doug Schuster Steven Worley Computational.
Live Access Server (LAS)
Steven Worley, Douglas Schuster,
CISL’s Research Data Archive (RDA) : Description and Methods
Comeaux and Worley, NSF/NCAR/SCD
Data Management Components for a Research Data Archive
Robert Dattore and Steven Worley
Successful Data Curation for Large Data Archives
Presentation transcript:

Introduction Downloading and sifting through large volumes of data stored in differing formats can be a time-consuming and sometimes frustrating process for researchers who would rather be spending that time finding answers to research questions. To assist more than 10K researchers annually through this process, the NCAR Research Data Archive (RDA) employs a combination of in-house developed and standards based access tools, including a recently deployed Unidata THREDDS Data Server (TDS). This presentation highlights the services provided through TDS and the expected benefits for end users. System Background The RDA provides a variety of access tools to serve a broad user community. Data subset requests can be submitted through in-house developed web interfaces and command line app clients for asynchronous processing. Resulting output files are are made available for download once processing is completed. To complement the asynchronous processing functionality and enhance the user experience, the RDA has started implementing the Unidata TDS for use on highly valued gridded dataset collections. The TDS uses a variety of standards-based services, including OPeNDAP and OGC protocols. These allow users to access virtual (aggregated) data files remotely through a variety of client tools, without needing to download the data first. Client tools can also directly create temporal, parameter, and spatial subsets, analyses, and other products. A growing tool set is accessing RDA data through the TDS (Figure 1). Acknowledgments The members of the NCAR Computational Information Systems Laboratory (CISL), Data Support Section (DSS) support the infrastructure and software described in this presentation. They include Joey Comeaux, Thomas Cram, Cecilia Banner, Bob Dattore, Dave Stepaniak, Chi-Fan Shih, and Steve Worley. Figure 1. RDA TDS Services and Access Pathways User Benefits from RDA TDS services Data extraction across multi-file datasets using single service point directives. No need to download large data files to perform a “first- look” analysis Visualizations and resulting subsets are created on-the-fly Following visual discovery the data can be easily exported to a local netCDF data file for further examination Services are standards based, use your favorite tool Multiple data access service protocols are available including OPeNDAP and OGC Metadata provided in NCML and ISO standards Archive data format transparent to the end user THREDDS metadata can be harvested and searched by distributed tools Asynchronous Request Metrics User metrics display a continued growth in use of customized request access options (Figure 2). This trend supports the need for addition access options including TDS. Figure 2.. Monthly customized RDA user access from the RDA web interface. Access options include data subset and file format conversion requests. Once sufficient data are available, usage metrics will also be compiled for the TDS data access methods. Kevin Manross, Zaihua Ji, and Doug Schuster CISL/DSS, National Center for Atmospheric Research, Boulder, CO, United States.,, Relevant Links RDA THREDDS Data Server Unidata THREDDS Data Server software home page ` OPeNDAP homepage OPeNDAP capable client data access/analysis tools OGC Protocol homepage OGC capable client data access/analysis tools OPeNDAP/THREDDS Access to the NCAR Research Data Archive Currently/Month 1 PB Data Processed 30 TB Data Delivered 3000 Requests 500 Users