The Arctic Observing Network and its Data Management Challenges Florence Fetterer (NSIDC/CIRES/CU), James A. Moore (NCAR/EOL), and the CADIS team Photo courtesy Andrew Mahoney

What is AON? “…a system of atmospheric, land- and ocean-based environmental monitoring capabilities--from ocean buoys to satellites--that will significantly advance our observations of Arctic environmental conditions. Data from the AON will enable the interagency U.S. government initiative--the Study of Environmental Arctic Change (SEARCH)--to get a handle on the wide-ranging series of significant and rapid changes occurring in the Arctic.” NSF Vision …a portal through which scientists can find all data relevant to a location or process; …all data have browse imagery and complete documentation; …time series or fields can be plotted online; …and all metadata are in a relational database so that multiple data sets and sources can be queried. Vision for an AON data portal Toward an Integrated Arctic Observing Network, NAS, Data Management chapter. ( og/11607.html)

AON at present (Google ‘AON Arctic EOL’ for more info)

The Cooperative Arctic Data and Information Service (CADIS) A joint NCAR, NSIDC, UCAR project to meet AON data management challenges Challenges include –Multidisciplinary data, in many formats –Diversity of archiving and access needs –Meeting PI’s expectations –Coordinating with/leveraging other EO CI and DM interoperability efforts

Meeting PI and Community Expectations Enter metadata quickly, efficiently, and without confusion Find my own data and display it Display it over or with other data sets of interest (that may or may not be in the CADIS system) Mathematically manipulate data Export in a choice of formats Easy to use, graphical interfaces Examples to emulate ArcticRIMS - NOAA ESRL PSD -

Coordinating With Other Earth Observation CI and Data Management Projects NSF funded data portals –Marine Geoscience Data System –Geosciences Network (GEON) –National Ecological Observatory Network (NEON) Other portals of note –Alaska Ocean Observing System (part of IOOS/GCOS) –ESA’s soon to be Group on Earth Observations GEOPortal The Marine Metadata Initiative DAMOCLES, with links to WMO information system “…if efforts are not coordinated, the observatory communities will continue to expend great resources developing their CI in isolation….While numerous CI tools and services already exist that can be leveraged, they represent different architectural approaches, typically targeted at a very narrow set of requirements, and are not well integrated with each other.” -- Shared Cyberinfrastructure for Environmental Observatories: Identifying Common Needs, Barriers, and Solutions. Whitepaper by Minsker and Butler, NCSA

Time pressure demands a pragmatic approach… First priority: A user friendly data and metadata entry tool –Build on existing tools – NCAR’s Community Data Portal for PI-managed data management –GCMD Doc-builder as a rough model for a metadata entry tool Metadata – A CADIS profile –Incorporates the International Polar Year profile: –Compatible with GCMD DIF, FGDC CSDGM and eventually, with ISO19115/19139 –Adds fields for improved discovery

AON Metadata Entry Form at CDP PIs enter metadata Context sensitive help Controlled vocabulary where possible Fields pre-populated where possible Can start with metadata from a data set that is ‘like’ yours What happens to the metadata after its entered? –In a metadata db –In a directory exposed to Web crawlers –Can be mapped to other standards –Native format is NCAR THREDDS

CADIS Data Hierarchy on the CDP Nested Collections

Individual Dataset PIs determine what constitutes a data set

Data in a standard format where possible NetCDF with CF conventions –Many existing tools for subsetting and mathematically manipulating large multidimensional data sets in NetCDF –Less experience with station or point data in netCDF Translators to get data into NetCDF from EASE-Grid, GRIB, and other formats are in development Request for data in BUFR (Arctic Reanalysis)

Visualization – Work toward addressing the basic use case Locate data –Map server –Google Earth or other virtual globe –Google-type search –Faceted search Visualize data –Visualization tools will build on NOAA PMEL’s Live Access Server (LAS/Ferret) and NCAR’s Integrated Data Viewer (IDV) –Virtual globes and/or GIS may be used

Summary Just completing first year of CADIS development Data/Metadata entry Portal testing underway Datasets into the archive beginning in early 2008 CADIS features will evolve with improvements in the Community Data Portal interface

The CADIS Team = NCAR CISL, NCAR EOL, UCAR, and NSIDC Grant PIs

Engineering Roadmap

Definitions The Community Data Portal (CDP) integrates a suite of technologies including THREDDS and OPeNDAP (described below), Open Archive Initiative (OAI) metadata sharing, transparent access to NCAR’s Mass Storage System, and a number of other resources. Under CDP architecture, data, metadata and services may be hosted on a system of separate servers interacting over the network, and metadata can be shared with other cooperating data centers and virtual organizations. CDP is Java-based and accessed via both browsers and applications. The AON portal will be a customization of the CDP running on an NSIDC server. Thematic Realtime Environmental Distributed Data Services (THREDDS) is for publishing and accessing data in netCDF or other self describing formats. Data services include: an OPeNDAP data server; an Open Geospatial Consortium Web Coverage Server, and a catalog that describes what data sets are available from a given THREDDS data server. THREDDS is written in Java, It works with data in many formats (e.g. netCDF, GRIB, HDF, ASCII etc.) and allows them to be described with many metadata standards (e.g. DIF, Dublin Core, and Climate and Forecast). THREDDS-aware desktop clients, such as IDL, MatLAB, or IDV, can access data in THREDDS catalogs (reached though the AON portal) directly. OPeNDAP

Definitions (Con’t) Open-source Project for a Network Data Access Protocol (OPeNDAP) is a protocol for requesting and transferring data over the Web using HTTP to frame requests and responses. Data are accessible remotely regardless of their local storage format. OPeNDAP is used to serve data in a variety of different formats, including NetCDF, GRIdded Binary (GRIB), and Hierarchical Data Format (HDF). (PMEL has a utility that allows in situ data like buoy data to be served via OPeNDAP as well). OPeNDAP servers (also known as DODS servers) located at institutions around the world provide model data, climatic data, satellite imagery, and ocean sensor results. The Internet Data Distribution (IDD) system makes data available in real-time. The user "subscribes" to data streams from a variety of sources and networks. The IDD then delivers the data as soon as they are available from each source using the Local Data Manager (LDM) software package to relay data from the source to the end user sites. The LDM supports flexible, site-specific configuration. We are envisioning the IASOA observatories using this technology as a trial for possible use by other AON observing systems. Currently IDD/LDM is used by 175 institutions for, e.g., satellite imagery, and National Weather Center output. The Unidata Integrated Data Viewer (IDV) is a software client for visualizing data (satellite imagery, gridded data, surface observations) within a single interface. IDV can display data that are in NetCDF, and will in the future handle data in HDF-5 as well. The Geosciences Network (GEON) is using IDV as its primary visualization tool.