1. Key new features in the research data archive
Presented by the Data Support Section (DSS) of SCD
Introduction ··················································
Steven Worley
ERA-40 Data ··················································
Joey Comeaux
NARR Data ····················································
Chi-Fan Shih
Near Real-time Data from the IDD ····················
Doug Schuster
Metadata: Building Data Discovery and Access ···
Bob Dattore
NSF/NCAR/SCD/DSS
5 February 2019

2. Research Data Archive definition
Collection of reference datasets used in atmospheric and related sciences
Over 600 datasets
Add new datasets annually
Maintained by 9 staff
Established 40 years ago
All RDA on the MSS
548K files
100.5 TB
NSF/NCAR/SCD/DSS
5 February 2019

3. RDA Content Many categories of data Observations
Model Output (NWP & Reanalyses)
Climatology Data
Station summary data
Satellite derived datasets
Global topography
Much more ….
NSF/NCAR/SCD/DSS
5 February 2019

4. Data User’s Functional Diagram for RDA Access
MSS
All RDA data
RDA(DSS)
Server
Some data
All Metadata
CDP
Standardized
Metadata
NCAR User
Internet
User
NSF/NCAR/SCD/DSS
5 February 2019

5. Data User’s Functional Diagram for RDA Access
MSS
All RDA data
RDA(DSS)
Server
Some data
All Metadata
CDP
Standardized
Metadata
NCAR User
Internet
User
metadata
NSF/NCAR/SCD/DSS
5 February 2019

6. Data User’s Functional Diagram for RDA Access
MSS
All RDA data
RDA(DSS)
Server
Some data
All Metadata
CDP
Standardized
Metadata
NCAR User
Internet
User
metadata
data
NSF/NCAR/SCD/DSS
5 February 2019

7. Data User’s Functional Diagram for RDA Access
MSS
All RDA data
RDA(DSS)
Server
Some data
All Metadata
CDP
Standardized
Metadata
NCAR User
Internet
User
metadata
data
future data
NSF/NCAR/SCD/DSS
5 February 2019

8. User Metrics, January-April 2005
Archive Source
Unique Users
Data Amount (TB)
NCAR MSS
242
19.1
RDA Server
1604
6.4
Highlight from RDA Server:
Dataset Description
Users
Amount (GB)
2.5° Tropospheric Analysis
123 36
1.0° Tropospheric Analysis
480
1884
Global Upper Air Observations
128 23
Global Surface Observations
154
220
The most recent year of each archive is online
NSF/NCAR/SCD/DSS
5 February 2019

9. ECMWF ERA-40 Data from NCAR by Joey Comeaux Data Support Section
NSF/NCAR/SCD/DSS
5 February 2019

10. Atmospheric model coupled to an Ocean Wave model
Model Specifics
Sept, 1957 – Aug, 2002
T159 spectral truncation
Reduced N80 (~125km)
60 Hybrid Levels
Ingested one of the largest archives of observational data ever assembled
Atmospheric model coupled to an Ocean Wave model
NSF/NCAR/SCD/DSS
5 February 2019

11. Analysis Fields on RDA web server
PRODUCTS
ANALYSIS TIMES
Daily at 00, 06, 12 and 18 UTC
FORMAT:
GRIB
LOCATION:
All Data on MSS
Analysis Fields on RDA web server
NSF/NCAR/SCD/DSS
5 February 2019

12. Products Horizontal Resolution Surface Pressure Level Model T159/N80
ds117.0
ds117.1
ds117.2
2.5
ds118.0
ds118.1
T106/N80
ds127.1
T85/N64
ds124.0
ds124.1
ds124.2
T159/N80 FCST
ds121.0
ds121.1
ds121.2
Values in table indicate DSS dataset IDs
Web pages available at dss.ucar.edu/datasets/dsnnn.n
Monthly Means available for ALL products
NSF/NCAR/SCD/DSS
5 February 2019

13. PRODUCTS Variable Lists
Surface
109 variables
Wind, temp, mslp, sfcp, cloud cover info, precip, fluxes radiation, stresses, Vertical Integrals, & more
Pressure Level
Z,T,W, RH,Q,Vort, Div, ozone mixing
Model Level
Same as pressure level + cloud liquid & ice water content, cloud cover
NSF/NCAR/SCD/DSS
5 February 2019

14. Other Products available 4X Daily at Model Resolution
Ocean Wave Analysis Fields
ds123.0
Isentropic
ds117.3
PV ±2 PVU
ds117.4
Ocean Wave Forecast Data
ds123.1
Chemical Transport
ds117.5
Net Tendency
ds117.1
Radiative Tendency
ds117.6
Feedback Records
??????
NSF/NCAR/SCD/DSS
5 February 2019

15. NSF/NCAR/SCD/DSS
5 February 2019

16. DSS ERA-40 web page More Info Available http://dss.ucar.edu/pub/era40
WEB links in all DSS talks available in handout
NSF/NCAR/SCD/DSS
5 February 2019

17. North American Regional Reanalysis NARR Data Chi-Fan Shih
Good afternoon. My name is Chi-Fan Shih. I’m going to give you a quick tour of our NARR data archive.

18. NARR Data Archive at NCAR
Introduction
Products
A Few Details
How to Access the Data
Summary
First of all, the URL of our NARR web site is My talk includes an introduction, brief description of different subgroups, a few details to be aware of, how to access the data and a summary.
NSF/NCAR/SCD/DSS
Chi-Fan Shih
5 February 2019

19. NARR & Eta-212 Domain NARR Archive NSF/NCAR/SCD/DSS
Before NARR data become available, Eta-212 archive is the only operational mesoscale product we have from NCEP. This slide shows the coverages of both Eta-212 and NARR. Next slide shows differences between NARR and Eta-212.
NSF/NCAR/SCD/DSS
5 February 2019

20. Introduction NARR Eta-212 32 km 40 km 29 levels 26 levels
Archive
Introduction
NARR Eta-212
32 km km
29 levels levels
Lambert-Conformal same
3 hourly same
May-continuing
Gridded Data: GRIB-1 same
Analysis Init, Anal, Fcst
Monthly Means
Climatologies
On-line, Mass Storage Mass Storage
NARR’s resolution is 32km in the horizontal direction and has 29 vertical levels. It is on Lambert-Conformal projection. It is 3 hourly and we have the data from 1979 to All data files are in GRIB edition 1 format. You need a GRIB decoder to read, unpack the data.
Currently we have 3-hourly analyses, monthly means and climatologies on-line as well as on mass storage systems.
NSF/NCAR/SCD/DSS
Chi-Fan Shih
5 February 2019

21. 3-hourly Data Products Raw Files from NCEP
NARR
Archive
Products
3-hourly Data
Raw Files from NCEP
- daily “a” & “b” tar files, 466 MB/day
- MSS
- Total volume: 4.2 TB,
Regrouped Files
- subgroups: 3D, clm, flx, sfc, pbl
- file sizes from 16 MB/day to 269 MB/day
- tar files, on-line, MSS
The 3-hourly analyses were downloaded from NCEP. The total data volume is about 4.2TB. We call those original files the “raw” files. They are on the mass storage systems only. To better serve data users, we have regrouped those “raw” files into 5 subgroups: 3D,clm,flx,sfc and pbl. Their filesizes range from 16MB/day to 269MB/day. In comparison to the 466MB/day of the “raw” files, the regrouped files are more manageable. Those regrouped data are available on-line or from mass storage in tar files.
NSF/NCAR/SCD/DSS
Chi-Fan Shih
5 February 2019

22. Products Monthly Means Climatologies (Kingtse.Mo@noaa.gov )
NARR
Archive
Products
Monthly Means
- averages of all data in a month, monthly averages of data every 3 hours
- from NCEP
- from 1979 to 2003, total volume 144 GB
- on-line, MSS
Climatologies )
- from 1979 to 2001, total volume 20 GB
- including monthly means
Both the monthly means and climatolgoies are also available on-line or from mass storage. They are archived in the same way as when we download them from NCEP.
NSF/NCAR/SCD/DSS
Chi-Fan Shih
5 February 2019

23. NARR
Archive
Products
Regrouped 3-hourly Data
3D: 10 variables, 269 MB/day,H,T,U,V,W,RH,.. on 29 pressure levels (from 1000mb to 100mb)
clm: 12 variables, 16 MB/day, atmospheric column precipitable water, convective cloud, water vapor zonal & meridional fluxes, ..
flx: 81 variables, 87 MB/day, U & V at 10m, soil moisture & temperature, radiation, .. not at surface
sfc: 44 variables, 50 MB/day, T, Pres, heat fluxes, radiation, .. at surface
pbl: 39 variables, 45 MB/day, T,U,V,W,.. in layers between surface and 180mb above ground
Here let me explain a little bit about the regrouped 3-hourly data files. The 3D files contain all 10 variables on 29 pressure levels. Those 10 variables include geopotential height, temperature, U, V and vertical wind, relative humidity, etc. The clm files have 12 variables and they are integrated throughout one atmospheric column. The flx files contain flux terms not at surface. The flux terms at the suface are saved in the “sfc” subgroup. The properties of near-surface layers between the ground and 180mb above the ground are saved in the “pbl” subgroup.
NSF/NCAR/SCD/DSS
Chi-Fan Shih
5 February 2019

24. A Few Details NARR has its own GRIB parameter table
Archive
A Few Details
NARR has its own GRIB parameter table
wgrib with NARR GRIB table to get correct parameter names
U, V of NARR are different from U, V from operational Eta output
Constant (or fixed ) fields are on-line
Before telling you how to get NARR data from DSS RDA server, here is a few reminder about NARR data. First, NARR has its own GRIB parameter table. Because of numerous new variables, you’ll not see their names if wrong GRIB table is used. For wgrib users, you can build wgrib with “user-defined” GRIB table option. That will assure you to get the correct NARR variable names. An important difference between NARR data and the operational Eta data is the wind. The U,V winds in NARR are true east-west, north-south winds. The U,V winds in operational Eta products are grid-relative. Finally, users who need the fixed fields used in NARR, such as surface geopotential height, surface land-sea coverage, etc., they are available on-line.
NSF/NCAR/SCD/DSS
Chi-Fan Shih
5 February 2019

25. NARR
Archive
How to Access the Data
NCAR Users - msrcp from MSS in /DSS/DS608.0/{3HRLY_TAR,MONTHLY,CLIMATOLOGIES,UCSD}/…
web –
ftp – to be implemented, contact
place a data order – tapes, etc., fees apply
small amount of data – ncdc.noaa.gov
How do you get the NARR data?
NSF/NCAR/SCD/DSS
Chi-Fan Shih
5 February 2019

26. If you go to our NARR web site, its top page looks like this
If you go to our NARR web site, its top page looks like this. The data flies are listed in the left frame. After a simple registion, you click on 3-hourly Data files. It leads you to select the year,month.
NSF/NCAR/SCD/DSS
5 February 2019

27. Clicking on, say 198002, you’ll see the list of regrouped tar files.
NSF/NCAR/SCD/DSS
5 February 2019

28. Clicking on the check boxes and “download files now”, the web download process will begin. To avoid downloading too many files for a 32-bit system, we have limited the total file size to be about 2GB. You’ll see a window poping up if this limit is exceeded.
NSF/NCAR/SCD/DSS
5 February 2019

29. NARR
Archive
Summary
All NARR data files are on-line on DSS RDA server as well as on MSS (in ds608.0 dataset)
ftp download from DSS RDA server will be available
Examples for NCAR users to retrieve MSS files will be added
NSF/NCAR/SCD/DSS
Chi-Fan Shih
5 February 2019

30. Near Real-time Data from the IDD by Doug Schuster Data Support Section
NSF/NCAR/SCD/DSS
5 February 2019

31. Near Real-time Data from the IDD
Dataset Background
Observational Data Products
Model Data Products
IDD Data Access
NSF/NCAR/SCD/DSS
5 February 2019

32. Unidata’s Internet Data Distribution Service
IDD
Distributes Meteorological Data to private and public institutions subscribing to the service via internet nodes.
Data is captured by subscribers and used for real-time weather analysis, forecasts, model initialization, misc. research, etc…
NSF/NCAR/SCD/DSS
5 February 2019

33. Near Real-time Data from the IDD
Contents
- Includes real-time observational and model
data captured and archived from the
IDD.
Purpose
- Provide a resource to examine events in a near real-time setting (e.g. examine last week’s tornado outbreak).
- Provide a permanent backup of the selected data to complement Unidata’s distribution service.
NSF/NCAR/SCD/DSS
5 February 2019

34. Observational Data, May2003 - current
Original reports, transmitted across the Global Telecommunication System (GTS), are captured and decoded by Unidata software.
NO QC of observations is performed.
The decoded reports are stored and archived in netCDF format.
Used to supplement DSS’s ADP and ISH (DATSAV) dataset archives.
NSF/NCAR/SCD/DSS
5 February 2019

35. Available Obs Data, May 2003 - current
COVERAGE
REPORT TYPE
FREQUENCY
Land Surface
SYNOP
3-Hourly
METAR
Hourly
Marine Surface
BUOY
Upper Air
RAOB
NSF/NCAR/SCD/DSS
5 February 2019

36. 00 UTC Upper Air Station Coverage
Sample 00Z IDD Upper Air Stations
NSF/NCAR/SCD/DSS
5 February 2019

37. Gridded Model Data, Dec 2002 - Current
Stored in GRIB format on standard NCEP transmission grids.
Used to supplement gridded and reanalysis datasets.
Includes initialization and forecast grids.
NSF/NCAR/SCD/DSS
5 February 2019

38. Available Model Data, Dec 2002 - Current
Init Times
Forecast Length
Coverage
Grid Res
RUC
3-hr int
3-hr int
CONUS
81km LC
Grid 211
ETA/NAM
00Z, 12Z
6-hr int
GFSNH
6-hr int
6-hr int
N. H.
381km PS Grid 201
GFSEXT
00Z
12-hr int
Global 5˚
Lat/Lon
NCEP ENS
6-hr int.
0-90˚ N
˚ E
1.5˚ Grids
39 and 40
ECMWF
12Z
24-hr int.
5˚ Mercator
NSF/NCAR/SCD/DSS
5 February 2019

39. IDD Data Access
Most Recent 3-months of data are available on RDA server.
RDA server archive populated daily.
RDA server archive copied to MSS weekly.
IDD Obs Data (May current, RDA server & MSS)
IDD Model Data (Dec current, RDA server & MSS)
NCEP Grid Definitions
NSF/NCAR/SCD/DSS
5 February 2019

40. Metadata: Building Data Discovery and Access Bob Dattore dattore@ucar
Metadata: Building Data Discovery and Access Bob Dattore
NSF/NCAR/SCD/DSS
5 February 2019

41. History of Metadata
DSS has been compiling metadata in digital form for 20+ years
Dataset-level metadata for all datasets in a quasi-standard format
File-level metadata for all data files on the NCAR MSS
Format of this information varies by the dataset specialist who creates/enters it
NSF/NCAR/SCD/DSS
5 February 2019

42. Dataset-Level Metadata
Identifies and locates datasets having required data
Allows searches by “key” fields (e.g. - date, parameter, project, platform, etc.)
Drives main methods of discovery:
Search Engines
Dataset Catalogs (precipitation, MM5)
Interaction with DSS Staff
NSF/NCAR/SCD/DSS
5 February 2019

43. File-Level Metadata
Identifies and locates files containing needed data
Gives information about the data (format, obs or gridded?, etc.)
Gives information about the files (COS-blocking, volume, etc.)
Drives access methods:
Read directly from MSS (msrcp)
Download from web servers (DSS, CDP)
NSF/NCAR/SCD/DSS
5 February 2019

44. Goals for Improvement Standardize all metadata
DSS standards (XML)
Controlled keyword lists
Common formats for describing data files
Easier to build software to process the metadata
Capture all the metadata necessary to completely describe our datasets and files
Allows us to map our metadata into other standards (THREDDS, DIF, etc.)
NSF/NCAR/SCD/DSS
5 February 2019

45. Goals for Improvement Use databases for quicker access
Faster searches for data discovery
Faster access to metadata by software (e.g. - program to subset a large dataset)
Create the ability to access information and data files over aggregations of data
NSF/NCAR/SCD/DSS
5 February 2019

46. NSF/NCAR/SCD/DSS
5 February 2019

47. NSF/NCAR/SCD/DSS
5 February 2019

48. NSF/NCAR/SCD/DSS
5 February 2019

49. NSF/NCAR/SCD/DSS
5 February 2019

50. NSF/NCAR/SCD/DSS
5 February 2019

51. NSF/NCAR/SCD/DSS
5 February 2019

52. NSF/NCAR/SCD/DSS
5 February 2019

53. Improvements to the MSS File Lists
Provide more information about each data file
Allow users to create customized lists (e.g. - show only the files for 2004, only files containing temperature, etc.)
NSF/NCAR/SCD/DSS
5 February 2019

54. The RDA on the CDP
The Community Data Portal is a collection of datasets from participating organizations
The next session, “New data portals for the community”, will talk about the CDP in more detail
All of our datasets are represented on the CDP
NSF/NCAR/SCD/DSS
5 February 2019

55. The RDA on the CDP
Datasets can be searched alongside those of other participants
As our metadata improves, so will the THREDDS catalogs
We are working to build MSS file lists for all datasets
Will allow NCAR users to access our MSS files through the CDP
NSF/NCAR/SCD/DSS
5 February 2019

56. NSF/NCAR/SCD/DSS
5 February 2019

57. Comments?
Questions?
NSF/NCAR/SCD/DSS
5 February 2019