1. The NOAA Operational Model Archive and Distribution System NOMADS The NOAA Operational Model Archive and Distribution System NOMADS Glenn K. Rutledge Meteorologist / Physical Scientist National Oceanic and Atmospheric Administration National Climatic Data Center Perspectives on Building Communities for Effective Development and Application of Cyberinfrastructure SuperComputing 2004 Pittsburgh PA, November 12, 2004 Image: Unidata idv

2. To overcome a deficiency in model data access, some of the Nations top scientists are actively engaged in a grass-roots framework to share data and research findings over the Internet. NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System. NOMADS is a distributed data services pilot for format independent access to climate and weather models and data. Overview

3. foster research within the geo-science communities (ocean, weather, and climate) to study multiple earth systems using collections of distributed data promote model evaluation and product development develop institutional partnerships via distributed open technologies provide distributed access to models and associated data NOMADS Goals The NOAA Operational Model Archive and Distribution System

4. The users experience is often frustrating— - What data of interest exist? - Are they going to be useful to me? - How can I obtain them in a usable form? Time and effort are wasted on data access and format issues. As a result atmosphere/ocean/climate data are under- utilized. Model inter-comparison nearly impossible. Model Data Access

5. Why Now? How can Agencies and institutions develop partnerships while allowing for attribution, with diverse goals and agendas? What are the goals facing the GeoScience community? How will Agencies and Institutions address interoperability? Have the scientific requirements been adequately defined? Community building with science driven requirements! Do top down approaches adequately promote science? Should it be system, data or both? Is it just access to high volume data (satellite, radar, and model)?

6. Climate model output and observations are vital to providing timely assessments of climate change and impacts: climate model output statistics from NWP run in quasi-real time to identify time-dependant biases in observations; assess the affect of missing data; for observing network design and operation, models can be used to guide the spatial and temporal sampling frequency to resolve distributions for specific variables (temp, precip). NOMADS Uses

7. Uses (cont.) Improved climate and weather assessments. Accurate estimates of future climate variability and trends. Long-term stewardship of NWP analysis and forecasts. Promote collaboration of GCM and NWP researchers using large data volumes by sub-setting. Inter-comparisons of weather and climate data sets.

8. CDC COLA co-PI FSL GFDL co-PI LLNL NCAR NCDC PI NCEP co-PI PMEL co-PI Unidata -THREDDS BADC CEOS DataGrid CEOP LEAD NASA/GCMD NASA/SI-ESIP NSSL w/UW/SSEC NWS/COMET/HQ GMU / Alabama / others The Partnerships

9. International Partners WCRP/WGCM JCS Data Management British Atmospheric Data Center & NERC Data Grid (e-Science) Climate Action Partnership" (CAP) with Commerce, Energy, State, and EPA with Australia BOM. GODAE, IOOS, GOOS, GCOS, HAOC, NVODS projects. European PRogram for Integrated Earth System Modeling (PRISM) Committee of Earth Observation Satellites (CEOS) Coordinated Enhanced Observing Period (CEOP) Interagency Working Group on Earth Observations (IWGEO)

10. Tools for Users Pare down large file sizes of high resolution data and products. (re-) Group different data sets to create needed products – such as initialization files for model development, analysis, and intercomparison. Subset the data: –in parameter space –in physical space –in temporal space

11. NOMADS uses the Open Source http based OPeNDAP. OPeNDAP is a binary-level protocol designed for the transport of scientific data subsets over the Internet. Data formats: cf, DIF, GRIB, GRIB2, BUFR, HDF, NetCDF, ascii,....libraries built as necessary. APIs: JAVA-OPeNDAP, C++-OPeNDAP, NetCDF, GRIB, BUFR, THREDDS, Python. Framework

12. NOAAPort Unidata IDD NCDC ftp Data & Directory structures “merged” Daily Data Ingest inter-comparison QC and Monitoring Index File generation Control and DODS metadata generation CVS Backup (code) HDSS/HAS Injection CEOS-Grid (exploratory) Obs, Eta, GFS, RUC Hi-Res GFS, Eta, NARR and GDAS Data Ingest* Dual Redundant Ingest Data Management Data Access DSI 6172 DSI 6173 DSI 6174 DSI 6175 NOMADS Access System Architecture * NOMADS provides NCDC NOAAPort ingest processing NCDC Archive AAB Ingest Processes DAB/NOMADS Ingest Processes AAB Archive Processes

13. NOMADS on-line volume: 16.7TB NOMADS Archive Volume: ~400TB+ Data Ingest -NOAAPort Direct -NCEP ftp -Unidata IDD Data Access Data Management NOMADS HAS Interface / QC / Metadata NOMADS System Architecture NSA_GKR_Mar04 Globus

14. Data Portals GDS LAS * GrADS, Ferret, MatLab, IDL, IDV, Web browsers or any OPeNDAP enabled client NSDL, GCMD, and THREDDS catalog / search Services NCDC Archive Data ingest (dual) 150K+ grids/day ESP The Grids include: CEOS-DataGrid, DOE’s ESG, NASA’s IPG, ADG, & UK’s NERC DataGrid Exploratory Grid effort CEOS Server * In process The Big Picture

15. NCDC NOMADS Archive NCEP NWP Operational Model Output from NOAAPort POR: 2002 to Real-Time (to 1999 exists) Eta (12km) and Global Forecast Models (1 degree) RUC-II 40km; (FSL NOMADS has real-time 20km) Limited Navy NOGAPS (1/2 degree FY05) NCDC Reference Data Sets (Reynolds SST’s, GHCN) NCEP Real-Time NOMADS Global Forecast System GFS (AVN/MRF) (1 degree) Hourly Eta at 12km Regional Spectral Model (RSM) Climate Data Assimilation System (CDAS) AMIP Climate Monitoring NCEP/NCAR Global Reanalysis 1&2 NCDC/NCEP Data Availability

16. Period of Record for NOMADS Datasets As of November 05, 2004 Model SystemGridOnline AvailabilityAvailability from deep archive GFS-AVNALLJune 01, 2003 to PresentMay 01, 2002 to May 31, 2002 GFS-MRF201June 01, 2003 to PresentMay 01, 2002 to May 31, 2002 202 203July 01, 2003 to PresentJanuary 17, 2003 to May 31, 2003 205 Early-ETA212June 01, 2003 to PresentJuly 27, 2002 to May 31, 2003 Meso-ETA211June 01, 2003 to PresentJuly 27, 2002 to May 31, 2003 212 215 218June 01, 2003 to PresentOctober 02, 2002 to May 31, 2003 RUC211June 01, 2003 to PresentJuly 27, 2002 to May 31, 2003 RUC2536June 01, 2003 to PresentJuly 27, 2002 to May 31, 2003 MESO-ETA High Resolution 003May 02, 2004 to PresentMarch 01, 2004 to May 01, 2004 GFS-AVN High Resolution 218June 01, 2004 to PresentMarch 01, 2004 to May 31, 2004 North American Regional Reanalysis 221January 01, 1979 to December 31, 2003This dataset is completely online

17. NOAAPort Numerical Weather Prediction Models on NOMADS Available on NOMADS ModelGridDomainResolutionCyclesForecastsLevels and [variables] Global Forecast System Aviation (GFS-AVN) 201 Northern Hemisphere 381.0 Km6 Hours6 Hrs. to 120. 10 Pressure Levels [6], Tropopause [5], and basic surface variables [7] 202North America190.5 Km6 Hours6 Hrs. to 120.10 Pressure Levels [6], Surface variables [14] 203North Pacific190.5 Km6 Hours6 Hrs. to 120. 10 Pressure Levels [7], Tropopause [5], and basic surface variables [10] 211United States81.27 Km6 Hours 6 Hrs. to 180. 12 Hrs. to 240 29 Pressure Levels [7], Tropopause [5], mb above ground [4], and surface variables [15] 213North America95.25 Km6 Hours6 Hrs. to 120.10 Pressure Levels [7], Tropopause [4], and surface variables [19] Global Forecast System Medium Range Forecast (GFS-MRF) 201 Northern Hemisphere 381.0 KmDaily 00z12 Hrs to 2407 Pressure Levels [6], and basic surface variables [9] 202North America190.5 KmDaily 00z12 Hrs to 2407 Pressure Levels [6], and basic surface variables [9] 203North Pacific190.5 KmDaily 00z12 Hrs to 2407 Pressure Levels [6], and basic surface variables [9] 205Atlantic Basin190.5 KmDaily 00z12 Hrs to 2407 Pressure Levels [6], and basic surface variables [9] Early-ETA212United States40.64 Km12 Hours3 Hrs. to 6013 Surface Variables Meso-ETA 211United States81.27 Km12 Hours6 Hrs. to 60 19 Vertical Levels [6], Layered variables [5], and 17 surface variables including convective parameters [17] 212United States40.64 Km6 Hours3 Hrs. to 60 29 Pressure Levels [6], Layered variables [2], Surface variables [10], and others [2] 215United States20.31 Km6 Hours3 Hrs. to 60 Layered variables [5], and surface variables including convective parameters [27] 218United States12.19 Km6 Hours3 Hrs. to 84Surface variables including convective parameters [14] RUC211United States81.27 KmHourly3 Hrs. to 12 19 Vertical Levels [6], Tropopause [4], Layered variables [4], basic surface variables [7], and others [6] RUC2236United States40.64 KmHourly 1 Hr. to 9, then 12 19 Vertical Levels [6], Tropopause [4], surface variables including convective parameters [29], and others [5]

18. NCEP High Resolution Models Available on NOMADS MESO- ETA 218 United States 12.19 Km 6 Hours 3 Hrs. to 60 141 total variables, with 39 Vertical Levels [8] GFS- AVN 003Global 1 degree Lat/Lon 6 Hours 3 Hrs. to 180 139 Total variables, with 26 Vertical Levels [8] North American Regional Reanalysis Available on NOMADS NARR221 North America 32.46 Km 3 HoursNone 29 Vertical Levels [8], with 187 Total Variables

19. Global Spectral Forecast Model and the Spectral Statistical Interpolation Cycling Analysis System (GDAS): - NOAA-15/16 AMSU-A/B TOVS 1B Radiances (IEEE) - Analysis Bias Corrected Information / Obs Toss List - SFC U/A, ACRS, Aircft (BUFR) - 6HR fcst guess from previous run (BUFR) - ERSCAT Sat obs / HIRS 14/15, MSU TOVS (IEEE) - Guess prep and and fcst guess output (BUFR) - Analysis ready QC’ed Obs. (prepBUFR) - Profiler, TOVS, Wind Obs. (BUFR) - SFC Analysis Restart Files - SST’s (GRIB), Radar VAD Winds (BUFR) Model Input: GDAS

20. NOMADS saves the minimum data necessary to regenerate model output products as close as possible to NCEP operations. The analysis files will be in the models own coordinate system. Files are constructed with computer and computational efficiency in mind, and not in standard coordinate systems. Programs to convert these files will be made available: spectral to gaussian gaussian to lat/lon sigma to pressure Model Input: GDAS (cont.)

21. N.A. Regional Reanalysis Create a long-term set of consistent climate data on a Create a long-term set of consistent climate data on a regional scale on a North American domain regional scale on a North American domain Superior to NCEP/NCAR Global Reanalysis (GR): Superior to NCEP/NCAR Global Reanalysis (GR): use of a regional model (the Eta model) use of a regional model (the Eta model) advances in modeling and data assimilation since advances in modeling and data assimilation since 1995 especially: 1995 especially: Precipitation assimilation Precipitation assimilation Direct assimilation of radiances Direct assimilation of radiances Land-surface (NOAH) model updates Land-surface (NOAH) model updates

22. Assimilation of Hourly Precipitation Hourly 4-km radar/gage analysis (Stage IV) Cold Season Processes (Koren et al 1999) patchy snow cover; frozen soil (new state variable); snow density (new state variable) Bare Soil Evaporation Refinements parameterize upper sfc crust cap on evap Soil Heat Flux new soil thermal conductivity (Peters-Lidard ‘98) under snowpack (Lunardini, 1981) vegetation reduction of thermal cond. Surface Characterization maximum snow albedo database (Robinson ’85) dynamic thermal roughness length refinements Vegetation deeper rooting depth in forests canopy resistance refinements NARR- Eta-NOAH Upgrades

23. NOMADS Archive and Users Existing and Projected Volume * 5-YR retention of fcsts. Long term for anal. 06 07 09 09 10 2004 Month

24. NCDC Web Interface Three primary methods for data access: Web Interface GDS OPeNDAP ftp w/ on the fly Grib subsetting On-line or Off-line (archive) Server-side data computations.

25. NCDC Web Interface (cont.) The NCDC Web Interface originally developed at NCEP: NOMADS leverages efforts across the community. Promoting Model Collaborations

26. NOMADS “Web Plotter” NCDC NOMADS ingests 150K grids day. POR 2002 to present. Any one of these accessible in seconds Via:OpENDAP GDS ftp Web Plotter LAS (soon)

27. NCDC Reference Datasets NCDC reference And others datasets also available: CARDS (IGRA) GHCN NARR Ocean WAVE Accessible via OPeNDAP Promoting Model to Obs. Intercomparisons

31. Web Interface: click -plot -retrieve -analyze... DODS+Web: wget -O $RESULT http://nomads.ncdc.noaa.gov:9090/dods/gdas/gdas2003092118/.ascii? ulwrfsfc[1:1][140:140][200:200] http://nomads.ncdc.noaa.gov:9090/dods/gdas/gdas2003092118/.ascii? ulwrfsfc[1:1][140:140][200:200] The above W3C Web data request returns OLR at a specific lat/lon for a specific dtg. Time to receipt ~1 sec Volume: 200bytes Traditional acceess: 1) ftp the data: 2.4Mb 2) decode (degrib) and format the data 3) develop code to locate specific variable, earth located point, and time NOMADS Clients: open (url) (variable) (...) The power and ease of NOMADS scripts...... Using NOMADS

32. Outgoing Longwave Radiation Flux SFC Watts/m**2 Wget w/ DODS expression

33. * January Mean 500 Height (1981 to 1989) minus (1990 to 1998) * Mean & Standard Deviation for all 10 ensembles * Time required: 60 secs 'reinit' '!date' * baseURL = 'http://motherlode.ucar.edu:9090/dods/_expr_' * GKR 2/13/03 New NCAR URL baseURL = 'http://dataportal.ucar.edu:9191/dods/' expr = 'ave(z,t=387,t=483,12)-ave(z,t=495,t=591,12)' xdim = '0:360' ydim = '20:90' zdim = '500:500' tdim = '1nov1978:1nov1978' 'sdfopen 'baseURL'_expr_{C20C/C20C_A}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_B}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_C}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_D}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_E}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_F}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_G}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_H}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_I}{'expr'}{'xdim','ydim','zdim','tdim'}' 'sdfopen 'baseURL'_expr_{C20C/C20C_J}{'expr'}{'xdim','ydim','zdim','tdim'}' 'define resa = result.1' 'define resb = result.2' 'define resc = result.3' 'define resd = result.4' 'define rese = result.5' 'define resf = result.6' 'define resg = result.7' 'define resh = result.8' 'define resi = result.9' 'define resj = result.10' say 'got data' 'set lev 500' 'set lat 20 90' 'define mean = (resa + resb + resc + resd + rese + resf + resg + resh + resi + resj)/10' 'define d1 = (pow(resa-mean,2))' ; 'define d2 = (pow(resb-mean,2))' 'define d3 = (pow(resc-mean,2))' ; 'define d4 = (pow(resd-mean,2))' 'define d5 = (pow(rese-mean,2))' ; 'define d6 = (pow(resf-mean,2))' 'define d7 = (pow(resg-mean,2))' ; 'define d8 = (pow(resj-mean,2))' 'define d9 = (pow(resi-mean,2))' ; 'define d10 = (pow(resj-mean,2))' 'define stddev = pow((d1 + d2 + d3 + d4 + d5 + d6 + d7 + d8 + d9 + d10)/10,0.5)' 'set gxout shaded' 'set mproj nps' 'display mean' 'draw title January Mean 500 Height (1981 to 1989) minus (1990 to 1998)' 'set string 3 bc 1' 'draw string 5.5.5 Mean & Standard Deviation for all 10 ensembles: ‘C20C Climate of the 20th Century Folland/Kinter' *'cbarn' 'set gxout contour' 'set ccolor 0' 'display stddev' '!date' At left is the complete script for generating mean and sdev at 500mb analyzing 20 years of “Climate of the 20 th Century” from NCAR. Traditional vs. NOMADS method: Data volume transported:100Gb vs. 2Kb Time to access data:2 days vs. 60 sec Code development: days vs. minutes Fortran based LOC:1000 vs. 50 LOC

34. NOMADS Ensemble Probabilities on the fly NOMADS Ensemble Access URL is: http://nomad3.ncep.noaa.gov:9090/dods/enshires/archive/ens20040809/ensc0_http://nomad3.ncep.noaa.gov:9090/dods/enshires/archive/ens20040809/ensc0_ 00z_1x1.ascii?pratesfc[3:3][125:125][277:277] Enabling private sector access: An example No need for image generation of ensembles... OPeNDAP constraint expression

35. Providers: NCEP

36. Providers: FSL

37. Providers: GFDL

38. Providers: NASA GCMD

39. Providers: UCAR Unidata

40. GO-ESSP A super-set of the original NOMADS group of data managers has formed the Global Organization for Earth Systems Science Portals GO-ESSP http://esportal.gfdl.noaa.gov Unidata ESG (NCAR, LLNL) OPeNDAP COLA NOMADS (GFDL, PMEL, NCDC, NCEP, others) NASA/GCMD BADC, BODC WMO Advancing Collaborations

41. GO-ESSP The Global Organization for Earth System Science Portal (GO-ESSP) is a collaboration designed to build the infrastructure needed to create web portals to provide access to observed and simulated data within the climate and weather communities. The infrastructure created within GO-ESSP will provide a flexible framework that will allow interoperability between front-end and back- end software components. GO-ESSP is an international collaboration involving software developers from both Europe and the United States. Advancing Collaborations

42. Earth Observation Summit Affirmed need for timely, quality, long-term, global information as a basis for sound decision making. Recognized need to support: 1)Comprehensive, coordinated, and sustained Earth observation system or systems; 2)Coordinated effort to address capacity-building needs related to Earth observations; 3)Exchange of observations in a full and open manner with minimum time delay and minimum cost; and 4)Preparation of a 10-year Implementation Plan, building on existing systems and initiatives by European ministerial in late 2004 Established ad hoc Group on Earth Observations (GEO) to develop Plan Invited other governments to join. Group on Earth bservations Group on Earth bservations

43. IEOS Data Management Group on Earth bservations Group on Earth bservations In draft form-community input needed!

44. Operational Forecasting- Ensemble Predictions: flow-dependant prediction of weather and climate risk- nowcasting, medium range and seasonal. What’s Next? 1/2 Atmospheric and Oceanic Research- Scalar and Vector processing and Workstation models Model output statistics; data assimilation techniques Global Climate Change and Advanced Analysis- Clouds, initial conditions, true coupled simulations. Long term climate monitoring: in-situ analysis, trends, data homogeneity, extremes, downscaling, reducing uncertainty... On-demand Data Mining and Product Generation.

45. What’s Next 2/2 Data extraction into high volume data archives for the generation of products “on-demand”. Advanced data mining algorithms for pre-generation, or executed by (authorized) users also on-demand. Access to mined physical processes or signatures thru data mining. Search and location tools, portals, and long-term metadata management.

46. For NOMADS Program Information see: http://www.ncdc.noaa.gov/oa/climate/nomads/nomads.html For NOMADS Model Data Access: NOAA NCDC Main Page  Upper-Air  Data and Products http://nomads.ncdc.noaa.gov Or contact: Glenn.Rutledge @ noaa.gov Selected Publications on NOMADS and distributed data access: http://www.ncdc.noaa.gov/oa/model/publications/publications.html For more information... QUESTIONS ?