Distribution of Real-Time and Retrospective NCEP Model Data Sets RT-NOMADS Jordan C. Alpert Jun Wang Glenn Alpert 9/5/03 where the nation’s climate and weather services begin NOAA Operational Model Archive and Distribution System

NOMADS is an agreement between agencies who participate to have common … Data and observation distribution software, format independent and description methodology, Data and observation distribution software, format independent and description methodology, Documentation and organization framework, Documentation and organization framework, Forum to plan and organize Forum to plan and organize Funding avenue to direct $. Funding avenue to direct $. … for university, federal agencies and organizations, a mechanism to obtain support for the dissemination of their data sets.

Participants U.S.- NOAA, DOE, NASA, NCAR, Unidata, COLA, OPeNDAP Project, & Several Universities International- CEOS NERC ESG UK BADC BOM / Dept. of State via CAP

NOMADS and the NCEP Real Time Component Wider information dissemination through web pages and other avenues. Wider information dissemination through web pages and other avenues. Framework to create specialized and value added products needs access to weather data. Framework to create specialized and value added products needs access to weather data. Facilitate communication among commercial, public, research community and federal agencies. Facilitate communication among commercial, public, research community and federal agencies. NOMADS is a system for participants to interact with others using common protocols and formats. Existing differences in details are mollified by software. NOMADS is a system for participants to interact with others using common protocols and formats. Existing differences in details are mollified by software. A solution for Recommendation 5 of “Fair Weather: Effective Partnership in Weather and Climate Services”

RT-NOMADS at NCEP Distributed Ocean Data Server - OPEN_DAP (DODS) based on metadata file descriptors and XML. Distributed Ocean Data Server - OPEN_DAP (DODS) based on metadata file descriptors and XML. Web based ftp, the NCEP ftp2u client and “Great Displays” (pdisp), a grid analysis and display client. Web based ftp, the NCEP ftp2u client and “Great Displays” (pdisp), a grid analysis and display client.ftp2u Model initialization ingredients, “The minimum set” for model rerun/retrospective analysis ready observations & restart files. Model initialization ingredients, “The minimum set” for model rerun/retrospective analysis ready observations & restart files. The Real Time NCEP Operational model run history. The Real Time NCEP Operational model run history. Consists of

RT-NOMADS is a service for science, development and public access. To distribute NCEP’s Operational data set to researchers and the public. To provide server software to NOAA projects who need to serve their data through web services. NCEP NOMADS servers are the intermediary between NCEP operational computers and NCDC NOMADS archive servers. NCEP NOMADS servers are the intermediary between NCEP operational computers and NCDC NOMADS archive servers.

Tools for Users on the server Pare down large file sizes of high resolution initialization and model results. Pare down large file sizes of high resolution initialization and model results. (re-) Group different data sets to create needed products – such as initialization files for model development. (re-) Group different data sets to create needed products – such as initialization files for model development. Subset the data sets in parameter space Subset the data sets in parameter space Subset the data sets in physical space Subset the data sets in physical space (subset the grid in 4 dimensions) (subset the grid in 4 dimensions)

NCEP Real-Time NOMADS Server Real time. Real time. Operational Suite of Model products. Operational Suite of Model products. Forecast information for model comparison. Forecast information for model comparison. Operational initialization and observations. Operational initialization and observations. Maintenance of codes. Maintenance of codes. Utilities for data manipulation and grid transformation & sub-setting. Utilities for data manipulation and grid transformation & sub-setting. Documentation. Documentation. Serves as transfer point to NCDC for NOMADS archive of reruns and retrospectives. Serves as transfer point to NCDC for NOMADS archive of reruns and retrospectives.

The Server – Client distributed relationship: A Common Sense Approach to Data Distribution ftp2u, Great Displays, and GDS allow sub-setting of data by parameter, by 3D region, and by time. ftp2u, Great Displays, and GDS allow sub-setting of data by parameter, by 3D region, and by time. ftp2u Allows for results with low band width! Eg., NASA/GSFC takes 1 hour to ftp entire GFS (AVN) forecasts or 13 seconds to download their desired fields (ftp2u). Allows for results with low band width! Eg., NASA/GSFC takes 1 hour to ftp entire GFS (AVN) forecasts or 13 seconds to download their desired fields (ftp2u).ftp2u GDS: NOMADS participants serve up their data sets through a client-server relationship, that is, the data sets are internet ready and the display is done by the client. GDS: NOMADS participants serve up their data sets through a client-server relationship, that is, the data sets are internet ready and the display is done by the client. Examples of Clients :

Extending NOMADS to Real Time The A in NOMADS is for Archive. The A in NOMADS is for Archive. NCEP historical cases for rerun/retrospective case studies of ocean, atmosphere and climate. NCEP historical cases for rerun/retrospective case studies of ocean, atmosphere and climate. NCEP Real Time model output in the NOMADS framework … so there is a seamless access to these important data sets at NCEP and at NCDC! NCEP Real Time model output in the NOMADS framework … so there is a seamless access to these important data sets at NCEP and at NCDC! Use of benchmarks to characterize system loads in order to determine costs and give NOAA administration the information it needs to triage data resources. Use of benchmarks to characterize system loads in order to determine costs and give NOAA administration the information it needs to triage data resources. RT-NOMADS provides the framework for long term support to be available to keep up with modeling advances.

RT-NOMADS is a prototype Provide proof of the concept that distributed systems can serve data sets across agencies for modeling science, the university community and. Provide proof of the concept that distributed systems can serve data sets across agencies for modeling science, the university community and. Specify an Operational system for NOAA/TOC, Specify an Operational system for NOAA/TOC, Provide current state of the art operational capability for serving observations and data to Forecast Offices, Provide current state of the art operational capability for serving observations and data to Forecast Offices, Goals:

Data Application Protocol (DAP) using…

… GrADS client …. commands Sdfopen t = 1 'set gxout shaded' while (t <= 1068) 'set t 't 'set t 't 'q time' 'q time' date=subwrd(result,3) date=subwrd(result,3) 'draw title hgtprs 'date' 't 'draw title hgtprs 'date' 't 'd hgtprs' 'd hgtprs' '/home/wd23ja/grads/lib/cbarn.gs' '/home/wd23ja/grads/lib/cbarn.gs' 'swap' 'swap' t = t + 1 t = t + 1endwhile

Getting data without GrADS or a commercial client: 0][0:0][30:130] [290:290] #the order of the square bracked values is: # [time][level][lat][lon] # where lat is measured from the south pole (0) # to equator (90) to NP (180) (Use a “wget” and the URL with the ascii response in cron or a cgi-bin script provides the needed number(s) when and where needed.)

NOMADS clients at NCEP Real Time Operational model data sets from GFS, and ETA from present to one month ago. Real Time Operational model data sets from GFS, and ETA from present to one month ago. Attention to accumulation variable differences and high and low resolution. Attention to accumulation variable differences and high and low resolution. Analysis pgb, restart files. Analysis pgb, restart files. Recent (“real time”) climate runs, ie., RSM. Recent (“real time”) climate runs, ie., RSM. Observations (soon). Observations (soon). Reanalysis-2 (and 1) Reanalysis-2 (and 1) AMIP type runs (Suru’s finest runs eg., 28 and 64 levels ), sudden warming, …. AMIP type runs (Suru’s finest runs eg., 28 and 64 levels ), sudden warming, …. “Great Displays” (pdisp) and ftp2uftp2u by Ebisuzaki and modified by Wang

MATLAB: A Commerical Client server=‘nomad2:9090’; directory=‘/dods/gdas/rotating/’; fext=‘ascii’; model=‘gdas’; datestr=‘YYYYMMDDHH’; varname=‘hgt’; varlevs=‘1’;Arrc=[‘[‘,t1,’:’,t2,’][‘,nlev1,’:’,nlev2,’][‘,ilat1,’:’,ilat2,’][‘,ilon1,’:’,ilon2,’]’];urlstring=[ url = java.net.URL(urlstring); % connect to web page Try stream = openStream(url) stream = openStream(url) ireader = java.io.InputStreamReader(stream); ireader = java.io.InputStreamReader(stream); breader = java.io.BufferedReader(ireader); breader = java.io.BufferedReader(ireader); Line = readLine(breader); disp(line);

NCEP Data Sets NOAAPORT Operations. NOAAPORT Operations. Continuation of NCEP real time model component. Continuation of NCEP real time model component. NCEP model initialization ingredients, “The minimum set” for model rerun/retrospective analysis ready observations & restart files. NCEP model initialization ingredients, “The minimum set” for model rerun/retrospective analysis ready observations & restart files. Archives at NCDC

Model Data Requirements from CLASS Current Archive: NOAAPort:2TB/yr GDAS/SSI:400Gb/yr NARR10Tb Total12.5Tb FY04 NCEP Hi-Res:+10TB/yr* Total 22.5Tb * 5 year retention only HDSS 23TB/yr NOAAPort Gridded Output NCEP Hi-Res Eta/Global and NARR NCEP Global Assimilation Spectral Statistical GDAS O/I North American Regional Reanalysis Legend Green- Existing Archive Blue- Expected FY03/04 Rutledge and Alpert 6/03 FY03

Model Data Req’s - NWP and GCM’s GCM- Global Climate Model Reanal- N. Am. Regional Reanalysis NWP- Numerical Weather Prediction * 5-year rotating NWP archive Alpert & Rutledge 6/03

RT NOMADS Server at NCEP Two 1TB RAIDZONE servers Two 1TB RAIDZONE servers (backup) (backup) Documentation web page under construction Documentation web page under construction New disk space on order…. New disk space on order….

Non-Operational “Projects” can become NOMADS participants to display their data sets: Install scripts and clients, ftp2u and “Great Displays” (pdisp) for a GDS framework, Install scripts and clients, ftp2u and “Great Displays” (pdisp) for a GDS framework,ftp2u Setup GDS XML file – see example on web page, for example,… Setup GDS XML file – see example on web page, for example,… Including GDS administration with updates and new additions, Including GDS administration with updates and new additions, “newxml” update scripts – to automatically add new directories and files. “newxml” update scripts – to automatically add new directories and files. NOMADS gives guidance to:

(RT) NOAMDS data sets …. One data set (e.g, GRIB) provides the data for all access applications. One data set (e.g, GRIB) provides the data for all access applications. GraDS control files (grib2ctl) and index files (gribmap) create unique metadata description and random access. GraDS control files (grib2ctl) and index files (gribmap) create unique metadata description and random access. GDS can read HDF, NETCDF* as well as other formats. GDS can read HDF, NETCDF* as well as other formats.

NOMADS and the NCEP Real Time Component NWS should make its data and products available in Internet-accessible digital form. NWS should make its data and products available in Internet-accessible digital form. Information held in digital databases should be based on widely recognized standards, formats, and metadata descriptions to ensure that data from different observing platforms, databases and models can be integrated and used by all interested parties in the weather and climate enterprise. Information held in digital databases should be based on widely recognized standards, formats, and metadata descriptions to ensure that data from different observing platforms, databases and models can be integrated and used by all interested parties in the weather and climate enterprise. A solution for Recommendation 5 of “Fair Weather: Effective Partnership in Weather and Climate Services”