1. Jordan C. Alpert/NCEP/EMC jordan.alpert@noaa.govjordan.alpert@noaa.gov Glenn Rutledge/NCDC Jun Wang/SAIC where the nation’s climate and weather services begin NOAA Operational Model Archive Distribution System NOMADS Toward an Operational service orientated architecture EMC 5 th GO-ESSP Earth Science Portal Workshop, June 20, 2006 at Lawrence Livermore National Laboratory NCEP/Environmental Modeling Center “National”

2. NOMADS Web Page Completely re-designed. Uses “php” Easier navigation Better Documentation NOMADS is renamed: NOAA National…

3. Vision “ NOMADS is a distributed service oriented architecture, a system-of- systems integration based on using loosely coupled connections among independent systems to create a scalable, extensible, interoperable, reliable, and secure framework.” … reliable? What is service oriented architecture?

4. What is NOMADS? (con’t) Real Time systems at NCEP and NCDC and archives at NCDC make for seamless past to present data availability. A digital archive of NOAA’s operational weather models, and an “innovative data access philosophy (read free) to promote interoperable access across the geosciences” (BAMS, Rutledge et. al., 2006). A Pilot project at NCEP that advances integration (read access) of real time Operational model data. NOMADS: An integrator of common web services infrastructure to support the discovery, access and transport of data (NOAA GEO-IDE Document) User/client applications developed at EMC, NCEP’s development division, and continued development at NCDC.

5. foster research within the geo-science communities (ocean, weather, and climate) to study multiple earth systems using collections of distributed data, promote model evaluation and product development, develop institutional partnerships via distributed open technologies. provide distributed access to models and associated data, NOMADS Goals The NOAA Operational Model Archive and Distribution System

6. Goals To take advantage of internet technology opportunities. Improve efficiency and reduce costs by bridging the barriers between existing, independent “stove pipe” systems. Integrate the data management activities of NOAA projects. Individual components retain responsibility and authority within the context of a systematic set of principles. Develop and adopt standards for metadata, data discovery and data transports, formats and protocols. Service oriented Goals (DMIT)

7. To provide Operational Services where security, timeliness (time critical) and reliability are paramount. Operational public access services for public access (includes university/research/scientist) to data, products and information services. Scientific services where efficient and flexible discovery and access to data sets is required. Commercial value-added services and user client applications. The Operational (reliability) problem

8. NOAA/NCEP Operational ftp services for model data (What is Operational today). Two operational centers (ftpprd… and tgftp…) Standard ftp servers Load balance, Fail over, State of the art hardware ($) and 24/7 support. A little difficult to navigate (e.g, file naming convention) Entire file (0.25GB) download necessary. Holdings not complete (0.5 degree GFS, NAM hourly, ensembles…?) Complete and “fast ftp” (Partial ftp transfers) for all data set holdings is imminent.

10. ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.gfs_CY.00/RD.20060615/PT.grid_DF.g r1 Example of tgftp location for GFS model (first entry) files from June 15: Excerpt from the ftpprd inventory description: Inventory of File gfs.t00z.pgrbanl Model: GFS Cycle: 00 UTC Forecast: 0 HRS Number of Records: 267 Grid Identification : 3 Number Level/Layer Parameter Description 0001 1000 ISBL HGT 1000 hPa isobaric level Geopotential height gpm] 0002 975 ISBL HGT 975 hPa isobaric level Geopotential height [gpm] 0003 950 ISBL HGT 950 hPa isobaric level Geopotential height [gpm] An Excerpt from the list of 500 files, note name and size: File: fh.0030_tl.press_gr.onedeg 26562 KB 06/15/2006 File: fh.0030_tl.sflux 42981 KB 06/15/2006 What the user sees at the operational ftp site:

11. Toward Operational services We know how to do the hardware part. $ = load balance, fail over, 24/7, redundancy What about users usage of the applications, e.g., DAP and GDS and others Operationalize data flow to NCDC from NCEP. GRIB2 with jpeg packing increases CPU resources needed. Software to handle sub-seting and GRIB2 GDS, OPeNDAP, and other applications need GRIB2 software strategy. Implement with partial http transfers and index files (first step).

12. User Statistics Highest month: 4.3Tb w/3.8 million downloads – NCEP has ~1-5Tb /month NCDC Only NCDC Only BAMS Paper

13. Real Time (NCEP) nomad3 server GDS/OPeN(DAP) DODS: typically 250,000 queries/day from the GDS log (already /2). pdisp (“Great Displays”) 2694/day. http access_log ~200,000 hits/day ftp2u (fast ftp index download) over 5/11 – 6/15 8,623* repackaged GRIB downloads/day. Need better ways of evaluating use for good health and for comparison of servers.

14. Real Time (NCEP) NOMADS Server Users query files that are not present in tight loops. During the 12 hours [00Z-12Z] on June 1, of the 371 different users who queried the httpd server error_log, the top 7 users with the most queries are listed: #queries User 1579 hawaii.edu 1616 abo.wanadoo.fr 1870 natpool.mwn.de 3293 accuweather.com 4039 saildocs.com 7864 hmg.inpg.fr 100465 labsolar.ufsc.br A solution: Check log file (error_log) user access rate. If the rate criteria (6/minute) is exceeded then place user’s IP address in IP tables to block access for that IP address for a period of time (10 minutes). Placing users in a “penalty box” when they are repeatedly accessing the server in tight loops is the same as users placing a “sleep 600” command or wait 10 minutes in their unix script for loop. This represents a throttle that can be applied to other situations such as many GDS queries when one is needed.

15. NCEP Dual Ingest & QC Other Portals GrADS Data Server Live Access Server GrADS, Ferret, MatLab, IDL, IDV, Web browsers or any OPeNDAP enabled client THREDDS Data Server Project ACCESS: NASA, GMU, NOAA (NCDC), GMU & OPeNDAP.org NCDC Archive CEOS-Grid Web Browse & Compute: Sister servers Exploratory Grid Projects w/ Globus The NOMADS System Design Multiple paths to format independent data access

16. Search, Discovery, Access, and Analysis A metadata server, DIMES (Distributed Metadata Server) integrated with GDS to form MIDAS (Kafatos, Yang, Zhao) : Metadata Integrated Data Analysis Server Content consistency between data server and metadata server. Interactive metadata search and data access and analysis. Potential for more effective and efficient searching. Potential for better interfaces. More seamless transition from data search to data access. Uses XML to represent, store, retrieve and interoperate metadata with minimum semantic enforcement. Contains a metadata model, XML query engine, web-based prototype interface. Enables the application power of Grads and GDS server side data analysis.

17. One can search by Time resolution (forecast time), Spatial resolution, Search_Space (region), Search_Time (Cycle), and Text (by variable and other Title text in the Metadata file).

19. After selection of the Time and Space Box, the user can generate an “sdfopen” URL command including functionality, e.g, time or space averaging, for further processing or generate the DODS URL and run GrADS client script and interactive commands.

20. Conclusions and Future Plans Incorporate NOMADS functionality into CLASS (Comprehensive Large Array Stewardship system) Enhance the NCEP real time servers at NCDC and place in NCDC operations. Operationalize the data flow from NCEP to NCDC. Existing Operational NOAA servers can be enhanced with NOMADS service orientated architecture and accepted by operational environment administrators. Support working prototypes for proof of concept and continued development toward improved services.