1. TIGGE Data Archive and Access System at NCAR 5th GIFS-TIGGE Working Group South African Weather Service Pretoria 11-13 March 2008 Steven Worley Doug Schuster Dave Stepaniak -- Others --

2. Topic Outline l Archive Summary l Provider Summary l Improved Data Handling at NCAR l NCAR’s portal and access l Next Steps - Future

3. Archive Summary l Online Data –Period, most recent three weeks –~ 3 TB, public products –~ 6 TB, data preparation, subsetting l Offline Data –Full period of record –~ 80 TB, NCAR Mass Storage System l Provide access to both

4. Provider Summary l Geographic and methods l Metrics

5. TIGGE Archive Centers and Data Providers Archive Centre Current Data Provider NCAR NCEP CMC UKMO ECMWF MeteoFrance JMA KMA CMA BoM CPTEC IDD/LDM HTTP FTP IDD/LDM Internet Data Distribution / Local Data Manager Commodity internet application to send and receive data

6. Summary of Data Providers, March ‘08

7. VariableLvLECWFUKMOJMANCEPCMACMCBOMMetFKMACPTC Geopotential ZPL Specific HPL T U-velocityPL V-velocityPL Potential VorPT Potential TPV U-velocityPV V-VelocityPV U 10mSL V 10mSL CAPESL Conv. Inhib.SL Land-seaSL Mean SLPSL Orog.SL Skin TSL Snow D. H20SL Snow F. H20SL TIGGE Archive Completeness, 02/2008 PL = Pressure Level, PT = 320K Theta Level, SL = Single/Surface Level

8. VariableLvLECWFUKMOJMANCEPCMACMCBOMMetFKMACPTC Soil Moist.SL Soil TSL Sunshine D.SL Surf. DPTSL Surf. ATmaxSL Surf. ATminSL Surf. ATSL Surf. PSL LW Rad. OutSL LH fluxSL Net RadSL Net Therm. RadSL Sensible Rad.SL Cloud CovSL Column WaterSL PrecipitationSL Wilt. PointSL Field Cap.SL TIGGE Archive Completeness, 02/2008 PL = Pressure Level, PT = 320K Theta Level, SL = Single/Surface Level

9. New Data Handling Hardware @ NCAR l Why ? –NCEP data processing n Operational (non-TIGGE) grids via network (59 parameters) n Create TIGGE compliant grids, mainly metadata, some re-computation (e.g. precip.) n FTP upload 11 parameters from NCDC - work in progress n Re-distribute to ECMWF and CMA –Data ingest from all providers, 240 GB/day n NCAR has maximum task, it produces nothing and receives all –New security model at NCAR n Forced by TeraGrid –networked based sharing of data and computational cycles l Cost –$120K

10. The Data Management Challenges Anomalies between centers l Native horizontal resolution l Number of ensemble members l Number of forecast initialization times (1, 2, 4x daily) l Forecast increment time period l Forecast length l Number of fields provided l Internal file compression (e.g. jpg) was not specified Influence on Research l User interface must inform users - commonalities constrain research

11. NCAR TIGGE Portal Address: http://tigge.ucar.edu http://tigge.ucar.edu Main Features –Registration and Login –Get Data –User Tools –Documentation and Linkages –Technical and Community Supported Help

12. User Interface Registration and Login –Required per international agreement n Users electronically accept conditions for usage –We capture metrics for n Name, email, organization name, organization type (univ., gov.,), and country n Who, what, when files were downloaded

13. User Interface Get Forecast Data l Two Selection Interfaces –File Granularity n Developed First –Parameter Granularity n Added later

14. Dates Center File Type Forecast Time Forecast Duration

15. Get Forecast Data NCAR online file archive Selection options Center(s) Date File type (sl, pl, etc) Initialization time Forecast length Download Options Point and click using browser, one file at a time Script to run on local machine User and password encrypted ‘wget’ commands background process to access all files User customized files Selection options Same as for files, plus Parameter Regridding Spatial subsets Formats, GRIB2, netCDF Delayed ModeReal Time Two User Interfaces

16. Tools l Collection of tools with growing maturity –Contributors n NCAR n ECMWF n NOAA n Unidata l New API, GRIB2 to netCDF –Developed as ECMWF/NCAR collaboration –Need to deploy in the NCAR system

17. Tools, example; NCAR NCL

18. User Help Two modes l Technical assistance directly from TIGGE staff at NCAR via email –Could originate from the portal l Open community website forum, including subscription email –Enrollees can post questions, give answers, and share ideas and experiences –Provided by Unidata

19. NCAR TIGGE archive usage 1.1 TB, 95 K files 1.1 TB, 95 K files 70 Unique data users 70 Unique data users User Profile A few users taking most volume 30 UU used parameter subsetting Most users take small amounts of data on single visit Not sufficient for relevant research service

20. TIGGE data usage l Want public access to grow –Web advertising –Helpful Idea - User Working Group meeting in conjunction 3rd THORPEX Science Symposium n THORPEX Exec. Com., 7th February Meeting. –Publication, possibly EOS/AGU –More presentations

21. Future @ NCAR, Phase I Completion l Finish new system installation l Verify and add NCEP parameters prepared at NCDC l Complete advanced subsetting features –Spatial, grid interpolation, and user selected output format (GRIB2 and NetCDF) l Continue data analysis tool development l Quasi-automatic user access to long-term TIGGE holdings from the NCAR MSS

22. Future @ NCAR, add LAMS l Focus on Americas and possibly one or two others l Adapt access system for non-global domains l Much to be discussed

23. Future @ NCAR, Phase II Participation has limiting dependencies –Finish Phase I –Demonstrated need from research and education communities –Requirements implied by –Requirements implied by “data..accessible via common web user interface”? n Single user portal => uniform web service protocol system- wide, interoperable across data, transport, and search n Track development, not contribute resources to R&D –Need funding path for all but very simple adaptations n WIS demonstrator project, WMO surplus funds? n THORPEX Exec. Com., 7th February Meeting.

24. Acknowledgements l International data providers, excellent collaborators –TIGGE compliant data easily entered into the TIGGE archive system –Thanks for all the hard work l International Archive Centers, excellent partners –Very prompt actions –Share much software/knowledge/experience –Demonstrated a new level of International data provision –NCAR has benefited l Users have been patient –This has been a Super-sized challenge l NCAR staff are performing outstandingly Thanks to SAWS for hosting the 5th GIFS-TIGGE WG

25. END Steven Worley worley@ucar.edu