1. Growing and Future Datasets in the SCD Research Data Archives for NSF SCD Review Panel 16 October 2001 Steven Worley Scientific Computing Division Data Support Section

2. Outline Introduction Extant Growing Archives Data Archive Assistance for the Research Community Future Archives

3. Introduction Components of a good archive –Maintained in a reliable system (MSS) –Clear and concise information interface Complete discovery metadata –Convenient data access for many users Local computing platforms Transfer to remote computing platforms –Consultants for assistance Guidance to the best products, sometimes within multi-product complex collections

4. Components, continued –Underlying archive with rich content Many historical reference datasets - have 100’s of these, but not discussed here Relevant new and frequently updated datasets Focus today: Growing and Future datasets

5. Global Observations P.O.R# YrsIncep. DateComments Rawinsondes1946-on551967Upper Air Pibals1942-on591973 Upper Air, wind Aircraft1947-on521973 USAF and Commer. Sat. cloud wind drift1967-on341973 GOES and GTS Satellite Soundings1969-92251973 TOVS + irradiance Surface Synoptic1948-on531975 some much older Ocean Surface1794-on2031981COADS Usages: Input for global atmospheric reanalysis Basic long term climate assessment and case studies

6. Operational and Composite Analyses Special analyses were discontinued when global operational analyses became very good Daily SLP is a small but very popular dataset, e.g. NAO evaluations

7. Highlights up to date, 1985 – June 2001 different temporal resolutions, 6 hr to 1 mn different spatial resolutions, ~ 1 degree to 2.5 degree many atmospheric levels and variables Details and Drawbacks Distribution Restriction; U.S. non-profits and UCAR members only. Cost, increasing and unpredictable $11K in 1999, $16K in 2000, $19K in 2001 We get only modest resolution (T106, N80), T319 and N256 are available – again cost is an issue.

8. Highlights Very current, FNL 1.0 is done daily High resolution N. America, ETA at 40km No cost or distrib. restrictions from NCEP - GREAT

9. Reanalyses P.O.R# YrsIncep. Date NCEP/NCAR Reanalysis I1948-06/2001531994 ECMWF ERA-151979-1993151994 NCEP Reanalysis II1979-06/2001221998 Notes: ERA-15 is terminal, ERA-40 is under development now NCEP II, experimental run (testing new data and schemes)

10. Outstanding Features Three different coordinate surfaces Very long analysis Unrestricted distribution CD-ROMS are very popular

11. Countries Receiving Reanalysis CD-ROMs Highlights Over 8900 CD-ROMs 1997-09/2001 Top 13; U.S. 46%, Japan 11%, (Canada, UK) @ 4%, (Germany, India) @ 3%, (Australia, S.Korea, Spain, Mexico, Norway, Russia, France) @ 2%

12. Other Users, Jan.-Sep. 2001 35 Received CD-ROMs 80 Custom data orders (FTP, tape) 906 Data downloads from the online server 406 taking more than one file (66 GB) 678 Served

13. Data Archive Assistance for the Research Community Who is the Research Community –International –University –Other UCAR Research Programs Why? –Helps others achieve goals –Provides additional resources at NCAR –Can lead to future opportunities

14. International collaboration GCIP Model Data Center High res. atmos. models focused on energy and hydrology cycles – many surface boundary layer data. GCIP: GEWEX Continental-Scale International Project / GEWEX : Global Energy and Water Cycle Exper. Critical data for N. American mesoscale studies Complete archive is approx. 1 Terabyte JOSS/UCAR has many of the GCIP observations Eta –NCEP3 hr40 km 25 lvs 5/1995 – 7/2001 MAPS – FSL NOAA 3 hr40 km 5 lvs 8/1996 - 7/2001 GEM – Canadian 6 hr41 km 28 lvs 4/1997 – 6/2001

15. University collaboration MICOM; Miami Isopynic Coordinate Ocean Model, 1/12 th degree 70N to 28 S, 16-20 layers COADS Clim. Forcing 6 yrs305 Gigabytes ECMWF Clim. Forcing 2 yrs164 Gigabytes ECMWF Daily Forcing 5 yrs415 Gigabytes (1979-1983) U. Miami, Ocean Model Data Why? Needed help getting the data to users How? –Web order interface –Automatic processes to stage the data from the MSS and create subsets –Data then staged for FTP pickup 6-yr Mean T at 5 meters

16. UCAR research collaboration GTS: GPS Science and Technology Program, leveraging GPS satellites for science SuomiNet Data from GST GPS satellite signals at receivers Estimate integrated water vapor Total electron count (strato.) Why? The GST project is focused on real-time data capture and provision We want to preserve the archive for long term studies in the future Receiver Sites How? GST staff stage data to the MSS SCD staff perform archive maintenance and access

17. UCAR research collaboration Support real-time data services Unidata’s (UCAR) role –Runs full capacity IDD/LDM application –Serves ± 20 universities downstream SCD’s role –24 hr x 7 day operation monitoring –Routine system back up –Data archive backup and maintenance Covers one year – done daily 2.3 GB per day to the MSS Observations, NCEP model data, wind profilers

18. Future Data Collection Rescue of African Data (NOAA/FSU) –Upper air and surface data –Digitize and save magnetic tapes –Why? Improve coverage Rescue of Russian Ocean Observations (NSF) –Digitize six million marine surface data, 1937-1993 –“Cold War” archive – global coverage, 60’s and 70’s Map reanalysis model QC back onto original data –Merge many sources of observations together Many others –E.g. more global river flow data Observations;

19. Future Data Collection ECMWF’s ERA-40 –1957-2001 –SCD has provided observational datasets –Three time periods computed at once, completion approx. late 2002. –T159 (~82km), 60 levels, and 6 hr resolutions –15 TB of data –Cost = $100K, and restricted distribution Reanalyses;

20. Future Data Collection NCEP N. America Mesoscale –Probably, 30 km, 25-30 levels –Starting late 2001, maybe later – Based on the ETA model Next U.S. Global Reanalysis –Based on NCEP experiments and ERA40 results –Might be under a combined NASA and NOAA project Reanalyses; Generally: Stay informed about science activities, sometimes through participation, and then collect new data sources as they emerge.

21. Key Summary Points Update many archives to support research –Observational collections –Operational Analyses –Reanalyses Collaborations to promote great science –International –University –NCAR/UCAR Have plans to collect emerging new data resources