1. Data to Support Ocean-Atmosphere Research NCAR Research Data Archive (RDA), http://www.dss.ucar.edu Zaihua Ji, NCAR Steven Worley, NCAR Scott Woodruff, NOAA/ESRL Fourth International Ocean-Atmosphere Conference Fourth International Ocean-Atmosphere Conference July 4 - 6, 2007, Qingdao, China

2. 2 Presentation Outline Introduction Introduction Overview Overview Focus, Data Relevant to Ocean- atmosphere Research Focus, Data Relevant to Ocean- atmosphere Research RDA Data Archive and Management RDA Data Archive and Management RDA Data Usage Statistics RDA Data Usage Statistics Explore Data Archived in RDA Explore Data Archived in RDA Future Developments Future Developments Summary Summary

3. 3 Introduction Definition, Research Data Archive (RDA) Dataset collection to support atmospheric and related geosciences research Hosted by the Computational and Information Systems Laboratory (CISL) at National Center For Atmospheric Research (NCAR) Built with data curation and stewardship over more than 40 years Presently maintained by eight people, educated in meteorology and oceanography Basic Scale metrics + 500 datasets + 500 datasets + 800K files + 800K files + 120 TB + 120 TB

4. 4 Introduction Why RDA Access is Unrestricted Our user community is best served if no restrictions exists Our user community is best served if no restrictions exists Research needs a very large array of data types Research needs a very large array of data types Dataset development requires experts - no one can do it all Dataset development requires experts - no one can do it all Many studies require global data - everyone can make a contribution and everyone should benefit. Many studies require global data - everyone can make a contribution and everyone should benefit. Data Management alone has significant associated costs (people, storage, servers, etc) Data Management alone has significant associated costs (people, storage, servers, etc) There are near zero funds to purchase data There are near zero funds to purchase data If service systems are well designed many people can be economically served If service systems are well designed many people can be economically served Foremost Foremost We know data sharing works well and we want to be a “best practices” example in the international research community We know data sharing works well and we want to be a “best practices” example in the international research community We have had successful data collaborations and exchanges with China Focused on building and improving ICOADS

5. 5 Overview - Data Relevant to Ocean-Atmosphere Research Marine observations International Comprehensive Ocean-Atmosphere Data Set (ICOADS) International Comprehensive Ocean-Atmosphere Data Set (ICOADS) Marine surface observations from ships, moored and drifting buoys, and fixed platforms Marine surface observations from ships, moored and drifting buoys, and fixed platforms Very long time series, 1794-2005, over 200 million obs. Very long time series, 1794-2005, over 200 million obs. Merging and QA/QC of many different datasets Merging and QA/QC of many different datasets 20 + year collaborative project between NOAA and NCAR, now receiving more international collaboration 20 + year collaborative project between NOAA and NCAR, now receiving more international collaboration More assistance is welcomed More assistance is welcomed RDA provides two product lines RDA provides two product lines Observations Observations Monthly summary statistics in 1x1 and 2x2 lat. lon. boxes Monthly summary statistics in 1x1 and 2x2 lat. lon. boxes ICOADS is a source for higher level analyzed products, e.g. ICOADS is a source for higher level analyzed products, e.g. SST analyses SST analyses Global atmospheric reanalyses Global atmospheric reanalyses

6. 6 Overview - Data Relevant to Ocean-Atmosphere Research Example of ICOADS and products SST: Dec 1942 ICOADS and two analyses HadISST from UKMO ERSST from NOAA ICOADS ERSSTHadISST

7. 7 Overview - Data Relevant to Ocean-Atmosphere Research ICOADS monthly summary statistics 10 statistics, e.g. median, mean, nobs… 23 observed and derived parameters no bias corrections All source observations are available in a similar way - derive your own statistics

8. 8 Overview - Data Relevant to Ocean-Atmosphere Research WOD Approximately, 1900 and forward Approximately, 1900 and forward Ocean profile observations from many international sources Ocean profile observations from many international sources Many types (XBT, CTD, ARGO, bottle stations, etc) Many types (XBT, CTD, ARGO, bottle stations, etc) Quality checked and on observed and standard depth levels Quality checked and on observed and standard depth levels World Ocean Database (WOD) and World Ocean Atlas (WOA) Prepared by Ocean Climate Laboratory NODC, Sydney Levitus and collaborators First edition 1982, now at 2005 Worlds largest collection Profiling float data added WOD01  WOD05

9. 9 Overview - Data Relevant to Ocean-Atmosphere Research WOA analyzed data on standard levels, 1 degree resolution analyzed data on standard levels, 1 degree resolution climatology, seasonal, monthly products (T, S, O2, nutrients, etc..) climatology, seasonal, monthly products (T, S, O2, nutrients, etc..) Some special products Some special products annual heat content annual heat content mix-layer depth mix-layer depth high resolution (.25 degree) T and S climatology high resolution (.25 degree) T and S climatology

10. 10 Overview - Data Relevant to Ocean-Atmosphere Research Marine Analyses - SST NOAA Extended Reconstructed SST Analysis – 1854 - ongoing, monthly, 2°x 2°, updated monthly – ICOADS and real-time obs. Hadley Centre HadISST –1850 - ongoing, monthly, 1°x 1°, updated monthly – ICOADS, real-time obs., AVHRR sat. blend for modern period NOAA Optimum Interpolation Daily SST Analysis –1/4 –1/4 °x1/4 °, 1985 - ongoing, two products, updated monthly n ICOADS, real-time obs, AVHRR satellite n ICOADS, real-time obs., AVHRR and AMSR satellite NOAA Optimum Interpolation SST Analysis – 1981- ongoing, weekly, monthly, 1°x 1°, updated monthly – ICOADS, real-time obs., AVHRR sat. blend n (May ‘07 in background)

11. 11 Overview - Data Relevant to Ocean-Atmosphere Research Marine Analyses - Wind Global QSCAT/NCEP Blended Analysis Created by Colorado Research Associates 0.5° x 0.5 °, 4x daily, 1999 - ongoing Vector wind components, and curl New versions are prepared when QSCAT algorithms are improved. Currently, a new version is under development. Many other good sources from Atmospheric Reanalysis datasets To be presented later

12. 12 Overview - Data Relevant to Ocean-Atmosphere Research Marine Analyses - Waves Global and North Atlantic Atlas of Ocean Waves Created by Sergey Gulev 2° x 2 ° and 5° x 5°, monthly, 1958 - 1997 Significant wave height Derived from ICOADS visual wave observations ERA-40 1.5° x 1.5 °, 4x daily, 1957 - 2002 2-D wave spectra are available for 12 directions and 25 frequencies

13. 13 Overview - Data Relevant to Ocean-Atmosphere Research Marine Analyses - Fluxes Global Objectively Analyzed Air-Sea Fluxes (OAFlux) Created by Yu and Weller 1° x 1°, daily and monthly Latent and sensible flux, 1981-2002, net short and long wave radiation 1983-2002 Derived from in situ observations, satellite data, and NWP and reanalysis output Update is planned for 2003 and onward Ocean-atmosphere heat, momentum and freshwater flux Created by Josey et al. 1° x 1°, climatology Derived from in situ observations and adjusted to match ocean heat transport estimates

14. 14 Overview - Data Relevant to Ocean-Atmosphere Research Marine Analyses - Atmospheric Reanalyses Sources Many useful surface products at different resolutions. Too many to cover here Main dataset title and temporal coverage NCEP/NCAR Reanalysis, 1948 - ongoing NCEP/DOE Reanalysis, 1979 - ongoing ERA-40, 1957-2002 JRA-25, forthcoming from JMA, 1979 - ongoing MERRA, forthcoming from NASA, 1979 - ongoing Important note about ERA-40 By agreement RDA can distribute only in U.S., and to a few universities in Canada. Copy outside this domain is prohibited

15. 15 Overview - Data Relevant to Ocean-Atmosphere Research Marine Analyses - Ocean Reanalyses NCEP Global Ocean Data Assimilation System (GODAS) 1° x 1°, pentad and monthly, 1980 - ongoing 1/3 ° resolution ± 10° of equator 40 vertical levels, parameters (T, S, ocean current, etc) GODAS Annual mix layer depth

16. 16 Overview - Data Relevant to Ocean-Atmosphere Research NWP Operational Output NCEP Global Final Analysis (FNL) 1° x 1°, 4x daily, 1999 - ongoing Updated monthly ECMWF Global Advanced Operational Surface Analysis 1.125 ° x 1.125°, 4x daily, 1985 - ongoing Updated bi-annually TIGGE (THORPEX Interactive Global Grand Ensemble) Analyses and forecasts from international NWP centers Now CMA, ECMWF, NCEP, UKMO and JMA; will add 5 more centers native model resolutions (approx. 1°), 1x to 4x daily, all ensemble members Parameters; 10m winds, temperatures, surface radiation and fluxes, etc, Update, near real time, 2-weeks data online

17. 17 RDA Data Archive and Management l RDA Data Archive Components l Creating Dataset Metadata l Populating Datasets with Data l Updating Datasets l Technology of data archiving l Accumulation of data archived in RDA

18. 18 RDA Data Archive Components

19. 19 Creating Dataset Metadata

20. 20 Populating Dataset with Data

21. 21 Updating Datasets

22. 22 Archiving Data with State of Art Technology l Hybrid archiving system with Database Management System (DBMS), Database for short, and file management system l Comprehensive archiving utility: gathering data, archiving to MSS and/or online service, recording file metadata, and publishing data filelists l Automated and complete for periodic data updates: downloading, processing/validating, and archiving data l All archiving, web and DBMS utilities are based upon open source software l Both onsite and offsite backup of whole RDA

23. 23 Data Archived In RDA (MSS: 121.5 TB, Online: 14.9 TB; By 2007-05)

24. 24 RDA Data Usage Statistics l How data are accessed l What data are accessed l Who uses the RDA data

25. 25 How Data Are Accessed (Downloaded between 2006-06-01 & 2007-05-31)

26. 26 How Data Are Accessed (Between 2006-06-01 & 2007-05-31)

27. 27 Top 10 Datasets Accessed by Most Users (between 2006-06-01 & 2007-05-31)

28. 28 Top 10 Datasets by Download Volume (between 2006-06-01 & 2007-05-31)

29. 29 Who uses RDA Data (Accessed Between 2006-06-01 & 2007-05-31)

30. 30 Top 20 Countries/Areas by Most Users (Online data Between 2006-06-01 & 2007-05-31)

31. 31 Top 10 Datasets by Most Users From P.R.China, Taiwan & Hong Kong (Online Data Between 2006-06-01 & 2007-05-31)

32. 32 Monthly Online Users From P.R.China, Taiwan & Hong Kong (Between 2005-01 & 2007-05)

33. 33 Monthly Online Data by Users From P.R.China, Taiwan & Hong Kong (Downloaded Between 2005-01 & 2007-05)

34. 34 Explore Data Archived in RDA l Registration l Identify dataset l Get the data –Online data –MSS Filelist –Subsetting –Questions and special requests

35. 35 Register Yourself on RDA Server

36. 36 Identify Datasets You Want

37. 37 How to Access Data For A Specified Dataset

38. 38 How to Access Data For A Specified Dataset

39. 39 Access Online Data … …

40. 40 MSS File List … …

41. 41 Data Subsetting … …

42. 42 Future Developments l New data search tool based on standardized metadata, GCMD l Browse tool to identify the files needed, based on DB and file metadata queries l Expanded RDA content, e.g. JRA-25, more ocean reanalysis l User feedback and information sharing at the dataset level using web forum technology l More user subsetting and format conversion l More open archiving and access service for NSF projects –Need specific inclusion and management policies l Modernization of web page look and function

43. 43 Summary l NCAR’s RDA is an open international data resource l We encourage and facilitate data exchanges l We continually build and improved access to the RDA l Modern DB systems are used for management and public data service l Modern data management methods (DB’s and automation) allow: – –Systematic growth – –Systematic and secure archiving – –Accurate usage assessment metrics l We are always open to suggestions for new data resources and service improvements from the users

44. 44 Thank You Questions?Contact: l http://dss.ucar.edu/index.html http://dss.ucar.edu/index.html l Zaihua Ji, zji@ucar.edu