1. The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin, NOAA/PMEL and the NOAA UAF team

2. NOAA UAF-grid 2 Enterprise-wide integration of data is a very hard problem!

3. NOAA UAF-grid 3 Heterogeneous requirements Data Providers with different outlooks (different "corporate cultures”) Data Users with diverse needs and skills Corporate cultures and people’s habits are notoriously difficult to change … and by the way, getting funds for integration is often a hard sell

4. NOAA UAF-grid 4 Integrating NOAA’s data is such a challenge weather forecast (time critical) fisheries management (regulatory aspects) nautical charting climate, ocean, atmosphere monitoring … modeling & observations research & operations

5. NOAA UAF-grid 5 The accepted approach: build a “system of systems” Connect existing data systems using a Service Oriented Architecture (“wrap” existing systems with loosely coupled, standardized services)

6. NOAA UAF-grid 6 Global Earth Observation Integrated Data Environment GEO-IDE Con-ops outlines such a plan 2005-06, 66 pages

7. NOAA UAF-grid 7 Seed funding ٨ available this year How do you approach a very big problem with a very small amount of money? finally

8. NOAA UAF-grid 8 1.Define requirements 2.Create enterprise architecture documents 3.Assemble volunteer teams to implement NOT ! NOT !

9. NOAA UAF-grid 9 An alternative approach Don't Solve Problems -- Copy Success "Switch: How to Change Things When Change Is Hard", Chip and Dan Heath (psychologists) http://www.fastcompany.com/magazine/142/switch-how-to- change-things-when-change-is-hard.html http://www.fastcompany.com/magazine/142/switch-how-to- change-things-when-change-is-hard.html

10. Feb. '10 - Ocean Sciences NOAA UAF-grid 10 Projects: (too many to name) Data formats: netCDFGRIBHDF Service stack: netCDF-CF-DAP-THREDDS-WMS Applications: MatlabArcGISFerret GrADS Google Earth IDV LAS ERDDAP… Users: (too many to name) …

11. NOAA UAF-grid 11 UAF (*) -grid (initial focus: gridded data) –netCDF –CF –OPeNDAP –THREDDS –WMS (& WCS) (*) Unified Access Framework -----an abstract data model -----metadata conventions --network transport --network transport --network topology --network topology --rendering --rendering

12. NOAA UAF-grid 12 Who is using this approach? Climate, ocean, atmosphere modelers –IPCC, GFDL, NCAR, … Satellite programs –GHRSST, PathFinder, CoastWatch, … NCEP weather and ocean forecasts (GRIB via NOMADS) Coastal (“HF”) radar A growing list of observations programs –Argo, OceanSites, SAMOS&GOSUD (soon), …

13. NOAA UAF-grid 13 Some rough spots need better compliance with CF conventions "files" need to be aggregated into logical datasets need richer discovery-level metadata

14. NOAA UAF-grid 14 NcML a tool for fixing compatibility problems e.g. adding a CF “standard_name” attribute to GRIB data The file, itself, is untouched. But access to the file through the services is made to conform to standards.

15. NOAA UAF-grid 15 more NcML e.g. aggregating 3 years of data into 1 </aggregation> The files are untouched. But the services see a single logical file.

16. NOAA UAF-grid 16 How users will find UAF datasets Less mature … UAF web crawler extracts THREDDS metadata –Ingests into a dbase –other metadata also ingested as available ISO 19115 records generated Datasets and services registered –GCMD, Geospatial One-Stop, GEOSS, data.gov Much metadata work will remain beyond year 1.

17. NOAA UAF-grid 17 The emerging UAF The UAF THREDDS catalog: ~400 “clean” datasets.. growing as the team aggregates and configures using ncML –team members become coaches  lasting solutions must be implemented at the data provider sites The UAF Team –8-10 active participants –self-selected 5 months into the project …

18. NOAA UAF-grid 18 The emerging UAF Users access data through their client applications –Without downloading files Configuration of clients is underway Some examples … Take differences Scripts Live Access Server (LAS)

19. NOAA UAF-grid 19 Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32 Desktop access in Matlab

20. NOAA UAF-grid 20 Desktop access in Ferret

21. NOAA UAF-grid 21 Desktop access in GrADS

22. NOAA UAF-grid 22 Desktop access in IDV

23. NOAA UAF-grid 23 Access in ArcGIS using the Environmental Data Connector (EDC)

24. NOAA UAF-grid 24 Access in Google Earth

25. NOAA UAF-grid 25 WMS provides access to many GIS applications

26. NOAA UAF-grid 26 Browser access using ERDDAP. And from there …

27. NOAA UAF-grid 27.mat REST URL access to data subsets in several simple formats (accessible through home-grown scripting of many types).kml.mat.nc

28. NOAA UAF-grid 28 Next year’s target: observational data Collections of time series & profiles are already on-line UAF home page … coming soon

29. NOAA UAF-grid 29 “No standard should be approved without having been used to implement a few projects of realistic complexity.” The Rise and Fall of CORBA (Henning, 2006) The UAF strategy builds on historical lessons

30. NOAA UAF-grid 30 Thank you