1. The Unified Access Framework (UAF) Building NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) one step at a time Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), Lewis McCulloch (HQ/TPIO) and the NOAA UAF team Kenneth.Casey@noaa.gov Julie.Bosch@noaa.gov Tina.Chang@noaa.gov Scott.Cross@noaa.gov Roy.Mendelssohn@noaa.gov Steven.C.Hankin@noaa.gov Jeremy.Malczyk@noaa.gov Jordan.Alpert@noaa.gov Jim.Sargent@noaa.gov Jeff.deLaBeaujardiere@noaa.gov Ted.Habermann@noaa.gov John.Relph@noaa.gov Bob.Simons@noaa.gov David.Neufeld@noaa.gov Upendra.Dadi@noaa.gov rsignell@usgs.gov Phil.Cogbill@noaa.gov Glenn.Rutledge@noaa.gov Mike.Grogan@noaa.gov Jeff.Budai@noaa.gov

2. Dec. 20102 Enterprise-wide integration of data is a very hard problem! (i.e. adopting shared approaches to achieve data interoperability) NOAA/UAF-grid

3. Dec. 2010 3 NOAA/UAF-grid Why? Different fields have different concepts of ‘data’ … and have developed solutions that make sense to them.  Getting people (and organizations) to change habits is difficult!

4. Dec. 2010 4 NOAA-world NOAA/UAF-grid weather forecast (time critical) fisheries management (regulatory concerns) nautical charting climate, ocean, atmosphere research … the list goes on … … the list goes on … NOAA management has been reluctant to invest funds in Agency-wide data integration

5. Dec. 20105 The accepted approach: build a “system of systems” “Wrap” existing systems with loosely coupled, standardized services  a Service Oriented Architecture NOAA/UAF-grid

6. Dec. 20106 Global Earth Observation Integrated Data Environment GEO-IDE Con-ops outlines such a plan 2005-06, 66 pages (by “DMIT”) NOAA/UAF-grid

7. Dec. 20107 Seed funding ٨ available last year Question: How do you approach a very big problem with a very small amount of money? finally NOAA/UAF-grid

8. Dec. 20108 1.Generate use cases 2.Define requirements 3.Write a Concept of Operations 4. … and an Implementation Plan 5.Assemble volunteer teams to implement NOT ! NOT ! NOAA/UAF-grid Tried and true approach …

9. Dec. 20109 An alternative (‘agile’) approach Don't Solve Problems -- Copy Success "Switch: How to Change Things When Change Is Hard", Chip and Dan Heath (psychologists), 2010 NOAA/UAF-grid

10. What “success” did UAF chose to copy? Year 1 focused on gridded datasets. Dec. 201010 Projects: (too many to name) Data formats: netCDFGRIBHDF Service stack: netCDF-CF-DAP-THREDDS-WMS Applications: MatlabArcGISFerret GrADS Google Earth IDV LAS ERDDAP… Users: (too many to name) …

11. Dec. 201011 UAF (*) -grid (initial focus: gridded data) –netCDF –CF –OPeNDAP –THREDDS –WMS & WCS (*) Unified Access Framework --file format/API/data model --file format/API/data model --metadata conventions --metadata conventions --network transport --network transport --network topology --network topology -- OGC rendering and transport -- OGC rendering and transport NOAA/UAF-grid

12. THREDDS “network topology” a tree defined as distributed XML Dec. 2010 NOAA/UAF-grid 12 NOMADS GEO-IDE/UAF NOAANOAA Affiliated NMFSOARNWSNESDIS NODC NGDC GFDL PMEL AOML OCO PFEG NDBC ESRL Coastwatch IOOS National Partners IOOS Regional Partners NAVO AOOS NANOOS CENCOOS SCCOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS

13. Dec. 201013 Who is using this approach? Modelers –IPCC, GFDL, NCAR, … Satellite programs –GHRSST, PathFinder, CoastWatch, … NCEP weather and ocean forecasts –GRIB files served via NOMADS Coastal (“HF”) radar A growing list of observations programs –Argo, OceanSites, … Pending adoption by OGC NOAA/UAF-grid

14. Dec. 201014 Some rough spots compliance with CF conventions is inconsistent individual files often not aggregated into logical datasets metadata are often in need of enhancement NOAA/UAF-grid

15. Dec. 201015 … but excellent repair tools exist: NcML and “virtual files” e.g. Improve CF compliance by adding “standard_name” attribute to GRIB data The file, itself, is untouched. But access to the file through the services is made to conform to standards. NOAA/UAF-grid

16. Dec. 201016 NcML for aggregation e.g. Aggregate three 1-year files of the same (say) model run </aggregation> The files are untouched, but services see a single logical file. NOAA/UAF-grid

17. Dec. 201017 The emerging UAF The UAF THREDDS catalog: ~500 “clean” datasets.. growing as the team aggregates and configures using ncML –team members become coaches  lasting solutions must be implemented at the data provider sites The UAF Team –8-10 active participants –self-selected (volunteers) 1 year into the project … NOAA/UAF-grid

18. Dec. 201018 How to reach users? Through their preferred tools (without downloading files) NOAA/UAF-grid

19. Dec. 201019 Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32 Desktop access in Matlab NOAA/UAF-grid

20. Dec. 201020 Access in ArcGIS using the Environmental Data Connector (EDC) NOAA/UAF-grid

21. Dec. 201021 Desktop access in Ferret NOAA/UAF-grid

22. Dec. 201022 Desktop access in GrADS NOAA/UAF-grid

23. Dec. 201023 Desktop access in IDV NOAA/UAF-grid

24. Web browser examples … Dec. 201024 Take differences Google Earth Scripts Live Access Server (LAS) NOAA/UAF-grid

25. Dec. 2010 NOAA/UAF-grid 25 Features recently added to LAS

26. Dec. 201026 WMS can be used by many GIS applications NOAA/UAF-grid

27. Dec. 201027 Browser access using ERDDAP … and from there …

28. Dec. 201028.mat REST URL access to data subsets in several simple formats (accessible through home-grown scripting of many types).kml.mat.nc NOAA/UAF-grid

29. Dec. 201029 How can users find the datasets? (less mature) 1. Web crawlers extract metadata from files (and elsewhere) extract metadata from files (and elsewhere) ingest into a data bases ingest into a data bases and from there: and from there: 2. formal metadata records generated (ISO 19115) 3. data discovery portals harnessed (e.g. RAMADDA, GeoNetwork, GI-Cat, …) 4. metadata made available for harvesting –GCMD, Geospatial One-Stop, GEOSS, data.gov NOAA/UAF-grid

30. Dec. 2010 NOAA/UAF-grid 30 UAF home page Instructions for end users: how to access data through their favorite applications UAF is experimenting to see how documentation may be shared by other projects…

31. Dec. 2010 NOAA/UAF-grid 31 netCDF Java Tools LAS ERDDAP GODIVA From the THREDDS catalog “viewers” may also be called up

32. Dec. 2010 NOAA/UAF-grid 32 File-internal metadata accessible directly from THREDDS catalogs

33. Dec. 201033 The coming year’s target: Integrating “1-d grids” (observations) Collections of time series & profiles in netCDF-CF format are already on-line NOAA/UAF-grid

34. Dec. 201034 UAF team met just 3 weeks ago (Seattle) NOAA/UAF-grid Preliminary plans: Some initial target collections of in situ obs: Argo, OceanSites, ICOADS stream gauge data from USGS atmospheric profilers (NCDC) … Common denominators for accessing the collections: UAF year-1 techniques allow 1-at-a-time access ERDDAP tables (ASCII and OPeNDAP Sequence) a gateway for IOOS/SOS, RDBMS, DAPPER, … Year 1 metadata techniques largely applicable to obs  Still largely volunteer-powered

35. Dec. 201035 “No standard should be approved without having been used to implement a few projects of realistic complexity.” Beware: “… specifications end up as the kitchen sink of every feature thought of” (The Achilles' heel of committee processes: complexity and boat.) The Rise and Fall of CORBA (Henning, 2006) The UAF strategy builds on historical lessons NOAA/UAF-grid

36. Dec. 201036 Thank you NOAA/UAF-grid