1. The Unified Access Framework (UAF) Philosophy, progress, and plans DAARWG Meeting, Seattle, Nov. 2011 Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), and the NOAA UAF team Kenneth.Casey@noaa.gov Julie.Bosch@noaa.gov Tina.Chang@noaa.gov Scott.Cross@noaa.gov Roy.Mendelssohn@noaa.gov Steven.C.Hankin@noaa.gov Jordan.Alpert@noaa.gov Jim.Sargent@noaa.gov Ted.Habermann@noaa.gov John.Relph@noaa.gov Bob.Simons@noaa.gov David.Neufeld@noaa.gov Upendra.Dadi@noaa.gov Rich Signell (rsignell@usgs.gov ) Phil.Cogbill@noaa.gov Glenn.Rutledge@noaa.gov Mike.Grogan@noaa.gov Jeff.Budai@noaa.gov

2. Review … continuing into new material Nov. 2011 NOAA/UAF 2

3. Nov. 20113 GEO-IDE a strategy for enterprise- wide integration of NOAA data (i.e. adopting standards and practices to achieve data interoperability) NOAA/UAF 2005-06 66 pgs (by “DMIT”) Why is this a hard problem?

4. Nov. 2011 4 NOAA-world NOAA/UAF weather forecast (time critical) fisheries management (regulatory concerns) nautical charting climate, ocean, atmosphere research response and restoration … the list goes on … … the list goes on …

5. Nov. 2011 5 NOAA/UAF Different disciplines have different concepts of ‘data’. Each develops solutions that make sense to them.  Getting people (and organizations) to change habits is difficult! (and data management has often been an after thought.)

6. Nov. 20116 The accepted approach: build a “system of systems” Wrap existing systems with loosely coupled, standardized services  a Service Oriented Architecture NOAA/UAF GEO-IDE Con-ops outlines such a plan. How to build it with a largely volunteer team?

7. Nov. 20117 1.Generate use cases 2.Define requirements 3.Write a Concept of Operations 4. … and an Implementation Plan 5.Assemble (volunteer) teams to implement NOT ! NOT ! NOAA/UAF Tried and true approach …

8. Nov. 20118 An alternative (‘agile’) approach Don't Solve Problems -- Copy Success -- NOAA/UAF

9. Nov. 20119 Why ‘agile’ is attractive Because inevitably: funding is much smaller than needed; collaborations are more difficult than anticipated; infrastructure is being built on a background of rapidly evolving technology NOAA/UAF

10. Nov. 201110 Why ‘agile’ is attractive NOAA/UAF infrastructure is being built on a background of rapidly evolving technology Change equals risk. Mitigate risk by following a strategy of incremental change that serves users (increasingly) well at every stage of evolution. “Visualize a decade away. Build things that work today.”

11. Agile Principles (condensed from the ‘Agile Manifesto’) Working software is the meaningful measure of progress: ensure early, continuous, frequent releases Emphasize simplicity - maximize the work not done Build projects around motivated individuals. Give them the support they need and entrust them The best architectures, requirements, and designs emerge from self-organizing teams Welcome changing requirements Scientists ( ‘business people’ ) and developers must work together on a daily basis Nov. 2011 NOAA/UAF 11

12. Nov. 2011 NOAA/UAF 12 Lemma “Don’t let perfect be the enemy of good”

13. What “success” did UAF chose to copy? Year 1 focused on gridded datasets. Nov. 201113 Projects: (too many to name) Data formats: netCDFGRIBHDF Service stack: netCDF-CF-DAP-THREDDS (WMS) Applications: MatlabArcGISFerret GrADS Google Earth IDV LAS ERDDAP… Users: (too many to name) … NOAA/UAF

14. Nov. 201114 Who is providing data this way? Modelers –AR4&5, GFDL, NCAR, … Satellite programs –GHRSST, PathFinder, CoastWatch, … NCEP weather and ocean forecasts –GRIB files served via NOMADS Coastal (“HF”) radar A growing list of observations programs –Argo, OceanSites, tide gauges, … Adoption by OGC is well underway NOAA/UAF

15. Nov. 201115 How to reach users? (without downloading files) NOAA/UAF Through their preferred tools

16. Nov. 201116 Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32 Desktop access in Matlab NOAA/UAF

17. Nov. 201117 Access in ArcGIS using the Environmental Data Connector (EDC) NOAA/UAF

18. Nov. 201118 Desktop access in Ferret NOAA/UAF

19. Nov. 201119 Desktop access in GrADS NOAA/UAF

20. Nov. 2011 NOAA/UAF 20 UAF home page Instructions for end users: how to access data through their favorite applications UAF is experimenting to see how documentation may be shared by other projects…

21. Under the hood Projects with data to provide –make their data available as netCDF-CF (or other TDS-compatible format) –host a THREDDS / OPeNDAP server Lets look at the tools we have to link projects together … Nov. 2011 NOAA/UAF 21

22. UAF “network topology” a tree defined in THREDDS (XML) Nov. 2011 NOAA/UAF 22 NOMADS GEO-IDE/UAF NOAANOAA Affiliated NMFSOARNWSNESDIS NODC NGDC GFDL PMEL AOML OCO PFEG NDBC ESRL Coastwatch IOOS National Partners IOOS Regional Partners NAVO AOOS NANOOS CENCOOS SCCOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS

23. Nov. 201123 Not so fast, kiddo! compliance with CF conventions is inconsistent files commonly are not aggregated into logical datasets metadata are often in need of enhancement NOAA/UAF Perspective – this is not an unusual situation: Standards compliance problems are *the norm* Divergent dialects often pile up (e.g. GRIB, BUFR) UAF tools offer a solution …

24. Nov. 201124 ‘NcML’ can be used to repair problems (*) e.g. Improve CF compliance by adding “standard_name” attribute to GRIB data The file, itself, is untouched. The ‘virtual file’ seen through the services conforms to standards. (*) IOServiceprovider modules also important … not discussed here NOAA/UAF

25. Nov. 201125 NcML for aggregation e.g. Aggregate three 1-year files of the same (say) model run </aggregation> A long time series ‘virtual file’ is seen through the services. NOAA/UAF

26. UAF Team members are helping data managers, person-to-person, to improve the data services from their projects (a gradual, but important process) (a gradual, but important process) in parallel … in parallel … Nov. 2011 NOAA/UAF 26

27. Developing the UAF Catalog Cleaner (a ‘web crawler’) Nov. 2011 NOAA/UAF 27 NOMADS UAF ‘RAW’ catalog NOAANOAA Affiliated NMFSOARNWSNESDIS NODC NGDC GFDL PMEL AOML OCO PFEG NDBC ESRL Coastwatch IOOS National Partners IOOS Regional Partners NAVO AOOS NANOOS CENCOOS SCCOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS NOMADS UAF ‘CLEAN’ catalog NOAANOAA Affiliated NMFSOARNWSNESDIS NODC NGDC GFDL PMEL AOML OCO PFEG NDBC ESRL Coastwatch IOOS National Partners IOOS Regional Partners NAVO AOOS NANOOS CENCOOS SCCOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS ‘RAW’ ‘CLEAN’

28. The Catalog Cleaner 1. Crawl the raw catalog 2. Extracts metadata from the files, themselves, and from THREDDS into a relational database 3. Process the data base to detect aggregations, etc. 4. Create new THREDDS XML that is aggregated and metadata-cleaned Nov. 2011 NOAA/UAF 28

29. ‘09 Carbon Tracker files from ESRL Nov. 2011 NOAA/UAF 29 ‘raw catalog’ snippit Optional documentation (beyond what is inside the file) Services: Usually OPeNDAP. Often WMS. Sometimes more. Optional viewers

30. Nov. 2011 NOAA/UAF 30 ‘clean catalog’ -- same data, but … augmented with Uniform services Uniform viewers improved metadata uniform services uniform viewers

31. Nov. 2011 NOAA/UAF 31 Uniform services ISO-standard metadata metadata quality assessment Simple interface to get a subset

32. Nov. 2011 NOAA/UAF 32

33. Nov. 2011 NOAA/UAF 33

34. Nov. 2011 NOAA/UAF 34 Godiva2 uses the WMS map service (as do other GIS apps)

35. Nov. 201135 NOAA/UAF Google Earth Property- property plots Sections and Hofmullers Vector plots Segue to desktop tools Line plots Animations Differencing Analyses

36. Nov. 201136 and ERDDAP provides … NOAA/UAF

37. Nov. 201137.mat REST URL access to data subsets in several formats (accessible through home-grown scripting of many types).kml.mat.nc NOAA/UAF

38. Nov. 2011 NOAA/UAF 38 ISO-standard metadata Data Discovery Crawl the clean catalog. Create ISO metadata

39. Agile principle: “Maximize the work not done” Who has already built reasonably mature discovery portals (preferably free) ? Unidata’s RAMADDAUnidata’s RAMADDA GI-CAT (able to crawl THREDDS catalog)GI-CAT (able to crawl THREDDS catalog) Geoportal (able to harvest ISO metadata)Geoportal (able to harvest ISO metadata) Nov. 2011 NOAA/UAF 39

40. Data Discovery using ESRI Geoportal NODC is leading the UAF Geoportal investigations

41. Enter Search terms Data Discovery using ESRI Geoportal

42. Search Results Data Discovery using ESRI Geoportal

43. Refined Search New Results Data Discovery using ESRI Geoportal

44. Expand Selection Data Discovery using ESRI Geoportal

45. Available Services Data Discovery using ESRI Geoportal

46. Available Services Data Discovery using ESRI Geoportal

47. View metadata details about the dataset

48. Available Services Data Discovery using ESRI Geoportal

49. Direct link into the THREDDS Data Server

50. Available Services Data Discovery using ESRI Geoportal

51. This seque to LAS demonstrates how GeoPortal can be trained to recognize a uniform set of UAF viewers

52. Nov. 2011 NOAA/UAF 52 Full disclosure: Geoportal is very powerful, but … difficult to set up immature user interface

53. Nov. 201153 In situ observations NOAA/UAF are characterized by their ‘feature type’ (a time series, a track line, a vertical profile, …) individual feature instances are degenerate (1D) grids the UAF framework just described already handles individual 1D features many data providers are creating such files (Argo, OceanSites, …). Growth is robust. but users want to interact with collections of features standards and practices are in flux

54. Nov. 201154 Collections of in situ observations NOAA/UAF Candidates: 1. CF ‘Discrete Geometries’ Chapter 9 of CF 1.6, May 2011 the ‘natural’ (?) extension for UAF but … a new API yet to be released (TDS 4.3) Java support only few tools available

55. Nov. 201155 Collections of in situ observations NOAA/UAF Candidates: 2.‘DAPPER’ (OPeNDAP Sequences) solid technology good prototype tools and examples already years ago but … poor standardization of semantics (conventions for lat, long, time, …) flagging momentum …

56. Nov. 201156 Collections of in situ observations NOAA/UAF Candidates: 3.OGC Web Feature Service (WFS) much momentum in the GIS community but … by design little standardization of semantics communities must develop ‘application schemas’ success rates poor, momentum flagging …

57. Nov. 201157 Collections of in situ observations NOAA/UAF Candidates: 4.OGC Sensor Observation Service (SOS) significant momentum in the earth science community but … only the request protocol is standardized many choices for semantics: “SWE common”, KML, custom ASCII/JSON/etc. few tools, interoperability concerns

58. Nov. 201158 Collections of in situ observations NOAA/UAF What to do? Agile philosophy: Maximize the work not done Don’t let perfect be the enemy of good UAF isn’t resourced to undertake R&D ($ + risk)

59. Nov. 201159 Collections of in situ observations NOAA/UAF UAF approach: 1.Explore the potentials of ERDDAP. It provides IOOS-compatible SOS ‘TableDAP’ and other R.E.S.T. outputs commitment to support many protocols 2.Where data are in databases: install ERDDAP e.g. OSMC (in situ ocean obs) 3.Prototype TDS 4.3 / CF Discrete Geometries Utilize from ERDDAP and LAS (ArcGIS soon after?)

60. Nov. 201160 Collections of in situ observations NOAA/UAF UAF approach: 4.Vigorously promote 1D CF features Here’s why: Available today, understood by data providers, momentum exists Compatible with UAF tools and practices Provides immediate value to users

61. Nov. 201161 Collections of in situ observations NOAA/UAF UAF approach: 4.Vigorously promote 1D CF features  Efforts will pay off (low risk) Gets the data ‘out there’ (open, available) Metadata-rich Discoverable through standard frameworks Future TDS versions will offer aggregation  miracles will happen for free: SOS, improved discovery, etc.

62. To increase data integration, what question should we be asking? Is it: “How do I build a system that will […]?” Nov. 2011 NOAA/UAF 62 Or is it: “How do we most rapidly narrow the gap between what we know technology is capable of delivering, and what our actual practices are today?” What is the right balance of strategies: ‘Design and build’ vs. ‘Agile’ ? ‘Design and build’ vs. ‘Agile’ ?

63. Nov. 201163 Discussion NOAA/UAF