The Unified Access Framework (UAF) Building NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) one step at a time Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), Lewis McCulloch (HQ/TPIO) and the NOAA UAF team

Dec Enterprise-wide integration of data is a very hard problem! (i.e. adopting shared approaches to achieve data interoperability) NOAA/UAF-grid

Dec NOAA/UAF-grid Why? Different fields have different concepts of ‘data’ … and have developed solutions that make sense to them.  Getting people (and organizations) to change habits is difficult!

Dec NOAA-world NOAA/UAF-grid weather forecast (time critical) fisheries management (regulatory concerns) nautical charting climate, ocean, atmosphere research … the list goes on … … the list goes on … NOAA management has been reluctant to invest funds in Agency-wide data integration

Dec The accepted approach: build a “system of systems” “Wrap” existing systems with loosely coupled, standardized services  a Service Oriented Architecture NOAA/UAF-grid

Dec Global Earth Observation Integrated Data Environment GEO-IDE Con-ops outlines such a plan , 66 pages (by “DMIT”) NOAA/UAF-grid

Dec Seed funding ٨ available last year Question: How do you approach a very big problem with a very small amount of money? finally NOAA/UAF-grid

Dec Generate use cases 2.Define requirements 3.Write a Concept of Operations 4. … and an Implementation Plan 5.Assemble volunteer teams to implement NOT ! NOT ! NOAA/UAF-grid Tried and true approach …

Dec An alternative (‘agile’) approach Don't Solve Problems -- Copy Success "Switch: How to Change Things When Change Is Hard", Chip and Dan Heath (psychologists), 2010 NOAA/UAF-grid

What “success” did UAF chose to copy? Year 1 focused on gridded datasets. Dec Projects: (too many to name) Data formats: netCDFGRIBHDF Service stack: netCDF-CF-DAP-THREDDS-WMS Applications: MatlabArcGISFerret GrADS Google Earth IDV LAS ERDDAP… Users: (too many to name) …

Dec UAF (*) -grid (initial focus: gridded data) –netCDF –CF –OPeNDAP –THREDDS –WMS & WCS (*) Unified Access Framework --file format/API/data model --file format/API/data model --metadata conventions --metadata conventions --network transport --network transport --network topology --network topology -- OGC rendering and transport -- OGC rendering and transport NOAA/UAF-grid

THREDDS “network topology” a tree defined as distributed XML Dec NOAA/UAF-grid 12 NOMADS GEO-IDE/UAF NOAANOAA Affiliated NMFSOARNWSNESDIS NODC NGDC GFDL PMEL AOML OCO PFEG NDBC ESRL Coastwatch IOOS National Partners IOOS Regional Partners NAVO AOOS NANOOS CENCOOS SCCOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS

Dec Who is using this approach? Modelers –IPCC, GFDL, NCAR, … Satellite programs –GHRSST, PathFinder, CoastWatch, … NCEP weather and ocean forecasts –GRIB files served via NOMADS Coastal (“HF”) radar A growing list of observations programs –Argo, OceanSites, … Pending adoption by OGC NOAA/UAF-grid

Dec Some rough spots compliance with CF conventions is inconsistent individual files often not aggregated into logical datasets metadata are often in need of enhancement NOAA/UAF-grid

Dec … but excellent repair tools exist: NcML and “virtual files” e.g. Improve CF compliance by adding “standard_name” attribute to GRIB data The file, itself, is untouched. But access to the file through the services is made to conform to standards. NOAA/UAF-grid

Dec NcML for aggregation e.g. Aggregate three 1-year files of the same (say) model run </aggregation> The files are untouched, but services see a single logical file. NOAA/UAF-grid

Dec The emerging UAF The UAF THREDDS catalog: ~500 “clean” datasets.. growing as the team aggregates and configures using ncML –team members become coaches  lasting solutions must be implemented at the data provider sites The UAF Team –8-10 active participants –self-selected (volunteers) 1 year into the project … NOAA/UAF-grid

Dec How to reach users? Through their preferred tools (without downloading files) NOAA/UAF-grid

Dec Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32 Desktop access in Matlab NOAA/UAF-grid

Dec Access in ArcGIS using the Environmental Data Connector (EDC) NOAA/UAF-grid

Dec Desktop access in Ferret NOAA/UAF-grid

Dec Desktop access in GrADS NOAA/UAF-grid

Dec Desktop access in IDV NOAA/UAF-grid

Web browser examples … Dec Take differences Google Earth Scripts Live Access Server (LAS) NOAA/UAF-grid

Dec NOAA/UAF-grid 25 Features recently added to LAS

Dec WMS can be used by many GIS applications NOAA/UAF-grid

Dec Browser access using ERDDAP … and from there …

Dec mat REST URL access to data subsets in several simple formats (accessible through home-grown scripting of many types).kml.mat.nc NOAA/UAF-grid

Dec How can users find the datasets? (less mature) 1. Web crawlers extract metadata from files (and elsewhere) extract metadata from files (and elsewhere) ingest into a data bases ingest into a data bases and from there: and from there: 2. formal metadata records generated (ISO 19115) 3. data discovery portals harnessed (e.g. RAMADDA, GeoNetwork, GI-Cat, …) 4. metadata made available for harvesting –GCMD, Geospatial One-Stop, GEOSS, data.gov NOAA/UAF-grid

Dec NOAA/UAF-grid 30 UAF home page Instructions for end users: how to access data through their favorite applications UAF is experimenting to see how documentation may be shared by other projects…

Dec NOAA/UAF-grid 31 netCDF Java Tools LAS ERDDAP GODIVA From the THREDDS catalog “viewers” may also be called up

Dec NOAA/UAF-grid 32 File-internal metadata accessible directly from THREDDS catalogs

Dec The coming year’s target: Integrating “1-d grids” (observations) Collections of time series & profiles in netCDF-CF format are already on-line NOAA/UAF-grid

Dec UAF team met just 3 weeks ago (Seattle) NOAA/UAF-grid Preliminary plans: Some initial target collections of in situ obs: Argo, OceanSites, ICOADS stream gauge data from USGS atmospheric profilers (NCDC) … Common denominators for accessing the collections: UAF year-1 techniques allow 1-at-a-time access ERDDAP tables (ASCII and OPeNDAP Sequence) a gateway for IOOS/SOS, RDBMS, DAPPER, … Year 1 metadata techniques largely applicable to obs  Still largely volunteer-powered

Dec “No standard should be approved without having been used to implement a few projects of realistic complexity.” Beware: “… specifications end up as the kitchen sink of every feature thought of” (The Achilles' heel of committee processes: complexity and boat.) The Rise and Fall of CORBA (Henning, 2006) The UAF strategy builds on historical lessons NOAA/UAF-grid

Dec Thank you NOAA/UAF-grid