1. Catalog-driven workflows using CSW Rich Signell, USGS, Woods Hole, MA, USA Filipe Fernandes, SECOORA, Brazil Kyle Wilcox, Axiom Data Science, Wickford, RI, USA ESIP Winter Meeting, Washington, DC 2016-01-08 Rich Signell, USGS, Woods Hole, MA, USA Filipe Fernandes, SECOORA, Brazil Kyle Wilcox, Axiom Data Science, Wickford, RI, USA ESIP Winter Meeting, Washington, DC 2016-01-08

2. The 4 th Network Layer: Data “We need an end-to-end, layer-by-layer, designed information technology … that are composed of no more than a stack of protocols” “We need open standards… and above all, we need to teach scientists to work in this new layer of data” “We need an end-to-end, layer-by-layer, designed information technology … that are composed of no more than a stack of protocols” “We need open standards… and above all, we need to teach scientists to work in this new layer of data” 2 From the essay: “I have seen the Paradigm Shift, and It Is Us”, byJohn Wilbanks, in the book “The Fourth Paradigm” Data Web TCP/IP Ethernet

3. US Integrated Ocean Observing System (IOOS ® ) Global ComponentGlobal Component Coastal ComponentCoastal Component  17 Federal Agencies  11 Regional Associations

4. IOOS Core Principles Adopt open standards & practices Avoid customer-specific stovepipes Standardized access services implemented at data providers Adopt open standards & practices Avoid customer-specific stovepipes Standardized access services implemented at data providers 4 Customer Web access service DataProvider Observations Models

5. Numerical model Output

6. Time Series, Trajectories Meteorology and Wave Buoy in the Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution

7. IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML Common Data Model OPeNDAP NetCDF Subset THREDDS Data Server Standardized (CF-1.6, SGRID-0.1, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python EDC NetCDF-Java SOS Geoportal Server GeoNetwork CKAN Observed data (buoy, gauge, ADCP, glider) Web Portals pycsw NcML Grid TimeSeries Profile Trajectory TimeSeriesProfile Sgrid Ugrid Nonstandard Data Files Catalog Services Rectilinear ERDDAP WCS

8. Catalog Search 8

9. Interoperable Access in Python (Iris)

10. IOOS System Test

11. 2015 Boston Light Swim 2015 Aug 15, 7:00 am start 8 mile swim No wet suit How cold will the water be?

12. NECOFS Massbay Forecast

13. Reproducible Jupyter Notebook Go to https://github.com/ocefpaf/boston_light_swim, click on “launch binder” to run on cloudhttps://github.com/ocefpaf/boston_light_swim

17. Final Result

18. 18

19. 19

20. pycsw 20

21. Workflow for the USGS CMG Portal 21

22. Workflow (3/3) Axiom Data Science –Runs a CSW search (in a cron job) on the modeling groups pycsw services, filtering on datasets that contain a project called “CMG_Portal” –Datasets that have valid WMS services are added to the portal See for details of the workflow Axiom Data Science –Runs a CSW search (in a cron job) on the modeling groups pycsw services, filtering on datasets that contain a project called “CMG_Portal” –Datasets that have valid WMS services are added to the portal See for details of the workflow 22

23. 23

24. WMS-driven Model Viewing Portal

25. 25

26. Interoperable access in Matlab (nctoolbox)

27. 27

28. 28

29. Catalog-driven dynamic portals 29

30. 30

31. Benefits of catalog-driven applications Dynamically adapt to new or changing data Find the machine-to-machine issues –Easy problems that can be fixed in minutes to day –Harder problems to guide future work Fixes for your workflow benefit everyone Build success stories Create reproducible workflows that others can learn from, expand on, or transform Standardized workflows help develop the 4 th network layer for data Dynamically adapt to new or changing data Find the machine-to-machine issues –Easy problems that can be fixed in minutes to day –Harder problems to guide future work Fixes for your workflow benefit everyone Build success stories Create reproducible workflows that others can learn from, expand on, or transform Standardized workflows help develop the 4 th network layer for data