1. National Center for Supercomputing Applications University of Illinois at Urbana–Champaign Data Management for the Mississippi River Luigi Marini September 2nd, 2015

2. Great Lakes to Gulf (GLTG) Virtual Observatory Build on existing technology stack Live data streaming of 5 GREON buoys on the Mississippi River, and 2 GREON buoys in Illinois watersheds Historical data from the USGS, state agencies, water utilities Collaboration and funding:

3. Outline Goals Data sources Features Software Architecture Data Ingestion

4. Scalable Data Management Ability to grow in data size and data types Long Term Archive Geospatial Database Raw Data Geospatial Cache Long Term Archive Visualize Search Retrieve

5. Historical and Real Time Data 1 timeContinuous

6. DATA SOURCES CURRENTLY AVAILABLE

7. Water Quality Portal (USGS, USEPA, USDA) Data accessed via web services Parameters Nitrogen nitrate All other nutrients (coming soon) Other water quality parameters (coming soon)

8. United States Geological Survey Daily measurement Water discharge Nitrate nitrogen Nitrate Nitrogen Load Nitrate Nitrogen Cumulative Load Other water quality parameters (coming soon)

9. U.S. Army Corps of Engineers Long term Monitoring program

10. Great Rivers Ecological Observation Network Parameters water temperature, oxygen, conductivity, turbidity, chlorophyll-a, blue-green algae, nitrate, weather information Data streamed near real time 7 GREON platforms are currently deployed, positioned at La Crosse, Wi (2), Alton, IL (2), Cape Girardeau, MO (1) Carlisle Lake, IL (1), Lake Decatur, IL (1)

11. National Oceanic and Atmospheric Administration Data streamed near real time Temperature (More water quality parameters coming soon)

12. Other Data Sources?

13. FEATURES

14. http://gltg.ncsa.edu

15. Explore the Data

16. Search

17. Search Results

18. Data Archive

19. Provenance Trail

21. SOFTWARE ARCHITECTURE

22. Flexible and Robust Backend Historical Data Time Location Based Space Flexibility Flexible Data Representation RequirementsDimensions Discovery Slice and Dice Across All Dimensions

23. Flexible and Robust Backend Interact with backend in the most natural way for the browser Do some processing (e.g. averages) in the browser Not require mapping between formats Clear separation between client and server HTTP-based web service JavaScript Object Notation (JSON) GeoJSON! location & flexible attributes What about time? Browser GeoDashboard Server

24. Datapoint ISO 8601 With timezone Open JSON document GeoJSON Altitude

25. Datapoint

26. Data API Datapoint Stream Sensor/Site

27. Search HTTP GET /geostreams/datapoints? geocode=46.07323062540838,-83.232421875,74.759& since=2010-01-01 12:00:00& until=2010-06-01 12:00:00& sources=epa& attributes=NNN, mg/l& attributes=Si, mg/l& attributes=TP, ug/l& format=json/csv

28. DATA INGESTION

29. Add Datapoint HTTP POST /geostreams/datapoints In body:

30. Data Archive and Geotemporal API Imaginations unbound Raw file ER58.cnv Raw file ER63s.cnv Upload Data Archive GREON software (datalogger) Geotemporal API External Catalog Analytics Code Sensor

31. Data Proxy Geotemporal API ScriptService

32. Data Proxy Services can be written in any language HTML / JSON The most common standards on the web Caching the data For fast queries Each data point carries with it where it came from

33. Open source Software https://opensource.ncsa.illinois.edu

34. Questions? Contact Michael Brennan Live Site https://gltg.ncsa.illinois.edu