National Center for Supercomputing Applications University of Illinois at Urbana–Champaign Data Management for the Mississippi River Luigi Marini September.

Slides:

Advertisements

Similar presentations

HydroServer A Platform for Publishing Space- Time Hydrologic Datasets Support EAR CUAHSI HIS Sharing hydrologic data Jeffery.

Advertisements

C van Ingen, D Agarwal, M Goode, J Gupchup, J Hunt, R Leonardson, M Rodriguez, N Li Berkeley Water Center John Hopkins University Lawrence Berkeley Laboratory.

Linking HIS and GIS How to support the objective, transparent and robust calculation and publication of SWSI? Jeffery S. Horsburgh CUAHSI HIS Sharing hydrologic.

Components of an Integrated Environmental Observatory Information System Cyberinfrastructure to Support Publication of Water Resources Data Jeffery S.

This work was funded by the U.S. National Science Foundation under grant EAR Any opinions, findings and conclusions or recommendations expressed.

Development of a Community Hydrologic Information System Jeffery S. Horsburgh Utah State University David G. Tarboton Utah State University.

Using GIS in Creating an End-to- End System for Publishing Environmental Observations Data Jeffery S. Horsburgh David G. Tarboton, David R. Maidment, Ilya.

Introducing the CUAHSI Hydrologic Information System Desktop Application (HydroDesktop) and Open Development Community Jiří Kadlec, Daniel Ames, Teva Velupillai.

Deployment and Evaluation of an Observations Data Model Jeffery S Horsburgh David G Tarboton Ilya Zaslavsky David R. Maidment David Valentine

SAN DIEGO SUPERCOMPUTER CENTER Developing a CUAHSI HIS Data Node, as part of Cyberinfrastructure for the Hydrologic Sciences David Valentine Ilya Zaslavsky.

0-1 Team ?? Status Report (1 of 3) Client Contact –Point 1 –Point 2 Team Meetings –Point 1 –Point 2 Team Organization –Point 1 –Point 2 Team 1: Auraria.

An End-to-End System for Publishing Environmental Observations Data Jeffery S. Horsburgh David K. Stevens, David G. Tarboton, Nancy O. Mesner, Amber Spackman.

Introduction to Building a BI Solution 권오주 OLAPForum

Tools for Publishing Environmental Observations on the Internet Justin Berger, Undergraduate Researcher Jeff Horsburgh, Faculty Mentor David Tarboton,

U.S. Department of the Interior U.S. Geological Survey U.S. National Water Census “Cyber – Platform” Update Progress and challenges to overcome in realizing.

Data for Water Resource Management Module 14, part A – Data types and sources.

About CUAHSI The Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) is an organization representing 120+ universities.

Ocean Sciences What is CUAHSI? CUAHSI – Consortium of Universities for the Advancement of Hydrologic Science, Inc Formed in 2001 as a legal entity Program.

Evolution to CIMI Charles (Cal) Loomis & Mohammed Airaj LAL, Univ. Paris-Sud, CNRS/IN2P3 29 August 2013.

Information Requirements for Integrating Spatially Discrete, Feature- Based Earth Observations Jeffery S. Horsburgh Anthony Aufdenkampe, Kerstin Lehnert,

Lecture 3: Geospatial Web Services From Web sites to Web services Geospatial Web service functions Web service types Interoperability and geospatial service.

AIRNow-International The future of the United States real-time air quality reporting and forecasting program and GEOSS participation John E. White U.S.

WPS Application Patterns at the Workshop “Models For Scientific Exploitation Of EO Data” ESRIN, October 2012 Albert Remke & Daniel Nüst 52°North Initiative.

UNCW Ocean Observing: Providing Infrastructure, Data, and Products to Support State and Federal Agency Needs AQUARIUS NOAA’s Undersea Research Center (NIUST,

The GeoConnections Discovery Portal Michael Robson MacDonald Dettwiler and Associates Brian McLeod, Michael Adair Natural Resources Canada.

Exercises: Organizing, Loading, and Managing Point Observations Using HydroServer Support EAR CUAHSI HIS Sharing hydrologic data

Is Apache CouchDB for you?

U.S. Environmental Protection Agency WATERS Status Update

Water Web Services David R. Maidment Center for Research in Water Resources University of Texas at Austin Open Waters Symposium Delft, the Netherlands.

Introduction to Apache OODT Yang Li Mar 9, What is OODT Object Oriented Data Technology Science data management Archiving Systems that span scientific.

National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Dynamic Virtual Observatories James Myers, Luigi Marini, Rob.

Designing Web Applications Thaddeus Diamond CPSC 112.

BioData a new bioassessment database for the USGS Briefing for the CDI

Water and Catchment Data Services David R. Maidment Center for Research in Water Resources University of Texas at Austin River Science Symposium Swansea,

Ocean Observatories Initiative Data Management (DM) Subsystem Overview Michael Meisinger September 29, 2009.

1 Factors influencing the dynamics of excessive algal blooms Richard F. Ambrose Environmental Science and Engineering Program Department of Environmental.

1 MSCS 237 Overview of web technologies (A specific type of distributed systems)

NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.

CUAHSI HIS Features of Observations Data Model. NWIS ArcGIS Excel NCAR Trends NAWQA Storet NCDC Ameriflux Matlab AccessSAS Fortran Visual Basic C/C++

Web Pages with Features. Features on Web Pages Interactive Pages –Shows current date, get server’s IP, interactive quizzes Processing Forms –Serach a.

1 Web Services USGS/EPA Collaboration. 2 Overview  EPA and USGS are working together to develop a commons suite of web services  These services will.

Mercury – A Service Oriented Web-based system for finding and retrieving Biogeochemical, Ecological and other land- based data National Aeronautics and.

Module: Software Engineering of Web Applications Chapter 2: Technologies 1.

U.S. Environmental Protection Agency Central Data Exchange Pilot Project Promoting Geospatial Data Exchange Between EPA and State Partners. April 25, 2007.

Data Model / Database Implementation (continued) Jeffery S. Horsburgh Hydroinformatics Fall 2014 This work was funded by National Science Foundation Grants.

CUAHSI HIS: Science Challenges Linking small integrated research sites (

U.S. Department of the Interior U.S. Geological Survey Water Data Sharing Using OGC Standards and the Internet David Blodgett USGS Center for Integrated.

The Virtual Observatory and Ecological Informatics System (VOEIS): Using RESTful architecture and an extensible data model to provide a unique data management.

Google Analytics Graham Triggs Head of Repository Systems, Symplectic.

National Center for Supercomputing Applications University of Illinois at Urbana–Champaign Great Lakes to Gulf Virtual Observatory Marcus Slavenas September.

Proposal of Satellite Data Center India Meteorological Department A.K.Sharma (Chairman), Virendera Singh (Member), R.K.Giri (Member) and N.Puviarasan (Member.

Grid Services for Digital Archive Tao-Sheng Chen Academia Sinica Computing Centre

Developing a community hydrologic information system David G Tarboton David R. Maidment (PI) Ilya Zaslavsky Michael Piasecki Jon Goodall

Intermountain West Data Warehouse - Western Air Quality Study

The CUAHSI Hydrologic Information System Spatial Data Publication Platform David Tarboton, Jeff Horsburgh, David Maidment, Dan Ames, Jon Goodall, Richard.

Connected Infrastructure

Using GIS in Creating an End-to-End System for Publishing Environmental Observations Data Jeffery S. Horsburgh David G. Tarboton, David R. Maidment, Ilya.

Sharing Hydrologic Data with the CUAHSI* Hydrologic Information System

Web Development Web Servers.

Jeffery S. Horsburgh Utah State University

REST: Web Services Abel Sanchez.

REST- Representational State Transfer Enn Õunapuu

Lecture 8 Database Implementation

Steering Group Member, Link Digital

Connected Infrastructure

KISTERS TimeSeries HUB

21 November Data Science Capabilities

Introduction to Portal for ArcGIS

Chengyu Sun California State University, Los Angeles

IGARSS 2019 Dr. Ingo Simonis July 2019

Presentation transcript:

National Center for Supercomputing Applications University of Illinois at Urbana–Champaign Data Management for the Mississippi River Luigi Marini September 2nd, 2015

Great Lakes to Gulf (GLTG) Virtual Observatory Build on existing technology stack Live data streaming of 5 GREON buoys on the Mississippi River, and 2 GREON buoys in Illinois watersheds Historical data from the USGS, state agencies, water utilities Collaboration and funding:

Outline Goals Data sources Features Software Architecture Data Ingestion

Scalable Data Management Ability to grow in data size and data types Long Term Archive Geospatial Database Raw Data Geospatial Cache Long Term Archive Visualize Search Retrieve

Historical and Real Time Data 1 timeContinuous

DATA SOURCES CURRENTLY AVAILABLE

Water Quality Portal (USGS, USEPA, USDA) Data accessed via web services Parameters Nitrogen nitrate All other nutrients (coming soon) Other water quality parameters (coming soon)

United States Geological Survey Daily measurement Water discharge Nitrate nitrogen Nitrate Nitrogen Load Nitrate Nitrogen Cumulative Load Other water quality parameters (coming soon)

U.S. Army Corps of Engineers Long term Monitoring program

Great Rivers Ecological Observation Network Parameters water temperature, oxygen, conductivity, turbidity, chlorophyll-a, blue-green algae, nitrate, weather information Data streamed near real time 7 GREON platforms are currently deployed, positioned at La Crosse, Wi (2), Alton, IL (2), Cape Girardeau, MO (1) Carlisle Lake, IL (1), Lake Decatur, IL (1)

National Oceanic and Atmospheric Administration Data streamed near real time Temperature (More water quality parameters coming soon)

Other Data Sources?

FEATURES

Explore the Data

Search

Search Results

Data Archive

Provenance Trail

SOFTWARE ARCHITECTURE

Flexible and Robust Backend Historical Data Time Location Based Space Flexibility Flexible Data Representation RequirementsDimensions Discovery Slice and Dice Across All Dimensions

Flexible and Robust Backend Interact with backend in the most natural way for the browser Do some processing (e.g. averages) in the browser Not require mapping between formats Clear separation between client and server HTTP-based web service JavaScript Object Notation (JSON) GeoJSON! location & flexible attributes What about time? Browser GeoDashboard Server

Datapoint ISO 8601 With timezone Open JSON document GeoJSON Altitude

Datapoint

Data API Datapoint Stream Sensor/Site

Search HTTP GET /geostreams/datapoints? geocode= , ,74.759& since= :00:00& until= :00:00& sources=epa& attributes=NNN, mg/l& attributes=Si, mg/l& attributes=TP, ug/l& format=json/csv

DATA INGESTION

Add Datapoint HTTP POST /geostreams/datapoints In body:

Data Archive and Geotemporal API Imaginations unbound Raw file ER58.cnv Raw file ER63s.cnv Upload Data Archive GREON software (datalogger) Geotemporal API External Catalog Analytics Code Sensor

Data Proxy Geotemporal API ScriptService

Data Proxy Services can be written in any language HTML / JSON The most common standards on the web Caching the data For fast queries Each data point carries with it where it came from

Open source Software

Questions? Contact Michael Brennan Live Site