A Services Oriented Architecture for Water Resources Data David R. Maidment Center for Research in Water Resources University of Texas at Austin

A Services Oriented Architecture for Water Resources Data CUAHSI and WATERS WaterML and WaterOneFlow Hydrologic Information Server Modeling services

A Services Oriented Architecture for Water Resources Data CUAHSI and WATERS WaterML and WaterOneFlow Hydrologic Information Server Modeling services

Ocean Sciences What is CUAHSI? CUAHSI – Consortium of Universities for the Advancement of Hydrologic Science, Inc Formed in 2001 as a legal entity Program office in Washington (5 staff) NSF supports CUAHSI to develop infrastructure and services to advance hydrologic science in US universities Earth Sciences Atmospheric Sciences UCAR CUAHSI Unidata HIS National Science Foundation Geosciences Directorate

CUAHSI Member Institutions 112 US Universities as of September 2007

HIS Team WATERS Testbed WATERS Network Information System NSF has funded work at 11 testbed sites, each with its own science agenda. HIS supplies the common information system Minnehaha Creek

Super Computer Centers: NCSA, TACC Domain Sciences: Unidata, NCAR LTER, CZEN GEON Government: USGS, EPA, NCDC Industry: ESRI, Kisters, Microsoft CUAHSI HIS HIS Team and its Cyberinfrastructure Partners HIS Team: Texas, SDSC, Utah, Drexel, Duke

Super computer Centers: NCSA, TACC Domain Sciences: Unidata, NCAR LTER, GEON Government: USGS, EPA, NCDC Industry: ESRI, Kisters, Microsoft HIS Team WATERS Testbed WATERS Network Information System CUAHSI HIS HIS, WATERS and the CUAHSI Community

Super computer Centers: NCSA, TACC Domain Sciences: Unidata, NCAR LTER, GEON Government: USGS, EPA, NCDC, USDA Industry: ESRI, Kisters, OpenMI HIS Team WATERS Testbed WATERS Network Information System CUAHSI HIS International Partners CSIRO and Bureau of Meteorology Australian Water Resources Information System European Commission Water database design and model integration (OpenMI)

HIS Goals Hydrologic Data Access – providing better access to a large volume of high quality hydrologic data across the nation Support for Observatories – integrating local observations by academic investigators with hydrologic data for a region Advancement of Hydrologic Science – modeling and analysis of “hydrology in a dynamic earth” Hydrologic Education – bringing more data into the classroom

A Services Oriented Architecture for Water Resources Data CUAHSI and WATERS WaterML and WaterOneFlow Hydrologic Information Server Modeling services

Definition The CUAHSI Hydrologic Information System (HIS) is a geographically distributed network of data sources and functions that are integrated using web services so that they operate as a connected whole.

Services Oriented Architecture Service-oriented Architecture (SOA) is an architectural design pattern that concerns itself with defining loosely-coupled relationships between producers and consumers. A major focus of Web services is to make functional building blocks accessible over standard Internet protocols that are independent from platforms and programming languages. The Web Services Description Language (WSDL, pronounced 'wiz-dəl' or spelled out, 'W- S-D-L') is an XML-based language that provides a model for describing Web services.XMLWeb services (from Wikipedia) Defined by the World Wide Web Consortium (W3C)

Web Pages and Web Services Uses Hypertext Markup Language (HTML) Uses WaterML (an eXtended Markup Language for water data)

Rainfall & Snow Water quantity and quality Remote sensing Water Data Modeling Meteorology Soil water

Water Data Web Sites

NWISWeb site output # agency_cd Agency Code # site_no USGS station number # dv_dt date of daily mean streamflow # dv_va daily mean streamflow value, in cubic-feet per-second # dv_cd daily mean streamflow value qualification code # # Sites in this file include: # USGS NEUSE RIVER NEAR CLAYTON, NC # agency_cdsite_nodv_dtdv_vadv_cd USGS USGS USGS USGS USGS USGS USGS USGS USGS USGS USGS Time series of streamflow at a gaging station USGS has committed to supporting CUAHSI’s GetValues function

Observation Stations Ameriflux Towers (NASA & DOE)NOAA Automated Surface Observing System USGS National Water Information SystemNOAA Climate Reference Network Map for the US

Water Quality Measurement Sites in EPA Storet Substantial variation in data availability from states Data from Bora Beran, Drexel University

Water Quality Measurement Sites from Texas Commission for Environmental Quality (TCEQ)

Geographic Integration of Storet and TCEQ Data in HIS

Observations Catalog Specifies what variables are measured at each site, over what time interval, and how many observations of each variable are available

Point Observations Information Model Data Source Network Sites Variables Values {Value, Time, Qualifier, Offset} USGS Streamflow gages Neuse River near Clayton, NC Discharge, stage (Daily or instantaneous) 206 cfs, 13 August 2006 A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time A qualifier is a symbol that provides additional information about the value An offset allows specification of measurements at various depths in water GetSites GetSiteInfo GetVariables GetVariableInfo GetValues

Locations Variable Codes Date Ranges WaterML and WaterOneFlow GetSiteInfo GetVariableInfo GetValues WaterOneFlow Web Service Client STORET NAM NWIS Data Repositories Data EXTRACT TRANSFORM LOAD WaterML WaterML is an XML language for communicating water data WaterOneFlow is a set of web services based on WaterML

WaterOneFlow Set of query functions Returns data in WaterML

NWIS ArcGIS Excel NCAR Unidata NASA Storet NCDC Ameriflux Matlab AccessJava Fortran Visual Basic C/C++ Some operational services CUAHSI Web Services Data Sources Applications Extract Transform Load

GetValues for Dissolved Oxygen at Minnehaha Creek

A Services Oriented Architecture for Water Resources Data CUAHSI and WATERS WaterML and WaterOneFlow Hydrologic Information Server Modeling services

Hydrologic Information Server Supports data discovery, delivery and publication –Data discovery – how do I find the data I want? Map interface and observations catalogs Metadata based Search –Data delivery – how do I acquire the data I want? Use web services or retrieve from local database –Data Publication – how do I publish my observation data? Use Observations Data Model

Water Resource Regions and HUC’s

National Hydrography Dataset

NHDPlus for Region 17E

NHDPlus Reach Catchments ~ 3km 2 About 1000 reach catchments in each 8-digit HUC Average reach length = 2km 2.3 million reaches for continental US

Reach Attributes Slope Elevation Mean annual flow –Corresponding velocity Drainage area % of upstream drainage area in different land uses Stream order

Observations Catalog Specifies what variables are measured at each site, over what time interval, and how many observations of each variable are available

Hydrologic Information Server Microsoft SQLServer Relational Database Observations Data Geospatial Data GetSites GetSiteInfo GetVariables GetVariableInfo GetValues DASH – data access system for hydrologyWaterOneFlow services ArcGIS Server

Minnehaha Creek Experimental Site

Water Temperature (°C) Dissolved Oxygen (mg/L) Depth (m) Dissolved Oxygen (% saturation)

Data Heterogeneity Syntactic mediation –Heterogeneity of format –Use WaterML to get data into the same format Semantic mediation –Heterogeneity of meaning –Each water data source uses its own vocabulary –Match these up with a common controlled vocabulary –Make standard scientific data queries and have these automatically translated into specific queries on each data source

Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them Objective NWIS NARR NAWQA NAM-12 request request return return What we are doing now ….. Michael Piasecki Drexel University

Semantic Mediator What we would like to do ….. NWIS NAWQA NARR generic request GetValues GetValues HODM Michael Piasecki Drexel University

Hydroseek Supports search by location and type of data across multiple observation networks including NWIS and Storet

A Services Oriented Architecture for Water Resources Data CUAHSI and WATERS WaterML and WaterOneFlow Hydrologic Information Server Modeling services

Project sponsored by the European Commission to promote integration of water models within the Water Framework Directive Software standards for model linking Uses model core as an “engine”

OpenMI Conceptual Framework VALUES All values are referenced in a what-where-when framework, allowing different data resources or models to communicate data Space, L Time, T Variables, V D An application of the data cube to integrate simulation models Jon Goodall, Duke University

Typical model architecture Application User interface + engine Engine Simulates a process – flow in a channel Accepts input Provides output Model An engine set up to represent a particular location e.g. a reach of the Thames Engine Output data Input data Model application Run Write Read User interface

AcceptsProvides Rainfall (mm) Runoff (m 3 /s) Temperature (Deg C) Evaporation (mm) AcceptsProvides Upstream Inflow (m 3 /s) Outflow (m 3 /s) Lateral inflow (m 3 /s) Abstractions (m 3 /s) Discharges (m 3 /s) River Model Linking modelled quantities

Data transfer at run time Rainfall runoff Output data Input data User interface River Output data Input data User interface GetValues(..)

Models for the processes River (InfoWorks RS) Rainfall (database) Sewer (Mouse) RR (Sobek-Rainfall -Runoff)

Data exchange 3 Rainfall.GetValues River (InfoWorks-RS) Rainfall (database) Sewer (Mouse) 2 RR.GetValues 7 RR.GetValues RR (Sobek-Rainfall -Runoff) 1 Trigger.GetValues 6 Sewer.GetValues call data

Coupling the HIS with Models using OpenMI ODM Observations Data Model WaterOneFlow Web Services Water Markup Language WOFWaterML MODFLOW HEC-RAS Others OpenMI SWAT HSFP “academic models”

Linking the HIS with HEC-RAS via OpenMI HIS WaterOneFlow Web Service for NWIS Realtime Streamflow A HEC-RAS Model Workflow “trigger” Using real-time streamflow from the HIS as a boundary condition for a HEC-RAS simulation.

SOAP-based Web Service for Models HEC-RAS USGS NWIS WSDL Simulation Model Database OpenMI Workflow In an effort to build cyberinfrastructure for the hydrologic sciences, we are extending OpenMI to utilize models as web services.

Extending OpenMI for Distributed Computing Connects to remote database via web services Connects to remote model via web services Goal: To allow a modeler to create a workflow from OpenMI components that wrap web services. Model linkage designed on client machine

Conclusion The CUAHSI Hydrologic Information System (HIS) is a geographically distributed network of hydrologic data sources and functions that are integrated using web services so that they function as a connected whole. For more information: