Hydrologic Data and Modeling: Towards Hydrologic Information Science David R. Maidment Center for Research in Water Resources University of Texas at Austin EPSCorR, Vermont November 10, 2008

Hydrologic Data and Modeling New knowledge in hydrology Hydrologic data Hydrologic modeling Hydrologic information systems

Hydrologic Data and Modeling New knowledge in hydrology Hydrologic data Hydrologic modeling Hydrologic information systems

How is new knowledge discovered? By deduction from existing knowledge By experiment in a laboratory By observation of the natural environment After completing the Handbook of Hydrology in 1993, I asked myself the question: how is new knowledge discovered in hydrology? I concluded:

Deduction – Isaac Newton Deduction is the classical path of mathematical physics –Given a set of axioms –Then by a logical process –Derive a new principle or equation In hydrology, the St Venant equations for open channel flow and Richard’s equation for unsaturated flow in soils were derived in this way. (1687) Three laws of motion and law of gravitation

Experiment – Louis Pasteur Experiment is the classical path of laboratory science – a simplified view of the natural world is replicated under controlled conditions In hydrology, Darcy’s law for flow in a porous medium was found this way. Pasteur showed that microorganisms cause disease & discovered vaccination Foundations of scientific medicine

Observation – Charles Darwin Observation – direct viewing and characterization of patterns and phenomena in the natural environment In hydrology, Horton discovered stream scaling laws by interpretation of stream maps Published Nov 24, 1859 Most accessible book of great scientific imagination ever written

Conclusion for Hydrology Deduction and experiment are important, but hydrology is primarily an observational science discharge, water quality, groundwater, measurement data collected to support this.

Great Eras of Synthesis Scientific progress occurs continuously, but there are great eras of synthesis – many developments happening at once that fuse into knowledge and fundamentally change the science Physics (relativity, structure of the atom, quantum mechanics) Geology (observations of seafloor magnetism lead to plate tectonics) Hydrology (synthesis of water observations leads to knowledge synthesis) 2020

Hydrologic Science Hydrologic conditions (Fluxes, flows, concentrations) Hydrologic Process Science (Equations, simulation models, prediction) Hydrologic Information Science (Observations, data models, visualization Hydrologic environment (Physical earth) Physical laws and principles (Mass, momentum, energy, chemistry) It is as important to represent hydrologic environments precisely with data as it is to represent hydrologic processes with equations

Hydrologic Data and Modeling New knowledge in hydrology Hydrologic data Hydrologic modeling Hydrologic information systems

CUAHSI Member Institutions 122 Universities as of July 2008 (and CSIRO!)

HIS Team and Collaborators University of Texas at Austin – David Maidment, Tim Whiteaker, Ernest To, Bryan Enslein, Kate Marney San Diego Supercomputer Center – Ilya Zaslavsky, David Valentine, Tom Whitenack Utah State University – David Tarboton, Jeff Horsburgh, Kim Schreuders, Justin Berger Drexel University – Michael Piasecki, Yoori Choi University of South Carolina – Jon Goodall, Tony Castronova CUAHSI Program Office – Rick Hooper, David Kirschtel, Conrad Matiuk National Science Foundation Grant EAR

HIS Goals Data Access – providing better access to a large volume of high quality hydrologic data; Hydrologic Observatories – storing and synthesizing hydrologic data for a region; Hydrologic Science – providing a stronger hydrologic information infrastructure; Hydrologic Education – bringing more hydrologic data into the classroom.

HIS Overview Report Summarizes the conceptual framework, methodology, and application tools for HIS version 1.1 Shows how to develop and publish a CUAHSI Water Data Service Available at:

Rainfall & Snow Water quantity and quality Remote sensing Water Data Modeling Meteorology Soil water

Water Data Web Sites

HTML as a Web Language Text and Pictures in Web Browser Vermont EPSCoR --> HyperText Markup Language

WaterML as a Web Language Discharge of the San Marcos River at Luling, TX June 28 - July 18, 2002 Streamflow data in WaterML language

Services-Oriented Architecture for Water Data Links geographically distributed information servers through internet Web Services Description Language (WSDL from W3C) We designed WaterML as a web services language for water data Functions for computer to computer interaction HIS Servers in the WATERS Network HIS Central at San Diego Supercomputer Center Web Services

HIS Central National Water Metadata Catalog WaterML Get Data Get Metadata

CUAHSI Point Observation Data Services 1.Data Loading –Put data into the CUAHSI Observations Data Model 2.Data Publishing –Provide web services access to the data 3.Data Indexing –Summarize the data in a centralized cataloging system

CUAHSI Point Observation Data Services 1.Data Loading –Put data into the CUAHSI Observations Data Model 2.Data Publishing –Provide web services access to the data 3.Data Indexing –Summarize the data in a centralized cataloging system

Data Values – indexed by “What-where- when” Space, S Time, T Variables, V s t ViVi v i (s,t) “Where” “What” “When” A data value

Data Values Table Space, S Time, T Variables, V s t ViVi v i (s,t)

Observations Data Model Horsburgh, J. S., D. G. Tarboton, D. R. Maidment and I. Zaslavsky, (2008), "A Relational Model for Environmental and Water Resources Data," Water Resour. Res., 44: W05406, doi: /2007WR

11 WATERS Network test bed projects 16 ODM instances (some test beds have more than one ODM instance) Data from 1246 sites, of these, 167 sites are operated by WATERS investigators National Hydrologic Information Server San Diego Supercomputer Center HIS Implementation in WATERS Network Information System

CUAHSI Point Observation Data Services 1.Data Loading –Put data into the CUAHSI Observations Data Model 2.Data Publishing –Provide web services access to the data 3.Data Indexing –Summarize the data in a centralized cataloging system

Point Observations Information Model Data Source Network Sites Variables Values {Value, Time, Metadata} Utah State Univ Little Bear River Little Bear River at Mendon Rd Dissolved Oxygen 9.78 mg/L, 1 October 2007, 5PM A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time A metadata quantity provides additional information about the value GetSites GetSiteInfo GetVariableInfo GetValues

Assemble Data From Different Sources Ingest data using ODM Data Loader Load Newly Formatted Data into ODM Tables in MS SQL/Server Wrap ODM with WaterML Web Services for Online Publication Utah State University University of Florida Texas A&M Corpus Christi Publishing an ODM Water Data Service USU ODM UFL ODM TAMUCC ODM Observations Data Model (ODM) WaterML

Snotel DataValues Snotel METADATA ODM WaterML Metadata From: ODM Database in San Diego, CA Snotel Web Site in Portland, OR Snotel Water Data Service Publishing a Hybrid Water Data Service Snotel Metadata are Transferred to the ODM Web Services can both Query the ODM for Metadata and use a Web Scraper for Data Values Calling the WSDL Returns Metadata and Data Values as if from the same Database Get Values from:

Locations Variable Codes Date Ranges WaterML and WaterOneFlow GetSiteInfo GetVariableInfo GetValues WaterOneFlow Web Service Client Penn State Utah State NWIS Data Repositories Data EXTRACT TRANSFORM LOAD WaterML WaterML is an XML language for communicating water data WaterOneFlow is a set of web services based on WaterML

WaterOneFlow Set of query functions Returns data in WaterML

CUAHSI Point Observation Data Services 1.Data Loading –Put data into the CUAHSI Observations Data Model 2.Data Publishing –Provide web services access to the data 3.Data Indexing –Summarize the data in a centralized cataloging system

Data Series – Metadata description Space Variable, V i Site, S j End Date Time, t 2 Begin Date Time, t 1 Time Variables Count, C There are C measurements of Variable V i at Site S j from time t 1 to time t 2

Series Catalog Space Variable, V i Site, S j End Date Time, t 2 Begin Date Time, t 1 Time Variables Count, C ViVi SjSj t2t2 t1t1 C

Texas Hydrologic Information System Sponsored by the Texas Water Development Board and using CUAHSI technology for state and local data sources (using state funding)

CUAHSI National Water Metadata Catalog Indexes: 50 observation networks 1.75 million sites 8.38 million time series 342 million data values NWIS STORET TCEQ

Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them Data Searching NWIS NARR NAWQA NAM-12 request request return return Searching each data source separately Michael Piasecki Drexel University

Semantic Mediation Searching all data sources collectively NWIS NAWQA NARR generic request GetValues GetValues HODM Michael Piasecki Drexel University

Hydroseek Supports search by location and type of data across multiple observation networks including NWIS and Storet Bora Beran, Drexel

HydroTagger Ontology: A hierarchy of concepts Each Variable in your data is connected to a corresponding Concept

NWIS ArcGIS Excel Academic Unidata NASA Storet NCDC Snotel Matlab Java Visual Basic Operational services CUAHSI Web Services Data Sources Applications Extract Transform Load

HydroExcel

HydroGET: An ArcGIS Web Service Client

Direct analysis from your favorite analysis environment. e.g. Matlab % create NWIS Class and an instance of the class createClassFromWsdl(' /NWIS/DailyValues.asmx?WSDL'); WS = NWISDailyValues; % GetValues to get the data siteid='NWIS: '; bdate=' T00:00:00'; edate=' T00:00:00'; variable='NWIS:00060'; valuesxml=GetValues(WS,siteid,variable,bdate,edate,'');

National Water Metadata Catalog Synthesis and communication of the nation’s water data HydroseekWaterML Government Water Data Academic Water Data

Hydrologic Data and Modeling New knowledge in hydrology Hydrologic data Hydrologic modeling Hydrologic information systems

Project sponsored by the European Commission to promote integration of water models within the Water Framework Directive Software standards for model linking Uses model core as an “engine”

OpenMI – Links Data and Simulation Models CUAHSI Observations Data Model as an OpenMI component Simple River Model Trigger (identifies what value should be calculated)

Typical model architecture Application User interface + engine Engine Simulates a process – flow in a channel Accepts input Provides output Model An engine set up to represent a particular location e.g. a reach of the Thames Engine Output data Input data Model application Run Write Read User interface

AcceptsProvides Rainfall (mm) Runoff (m 3 /s) Temperature (Deg C) Evaporation (mm) AcceptsProvides Upstream Inflow (m 3 /s) Outflow (m 3 /s) Lateral inflow (m 3 /s) Abstractions (m 3 /s) Discharges (m 3 /s) River Model Linking modelled quantities

Data transfer at run time Rainfall runoff Output data Input data User interface River Output data Input data User interface GetValues(..)

Models for the processes River (InfoWorks RS) Rainfall (database) Sewer (Mouse) RR (Sobek-Rainfall -Runoff)

Data exchange 3 Rainfall.GetValues River (InfoWorks-RS) Rainfall (database) Sewer (Mouse) 2 RR.GetValues 7 RR.GetValues RR (Sobek-Rainfall -Runoff) 1 Trigger.GetValues 6 Sewer.GetValues call data

Hydrologic Data and Modeling New knowledge in hydrology Hydrologic data Hydrologic modeling Hydrologic information systems

Space, L Time, T Variable, V D Data Cube – What, Where, When “What” “Where” “When” A data value

Continuous Space-Time Data Model -- NetCDF Space, L Time, T Variables, V D Coordinate dimensions {X} Variable dimensions {Y}

Space, FeatureID Time, TSDateTime Variables, TSTypeID TSValue Discrete Space-Time Data Model

Geostatistics Time Series Analysis Multivariate analysis Hydrologic Statistics How do we understand space-time correlation fields of many variables?

CUAHSI Hydrologic Information Systems A system for integrating water data and models CUAHSI HIS team invites EPSCoR scientists to publish their data using CUAHSI Water Data Services and to help us build HIS Desktop during 2009 Observations ModelsClimate GIS