Sharing and publishing data using CUAHSI HIS Outline HIS data publication system WaterML and WaterOneFlow web services Observations data model (ODM) Data loading Data editing and quality control Controlled vocabularies HIS central registration and tagging

HIS Data Publication System Query, Visualize, and Edit data using ODM Tools Analysis Access Discovery Hydroseek GIS Matlab Splus R IDL Java C++ VB HydroExcel HydroGet HydroLink HydroObjects ODM Database Service Registry Hydrotagger GetSites GetSiteInfo GetVariableInfo GetValues WaterOneFlow Web Service WaterML Streaming Data Loader Base Station Computer(s) Telemetry Network Sensors Harvester ODM Data Loader Water Metadata Catalog Excel ODM Text ODM Contribute your ODM HIS Central

Steps in publishing data Establish an HIS Server Load observations into an ODM database Provide access to data through web services (http://<your-server>/<your-network>/cuahsi_1_0.asmx?WSDL) Index the resulting water data service at HIS Central (http://hiscentral.cuahsi.org)

Establishing an HIS Server Windows server platform Base Software: Microsoft SQL and ArcGIS Server HIS Server applications WaterOneFlow web services ODM + tools DASH HIS Data http://his.cuahsi.org/hisserver.html

Load Observations into an ODM Database Groundwater levels Streamflow ODM Soil moisture data Precipitation & Climate Water Quality Flux tower data

Outline HIS data publication system WaterML and WaterOneFlow web services Observations data model (ODM) Data loading Data editing and quality control Controlled vocabularies HIS central registration and tagging

WaterML and WaterOneFlow Locations Variables Time TCEQ Data GetSiteInfo GetVariableInfo GetValues UT Data WaterML Data USGS WaterOneFlow Web Service Data Repositories Client TRANSFORM EXTRACT LOAD WaterML is an XML language for communicating water data WaterOneFlow is a set of web services based on WaterML Slide from David Valentine

WaterOneFlow Web Services Web Application: Data Portal Your application Excel, ArcGIS, Matlab Fortran, C/C++, Visual Basic Hydrologic model ……………. Your operating system Windows, Unix, Linux, Mac Internet Simple Object Access Protocol Web Services Library Slide from David Valentine

WaterOneFlow Set of query functions Returns data in WaterML NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites) Slide from David Valentine 9

WaterML design principles Goal - capture semantics of hydrologic observations discovery and retrieval Role - exchange schema for CUAHSI web services Driven by Hydrologists (community review) ODM USGS NWIS, EPA STORET, Academic Sources Conformance with Open Geospatial Consortium standards. http://www.opengeospatial.org/ For XSD pros, the WaterML schema is at http://his.cuahsi.org/wofws.html Slide from David Valentine

Point Observations Information Model Utah State University Data Source Little Bear River Network GetSites GetSiteInfo Little Bear River at Mendon Rd Sites GetVariableInfo Dissolved Oxygen Variables GetValues 9.78 mg/L, 1 October 2007, 6PM Values {Value, Time, Qualifier, Offset} A data source operates and provides data to an observation network A network is a set of observation sites (stored in a single ODM instance) A site is a point location where one or more variables are measured A variable is a measured property (e.g. describing the flow or quality of water) A value is an observation of a variable at a particular time A qualifier is a symbol that provides additional information about the value An offset allows specification of measurements at various depths in water 11

Building Blocks of WaterML Responses Response Types Key Elements site sourceInfo seriesCatalog variable value queryInfo - Sites - Variables - TimeSeries GetSites GetSiteInfo GetVariableInfo GetValues Slide from David Valentine

Sites response queryInfo name code site seriesCatalog Series how many location seriesCatalog variables Series how many when TimePeriodType Slide from David Valentine 13

VariablesResponseType variable – same as in series element Code, name, units Sites Variables Values Slide from David Valentine 14

GetValues response - timeSeries queryInfo timeSeries sourceInfo – “where” variable – “what” values Sites Variables Values Slide from David Valentine 15

Values Each time series value recorded in value element Timestamp, plus metadata for the value, recorded in element’s attributes qualifier ISO Time value Slide from David Valentine 16

Outline HIS data publication system WaterML and WaterOneFlow web services Observations data model (ODM) Data loading Data editing and quality control Controlled vocabularies HIS central registration and tagging

Why an Observations Data Model Syntactic heterogeneity (File types and formats) Semantic heterogeneity Language for observation attributes (structural) Language to encode observation attribute values (contextual) Publishing and sharing research data Metadata to facilitate unambiguous interpretation Enhance analysis capability

Scope Focus on Hydrologic Observations made at a point Exclude Remote sensing or grid data. These are part of a digital watershed but not suitable for an atomic database model and individual value queries Primarily store raw observations and simple derived information to get data into its most usable form. Limit inclusion of extensively synthesized information and model outputs at this stage.

What are the basic attributes to be associated with each single data value and how can these best be organized? Value DateTime Variable Location Units Interval (support) Accuracy Offset OffsetType/ Reference Point Source/Organization Censoring Data Qualifying Comments Method Quality Control Level Sample Medium Value Type Data Type

CUAHSI Observations Data Model Streamflow Groundwater levels A relational database at the single observation level (atomic model) Stores observation data made at points Metadata for unambiguous interpretation Traceable heritage from raw measurements to usable information Standard format for data sharing Cross dimension retrieval and analysis Precipitation & Climate Soil moisture data Flux tower data Water Quality Space, S Time, T Variables, V s t Vi vi (s,t) “Where” “What” “When” A data value

CUAHSI Observations Data Model http://www.cuahsi.org/his/odm.html

Site Attributes SiteCode, e.g. NWIS:10109000 SiteName, e.g. Logan River Near Logan, UT Latitude, Longitude Geographic coordinates of site LatLongDatum Spatial reference system of latitude and longitude Elevation_m Elevation of the site VerticalDatum Datum of the site elevation Local X, Local Y Local coordinates of site LocalProjection Spatial reference system of local coordinates PosAccuracy_m Positional Accuracy State, e.g. Utah County, e.g. Cache

Observations Data Model Independent of, but can be coupled to Geographic Representation ODM Arc Hydro Feature Waterbody HydroID HydroCode FType Name AreaSqKm JunctionID HydroPoint Watershed DrainID NextDownID ComplexEdgeFeature EdgeType Flowline Shoreline HydroEdge ReachCode LengthKm LengthDown FlowDir Enabled SimpleJunctionFeature 1 HydroJunction DrainArea AncillaryRole * HydroNetwork Observations Data Model Sites 1 1 SiteID SiteCode SiteName OR Latitude Longitude … CouplingTable 1 SiteID HydroID 1

Variable attributes Flow m3/s VariableName, e.g. discharge Cubic meters per second Flow m3/s VariableName, e.g. discharge VariableCode, e.g. NWIS:0060 SampleMedium, e.g. water ValueType, e.g. field observation, laboratory sample IsRegular, e.g. Yes for regular or No for intermittent TimeSupport (averaging interval for observation) DataType, e.g. Continuous, Instantaneous, Categorical GeneralCategory, e.g. Climate, Water Quality NoDataValue, e.g. -9999

Scale issues in the interpretation of data The scale triplet a) Extent b) Spacing c) Support length or time quantity length or time quantity length or time quantity From: Blöschl, G., (1996), Scale and Scaling in Hydrology, Habilitationsschrift, Weiner Mitteilungen Wasser Abwasser Gewasser, Wien, 346 p.

The effect of sampling for measurement scales not commensurate with the process scale (b) extent too small – trend (c) support too large – smoothing out (a) spacing too large – noise (aliasing) From: Blöschl, G., (1996), Scale and Scaling in Hydrology, Habilitationsschrift, Weiner Mitteilungen Wasser Abwasser Gewasser, Wien, 346 p.

Discharge, Stage, Concentration and Daily Average Example

Data Types Continuous (Frequent sampling - fine spacing) Sporadic (Spot sampling - coarse spacing) Cumulative Incremental Average Maximum Minimum Constant over Interval Categorical

15 min Precipitation from NCDC NCDC Precipitation Example Concepts: Use of nodata value in Value field of Values table (denotes beginning and ending of a no data period) Multiple observations of multiple variables at a single site Storage of incremental data with different time support (15 minute incremental vs. 24 hour incremental) Use of data qualifying comments Use of end of interval accumulation of data (the sum of precipitation for a day is reported at the end of the day – midnight of the next day) Relationships: Incomplete or Inexact daily total occurring. Value is not a true 24-hour amount. One or more periods are missing and/or an accumulated amount has begun but not ended during the daily period.

Irregularly sampled groundwater level Groundwater Level Example Concepts: Multiple observations of a single variable at a single site made by a single source Use of a quality control level to qualify data Relationships: Relationship between the Sites table and the Values table on SiteID Relationship between the Values table and the Variables table on VariableID Relationship between the Values table and the Sources table on SourceID Relationship between the Values table and the QualityControlLevelDefinitions table on QualityControlLevel

Offset OffsetValue Distance from a datum or control point at which an observation was made OffsetType defines the type of offset, e.g. distance below water level, distance above ground surface, or distance from bank of river

Water Chemistry from a profile in a lake Water Chemistry From a Lake Profile Concepts: Grouped observations (all observations in one reservoir profile) Observations made using an offset (observations made at multiple depths below the surface of a reservoir) Observations made using a specific method (observations made using a particular field instrument) Relationships: Relationship between Values table and the Variables table on VariableID Relationship between Values table and OffestTypes table on OffsetTypeID Relationship between Values table and Methods table on MethodID Relationship between Variables table and Units table on UnitID Relationship between GroupDescriptions table and Groups table on GroupID Relationship between OffsetTypes table and Units table on UnitID and OffsetUnitID

Groups and Derived From Associations

Stage and Streamflow Example Discharge Derived from Gage Height Concepts: Data derived from other data – single data point derived from a single observation (discharge from stage) Data derived using a specific method (discharge from stage using rating curve) Relationships: Relationships between Values table and DerivedFrom table on DerivedFromID and ValueID Relationship between Values table and Variables table on VariableID Relationship between Values table and Methods table on MethodID Relationship between Variables table and Units table on UnitID

Daily Average Discharge Example Daily Average Discharge Derived from 15 Minute Discharge Data Concepts: Data derived from other data – single data point derived from multiple observations (daily average from 15 minute instantaneous observations) Data derived using a method (daily average by averaging 15 minute instantaneous observations) Relationships: Relationship between Values table and DerivedFrom table on DerivedFromID Relationship between Values table and Methods table on MethodID Relationship between Values table and Variables table on VariableID Relationship between Variables table and Units table on UnitID

Methods and Samples Method specifies the method whereby an observation is measured, e.g. Streamflow using a V notch weir, TDS using a Hydrolab, sample collected in auto-sampler SampleID is used for observations based on the laboratory analysis of a physical sample and identifies the sample from which the observation was derived. This keys to a unique LabSampleID (e.g. bottle number) and name and description of the analytical method used by a processing lab.

Water Chemistry from Laboratory Sample

Low Accuracy, but precise ValueAccuracy A numeric value that quantifies measurement accuracy defined as the nearness of a measurement to the standard or true value. This may be quantified as an average or root mean square error relative to the true value. Since the true value is not known this may should be estimated based on knowledge of the method and measurement instrument. Accuracy is distinct from precision which quantifies reproducibility, but does not refer to the standard or true value. ValueAccuracy Accurate Low Accuracy Low Accuracy, but precise

Data Quality Qualifier Code and Description provides qualifying information about the observations, e.g. Estimated, Provisional, Derived, Holding time for analysis exceeded QualityControlLevel records the level of quality control that the data has been subjected to. - Level 0. Raw Data - Level 1. Quality Controlled Data - Level 2. Derived Products - Level 3. Interpreted Products - Level 4. Knowledge Products

Series of Observations A “Data Series” is a set of all the observations of a particular variable at a site. The SeriesCatalog is programmatically generated to provide users with the ability to do data discovery (i.e. what data is available and where) without formulating complex queries or hitting the DataValues table which can get very large.

Outline HIS data publication system WaterML and WaterOneFlow web services Observations data model (ODM) Data loading Data editing and quality control Controlled vocabularies HIS central registration and tagging

Loading data into ODM Interactive OD Data Loader (OD Loader) Loads data from spreadsheets and comma separated tables in simple format Scheduled Data Loader (SDL) Loads data from datalogger files on a prescribed schedule. Interactive configuration SQL Server Integration Services (SSIS) Microsoft application accompanying SQL Server useful for programming complex loading or data management functions SDL SSIS

Central Observations Database (ODM) Base Station Computer ODM Streaming Data Loader Internet Sensor Network Remote Monitoring Sites Data discovery, visualization, and analysis through Internet enabled applications Radio Repeaters Applications Central Observations Database From Jeff Horsburgh

ODM Streaming Data Loader Loading the Little Bear Sensor Data Into ODM ODM Streaming Data Loader Streaming Data Text Files ODM SDL Mapping Wizard Automate the data loading process via scheduled updates Map datalogger files to the ODM schema and controlled vocabularies XML Config File ODM SDL Import Application ODM SDL manages the periodic insertion of the streaming data into the ODM database using the mappings stored in the XML configuration file. Base Station Computer(s) ODM From Jeff Horsburgh

CUAHSI Observations Data Model 1 3 2 At last … Work from Out to In 4 5 6 7 And don’t forget … CUAHSI Observations Data Model http://www.cuahsi.org/his/odm.html

Managing Data Within ODM - ODM Tools Query and export – export data series and metadata Visualize – plot and summarize data series Edit – delete, modify, adjust, interpolate, average, etc.

Outline HIS data publication system WaterML and WaterOneFlow web services Observations data model (ODM) Data loading Data editing and quality control Controlled vocabularies HIS central registration and tagging

Syntactic Heterogeneity Multiple Data Sources With Multiple Formats Excel Files Text Files ODM Observations Database Access Files Data Logger Files From Jeff Horsburgh

Semantic Heterogeneity General Description of Attribute USGS NWISa EPA STORETb Structural Heterogeneity Code for location at which data are collected "site_no" "Station ID" Name of location at which data are collected "Site" OR "Gage" "Station Name" Code for measured variable "Parameter" ?c Name of measured variable "Description" "Characteristic Name" Time at which the observation was made "datetime" "Activity Start" Code that identifies the agency that collected the data "agency_cd" "Org ID" Contextual Semantic Heterogeneity "Discharge" "Flow" Units of measured variable "cubic feet per second" "cfs" "2008-01-01" "2006-04-04 00:00:00" Latitude of location at which data are collected "41°44'36" "41.7188889" Type of monitoring site "Spring, Estuary, Lake, Surface Water" "River/Stream" a United States Geological Survey National Water Information System (http://waterdata.usgs.gov/nwis/). b United States Environmental Protection Agency Storage and Retrieval System (http://www.epa.gov/storet/). c An equivalent to the USGS parameter code does not exist in data retrieved from EPA STORET. From Jeff Horsburgh

Overcoming Semantic Heterogeneity ODM Controlled Vocabulary System ODM CV central database Online submission and editing of CV terms Web services for broadcasting CVs ODM VariableNameCV Term … Sunshine duration Temperature Turbidity Variable Name Investigator 1: “Temperature, water” Investigator 2: “Water Temperature” Investigator 3: “Temperature” Investigator 4: “Temp.” From Jeff Horsburgh

Dynamic controlled vocabulary moderation system ODM Data Manager ODM Website ODM Tools ODM Controlled Vocabulary Moderator XML Master ODM Controlled Vocabulary Local ODM Database ODM Controlled Vocabulary Web Services Local Server http://his.cuahsi.org/mastercvreg.html From Jeff Horsburgh

Outline HIS data publication system WaterML and WaterOneFlow web services Observations data model (ODM) Data loading Data editing and quality control Controlled vocabularies HIS central registration and tagging

Registering Web Services with HIS Central Listing of all public data services Enables applications like Hydroseek to discover data

Tagging Variables for Data Discovery Through a Metadata Catalog Ontology: A hierarchy of concepts Each Variable in your data is connected to a corresponding Concept From Michael Piasecki 55

Tagging variables in Ontology Steps The WSDL for a set of ODM web services is registered in the WSDL Registry The “harvester” jumps into action and trawls through the web services at the WSDL to find and identify new variables It returns i) data updating information and ii) variable names used and compares these to those used by HydroSeek. WATERS Network Information System From Michael Piasecki 9/19/2018 Department of Civil, Architectural & Environmental Engineering Department of Civil, Architectural & Environmental Engineering 56

Mapping onto Ontology Steps contd. New variables are manually mapped onto appropriate ontology concept. HydroSeek catalogue is updated. From Michael Piasecki 9/19/2018 Department of Civil, Architectural & Environmental Engineering Department of Civil, Architectural & Environmental Engineering 57

Hydroseek http://www.hydroseek.org Supports search by location and type of data across multiple observation networks including NWIS, Storet, and university data

Summary Generic method for publishing observational data Supports many types of point observational data Overcomes syntactic and semantic heterogeneity using a standard data model and controlled vocabularies Supports a national network of observatory test beds but can grow! Web services provide programmatic machine access to data Work with the data in your data analysis software of choice Internet-based applications provide user interfaces for the data and geographic context for monitoring sites