1. Sharing and publishing data using CUAHSI HIS
Outline
HIS data publication system
WaterML and WaterOneFlow web services
Observations data model (ODM)
Data loading
Data editing and quality control
Controlled vocabularies
HIS central registration and tagging

2. HIS Data Publication System
Query, Visualize, and Edit data using ODM Tools
Analysis
Access
Discovery
Hydroseek
GIS
Matlab
Splus R
IDL
Java
C++ VB
HydroExcel
HydroGet
HydroLink
HydroObjects
ODM
Database
Service Registry
Hydrotagger
GetSites
GetSiteInfo
GetVariableInfo
GetValues
WaterOneFlow
Web Service
WaterML
Streaming Data Loader
Base Station
Computer(s)
Telemetry Network
Sensors
Harvester
ODM Data Loader
Water Metadata Catalog
Excel
ODM
Text
ODM
Contribute your ODM
HIS Central

3. Steps in publishing data
Establish an HIS Server
Load observations into an ODM database
Provide access to data through web services (
Index the resulting water data service at HIS Central (

4. Establishing an HIS Server
Windows server platform
Base Software: Microsoft SQL and ArcGIS Server
HIS Server applications
WaterOneFlow web services
ODM + tools
DASH
HIS Data

5. Load Observations into an ODM Database
Groundwater
levels
Streamflow
ODM
Soil
moisture
data
Precipitation
& Climate
Water Quality
Flux tower
data

6. Outline HIS data publication system
WaterML and WaterOneFlow web services
Observations data model (ODM)
Data loading
Data editing and quality control
Controlled vocabularies
HIS central registration and tagging

7. WaterML and WaterOneFlow
Locations
Variables
Time
TCEQ
Data
GetSiteInfo
GetVariableInfo
GetValues UT
Data
WaterML
Data
USGS
WaterOneFlow
Web Service
Data
Repositories
Client
TRANSFORM
EXTRACT
LOAD
WaterML is an XML language for communicating water data
WaterOneFlow is a set of web services based on WaterML
Slide from David Valentine

8. WaterOneFlow Web Services
Web Application: Data Portal
Your application
Excel, ArcGIS, Matlab
Fortran, C/C++, Visual Basic
Hydrologic model
…………….
Your operating system
Windows, Unix, Linux, Mac
Internet
Simple Object Access Protocol
Web Services
Library
Slide from David Valentine

9. WaterOneFlow Set of query functions Returns data in WaterML
NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)
Slide from David Valentine 9

10. WaterML design principles
Goal - capture semantics of hydrologic observations discovery and retrieval
Role - exchange schema for CUAHSI web services
Driven by
Hydrologists (community review)
ODM
USGS NWIS, EPA STORET, Academic Sources
Conformance with Open Geospatial Consortium standards.
For XSD pros, the WaterML schema is at
Slide from David Valentine

11. Point Observations Information Model
Utah State
University
Data Source
Little Bear River
Network
GetSites
GetSiteInfo
Little Bear River at Mendon Rd
Sites
GetVariableInfo
Dissolved Oxygen
Variables
GetValues
9.78 mg/L, 1 October 2007, 6PM
Values
{Value, Time, Qualifier, Offset}
A data source operates and provides data to an observation network
A network is a set of observation sites (stored in a single ODM instance)
A site is a point location where one or more variables are measured
A variable is a measured property (e.g. describing the flow or quality of water)
A value is an observation of a variable at a particular time
A qualifier is a symbol that provides additional information about the value
An offset allows specification of measurements at various depths in water 11

12. Building Blocks of WaterML Responses
Response Types
Key Elements
site
sourceInfo
seriesCatalog
variable
value
queryInfo
- Sites
- Variables
- TimeSeries
GetSites
GetSiteInfo
GetVariableInfo
GetValues
Slide from David Valentine

13. Sites response queryInfo name code site seriesCatalog Series how many
location
seriesCatalog
variables
Series
how many
when
TimePeriodType
Slide from David Valentine 13

14. VariablesResponseType
variable – same as in series element
Code, name, units
Sites
Variables
Values
Slide from David Valentine 14

15. GetValues response - timeSeries
queryInfo
timeSeries
sourceInfo – “where”
variable – “what”
values
Sites
Variables
Values
Slide from David Valentine 15

16. Values Each time series value recorded in value element
Timestamp, plus metadata for the value, recorded in element’s attributes
qualifier
ISO Time
value
Slide from David Valentine 16

17. Outline HIS data publication system
WaterML and WaterOneFlow web services
Observations data model (ODM)
Data loading
Data editing and quality control
Controlled vocabularies
HIS central registration and tagging

18. Why an Observations Data Model
Syntactic heterogeneity (File types and formats)
Semantic heterogeneity
Language for observation attributes (structural)
Language to encode observation attribute values (contextual)
Publishing and sharing research data
Metadata to facilitate unambiguous interpretation
Enhance analysis capability

19. Scope Focus on Hydrologic Observations made at a point
Exclude Remote sensing or grid data. These are part of a digital watershed but not suitable for an atomic database model and individual value queries
Primarily store raw observations and simple derived information to get data into its most usable form.
Limit inclusion of extensively synthesized information and model outputs at this stage.

20. What are the basic attributes to be associated with each single data value and how can these best be organized?
Value
DateTime
Variable
Location
Units
Interval (support)
Accuracy
Offset
OffsetType/
Reference Point
Source/Organization
Censoring
Data Qualifying Comments
Method
Quality Control Level
Sample Medium
Value Type
Data Type

21. CUAHSI Observations Data Model
Streamflow
Groundwater
levels
A relational database at the single observation level (atomic model)
Stores observation data made at points
Metadata for unambiguous interpretation
Traceable heritage from raw measurements to usable information
Standard format for data sharing
Cross dimension retrieval and analysis
Precipitation
& Climate
Soil
moisture
data
Flux tower
data
Water Quality
Space, S
Time, T
Variables, V s t Vi
vi (s,t)
“Where”
“What”
“When”
A data value

22. CUAHSI Observations Data Model

23. Site Attributes SiteCode, e.g. NWIS:10109000
SiteName, e.g. Logan River Near Logan, UT
Latitude, Longitude Geographic coordinates of site
LatLongDatum Spatial reference system of latitude and longitude
Elevation_m Elevation of the site
VerticalDatum Datum of the site elevation
Local X, Local Y Local coordinates of site
LocalProjection Spatial reference system of local coordinates
PosAccuracy_m Positional Accuracy
State, e.g. Utah
County, e.g. Cache

24. Observations Data Model
Independent of, but can be coupled to Geographic Representation
ODM
Arc Hydro
Feature
Waterbody
HydroID
HydroCode
FType
Name
AreaSqKm
JunctionID
HydroPoint
Watershed
DrainID
NextDownID
ComplexEdgeFeature
EdgeType
Flowline
Shoreline
HydroEdge
ReachCode
LengthKm
LengthDown
FlowDir
Enabled
SimpleJunctionFeature 1
HydroJunction
DrainArea
AncillaryRole *
HydroNetwork
Observations Data Model
Sites 1 1
SiteID
SiteCode
SiteName OR
Latitude
Longitude …
CouplingTable 1
SiteID
HydroID 1

25. Variable attributes Flow m3/s VariableName, e.g. discharge
Cubic meters per second
Flow
m3/s
VariableName, e.g. discharge
VariableCode, e.g. NWIS:0060
SampleMedium, e.g. water
ValueType, e.g. field observation, laboratory sample
IsRegular, e.g. Yes for regular or No for intermittent
TimeSupport (averaging interval for observation)
DataType, e.g. Continuous, Instantaneous, Categorical
GeneralCategory, e.g. Climate, Water Quality
NoDataValue, e.g

26. Scale issues in the interpretation of data
The scale triplet
a) Extent
b) Spacing
c) Support
length or time
quantity
length or time
quantity
length or time
quantity
From: Blöschl, G., (1996), Scale and Scaling in Hydrology, Habilitationsschrift, Weiner Mitteilungen Wasser Abwasser Gewasser, Wien, 346 p.

27. The effect of sampling for measurement scales not commensurate with the process scale
(b) extent too small – trend
(c) support too large – smoothing out
(a) spacing too large – noise (aliasing)
From: Blöschl, G., (1996), Scale and Scaling in Hydrology, Habilitationsschrift, Weiner Mitteilungen Wasser Abwasser Gewasser, Wien, 346 p.

28. Discharge, Stage, Concentration and Daily Average Example

29. Data Types Continuous (Frequent sampling - fine spacing)
Sporadic (Spot sampling - coarse spacing)
Cumulative
Incremental
Average
Maximum
Minimum
Constant over Interval
Categorical

30. 15 min Precipitation from NCDC
NCDC Precipitation Example
Concepts:
Use of nodata value in Value field of Values table (denotes beginning and ending of a no data period)
Multiple observations of multiple variables at a single site
Storage of incremental data with different time support (15 minute incremental vs. 24 hour incremental)
Use of data qualifying comments
Use of end of interval accumulation of data (the sum of precipitation for a day is reported at the end of the day – midnight of the next day)
Relationships:
Incomplete or Inexact daily total occurring. Value is not a true 24-hour amount. One or more periods are missing and/or an accumulated amount has begun but not ended during the daily period.

31. Irregularly sampled groundwater level
Groundwater Level Example
Concepts:
Multiple observations of a single variable at a single site made by a single source
Use of a quality control level to qualify data
Relationships:
Relationship between the Sites table and the Values table on SiteID
Relationship between the Values table and the Variables table on VariableID
Relationship between the Values table and the Sources table on SourceID
Relationship between the Values table and the QualityControlLevelDefinitions table on QualityControlLevel

32. Offset
OffsetValue
Distance from a datum or control point at which an observation was made
OffsetType defines the type of offset, e.g. distance below water level, distance above ground surface, or distance from bank of river

33. Water Chemistry from a profile in a lake
Water Chemistry From a Lake Profile
Concepts:
Grouped observations (all observations in one reservoir profile)
Observations made using an offset (observations made at multiple depths below the surface of a reservoir)
Observations made using a specific method (observations made using a particular field instrument)
Relationships:
Relationship between Values table and the Variables table on VariableID
Relationship between Values table and OffestTypes table on OffsetTypeID
Relationship between Values table and Methods table on MethodID
Relationship between Variables table and Units table on UnitID
Relationship between GroupDescriptions table and Groups table on GroupID
Relationship between OffsetTypes table and Units table on UnitID and OffsetUnitID

34. Groups and Derived From Associations

35. Stage and Streamflow Example
Discharge Derived from Gage Height
Concepts:
Data derived from other data – single data point derived from a single observation (discharge from stage)
Data derived using a specific method (discharge from stage using rating curve)
Relationships:
Relationships between Values table and DerivedFrom table on DerivedFromID and ValueID
Relationship between Values table and Variables table on VariableID
Relationship between Values table and Methods table on MethodID
Relationship between Variables table and Units table on UnitID

36. Daily Average Discharge Example Daily Average Discharge Derived from 15 Minute Discharge Data
Concepts:
Data derived from other data – single data point derived from multiple observations (daily average from 15 minute instantaneous observations)
Data derived using a method (daily average by averaging 15 minute instantaneous observations)
Relationships:
Relationship between Values table and DerivedFrom table on DerivedFromID
Relationship between Values table and Methods table on MethodID
Relationship between Values table and Variables table on VariableID
Relationship between Variables table and Units table on UnitID

37. Methods and Samples
Method specifies the method whereby an observation is measured, e.g. Streamflow using a V notch weir, TDS using a Hydrolab, sample collected in auto-sampler
SampleID is used for observations based on the laboratory analysis of a physical sample and identifies the sample from which the observation was derived. This keys to a unique LabSampleID (e.g. bottle number) and name and description of the analytical method used by a processing lab.

38. Water Chemistry from Laboratory Sample

39. Low Accuracy, but precise
ValueAccuracy
A numeric value that quantifies measurement accuracy defined as the nearness of a measurement to the standard or true value. This may be quantified as an average or root mean square error relative to the true value. Since the true value is not known this may should be estimated based on knowledge of the method and measurement instrument. Accuracy is distinct from precision which quantifies reproducibility, but does not refer to the standard or true value.
ValueAccuracy
Accurate
Low Accuracy
Low Accuracy, but precise

40. Data Quality
Qualifier Code and Description provides qualifying information about the observations, e.g. Estimated, Provisional, Derived, Holding time for analysis exceeded
QualityControlLevel records the level of quality control that the data has been subjected to. - Level 0. Raw Data - Level 1. Quality Controlled Data - Level 2. Derived Products - Level 3. Interpreted Products - Level 4. Knowledge Products

41. Series of Observations
A “Data Series” is a set of all the observations of a particular variable at a site.
The SeriesCatalog is programmatically generated to provide users with the ability to do data discovery (i.e. what data is available and where) without formulating complex queries or hitting the DataValues table which can get very large.

42. Outline HIS data publication system
WaterML and WaterOneFlow web services
Observations data model (ODM)
Data loading
Data editing and quality control
Controlled vocabularies
HIS central registration and tagging

43. Loading data into ODM Interactive OD Data Loader (OD Loader)
Loads data from spreadsheets and comma separated tables in simple format
Scheduled Data Loader (SDL)
Loads data from datalogger files on a prescribed schedule.
Interactive configuration
SQL Server Integration Services (SSIS)
Microsoft application accompanying SQL Server useful for programming complex loading or data management functions
SDL
SSIS

44. Central Observations Database
(ODM)
Base Station
Computer
ODM Streaming
Data Loader
Internet
Sensor Network
Remote Monitoring Sites
Data discovery, visualization, and analysis through Internet enabled applications
Radio Repeaters
Applications
Central Observations Database
From Jeff Horsburgh

45. ODM Streaming Data Loader
Loading the
Little Bear
Sensor Data
Into ODM
ODM Streaming Data Loader
Streaming Data Text Files
ODM SDL Mapping Wizard
Automate the data loading process via scheduled updates
Map datalogger files to the ODM schema and controlled vocabularies
XML Config File
ODM SDL Import Application
ODM SDL manages the periodic insertion of the streaming data into the ODM database using the mappings stored in the XML configuration file.
Base Station
Computer(s)
ODM
From Jeff Horsburgh

46. CUAHSI Observations Data Model1 3 2
At last …
Work from Out to In 4 5 6 7
And don’t forget …
CUAHSI Observations Data Model

47. Managing Data Within ODM - ODM Tools
Query and export – export data series and metadata
Visualize – plot and summarize data series
Edit – delete, modify, adjust, interpolate, average, etc.

48. Outline HIS data publication system
WaterML and WaterOneFlow web services
Observations data model (ODM)
Data loading
Data editing and quality control
Controlled vocabularies
HIS central registration and tagging

49. Syntactic Heterogeneity
Multiple Data Sources
With Multiple Formats
Excel
Files
Text
Files
ODM Observations
Database
Access
Files
Data Logger
Files
From Jeff Horsburgh

50. Semantic Heterogeneity
General Description of Attribute
USGS NWISa
EPA STORETb
Structural Heterogeneity
Code for location at which data are collected
"site_no"
"Station ID"
Name of location at which data are collected
"Site" OR "Gage"
"Station Name"
Code for measured variable
"Parameter" ?c
Name of measured variable
"Description"
"Characteristic Name"
Time at which the observation was made
"datetime"
"Activity Start"
Code that identifies the agency that collected the data
"agency_cd"
"Org ID"
Contextual Semantic Heterogeneity
"Discharge"
"Flow"
Units of measured variable
"cubic feet per second"
"cfs"
" "
" :00:00"
Latitude of location at which data are collected
"41°44'36"
" "
Type of monitoring site
"Spring, Estuary, Lake, Surface Water"
"River/Stream"
a United States Geological Survey National Water Information System (
b United States Environmental Protection Agency Storage and Retrieval System (
c An equivalent to the USGS parameter code does not exist in data retrieved from EPA STORET.
From Jeff Horsburgh

51. Overcoming Semantic Heterogeneity
ODM Controlled Vocabulary System
ODM CV central database
Online submission and editing of CV terms
Web services for broadcasting CVs
ODM VariableNameCV
Term …
Sunshine duration
Temperature
Turbidity
Variable Name
Investigator 1:
“Temperature, water”
Investigator 2:
“Water Temperature”
Investigator 3:
“Temperature”
Investigator 4:
“Temp.”
From Jeff Horsburgh

52. Dynamic controlled vocabulary moderation system
ODM Data Manager
ODM Website
ODM
Tools
ODM Controlled
Vocabulary Moderator
XML
Master ODM Controlled Vocabulary
Local ODM
Database
ODM
Controlled Vocabulary
Web Services
Local
Server
From Jeff Horsburgh

53. Outline HIS data publication system
WaterML and WaterOneFlow web services
Observations data model (ODM)
Data loading
Data editing and quality control
Controlled vocabularies
HIS central registration and tagging

54. Registering Web Services with HIS Central
Listing of all public data services
Enables applications like Hydroseek to discover data

55. Tagging Variables for Data Discovery Through a Metadata Catalog
Ontology: A hierarchy of concepts
Each Variable in your data is connected to a corresponding Concept
From Michael Piasecki 55

56. Tagging variables in Ontology
Steps
The WSDL for a set of ODM web services is registered in the WSDL Registry
The “harvester” jumps into action and trawls through the web services at the WSDL to find and identify new variables
It returns i) data updating information and ii) variable names used and compares these to those used by HydroSeek.
WATERS Network Information System
From Michael Piasecki
9/19/2018
Department of Civil, Architectural & Environmental Engineering
Department of Civil, Architectural & Environmental Engineering 56

57. Mapping onto Ontology Steps contd.
New variables are manually mapped onto appropriate ontology concept.
HydroSeek catalogue is updated.
From Michael Piasecki
9/19/2018
Department of Civil, Architectural & Environmental Engineering
Department of Civil, Architectural & Environmental Engineering 57

58. Hydroseek http://www.hydroseek.org
Supports search by location and type of data across multiple observation networks including NWIS, Storet, and university data

59. Summary Generic method for publishing observational data
Supports many types of point observational data
Overcomes syntactic and semantic heterogeneity using a standard data model and controlled vocabularies
Supports a national network of observatory test beds but can grow!
Web services provide programmatic machine access to data
Work with the data in your data analysis software of choice
Internet-based applications provide user interfaces for the data and geographic context for monitoring sites