1. CUAHSI-Hydrologic Information Systems
UCAR
CUAHSI – Consortium of Universities for the Advancement of Hydrologic Science, Inc
Formed in 2001 as a legal entity
Program office in Washington (5 staff)
Supported by the National Science Foundation
Unidata
Atmospheric
Sciences
Earth
Sciences
Ocean
Sciences
CUAHSI
National Science Foundation
Geosciences Directorate
HIS

2. CUAHSI Member Institutions
115 Universities as of August 2006

3. CUAHSI Mission: To provide infrastructure and services to advance the development of hydrologic science and education

4. Common Vision: WATERS Network
Informatics
Observatories/ Environmental Field Facilities
Sensors and Measurement Facility
Synthesis
A combined CLEANER-CUAHSI effort

6. Definition
The CUAHSI Hydrologic Information System (HIS) is a geographically distributed network of hydrologic data sources and functions that are integrated using web services so that they function as a connected whole.

7. Goals better Data Access support for Hydrologic Observatories
advancement of Hydrologic Science
enabling Hydrologic Education

8. CUAHSI HIS Project Team
Project co-PI
Collaborator

9. Modeling, Analysis and Visualization
CUAHSI Hydrologic Information System
Experiments
Monitoring
1. Assemble data from many sources
Information Sources
Remote sensing
GIS
Climate models
2. Integrate data into a coherent structure
Hydrologic Information Model
Modeling, Analysis and Visualization
3. Do science
Hypothesis testing
Statistics
Simulation
Data
Assimilation

10. HIS User Assessment First survey done for HIS White Paper (2003)
HIS Symposium in March – 4 institutional surveys and a survey of participants
CUAHSI Web Surveyor – online questionnaire (75 responses from 38 institutions)
Summary paper

11. Please rank these four HIS service categories for helping you.
Value Score (counting 4 for first, 4 for second, 2 for third and 1 for fourth).
Conclusion: Data services are the highest priority

12. % of time spent preparing data

13. Which operating systems do you use for your research
Which operating systems do you use for your research? If you use more than one operating system, select all that apply.

14. Please indicate one dataset that you believe would most benefit from increased ease of access through a Hydrologic Information System (HIS).
Conclusion: EPA STORET Water Quality, Streamflow and Remote Sensing Data are perceived to be able to benefit from improved access.
I am surprised USGS streamflow is up there. Is this an indication of importance over difficulty?

15. How we use software (Austin Symposium)

16. Conclusion: High priorities are: - Data formats - Metadata
Which of the following data analysis difficulties are most important for HIS to address?
Conclusion: High priorities are:
- Data formats
- Metadata
- Irregular time steps
Value Score (counting 3 for first, 2 for second and 1 for third).

17. How we use software (Web Surveyor)
Programming (85% of respondents): Fortran, C/C++, Visual Basic
Data Management (93%): Excel, MS Access
GIS (93%): ArcGIS
Mathematics/Statistics (98%): Excel, Matlab, SAS, variety of other systems
Hydrologic models (80%): Modflow, HEC models
A general, simple, standard, and open interface that could connect with many systems is the only way to accommodate all these
Dat

18. Water quantity and quality
Water Data
Water quantity and quality
Soil water
Rainfall & Snow
Modeling
Meteorology
Remote sensing

19. Water Data Web Sites

20. “Digital Watershed”
How can hydrologists integrate observed and modeled data from various sources into a single description of the environment?
A digital watershed is a synthesis of hydrologic observation
data, geospatial data, remote sensing data and weather
and climate data into a connected database for a hydrologic region

21. HDAS Web portal Interface Web services interface
Information input, display, query and output services
Preliminary data exploration and discovery. See what is available and perform exploratory analyses
Downloads
Uploads
GIS
Matlab
IDL
Splus, R
Excel
Programming (Fortran, C, VB)
Web services interface
HTML -XML
3rd party servers
e.g. USGS, NCDC
Data access through web services
WaterOneFlow Web Services
WSDL - SOAP
Data storage through web services
Hydrologic Information System Service Oriented Architecture
Observatory servers
SDSC HIS servers

22. Applications and Services
Web application: Data Portal
Your application
Excel, ArcGIS, Matlab
Fortran, C/C++, Visual Basic
Hydrologic model
…………….
Your operating system
Windows, Unix, Linux, Mac
Internet
Web Services
Library

23. CUAHSI Hydrologic Data Access System
NCDC
NASA
EPA
NWS
Observatory Data
USGS
Arc Hydro Server will be a customization of ArcGIS Server 9.2 for serving water observational data
David – This slide implies that the CUAHSI website is the path through which people would access this data. I’m not sure that this is really your intent, since the fed agencies are not CUAHSI members and in the long run would probably access this data and use these tools through another path.
But the line at the bottom of the slide should appear somewhere, as it is one of the key messages, if not THE key message.
How do you feel about changing the labeling on this slide to be more consistent since it’s a fed agency audience. Either list agencies, list data sources, or list both.
So USGS could be listed as USGS, as NWIS, or as USGS NWIS
EPA as EPA, STORET or EPA STORET
NCDC as NOAA, NCDC or NOAA NCDC
NWS as NOAA-NWS or NDFD, or NOAA NDFD
NASA – any specific datasets or websites like Giovanni?
A common data window for accessing, viewing and downloading hydrologic information

24. Utah State University Streamflow Analyst

25. Data Sources Extract Transform CUAHSI Web Services Load Applications
NASA
Storet
Ameriflux
Extract
NCDC
Unidata
NWIS
NCAR
Transform
CUAHSI Web Services
Excel
Visual Basic
Shauna – the ones on top are a bunch of web sites providing data that go through the CUAHSI Web Services (basically some software) which allows all this data to be used in all these other software packages listed across the bottom. The ones in red are the ones that are connections that are already built. It would look good to redo this in ESRI style.
ArcGIS
C/C++
Load
Matlab
Fortran
Access
Java
Applications
Some operational services

26. CUAHSI Hydrologic Information System Levels
National HIS – San Diego Supercomputer Center
Map interface, observations catalogs and web services for national data sources; integration of information from workgroups
HIS Server
Workgroup HIS – research group or observatory
Map interface, observations catalogs and web services for regional data sources; observations databases and web services for individual investigator data
Personal HIS – an individual hydrologic scientist
HIS Analyst
Application templates and HydroObjects for direct ingestion of data into analysis environments: Excel, ArcGIS, Matlab, programming languages; MyDB for storage of analysis data

27. HIS Server Supports data discovery, delivery and publication
Data discovery – how do I find the data I want?
Map interface and observations catalogs
Metadata based Search
Data delivery – how do I acquire the data I want?
Use web services or retrieve from local database
Data Publication – how do I publish my observation data?
Use Observations Data Model

28. and how many observations of each variable are available
Observations Catalog
Specifies what variables are measured at each site, over what time interval,
and how many observations of each variable are available

29. HIS Server Architecture
Map front end – ArcGIS Server 9.2 (being programmed by ESRI Water Resources for CUAHSI)
Relational database – SQL/Server 2005 or Express
Web services library – VB.Net programs accessed as a Web Service Description Language (WSDL)

30. National and Workgroup HIS
National HIS
Workgroup HIS
National HIS has a polygon
in it marking the region of
coverage of a workgroup HIS
server
Workgroup HIS has local
observations catalogs for coverage of national data sources in its region. These local catalogs are partitioned from the national observations catalogs.
For HIS 1.0 the National and Workgroup HIS servers will not be dynamically connected.

31. Hydrologic Science
It is as important to represent hydrologic environments precisely with
data as it is to represent hydrologic processes with equations
Physical laws and principles
(Mass, momentum, energy, chemistry)
Hydrologic Process Science
(Equations, simulation models, prediction)
Hydrologic conditions
(Fluxes, flows, concentrations)
Hydrologic Information Science
(Observations, data models, visualization
Hydrologic environment
(Dynamic earth)

32. Continuous Space-Time Model – NetCDF (Unidata)
Time, T
Coordinate
dimensions
{X} D
Space, L
Variable dimensions
{Y}
Variables, V

33. Discrete Space-Time Data Model ArcHydro
Time, TSDateTime
TSValue
Space, FeatureID
Variables, TSTypeID

34. HydroVolumes
Take a watershed and extrude it vertically into the atmosphere
and subsurface
A hydrovolume is “a volume in space through which water, energy and mass flow, are stored internally, and transformed”

35. Watershed Hydrovolumes
Geovolume is the
portion of a hydrovolume
that contains solid
earth materials
USGS Gaging stations

36. Stream channel Hydrovolumes

37. Geospatial Time Series
Properties
(Type)
A Value-Time array
Value
Time
A time series that knows what
geographic feature it describes
and what type of time series it is
Shape

38. Terrain Data Models
Grid
TIN
Contour and flowline

39. Neuse Basin: Coastal aquifer system
Section line
Beaufort Aquifer
* From USGS, Water Resources Data Report of North Carolina for WY 2002

40. Neuse Groundwater Geovolumes of hydrogeologic units
from US Geological survey (GMS)

41. Create a 3 dimensional representation
Geovolume
Each cell in the 2D representation is transformed into a 3D object
Geovolume with model cells

42. The Demands METADATA Air-Q HSPF MM5 NCDC USGS NWIS NCEP NWS NGDC
Page 3
The Demands
Sensor Arrays
Numerical Models
Prediction
HSPF
Air-Q
MM5
Data Centers
NCDC
USGS
NWIS
NCEP
NWS
NGDC
METADATA
Individual
Samples
Drexel University, College of Engineering

43. Hydrologic Processes Sedimentation
Page 21
Hydrologic Metadata
Upper Hydrologic Ontology
We currently have
What we need is
Many More
ISO Temporal Objects
ARCHydro
ISO Geospatial
Many More
USGS Hydrologic Unit Code
ISO Units/Conversion
Hydrologic Processes Sedimentation
Many More
Michael Piasecki is our
expert in this subject!
Many More
Ontology Examples
Drexel University, College of Engineering

44. CUAHSI Observations Data Model
A relational database stored in Access, PostgreSQL, SQLServer, ….
Stores observation data made at points
Access data through web interfaces
Fill using automated data harvesting
Streamflow
Groundwater
levels
Precipitation
& Climate
Soil
moisture
data
Water Quality
Flux tower
data

45. Purposes Premise Hydrologic Observations Data System to Enhance
Retrieval
Integrated Analysis
Multiple Investigators
Standard and Scalable Format for Sharing
Ancillary information (metadata) to allow unambiguous interpretation and use – incorporating uncertainty
Traceable heritage from raw measurements to usable information – quality control levels
Premise
A relational database at the single observation level (atomic model)
Querying capability
Cross dimension retrieval and analysis

46. Community Design Requirements (from comments of 22 reviewers)
Incorporate sufficient metadata to identify provenance and give exact definition of data for unambiguous interpretation
Spatial location of measurements
Scale of measurements
Depth/Offset Information
Censored data
Classification of data type to guide appropriate interpretation
Continuous
Indication of gaps
Indicate data quality

47. Scale issues in the interpretation of data
The scale triplet
a) Extent
b) Spacing
c) Support
From: Blöschl, G., (1996), Scale and Scaling in Hydrology, Habilitationsschrift, Weiner Mitteilungen Wasser Abwasser Gewasser, Wien, 346 p.

48. Hydrologic Observations Data Model
What are the basic attributes to be associated with each single observation and how can these best be organized?
Data Source and Network
Controlled Vocabulary Tables
Sites
Variables
Values
Metadata
e.g. mg/kg, cfs
e.g. depth
Streamflow
Depth of snow pack
Landuse, Vegetation
e.g. Non-detect,Estimated,
Windspeed, Precipitation
A data source operates an observation network
A network is a set of observation sites
A site is a point location where one or more variables are measured
A variable is a property describing the flow or quality of water
A value is an observation of a variable at a particular time
Metadata provide information about the context of the observation.
Data Discovery
Data Delivery
See
Ernest To
Center for Research in Water Resources
University of Texas at Austin

49. Hydrologic Observations Data Model
Independent of, but coupled to Geographic Representation
HODM
Arc Hydro
Feature
Waterbody
HydroID
HydroCode
FType
Name
AreaSqKm
JunctionID
HydroPoint
Watershed
DrainID
NextDownID
ComplexEdgeFeature
EdgeType
Flowline
Shoreline
HydroEdge
ReachCode
LengthKm
LengthDown
FlowDir
Enabled
SimpleJunctionFeature 1
HydroJunction
DrainArea
AncillaryRole *
HydroNetwork
Hydrologic Observations Data Model
MonitoringPoint 1 1
SiteID
SiteCode
SiteName OR
Latitude
Longitude …
CouplingTable
SiteID (GUID)
HydroID (Integer) 1
The Arc Hydro framework is built on a Hydro Network made up of HydroEdges (stream lines) and HydroJunctions (points of interest on the lines). The watersheds, waterbodies and hydropoints are connected to the hydronetwork using relationships with the hydro junctions (blue lines in the diagram). 1

50. NHDPlus as a starting point for geographic representation
Slope
Elevation
Mean annual flow
Corresponding velocity
Drainage area
% of upstream drainage area in different land uses
Stream order

51. Variable attributes L3/T m3/s VariableName, e.g. discharge
Cubic meters per second
L3/T
m3/s
VariableName, e.g. discharge
VariableCode, e.g. 0060
SampleMedium, e.g. water
Valuetype, e.g. field observation, laboratory sample
IsRegular, e.g. Yes for regular or No for intermittent
TimeSupport (averaging interval for observation)
DataType, e.g. Continuous, Instantaneous, Categorical
GeneralCategory, e.g. Climate, Water Quality
NoDataValue, e.g

52. Data Types Continuous (Frequent sampling - fine spacing)
Instantaneous (Spot sampling - coarse spacing)
Cumulative
Incremental
Average
Maximum
Minimum
Constant over Interval
Categorical

53. Groups and Derived From Associations

54. Stage and Streamflow Example

55. Daily Average Discharge Example Daily Average Discharge Derived from 15 Minute Discharge Data

56. Offset
OffsetValue
Distance from a datum or control point at which an observation was made
OffsetType defines the type of offset, e.g. distance below water level, distance above ground surface, or distance from bank of river

57. Water Chemistry from a profile in a lake

58. Methods and Samples
Method specifies the method whereby an observation is measured, e.g. Streamflow using a V notch weir, TDS using a Hydrolab, sample collected in auto-sampler
SampleID is used for observations based on the laboratory analysis of a physical sample and identifies the sample from which the observation was derived. This keys to a unique LabSampleID (e.g. bottle number) and name and description of the analytical method used by a processing lab.

59. Accuracy and Precision
ObsAccuracyStdDev
Numeric value that expresses measurement accuracy as the standard deviation of each specific observation

60. Observation Series
An Series is a set of all the observations of a particular variable at one place, i.e. with unique SiteID. The ObservationSeriesCatalog is programatically generated to provide a means by which a user can get simple descriptive information about the variables observed at a location.

61. Data Quality
Qualifier Code and Description provides qualifying information about the observations, e.g. Estimated, Provisional, Derived, Holding time for analysis exceeded
QualityControlLevel records the level of quality control that the data has been subjected to. - Level 0. Raw Data - Level 1. Quality Controlled Data - Level 2. Derived Products - Level 3. Interpreted Products - Level 4. Knowledge Products

62. 15 min Precipitation from NCDC

63. Irregularly sampled groundwater level

64. How Excel connects to ODM
HydroObjects
CUAHSI Web service
Obtains inputs for CUAHSI web methods from relevant cells.
Available Web methods are GetSiteInfo, GetVariableInfo GetValues methods.
parses user inputs into a standardized CUAHSI web method request.
converts standardized request to SQLquery.
SQL query
Observations
Data
Model
Response
converts response to a standardized XML.
imports VB object into Excel and graphs it
converts XML to VB object

65. Example: Matlab use of CUAHSI Web Services
% create NWIS class and an instance of the class.
createClassFromWsdl('
svsNWIS = NWIS;
xmlSites=GetSites(svsNWIS); % Could parse to identify sites to work with.
SiteID=' '; % Here specify a SiteID to use
% Call the GetSiteInfo function
xmlSiteInfo=GetSiteInfo(svsNWIS,SiteID)
% Parse the XML that is returned to learn the variables recorded there
structSiteInfo=parse_xml(xmlSiteInfo)
… (non trivial)
% Call the GetVariableInfo function to get details about each variable
xmlVarInfo=GetVariableInfo(svsNWIS,varcodes(i));
structVarInfo=parse_xml(xmlVarInfo);
% Parse to write results to html file for display
… (non trivial)

66. NWIS Site Information Generated using Web Services in Matlab

67. Retrieve Data using GetValues
xmlVals=GetValues(svsNWIS,SiteID,varcodes(1),D1,D2);
% Parse the xml string that is returned into matrices and plot
strValues=parse_xml(xmlVals);
… (non trivial)
plot(dn,Q);datetick;

68. Conclusions
HIS = a geographically distributed system of web-connected data and functions
Hydrologic Data Access System is a significant technological innovation
Emerging understanding of digital watershed structure and functions
Beginnings of hydrologic information science and shared data models with neighboring sciences
Web services provide access to HIS capability from within a users preferred analysis environment