1. Ocean Sciences CUAHSI-Hydrologic Information Systems CUAHSI – Consortium of Universities for the Advancement of Hydrologic Science, Inc Formed in 2001 as a legal entity Program office in Washington (5 staff) Supported by the National Science Foundation Earth Sciences Atmospheric Sciences UCAR CUAHSI Unidata HIS National Science Foundation Geosciences Directorate

2. CUAHSI Member Institutions 115 Universities as of August 2006

3. CUAHSI Mission: To provide infrastructure and services to advance the development of hydrologic science and education

4. Common Vision: WATERS Network Sensors and Measurement Facility Synthesis Informatics Observatories/ Environmental Field Facilities A combined CLEANER-CUAHSI effort

6. Definition The CUAHSI Hydrologic Information System (HIS) is a geographically distributed network of hydrologic data sources and functions that are integrated using web services so that they function as a connected whole.

7. Goals better Data Access support for Hydrologic Observatories advancement of Hydrologic Science enabling Hydrologic Education

8. Project co-PI Collaborator CUAHSI HIS Project Team

9. Information Sources Modeling, Analysis and Visualization Hydrologic Information Model CUAHSI Hydrologic Information System GIS Experiments Simulation Monitoring Climate models 2. Integrate data into a coherent structure 3. Do science 1. Assemble data from many sources Hypothesis testing Data Assimilation Remote sensing Statistics

10. HIS User Assessment First survey done for HIS White Paper (2003) HIS Symposium in March – 4 institutional surveys and a survey of participants CUAHSI Web Surveyor – online questionnaire (75 responses from 38 institutions) Summary paper

11. Please rank these four HIS service categories for helping you. Conclusion: Data services are the highest priority Value Score (counting 4 for first, 4 for second, 2 for third and 1 for fourth).

12. % of time spent preparing data

13. Which operating systems do you use for your research? If you use more than one operating system, select all that apply.

14. Please indicate one dataset that you believe would most benefit from increased ease of access through a Hydrologic Information System (HIS). Conclusion: EPA STORET Water Quality, Streamflow and Remote Sensing Data are perceived to be able to benefit from improved access. I am surprised USGS streamflow is up there. Is this an indication of importance over difficulty?

15. How we use software (Austin Symposium)

16. Which of the following data analysis difficulties are most important for HIS to address? Conclusion: High priorities are: - Data formats - Metadata - Irregular time steps Value Score (counting 3 for first, 2 for second and 1 for third).

17. How we use software (Web Surveyor) Programming (85% of respondents): Fortran, C/C++, Visual Basic Data Management (93%): Excel, MS Access GIS (93%): ArcGIS Mathematics/Statistics (98%): Excel, Matlab, SAS, variety of other systems Hydrologic models (80%): Modflow, HEC models A general, simple, standard, and open interface that could connect with many systems is the only way to accommodate all these

18. Rainfall & Snow Water quantity and quality Remote sensing Water Data Modeling Meteorology Soil water

19. Water Data Web Sites

20. “Digital Watershed” How can hydrologists integrate observed and modeled data from various sources into a single description of the environment? A digital watershed is a synthesis of hydrologic observation data, geospatial data, remote sensing data and weather and climate data into a connected database for a hydrologic region

21. WaterOneFlow Web Services Data accessthrough web services Data storage through web services Downloads Uploads Observatory servers SDSC HIS servers 3 rd party servers e.g. USGS, NCDC GIS Matlab IDL Splus, R Excel Programming (Fortran, C, VB) Web services interface HDAS Web portal Interface Information input, display, query and output services Preliminary data exploration and discovery. See what is available and perform exploratory analyses HTML -XML WSDL - SOAP

22. Applications and Services Web Services Library Web application: Data Portal Your application Excel, ArcGIS, Matlab Fortran, C/C++, Visual Basic Hydrologic model ……………. Your operating system Windows, Unix, Linux, Mac Internet

23. CUAHSI Hydrologic Data Access System A common data window for accessing, viewing and downloading hydrologic information USGS NASANCDC EPANWS Observatory Data http://river.sdsc.edu/HDAS Arc Hydro Server will be a customization of ArcGIS Server 9.2 for serving water observational data

24. Utah State University Streamflow Analyst

25. NWIS ArcGIS Excel NCAR Unidata NASA Storet NCDC Ameriflux Matlab AccessJava Fortran Visual Basic C/C++ Some operational services CUAHSI Web Services Data Sources Applications Extract Transform Load http://www.cuahsi.org/his/

26. CUAHSI Hydrologic Information System Levels National HIS – San Diego Supercomputer Center Workgroup HIS – research group or observatory Personal HIS – an individual hydrologic scientist HIS Server HIS Analyst Map interface, observations catalogs and web services for national data sources; integration of information from workgroups Map interface, observations catalogs and web services for regional data sources; observations databases and web services for individual investigator data Application templates and HydroObjects for direct ingestion of data into analysis environments: Excel, ArcGIS, Matlab, programming languages; MyDB for storage of analysis data

27. HIS Server Supports data discovery, delivery and publication –Data discovery – how do I find the data I want? Map interface and observations catalogs Metadata based Search –Data delivery – how do I acquire the data I want? Use web services or retrieve from local database –Data Publication – how do I publish my observation data? Use Observations Data Model

28. Observations Catalog Specifies what variables are measured at each site, over what time interval, and how many observations of each variable are available

29. HIS Server Architecture Map front end – ArcGIS Server 9.2 (being programmed by ESRI Water Resources for CUAHSI) Relational database – SQL/Server 2005 or Express Web services library – VB.Net programs accessed as a Web Service Description Language (WSDL)

30. National and Workgroup HIS National HIS has a polygon in it marking the region of coverage of a workgroup HIS server Workgroup HIS has local observations catalogs for coverage of national data sources in its region. These local catalogs are partitioned from the national observations catalogs. For HIS 1.0 the National and Workgroup HIS servers will not be dynamically connected. National HISWorkgroup HIS

31. Hydrologic Science Hydrologic conditions (Fluxes, flows, concentrations) Hydrologic Process Science (Equations, simulation models, prediction) Hydrologic Information Science (Observations, data models, visualization Hydrologic environment (Dynamic earth) Physical laws and principles (Mass, momentum, energy, chemistry) It is as important to represent hydrologic environments precisely with data as it is to represent hydrologic processes with equations

32. Continuous Space-Time Model – NetCDF (Unidata) Space, L Time, T Variables, V D Coordinate dimensions {X} Variable dimensions {Y}

33. Space, FeatureID Time, TSDateTime Variables, TSTypeID TSValue Discrete Space-Time Data Model ArcHydro

34. HydroVolumes Take a watershed and extrude it vertically into the atmosphere and subsurface A hydrovolume is “a volume in space through which water, energy and mass flow, are stored internally, and transformed”

35. Watershed Hydrovolumes Geovolume is the portion of a hydrovolume that contains solid earth materials USGS Gaging stations Hydrovolume

36. Stream channel Hydrovolumes

37. Geospatial Time Series Value Time Shape Time Series Properties (Type) A Value-Time array A time series that knows what geographic feature it describes and what type of time series it is

38. Terrain Data Models Grid Contour and flowline TIN

39. Neuse Basin: Coastal aquifer system * From USGS, Water Resources Data Report of North Carolina for WY 2002 Section line Beaufort Aquifer

40. Neuse Groundwater Geovolumes of hydrogeologic units from US Geological survey (GMS)

41. Create a 3 dimensional representation Geovolume Each cell in the 2D representation is transformed into a 3D object Geovolume with model cells

42. Page 3 Drexel University, College of Engineering Data Centers NCDC USGS NWIS NCEP NWS NGDC Sensor Arrays Numerical Models Prediction HSPF Air-Q MM5 Individual Samples METADATA The Demands

43. ISO 19103 Units/Conversion Page 21 Drexel University, College of Engineering Ontology Examples Hydrologic Metadata We currently have ISO 19108 Temporal ObjectsUSGS Hydrologic Unit CodeISO 19115 Geospatial Hydrologic Processes Sedimentation ARCHydro What we need is Many More Upper Hydrologic Ontology Michael Piasecki is our expert in this subject!

44. CUAHSI Observations Data Model A relational database stored in Access, PostgreSQL, SQLServer, …. Stores observation data made at points Access data through web interfaces Fill using automated data harvesting Streamflow Flux tower data Precipitation & Climate Groundwater levels Water Quality Soil moisture data

45. Purposes Hydrologic Observations Data System to Enhance –Retrieval –Integrated Analysis –Multiple Investigators Standard and Scalable Format for Sharing Ancillary information (metadata) to allow unambiguous interpretation and use – incorporating uncertainty Traceable heritage from raw measurements to usable information – quality control levels Premise A relational database at the single observation level (atomic model) –Querying capability –Cross dimension retrieval and analysis

46. Community Design Requirements (from comments of 22 reviewers) Incorporate sufficient metadata to identify provenance and give exact definition of data for unambiguous interpretation Spatial location of measurements Scale of measurements Depth/Offset Information Censored data Classification of data type to guide appropriate interpretation –Continuous –Indication of gaps Indicate data quality

47. Scale issues in the interpretation of data The scale triplet From: Blöschl, G., (1996), Scale and Scaling in Hydrology, Habilitationsschrift, Weiner Mitteilungen Wasser Abwasser Gewasser, Wien, 346 p. a) Extentb) Spacing c) Support

48. Hydrologic Observations Data Model What are the basic attributes to be associated with each single observation and how can these best be organized? See CUAHSI Community Hydrologic Observations Data Model Working Design Specifications Document http://www.cuahsi.org/his/documentation.html http://www.cuahsi.org/his/documentation.html

49. Ernest To Center for Research in Water Resources University of Texas at Austin 20061011 What are the basic attributes to be associated with each single observation and how can these best be organized? A data source operates an observation network A network is a set of observation sites Data Source and Network SitesVariablesValuesMetadata Depth of snow pack Streamflow Landuse, Vegetation Windspeed, Precipitation Data Delivery Controlled Vocabulary Tables e.g. mg/kg, cfs e.g. depth e.g. Non-detect,Estimated, A site is a point location where one or more variables are measured Metadata provide information about the context of the observation. A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time Data Discovery Hydrologic Observations Data Model See http://www.cuahsi.org/his/documentation.htmlhttp://www.cuahsi.org/his/documentation.html

50. 1 1 CouplingTable SiteID (GUID) HydroID (Integer) MonitoringPoint SiteID SiteCode SiteName Latitude Longitude … Hydrologic Observations Data Model 1 1 OR Independent of, but coupled to Geographic Representation HODM Arc Hydro

51. NHDPlus as a starting point for geographic representation Slope Elevation Mean annual flow –Corresponding velocity Drainage area % of upstream drainage area in different land uses Stream order

52. Variable attributes VariableName, e.g. discharge VariableCode, e.g. 0060 SampleMedium, e.g. water Valuetype, e.g. field observation, laboratory sample IsRegular, e.g. Yes for regular or No for intermittent TimeSupport (averaging interval for observation) DataType, e.g. Continuous, Instantaneous, Categorical GeneralCategory, e.g. Climate, Water Quality NoDataValue, e.g. -9999 m 3 /s L 3 /T Cubic meters per second

53. Data Types Continuous (Frequent sampling - fine spacing) Instantaneous (Spot sampling - coarse spacing) Cumulative Incremental Average Maximum Minimum Constant over Interval Categorical

54. Groups and Derived From Associations

55. Stage and Streamflow Example

56. Daily Average Discharge Example Daily Average Discharge Derived from 15 Minute Discharge Data

57. Offset OffsetValue Distance from a datum or control point at which an observation was made OffsetType defines the type of offset, e.g. distance below water level, distance above ground surface, or distance from bank of river

58. Water Chemistry from a profile in a lake

59. Methods and Samples Method specifies the method whereby an observation is measured, e.g. Streamflow using a V notch weir, TDS using a Hydrolab, sample collected in auto-sampler SampleID is used for observations based on the laboratory analysis of a physical sample and identifies the sample from which the observation was derived. This keys to a unique LabSampleID (e.g. bottle number) and name and description of the analytical method used by a processing lab.

60. Accuracy and Precision ObsAccuracyStdDev Numeric value that expresses measurement accuracy as the standard deviation of each specific observation

61. Observation Series An Series is a set of all the observations of a particular variable at one place, i.e. with unique SiteID. The ObservationSeriesCatalog is programatically generated to provide a means by which a user can get simple descriptive information about the variables observed at a location.

62. Data Quality Qualifier Code and Description provides qualifying information about the observations, e.g. Estimated, Provisional, Derived, Holding time for analysis exceeded QualityControlLevel records the level of quality control that the data has been subjected to. - Level 0. Raw Data - Level 1. Quality Controlled Data - Level 2. Derived Products - Level 3. Interpreted Products - Level 4. Knowledge Products

63. 15 min Precipitation from NCDC

64. Irregularly sampled groundwater level

65. ExcelCUAHSI Web service How Excel connects to ODM Obtains inputs for CUAHSI web methods from relevant cells. Available Web methods are GetSiteInfo, GetVariableInfo GetValues methods. converts standardized request to SQLquery. imports VB object into Excel and graphs it converts response to a standardized XML. Observations Data Model SQL query Response HydroObjects converts XML to VB object parses user inputs into a standardized CUAHSI web method request.

66. Example: Matlab use of CUAHSI Web Services % create NWIS class and an instance of the class. createClassFromWsdl('http://river.sdsc.edu/NWISTS/nwis.asmx?WSDL'); svsNWIS = NWIS; xmlSites=GetSites(svsNWIS); % Could parse to identify sites to work with. SiteID='10109000'; % Here specify a SiteID to use % Call the GetSiteInfo function xmlSiteInfo=GetSiteInfo(svsNWIS,SiteID) % Parse the XML that is returned to learn the variables recorded there structSiteInfo=parse_xml(xmlSiteInfo) … (non trivial) % Call the GetVariableInfo function to get details about each variable xmlVarInfo=GetVariableInfo(svsNWIS,varcodes(i)); structVarInfo=parse_xml(xmlVarInfo); % Parse to write results to html file for display … (non trivial)

67. NWIS Site Information Generated using Web Services in Matlab

68. Retrieve Data using GetValues xmlVals=GetValues(svsNWIS,SiteID,varcodes(1),D1,D2); % Parse the xml string that is returned into matrices and plot strValues=parse_xml(xmlVals); … (non trivial) plot(dn,Q);datetick;

69. Conclusions HIS = a geographically distributed system of web-connected data and functions Hydrologic Data Access System is a significant technological innovation Emerging understanding of digital watershed structure and functions Beginnings of hydrologic information science and shared data models with neighboring sciences Web services provide access to HIS capability from within a users preferred analysis environment