Hydroinformatics Lecture: HydroServer .NET/PHP CUAHSI HIS Sharing hydrologic data Hydroinformatics Lecture: HydroServer .NET/PHP http://his.cuahsi.org/ Daniel P. Ames and Jeffery S. Horsburgh The CUAHSI HIS is Supported by NSF Grant# EAR 0622374

CUAHSI HIS Metadata Services Metadata Search Data Services The CUAHSI Hydrologic Information System (HIS) is an internet based system to support the sharing of hydrologic data. It is comprised of hydrologic databases and servers connected through web services as well as software for data publication, discovery and access. Data Discovery and Integration platform Data Publication platform Data Synthesis and Research platform Data Services Metadata Services Metadata Search HIS Central HydroDesktop HydroServer Service registration Catalog harvesting Service and data theme metadata Data carts Water Data Services Spatial Data Services Like search portals Google, Yahoo, Bing Like browsers Like web servers Like HTML Welcome to our week for discussing HydroServer and HydroServer Lite as tools for publishing your data from an ODM using the Water One Flow web services and web based user interfaces. As you recall, the CUAHSI Hydrologic Information System is comprised of three components which follow the internet’s common Services Oriented Architecture (SOA). HydroServer and similar software tools plays an important role in the CUAHSI HIS SOA as the primary data provider in the system. It’s role is similar to that of a web server, though the HydroServer needs NO user interface to function on the HIS network since it’s primary role is to share data through web services.

HydroServer Goals A platform for organizing and publishing space-time hydrologic datasets that: Provides local control of data Makes data universally available Is autonomous – e.g., functional independent of the rest of HIS Put the data service as close to the data producer as possible. Share the data as widely and openly as possible. Some of the key goals of the HydroServer effort are given here. Serve as a useful data store regardless of the fate of CUAHSI HIS.

Capabilities Database Configuration Tool Add on Visualization Capabilities KEY HydroServer Functionality: Read ODM Share it with WebServices Capabilities Database Configuration Tool HydroServer Capabilities Database WaterOneFlow Services Add on GIS Services through ArcGIS Server Spatial Services HydroServer software can be found at hydroserver.codeplex.com WaterOneFlow WaterOneFlow WaterOneFlow ODM ODM ODM ODM Databases and Web Services ArcGIS Server Spatial Data Services

HydroServer Key Concepts: Regions A region is an experimental watershed, research site, etc. for which data are published Little Bear River Experimental Watershed Reynolds Creek Experimental Watershed T.W. Daniels Experimental Forest Organizing concept for HydroServer One or more ODM databases/WaterOneFlow services for a region One or more geospatial data services for a region

HydroServer Key Concepts: ODM Soil moisture data Streamflow Flux tower data Groundwater levels Water Quality Precipitation & Climate Hopefully by now you are all familiar with the ODM database model. This database model is at the core of HydroServer and stores the data that are shared through web services. A relational database at the single observation level Metadata for unambiguous interpretation Traceable heritage from raw measurements to usable information Promote syntactic and semantic consistency Cross dimension retrieval and analysis Horsburgh, J. S., D. G. Tarboton, D. R. Maidment, and I. Zaslavsky (2008), A relational model for environmental and water resources data, Water Resources Research, 44, W05406, doi:10.1029/2007WR006392.

ODM 1.1.1 Let’s take another closer look at ODM.

What are the basic attributes to be associated with each single data value and how can these best be organized? DateTime Interval (support) Space, S Time, T Variables, V s t Vi vi (s,t) “Where” “What” “When” A data value Units Accuracy Censoring Qualifying comments Variable Method Quality Control Level Sample Medium Value Type Data Type Source/Organization Location Feature of interest

Data Series – Metadata Description Space Variable, Vi Site, Sj End Date Time, t2 Begin Date Time, t1 Time Variables Count, C There are C measurements of Variable Vi at Site Sj from time t1 to time t2

Series Catalog Sj Space Variable, Vi Site, Sj End Date Time, t2 Begin Date Time, t1 Time Variables Count, C Vi t1 t2 C

ODM Controlled Vocabularies HIS CV Website ODM Data Manager ODM Tools ODM Controlled Vocabulary Moderator XML Master ODM Controlled Vocabulary Local ODM Database ODM Controlled Vocabulary Web Services Local HydroServer

MOST IMPORTANT WAY: Using WaterML and WaterOneFlow How do we get data out? MOST IMPORTANT WAY: Using WaterML and WaterOneFlow Set of query functions Returns data in WaterML WaterML is an XML schema for communicating water data WaterOneFlow is a set of web services based on WaterML Let’s look at an example… http://river.sdsc.edu/wateroneflow/NWIS/UnitValues.asmx We will explore this example a bit to see what the services do. Show get sites and get site info. Slide from David Valentine 12

For example: to get a site, send this “SOAP” request to the server: Specify the site and an optional authorization token

Get this response: Result returns in this string. We may look at a couple more examples online. Result returns in this string.

2 Flavors of HydroServer: HydroServer (.NET) http://hydroserver.codeplex.com Model implementation of the HydroServer CUAHSI HIS specification Runs on Windows servers Requires Microsoft SQL Server Uses ArcGIS Server Uses Streaming Data Loader, Data Loader and SQL tools to manage the database. These are the two flavors of HydroServer that we will discuss in this class. Both are free and open source. Both can be downloaded from the web and installed on your own server. Both can be registered on the CUAHSI HIS system.

Loading Data into ODM Interactive ODM Data Loader Loads data from spreadsheets and comma separated tables in simple format Streaming Data Loader (SDL) Loads data from datalogger files on a prescribed schedule Interactive configuration SQL Server Integration Services (SSIS) Microsoft application accompanying SQL Server useful for programming complex loading or data management functions ODM SDL SSIS

Editing Data in HydroServer using “ODM Tools” Query and export – export data series and metadata Visualize – plot and summarize data series Edit – delete, modify, adjust, interpolate, average, etc.

HydroServer Web Data Loader The general format for input to the ODMWDL is a single file containing a table with a one row header that uses ODM field names in the header, followed by the data in subsequent rows.

Where do we get these tools? ODM Tools, Streaming Data Loader, etc. can all be downloaded from the HydroServer web site: Web Data Loader… ODM Tools Data Loader Streaming Data Loader

Publication of Geospatial (GIS) Datasets Publishing spatial datasets using ArcGIS Server Using OGC standards that can be consumed by a number of GIS clients WMS, WFS, WCS

Data Presentation Via a Map Interface Internet Map Server built using ArcGIS Web browser client Combine spatial data and observational data Launch data visualization tools Based on a “Region” http://maps.usu.edu/map/

Data Preview, Visualization, and Analysis Time Series Analyst Web Browser Client Multiple ODM Database Support Variety of plot types Descriptive statistics Linked to the map application Data preview and download Let’s try playing with the time series analyst on the web site. Let’s try it! http://icewater.usu.edu/tsa/

HydroServer Website HydroServer home page Observational data services GIS data services Online map Time Series Analyst Data Query and Download http://icewater.usu.edu

HydroServer Capabilities Database Configuration Tool HydroServer Capabilities Database Ties together Map Services and HIS Services HydroServer Capabilities Database Configuration Tool Simple Windows application for creating metadata records in your database Associating Regions with services and maps

HydroServer Capabilities Web Service Publish capabilities of each HydroServer Listing of published observational data services Listing of published spatial data services Supports automatic cataloging of available services at HIS Central Makes HydroServers self describing Catalog Server Desktop Metadata Services Catalog Services Data Services

How do I create a HydroServer? Get a Windows Server Machine with IIS and .Net Framework Install Microsoft SQL Server Install HydroServer Software from his.cuahsi.org ODM and Utilities WaterOneFlow web services HydroServer Website, TSA, Map, Capabilities Install ArcGIS Create data services and document them in the capabilities database Register your services with HIS Central and register your variables with the HIS Ontology so your data will be searchable

HydroServer System Requirements Required Hardware (Generally less than $5000) Can be one or more physical or virtual machine(s) Dual or quad-core processor at 2+ GHz > 4 GB of RAM > 500 GB of hard drive space Network adapter Required Commercial Software Microsoft Windows 2003 or 2008 Server (preferred) IIS Web Server .NET Framework Version 3.5 Microsoft SQL Server 2005 or 2008 (preferred) Any edition – Express (free), Standard, Enterprise ESRI ArcGIS Server 9.3.1

2 Flavors of HydroServer: HydroServer PHP (HydroServer Lite) http://hydroserverlite.codeplex.com Simplified implementation of the CUAHSI HIS specification (all required services are supported). Runs on the free LAMP stack (Linux, Apache, MySQL, PHP) Uses front end web pages to load data in the database. These are the two flavors of HydroServer that we will discuss in this class. Both are free and open source. Both can be downloaded from the web and installed on your own server. Both can be registered on the CUAHSI HIS system.

What is HydroServer Lite? Lots of the same ideas, but different implementation PHP and MySQL mean easy/fast/cheap setup Focus on easy extends to user interfaces. Developer community at http://hydroserverlite.codeplex.com/ Let’s try it out: http://worldwater.byu.edu/ Browse data from any database, upload data using the “sandbox” database.

Can Citizen Science Data be Shared and Intermingled with “Real Data”?

(Interactive Graphing Tool…) (Custom Banner…) (Map Based Interface…) (Interactive Graphing Tool…)

HydroServers for the Developing World http://worldwater.byu.edu Custom PHP-based Catalog for Developing Countries

Internationalization Effort

CUAHSI HIS Sharing hydrologic data Build your own HydroServer Lite instance by following the instructions at : http://hydroserverlite.codeplex.com/ Support EAR 0622374