1. CUAHSI HIS Service Oriented Architecture, Web Services, WaterML

2. Test bed HIS Servers Central HIS servers GIS Matlab IDL Splus, R Excel Programming (Fortran, C, VB) Desktop clients Customizable web interface (DASH) HTML - XML WSDL - SOAP Hydrologic Information System Service Oriented Architecture Modeling (OpenMI) Global search (Hydroseek) WaterOneFlow Web Services, WaterML Controlled vocabularies Metadata catalogs Ontology ETL services HIS Lite Servers External data providers Deployment to test beds Other popular online clients ODM DataLoader Streaming Data Loading Ontology tagging (Hydrotagger) WSDL and ODM registration Data publishing ODMTools Server config tools

3. Point Observations Information Model A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water An observation series is an array of observations at a given site, for a given variable, with start time and end time A value is an observation of a variable at a particular time A qualifier is a symbol that provides additional information about the value Data Source Network Sites Observation Series Values {Value, Time, Qualifier} USGS Streamflow gages Neuse River near Clayton, NC Discharge, stage, start, end (Daily or instantaneous) 206 cfs, 13 August 2006 Return network information, and variable information within the network Return site information, including a series catalog of variables measured at a site with their periods of record Return time series of values

4. WaterML design principles Driven largely by hydrologists; the goal is to capture semantics of hydrologic observations discovery and retrieval Relies to a large extent on the information model as in ODM (Observations Data Model), and terms are aligned as much as possible –Several community reviews since 2005 Driven by data served by USGS NWIS, EPA STORET, multiple individual PI-collected observations Is no more than an exchange schema for CUAHSI web services The least barrier for adoption by hydrologists A fairly simple and rigid schema tuned to the current implementation Conformance with OGC specs not in the initial scope – but working with OGC on this

5. - From different database structures, data collection procedures, quality control, access mechanisms  to uniform signatures … Water Markup Language - Tested in different environments - Standards-based - Can support advanced interfaces via harvested catalogs - Accessible to community - Templates for development of new services - Optimized, error handling, memory management, versioning, run from fast servers - Working with agencies on setting up services and updating site files

6. Locations Variable Codes Date Ranges WaterML and WaterOneFlow GetSiteInfo GetVariableInfo GetValues WaterOneFlow Web Service Client STORET NAM NWIS Data Repositories Data EXTRACT TRANSFORM LOAD WaterML WaterML is an XML language for communicating water data WaterOneFlow is a set of web services based on WaterML

7. WaterOneFlow Set of query functions Returns data in WaterML NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)

8. WaterML key elements Response Types –SiteInfo –Variables –TimeSeries Key Elements –site –sourceInfo –seriesCatalog –variable –timeSeries values –queryInfo GetValues GetVariableInfo GetSiteInfo

9. variables variablesResponse variable 1 many timeSeriesqueryInfo criteria timeSeriesResponse variable sourceInfo queryURL values site queryInfo criteria sitesResponse seriesCatalog siteInfo queryURL variable series variableTimeInterval 1 many Structure of responses

10. More Information about WaterML… next 20 slides… Or check the specification online at http://www.opengeospatial.org/standards/dp

11. DatasetInfoTypeSiteInfoType Elements Defining Spatial Location for observation sitesfor continuous surfaces (other site information) SourceInfoType GeogLocationType (other dataset information) LatLonPointType child elements GeogLocationType LatLonBoxType LatLonPointType

12. SiteInfoResponseType Namespaces queryInfo site Network Sites Variables

13. queryInfo example user parameters query URL Parameters sent to service URLs called (if external resource)

14. siteInfo Name Site Code Location

15. geoLocation geogLocation – geographic coordinates –LatLon point –LatLon box localSiteXY – projected coordinates

16. series variable – what is measured valueCount – how many measurements variableTimeInterval – when is it measured TimePeriodType

17. variable variableCode – global identifier variableName units Sites Variables Values TimePeriodType

18. variableTimeInterval TimePeriodType – date range (including “last n days” TimeInstantType – single measurement Compare with…

19. SiteInfo response queryInfo site name code location seriesCatalog variables Series how many when TimePeriodType

20. VariablesResponseType variable – same as in series element Code, name, units Sites Variables Values

21. TimeSeriesResponseType queryInfo timeSeries –sourceInfo – “where” –variable – “what” –values Sites Variables Values

22. sourceInfo SiteInfoType –Same as siteInfo element –code, name, location DataSetInfoType –For data continuous in space –LatLonPointType –LatLonBoxType

23. values Each time series value recorded in value element Timestamp, plus metadata for the value, recorded in element’s attributes ISO Timevalue qualifier

24. value metadata examples qualifiers censorCode (lt, gt, nc) qualityControlLevel (Raw, QC’d, etc.) methodID offset –offsetValue –offsetUnitsAbbreviation –offsetDescription –offsetUnitsCode

25. TimeSeries response queryInfo location variable values

26. OGC Harmonization Best Practices WaterML text includes steps for harmonizing with GML/O&M Align spatial feature descriptions (e.g. using gml:Point, gml:Envelope) Align service signatures (getCapabilities) Align terminology with O&M

27. Newest Developments USGS NWIS Daily Values web service EPA WQX services NCDC prototyping WaterML as format for data delivery

28. USGS Values –SDSC hosts a database catalog of USGS sites and series information –GetValues method now hosted at USGS Follows the CUAHSI Webservices, and returns WaterML TimeSeriesResponse Our service now proxies the USGS service instead of scraping the web site –More services to be developed (Real Time is next)

29. EPA Web Services EPA now provides web services http://www.epa.gov/storet/web_services.html http://www.epa.gov/storet/web_services.html The web services uses WQX, an implementation of the Environmental Sampling, Analysis and Results data standard. Using EPA Webservices, instead of scrapping is over an order of magnitude increase. Issues: WQX is based on the EPA data model (org-(Analysis-Location-(Result))) whereas WaterML is time-series oriented (Site-Variable- (Result). –We are working on mapping WQX in WaterML.

30. EPA WXQ

31. Mapping WQX results to WaterML TimeSeries Organization –Activity ActivityDescription –ActivityStartDate –Details MonitoringLocation –StationID and Name only Result –Result Details »CharacteristicName (variable) »[ResultMeasureValue,Unit] ResultMeasure (DataValue) »Qualifier –BiologicalResultDescription »Details –ResultLabInformation »AnalysisStartDate –ResultAnalyticalMethod –Activity ActivityDescription –ActivityStartDate TimeSeries Site SiteInfo Variable VariableName Units Values DataValue DateTime Value Qualifiers Method DataValue DateTime Value Qualifiers Method Qualifier Methods WQX WaterML call to StationWebService WQX-Each activity produces one (or more) WaterML DataValue

32. StoretResultService GetResults(Organization, MonitoringLocation, … CharacteristicName)

33. Hydrologic Information System Server: Software Stack, Deployment, Operation

34. Test bed HIS Servers Central HIS servers GIS Matlab IDL Splus, R Excel Programming (Fortran, C, VB) Desktop clients Customizable web interface (DASH) HTML - XML WSDL - SOAP Hydrologic Information System Service Oriented Architecture Modeling (OpenMI) Global search (Hydroseek) WaterOneFlow Web Services, WaterML Controlled vocabularies Metadata catalogs Ontology ETL services HIS Lite Servers External data providers Deployment to test beds Other popular online clients ODM DataLoader Streaming Data Loading Ontology tagging (Hydrotagger) WSDL and ODM registration Data publishing ODMTools Server config tools

35. Hydrologic Information Server Supports data discovery, delivery and publication –Data discovery – how do I find the data I want? Map interface and observations catalogs Metadata based Search –Data delivery – how do I acquire the data I want? Use web services or retrieve from local database –Data Publication – how do I publish my observation data? Use Observations Data Model

36. Hydrologic Information Server Microsoft SQLServer Relational Database Observations Data & Catalogs Geospatial Data GetSites GetSiteInfo GetVariables GetVariableInfo GetValues DASH – data access system for hydrologyWaterOneFlow services ArcGIS Server

37. Deployment Overview ► HIS Server Machines are staged at SDSC  Base software components installed (Microsoft, ESRI)  All HIS components installed: ► ODM ► Web Services and templates ► DASH = Data Access System for Hydrology ► + additional tools ► Servers can be accessed remotely, before shipped to testbeds

38. Windows 2003 Server 4 GB Ram 500 GB Disk Quad Core CPU SQLServer 2005 IIS (Internet Information Server) DASH ArcGIS Server Visual Studio 2005 WaterOneFlow Web Services ArcGIS 9.2 GIS Data Mxd Services ODDataLoader ODM tools COTS Software HIS Applications Data NWIS DV NWIS IID Your ODM

40. C: 50Gb O: 230Gb P: 180Gb System Disk Operating System Program Files - SQL Server - IIS - Visual Studio - ArcGIS - ArcGIS Server Data Disk GIS Data SQL Data (ODM) HIS software -WaterOneFlow Web Services - DASH - ODM Data Loader -ODM Tools -WSTestPage

41. SQL Server ODMs and catalogs. All instances exposed as ODM (i.e. have standard ODM tables or views: Sites, Variables, SeriesCatalog, etc.) NWIS-IID NWIS-DV ASOS STORET TCEQ BearRiver... Spatial store Geodatabase or collection of shapefiles or both NWIS-IID points NWIS-DV points ASOS points STORET points TCEQ points BearRiver points... My new ODM My new points More databases More synced layers DASH Web Application Background layers (can be in the same or separate spatial store) WOF services Web services from a common template NWIS-IID WS NWIS-DV WS ASOS WS STORET WS TCEQ WS BearRiver WS... My new WS More WS from ODM-WS template USGS NCDC EPA TCEQ Web Configuration file Stores information about registered networks MXD Stores information about layers WSDLs, web service URLs Connection strings Layer info, symbology, etc. ODM DataLoader 2 6 5 3 1 4 WORKGROUP HIS SERVER ORGANIZATION STEPS FOR REGISTERING OBSERVATION DATA

42. New network registration steps Using the ODM DataLoader or other, load your data into a blank ODM instance (this will create all ODM tables that HIS relies on) Copy Web Services template to a new folder, edit the template web.config file to point to the new ODM, test to make sure the new service works as expected Create a point layer (a feature class in GDB, or a shapefile) from the new ODM’s Sites table using the GetSitesTool. Add the point layer to the MXD document, specify symbology, scale-dependent rendering, etc. Add information about the new ODM, the associated web service, and the associated point layer, to HIS configuration file (see the first slide for the exact content) Restart the HIS service Register and test the new service at the HIS Central: http://water.sdsc.edu/centralhis/ 6 5 4 3 2 1 7

43. Administration and Updates ► Admin accounts: local + remote (for SDSC, troubleshooting) ► Updating software:  DASH and ODM Tools ► New versions on web site, with installation instructions  ODM Data Loader ► ClickOnce deployment  Regular software updates and patches for COTS ► Need to first try at SDSC; post on wiki ► Updating databases:  Regularly updated at SDSC; available to workgroups via web services and direct connection to disrupter.sdsc.edu ► Updating web services  New templates posted on web site, with instructions Information for developers: river.sdsc.edu/wiki/

46. Challenges: information model (1/3) Sites –STORET has stations, and measurement points, at various offsets… –Site metadata lacking and inconsistent (e.g. 2/3 no HUC info, 1/3 no state/county info); agency site files need to be upgraded to ODM… –A groundwater site is different than a stream gauge… Censored values –Values have qualifiers, such as “less than”, “censored”, etc. – per value. Sometimes mixed data types.. Units –There are multiple renditions of the same units, even within one repository –There may be several units for the same parameter code (STORET) –If no value recorded – there are no units?? –Unit multipliers E.g. NCDC ASOS keeps measurements as integers, and provides a multiplier for each variable Sources –STORET requires organization IDs (which collected data for STORET) in addition to site IDs Time stamps: ISO 8601 –Many in local times; how to convert? (UTC offsets given lat/lon and date?)

47. Values retrieval –USGS: by site, variable, time range –EPA: by organization-site, variable, medium, units, time range –NCDC: fewer variables, period of record applies to site, not to seriesCatalog Now: web services stated to appear at agencies – more later Variable semantics –Variable names and measurement methods don’t match E.g. NWIS parameter # 625 is labeled ‘ammonia + organic nitrogen‘, Kjeldahl method is used for determination but not mentioned in parameter description. In STORET this parameter is referred to as Kjeldahl Nitrogen. –One-to-one mapping not always possible E.g. NWIS: ‘bed sediment’ and ‘suspended sediment’ medium types vs. STORET’s ‘sediment’.  Ontology tagging, semantic mediation (to be presented later) Challenges: information model (2/3)

48. Several modes of ODM publication –Setup your own HIS Server (or a virtual server at SDSC), register data to Central HIS –Prepare ODM and submit it to Central HIS for publication –Submit raw data to Central HIS for publication What are your preferences? Security of the published data, and user agreements Simple tools for loading and massaging data in ODM –From static files –Streaming data, from different sensors Who is responsible for data quality and re-usability? –Data shall be tagged with ontologies, conform with controlled vocabularies, discoverable. –Data shall be curated and annotated How we extend the model to other data types Challenges: data publication (3/3)

49. Additional materials not covered earlier… On integration with RBNB On CUAHSI HIS role as the community mediator On data cubes Work with us…

50. Scalable, secure, programmable, versatile for different data types and vendor interfaces, developer community, with many applications written (e.g. data viewers and plug-ins), open source, high performance streaming (10mb/s, 1000 frames/s) Typical scenarios: DBMS Monitoring and management apps CS loggers NI loggers Other proprietary or in-house CS loggers Loggernet RBNB DataTurbine (Ring Buffered Network Bus)

51. CUAHSI HIS as a mediator across multiple agency and PI data –Keeps identifiers for sites, variables, etc. across observation networks –Manages and publishes controlled vocabularies, and provides vocabulary/ontology management and update tools –Provides common structural definitions for data interchange –Provides a sample protocol implementation –Governance framework: a consortium of universities, MOUs with federal agencies, collaboration with key commercial partners, led by renowned hydrologists, and NSF support for core development and test beds

52. US Map of USGS Observations Antarctica Puerto Rico Hawaii Alaska

53. Different types of nutrients by decade: Available Data Total

54. Some physical properties by decade: Available Data Total

55. Same without discharge, gage height, temperature and precipitation (the four most common, in that order): Available Data Total

56. Work with us Building a larger network of hydrologic data servers –Install a HIS server at your site –Contribute data and services –Join development team –Create hydrologic mashups –Provide feedback Web sites: –www.cuahsi.org/his/ project info, downloadable software, manuals, workbookwww.cuahsi.org/his/ –river.sdsc.edu/wiki/ developer notes, issues/bugs tracking, CVS instructions, etc.river.sdsc.edu/wiki/ –CUAHSI Web Services http://river.sdsc.edu/Wiki/CUAHSI WebServices.ashxhttp://river.sdsc.edu/Wiki/CUAHSI WebServices.ashx –EPA Web services http://www.epa.gov/storet/web_services.html http://www.epa.gov/storet/web_services.html –USGS (non-production) http://waterservices.usgs.gov/NWISQuery/GetWSService?wsdl http://waterservices.usgs.gov/NWISQuery/GetWSService?wsdl