Department of Civil, Architectural & Environmental Engineering HYDROSEEK and HYDROTAGGER A Search Engine for Hydrologists GIS in Water Resources Lecture M. Piasecki November, 2007 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering 9/15/2018 Lecture Demo of HydroSeek What are the search criteria? Functionality of the Engine Interface Data Sources Common Sources Common Problems (Completeness, Syntax, Semantics) Ontologies Ontology details Concept-to-data variable tagging Architecture Flow Chart Technologies used Demo of HydroTagger Why the Tagging? Technologies 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering www.HydroSeek.org 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering 9/15/2018 HIS Goals Hydrologic Data Access System – better access to a large volume of high quality hydrologic data Support for Observatories – synthesizing hydrologic data for a region Advancement of Hydrologic Science – data modeling and advanced analysis Hydrologic Education – better data in the classroom, basin-focused teaching 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering 9/15/2018 Objective Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them What we are doing now ….. NWIS request return request return return request NAWQA NAM-12 return request return request request return request return request return NARR 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering 9/15/2018 What we would like to do ….. GetValues Semantic Mediator NWIS GetValues GetValues GetValues generic request GetValues NAWQA GetValues NARR GetValues HODM GetValues 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Data sources… USGS EPA CIMS TCEQ NADP 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Spatial Coverage STORET has 758 sites in Texas, TCEQ has 8407. STORET has 47,602 sites in Florida, NWIS has 27,906. NWIS has 121,545 in Minnesota, STORET has 22,260. 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Data Availability 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Temporal Coverage 2003-2007 1977-2003 1957-1977 Nitrogen 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Interface Problem NWIS ~175 form elements on a single page STORET + NWIS + TCEQ + CIMS = ??? A drop down menu ∞ String search across parameter list? How about synonyms? ‘Elevation, water surface’ vs. ‘stage height’ 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Completeness Problem: Metadata Catalog Better query performance Freedom Fewer errors Total Number of Sites 274,918 Sites with geographic coordinates 274,435 Sites with State/County information 273,113 Sites with Hydrologic Unit Codes 128,646 Availability of geographic identifiers for stations in EPA STORET 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Heterogeneity Problem Syntax E.g. date & time formats, Gregorian versus Julian Data format/structure E.g. XML, HTML, tab/tilde/comma separated text, gunzipped tar balls… Semantics more ….. 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Issues with Semantics Hyponymy Parameter “Groundwater level”, “Stream stage”, “Reservoir level” versus “Water level” Pseudo hyponymy due to lack of metadata Parameter “Manganese, 6N hydrochloric acid extracted, recoverable, dry weight, milligrams per kilogram” versus “Manganese, milligrams per kilogram” Synonymy ‘Total Kjeldahl Nitrogen’ vs. ‘Ammonia+Organic Nitrogen’ 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Search Strategy Search  Fine tune  Retrieve rather than Search  Retrieve avoid ‘high precision, low recall’ and ‘low precision, high recall’ problems. 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Layered Ontology Model 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Core Navigation Compound 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Knowledge Base Supports classification of search results Entities in the ontology are associated with measured variables in a relational database Helps solving semantic heterogeneity issues between data repositories OWL Ontologies ‘Escherichia coli’ = ‘E. coli’ ‘E. coli’ is-a ‘Indicator Organism’ ‘Copper’ is-a ‘Micronutrient’ ‘Copper’ isMeasuredIn ‘Medium’ ‘Medium’ = {Water, Soil…} ‘Micronutrient’ is-a ‘Nutrient’ 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Point Observations Information Model 9/15/2018 http://www.cuahsi.org/his/webservices.html Point Observations Information Model USGS Data Source Streamflow gages Network GetSites GetSiteInfo Neuse River near Clayton, NC Sites GetVariables Discharge, stage (Daily or instantaneous) Variables GetVariableInfo GetValues Values 206 cfs, 13 August 2006 {Value, Time, Qualifier, Offset} A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time A qualifier is a symbol that provides additional information about the value An offset allows specification of measurements at various depths in water 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Hydroseek Webservices Most Hydroseek functions are available as web services (SOAP) Support for queries using GlobalChangeMasterDirectory GCMD keywords Supports output in GeographyMarkupLanguage GML as well as WaterML MicroSoft Server VirtualEarth Map San Diego Supercomputer Center Server Native Services HydroSeek EPA STORET WaterOneFlow USGS Daily WaterOneFlow Drexel Server WaterOneFlow CIMS WaterOneFlow USGS Realtime WaterOneFlow TCEQ 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering GetStations Request BoundingBox Response 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering GetStationsByHU Request HUC_Code Response 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering GetStationCatalogueFiltered Request Response 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering GetStationCatalogue Request Response 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Allows searching multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them Modular & extensible Architecture Outline Inside the CUAHSI HOD Module 9/15/2018 Department of Civil, Architectural & Environmental Engineering

The Database-Ontology Link www.HdyroTagger.org 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering 1) MappingsApproved_Table 2) FrequentUpDates_Table HydroSeek ODM needed an upgrade, i.e. additional tables. 9/15/2018 Department of Civil, Architectural & Environmental Engineering

How does the Tagging work? Step 1 Users need to register on the web-site first before they can use the HydroTagger. When registering select the testbed site you are affiliated with. Each testbed site needs ONE administrator who can then admit additional users for that specific testbed site. Please send an email to identify the designated tagger site administrator so we can promote that person to the role. 9/15/2018 Department of Civil, Architectural & Environmental Engineering

How does the Tagging work? Step 2 The “Sniffer” jumps into action and trawls through the testbed sites to find and identify new variablenames (once a week, currently every Sunday night) It does so by using the regular web-services published through the WSDL (no “hacking”!!!) It returns i) data updating information and ii) variablenames used and compares these to those used by HydroSeek. WATERS Network Information System 9/15/2018 Department of Civil, Architectural & Environmental Engineering

How does the Tagging work? Step 3 The Tagger now updates the HydroSeek catalogue (an amalgamation of all 10 testbed catalogues) with the newly found data entries. If it finds a new variablename (introduced during the data loading process using the Data-Loader), it puts it into a table and offers it up to he HydroTagger GUI for semantic Tagging. 9/15/2018 Department of Civil, Architectural & Environmental Engineering

Department of Civil, Architectural & Environmental Engineering Thank you…Questions? 9/15/2018 Department of Civil, Architectural & Environmental Engineering