Using Python to Retrieve Data from the CUAHSI HIS Web Services Jon Goodall Hydroinformatics Fall 2014 This work was funded by National Science Foundation Grants EPS and EPS

Class Plan Introduction (setting this material within context) Set up – Software requirements In class demos – Example 1: Getting the site name for a USGS NWIS station using the CUAHSI HIS Web Services – Example 2: Getting the minimum streamflow over the past five days for a USGS NWIS station using the CUAHSI HIS Web Services Challenge problems Wrap up – Further reading – Thursday’s class

Big Picture Context Data life cycle -> data modeling -> database design -> database implementation -> ODM -> SQL Querying of an ODM -> Python programming against an ODM -> sharing data from an ODM using CUAHSI HIS Web services (WaterOneFlow) and WaterML -> Accessing CUAHSI HIS Web services using HydroDesktop -> This week: Accessing CUAHSI HIS Web services using Python

Set up …

Install the suds package Start Canopy Open the Package Manager Search for the ‘suds 0.4-1’ package Install this package Requires you to sign in with an account that has an academic license. Close the package manager

Test the installation of the suds package Within Canopy – Open the Editor – At the command line, type ‘import suds’ as shown below – In [1]: import suds – If you don’t get an error, then suds is installed correctly. – If you do get an error, repeat the instructions to install the suds package.

What is the suds package? “Suds is a lightweight SOAP Python client for consuming Web Services." SOAP and WSDL are standards for creating web services. You don’t need to know the details behind these standards, but if you are interested, Wikipedia has a good summary of both: – SOAP: – WSDL: tion_Languagehttp://en.wikipedia.org/wiki/Web_Services_Descrip tion_Language

In class examples …

Example 1: Get the site name for a USGS NWIS station using the CUAHSI HIS Web Services Use the GetSiteInfo method on the CUAHSI HIS WaterOneFlow web services – eInfo eInfo Use the suds Client object to call the web service method – We will use siteCode = USGS: Suds will automatically parse the WaterML response from the web service call. We will need to find the siteName property in the response and print it to the console. The answer for USGS: is: LOGAN RIVER ABOVE STATE DAM, NEAR LOGAN, UT I will walk you through building the code in class, but you can download an up-to-date version of the code here.here

Pros and Cons of using Web Services vs a local ODM database Pros: – Access to the entire NWIS database without the need to store the data locally – No need to keep local local in sync with USGS version Cons: – Requires Internet connection – Speed: Getting data via web services will almost certainly be slower than getting data from your own database – Data is outside your control: breaking changes, unavailable services, etc. – (Some of these could be improved with a data caching strategy)

Example 2: Getting the minimum streamflow over the past five days for a USGS NWIS station using the CUAHSI HIS Web Services Use the GetValuesObject method on the CUAHSI HIS WaterOneFlow web services – Like example 1, Use the suds Client object to call the web service method We will use siteCode = USGS: and ParameterCode = USGS= DateTimes should be in the format YYYY-MM-DD. We will extract the values and dateTimes and create a Pandas Series object to store the time series. We will use the min and idxmin methods on the Series object to get the minimum streamflow and datetime when the minimum streamflow occurred. – – I will walk through this in class, but you can download an up-to-date version of this code here.here Note: This works with Pandas version and with version , but may cause errors with earlier versions of Pandas. If you get an error, check your version of Pandas in the Canopy Package Manager and upgrade the version of Pandas if needed.

Using these principles for other WaterOneFlow Web services CUAHSI HIS Central lists other available web services that can be accessed in a similar way – – There are over 400 billion observations available through HIS Central! The HydroServers you created last week with Dr. Ames can also be accessed using this same approach. – If you would like to do this for your class project, get in touch with Dr. Ames.

Challenge problems …

Coding Challenges For the same station (USGS: ), modify the example 2 script so that it prints the daily min, max, and average streamflow for the past 5 days. (time permitting) Modify your script so that it prints out the min, max, and average streamflow for EACH DAY during the past 5 days  this is part of what you will need to do for Assignment 8.

Wrap up …

Further reading Details on the CUAHSI HIS web services are available here: – Read about WaterOneFlow (the WSDL) to learn more about the methods – Read about WaterML version 1.0 to learn more about the structure of the responses The WaterML example response are particularly helpful to seeing the structure of the response – Note that some services will return WaterML in version 1.1 format. The code shown in this lecture may require slight modification to work with these services.

Thursday’s Class Dr. Horsburgh will be leading class. Introduce Assignment 8. Work on the assignment in class.