1. Using Python to Retrieve Data from the CUAHSI HIS Web Services Jeffery S. Horsburgh Hydroinformatics Fall 2015 This work was funded by National Science Foundation Grants EPS 1135482 and EPS 1208732 Slides adapted from original version by Jon Goodall University of Virginia, Hydroinformatics, Fall 2014

2. Objectives Discover and access data from major hydrologic data sources Create reproducible data visualizations Write and execute computer code to automate difficult and repetitive data related tasks Manipulate data and transform it across file systems, flat files, databases, programming languages, etc. Retrieve and use data from Web services

3. Class Plan Introduction (setting this material within context) Set up – Software requirements In class demos – Example 1: Getting the site name for a USGS NWIS station using the CUAHSI HIS Web Services – Example 2: Getting the minimum streamflow over the past 5 days for a USGS NWIS station using the CUAHSI HIS Web Services – Example 3: Making a plot of the streamflow data for the past 5 days Challenge problems Wrap up

4. Big Picture Context 1.Data life cycle 2.Data modeling 3.Database design 4.Database implementation and ODM 5.SQL Querying of an ODM database 6.Python programming against an ODM database 7.Sharing data from an ODM using CUAHSI HIS Web services (WaterOneFlow) and WaterML 8.Accessing CUAHSI HIS Web services using HydroDesktop 9.This week: Accessing CUAHSI HIS Web services using Python

5. Set up …

6. Required Packages “suds” – a package for making requests to SOAP web services (https://fedorahosted.org/suds/)https://fedorahosted.org/suds/) “pandas” – A data analysis library with high performance data structures (http://pandas.pydata.org/)http://pandas.pydata.org/) “matplotlib” – A package for scientific plotting (http://matplotlib.org/)http://matplotlib.org/)

7. What is the “suds” package? “Suds is a lightweight SOAP Python client for consuming Web Services.” https://fedorahosted.org/suds/ https://fedorahosted.org/suds/ SOAP and WSDL are standards for creating web services. You don’t need to know the details behind these standards, but if you are interested, Wikipedia has a good summary of both: – SOAP: http://en.wikipedia.org/wiki/SOAPhttp://en.wikipedia.org/wiki/SOAP – WSDL:http://en.wikipedia.org/wiki/Web_Services_De scription_Languagehttp://en.wikipedia.org/wiki/Web_Services_De scription_Language

8. What is the “pandas” package? pandas is an open source library providing high performance data structures Some pandas data structures you may be interested in: – Series – a one-dimensional, labeled array capable of holding any data type – axis labels are collectively referred to as the “index” – DataFrame – a 2-dimensional data structure with columns of potentially different types (essentially a high performance table object)

9. In class examples …

10. Example 1: Get the site name for a USGS NWIS gage station using the CUAHSI HIS Web Services Use the GetSiteInfoObject method on the CUAHSI HIS WaterOneFlow USGS Unit Values web service: – http://hydroportal.cuahsi.org/nwisuv/cuahsi_1_1.asmx http://hydroportal.cuahsi.org/nwisuv/cuahsi_1_1.asmx Use the suds Client object to call the web service method – We will use siteCode = “USGSUV:10109000” – Suds will automatically parse the WaterML response from the web service call. – We will need to find the siteName property in the response and print it to the console. The answer for “NWISUV:10109000” is: LOGAN RIVER ABOVE STATE DAM, NEAR LOGAN, UT

11. Example 2: Getting the minimum streamflow over the past five days for a USGS NWIS station using the CUAHSI HIS Web Services Use the GetValuesObject method on the CUAHSI HIS WaterOneFlow web services – http://hydroportal.cuahsi.org/nwisuv/cuahsi_1_1.asmx http://hydroportal.cuahsi.org/nwisuv/cuahsi_1_1.asmx Like example 1, use the suds Client object to call the web service method We will use siteCode = “USGSUV:10109000 and ParameterCode = USGSUV:00060. DateTimes should be in the format “YYYY-MM-DD”. – Example GetValuesObject web service call in a browser that returns a WaterML file: http://hydroportal.cuahsi.org/nwisuv/cuahsi_1_1.asmx/GetValuesObject?location=NWISUV:101090 00&variable=NWISUV:00060&startDate=2015-10-20&endDate=2015-10-31&authToken= http://hydroportal.cuahsi.org/nwisuv/cuahsi_1_1.asmx/GetValuesObject?location=NWISUV:101090 00&variable=NWISUV:00060&startDate=2015-10-20&endDate=2015-10-31&authToken We will extract the values and dateTimes and create a pandas Series object to store the time series We will use the min() and idxmin() methods on the Series object to get the minimum streamflow and datetime when the minimum streamflow occurred. – http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.min.html http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.min.html Note: I tested this with Pandas version 0.17.0. If you get an error, check your version of Pandas in the PyCharm Package Manager and upgrade the version of Pandas if needed.

12. Example 3: Create a Time Series Plot of Streamflow Values for the Past 5 Days Use the GetValuesObject method on the CUAHSI HIS WaterOneFlow web services – http://hydroportal.cuahsi.org/nwisuv/cuahsi_1_1.asmx http://hydroportal.cuahsi.org/nwisuv/cuahsi_1_1.asmx Like examples 1 and 2, use the suds client object to call the web service method We will use SiteCode = USGSUV:10109000 and ParameterCode = USGSUV:00060. DateTimes should be in the format YYYY-MM-DD. We will extract the values and dateTimes and create a Pandas Series object to store the time series. We will create a “figure” object within which we can create our time series plot – http://matplotlib.org/api/figure_api.html http://matplotlib.org/api/figure_api.html We will use the plot() method on the pandas Series object to plot the time series. – http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.plot.html http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.plot.html Note: I tested this with Pandas version 0.17.0. If you get an error, check your version of Pandas in the PyCharm Package Manager and upgrade the version of Pandas if needed.

13. Using these principles for other WaterOneFlow Web services CUAHSI HIS Central lists other available web services that can be accessed in a similar way – http://hiscentral.cuahsi.org/ http://hiscentral.cuahsi.org/ – There are over 400 billion observations available through HIS Central! The HydroServers you created last week with Dr. Ames can also be accessed using this same approach

14. Challenge problems …

15. Coding Challenges For the same station (NWISUV:10109000), modify the example 2 script so that it prints the daily min, max, and average streamflow for the past 5 days. (time permitting) Modify your script so that it prints out the min, max, and average streamflow for EACH DAY during the past 5 days  this is part of what you will need to do for Assignment 8.

16. Wrap up …

17. Pros and Cons of using Web Services vs. a local ODM database Pros: – Access to the entire database on which the service is based without the need to store the data locally – No need to keep local local in sync with USGS version Cons: – Requires Internet connection – Speed: Getting data via web services will almost certainly be slower than getting data from your own database – Data is outside your control: breaking changes, unavailable services, etc. – (Some of these could be improved with a data caching strategy)

18. Summary You can use Python to automate the retrieval of hydrologic data via web services The “suds” package enables you to retrieve data as Python objects “pandas” has some nice data structures that make analysis and visualization easier “matplotlib” allows you to make nice plots

19. Thursday’s Class Introduce Assignment 7 Work on the assignment in class