Using Python to Interact with the EPA WATERS Web Services Jon Goodall Hydroinformatics Fall 2014 This work was funded by National Science Foundation Grants EPS 1135482 and EPS 1208732

EPA Waters “The Watershed Assessment, Tracking & Environmental Results System (WATERS) unites water quality information previously available only from several independent and unconnected databases.” http://water.epa.gov/scitech/datait/tools/waters/

WATERS Web Services Description: “WATERS provides a suite of interoperable services that expose components that perform complex analysis and supporting strategic datasets, such as NHD, NHDPlus, and WBD.” Source: http://water.epa.gov/scitech/datait/tools/waters/services/index.cfm

Web and Database Services The WATERS Web and Database services provide open interfaces to complex analyses. These services make extensive use of the National Hydrography Dataset (NHD) Designed as loosely coupled modular units, the services are developed in a common architecture. The HTTP Services provided REST-like access to the WATERS database services. Source: http://water.epa.gov/scitech/datait/tools/waters/services/index.cfm

NHD = National Hydrography Dataset http://nhd.usgs.gov

REST REST = Representational State Transfer You can think of this as an alternative to SOAP that we discussed and you used previously in this course. Most people seem to prefer REST because you can simply enter a URL into a web browser and see the response. The response is often structured text that uses either XML or JSON. Wikipedia page has further details if you are interested: http://en.wikipedia.org/wiki/Representational_state_transfer

What is JSON? JSON = JavaScript Object Notation A way of structuring data in a machine-readable way. An alternative to XML Thread on Stackoverflow.com on XML vs JSON: http://stackoverflow.com/questions/3536893/what-are-the-pros-and-cons-of-xml-and-json You can read more details on the Wikipedia page: http://en.wikipedia.org/wiki/JSON

Back to the EPA WATERS Web Services… You can find a list of the available services with documentation here: http://water.epa.gov/scitech/datait/tools/waters/services/index.cfm We will use the Name Service in today’s class as an example: http://water.epa.gov/scitech/datait/tools/waters/services/name_service.cfm

Name Service Name service documentation is here: http://water.epa.gov/scitech/datait/tools/waters/services/name_service.cfm Brief description: “Advanced query function to find water feature names and provide general location information for NHDPlus features.”

How do you use these services? Some example use cases of the WATERS services are here: http://codepen.io/WATERS_SUPPORT/public/ I used Firebug, a web development plug-in to Firefox, to see the web service calls used by the example software (see screen shot on next page and I will demonstrate in class). http://getfirebug.com/ I was told, but did not test myself, that the Chrome DevTools could also be used for this purpose. https://developer.chrome.com/devtools

The URL for the service http://ofmpub.epa.gov/waters10/Name.Service?pFullText=James+River&pFullTextRegex=&pBasename=&pBasenameRegex=&pHydrography=&pHydrographyRegex=&pDirectional=&pDirectionalRegex=&pOperator=EQ&pQueryLimit=&pJWThreshold=90&pResolution=3&pSourceTable=nhdflowline&pState=VA&pStateMod=%2C&pCountyFips5=&pCountyFips5Mod=%2C&pSubbasin=&pSubbasinMod=%2C&pGnisClass=&pGnisClassMod=%2C&pFtype=&pBreakBySubbasin=false&pBreakByFcode=false&optNHDPlusDataset=2.1&optCache=1415283785917&optJSONPCallback=success

Response from Web Service success({ "output":{ "feature_id":null,"results":[{ "styleUrl":"#.namesrv_results","feature_id":"1488853","feature_name":"JAMES RIVER","feature_class":"STREAM","primary_state":"VA","primary_state_fips":"51","primary_county":"ISLE OF WIGHT","primary_county_fips":"093","gnis_primary_stem":"JAMES","gnis_primary_hydro":"RIVER","gnis_primary_dir":null,"gnis_secondary_stem":null,"gnis_secondary_hydro":null,"gnis_secondary_dir":null,"resolution":3,"source_schema":"NHDPLUS","source_table":"NHDFLOWLINE_NP21","subbasin":null,"ftype":null,"fcode":null,"gnis_centroid_geom":{"type":"Point","coordinates":[-76.4435588,36.9415369]},"mbr_geom":{"type":"Polygon","coordinates":[[[-79.8541445197531,36.9403890091464],[-76.3481893915265,36.9403890091464],[-76.3481893915265,37.7966000744348],[-79.8541445197531,37.7966000744348],[-79.8541445197531,36.9403890091464]]]},"shape":null}]},"status":{ "submission_id":null,"status_code":0,"status_message":null,"execution_time":0.129432,"output_bytes":853}})

Comments on URL and response In the URL you will see pFullText=James+River pSourceTable=nhdflowline pState=VA These are the search criteria I entered in the CodePen interface In the response you will see "gnis_centroid_geom":{"type":"Point","coordinates":[-76.4435588,36.9415369]}, “Geometry representing the centroid of the result record as defined by GNIS.” "mbr_geom":{"type":"Polygon","coordinates":[[[-79.8541445197531,36.9403890091464],[-76.3481893915265,36.9403890091464],[-76.3481893915265,37.7966000744348],[-79.8541445197531,37.7966000744348],[-79.8541445197531,36.9403890091464]]]},"shape":null}]}, “Geometry representing the maximum bounding rectangle of the matching NHD feature. Note that in the case that the matching NHD record is a single point, that point will be stored in this field.”

Testing the Result using Google Maps Note that you need to flip the coordinates provided by the service when you input them into Google maps

Challenge Problem #1 Change the URL provided earlier to find a water feature of interest to you. Enter the coordinates returned by the service into Google Maps to verify that you got what you expected.

Repeat Using Python Suppose you wanted to create a “search tool” capable of geolocating an NHD feature given its name and state. We have already shown how to get the response from a URL using Python import urllib2 response = urllib2.urlopen(<URL String Here>).read() There are JSON libraries in Python we can use to extract the lat/lon coordinates of the centroid point https://docs.python.org/2/library/json.html

Challenge Problem #2 Write a Python script that returns the centroid coordinates for an NHD feature given its name and location. My solution is in the class Github site: https://github.com/goodalljl/hydroinformatics_class

What you learned today… You were introduced to REST Can you describe the differences between REST and SOAP? You were introduced to JSON Can you describe the differences between JSON and XML? You were introduced to the concept of using Web services for analytics vs. simply providing data

Plan for Thursday’s Class We will look at other services in the EPA WATER Web Services I will discuss your assignment that will ask you to further evaluate and build on the EPA WATER Web Services. The USGS/EPA team behind these want to hear your opinion! With your permission, we will share your assignment solution with the NHD development team.