Water Quality Portal Web Services Guide

The Water Quality Data Portal (WQP) provides an easy way to access data stored in various large water quality databases through form-based queries as well as through standalone web services. The form interface and the web services offer equivalent input parameters and output formats. Submit a web service request to bypass the WQP form interface. For more information on the WQP input parameters and data retrievals, see the User Guide.

HTTPS only on the Water Quality Portal

USGS-hosted websites, which includes the Water Quality Portal, will be switching to the secure and encrypted HTTPS protocol during December, 2016.  For most users, the changes won’t be noticeable other than a visual change in web browsers indicating a private connection.  But users who write or maintain software that uses USGS web services could be impacted and should read on for details.

After this switch, requests for USGS web resources using the HTTP protocol will receive one of the 3xx Redirection HTTP status codes indicating an equivalent HTTPS URL.  Software that receives this response likely falls into one of these categories:

1.  No change required.  Many underlying software libraries transparently process redirects and switch to the HTTPS protocol with no required programming changes.  The additional redirect will reduce performance slightly, however, so an optimization would be changing all USGS URLs to HTTPS.
2.  Does not follow redirects automatically, change required.  Some software libraries may report redirects as errors instead of following them.  For this, change all USGS URLs to HTTPS, or alter the software to follow the redirect response.
3.  Supports HTTPS only with additional configuration, change required.  Underlying software libraries might require additional configuration, installed modules, or invocation options to support HTTPS.
4.  Does not support HTTPS, potentially major change required.  Some software libraries, particularly older ones, might not be capable of supporting HTTPS.  For this, alternate libraries can be sought or the software can be ported to another language or platform that supports HTTPS.

Software developers may test a software library’s redirect and HTTPS support by retrieving an arbitrary web resource, including this URL that generates a 3xx redirect to an HTTPS URL: http://https.cio.gov/ .   The Water Quality Portal's Web Services already support HTTPS, and you can make the change to you scripts today.  If you have issues, please contact us at http://www.waterqualitydata.us/contact_us/

This conversion is part of a federal-government-wide effort observing that “the American people expect government websites to be secure and their interactions with those websites to be private.”  

Using a web service client

The Water Quality Portal has several language-specific web service tools that may make it easier to use the WQP services if you are already using a scripting language, as the application will do the heavy lifting to get the data into a familiar data format.

• R- The dataRetrevial package is a powerful tool that facilitates downloading data from the WQP and from a number of USGS NWIS services
  • https://github.com/USGS-R/dataRetrieval
  • Tutorial: https://owi.usgs.gov/R/dataRetrieval.html
  • available on CRAN: https://cran.r-project.org/web/packages/dataRetrieval/index.html
• Python- pyWQP at https://github.com/USGS-CIDA/pyWQP  This client has limited support that this time.

Submitting a Web Services Request

The WQP web services may be queried using a RESTlike (REpresentational State Transfer) technique. This method will retrieve the same data as the WQP form when the same retrieval parameters are specified. The basic building blocks for querying the services are outlined in the following series of tables. Most non-alphanumeric characters (such as punctuation) must be "url-encoded", (for example: space is "%20").

Parameters must be appended to the base Uniform Resource Locator (URL) for the RESTlike WQP web services. The base URL is constructed differently depending on the type of information being requested. There are two options - one base URL for downloading sites and one base URL for downloading results:

• Base URL for downloading sites -- https://www.waterqualitydata.us/data/Station/search?
• Base URL for downloading results -- https://www.waterqualitydata.us/data/Result/search?
• Base URL for downloading activity data – https://www.waterqualitydata.us/data/Activity/search?
• Base URL for downloading activitymetric data – https://www.waterqualitydata.us/data/ActivityMetric/search?
  

Construct a RESTlike web service query by concatenating the base URL with the desired parameters and arguments (Table 1).  At least one parameter-argument pair must be specified. Separate multiple parameter-argument pairs with an ampersand ("&"). Unneeded web service parameters may be omitted. If no mime type is specified, the retrieval will default to WQX-XML format. Station and Result retrieval data elements are output in a consistent format and nomenclature. See the User Guide for a list of elements included in the station and result retrievals.

Table 1. URL-encoded retrieval parameters and arguments for WQP web services [parameter names are insensitive only to the leading capital letter].

Web Service Request Examples

All examples include <remove this string> to avoid spurious web service calls from web crawler robots

The following example REST web service request retrieves sites from Oklahoma County, Oklahoma, where Atrazine was measured in XML format, zipped.

https://www.water<remove this string>qualitydata.us/data/Station/search?countycode=US%3A40%3A109&characteristicName=Atrazine&mimeType=xml&zip=yes

This REST web service request retrieves contained within a bounding box where Caffeine was measured in KML format, zipped.

https://www.water<remove this string>qualitydata.us/data/Station/search?characteristicName=Caffeine&mimeType=kml&bBox=-92.8,44.2,-88.9,46.0&zip=yes

Continuing from above, this REST web service request retrieves the Caffeine sample results from sites contained within a bounding box and collected on, or after, 10-01-2006 in MS-Excel format (zipped).

https://www.water<remove this string>qualitydata.us/data/Result/search?characteristicName=Caffeine&bBox=-92.8,44.2,-88.9,46.0&startDateLo=10-01-2006&mimeType=xlsx&zip=yes

Using POST instead of GET

Water Quality Portal services can be accessed using an http POST, with a JSON payload, instead of an http GET.  This is particularly helpful when submitting large lists of sites.

An example JSON submittal looks like this:

{
"organization": ["WIDNR_WQX"],
"siteid": ["WIDNR_WQX-133003", "WIDNR_WQX-133398", "WIDNR_WQX-133486"]
}

and an example curl with that data looks like this:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/zip' -d '{"organization":["WIDNR_WQX"],"siteid":["WIDNR_WQX-133003","WIDNR_WQX-133398","WIDNR_WQX-133486"]} \ 
 ' 'https://www.waterqualitydata.us/data/Station/search?mimeType=csv&zip=yes'

If there are any problems, they are returned in the response via the warning header(s). See RFC 2616 Section 14 for the format of the warning header. 

Web service Request Pre-Validation

A REST web service request may not be valid for all data providers. A web service user may perform a quick validation before submitting a request to ensure that all parameters are valid before making an expensive request for actual data. For example, the WQP mini-portal form does this before an actual data submission, and pops up a prompt to the user if problems are found.

A REST web service validation service for sites and results requests are invoked by using the exact same RESTlike URL as an actual request but using a HTTP HEAD method rather than the usual GET or POST. See the W3C specification RFC 2616 Section 9 for an explanation of HTTP methods. The exact way to invoke a URL via a HEAD request depends on your programming language, but an example is provided below for JavaScript:

function makeRequest(url) {
  	var http = new XMLHttpRequest();
  	http.open('HEAD', url, false);
  	http.send();
  	var warningHeader = http.getResponseHeader("Warning");
  	alert(warningHeader);
}

Another option is using a command line. For example, a HEAD request can be issued as:

curl -I 'https://www.water<remove this string>qualitydata.us/data/Result/search?bBox=-92.8,44.2,-88.9,46.0&startDateLo=10-01-2011 &mimeType=csv&characteristicName=Dissolved%20oxygen%20(DO)' 

If there are any problems, they are returned in the response via the warning header(s). See RFC 2616 Section 14 for the format of the warning header. 

NOTE: The same warning header(s) and count headers are returned on the GET request as well, so it is important to issue the command with 'debug' or 'verbose' diagnostic option turned on, and then inspect the header to ensure the service is performing as expected. For example:

wget -d -O - 'https://www.water<remove this string>qualitydata.us/data/Result/search?&bBox=-92.8,44.2,-88.9,46.0 &startDateLo=10-01-2011&mimeType=csv&characteristicName=Dissolved%20oxygen%20(DO)'
or 
curl -v  'https://www.water<remove this string>qualitydata.us/data/Result/search?&bBox=-92.8,44.2,-88.9,46.0 &startDateLo=10-01-2011&mimeType=csv&characteristicName=Dissolved%20oxygen%20(DO)'

Either statement returns the following (indicating no NWIS results are returned):

Total-Site-Count: 170
STORET-Site-Count: 170
Warning: 299 NWIS "[characteristicName]The value of characteristicName=Dissolved 
                     oxygen (DO) is not in the list of enumerated values"

Using OGC Services to map Water Quality Portal sites based on Water Quality Portal Parameters

The Water Quality Portal has an endpoint for generating OGC-Compliant Geospatial web services.  At this time, we support Web Mapping Service (WMS) version 1.1.1 and Web Feature Service (WFS) version 1.1.1.  The service is a customized version of the WMS and WFS services provided by Geoserver.  The Geoserver WMS Reference and WFS Reference will be helpful for getting started.  The base URL for both the WMS and WFS Services is 

 https://www.waterqualitydata.us/ogcservices/{wms|wfs}

You will want to use web mapping tools, such as Leaflet or Openlayers to be able to easily connect to these web map services.  GIS tools such as ARC GIS and QGIS should also be able to connect to these web maps services, though they have not been tested.

In a nutshell, both WFS and WMS services require an additional parameter called "SearchParams."  SearchParams is a URL-encoded version of the Water Quality Portal search parameters that you pass along with the rest of the standard OGC query parameters.  The service supports calls that return up to up to 250,000 sites.  Calls that return large numbers of sites will take longer to load for the first time, as a cache is populated.  After that, the service should respond to requests with response rates in the tens to hundreds of milliseconds.  The Cache is cleared after new data is loaded to the portal, generally once a day at night.

At this time, we support WMS GetMap, WMS GetFeatureInfo, and WMS GetLegendGraphic.

WMS Getmap 

• For detailed information on constructing a WMS getmap request, please refer to the GetMap documentation at the Geoserver site.  

• Here is an example GetMap Request for stream sites that have samples for atrazine.  the 

https://www.waterqualitydata.us/ogcservices/wms?

SERVICE=WMS

&REQUEST=GetMap

&VERSION=1.1.1

&LAYERS=wqp_sites

&STYLES=wqp_sources

&FORMAT=image%2Fpng

&TRANSPARENT=true

&HEIGHT=256

&WIDTH=256

&SEARCHPARAMS=countrycode%3AUS%3BcharacteristicName%3AAtrazine

&SRS=EPSG%3A3857

&BBOX=-15028131.257091932,-7.081154551613622e-10,-10018754.171394622,5009377.085697313

Looking Up Domain Values through Web Services

The domains for retrieval parameter arguments may vary over time, as new data are added. The domain of values for the parameter arguments may be retrieved with a REST query. Use the domain-value base URL concatenated with the parameter name and arguments as shown in Table 2.  

Base URL for looking up domain values -- https://www.waterqualitydata.us/Codes/{endpointName}?{parameter}

There needs to be at least one argument in the web service call. If you want all domain values, you can just specify the mimetype: e.g. mimeType=json

Table 2. -- Domain values web service parameters and arguments