The Water Quality Data Portal (WQP) provides an easy way to access data stored in various large water quality databases through form-based queries as well as through standalone web services. The form interface and the web services offer equivalent input parameters and output formats. Submit a web service request to bypass the WQP form interface. For more information on the WQP input parameters and data retrievals, see the User Guide.
USGS-hosted websites, which includes the Water Quality Portal, will be switching to the secure and encrypted HTTPS protocol during December, 2016. For most users, the changes won’t be noticeable other than a visual change in web browsers indicating a private connection. But users who write or maintain software that uses USGS web services could be impacted and should read on for details.
After this switch, requests for USGS web resources using the HTTP protocol will receive one of the 3xx Redirection HTTP status codes indicating an equivalent HTTPS URL. Software that receives this response likely falls into one of these categories:
Software developers may test a software library’s redirect and HTTPS support by retrieving an arbitrary web resource, including this URL that generates a 3xx redirect to an HTTPS URL: http://https.cio.gov/ . The Water Quality Portal's Web Services already support HTTPS, and you can make the change to you scripts today. If you have issues, please contact us at http://www.waterqualitydata.us/contact_us/
This conversion is part of a federal-government-wide effort observing that “the American people expect government websites to be secure and their interactions with those websites to be private.”
The Water Quality Portal has several language-specific web service tools that may make it easier to use the WQP services if you are already using a scripting language, as the application will do the heavy lifting to get the data into a familiar data format.
The WQP web services may be queried using a RESTlike (REpresentational State Transfer) technique. This method will retrieve the same data as the WQP form when the same retrieval parameters are specified. The basic building blocks for querying the services are outlined in the following series of tables. Most non-alphanumeric characters (such as punctuation) must be "url-encoded", (for example: space is "%20").
Parameters must be appended to the base Uniform Resource Locator (URL) for the RESTlike WQP web services. The base URL is constructed differently depending on the type of information being requested. There are two options - one base URL for downloading sites and one base URL for downloading results:
Construct a RESTlike web service query by concatenating the base URL with the desired parameters and arguments (Table 1). At least one parameter-argument pair must be specified. Separate multiple parameter-argument pairs with an ampersand ("&"). Unneeded web service parameters may be omitted. If no mime type is specified, the retrieval will default to WQX-XML format. Station and Result retrieval data elements are output in a consistent format and nomenclature. See the User Guide for a list of elements included in the station and result retrievals.
Table 1. URL-encoded retrieval parameters and arguments for WQP web services [parameter names are insensitive only to the leading capital letter].
REST parameter | Argument | Discussion |
---|---|---|
bBox | Western-most longitude, Southern-most latitude, Eastern-most longitude, and Northern-most longitude separated by commas, expressed in decimal degrees, WGS84, and longitudes west of Greenwich are negative. (Example: bBox=-92.8,44.2,-88.9,46.0) | These four arguments are used together to form a quadrant of the Earth's surface for locating data-collection stations. Many stations outside the continental US do not have latitude and longitude referenced to WGS84 and therefore cannot be found using these parameters. Other stations are not associated with latitude and longitude due to Homeland Security concerns. |
lat | Latitude for radial search, expressed in decimal degrees,WGS84 | These three arguments are used together to form a circle on the Earth's surface for locating data-collection stations. Many stations outside the continental US do not have latitude and longitude referenced to WGS84 and therefore cannot be found using these parameters. |
long | Longitude for radial search, expressed in decimal degrees, WGS84 | |
within | Distance for radial search, expressed in decimal miles | |
countrycode | Two-character Federal Information Processing Standard (FIPS) country code. (see domain service for available codes) | FIPS country codes were established by the National Institute of Standards, publication 5-2. |
statecode | Two-character Federal Information Processing Standard (FIPS) country code, followed by a URL-encoded colon ("%3A"), followed by a two-digit FIPS state code. (seedomain service for available codes) | FIPS state codes were established by the National Institute of Standards, publication 5-2. |
countycode | Two-character Federal Information Processing Standard (FIPS) country code, followed by a URL-encoded colon ("%3A"), followed by a two-digit FIPS state code, followed by a URL-encoded colon ("%3A"), followed by a three-digit FIPS county code. (see domain service for available codes) | FIPS county codes were established by the National Institute of Standards, publication 5-2. |
siteType | One or more case-sensitive site types, separated by semicolons. (see domain service for available site types) | Restrict retrieval to stations with specified site type (location in the hydrologic cycle). The MonitoringLocationTypeName for individual records may provide more detailed information about the type of individual stations. |
organization | For USGS organization IDs, append an upper-case postal-service state abbreviation to "USGS-" to identify the USGS office managing the data collection station records. However, a few US states are serviced by one USGS office.
| USGS offices sometimes provide data for stations outside the political boundaries associated with the office's organization code. Use the statecode or countycode arguments to search for stations located within those political boundaries. |
siteid | Concatenate an agency code, a hyphen ("-"), and a site-identification number. | Each data collection station is assigned a unique site-identification number. Other agencies often use different site identification numbers for the same stations. |
huc | One or more eight-digit hydrologic units, delimited by semicolons. | Hydrologic unit codes identify surface watersheds. The lists and maps of hydrologic units are available from USGS. |
sampleMedia | One or more case-sensitive sample media, separated by semicolons. (see domain service for available sample media) | Sample media are broad general classes, and may be subdivided in the retrieved data. Examine the data elements ActivityMediaName, ActivityMediaSubdivisionName, and ResultSampleFractionText for more detailed information. |
characteristicType | One or more case-sensitive characteristic types (groupings) separated by semicolons. (see domain service for available characteristic types) | These groups will be expanded as part of the ongoing collaboration between USGS and USEPA. |
characteristicName | One or more case-sensitive characteristic names, separated by semicolons. (see domain service for available characteristic names) | Characteristic names identify different types of environmental measurements. The names are derived from the USEPA Substance Registry System (SRS). USGS uses parameter codes for the same purpose and has associated most parameters to SRS names. |
pCode | One or more five-digit USGS parameter codes, separated by semicolons. | |
activityId | One or more case-sensitive activity IDs, separated by semicolons. | Designator that uniquely identifies an activity within an organization. |
startDateLo | Date of earliest desired data-collection activity, expressed as MM-DD-YYYY | These two parameters, used together or individually, restrict the retrieval to data-collection activities within a range of dates. |
startDateHi | Date of last desired data-collection activity, expressed as MM-DD-YYYY | |
mimeType | xml | Output format is XML compatible with WQX-Outbound schema. This is the default format, and if a mimeType is not specified, the data will be in XML format |
xlsx | Output format is xlsx compatible with MS-Excel 2007 and greater | |
csv | Output format is comma-separated columns | |
tsv|tab | Output format is tab-separated columns | |
geojson | Output format is a narrow | |
kml | Output format is KML compatible with Google Earth. This option is not available for the results service. | |
kmz | Output format is kmz, a compressed form of kml compatible with Google Earth. This option is not available for the results service. | |
Zip | yes | Include the parameter to stream compressed data. Compression often greatly increases throughput, thus expediting the request. Kml files will be returned in the kml-specific zip format, .kmz. |
providers | STORET|NWIS|STEWARDS (also see domain service for available codes) | By default, requests are submitted to all the data providers. However, a particular provider may be specified using this parameter. |
sorted | yes|no | By default, tabular data are sorted by organization, monitoringLocationID, and (for results) activityID. However, sorting increases response time significantly, sometimes by orders of magnitude. If you are doing your own sorting after download, set sorted=no. For large downloads (over 5 million rows) sorting is disabled by default to ensure reasonable response times. XML requests are always sorted to accommodate the WQX data schema |
dataProfile | biological | Only affects results endpoint at this time. The biological dataProfile returns an extended set of columns that further describe biological data |
DEPRECATED COMMANDS | ||
command.avoid | STORET|NWIS | Deprecated! By default, requests are submitted to all the data providers. This deprecated command is replaced by "providers." To support legacy applications, command.avoid=NWIS maps to providers=STORET, and command.avoid=STORET maps to providers=NWIS |
mimeType | xls | Deprecated in favor of xlsx. The xls format had a limit of 63,000 rows, not practical for most wqp downloads. |
The following example REST web service request retrieves sites from Oklahoma County, Oklahoma, where Atrazine was measured in XML format, zipped.
https://www.water<remove this string>qualitydata.us/data/Station/search?countycode=US%3A40%3A109&characteristicName=Atrazine&mimeType=xml&zip=yes
This REST web service request retrieves contained within a bounding box where Caffeine was measured in KML format, zipped.
https://www.water<remove this string>qualitydata.us/data/Station/search?characteristicName=Caffeine&mimeType=kml&bBox=-92.8,44.2,-88.9,46.0&zip=yes
Continuing from above, this REST web service request retrieves the Caffeine sample results from sites contained within a bounding box and collected on, or after, 10-01-2006 in MS-Excel format (zipped).
https://www.water<remove this string>qualitydata.us/data/Result/search?characteristicName=Caffeine&bBox=-92.8,44.2,-88.9,46.0&startDateLo=10-01-2006&mimeType=xlsx&zip=yes
Water Quality Portal services can be accessed using an http POST, with a JSON payload, instead of an http GET. This is particularly helpful when submitting large lists of sites.
An example JSON submittal looks like this:
{
"organization": ["WIDNR_WQX"],
"siteid": ["WIDNR_WQX-133003", "WIDNR_WQX-133398", "WIDNR_WQX-133486"]
}
and an example curl with that data looks like this:
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/zip' -d '{"organization":["WIDNR_WQX"],"siteid":["WIDNR_WQX-133003","WIDNR_WQX-133398","WIDNR_WQX-133486"]} \ ' 'https://www.waterqualitydata.us/data/Station/search?mimeType=csv&zip=yes'
If there are any problems, they are returned in the response via the warning header(s). See RFC 2616 Section 14 for the format of the warning header.
A REST web service request may not be valid for all data providers. A web service user may perform a quick validation before submitting a request to ensure that all parameters are valid before making an expensive request for actual data. For example, the WQP mini-portal form does this before an actual data submission, and pops up a prompt to the user if problems are found.
A REST web service validation service for sites and results requests are invoked by using the exact same RESTlike URL as an actual request but using a HTTP HEAD method rather than the usual GET or POST. See the W3C specification RFC 2616 Section 9 for an explanation of HTTP methods. The exact way to invoke a URL via a HEAD request depends on your programming language, but an example is provided below for JavaScript:
function makeRequest(url) { var http = new XMLHttpRequest(); http.open('HEAD', url, false); http.send(); var warningHeader = http.getResponseHeader("Warning"); alert(warningHeader); }
Another option is using a command line. For example, a HEAD request can be issued as:
curl -I 'https://www.water<remove this string>qualitydata.us/data/Result/search?bBox=-92.8,44.2,-88.9,46.0&startDateLo=10-01-2011 &mimeType=csv&characteristicName=Dissolved%20oxygen%20(DO)'
If there are any problems, they are returned in the response via the warning header(s). See RFC 2616 Section 14 for the format of the warning header.
NOTE: The same warning header(s) and count headers are returned on the GET request as well, so it is important to issue the command with 'debug' or 'verbose' diagnostic option turned on, and then inspect the header to ensure the service is performing as expected. For example:
wget -d -O - 'https://www.water<remove this string>qualitydata.us/data/Result/search?&bBox=-92.8,44.2,-88.9,46.0 &startDateLo=10-01-2011&mimeType=csv&characteristicName=Dissolved%20oxygen%20(DO)' or curl -v 'https://www.water<remove this string>qualitydata.us/data/Result/search?&bBox=-92.8,44.2,-88.9,46.0 &startDateLo=10-01-2011&mimeType=csv&characteristicName=Dissolved%20oxygen%20(DO)'
Either statement returns the following (indicating no NWIS results are returned):
Total-Site-Count: 170 STORET-Site-Count: 170 Warning: 299 NWIS "[characteristicName]The value of characteristicName=Dissolved oxygen (DO) is not in the list of enumerated values"
The Water Quality Portal has an endpoint for generating OGC-Compliant Geospatial web services. At this time, we support Web Mapping Service (WMS) version 1.1.1 and Web Feature Service (WFS) version 1.1.1. The service is a customized version of the WMS and WFS services provided by Geoserver. The Geoserver WMS Reference and WFS Reference will be helpful for getting started. The base URL for both the WMS and WFS Services is
https://www.waterqualitydata.us/ogcservices/{wms|wfs}
You will want to use web mapping tools, such as Leaflet or Openlayers to be able to easily connect to these web map services. GIS tools such as ARC GIS and QGIS should also be able to connect to these web maps services, though they have not been tested.
In a nutshell, both WFS and WMS services require an additional parameter called "SearchParams." SearchParams is a URL-encoded version of the Water Quality Portal search parameters that you pass along with the rest of the standard OGC query parameters. The service supports calls that return up to up to 250,000 sites. Calls that return large numbers of sites will take longer to load for the first time, as a cache is populated. After that, the service should respond to requests with response rates in the tens to hundreds of milliseconds. The Cache is cleared after new data is loaded to the portal, generally once a day at night.
At this time, we support WMS GetMap, WMS GetFeatureInfo, and WMS GetLegendGraphic.
For detailed information on constructing a WMS getmap request, please refer to the GetMap documentation at the Geoserver site.
Here is an example GetMap Request for stream sites that have samples for atrazine. the
https://www.waterqualitydata.us/ogcservices/wms?
SERVICE=WMS
&REQUEST=GetMap
&VERSION=1.1.1
&LAYERS=wqp_sites
&STYLES=wqp_sources
&FORMAT=image%2Fpng
&TRANSPARENT=true
&HEIGHT=256
&WIDTH=256
&SEARCHPARAMS=countrycode%3AUS%3BcharacteristicName%3AAtrazine
&SRS=EPSG%3A3857
&BBOX=-15028131.257091932,-7.081154551613622e-10,-10018754.171394622,5009377.085697313
Base URL for looking up domain values -- https://www.waterqualitydata.us/Codes/{endpointName}?{parameter}
{endpointName} | REST parameter | Argument | Discussion | Example |
---|---|---|---|---|
Common parameters for all domain values web services | mimeType | xml|json | returns either XML or json. Default is xml | https://waterqualitydata.us/Codes/characteristicname?text=ph&pagesize=20&pagenumber=1&mimeType=json |
pagenumber | page number (1,2 etc) | allows for results to be paginated (especially useful for endpoints with many valid responses, allows for infinite scrolling). Use along with pagesize | ||
pagesize | e.g. 20 | number of results to return per page | ||
text | e.g. ph | text to match to endpoint results. This is straight string matching. When the text parameter is used, the results are returned sorted by length | ||
Endpoints with unique query parameters in addition to common query parameters | ||||
countrycode | FIPS country codes | |||
statecode | countrycode | A FIPS country code (e.g. US) | FIPS state codes. A FIPS country code argument is appended so that the URL ends as /statecode?countrycode=US | https://www.waterqualitydata.us/Codes/statecode?countrycode=US |
countycode | statecode | A FIPS statecode (e.g. statecode=US:01;US:04) | FIPS county codes. A FIPS statecode argument is appended so that the URL ends as /countycode?statecode=US:01;US:04 | https://www.waterqualitydata.us/Codes/countycode?statecode=US:01;US:04 |
Sitetype | Available site types | https://www.waterqualitydata.us/Codes/Sitetype?mimeType=json | ||
Organization | Available organization IDs | https://www.waterqualitydata.us/Codes/Organization?mimeType=xml | ||
Samplemedia | Sample media | https://www.waterqualitydata.us/Codes/Samplemedia?mimeType=xml | ||
Characteristictype | Characteristic types (groups) | https://www.waterqualitydata.us/Codes/Characteristictype?mimeType=xml | ||
Characteristicname | Characteristic names. A good choice for using paginated results so that hundreds of results are not returned. | https://www.waterqualitydata.us/Codes/Characteristicname?mimeType=xml | ||
providers | The names of the Data Sources for the Water Quality Portal | https://www.waterqualitydata.us/Codes/providers?mimeType=xml |