Metadata-Version: 2.4
Name: rws-waterinfo
Version: 1.2.0
Summary: A Python wrapper for the Rijkswaterstaat WaterWebservices (ddapi20) API. Simplifies downloading observation data, forecasts, tidal extremes, chemical concentrations and more.
Home-page: https://gitlab.com/st14n/rws-waterinfo
Author: RWS | Stijn Schmitz
Author-email: stijn.schmitz@rws.nl
Project-URL: Documentation, https://st14n.gitlab.io/rws-waterinfo/
Project-URL: Source, https://gitlab.com/st14n/rws-waterinfo
Project-URL: Issues, https://gitlab.com/st14n/rws-waterinfo/-/work_items
Keywords: Rijkswaterstaat waterinfo ddapi20 hydrology water
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas>=1.2
Requires-Dist: requests
Provides-Extra: dev
Requires-Dist: bandit; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: flake8-bugbear; extra == "dev"
Requires-Dist: flake8-comprehensions; extra == "dev"
Requires-Dist: flake8-docstrings; extra == "dev"
Requires-Dist: flake8-polyfill; extra == "dev"
Requires-Dist: isort; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: pip-audit; extra == "dev"
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: pylint; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: radon; extra == "dev"
Requires-Dist: rstcheck[sphinx]; extra == "dev"
Requires-Dist: sh; extra == "dev"
Requires-Dist: types-requests; extra == "dev"
Provides-Extra: gui
Requires-Dist: streamlit>=1.46; extra == "gui"
Requires-Dist: plotly>=5.0; extra == "gui"
Requires-Dist: openpyxl; extra == "gui"
Requires-Dist: pandas>=2.0; extra == "gui"
Provides-Extra: doc
Requires-Dist: pydata-sphinx-theme; extra == "doc"
Requires-Dist: sphinx; extra == "doc"
Requires-Dist: sphinx-autodoc-typehints; extra == "doc"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: summary

# rws_waterinfo

A Python wrapper for the Rijkswaterstaat [WaterWebservices (ddapi20)](https://rijkswaterstaatdata.nl/waterdata/) API.  
Simplifies downloading observation data, forecasts, tidal extremes, chemical concentrations and more from the new WADAR archive.

**Documentation:** <https://st14n.gitlab.io/rws-waterinfo/> · **PyPI:** <https://pypi.org/project/rws-waterinfo/>

**Key changes vs. the classic WaterWebservices:**
- Location codes are unified (e.g. `ameland.nes`, `hoekvanholland`) — no X/Y coordinates needed.
- `ProcesType` distinguishes measured data (`meting`), forecasts (`verwachting`) and astronomical tides (`astronomisch`). Default is `meting`.
- All columns from the API are returned without filtering.
- Dict-based request format gives full control over every API parameter.

---

## What's inside / Wat zit erin

| Capability | What | Needs |
|---|---|---|
| [Catalog](#csv-catalog-and-regions) | `get_catalog()`; a local, versioned CSV copy with `load_catalog()`, `update_catalog()`, `catalog_info()`, `filter_catalog()` | base |
| [Download](#request-formats) | `get_data()`: list or dict requests, `parallel`/`max_workers`, `columns=`, `progress_callback=`, `normen=`, `filepath=`; failed requests in `df.attrs`; `update_dataframe()` | base (network) |
| [Regions](#csv-catalog-and-regions) | KRW waterlichaam and province per location (`assign_regions`, `region_geojson`), filter on them | base, polygons bundled |
| [Norms](#water-quality-norms) | Bkl norms and the KRW stoffen koppeltabel bundled; RIVM and KRW-doelen (WKP) fetched on request (`update_normen`); `assign_normen`, `normtoetsing`, the rapportagegrens choice (`half`/`nul`/`waarde`) | base; optional layers need network once |
| [Observation helpers](#checking-for-duplicate-measurements) | `find_duplicates()`, `tidal_extremes()`, `last_failed_requests()` | base |
| [Water quality analysis](#water-quality-analysis-waterkwaliteit) | `rws_waterinfo.waterkwaliteit`: normtoetsing (JG-MKN/MAC-MKN) against the KRW stoffen koppeltabel (default) or the Bkl (`normbron`), trends (Mann-Kendall, Sen's slope), PFAS/PEQ, stofgroepen and stofgroep sets, correlaties, risico, kaart (GeoJSON), bovenstroom, datakwaliteit, Excel report, charts. **NOT VERIFIED** - see the disclaimer | base; charts need plotly, report needs openpyxl (both in `[gui]`) |
| [Charts](#charts-visualisatie) | `rws_waterinfo.visualisatie`: the WaterTrends chart modes (Tijdreeks with min-max band, Multi-locatie, Locatievergelijking, Jaarvergelijking, Multi-grootheid, stofgroep pages) with filters, IQR outlier correction, aggregation, trend and norm lines | base; figures need plotly (`[gui]`) |
| [Command line](#command-line) | `rws-waterinfo catalogus / download / normen / waterkwaliteit` | base (`.xlsx` output needs openpyxl) |
| [GUI](#gui) | `rws-waterinfo-gui`, five steps: locations (filters, map and table clicks), series (also stofgroep sets), period (quick picks) and download, results in five chart modes with filters/outlier correction/aggregation and export, and the Waterkwaliteit page (nine sections, norm source choice) | `pip install "rws-waterinfo[gui]"` |
| [Adapting](#adapting-and-extending) | where the code lives and how to change a chart or a calculation (`doc/adapteren.rst`) | - |
| [Example scripts](#example-scripts) | `example.py`, `examples/waterkwaliteit_voorbeeld.py`, `examples/visualisatie_voorbeeld.py` (offline sample included) | in the source distribution / repository |

---

## Install

```
pip install rws-waterinfo          # library: pandas, numpy, requests
pip install "rws-waterinfo[gui]"   # plus the optional GUI (Streamlit, Plotly)
```

---

## Quick start

Snelstart. Install (above), then:

```python
import rws_waterinfo as rw

rw.set_logger("info")

# 1. Full catalog — all locations × parameters, including ProcesTypes
catalog = rw.get_catalog()
catalog.to_csv("catalog.csv", index=False)

# 2. Measured water levels (ProcesType='meting' is the default)
#    List format: [compartiment, eenheid, meetapparaat, grootheid, locatie, start, end]
#    Use "" to omit optional fields.
params = [
    ["OW", "", "", "WATHTE", "ameland.nes", "2025-12-01", "2026-01-01"],
]
df = rw.get_data(params, return_df=True, max_workers=10)

# 3. Local CSV catalog with KRW waterlichaam and province per location
catalog = rw.load_catalog()   # downloads once, then reads the CSV copy (~0.3 s)
waal = rw.filter_catalog(catalog, krw_waterlichaam="Bovenrijn, Waal", grootheid="WATHTE")
gelderland = rw.filter_catalog(catalog, provincie="Gelderland", zoekterm="debiet")
```

The same from the command line, and the GUI:

```
rws-waterinfo download --locatie ameland.nes --grootheid WATHTE --van 2025-12-01 --tot 2025-12-31 --uitvoer ameland.csv
rws-waterinfo-gui                      # needs pip install "rws-waterinfo[gui]"
```

Water quality analysis of chemistry (here: the bundled sample of the source
distribution; any `get_data()` result or its CSV works). **Not verified by
Rijkswaterstaat: an aid, not an official assessment - see the
[disclaimer](#water-quality-analysis-waterkwaliteit).**

```python
from rws_waterinfo import waterkwaliteit as wk

analyse = wk.analyse("examples/data/chemie_voorbeeld.csv.gz")   # or wk.analyse(df)
analyse.normtoetsing["samenvatting"]    # JG-MKN per year, MAC-MKN per sample (KRW-normtabel)
analyse.trends["trends"]                # Mann-Kendall + Sen's slope per location and substance
analyse.rapport("waterkwaliteit.xlsx")  # every section, explanation and disclaimer (openpyxl)
```

More: the [example scripts](#example-scripts).

### Charts (visualisatie)

The chart modes of the WaterTrends page of the netwerkmonitoring-tool, as
functions you can use and adapt (computation needs only pandas, the figures
plotly):

```python
import pandas as pd
from rws_waterinfo import visualisatie as vis

df = pd.read_csv("examples/data/chemie_voorbeeld.csv.gz")     # the bundled sample (or get_data())
df["Naam"] = df["Code"].str.title()
df[vis.data.REEKS] = df["Parameter.Omschrijving"] + " (" + df["Eenheid.Code"] + ")"
data, n = vis.corrigeer_uitbijters(vis.meetwaarden(df))        # IQR x3, interpolated
reeks = "zink (ug/l)"
vis.beschikbare_modi(data)                                       # Tijdreeks, Multi-locatie, ...
band = vis.min_max_band(data, reeks, "Week")
vis.figuren.tijdreeks_figuur(band, reeks, "ug/l").write_html("tijdreeks.html")
jaren = vis.jaarvergelijking(data, reeks, vis.lijn_namen(data, reeks)[0])
vis.figuren.jaarvergelijking_figuur(jaren, reeks)              # .show() in a notebook
```

Also `filter_seizoen`, `filter_kwaliteit`, `filter_variant` (instrument, depth),
`aggregeer`, `multi_grootheid`, `trendlijn`, `statistieken`, `normlijnen`. The
stofgroep sets (`waterkwaliteit.stofgroepen.stofgroep_reeksen`) pick all series of
a substance group. See the [documentation](https://st14n.gitlab.io/rws-waterinfo/visualisatie.html).

### Adapting and extending

Every chart and number of the GUI is a public function, so you can change how a chart
looks or how something is computed without forking: a figure builder returns a plotly
`Figure` you can restyle, the computed points can go into your own plotly code, an own
outlier rule only has to return the frame with the outliers marked, the norm lines are a
list you can extend, and the bundled norm table (`rws_waterinfo/data/`) can be edited in a
source checkout. Where the code lives and worked examples:
[the documentation](https://st14n.gitlab.io/rws-waterinfo/adapteren.html).

---

## GUI

```
pip install "rws-waterinfo[gui]"
rws-waterinfo-gui                      # or: python -m rws_waterinfo.gui
rws-waterinfo-gui --catalogus D:\data\catalogus --server.port 8600
```

A Streamlit app on top of the library (it uses the CSV catalog below, no database), in five
steps and a sidebar: **1 Locaties**: filter by province, KRW waterlichaam, compartiment,
grootheid or free text, see the locations on a map, and choose them by clicking a point on
the map or a cell in the table (a second click removes it). **2 Reeksen**: choose series
(one per `Parameter_Wat_Omschrijving`) or a whole stofgroep (such as PFAS or Metalen) as a
set. **3 Periode en downloaden**: quick picks (7 dagen to 5 jaar) or dates, a summary of the
API calls, and a download with a progress bar. **4 Resultaat**: the chart modes of
`rws_waterinfo.visualisatie` (Tijdreeks with min-max band, Multi-locatie, Locatievergelijking,
Jaarvergelijking, Multi-grootheid) with filters (quality code, season, instrument, depth),
IQR outlier correction (as on WaterTrends), aggregation, limit and trend lines, and an export to
CSV or Excel. **5 Waterkwaliteit**: see below. A user guide of every element:
[the documentation](https://st14n.gitlab.io/rws-waterinfo/gui.html).

With "Normen tonen en meenemen" (on by default) the chart shows the
[water quality norms](#water-quality-norms) as horizontal lines (JG-MKN
dashed, MAC-MKN dotted, drinkwaterinname dash-dot; indicative RIVM norms
lighter, the KRW-doel of the waterlichaam as a shaded "goed" band), the
overview lists the norm values, a table compares each year with the norms,
and the export gets the `Norm.*` columns. Values below the reporting limit
("<") count as half the limit, zero or as reported (sidebar choice; the
table and export keep the reported values). "Normen vernieuwen (RIVM/WKP)"
downloads the optional norm layers.

After a download with chemistry (CONCTTE/MASSFTE) a **Waterkwaliteit**
section appears, laid out as on the Waterkwaliteit page of the netwerkmonitoring-tool
(a row of sections, "Overige normen", two-step pickers), with a not-verified warning
at the top. The sidebar chooses the norm source (the KRW stoffen koppeltabel by default,
as in the netwerkmonitoring-tool; where it differs from the Bkl the normtoetsing says so,
or the legal Bkl norms): normtoetsing,
trends, PFAS/PEQ, map, risk, stofgroepen, correlations (grey diagonal),
upstream/downstream comparison and data quality, each with an explanation,
plus an Excel report. Its settings (norm layers, indicative norms, refreshing
the RIVM stofgroepen and norm layers) are in the sidebar under
"Waterkwaliteit"; the reporting-limit choice is shared with the charts.

## Command line

```
rws-waterinfo catalogus --vernieuw                      # download the catalog
rws-waterinfo catalogus --info
rws-waterinfo catalogus --provincie Zeeland --grootheid CONCTTE --zoek zink --normen --uitvoer zink.xlsx
rws-waterinfo download --locatie lobith.ponton eijsden --parameter Cd Zn --van 2024-01-01 --tot 2024-12-31 \
    --uitvoer metalen.csv --normen --rapportagegrens half --toetsing toetsing.xlsx
rws-waterinfo download --uit-csv zink.csv --van 2024-01-01 --tot 2024-12-31 --uitvoer zink_metingen.csv \
    --kolommen Code,Tijdstip,Meetwaarde.Waarde_Numeriek,Meetwaarde.Waarde_Limietsymbool
rws-waterinfo normen --parameter Cd PFOS                # print the norms
rws-waterinfo normen --vernieuwen --bron rivm wkp       # download the optional layers
rws-waterinfo waterkwaliteit --invoer metalen.csv --uitvoer rapport.xlsx
rws-waterinfo waterkwaliteit --locatie lobith.ponton --van 2015-01-01 --tot 2024-12-31 \
    --rapportagegrens half --uitvoer rapport.xlsx
rws-waterinfo --help                                    # Dutch help per subcommand
```

* `catalogus`: update, show (`--info`) or export (`--uitvoer .csv/.xlsx`) the
  CSV catalog, filtered with `--provincie`, `--krw`, `--locatie`,
  `--grootheid`, `--parameter`, `--compartiment`, `--zoek`; `--normen` adds
  the norm columns (`--normen-lagen bkl rivm wkp` chooses the layers).
* `download`: series chosen by location(s) plus grootheid/parameter/
  `--omschrijving` (text in `Parameter_Wat_Omschrijving`), or the rows of a
  CSV exported by `catalogus` (`--uit-csv`). `--rapportagegrens {half,nul,waarde}`
  replaces values below the reporting limit (and adds `Onder_rapportagegrens`);
  without it the values stay as reported. `--toetsing FILE` writes the yearly
  comparison with the norms. Exit code 1 when nothing was downloaded.
* `normen`: print or export the norm table, `--vernieuwen` the optional layers.
* `waterkwaliteit`: the [water quality analysis](#water-quality-analysis-waterkwaliteit)
  of a CSV/parquet (`--invoer`) or of a download (the same filters as
  `download`; without a choice of series all chemistry of the locations):
  prints the not-verified disclaimer, notices and a summary, and with
  `--uitvoer x.xlsx` writes the report. `--rapportagegrens`, `--normen-lagen`,
  `--indicatief`, `--hardheidsklasse`; `--vernieuwen` first refreshes the norm
  layers and the RIVM stofgroepen.

---

## CSV catalog and regions

`load_catalog()` keeps a copy of the catalog in the user's cache folder
(`%LOCALAPPDATA%\rws_waterinfo\catalogus` on Windows, `~/.cache/...` elsewhere,
or `$RWS_WATERINFO_CATALOG_DIR`), as three CSV files the way the API delivers
it (locations, AquoMetadata, combinations: ~8 MB instead of ~100 MB for one
wide CSV). It is downloaded again after `max_age_days` (default 7); when that
fails the stored copy is used. A new download replaces the stored copy in one
step, only once it is complete; an empty catalog, or one with less than half
the stored rows, is not stored (`update_catalog(force=True)` stores it
anyway). Several processes or app sessions can share the folder: updates wait
for each other, and a damaged copy is downloaded again.

```python
catalog = rw.load_catalog()                        # same columns as get_catalog() + regions
catalog = rw.load_catalog(refresh=True)            # force a fresh download
catalog = rw.load_catalog(categorical=True)        # ~20 MB in memory instead of ~500 MB
rw.catalog_info()                                  # downloaded_at, age_days, rows, locations
```

Every location gets a **KRW waterlichaam** (the 54 KRW surface water bodies of
the Rijkswateren) and a **province** (Kadaster, including the Waddenzee and
IJsselmeer): the polygon it lies in, or the nearest one within 1 km
(harbours, beach transects); stations far out at sea get none. The polygons
ship with the package; only numpy is used.

| Column | Example |
|---|---|
| `KRW_Waterlichaam.Code` / `.Omschrijving` / `.Afstand_m` | `NL93_8` / `Bovenrijn, Waal` / `0.0` |
| `Provincie.Code` / `.Omschrijving` / `.Afstand_m` | `PV25` / `Gelderland` / `0.0` |

```python
df = rw.load_catalog(regions=False)                  # any table with Lat/Lon, e.g. get_catalog() or get_data()
df = rw.assign_regions(df)
rw.region_names("krw"), rw.region_names("provincie")
rw.region_geojson("krw")                             # polygons for your own maps
```

---

## Water quality norms

```python
import rws_waterinfo as rw

rw.normen(parameter="Cd")                       # the norm table (long format)
catalog = rw.load_catalog(normen=True)          # norm columns per catalog row
df = rw.get_data(params, return_df=True, normen=True)
toetsing = rw.normtoetsing(df, rapportagegrens="half")   # per series and year
df_rg = rw.apply_rapportagegrens(df, "half")    # values below "<" as half the limit
rw.convert_unit(0.08, "ug/l", "ng/l")           # 80.0
```

`assign_normen(df)` (and `normen=True` on `get_data`, `get_data_parallel`
and `load_catalog`) adds, in the series' own unit:

| Column | Meaning |
|---|---|
| `Norm.JG_MKN`, `Norm.MAC_MKN` | legally binding annual-mean / maximum standard for the location's water type |
| `Norm.Biota` | for organism (OE) series, µg/kg wet weight |
| `Norm.Drinkwater`, `Norm.Drinkwater_min` | applies at drinking water intake points (`_min`: a lower bound such as pH 7,0) |
| `Norm.Indicatief_JG`, `Norm.Indicatief_MAC` | RIVM, policy-set or indicative (JG-MKN, else MTR; MAC) — optional layer |
| `Norm.KRW_goed_min`, `Norm.KRW_goed_max` | the "goed" class of the KRW doel of the location's waterlichaam — optional layer |
| `Norm.Eenheid`, `Norm.Watertype`, `Norm.Laag`, `Norm.Bron`, `Norm.Opmerking` | unit, water type, layer(s), source, notes (dissolved/total, hardness, DOC, sum parameters, how a KRW doel is assessed) |

Landoppervlaktewater (rivers, lakes) or andere oppervlaktewateren (transitional
and coastal waters) follows from the location's KRW waterlichaam. A norm is
only applied to the hoedanigheid it names (a dissolved-metal norm not to a
total-metal series). Cadmium depends on the water hardness class: without
`hardheidsklasse=` the strictest class is used, with a note. A norm of a sum
parameter (e.g. PFOS lineair + vertakt) is shown for its constituents, with a
note. Nothing is downloaded by `assign_normen` or `get_data`.

Layers:

* **bkl** (always, bundled `rws_waterinfo/data/normen.csv`): the omgevings-
  waarden of the Besluit kwaliteit leefomgeving (BWBR0041313) bijlage III
  (priority substances), IIIa (specific pollutants) and V (drinking water
  intake), keyed by the Aquo codes RWS uses (Aquo hulplijsten). Built by
  `scripts/build_normen.py`; sources and licences in
  `rws_waterinfo/data/NORMEN_BRONNEN.md`. Government information, free to reuse.
* **rivm** (optional): RIVM [Risico's van stoffen](https://rvszoeksysteem.rivm.nl/)
  — also policy-set and indicative norms (MTR, ...). Not bundled: RIVM allows
  non-commercial use with attribution ("Bron: RIVM, Risico's van stoffen").
  RIVM rows that repeat a Bkl value are left out; the Bkl row notes whether
  RIVM agrees.
* **wkp** (optional): the KRW-doelen per waterlichaam from the
  [Waterkwaliteitsportaal](https://wkp.rws.nl/) ("Bron: Informatiehuis Water /
  Waterkwaliteitsportaal") — chloride, fosfor/stikstof totaal, temperature,
  oxygen saturation, pH and transparency.

`rw.update_normen(bronnen=("rivm", "wkp"))` downloads the optional layers
into the user cache (`default_normen_dir()`, or `$RWS_WATERINFO_NORMEN_DIR`),
at most one request per second (RIVM takes a few minutes). A layer that fails
or looks implausible keeps its previous copy and raises `RuntimeError`.
`rw.normen_info()` shows what is cached. **Caveat:** both sources are
web applications without a documented API; when they change, the update
fails (the cached copy stays usable) until the parser is adapted.

Not included: sediment/soil norms of the Bkl, the generic drinking water
norms for PAH and plant protection products (no single Aquo parameter), and
the KRW assessment statistics (e.g. summer-half-year mean) — `normtoetsing`
compares yearly means and single values.

**KRW stoffen koppeltabel.** The Waterkwaliteit analysis tests against the
national KRW norm table ("KRW stoffen koppeltabel", Aquo / Informatiehuis
Water), bundled as `rws_waterinfo/data/krw_stoffen_koppeltabel.csv` (254 rows:
JG-MKN and MAC-MKN per substance for surface water; the table the
netwerkmonitoring-tool uses). `normbron="koppeltabel"` is the default of
`wk.analyse()`; `normbron="bkl"` tests against the legal Bkl values instead
(CLI `--normbron`, GUI sidebar "Normbron"). Where the two differ (arsenic
MAC-MKN 8.6 against 8 µg/l, p-xyleen JG-MKN 170 against 17 µg/l, ...) the
analysis names both (`wk.normen.bkl_afwijking`, the "Bkl-afwijking" column and
a dotted line in the detail chart). A reference table can be out of date and
the results are **not verified**: see the disclaimer and
`rws_waterinfo/data/NORMEN_BRONNEN.md`. `rw.normtoetsing` and
`rw.assign_normen` keep testing against the Bkl.

---

## Water quality analysis (waterkwaliteit)

> **NOT VERIFIED - NIET GEVERIFIEERD.** The results of `rws_waterinfo.waterkwaliteit`
> are **not verified or validated by Rijkswaterstaat**. They are an aid, not an official
> assessment (no official KRW/WFD or BKMW assessment). Check them against the official
> sources yourself. De uitkomsten zijn **niet geverifieerd of gevalideerd door
> Rijkswaterstaat**: een hulpmiddel, geen officieel oordeel (geen officiële KRW- of
> BKMW-toetsing). Controleer ze zelf aan de officiële bronnen.

Installed with the package, imported only when used (`import rws_waterinfo`
stays light); numpy and pandas only, plotly for the charts, openpyxl for the
Excel report. A one-time `UserWarning` repeats the disclaimer;
`wk.DISCLAIMER` holds the full text (also in the report and the GUI).

```python
import rws_waterinfo as rw
from rws_waterinfo import waterkwaliteit as wk

df = rw.get_data([{"locatie_code": "lobith.ponton", "grootheid_code": "CONCTTE",
                   "start_date": "2020-01-01", "end_date": "2024-12-31"}], return_df=True)
analyse = wk.analyse(df, rapportagegrens="half")   # nothing computed yet

analyse.normtoetsing["samenvatting"]   # per location x substance x norm
analyse.normtoetsing["niet_getoetst"]  # norm for another fraction or unit
analyse.trends["trends"]               # Mann-Kendall + Sen's slope (>= 5 years)
analyse.pfas["samenvatting"]           # PFOA-equivalents (RIVM RPF's)
analyse.stofgroepen["overzicht"]       # per substance group
analyse.correlatie("lobith.ponton")    # Pearson matrix; analyse.correlatie_figuur(...) with grey diagonal
analyse.risico["prioritering"]         # substances by exceedances
analyse.kaart["waterlichamen"]         # GeoJSON with the exceedance rollup
analyse.bovenstroom                    # candidate location pairs; analyse.vergelijk(a, b)
analyse.datakwaliteit                  # duplicates, censoring, quality codes, completeness
analyse.meldingen                      # e.g. optional sources not downloaded / older than 30 days
analyse.uitleg("trends")               # what a section computes (Dutch)
analyse.rapport("waterkwaliteit.xlsx")
```

* **Input**: `get_data()` output, its CSV, or a parquet/CSV export of the
  netwerkmonitoring-tool. Only grootheid CONCTTE and MASSFTE is chemistry;
  placeholder values (>= 999999) and negative concentrations are skipped and
  counted.
* **Rapportagegrens** (values "<"): in means and medians as half the limit
  (BKMW, default), 0 or the limit itself; a single value below the limit is
  always tested against the limit itself. Results that depend on unknown values
  below the limit are "niet toetsbaar" and do not count in percentages.
* **Normtoetsing**: against the KRW stoffen koppeltabel by default (as the
  netwerkmonitoring-tool does; `normbron="bkl"` tests against the legal Bkl
  norms of [Water quality norms](#water-quality-norms) instead, per series and
  location with the watertype from the KRW waterlichaam): JG-MKN on the
  annual mean per location, substance and calendar year, MAC-MKN per sample.
  Where the koppeltabel differs from the Bkl, both are shown.
  `indicatief=True` also tests against RIVM indicative norms (rivm layer).
* **Trends**: yearly medians, Mann-Kendall test and Sen's slope per year,
  identical to pymannkendall (without scipy).
* **PFAS/PEQ**: concentration (ng/l) x RPF summed per water sample; measured
  values only, so a lower bound. RPF table bundled
  (`rws_waterinfo/data/pfas_rpf.csv`, "Bron: RIVM").
* **Stofgroepen**: derived, not maintained by hand: the RIVM
  [Risico's van stoffen](https://rvszoeksysteem.rivm.nl/) substance groups
  (functional lists such as KRW prioritaire stoffen, chemical groups such as
  PFAS and PAK's), linked by CAS number or name, after
  `wk.update_stofgroepen()` (about 2 minutes, at most 1 request per second;
  non-commercial use, "Bron: RIVM, Risico's van stoffen"); plus fixed rules
  (element symbols, nutrients, keywords). On the RWS catalog's 832 CONCTTE
  parameters: 76% classified with the rules alone, 90% with the RIVM groups and
  the RIVM norm layer.
* **Performance**: one million measurements in about 5 s for all sections
  (laptop), results cached per section.

Nothing is downloaded automatically; `rw.update_normen()` and
`wk.update_stofgroepen()` refresh the optional sources on request and keep
the previous copy when a download fails. Sources: measurements Rijkswaterstaat;
norms KRW stoffen koppeltabel (Aquo / Informatiehuis Water), Bkl (BWBR0041313),
RIVM and Informatiehuis Water / Waterkwaliteitsportaal when used; substance
groups and RPF's RIVM.

---

## Performance and memory

- Downloads run in parallel (`max_workers`, default 10); periods are split
  into requests of at most a year, and further when the API reports too many
  observations.
- `columns=[...]` keeps only the columns you need, per request, as results
  arrive: the biggest memory saver for large downloads.
- `filepath=...` without `return_df` streams each result to the CSV and keeps
  nothing in memory (the file always gets all columns; `columns` only applies
  to the returned DataFrame).
- `progress_callback=lambda done, total: ...` reports progress; the console
  progress is silenced by `set_logger("warning")`.
- A request that times out on a long period is split in two; the timeouts are
  `rws_waterinfo.rws_waterinfo.OBSERVATION_TIMEOUT` (default `(10, 60)` s,
  connect/read) and `CATALOG_TIMEOUT`.

```python
df = rw.get_data(params, return_df=True,
                 columns=["Code", "Tijdstip", "Meetwaarde.Waarde_Numeriek"],
                 progress_callback=lambda done, total: print(done, "/", total))
```

### Failed requests

A request that fails (HTTP error, timeout, an API `Foutmelding`) adds no rows
and is logged. Which ones failed:

```python
df = rw.get_data(params, return_df=True)
df.attrs["failed_requests"]     # [{"locatie_code": ..., "start_date": ..., "error": ...}, ...]
rw.last_failed_requests()       # the same, also after get_data(filepath=...); per thread
```

---

## Request formats

### List format (backwards compatible, 7 items)

```python
# [compartiment, eenheid, meetapparaat, grootheid, locatie, start_date, end_date]
params = [
    ["OW", "", "", "WATHTE", "ameland.nes", "2025-12-01", "2025-12-08"],
]
df = rw.get_data(params, return_df=True)
```

### Dict format (recommended — full API control)

```python
params = [
    {
        "locatie_code":      "ameland.nes",       # required
        "start_date":        "2025-12-01",         # required
        "end_date":          "2026-01-01",         # required
        # --- optional fields (omit to leave unfiltered) ---
        "compartiment_code":          "OW",
        "eenheid_code":               "cm",
        "meetapparaat_code":          127,
        "grootheid_code":             "WATHTE",
        "proces_type":                "meting",    # overrides global default
        "parameter_code":             None,        # e.g. "Cd" for cadmium
        "hoedanigheid_code":          None,
        "typering_code":              None,
        "waardebewerkingsmethode_code": None,
        "groepering_code":            None,        # e.g. "GETETBRKD2" for tidal extremes
        "waardebepalingsmethode_code": None,       # e.g. "other:F137" (wind gust, see below)
        "biotaxon_code":              None,        # biology: a BioTaxon code
        "kwaliteitswaardecodes":      None,        # e.g. ["00","10","20","25","30","40"]
        "bemonsteringshoogtes":       None,        # e.g. ["-250"] for 2.5 m depth
        "opdrachtgevende_instanties": None,        # e.g. ["RIKZ_GOLVEN"]
    }
]
df = rw.get_data(params, return_df=True)
```

> **Filtering on codes** (`parameter_code`, `hoedanigheid_code`, `typering_code`,
> `waardebewerkingsmethode_code`, `groepering_code`): use the exact code as shown in
> `get_catalog()`, including "not applicable" codes such as `"NVT"` — the RWS data
> model has no empty value for these fields. Leaving a field as `None` (the default)
> just omits it from the request ("don't filter"), it does **not** match an empty
> value. Some catalog entries only differ by one of these codes (e.g. a raw `"NVT"`
> series vs. a derived `"GEM24H"` daily-average series for the same
> location/parameter) — pass the specific code to select just one.

---

## ProcesType

`ProcesType` is an extra filter that distinguishes:

| Value | Meaning |
|---|---|
| `"meting"` | Measured observations (default) |
| `"verwachting"` | Model forecasts (e.g. expected water levels, 48 h ahead) |
| `"astronomisch"` | Astronomical tidal predictions (no weather effect) |
| `None` | No filter — returns all types |

Set it globally for a whole batch:

```python
df = rw.get_data(params, return_df=True, proces_type="verwachting")
```

Or override it per request using the dict format:

```python
params = [
    {"locatie_code": "hoekvanholland", "compartiment_code": "OW",
     "grootheid_code": "WATHTE", "start_date": "2025-12-01",
     "end_date": "2025-12-08", "proces_type": "meting"},
    {"locatie_code": "hoekvanholland", "compartiment_code": "OW",
     "grootheid_code": "WATHTE", "start_date": "2025-12-01",
     "end_date": "2025-12-08", "proces_type": "verwachting"},
]
df_both = rw.get_data(params, return_df=True)
```

---

## Examples

### Example scripts

* [`example.py`](example.py): catalog, `get_data` in both request formats, saving and updating.
* [`examples/waterkwaliteit_voorbeeld.py`](examples/waterkwaliteit_voorbeeld.py): the whole
  water quality analysis (normtoetsing and norms, trends, PFAS/PEQ, stofgroepen,
  correlaties, risico, kaart, bovenstroom, datakwaliteit, charts and report).
  Runs offline on the bundled sample `examples/data/chemie_voorbeeld.csv.gz`, or
  on fresh data with `--download`. **Not verified** (see the disclaimer above).
  See [`examples/README.md`](examples/README.md).
* [`examples/visualisatie_voorbeeld.py`](examples/visualisatie_voorbeeld.py): every chart mode
  of `rws_waterinfo.visualisatie` written as HTML, offline on the same sample.
* The documentation pages [visualisatie](https://st14n.gitlab.io/rws-waterinfo/visualisatie.html),
  [waterkwaliteit](https://st14n.gitlab.io/rws-waterinfo/waterkwaliteit.html),
  [gui](https://st14n.gitlab.io/rws-waterinfo/gui.html) and
  [adapteren](https://st14n.gitlab.io/rws-waterinfo/adapteren.html) have runnable snippets.

### Tidal extremes (Groepering / mux)

Use `groepering_code` to request multiplex channel groups. Do **not** set `grootheid_code` when using `groepering_code`. When `groepering_code` is set and `proces_type` is not explicitly provided, the wrapper automatically sets `proces_type=None` (because grouped observations span multiple process types and filtering on `"meting"` would return nothing).

```python
params = [
    {"locatie_code": "ameland.nes", "groepering_code": "GETETBRKD2",
     "start_date": "2025-12-01", "end_date": "2026-01-01"},
]
df_tides = rw.get_data(params, return_df=True)
```

| Code | Description |
|---|---|
| `GETETBRKD2` | Calculated tidal extremes (NAP) |
| `GETETM2` | Measured tidal extremes (NAP) |
| `GETETBRKDMSL2` | Calculated tidal extremes offshore (MSL) |
| `GETETMSL2` | Measured tidal extremes offshore (MSL) |

The result holds two series per extreme: the water level and its kind
(`hoogwater` / `laagwater`, in `Meetwaarde.Waarde_Alfanumeriek`).
`rw.tidal_extremes(df_tides)` puts them side by side: one row per extreme with
`Waterhoogte` and `Type`.

### Wind gusts

The grootheid `WINDST` no longer exists (the catalog still lists it, without
data). Gusts are `WINDSHD` with a WaardeBepalingsMethode:

```python
params = [{"locatie_code": "uithuizerwad", "grootheid_code": "WINDSHD",
           "waardebepalingsmethode_code": "other:F137",
           "start_date": "2026-08-01", "end_date": "2026-08-03"}]
```

### Chemical concentrations

Combine `grootheid_code` (e.g. `CONCTTE`) with `parameter_code` (e.g. `Cd`).  
Set `proces_type=None` for chemistry — it is not applicable there.

```python
params = [
    {"locatie_code": "lobith.ponton", "compartiment_code": "OW",
     "grootheid_code": "CONCTTE", "parameter_code": "Cd",
     "start_date": "2023-01-01", "end_date": "2024-01-01",
     "proces_type": None},
]
df_cd = rw.get_data(params, return_df=True)
```

### Sampling depth filter

```python
params = [
    {"locatie_code": "hoekvanholland", "compartiment_code": "OW",
     "grootheid_code": "SALNTT", "start_date": "2025-12-01",
     "end_date": "2026-01-01", "bemonsteringshoogtes": ["-250"]},  # -2.5 m
]
```

### Quality code filter

```python
params = [
    {"locatie_code": "ameland.nes", "compartiment_code": "OW",
     "grootheid_code": "WATHTE", "start_date": "2025-12-01",
     "end_date": "2026-01-01",
     "kwaliteitswaardecodes": ["00", "10", "20", "25", "30", "40"]},
]
```

### Commissioning body filter

Prevents receiving data from other organisations that also measured at the same location.

```python
params = [
    {"locatie_code": "ameland.nes", "compartiment_code": "OW",
     "grootheid_code": "WATHTE", "start_date": "2025-12-01",
     "end_date": "2026-01-01",
     "opdrachtgevende_instanties": ["RIKZ_GOLVEN"]},
]
```

### Save to CSV and update later

```python
rw.get_data(params, filepath="waterlevel.csv")

# Extend the CSV to a newer date
df_updated = rw.update_dataframe("waterlevel.csv", till_date="2026-06-01")
df_updated.to_csv("waterlevel.csv", index=False)
```

`update_dataframe` asks again for each series in the file: per location,
compartiment, eenheid, meetapparaat and grootheid, and (when the file has
them) parameter, hoedanigheid, typering, WaardeBewerkingsMethode, ProcesType
and groepering. A file of e.g. several substances at one location, forecasts
or tidal extremes is extended with the same series.

### Checking for duplicate measurements

```python
dupes = rw.find_duplicates(df)   # rows that occur more than once, with a same_value column
```

---

## API reference

### `get_catalog() → DataFrame`

Downloads the full catalog (all location × parameter combinations).  
Includes `ProcesTypes`, `Grootheden`, `Compartimenten`, `Eenheden`, `Parameters`, `Hoedanigheden`, `Typeringen`, `WaardeBewerkingsMethoden` and more.

### `get_data(params, filepath=None, return_df=False, parallel=True, max_workers=10, proces_type="meting", progress_callback=None, columns=None, normen=False) → DataFrame | None`

Downloads observations. See [Request formats](#request-formats) above. Failed
requests: `df.attrs["failed_requests"]` and `last_failed_requests()`.
`normen=True` (or a list of layers) adds the [norm columns](#water-quality-norms).

### `load_catalog(path=None, max_age_days=7, refresh=False, regions=True, categorical=False, normen=False) → DataFrame`

The catalog from the local CSV copy; see [CSV catalog and regions](#csv-catalog-and-regions).
Also `update_catalog(path=None, regions=True, force=False)`, `catalog_info(path=None)`,
`default_catalog_dir()`.

### `filter_catalog(catalog, provincie=None, krw_waterlichaam=None, locatie=None, grootheid=None, parameter=None, compartiment=None, proces_type=None, zoekterm=None) → DataFrame`

Rows matching every given filter (names or codes; a string or a list).

### `assign_regions(df, layers=("krw", "provincie"), lat_col="Lat", lon_col="Lon", max_distance_m=None) → DataFrame`

Adds the KRW waterlichaam and/or province columns.

### `update_dataframe(filepath, till_date) → DataFrame`

Reads an existing CSV saved by `get_data()`, re-fetches the same series up to `till_date`, and appends the new rows.

### `tidal_extremes(df) → DataFrame`, `find_duplicates(df, keys=None) → DataFrame`, `last_failed_requests() → list`

See the examples above.

### Norms

`normen(parameter=None, normtype=None, watertype=None, lagen=None, path=None)`,
`assign_normen(df, watertype=None, hardheidsklasse=None, lagen=None, normen_dir=None)`,
`convert_unit(value, from_unit, to_unit)`,
`rapportagegrens_waarden(values, limietsymbool, rapportagegrens="half")`,
`apply_rapportagegrens(df, rapportagegrens="half")`,
`normtoetsing(df, rapportagegrens="half", groepen=None)`,
`update_normen(bronnen=("rivm", "wkp"), path=None, jaar=None)`, `normen_info(path=None)`,
`default_normen_dir()`. See [Water quality norms](#water-quality-norms).

### `waterkwaliteit.analyse(data, rapportagegrens="half", normen_lagen=None, **opties) → Analyse`

Lazily computed sections `normtoetsing`, `trends`, `pfas`, `risico`, `stofgroepen`,
`correlaties`, `bovenstroom`, `datakwaliteit`, `kaart`; `correlatie(loc)`,
`vergelijk(a, b)`, `rapport(path)`, `meldingen`, `uitleg(onderdeel)`, `disclaimer`.
Options: `normen_dir`, `indicatief`, `hardheidsklasse`, `kwaliteitscodes`,
`min_jaren`, `waterlichamen`, `normbron` (`"koppeltabel"`, the default, or `"bkl"`).
`bkl_normen` and `normen.bkl_afwijking` show where the koppeltabel differs from the Bkl. Also `waterkwaliteit.voorbereiden()`,
`waterkwaliteit.update_stofgroepen()` and the chart helpers in
`waterkwaliteit.grafieken`. See [Water quality analysis](#water-quality-analysis-waterkwaliteit).

### `set_logger(verbose)`

Controls logging verbosity: `"debug"`, `"info"`, `"warning"`, `"error"`,
`"critical"`, `"silent"` (case-insensitive) or a number. From `"warning"` up the
console download progress is silenced too.

---

## Known data issues (API side)

Reported to Rijkswaterstaat on the [WaterWebservices issue tracker](https://github.com/Rijkswaterstaat/WaterWebservices/issues);
these are in the data the API serves, not in this library:

- Duplicate timestamps in some series (e.g. Q at `maastricht.sintpieter`,
  `venlo`; WATHTE series; two `OpdrachtgevendeInstantie` for one series): #16, #22, #48, #49.
  `find_duplicates(df)` lists them; nothing is removed automatically.
- Tidal extreme types (`GETETM2`, ...) have `Waarde_Numeriek` 0.0; the value is
  in `Waarde_Alfanumeriek`: #24 (`tidal_extremes(df)` pairs them with the
  levels). Some Q series contain 0.0 values flagged as checked: #25.
- `WINDST` (wind gust) no longer exists; use `WINDSHD` with
  `waardebepalingsmethode_code` (other:F137 / F219 / F132): #44, #56 (the
  library logs this hint when a WINDST request returns nothing); gust codes
  missing since 2026-08-18: #59.
- The maximum number of observations per request (HTTP 400, "Het maximaal
  aantal waarnemingen ... is overschreden") was 263,088 per series on
  2026-10-20 (#14 mentions 160,000); the library splits such periods itself.
- The catalog's `Parameter_Wat_Omschrijving` can differ from the one in the
  observations: #26, #39, #40.
- `Groepering` and `BioTaxonType` are empty strings in the catalog, by design: #13.
- `OphalenWaarnemingen` honours `Hoedanigheid`, `Typering`, `Parameter`,
  `WaardeBewerkingsMethode`, `WaardeBepalingsMethode`, `BioTaxon` and
  `MeetApparaat` (verified 2026-10-20);
  `CheckWaarnemingenAanwezig` ignores `Hoedanigheid` and `BioTaxon`.

---

## Links

- WaterWebservices documentation (Rijkswaterstaat): <https://rijkswaterstaatdata.nl/waterdata/>
- API reference (Swagger UI): <https://ddapi20-waterwebservices.rijkswaterstaat.nl/swagger-ui/index.html>
- User manual (PDF): <https://rijkswaterstaatdata.nl/publish/pages/223004/distributielaagwebservices-sum-2v7-1-.pdf>
- Catalog: `rw.load_catalog()` or `rws-waterinfo catalogus --uitvoer catalogus.csv`
  (the Excel download of the catalog is currently not offered by Rijkswaterstaat)
- Location codes, old LMW/DONAR codes to the new ones: <https://wrd.rwskennisbron.nl/index.php/Id-048979a4-2037-4605-9e86-3a921f9733bb>
- Community: <https://github.com/Rijkswaterstaat/WaterWebservices/discussions>
- Github issuetracker ddapi20: <https://github.com/Rijkswaterstaat/WaterWebservices/issues>
