Metadata-Version: 2.4
Name: hestia_earth_distribution
Version: 0.6.1
Summary: Hestia's Distribution library
Home-page: https://gitlab.com/hestia-earth/hestia-distribution
Author: Hestia Team
Author-email: guillaume@hestia.earth
License: MIT
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: hestia-earth-schema>=35.0.1
Requires-Dist: hestia-earth-utils>=0.16.7
Provides-Extra: stats
Requires-Dist: cmdstanpy; extra == "stats"
Requires-Dist: arviz; extra == "stats"
Requires-Dist: scipy; extra == "stats"
Provides-Extra: plotting
Requires-Dist: arviz; extra == "plotting"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Hestia Distribution

Prior and posterior distributions of crop **yield** and **input use** (fertiliser, pesticide,
irrigation), per country and per product, for the Hestia platform. Use them to get an expected
range for a value, or to score how plausible a reported value is.

## What it does

For a `(country, product)` pair the library produces a `(mu, sigma)` normal distribution for
each quantity, built in two layers:

- **Prior** — the starting belief, derived from FAOSTAT. For yield it is the national yield
  time series; for inputs it is national input use. The mean is an area-weighted average over
  the years, and the spread is the year-to-year variation of that national figure, floored by
  a cross-country coefficient of variation so a country with a flat series is not given an
  implausibly tight interval. Priors are cheap and exist for every `(country, product)` FAO
  reports.

- **Posterior** — the prior updated with Hestia's own Cycle data for that pair, via Bayesian
  MCMC (a small Stan model sampled with cmdstanpy). Where enough real cycles exist the
  posterior is dominated by them and departs from the prior; where they don't, it stays close
  to the prior. A posterior is only published if the data can support it: at least
  `MIN_CYCLES` cycles (default 30) and a fit that converged (r̂ ≤ 1.01). Otherwise no
  posterior is written and **the prior stands** — so a consumer always has something to fall
  back to.

The read path (getting a distribution back out) is pure lookup: it reads pre-built files from
a local folder and needs no sampler. Only *generating* posteriors needs cmdstanpy and CmdStan.

## Install

```
pip install hestia_earth.distribution
```

Reading distributions needs only this. To **generate** posterior files, also install the
sampler:

```
pip install hestia_earth.distribution[stats]
./install-cmdstan.sh
```

[cmdstanpy](https://pypi.org/project/cmdstanpy/) drives
[CmdStan](https://mc-stan.org/users/interfaces/cmdstan), which is compiled on the machine
rather than installed from pip — hence the second step. It needs a C++ toolchain
(`build-essential` on Debian/Ubuntu, the Xcode command line tools on macOS) and takes a few
minutes the first time.

## Getting the files

The library reads pre-built distribution files from a local folder, set by
`DISTRIBUTION_DATA_FOLDER` (default `./data`). The files are published at
**https://api.hestia.earth/distribution/files** as a JSON list of download URLs — the four
FAO prior CSVs and one posterior CSV per country.

Run this once to populate the folder (no credentials needed):

```python
import json
import os
import urllib.request

DATA_FOLDER = os.getenv("DISTRIBUTION_DATA_FOLDER", "./data")

files = json.load(urllib.request.urlopen("https://api.hestia.earth/distribution/files"))
for group, subfolder in [("priorFiles", "prior_files"), ("posteriorFiles", "posterior_files")]:
    folder = os.path.join(DATA_FOLDER, subfolder)
    os.makedirs(folder, exist_ok=True)
    for f in files[group]:
        urllib.request.urlretrieve(f["url"], os.path.join(folder, f["name"]))
    print(f"downloaded {len(files[group])} {subfolder}")
```

That lays out `<DATA_FOLDER>/prior_files/` and `<DATA_FOLDER>/posterior_files/`, which is
exactly where the accessors below look. The download URLs are short-lived, so fetch the list
fresh each time rather than saving a URL to reuse. Each file is then read once per process and
cached, so checking many values costs one read per file, not one per lookup.

## Getting distribution data

Each quantity has its own module, and every one exposes the same two accessors:

- `get_prior(...)` — the FAO-derived prior, always available where FAO has data.
- `get_post(...)` — the posterior, or `(None, None)` when none was published for that pair.
  When it is `(None, None)`, fall back to `get_prior(...)`.

Both return a `(mu, sigma)` tuple.

```python
from hestia_earth.distribution.prior_yield import get_prior as get_prior_yield
from hestia_earth.distribution.posterior_yield import get_post as get_post_yield

get_prior_yield("GADM-GBR", "wheatGrain")   # -> (8061.2, 3736.7)   FAO prior
get_post_yield("GADM-GBR", "wheatGrain")    # -> (7900.4, 1970.5)   updated with Hestia cycles
get_post_yield("GADM-GBR", "oatGrain")      # -> (None, None)       no posterior; use the prior
```

The other quantities follow the same shape, differing only in what identifies the row:

```python
# Fertiliser: keyed by product AND input; the prior is per input, independent of product.
from hestia_earth.distribution.prior_fert import get_prior as get_prior_fert
from hestia_earth.distribution.posterior_fert import get_post as get_post_fert
get_prior_fert("GADM-GBR", "inorganicNitrogenFertiliserUnspecifiedKgN")               # (mu, sigma)
get_post_fert("GADM-GBR", "wheatGrain", "inorganicNitrogenFertiliserUnspecifiedKgN")  # (mu, sigma)

# Pesticide and irrigation: the prior is per country only.
from hestia_earth.distribution.prior_pest import get_prior as get_prior_pest
from hestia_earth.distribution.posterior_pest import get_post as get_post_pest
get_prior_pest("GADM-GBR")                  # (mu, sigma)
get_post_pest("GADM-GBR", "wheatGrain")     # (mu, sigma) or (None, None)

from hestia_earth.distribution.prior_irrigation import get_prior as get_prior_irri
from hestia_earth.distribution.posterior_irrigation import get_post as get_post_irri
get_prior_irri("GADM-GBR")                  # (mu, sigma)
get_post_irri("GADM-GBR", "wheatGrain")     # (mu, sigma) or (None, None)
```

`get_post` returns the mean of the posterior ensemble. If you need the full ensemble of draws
rather than its mean, call `get_post_ensemble(country_id, product_id)` from the same module,
which returns `(mu_ensemble, sd_ensemble)` as lists.

## Validating a value

A `(mu, sigma)` distribution gives a confidence interval directly: with the usual 95%
interval, a value is an outlier when it falls outside `mu ± 1.96 · sigma`. Prefer the
posterior and fall back to the prior:

```python
from hestia_earth.distribution.posterior_yield import get_post as get_post_yield
from hestia_earth.distribution.prior_yield import get_prior as get_prior_yield

def yield_interval(country_id, product_id, z=1.96):
    mu, sigma = get_post_yield(country_id, product_id)
    if mu is None:                                   # no posterior -> use the prior
        mu, sigma = get_prior_yield(country_id, product_id)
    if mu is None:                                   # no distribution at all
        return None
    return (max(mu - z * sigma, 0), mu + z * sigma)  # yield cannot be negative

yield_interval("GADM-GBR", "wheatGrain")   # -> (4038.2, 11762.7)
```

### Joint (multivariate) plausibility

The interval above checks each quantity on its own. To score a *combination* — e.g. "is 8500
kg/ha of wheat alongside 200 kg N/ha jointly plausible for the UK?" — use the multivariate
fit, which accounts for the correlation between yield and input use. This path needs `scipy`
(included in the `[stats]` extra) and reads the underlying cycle data.

```python
from hestia_earth.distribution.utils.MCMC_mv import calculate_fit_2d

# candidate follows the column order [Nitrogen (kg N), Grain yield (kg/ha)]
likelihood, _ranges = calculate_fit_2d([200, 8500], "GADM-GBR", "wheatGrain")
# likelihood ~ how plausible the pair is (Monte Carlo integration over the joint density);
# roughly, the fraction of observed samples it stands above. Above ~5% is acceptable.
```

## Generating distributions

Building prior and posterior files (rather than reading them) is done through each module's
`generate_*` and `update_all_post` functions — for example
`generate_prior_yield_file(overwrite=True)` and
`update_all_post(country_id, product_ids=..., overwrite=False)` in `posterior_yield`.
Generating posteriors samples the Stan model and therefore needs the `[stats]` extra and
CmdStan installed (see **Install**). Passing `product_ids` restricts the run to the products
that actually have cycles in a country, which is far cheaper than walking every product.

## Configuration reference

| Variable | Purpose | Default |
|---|---|---|
| `DISTRIBUTION_DATA_FOLDER` | folder the library reads distribution files from | `./data` |
| `DISTRIBUTION_MIN_CYCLES` | minimum cycles before a posterior is published | `30` |
| `DISTRIBUTION_POST_CACHE_SIZE` | how many countries' posterior files to hold in memory | `8` |
