Metadata-Version: 2.4
Name: asta-evidence
Version: 0.1.1
Summary: Config-driven analysis of agricultural evidence datasets
Author-email: Ritvik Sahajpal <ritvik@umd.edu>
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: pandas
Requires-Dist: numpy
Requires-Dist: matplotlib
Requires-Dist: seaborn
Requires-Dist: scienceplots
Requires-Dist: arrow
Requires-Dist: logzero
Requires-Dist: requests
Provides-Extra: maps
Requires-Dist: pygmt; extra == "maps"
Provides-Extra: papers
Requires-Dist: pyreadr; extra == "papers"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: ruff; extra == "dev"

# ASTA

Config-driven analysis of agricultural evidence datasets. The first dataset is
**ERA** (Evidence for Resilient Agriculture); the architecture assumes more will
follow.

## What it does

ERA pairs a management practice against a control and records an outcome — one
row per observation, 112,859 of them drawn from 2,011 studies across 52 African
countries. ASTA answers *which country × crop × practice × outcome combinations
the evidence base actually covers*, and how large the measured effects are.

It produces nine charts, four maps, and a CSV behind each one:

| | Figure |
|---|---|
| F1 | Most-studied country × product × practice × outcome combinations |
| F2 | Most-studied product × practice × outcome combinations |
| F3 | Crop × practice coverage heatmap |
| F4 | Practice × outcome coverage heatmap |
| F5 | Country × practice-theme heatmap |
| F6 | Practice co-occurrence — which practices get tested together |
| F7 | Evidence composition by product and outcome pillar |
| F8 | Effect of each practice on crop yield, with bootstrap CIs |
| F9 | Practice × outcome median effect heatmap |
| F10 | Outcome-pillar co-measurement — does one study measure productivity *and* mitigation? |
| F11 | UpSet of the exact outcome sets studies measure together |
| F12 | Outcome × outcome co-occurrence heatmap |
| M1 | Evidence volume by country (PyGMT choropleth) |
| M2 | Trial sites — 1,599 distinct locations |
| M3 | Small multiples by practice theme |
| M4 | Breadth of the evidence base by country |

## Install

The map stack is binary — PyGMT needs the GMT C library plus its DCW country
polygons and GSHHG shorelines, which pip cannot provide. Use pixi:

```bash
pixi install
pixi run python -c "import pygmt; pygmt.clib.Session()"   # proves GMT initialised
```

Charts alone work from a plain pip install (`pip install -e .`); only the maps
need conda-forge.

## Data

The ERA files are not in this repo — the compilation is 85 MB. Obtain them from
[CIAT/ERA_dev](https://github.com/CIAT/ERA_dev) and place them here:

```
inputs/ERA/
├── 01a. ERA_Compiled_full.csv
├── Practice_Codes.csv
└── Outcome_Codes.csv
```

Then point `dir_base` in `config/asta_base.txt` at your checkout.

> The source files are Windows-1252 encoded, not UTF-8. `config/era.txt` already
> declares this; a plain UTF-8 read fails on the accented characters in the
> practice definitions.

## Run

```bash
pixi run analysis
```

or

```python
from asta import analysis
analysis.run(["config/asta_base.txt", "config/era.txt"])
```

Output lands in a date-stamped tree:

```
outputs/era_combinations/<Month_DD_YYYY>/
├── plots/    F1–F9 as PNG + PDF
├── maps/     M1–M4 as PNG + PDF
└── tables/   the aggregation behind every figure
```

## Downloading the papers

```bash
pixi run download-papers
```

Attempts to fetch the **open-access** PDF behind each of the 2,011 studies —
Unpaywall first, OpenAlex second, the bibliography's own URL last, every
payload validated by PDF magic bytes. Nothing bypasses paywalls: publishers
that refuse scripted access (MDPI, Wiley, Elsevier return 403 regardless of
the request) are recorded as failures, not worked around.

The bibliography itself (`ERA_Bibliography.csv`) is a cache, built on first
run from the ERAg package's `.rda` — the Dataverse copy of the same table is
behind a guestbook the API cannot pass.

PDFs land in `inputs/ERA/papers/` under a standard name:

```
{DATASET}_{STUDYCODE}__{FirstAuthorSurname}_{Year}.pdf
e.g.  ERA_NJ0001__Abass_2018.pdf
```

Beside them, `manifest.csv` holds one row per study — downloaded or not —
with a truthful status (`downloaded`, `already_present`, `no_bibliography`,
`no_doi`, `no_oa_location`, `download_failed`, `not_a_pdf`, `not_attempted`)
plus the DOI, resolver, URL and failure detail. `missing.csv` is the subset
that did not download, with the reason. Reruns are idempotent: successes are
skipped, failures retried (`retry_failed`), and a paper fetched manually and
dropped in under its standard name is picked up as `already_present`.

Knobs live in `[PAPERS]` in `config/era.txt`; set `max_papers = 15` for a
smoke test.

## Configuration

`config/asta_base.txt` holds paths, logging, and the list of active datasets.
`config/era.txt` holds everything tunable about the ERA analysis. The knobs that
change conclusions:

| Option | Effect |
|---|---|
| `count_unit` | `studies` counts distinct publications; `rows` counts observations and lets a few large multi-treatment trials dominate |
| `top_n_combinations` | how many bars the ranked charts show |
| `min_studies_effect` | evidence floor before an effect size is reported at all |
| `include_animal_products` | `False` restricts every figure to `Product.Type == Plant` |

## Adding a dataset

Three edits, none of which touch existing code:

1. Add `asta/datasets/<NAME>.py` exposing `run(base, section="<NAME>")`.
2. Add a `[<NAME>]` section to a config file.
3. Add a branch to the dispatch loop in `asta/analysis.py` and list the dataset
   in `[DATASETS] datasets`.

`BaseASTA` (`asta/base.py`) already gives you config parsing, resolved paths, a
date-stamped output tree, a logger, and `save_table()`.

## Design notes

Colour is not ad hoc. `asta/style.py` holds a palette validated for
colour-vision deficiency all-pairs in both light and dark: three categorical
slots (the three ERA outcome pillars), a single-hue sequential ramp for counts,
and a blue–grey–red diverging ramp pinned at zero for signed effects. Counts
span three orders of magnitude, so count heatmaps use a log scale and label it.

Effect sizes are aggregated on `yi`, the log response ratio, and back-transformed
for display. The `pc` column is floored at −100 % with a 99th percentile of
+1991 %, so its median would misstate the centre.

The UpSet plot (F11) is hand-rolled on matplotlib rather than using the
`upsetplot` package, for two reasons: the package crashes under pandas 3 /
matplotlib 3.11 (deprecated inplace chained assignment, then a hard failure in
`ax.scatter`), and it builds and styles its own figure, which fights `style.py`.

F11 targets **outcomes**, not practices, and that choice is measured rather than
assumed. UpSet exists to reveal higher-order intersections; practice bundles do
not have them — 90.3 % of bundles are a single practice or a pair, and the
top 25 bundles contain no three-practice bundle at all (the first ranks 33rd), so
an UpSet there would duplicate F6 and F1. Outcome sets carry 39.5 % of studies at
three or more members, four times the practice-bundle share, and no other figure
crosses outcomes against each other.
