Metadata-Version: 2.4
Name: calm-benchmark
Version: 0.1.3
Summary: An open, calibrated benchmark for menopausal-status prediction from routine health markers
License-File: LICENSE
Requires-Python: >=3.14
Requires-Dist: beautifulsoup4>=4.15.0
Requires-Dist: pandas>=3.0.3
Requires-Dist: polars>=1.42.1
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pyreadstat>=1.3.5
Requires-Dist: scikit-learn>=1.9.0
Requires-Dist: structlog>=26.1.0
Requires-Dist: typer>=0.27.0
Description-Content-Type: text/markdown

# CALM

**C**alibrated **M**enopausal-status benchmark — an open, reproducible benchmark for predicting
menopausal status from routine, low-cost health markers.

> 0.1.3 · Python 3.14 · uv · MIT

CALM provides a rigorously specified benchmark for:
**premenopausal vs. postmenopausal** from routine markers (labs, body measures, demographics), with
**calibrated confidence**, an explicit **no-call** option, honest generalization reporting, and
end-to-end **provenance**. It is a benchmark engine — a library and CLI — **not an application**. It emits
versioned, auditable artifacts that downstream tools consume.

Out-of-training-range inputs are always withheld. This is documented univariate range checking rather
than a claim of multivariate novelty detection.

Reported metrics include deterministic bootstrap confidence intervals. Synthetic runs are explicitly
exploratory; the default real-data protocol is within-cycle frozen-test evaluation, not external validation.

---


## Install

Requires Python 3.14 and [uv](https://docs.astral.sh/uv/).

After the first PyPI release:

```bash
uv tool install calm-benchmark
calm-benchmark --help
```

For development from source:

```bash
git clone <repo> && cd calm
uv sync
uv run calm-benchmark --help
```

---

## Quickstart

```bash
# run the deterministic, offline benchmark demonstration
uv run calm-benchmark demo --output runs
```

The demo writes, under `runs/synthetic-demo/`, `metrics.json`, a reliability plot, a benchmark card, and
a complete `manifest.json`.

## Pinned public NHANES run

The public 2017–2018 run consumes the official XPT files already downloaded to `data/raw/`, verifies
their SHA-256 hashes, validates the cycle schema, and records source URLs and checksums in its manifest.

```bash
uv run calm-benchmark run-nhanes --config configs/default.toml --run-id nhanes-2017-2018
```

It uses the versioned `public_cessation` outcome: `RHQ031` and `RHD043` distinguish reported current
periods from menopause/change-of-life cessation and exclude reported hysterectomy. Public data redact
pregnancy and hysterectomy information for women aged 45 and older, so this run does not claim to
exclude surgical menopause in that age band. An RDC-only natural-menopause variant is outside this
public benchmark.

## Attribution and optional explanations

Each benchmark run emits `attributions.json`: deterministic, per-prediction feature-association scores
with model-hash provenance. They describe association under a fixed reference replacement, not cause.
The optional OpenAI explanation service lives in `services/explainer/`; it receives only a versioned CALM
hand-off and refuses withheld/OOD results, missing attributions, diagnosis, and treatment requests.

---

## The benchmark contract

- **Task** — binary premenopausal vs. postmenopausal (optional secondary: early natural menopause).
- **Split** — deterministic, stratified, content-hashed; the hash is the no-leakage proof.
- **Metrics** — balanced accuracy, per-class recall, F1, AUROC, PR-AUC, Brier and reliability, no-call
  rate, bootstrap confidence intervals, and subgroup breakdowns with denominators and eligibility.

See `DESIGN.md` for the full specification and domain rationale.

---

## Reproducibility & provenance

Every artifact carries a `Provenance` record (normalized-config hash, input hashes, tool and git
versions). Re-running the same config against the same pinned inputs yields identical hashes. NHANES
public-use data are U.S. public domain; the NCHS Data User Agreement notice is preserved in exports.

---

## Architecture at a glance

```
src/calm/
├── config/         # pydantic schemas + loading
├── domain/         # entities, value objects, invariants, registries (pure — no I/O)
├── ports/          # Protocol interfaces
├── adapters/       # NHANES source, synthetic source, codebook parsing
├── pipeline/       # acquire ingest cohort labels features split train calibrate evaluate report
├── orchestration/  # instrumented runner, run manifest
├── observability/  # logging, timing, hashing, provenance
└── cli.py          # Typer app
```

Full details in `DESIGN.md`. The AI-agent build process is governed by `AGENT.md`; the roadmap lives in
`TODOS.md`.

---

## Extending CALM

- **New cycle** — add a `CycleSpec` in config.
- **New label rule / outcome** — implement and register a `LabelRule`.
- **New data source** — implement the `DataSource` port.
- **New model or metric** — register a factory.

---

## Development

```bash
uv run ruff check && uv run ruff format --check
uv run mypy
uv run pytest --cov=calm
uv run pre-commit run -a
uv run mkdocs serve
```

Standards: full type hints, Google-style docstrings on the public API, no superfluous comments, unit
tests required, property-based tests for pipeline invariants, and a coverage gate on the core.

---

## Contributing

See `AGENT.md` for engineering conventions and `TODOS.md` for the roadmap. Pull requests must pass all
gates and include tests and a `CHANGELOG.md` entry.

---

## Versioning & changelog

Semantic Versioning; history in `CHANGELOG.md` (Keep a Changelog).

---

## License & disclaimer

MIT. **Research benchmark, not a medical device.** CALM produces population-derived predictions for
methodological research and must not be used for clinical or diagnostic decisions.

---

## Citation

```
CALM: A Calibrated Benchmark for Menopausal-status Prediction from Routine Health Markers. <year>. <repo>.
```
