Metadata-Version: 2.4
Name: metrics-adjuster
Version: 1.0.1
Summary: Adjusted group-aware clinical prediction metrics with Python API and CLI.
Project-URL: Homepage, https://github.com/SaehwanPark/metrics-adjuster
Project-URL: Repository, https://github.com/SaehwanPark/metrics-adjuster
Project-URL: Issues, https://github.com/SaehwanPark/metrics-adjuster/issues
Author: Saehwan Park
License-Expression: GPL-3.0-only
License-File: LICENSE
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: matplotlib>=3.10.9
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.1
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0.3
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pandas-stubs>=2.1; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0.12.20260518; extra == 'dev'
Description-Content-Type: text/markdown

# metrics-adjuster

`metrics-adjuster` computes conventional and adjusted group-aware metrics for binary prediction models. It provides a typed Python API, a command-line interface, and an end-to-end synthetic demo for users who want to validate the workflow before running it on their own data.

The codebase is organized around a small functional core: IO lives at the boundaries, public options are captured in typed configuration objects, and tests describe the expected contracts for validation and output shape.

## What it computes

For each requested adjusted metric, the package now returns the conventional metric and adjusted metric side by side:

| Output file/key | Conventional column | Adjusted column |
| --- | --- | --- |
| `aTPR` | `TPR` | `aTPR` |
| `aPPV` | `PPV` | `aPPV` |
| `aNB` | `NB` | `aNB` |
| `aHR` | `HR` | `aHR` |

Metrics are computed by group across caller-selected risk-score quantiles. The main pipeline is:

1. validate typed configuration and input columns
2. calibrate predicted risk within groups
3. estimate density ratios against a reference group
4. compute conventional and adjusted metrics at risk thresholds
5. optionally bootstrap adjusted-metric uncertainty summaries
6. optionally render a self-contained HTML report with tables and plots
7. return metric tables through the API or write one CSV per metric through the CLI

## Installation

Released versions are on [PyPI](https://pypi.org/project/metrics-adjuster/)
(Python 3.11+).

```bash
python -m pip install metrics-adjuster
```

That installs both the CLI and the Python API. Parquet input works out of the
box (`pyarrow` is included).

**Other setups:**

- uv project: `uv add metrics-adjuster`
- CLI only: `pipx install metrics-adjuster`
- conda or mamba env: run the pip command above inside your activated environment
- pinned release: `python -m pip install "metrics-adjuster==1.0.1"`

### From source

For checkout-based development, see [Development](#development).

To install the latest main branch from GitHub:

```bash
python -m pip install "metrics-adjuster @ git+https://github.com/SaehwanPark/metrics-adjuster.git"
```

In a uv project:

```bash
uv add "metrics-adjuster @ git+https://github.com/SaehwanPark/metrics-adjuster.git"
```

## Python API

```python
from metrics_adjuster import (
  CalibrationConfig,
  ColumnSpec,
  DensityRatioConfig,
  MetricConfig,
  MetricName,
  adjusted_metrics,
)
from metrics_adjuster.synthetic import generate_synthetic_metrics_data

frame = generate_synthetic_metrics_data(n=600, seed=2026)
config = MetricConfig(
  columns=ColumnSpec(
    group="group",
    response="outcome",
    risk="risk",
    id="patient_id",
  ),
  ref_group="ref",
  quantiles=(0.2, 0.4, 0.6, 0.8),
  metrics=tuple(MetricName),
  calibration=CalibrationConfig(degree=2, cv=False),
  density_ratio=DensityRatioConfig(degree=1, cv=False),
  random_state=2026,
)

result = adjusted_metrics(frame, config)
print(result.metrics["aTPR"])
```

For an end-user report, call `adjusted_metrics_report(...)`:

```python
from metrics_adjuster import ReportConfig, ReportLabelConfig, adjusted_metrics_report

bundle = adjusted_metrics_report(
  frame,
  config,
  ReportConfig(
    title="Adjusted metrics report",
    x_scale="log_odds",
    labels=ReportLabelConfig(
      columns={"group": "Cohort group"},
      groups={"group": {"ref": "Reference group"}},
      metrics={"aTPR": "True positive rate"},
    ),
  ),
)
html = bundle.html
table = bundle.metric_table
```

`result.metrics["aTPR"]` contains columns like:

```text
group,quantile,tau,TPR,aTPR
```

The return value is a `MetricFrames` object containing:

- `metrics`: a dictionary from adjusted metric name to `pandas.DataFrame`
- `bootstrap`: optional raw long-form bootstrap records when bootstrapping is enabled
- `as_dict()`: a compatibility-shaped dictionary for older integrations

## Compatibility API

Existing callers can keep using the previous function-style entry point:

```python
from metrics_adjuster import compute_adjusted_metrics
from metrics_adjuster.synthetic import generate_synthetic_metrics_data

frame = generate_synthetic_metrics_data(n=600, seed=2026)
result = compute_adjusted_metrics(
  df=frame,
  idvar="patient_id",
  group_col="group",
  ref_group="ref",
  response_col="outcome",
  orig_risk_col="risk",
  metrics=["aTPR"],
  quantiles=[0.2, 0.4, 0.6, 0.8],
  random_state=2026,
)
```

Historical import shims from the private development repository are preserved
under `legacy/` for migration research only and are not part of the supported
v1 package.

## CLI

Run the built-in synthetic demo:

```bash
uv run metrics-adjuster demo --output-dir demo_outputs
```

By default, the CLI computes `aTPR`. Pass `--metrics aTPR,aPPV,aNB,aHR`
when you want every supported metric.

Run on your own CSV or Parquet file:

```bash
uv run metrics-adjuster run \
  --input path/to/input.csv \
  --output-dir adjusted_metric_outputs \
  --group-col group \
  --ref-group ref \
  --response-col outcome \
  --risk-col risk \
  --id-col patient_id \
  --quantiles 0.2,0.4,0.6,0.8 \
  --metrics aTPR,aPPV,aNB,aHR \
  --seed 2026
```

The CLI writes one CSV per adjusted metric, such as `aTPR.csv` and `aPPV.csv`. Each file includes the conventional companion column as well as the adjusted metric column.

Add `--report` to write a self-contained `report.html` with compact metric tables,
calibrated probability density plots, and densities normalized to the reference
group:

```bash
uv run metrics-adjuster run \
  --input path/to/input.csv \
  --output-dir adjusted_metric_outputs \
  --group-col group \
  --ref-group ref \
  --response-col outcome \
  --risk-col risk \
  --report \
  --report-title "Adjusted metrics report" \
  --report-config-yaml path/to/report.yml
```

The optional report YAML can provide human-friendly display names and choose
plot x-axis scaling:

```yaml
x_scale: log_odds
labels:
  columns:
    Prior1245: Veteran Priority Group
  groups:
    Prior1245:
      "0": Default Priority
      "99": Priority Group 5
  metrics:
    aTPR: True positive rate
```

Without uv, use the same commands after installing the package, but drop the `uv run` prefix.

Numeric reference groups are supported through the public CLI. For example,
`--ref-group 0` will match an integer-valued group column such as `Prior1245`.

## Required input columns

Your input table must include:

- a group column, such as `group`, `sex`, or `race`
- a binary outcome column encoded as `0` and `1`
- a predicted risk/probability column
- optionally, a stable identifier column

The reference group named by `ref_group` must be present in the group column.

## End-to-end integration demo

The demo script is intentionally outside `src/` so it remains a runnable example rather than package code:

```bash
uv run python scripts/run_synthetic_integration.py \
  --output-dir demo_outputs \
  --n 600 \
  --seed 2026
```

It writes:

- `synthetic_metrics_data.csv`
- `aTPR.csv` with `TPR` and `aTPR`
- additional metric CSVs only when `--metrics` requests them

## Sampled real-data evaluation

For a bounded real-data report review, first prepare an ignored sampled input:

```bash
uv run python scripts/prepare_sampled_report_input.py \
  --input /path/to/input.parquet \
  --output-dir data/generated/va-can-2019-prior1245-hosp1y-sample50k \
  --group-col Prior1245 \
  --ref-group 0 \
  --response-col Hosp_1y \
  --risk-col pHosp_1y \
  --sample-size 50000 \
  --bootstrap-iterations 25 \
  --seed 20260521 \
  --report-config-yaml scripts/va_can_report_config.yml
```

Then reproduce the report through the public CLI:

```bash
uv run metrics-adjuster run \
  --input data/generated/va-can-2019-prior1245-hosp1y-sample50k/sample.parquet \
  --output-dir results/va-can-2019-prior1245-hosp1y-sample50k \
  --group-col Prior1245 \
  --ref-group 0 \
  --response-col Hosp_1y \
  --risk-col pHosp_1y \
  --quantiles 0.1,0.3,0.5,0.7,0.9 \
  --metrics aTPR,aPPV,aNB,aHR \
  --bootstrap \
  --n-boot 25 \
  --seed 20260521 \
  --report \
  --report-config-yaml scripts/va_can_report_config.yml
```

The preparation script reads only the required columns, samples rows
deterministically, writes row-level data under ignored `data/generated/`, and
records copy-pasteable commands in its README. Metric CSVs, `bootstrap.csv`, and
`report.html` are produced only by `metrics-adjuster run`. By default, the
generated CLI report command mirrors the generated-data path under `results/`;
pass `--report-output-dir` to the preparation script to record a different
report destination.

## Full user manuals

- [API Manual](docs/API_MANUAL.md)
- [CLI Manual](docs/CLI_MANUAL.md)

## Development

Preferred setup:

```bash
uv sync --extra dev
```

Common checks:

```bash
uv run pytest
uv run ruff check .
uv run mypy src/metrics_adjuster
```
