Metadata-Version: 2.4
Name: vernier
Version: 0.5.0
Classifier: Development Status :: 1 - Planning
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=15
Requires-Dist: rfdetr==1.6.5.post0 ; extra == 'real-models'
Requires-Dist: platformdirs>=4 ; extra == 'real-models'
Requires-Dist: torch>=2.4 ; extra == 'real-models'
Requires-Dist: transformers>=5.1 ; extra == 'real-models'
Requires-Dist: huggingface-hub>=0.27 ; extra == 'real-models'
Requires-Dist: pillow>=10 ; extra == 'real-models'
Requires-Dist: timm>=1.0 ; extra == 'real-models'
Requires-Dist: torchmetrics>=1.5 ; extra == 'real-models'
Requires-Dist: polars>=1.0 ; extra == 'tables'
Requires-Dist: torch>=2.4 ; extra == 'torch'
Requires-Dist: plotly>=6.0 ; extra == 'viz'
Provides-Extra: real-models
Provides-Extra: tables
Provides-Extra: torch
Provides-Extra: viz
License-File: LICENSE-APACHE
License-File: LICENSE-MIT
Summary: High-performance, parity-preserving COCO-style evaluation
Keywords: evaluation,metrics,computer-vision,detection,coco,object-detection
Author: The vernier authors
License: MIT OR Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://github.com/NoeFontana/vernier#readme
Project-URL: Homepage, https://github.com/NoeFontana/vernier
Project-URL: Issues, https://github.com/NoeFontana/vernier/issues
Project-URL: Repository, https://github.com/NoeFontana/vernier

# vernier

[![PyPI](https://img.shields.io/pypi/v/vernier.svg)](https://pypi.org/project/vernier/)
[![Python](https://img.shields.io/pypi/pyversions/vernier.svg)](https://pypi.org/project/vernier/)
[![Crates.io](https://img.shields.io/crates/v/vernier.svg)](https://crates.io/crates/vernier)
[![License](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue.svg)](#license)

**Fast, auditable evaluation for 2D vision models.** Detection, instance and
panoptic segmentation, semantic segmentation, keypoints, and LVIS, all in one
package with a Rust core, a Python API, and a standalone CLI.

- **Bit-exact** with `pycocotools==2.0.11`, `panopticapi`, `lvis-api` and
  `boundary-iou-api` in strict mode. Every upstream quirk has a documented
  disposition ([quirks survey](docs/engineering/pycocotools-quirks.md)).
- **Drop-in** for `pycocotools.cocoeval.COCOeval`: change one import, or none.
  Same constructor, same `params` mutations, same `eval` / `evalImgs` /
  `ious` / `stats` read back off the instance ([what carries
  over](docs/migrate/from-pycocotools.md#params-what-the-shim-honors-and-what-it-rejects)).
- **3.7–19× faster** than faster-coco-eval and pycocotools at equal CPU budget
  ([benchmarks](#performance)).
- **Built for real pipelines**: training-loop evaluation, multi-rank
  gathering, per-image tables, error decomposition, calibration, scenario
  slicing.

## Install

```sh
pip install vernier               # Python ≥ 3.10, abi3 wheels
cargo binstall vernier-cli        # standalone `vernier` binary, no Python needed
cargo add vernier                 # Rust library
```

## Quickstart: the COCO API you already know

`vernier.COCOeval` has the same constructor, the same
`evaluate() / accumulate() / summarize()` sequence, and the same `.stats`
as pycocotools. It defaults to `parity_mode="strict"`, so the output is
bit-identical (see [Status & validation](#status--validation) for what
that is measured against).

```python
from pycocotools.coco import COCO
from vernier import COCOeval  # was: from pycocotools.cocoeval import COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.loadRes("detections.json")

E = COCOeval(coco_gt, coco_dt, iouType="bbox")  # "segm" | "keypoints" | "boundary"
E.evaluate()
E.accumulate()
E.summarize()
```

**Code you can't edit** (mmdetection, detectron2, ultralytics, …): patch
the symbol before anything imports `pycocotools.cocoeval`.

```python
import vernier

unpatch = vernier.patch_pycocotools()  # pycocotools.cocoeval.COCOeval -> vernier
run_existing_eval()
unpatch()
```

The patch is explicit, reversible, and never happens on import. A context
manager (`vernier.adapters.patched_pycocotools`) and a one-fixture pytest
recipe are in the [pycocotools migration guide](docs/migrate/from-pycocotools.md#pytest-integration).

TorchMetrics' `MeanAveragePrecision` runs under the patch unchanged,
including `class_metrics=True` and `extended_summary=True`
(`tests/python/test_compat_torchmetrics.py` asserts the patched and
unpatched results are equal). Two `params` fields are the exception and
raise rather than diverge quietly: `imgIds` subsetting, and `areaRng`
(custom area ranges are a native-`Evaluator` feature, ADR-0040). The
[migration guide](docs/migrate/from-pycocotools.md#params-what-the-shim-honors-and-what-it-rejects)
has the field-by-field table.

## Recommended: the native API

The shim exists for migration. New code should use the native `Evaluator`:
immutable configuration, typed results, no pycocotools dependency, and
access to everything below.

```python
from pathlib import Path
from vernier.instance import Bbox, CocoDataset, Evaluator

gt = CocoDataset.from_json(Path("instances_val2017.json").read_bytes())
dt = Path("detections.json").read_bytes()

evaluator = Evaluator(iou=Bbox(), parity_mode="strict")
summary = evaluator.evaluate(gt, dt, num_threads=8)

print("\n".join(summary.pretty_lines()))  # the familiar 12-line table
ap = summary.stats[0]
```

> The native `Evaluator` defaults to `parity_mode="corrected"`, which applies
> the [itemized fixes](docs/engineering/pycocotools-quirks.md) to upstream
> bugs. Pass `"strict"` when your numbers must match published pycocotools
> results.

**Inside a training loop**, evaluate on a background worker and feed it
tensors directly (torch, JAX, CuPy, NumPy via DLPack, zero-copy):

```python
with evaluator.background(gt) as bg:
    for images, targets in val_loader:
        preds = model(images)
        bg.submit([{"image_id": int(t["image_id"]), **p} for t, p in zip(targets, preds)])
    summary = bg.finalize()
```

**Per-image and per-class diagnostics** as Polars DataFrames
(`pip install "vernier[tables]"`):

```python
result = evaluator.evaluate(gt, dt, tables="all")
result.per_class
```

### Paradigms

Pick the submodule that matches your model's output. They have different
data models and matching rules, so they are separate evaluators rather than
one class with a mode switch ([why](docs/explanation/three-paradigms.md)).

| Submodule | Input | Metrics |
| --- | --- | --- |
| `vernier.instance` | Scored detections: boxes, masks, keypoints | AP / AR (bbox, segm, boundary, OKS), LVIS federated AP |
| `vernier.panoptic` | Panoptic PNGs + `segments_info` | PQ / SQ / RQ, boundary PQ |
| `vernier.semantic` | Class-id label maps | mIoU, FWIoU, pixel accuracy, mean accuracy |

### Beyond the COCO API

| Need | Feature | Guide |
| --- | --- | --- |
| Evaluate without blocking training | `Evaluator.background(...)` | [how-to](docs/how-to/background-evaluator.md) |
| Evaluate across DDP ranks | `evaluate_to_partial` / `from_partials` | [how-to](docs/how-to/distributed-eval.md) |
| Find which images/classes regressed | `tables="all"` | [how-to](docs/how-to/result-tables.md) |
| Explain an AP gap | TIDE error decomposition, oLRP | [tutorial](docs/tutorials/debugging-with-tide.md) |
| Check score calibration | ECE / MCE / reliability (`calibration=True`) | [how-to](docs/how-to/calibration.md) |
| Metrics per weather, time of day, … | Manifest slicing, `vernier aggregate` (mPC / rPC) | [how-to](docs/how-to/scenario-slicing.md) |
| Non-standard IoU / recall / area grids | `iou_thresholds=`, `recall_thresholds=`, `area_ranges=` | [how-to](docs/how-to/custom-evaluation-grids.md) |

## CLI

A static binary for CI gates and robotics replay pipelines. Output is
byte-deterministic (sorted keys, no timestamps), so artifacts diff cleanly.

```sh
vernier eval --gt gt.json --dt dt.json --iou-type bbox                  # pycocotools-identical stdout
vernier eval --gt gt.json --dt dt.json --iou-type segm --emit json=result.json --threads 8
```

Exit codes: `0` success, `1` evaluation error, `2` invalid arguments.
Full reference: [`crates/vernier-cli`](crates/vernier-cli/README.md).

## Status & validation

Every row is checked by a parity harness that runs the reference
implementation and vernier on the same inputs. "Bit-exact" means
`parity_mode="strict"`, against the reference **as published on PyPI**
— vernier's own output is identical on x86-64 and ARM. (A pycocotools
you recompile from source can round bbox IoU 1-2 ULP differently on
ARM; see [ADR-0056](docs/adr/0056-pin-no-fp-contraction-for-bbox-iou.md)
and the [migration guide](docs/migrate/from-pycocotools.md#bit-for-bit-and-against-which-build).)

| Metric | Reference | Parity | Notes |
| --- | --- | --- | --- |
| bbox / segm / keypoints AP | `pycocotools==2.0.11` | bit-exact | |
| Boundary AP | `boundary-iou-api` | bit-exact | |
| LVIS federated AP | `lvis-api` 0.5.3 | bit-exact | full v1 val, bbox |
| Panoptic PQ, boundary PQ | `panopticapi` (single-core path) | bit-exact | Cityscapes panoptic deferred |
| Semantic mIoU / FWIoU / pAcc / mAcc | `mmseg.IoUMetric` v1.2.2 (vendored) | bit-exact on class marginals | [ADR-0036](docs/adr/0036-vendor-mmsegmentation-ioumetric.md) proposed; ADE20K-scale check pending |
| oLRP | clean-room NumPy oracle | ≤ 1e-9 | panoptic not supported |
| Calibration (ECE / MCE) | clean-room NumPy oracle | bit-exact | detection family only |
| TIDE thresholds (segm, boundary) | none | corrected only | [ADR-0022](docs/adr/0022-tide-thresholds.md) proposed |

Parity model: [ADR-0002](docs/adr/0002-three-tier-parity-model.md).
Library-by-library comparison: [`docs/comparison.md`](docs/comparison.md).

## Performance

Median wall time at a one-CPU budget (CPU/wall is 1.00 in every cell).
COCO val2017 throughout, except the LVIS row, which is LVIS v1 val.
Speedup is the other library's time divided by vernier's.

<!-- Hand-mirrored from docs/benchmarks.md (generated by tools/render_benchmarks.py).
     After a bench round, re-run the renderer, then update these numbers. -->

| Workload | vernier | vs pycocotools | vs faster-coco-eval | vs hotcoco |
| --- | ---: | ---: | ---: | ---: |
| bbox AP | 305 ms | 18.7× | 5.6× | 1.9× |
| segm AP | 931 ms | 7.2× | 3.7× | 1.5× |
| keypoints AP | 137 ms | 16.8× | 5.7× | 1.6× |
| boundary AP | 3.15 s | 19.8× ¹ | 16.9× | — |
| Panoptic PQ | 10.5 s | 3.3× ² | — | — |
| Semantic mIoU | 2.87 s | 14.1× ³ | — | — |
| LVIS v1 bbox AP | 2.50 s | 81.0× ⁴ | — | 1.6× |

¹ boundary-iou-api · ² panopticapi · ³ mmsegmentation · ⁴ lvis-api, with 13.6× lower peak memory (1.11 vs 15.08 GiB)

Peak RSS on the COCO cells is 200 MiB, against 274 MiB (hotcoco),
569 MiB (pycocotools) and 691 MiB (faster-coco-eval).

<details>
<summary>Thread scaling (<code>num_threads</code>, 8-vCPU host)</summary>

faster-coco-eval ≥ 1.8 and hotcoco are multi-threaded by default, so each
column compares equal CPU budgets.

| Workload | 1 | 2 | 4 | 8 | vs hotcoco @ 8 | vs faster-coco-eval @ 8 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| bbox | 306 ms | 221 ms | 146 ms | 131 ms | 2.8× | 11.4× |
| segm | 926 ms | 536 ms | 304 ms | 239 ms | 2.3× | 14.6× |
| boundary | 3.18 s | 1.68 s | 866 ms | 711 ms | — | 23.8× |
| keypoints | 139 ms | 90 ms | 60 ms | 52 ms | 3.1× | 14.7× |
| bbox, Objects365 (1.06M dets) † | 6.6 s | — | — | 3.2 s | 2.6× | OOM at ~30 GiB |

Strict-mode results are bit-identical across thread counts.
† Single-rep dev run; pycocotools takes ~6 min per rep at this size. The
faster-coco-eval OOM is carried forward from the previous round rather
than re-measured — reproducing a 30 GiB OOM destabilises the host
mid-round.

</details>

AMD EPYC-Milan KVM VPS (4 cores x 2 threads = 8 logical CPUs), harness
mode `release` (N=10 measurement reps + 2 warmup, randomised impl order,
5% relative-IQR gate), against `hotcoco==1.0.1`,
`faster-coco-eval==1.8.0` and `pycocotools==2.0.11`. Full methodology,
every baseline pin and the complete results:
[`docs/benchmarks.md`](docs/benchmarks.md).

## Documentation

- [Tutorials](docs/tutorials/): first evaluation, training-loop integration
- [Migration guides](docs/migrate/): pycocotools, faster-coco-eval,
  panopticapi, lvis-api, mmsegmentation
- [How-to guides](docs/how-to/) · [Reference](docs/reference/) ·
  [Design decisions (ADRs)](docs/adr/)

## Contributing

```sh
just lint && just test && just audit
```

See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the ADR workflow, vendoring
policy, and code style.

## License

Licensed under either of [Apache-2.0](LICENSE-APACHE) or [MIT](LICENSE-MIT)
at your option. Test-only reference implementations used for parity checks
are listed in [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). They are not
shipped in wheels or binaries.

