Metadata-Version: 2.4
Name: true-colon
Version: 0.1.1
Summary: TRUE-Colon: a deployment-oriented evaluation protocol for polyp detection on full-length colonoscopy procedures, measuring false-alert burden, detection latency and temporal reliability alongside localization accuracy.
Author-email: Sebastian Doerrich <sebastian.doerrich@uni-bamberg.de>
License: MIT
Project-URL: Paper, https://arxiv.org/abs/2608.13711
Project-URL: Homepage, https://github.com/sdoerrich97/true-colon
Project-URL: Weights, https://huggingface.co/collections/sdoerrich97/true-colon
Project-URL: Issues, https://github.com/sdoerrich97/true-colon/issues
Keywords: colonoscopy,polyp-detection,object-detection,medical-imaging,benchmark,evaluation,froc,computer-aided-detection
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<3,>=1.24
Requires-Dist: pycocotools<3,>=2.0.7
Provides-Extra: plots
Requires-Dist: matplotlib>=3.8; extra == "plots"
Provides-Extra: streaming
Requires-Dist: ijson>=3.3; extra == "streaming"
Provides-Extra: dev
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Provides-Extra: experiments
Requires-Dist: matplotlib>=3.8; extra == "experiments"
Requires-Dist: ijson>=3.3; extra == "experiments"
Requires-Dist: pillow>=10; extra == "experiments"
Requires-Dist: pyyaml>=6.0; extra == "experiments"
Requires-Dist: tqdm>=4.66; extra == "experiments"
Requires-Dist: natsort>=8.4; extra == "experiments"
Requires-Dist: pandas>=2.2; extra == "experiments"
Dynamic: license-file

# TRUE-Colon: Exposing a Consistent Transfer Asymmetry in Real-Time Polyp Detection @ MICCAI 2026 EndoLINA

<p align="center">
    [<a href="https://arxiv.org/abs/2608.13711">Preprint</a>]
    [<a href="https://pypi.org/project/true-colon/">PyPI</a>]
    [<a href="https://huggingface.co/collections/sdoerrich97/true-colon">Pretrained Weights</a>]
    [<a href="#citation">Citation</a>]
</p>

<p align="center">
  <img src="assets/qualitative_comparison.png" alt="Four detectors on the same REAL-Colon frames: ground truth in green, predictions in red" width="70%">
</p>
<p align="center">
  <em>The same seven REAL-Colon frames through all four detectors. Ground truth in
  green, predictions in red. The last rows are the point: on flat, blurred and
  obliquely-viewed lesions every architecture misses, and curated benchmarks contain
  almost none of them.</em>
</p>

## Overview

Computer-aided detection (CADe) systems for colonoscopy promise to reduce clinical miss
rates, yet reliable real-world deployment remains elusive. This translational gap stems
in part from a structural flaw in model development: the reliance on curated datasets
that under-represent the long negative stretches and procedure-related artifacts
characteristic of routine examinations. Training and evaluating architectures strictly on
these lesion-centric benchmarks creates an illusion of success, since such benchmarks
cannot capture clinically crucial metrics such as false-alert burden, detection latency,
and temporal reliability.

To expose this gap, we establish **TRUE-Colon**, a standardized benchmarking protocol that
measures these deployment characteristics alongside localization accuracy, and evaluate
four real-time architectures (Faster R-CNN, YOLOv8, YOLOv11, RT-DETR) across curated
benchmarks (SUN, PICCOLO) and 60 unedited, full-length procedures (REAL-Colon).

### Key Contributions

* **A standardized, deployment-oriented evaluation protocol.** Four complementary tiers
  measured at a *matched false-alert operating point*, so architectures are compared at
  equal clinical cost rather than at an arbitrary shared confidence value.
* **A consistent transfer asymmetry.** Models trained on curated clips collapse on full
  procedures (YOLOv11 mAP50 drops from 0.724 on SUN to 0.164 on REAL-Colon), while
  procedure-trained models largely retain their accuracy on curated benchmarks.
* **An accuracy-versus-throughput trade-off** that fixed-threshold comparisons obscure:
  at a matched false-alert burden the Transformer detector leads on sensitivity, latency
  and temporal persistence, while the convolutional detectors stay competitive at
  markedly higher throughput.

### What the protocol measures

| Tier | Question it answers | Why curated benchmarks cannot answer it |
|---|---|---|
| **Detection** | Are the boxes in the right place? | — (standard COCO mAP) |
| **Frame** | How often does it alert on a lesion-free frame? | Curated clips are almost all lesion-bearing |
| **Lesion** | How fast is the first detection, and does the box persist? | Requires contiguous, unedited video |
| **AFROC** | Sensitivity against the fraction of clean frames raising an alert | Needs a realistic negative base rate |

## Installation

```bash
pip install true-colon
```

The evaluator depends only on `numpy` and `pycocotools` — no deep-learning framework.
Optional extras:

```bash
pip install "true-colon[plots]"      # FROC / AFROC curve rendering
pip install "true-colon[streaming]"  # streaming ingest for very large prediction files
```

To reproduce the paper's experiments, clone and install the reproduction layer:

```bash
git clone https://github.com/sdoerrich97/true-colon.git
cd true-colon
pip install -e ".[experiments,plots,streaming]"
```

The full pipeline runs from `experiments/`: `prepare_data.py` to build the splits,
`train.py` and `predict.py` per detector, `evaluate.py` for the protocol, then
`aggregate_seeds.py` and `make_tables.py` for the paper's tables and figures. Each script
carries the published settings as its argparse defaults, so the commands below are
complete as written.

## Quick Start

TRUE-Colon evaluates **any** detector. It reads a COCO ground-truth file and a COCO
predictions file, so if your model can export COCO results, you can run the protocol on
it — no retraining, no GPU, no images required.

```python
import true_colon as tc

evaluator = tc.create_evaluator(
    dataset="realcolon",  # or None, and configure for your own data
    id_source="pred_string",  # "gt_integer" for Detectron2-style integer image ids
)
result = evaluator.evaluate_paths("ground_truth.json", "predictions.json")

print(f"mAP50            {result.coco.ap50:.3f}")
print(f"frame TPR / FPR  {result.frame.tpr:.3f} / {result.frame.fpr:.3f}")
print(f"lesions detected {result.lesion.summary.detected_any}/{result.lesion.summary.n_lesions}")
print(f"mean latency     {result.lesion.summary.mean_latency_frames:.1f} frames")
```

### Matching the operating point

Raw confidence scores are not comparable across architectures. To compare detectors at an
equal false-alert burden, find the threshold that hits a target frame-level false-positive
rate — **on your validation split**, then evaluate at it:

```python
matched = evaluator.match_operating_point(val_gt, val_preds, target_fpr=0.045)
print(matched.threshold, matched.achieved_fpr, matched.feasible)
```

From the command line:

```bash
python experiments/evaluate.py \
    --gt  ground_truth.json \
    --pred predictions.json \
    --dataset realcolon --id-source pred_string \
    --target-fpr 0.045 --out-dir outputs/
```

### Runnable notebooks

All three are committed with their outputs, so they read without being run:

| Notebook | What it shows |
| :--- | :--- |
| [`01_evaluate_your_detector`](examples/01_evaluate_your_detector.ipynb) | The protocol end to end on a synthetic procedure, and why one number hides three stories |
| [`02_threshold_matching`](examples/02_threshold_matching.ipynb) | Why a shared threshold is not a shared operating point, and the trap in fixing it |
| [`03_verify_annotations`](examples/03_verify_annotations.ipynb) | Checking that a dataset conversion preserved the boxes, including what a padding bug looks like |

## Tables and figures

Aggregate several seeds, then render. Nothing downstream of `aggregate_seeds.py`
recomputes a number, so a table cannot disagree with the results it came from:

```bash
python experiments/aggregate_seeds.py \
    --model yolov11 outputs/y11m_s0/**/results.json outputs/y11m_s42/**/results.json \
    --model rtdetr  outputs/rtdetr_s0/**/results.json \
    --out outputs/tables/realcolon.tsv

python experiments/make_tables.py outputs/tables/realcolon.tsv \
    --out outputs/tables/realcolon.tex

python experiments/figures/froc_afroc.py \
    --model "yolov11=outputs/y11m_s0/results.json,outputs/y11m_s42/results.json" \
    --out-dir outputs/figures/

python experiments/figures/lesion_bars.py \
    --model "yolov11=outputs/y11m_s0/results.json,outputs/y11m_s42/results.json" \
    --meta-root /data/realcolon --out-dir outputs/figures/
```

Model names, ordering, colours and metric labels come from
`experiments/configs/presentation.py`, so figures and tables cannot drift apart. The
markers on the FROC/AFROC curves default to the published matched thresholds rather than
being retyped per figure.

Averaging curves across seeds requires the same confidence grid, and a mismatch is an
error: point-by-point averaging of different grids yields a smooth curve that is an
average of unrelated operating points.

## Preparing the datasets

Evaluation needs only a ground-truth JSON and a predictions JSON, so nothing above
requires the raw datasets. Reproducing the *training* runs does. One CLI covers all three:

```bash
# REAL-Colon, the route behind the published YOLOv8 / YOLOv11 / RT-DETR checkpoints
python experiments/prepare_data.py realcolon \
    --src /data/realcolon --dst /data/prepared/realColon_640x640 --stats

# REAL-Colon, the route behind the published Faster R-CNN checkpoint
python experiments/prepare_data.py realcolon \
    --src /data/realcolon --dst /data/prepared/realcolon_frcnn --route coco-export

python experiments/prepare_data.py sun     --src /data/sun     --dst /data/prepared/sun_split
python experiments/prepare_data.py piccolo --src /data/piccolo --dst /data/prepared/piccolo_split
```

REAL-Colon has **two** published routes and they are not interchangeable: they write
different layouts and read `--negative-ratio` differently. The default reproduces the
canonical one. See `experiments/data/realcolon.py` for how the two were told apart.

Each run writes a `split_manifest.json` recording a checksum of every split's membership.
A seed only reproduces a split if the RNG behaves identically, which is an assumption
across Python versions rather than a guarantee, so check rather than trust:

```bash
python experiments/prepare_data.py realcolon \
    --src /data/realcolon --dst /data/prepared/realColon_640x640 --verify
```

`--verify` re-resolves membership without reading or writing a single frame, and exits
non-zero if the splits have changed. `--stats` prints a sanity report over a prepared
root: per-split frame and box counts, image/label agreement, and whether any lesion
appears in more than one split.

## Model Zoo

Nine REAL-Colon checkpoints behind the paper's main table, grouped in a
[HuggingFace Collection](https://huggingface.co/collections/sdoerrich97/true-colon).

| Detector | Training data | Input | Seeds | Identifier |
| :--- | :--- | :--- | :--- | :--- |
| **YOLOv8-M** | REAL-Colon | 640 | 0, 42, 123 | `yolov8` |
| **YOLOv11-M** | REAL-Colon | 640 | 0, 42, 123 | `yolov11` |
| **RT-DETR** | REAL-Colon | 640 | 0, 42, 123 | `rtdetr` |

```python
from experiments.reference.weights import download_checkpoint

path = download_checkpoint("yolov11", seed=42)
```

> **Not released.** The Faster R-CNN checkpoints and every SUN- and PICCOLO-trained model
> did not survive the original author's offboarding and could not be recovered, so the
> transfer-direction rows of the paper have no released artifact behind them. We would
> rather say this plainly than quietly ship a partial zoo.

## Reproducibility

Because the protocol is framework-agnostic and CPU-only, **every table in the paper can be
regenerated from stored prediction files inside a hash-locked container, with no GPU and
no dataset**:

```bash
docker build --target runtime -t true_colon:latest .
docker run --rm -v /path/to/artifacts:/data:ro -v $PWD/outputs:/outputs true_colon:latest \
    python experiments/evaluate.py --gt /data/gt.json --pred /data/predictions.json \
    --dataset realcolon --out-dir /outputs
```

Detector *training* is a different matter: Detectron2 and Ultralytics pin incompatible
torch builds, so they get their own out-of-lock Docker targets (`ultralytics`,
`detectron2`). Only the evaluation runtime is covered by `uv.lock` and the hashed
`requirements.txt`.

Ultralytics must be **>= 8.3.223**. Below that, RT-DETR's exported `predictions.json`
carries wrong box coordinates while its metrics still look plausible; the fix landed
upstream in that release. See
[`PATCH_NOTES.md`](experiments/reference/rtdetr_ultralytics/PATCH_NOTES.md).

### Training and inference

Both entry points take a prepared root and inject the dataset, seed and output directory,
so nothing is hardcoded:

```bash
docker build --target ultralytics -t true_colon:ultralytics .

# Train. The published runs used seeds 0, 42 and 123.
python experiments/train.py --detector yolov11 \
    --data /data/prepared/realColon_640x640 --seed 42 --out-dir /outputs/y11m_s42

# Export predictions, which is what evaluation consumes.
python experiments/predict.py --detector yolov11 \
    --data /data/prepared/realColon_640x640 --split test \
    --weights /outputs/y11m_s42/weights/best.pt --out-dir /outputs/y11m_s42
```

`--published-seed` fetches a released checkpoint from HuggingFace instead of `--weights`,
so predictions can be reproduced without retraining. `--print-config` on `train.py` shows
the resolved hyperparameters without starting a run.

Predictions are exported with a low confidence pre-filter (0.001) on purpose: thresholding
at export time would destroy the information the FROC, AFROC and matched operating-point
analyses need. The protocol thresholds at evaluation time instead.

Note that the four detectors do **not** all read the same prepared root. Faster R-CNN was
trained from the COCO-export route (`--route coco-export` above); the other three from the
letterbox route. `experiments/data/realcolon.py` explains how the two were told apart.

Evaluation itself is deterministic and seed-free. The published split seed (1000) and
training seeds (0, 42, 123) are frozen in `experiments/configs/_published.py`.

**Honest limitation.** This repository reproduces the *pipeline*, not the thesis's exact
numbers: the original datasets, checkpoints and result files were not recoverable, so no
published configuration can be re-executed and no drift figure is quoted. The port is
instead verified by a differential test against the original implementation, which matches
it exactly across every metric tier: the preparation routes, the split samplers and all
five metric tiers were checked frame-by-frame against the code that produced the paper.

## Project Structure

```
true_colon/     the protocol (the installed package; MIT, numpy + pycocotools only)
experiments/    reproduction layer (not in the wheel)
  data/         dataset preparation, one module per dataset, plus sanity checks
  reference/    the four detector baselines, each with its upstream LICENSE
  figures/      the paper's figure generators
  configs/      frozen published constants and the paper's presentation choices
examples/       runnable notebooks
```

## Citation

```bibtex
@inproceedings{doerrich2026truecolon,
  title         = {TRUE-Colon: Exposing a Consistent Transfer Asymmetry in Real-Time Polyp Detection},
  author        = {Doerrich, Sebastian and Schwab, Andreas Franz and Di Salvo, Francesco
                   and Rai, Shyam Nandan and Nguyen, Hanh Huyen My and Ledig, Christian},
  booktitle     = {MICCAI Workshop on Endoscopic Lesion Identification and Analysis (EndoLINA)},
  year          = {2026},
  eprint        = {2608.13711},
  archivePrefix = {arXiv},
  primaryClass  = {eess.IV},
  doi           = {10.48550/arXiv.2608.13711}
}
```

Sebastian Doerrich and Andreas Franz Schwab contributed equally. The LNCS proceedings
entry will add volume and page numbers; until then the arXiv record above is the
resolvable version.

## Licensing

The components carry different licenses and are deliberately kept distinct.

| Component | License |
| :--- | :--- |
| `true_colon/` and project-authored `experiments/`, `examples/` | **MIT** |
| `experiments/reference/yolo_ultralytics/`, `rtdetr_ultralytics/` | **AGPL-3.0** (Ultralytics) |
| `experiments/reference/frcnn_detectron2/` | **Apache-2.0** (Detectron2) |
| **Released model weights** | **AGPL-3.0** — trained with Ultralytics |
| REAL-Colon, PICCOLO datasets | CC BY 4.0 (not redistributed, apart from the frames in `assets/`) |
| SUN dataset | Research agreement required (not redistributed here) |

The protocol package never imports Ultralytics, which is what keeps it MIT-licensable.

The frames in `assets/qualitative_comparison.png` are from
[REAL-Colon](https://doi.org/10.1038/s41597-024-03359-0) (Biffi et al., *Scientific Data*
11:539, 2024), used and redistributed here under **CC BY 4.0**. The overlaid boxes are
this project's. No other dataset imagery is redistributed.
If you use the released weights, the AGPL's obligations apply to your use of them.

## Changelog

### v0.1.1
* Packaging and documentation only; no change to `true_colon` behaviour or to any
  reported number.
* Ship `experiments/figures/`, the paper's figure generators. An unanchored `.gitignore`
  rule had kept them out of v0.1.0, so the figure commands documented below did not work
  from a clean checkout.
* Correct the citation: six authors with Doerrich first, plus the arXiv eprint and DOI.
* Link the preprint, [arXiv:2608.13711](https://arxiv.org/abs/2608.13711).
* Add `CITATION.cff`.

### v0.1.0
* Initial release accompanying the MICCAI 2026 EndoLINA paper.
