Metadata-Version: 2.4
Name: imgad
Version: 0.1.0
Summary: Image anomaly detection with foundation-model features (CLIP, SigLIP, DINOv2) + classical detectors, Anomalib models, and statistical benchmarking
Author: Jónathan Heras
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/joheras/imgad
Project-URL: Repository, https://github.com/joheras/imgad
Project-URL: Issues, https://github.com/joheras/imgad/issues
Keywords: anomaly-detection,one-class,computer-vision,foundation-models,clip,dinov2,anomalib,patchcore,mvtec,benchmark
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: scipy>=1.9
Requires-Dist: scikit-learn>=1.2
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.40
Requires-Dist: pillow>=9.0
Requires-Dist: joblib>=1.2
Requires-Dist: pyyaml>=6.0
Requires-Dist: tqdm>=4.60
Provides-Extra: open-clip
Requires-Dist: open_clip_torch>=2.20; extra == "open-clip"
Provides-Extra: pyod
Requires-Dist: pyod>=1.1; extra == "pyod"
Provides-Extra: pandas
Requires-Dist: pandas>=1.5; extra == "pandas"
Provides-Extra: anomalib
Requires-Dist: anomalib>=2.0; extra == "anomalib"
Provides-Extra: report
Requires-Dist: matplotlib>=3.5; extra == "report"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: matplotlib>=3.5; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Provides-Extra: all
Requires-Dist: open_clip_torch>=2.20; extra == "all"
Requires-Dist: pyod>=1.1; extra == "all"
Requires-Dist: pandas>=1.5; extra == "all"
Requires-Dist: anomalib>=2.0; extra == "all"
Requires-Dist: matplotlib>=3.5; extra == "all"
Requires-Dist: pytest>=7; extra == "all"
Dynamic: license-file

# imgad — Image anomaly detection with foundation models, classical detectors and Anomalib

[![CI](https://github.com/joheras/imgad/actions/workflows/ci.yml/badge.svg)](https://github.com/joheras/imgad/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/imgad.svg)](https://pypi.org/project/imgad/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/joheras/imgad/blob/main/LICENSE)

A modular framework for image-level anomaly detection in the **one-class** setting (training uses
normal images only):

```
images ──► FeatureExtractor ──► preprocessing ──► classical detector ──► anomaly score
          (CLIP, SigLIP, DINOv2…   (l2, PCA,         (kNN, Mahalanobis,     (+ calibrated threshold)
           any HF model)            standard…)        LOF, OCSVM, IF…)
```

It also wraps end-to-end [Anomalib](https://github.com/open-edge-platform/anomalib) models (PatchCore,
PaDiM, EfficientAD…) with pixel-level anomaly maps, and includes a benchmarking module with repeated
runs, confidence intervals, statistical tests and automatic markdown reports.

## Installation

```bash
pip install imgad                    # core
pip install "imgad[anomalib]"        # + Anomalib models (PatchCore, PaDiM, EfficientAD…)
pip install "imgad[report]"          # + figures in benchmark reports (matplotlib)
pip install "imgad[all]"             # + open_clip, pyod, pandas, anomalib, matplotlib, pytest
```

From source: `git clone https://github.com/joheras/imgad && pip install -e "imgad[dev]"`.
Python ≥ 3.10.

## Quick start

```python
from imgad import AnomalyDetectionPipeline, HFFeatureExtractor, build_detector, data
from imgad.evaluation import evaluate_scores

ds = data.mvtec_ad("/path/to/mvtec_anomaly_detection", "bottle")

pipe = AnomalyDetectionPipeline(
    extractor=HFFeatureExtractor("openai/clip-vit-base-patch16"),
    detector=build_detector("knn", k=1),
    preprocessing=["l2"],
    cache_dir=".feature_cache",          # caches features per image
)
pipe.fit(ds.train_paths)                 # normal images only
scores = pipe.score(ds.test_paths)       # higher = more anomalous
preds  = pipe.predict(ds.test_paths)     # 0/1 with the calibrated threshold
print(evaluate_scores(ds.test_labels, scores, preds))

pipe.save("bottle.joblib")               # saves detector + config (not the model weights)
pipe = AnomalyDetectionPipeline.load("bottle.joblib")
```

## Inference with a trained model

The `.joblib` file stores the fitted detector, the preprocessing, the threshold, the extractor
configuration and metadata (number of training images, date, threshold origin). Hugging Face weights
are not stored: they are reloaded from the Hub (or the local HF cache) on `load`.

```python
from imgad import AnomalyPredictor

predictor = AnomalyPredictor.load("bottle.joblib")   # device auto (cuda/mps/cpu); device="cpu" to force it
r = predictor.predict_one("new.png")                 # path, PIL.Image or numpy array
r.is_anomaly, r.score, r.normal_percentile

results = predictor.predict("folder/")               # folder, list of paths/images, or a mix
predictor.save_results(results, "preds.csv")         # .csv or .json
for r in predictor.predict_iter("huge_folder/", chunk_size=256): ...   # constant memory
```

Each `Prediction` contains:

| Field | Meaning |
|---|---|
| `score` | raw score (detector-dependent scale; higher = more anomalous) |
| `is_anomaly` | `score > threshold` |
| `threshold` | threshold used |
| `normal_percentile` | fraction of normal training images with a lower score (0–1). Comparable across detectors: ~0.5 = typical normal image, 1.0 = rarer than any normal image seen |

**Tune the threshold** on a labelled validation set (no retraining):

```python
pipe = AnomalyDetectionPipeline.load("bottle.joblib")
pipe.calibrate_threshold(val_paths, val_labels, method="f1")   # f1 | youden | fpr | tpr
pipe.save("bottle_cal.joblib")
# Normal images only: method="fpr", target_fpr=0.01  -> ≤1% false alarms
```

Or on the fly: `AnomalyPredictor.load("bottle.joblib", threshold=0.02)`.

Shortcuts: the extractor can be an HF id (`"facebook/dinov2-base"`) and the detector a name
(`"mahalanobis"`) or a dict (`{"name": "knn", "k": 5}`).

## Feature extractors

`HFFeatureExtractor(model_name, pooling="auto", layer=None, normalize=True, ...)` works with any
vision model on the Hub that loads with `AutoModel`:

| Family | Examples | Default pooling |
|---|---|---|
| CLIP-style dual encoders | `openai/clip-*`, `google/siglip-*`, `google/siglip2-*`, `facebook/metaclip-*`, `kakaobrain/align-base`, `laion/CLIP-*` (HF format) | `projection` → `get_image_features` |
| Vision backbones | `facebook/dinov2-*`, `google/vit-*`, `microsoft/beit-*`, `facebook/convnext-*`, `microsoft/resnet-*` | `pooler` |

`pooling` options: `auto`, `projection`, `pooler`, `cls`, `mean` (mean of patch tokens), `cls+mean`.
`layer` selects intermediate layers (`layer=-4`, or `layer=[6, 9, 12]` to concatenate several), which
often capture local defects better than the more semantic last layer.

For **open_clip** checkpoints (`hf-hub:laion/...`, EVA-CLIP, DataComp…):
`OpenCLIPFeatureExtractor("hf-hub:laion/CLIP-ViT-B-32-laion2B-s34B-b79K")`.

## Detectors

All detectors share the same API (`fit`, `score`, `predict`) and the convention **higher score = more
anomalous**.

| Name | Method | Notes |
|---|---|---|
| `knn` | Distance to the k nearest normal samples | Very strong baseline with CLIP/DINO; `metric="cosine"` |
| `mahalanobis` / `gaussian` | Mahalanobis distance (Ledoit-Wolf, OAS, MCD) | Very good with foundation-model features |
| `centroid` | Distance to the mean embedding | Minimal cost |
| `kmeans` | Distance to the nearest centroid | Multimodal normality |
| `lof` | Local Outlier Factor (novelty mode) | |
| `gmm` | Negative log-likelihood under a Gaussian mixture | |
| `kde` | Negative KDE log-density | Better after `pca:<n>` to few dimensions |
| `ocsvm` | One-Class SVM | |
| `iforest` | Isolation Forest | |
| `pca` | PCA reconstruction error | |
| `ensemble` | Mean of normalised detectors (z-score or rank) | |
| `sklearn` | Wraps any sklearn estimator | `SklearnDetector(estimator=EllipticEnvelope())` |
| `pyod` | Any PyOD model | `{"name": "pyod", "model": "ECOD"}` |

**Threshold**: after `fit`, `threshold_` is set to the `threshold_quantile` (0.95 by default) of
*out-of-sample* scores of the normal data: leave-one-out for kNN/LOF and 5-fold cross-validation for
the rest. This avoids the optimistic threshold of detectors that overfit the training set (KDE, OCSVM,
PCA…). It can be disabled with `det.calibration = "train"`.

## Anomalib models (with anomaly maps)

`AnomalibPipeline` wraps any **one-class** [Anomalib](https://github.com/open-edge-platform/anomalib)
model (`patchcore`, `padim`, `efficient_ad`, `fastflow`, `stfpm`, `reverse_distillation`, `cflow`,
`draem`…; `imgad list` shows them) with the same API as the feature pipelines, and adds **pixel-level
anomaly maps**:

```python
from imgad import AnomalibPipeline, evaluate_maps

pc = AnomalibPipeline("patchcore", backbone="wide_resnet50_2", layers=["layer2", "layer3"], image_size=256)
pc.fit(ds.train_paths)                              # normal images only
scores, maps = pc.score_and_maps(ds.test_paths)     # maps: list of arrays (model resolution)
evaluate_maps(ds.test_masks, maps)                  # pixel_auroc, pixel_aupr, pixel_f1_max, aupro
pc.predict_masks(maps)                              # binary masks (map > pixel_threshold_)
pc.save("bottle_pc.joblib")                         # loaded by AnomalyPredictor.load / imgad predict as usual
```

- Anomalib provides the models and the training loop; its post-processing (normalisation and threshold
  learnt on a labelled validation set) is disabled. Scores and maps are the **raw** model outputs
  (higher = more anomalous).
- **Threshold**: as for the detectors, the `threshold_quantile` (0.95) of the scores of *unseen* normal
  images: `calibration_split=0.2` holds out that 20% of the training images (not used for training).
  `calibration_split=0` uses training scores (optimistic, especially for PatchCore). The pixel threshold
  (`pixel_threshold_`) is the same quantile of the maximum of each held-out normal map; with validation
  masks it can be tuned with `calibrate_pixel_threshold(images, masks)` (pixel F1).
  `calibrate_threshold` works as in the other pipelines.
- Images are resized to `image_size` **without cropping**, so the maps cover the whole image.
  `score_and_maps(..., size="original")` returns them at each image's own size.
- The `.joblib` stores the configuration **and the weights** (e.g. the PatchCore memory bank): nothing
  needs to be downloaded to load it.
- Model kwargs are those of the Anomalib constructor (`backbone`, `layers`, `coreset_sampling_ratio`,
  `pre_trained`…). For models trained over several epochs: `max_epochs` or
  `trainer_kwargs={"max_steps": ...}`. EfficientAD trains with batch size 1 and downloads imagenette.
- One-class models only; zero/few-shot models (WinCLIP, AnomalyDINO…) are not supported.

Inference with maps:

```python
predictor = AnomalyPredictor.load("bottle_pc.joblib")
res = predictor.predict("folder/", return_maps=True)   # r.anomaly_map (original size), r.anomalous_area
predictor.save_maps(res, "maps/")                      # <folder>_<name>_heatmap.png + _mask.png
predictor.save_results(res, "preds.csv")               # includes the heatmap and mask paths
```

Heatmaps use a fixed scale per model (blue = typical normal, middle of the scale = pixel threshold,
red = far above it), so normal images look blue instead of being rescaled.

## Preprocessing

List of steps applied before the detector (fitted on the normal data):
`"l2"`, `"standard"`, `"center"`, `"pca:0.95"` (explained variance) or `"pca:64"` (components),
`"pca_whiten:<n>"`, or any sklearn transformer.

## Data

```python
data.mvtec_ad(root, "screw")                     # MVTec AD / MVTec layout (converted VisA, etc.)
data.from_folders(train_normal, test_normal, [test_anom_1, test_anom_2])
data.one_vs_rest(root, normal_class="cat")       # semantic AD from a classification dataset
```

Each returns an `AnomalyDataset` with `train_paths`, `test_paths`, `test_labels`, `test_types` and, for
MVTec, `test_masks` (path of the ground-truth mask in `ground_truth/<type>/<name>_mask.png`, or `None`
for normal images).

## Evaluation and benchmarks

`evaluate_scores` returns AUROC, AUPR, F1-max and FPR@95%TPR (and F1/TPR/FPR at the threshold if
predictions are given). `per_type_auroc` breaks the AUROC down by defect type.

`evaluate_maps(masks, maps)` gives the localisation metrics: pixel-AUROC, pixel-AUPR, pixel-F1-max and
AUPRO (FPR ≤ 0.3, the standard MVTec AD metric). Masks are resized to each map's resolution.

`benchmark(extractors, detectors, datasets, preprocessings, anomalib_models=[...])` evaluates the full
grid, extracting features once per (extractor, dataset), plus each Anomalib model per dataset (with
pixel metrics when masks are available). In YAML, use the `anomalib:` key (see
`configs/mvtec_benchmark.yaml`).

## Benchmark reports

`imgad benchmark --report DIR` writes a markdown report with figures:

```bash
imgad benchmark --config configs/mvtec_benchmark.yaml --report report/ --seeds 0 1 2 --bootstrap 1000
```

```
report/
  report.md       # the report
  figures/*.png   # every figure referenced in the report
  results.csv     # one row per dataset × configuration × seed
  config.yaml     # configuration used
```

The report contains:

| Section | Content |
|---|---|
| Summary | best configuration, runner-up and significance, best extractors/detectors, hardest dataset |
| Data analysis | images per split, defect types, resolution, masks; example images with ground-truth masks; warnings (small or imbalanced test sets, few training images) |
| Methods | extractors (with dimension), preprocessing, detectors and Anomalib models with a short description; threshold protocol, metrics, repetitions and statistical tests |
| Results | ranking table and chart, extractor × detector heatmap, per-dataset results, ROC curves, score histograms with the threshold, per-defect-type AUROC, operating point, pixel-level metrics and example anomaly maps, spread across seeds, Friedman/Nemenyi/Wilcoxon (or DeLong) with a critical-difference diagram, computational cost |
| Discussion | extractor vs. detector effect, consistency across datasets, significance, hard defect types, Anomalib vs. global features, threshold calibration, localisation |
| Limitations and reproducibility | caveats derived from the setup, exact command, configuration and environment versions |

Figures need matplotlib (`pip install "imgad[report]"`); without it the report contains tables only.
The summary and discussion are generated from the numbers with simple rules: treat them as a first
draft to check and complete. To rebuild a report without rerunning the benchmark:

```bash
imgad benchmark --config cfg.yaml --scores-out scores.joblib
imgad report --scores scores.joblib --config cfg.yaml --out report/ --title "My benchmark"
```

From Python: `imgad.report.generate_report(rows, datasets, "report/", config=cfg)` with
`rows = benchmark(..., keep_scores=True)`.

## Statistical analysis: repetitions, confidence intervals and tests

**Repetitions.** `benchmark(..., seeds=[0, 1, 2, 3, 4], train_subsample=0.8)` runs every configuration
once per seed (column `seed`). The seed sets the detectors' `random_state` (unless the spec fixes it),
the calibration folds, the Anomalib seed and the **training subset** (`train_subsample`: a fraction if
float, a number of images if int). Without subsampling, deterministic detectors (kNN, Mahalanobis…)
give the same result for every seed. Features are extracted once and reused across seeds.

**Confidence intervals.** There are two sources of uncertainty, measured separately:

| Source | How | Where |
|---|---|---|
| Test sample | stratified bootstrap of the test images (`bootstrap=1000`) | `auroc_ci_low/high`, `aupr_ci_low/high` in each row |
| Training | repetitions with seeds + `train_subsample` | `stats.summarize_seeds(rows)`: mean, standard deviation and t-based CI of the mean |

**Statistical tests** (`imgad.stats`):

| Question | Test | Function |
|---|---|---|
| Does A have a higher AUROC than B **on this dataset**? | DeLong (correlated AUROCs, same images) | `delong_test(y, sa, sb)`, `delong_vs_baseline(rows, base)` |
| Same for AUPR, F1-max or another metric | paired bootstrap | `paired_bootstrap_test(y, sa, sb, metric="aupr")` |
| Are there differences between configurations **across datasets**? | Friedman (+ Iman-Davenport correction) | `compare_configs(rows)` |
| Which pairs differ? | Nemenyi (critical difference of ranks) and Wilcoxon signed-rank + Holm | `compare_configs(rows)["pairs"]`, `plot_cd_diagram` |

To compare across datasets, seeds are first averaged within each dataset, and each dataset (e.g. each
MVTec category) is a block, following Demšar (2006). Wilcoxon-Holm usually has more power with few
configurations, while Nemenyi is the classical post-hoc test after Friedman and the one drawn in the CD
diagram. DeLong needs per-image scores (`keep_scores=True`).

```python
from imgad import benchmark, stats

rows = benchmark(extractors, detectors, datasets, seeds=range(5), train_subsample=0.8,
                 bootstrap=1000, keep_scores=True, anomalib_models=["patchcore"])
stats.summarize_seeds(rows)                          # mean ± std per (dataset, config)
res = stats.compare_configs(rows, metric="auroc")    # also aupr, pixel_auroc, aupro, fpr@95tpr…
print(stats.report_markdown(res))                    # mean ranks, Friedman, CD, pairs with Holm
stats.plot_cd_diagram(res, "cd.png")
stats.delong_vs_baseline(rows, baseline=res["configs"][0])
```

From the CLI:

```bash
imgad benchmark --config configs/mvtec_benchmark.yaml --seeds 0 1 2 3 4 --train-subsample 0.8 \
                --bootstrap 1000 --out results.csv --scores-out scores.joblib --stats
imgad stats --results results.csv --metric auroc --cd-plot cd.png --out stats.md
imgad stats --scores scores.joblib --baseline "openai/clip-vit-base-patch16 | l2 | knn(k=1)"   # + DeLong
```

## CLI

```bash
imgad list
imgad fit --model openai/clip-vit-base-patch16 --detector knn -p k=1 --prep l2 \
          --train data/bottle/train/good --out bottle.joblib

# inference: images and/or folders; output to stdout, .csv or .json
imgad predict --model bottle.joblib new.png data/bottle/test --out preds.csv
imgad predict --model bottle.joblib folder/ --only-anomalies --threshold 0.02 --device cpu

# re-tune the threshold on labelled validation data (f1 | youden | fpr | tpr)
imgad calibrate --model bottle.joblib --normal val/good --anomalous val/defects --method f1 --out bottle_cal.joblib
imgad info --model bottle_cal.joblib         # configuration, threshold and metadata

imgad benchmark --config configs/mvtec_benchmark.yaml --out results.csv --report report/

# Anomalib models: -p sets the model kwargs; predict saves heatmaps and masks
imgad fit --anomalib patchcore -p backbone=wide_resnet50_2 "layers=[layer2,layer3]" \
          --image-size 256 --train data/bottle/train/good --out bottle_pc.joblib
imgad predict --model bottle_pc.joblib data/bottle/test --maps-dir maps/ --out preds.csv
```

## Extending

Registering your own detector or extractor makes it available by name in Python, YAML and the CLI (see
`examples/extending.py`):

```python
from imgad import BaseDetector, register_detector

@register_detector("my_detector")
class MyDetector(BaseDetector):
    def _fit(self, X): ...
    def _score(self, X): ...   # higher = more anomalous
```

## Example results

[`benchmarks/mvtec_4categories/`](https://github.com/joheras/imgad/tree/main/benchmarks/mvtec_4categories) contains a benchmark of 5 extractors
× 11 detectors on 4 MVTec AD categories (report, results CSV and configuration).

## Project layout

```
src/imgad/
  features/      base.py (interface, registry, cache) · huggingface.py · open_clip.py
  detectors/     base.py (interface, registry, calibration) · classical.py
  anomalib_backend.py (AnomalibPipeline) · maps.py (masks, heatmaps)
  preprocessing.py · pipeline.py · inference.py · data.py · evaluation.py · stats.py · report.py · cli.py
configs/         example benchmark YAML files
examples/        quickstart.py · inference.py · compare_models.py · extending.py · anomalib_models.py
benchmarks/      example benchmark results
tests/           pytest (offline: a tiny CLIP generated locally; untrained ResNet-18 for Anomalib)
```

## Possible next steps

- **Zero-/few-shot** detection with text prompts (WinCLIP, AnomalyCLIP) using the text encoder.
- Native PatchCore on patch features of foundation models (CLIP/DINOv2), to compare with Anomalib's (CNN).
- Test-time augmentation (TTA) and multi-scale aggregation.
- Few-shot sweep (`train_subsample` = 1, 2, 4, 8… images) reusing the seeds and the tests in `imgad.stats`.

## License

Distributed under the [Apache 2.0](https://github.com/joheras/imgad/blob/main/LICENSE) license.

## Acknowledgements

This work is partially supported by Agencia de Desarrollo Económico de La Rioja [ADER 2022-I-IDI-00015].
