Metadata-Version: 2.5
Name: vision-data-doctor
Version: 0.2.0
Summary: Diagnose vision datasets before they lie to you: duplicates, train/val leakage, annotation defects.
Project-URL: Homepage, https://github.com/shivpratapsinghpanwar/data-doctor
Author: Shiv Pratap Singh Panwar
License: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: >=3.10
Requires-Dist: pillow>=10.0
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# data-doctor

Diagnose vision datasets **before they lie to you**: duplicate images,
train/val leakage, and annotation defects — in one command, with no heavy
dependencies (Pillow only).

## Why

Metrics are only as honest as the split behind them. This tool exists because
of a real production dataset whose delivered train/val split contained **81
groups of byte-identical images spanning both sides** — the reference model
scored 0.75 recall on that split and substantially less on an honest one.
Nothing in the training stack warned about it. `data-doctor` makes that
check (and its cousins) a 10-second habit instead of a post-mortem.

## Install

```bash
pip install data-doctor
```

## Usage

**Find corrupt files, exact duplicates, and near-duplicates in a folder:**

```bash
data-doctor scan data/images --json report.json
```

**Check a train/val split for leakage** (byte-identical *and* perceptually
near-identical frames on both sides):

```bash
data-doctor leakage --train data/train --val data/val
```

```
train: 714 images | val: 231 images

  FAIL exact leaks (byte-identical in both splits): 81
         val/frame_0117.jpg == train/frame_0116.jpg
         ...
  FAIL near leaks (pixel-verified >= 90% similar): 12
         val/frame_0201.jpg ~~ train/frame_0200.jpg (99.4% similar)
         val/shot_114.jpg ~~ train/shot_113.jpg (98.7% similar, rot90)

  40.3% of the validation set is leaked from train.
  Metrics measured on this split overstate real performance.
```

**Structural checks on a COCO annotation file:**

```bash
data-doctor coco annotations/train.json --images data/images
```

Checks: duplicate image ids and file names, annotations referencing missing
images, unknown category references, degenerate boxes (zero width/height),
degenerate polygons (< 3 points), referenced files missing on disk, and a
count of zero-annotation images (hard negatives or missing labels — you
decide which, the tool makes sure you *see* them).

## Exit codes

`0` when clean, `1` when any check fails — drop it straight into CI:

```yaml
- run: data-doctor leakage --train data/train --val data/val
```

## Python API

```python
from data_doctor import scan_directory, check_leakage, check_coco

result = check_leakage(Path("data/train"), Path("data/val"))
print(result.leaked_val_fraction)
```

## How near-duplicate detection works

Three stages, so every reported duplicate is a concrete, verified claim —
not a fuzzy hash coincidence:

1. **Exact** — SHA-256 catches byte-identical copies.
2. **Nominate** — two perceptual hash families (difference hash and DCT
   pHash), computed over all 8 rotations/flips, propose candidate pairs.
   Candidates are found by 16-bit quadrant bucketing (pigeonhole: any two
   hashes within hamming distance 3 share an identical quadrant), so scans
   stay fast on large folders.
3. **Verify** — every candidate pair is confirmed on decoded pixels:
   grayscale thumbnails compared across the 8 dihedral orientations. Only
   pairs at or above the similarity floor are reported, each with its
   measured score and matching orientation (`99.4% similar, rot90`).

Hashes alone never convict — they only nominate. Tune recall with
`--near-threshold` (default 3) and precision with `--min-similarity`
(default 0.90). Rotated, flipped, re-encoded, and resized copies are all
caught; the JSON report carries every verified pair with its score.

## License

MIT
