Metadata-Version: 2.5
Name: argus-curator
Version: 0.5.0
Summary: Image curation and moderation: quality scoring, near-duplicate dedup, face clustering, and apparent-age, scene and garment tagging with auditable filters; standalone or as LoRA data prep
Project-URL: Homepage, https://github.com/smk762/argus-curator
Project-URL: Repository, https://github.com/smk762/argus-curator
Project-URL: Issues, https://github.com/smk762/argus-curator/issues
Project-URL: Changelog, https://github.com/smk762/argus-curator/blob/main/CHANGELOG.md
Author: smk762
License-Expression: MIT
License-File: LICENSE
Keywords: curation,dataset,deduplication,diffusion,face-clustering,flux,image-quality,insightface,lora,phash,sdxl,stable-diffusion,training
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: argus-cortex<0.5,>=0.4
Requires-Dist: imagehash>=4.3
Requires-Dist: numpy>=1.24
Requires-Dist: pillow>=10.0
Requires-Dist: pydantic>=2.6
Requires-Dist: structlog>=24.0
Provides-Extra: all
Requires-Dist: argus-cortex[server]<0.5,>=0.4; extra == 'all'
Requires-Dist: certifi>=2026.10.19; extra == 'all'
Requires-Dist: fastapi>=0.110.1; extra == 'all'
Requires-Dist: httpx>=0.27; extra == 'all'
Requires-Dist: insightface>=0.7.3; extra == 'all'
Requires-Dist: onnxruntime>=1.16; extra == 'all'
Requires-Dist: opencv-python-headless>=4.9; extra == 'all'
Requires-Dist: python-multipart>=0.0.7; extra == 'all'
Requires-Dist: rich>=13.0; extra == 'all'
Requires-Dist: scikit-learn>=1.3; extra == 'all'
Requires-Dist: typer>=0.12; extra == 'all'
Requires-Dist: uvicorn>=0.27; extra == 'all'
Provides-Extra: cli
Requires-Dist: rich>=13.0; extra == 'cli'
Requires-Dist: typer>=0.12; extra == 'cli'
Provides-Extra: dev
Requires-Dist: certifi>=2026.10.19; extra == 'dev'
Requires-Dist: httpx2>=2.7; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: faces
Requires-Dist: insightface>=0.7.3; extra == 'faces'
Requires-Dist: onnxruntime>=1.16; extra == 'faces'
Requires-Dist: opencv-python-headless>=4.9; extra == 'faces'
Requires-Dist: scikit-learn>=1.3; extra == 'faces'
Provides-Extra: fairface-convert
Requires-Dist: onnx>=1.16; extra == 'fairface-convert'
Requires-Dist: torch>=2.5; extra == 'fairface-convert'
Requires-Dist: torchvision>=0.20; extra == 'fairface-convert'
Provides-Extra: garments-convert
Requires-Dist: huggingface-hub>=0.23; extra == 'garments-convert'
Requires-Dist: onnx>=1.16; extra == 'garments-convert'
Requires-Dist: onnxscript>=0.2; extra == 'garments-convert'
Requires-Dist: torch>=2.6; extra == 'garments-convert'
Requires-Dist: transformers>=4.45; extra == 'garments-convert'
Provides-Extra: gpu
Requires-Dist: insightface>=0.7.3; extra == 'gpu'
Requires-Dist: onnxruntime-gpu>=1.16; extra == 'gpu'
Requires-Dist: opencv-python-headless>=4.9; extra == 'gpu'
Requires-Dist: scikit-learn>=1.3; extra == 'gpu'
Provides-Extra: mivolo-convert
Requires-Dist: huggingface-hub>=0.23; extra == 'mivolo-convert'
Requires-Dist: onnx>=1.16; extra == 'mivolo-convert'
Requires-Dist: onnxscript>=0.2; extra == 'mivolo-convert'
Requires-Dist: safetensors>=0.4; extra == 'mivolo-convert'
Requires-Dist: timm>=1.0; extra == 'mivolo-convert'
Requires-Dist: torch>=2.6; extra == 'mivolo-convert'
Provides-Extra: places365-convert
Requires-Dist: onnx>=1.16; extra == 'places365-convert'
Requires-Dist: torch>=2.5; extra == 'places365-convert'
Requires-Dist: torchvision>=0.20; extra == 'places365-convert'
Provides-Extra: scenes
Requires-Dist: onnxruntime>=1.16; extra == 'scenes'
Provides-Extra: server
Requires-Dist: argus-cortex[server]<0.5,>=0.4; extra == 'server'
Requires-Dist: certifi>=2026.10.19; extra == 'server'
Requires-Dist: fastapi>=0.110.1; extra == 'server'
Requires-Dist: httpx>=0.27; extra == 'server'
Requires-Dist: python-multipart>=0.0.7; extra == 'server'
Requires-Dist: uvicorn>=0.27; extra == 'server'
Description-Content-Type: text/markdown

# Argus Curator

[![PyPI](https://img.shields.io/pypi/v/argus-curator)](https://pypi.org/project/argus-curator/)
[![Python](https://img.shields.io/pypi/pyversions/argus-curator)](https://pypi.org/project/argus-curator/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![CI](https://github.com/smk762/argus-curator/actions/workflows/ci.yml/badge.svg)](https://github.com/smk762/argus-curator/actions/workflows/ci.yml)

Image curation and moderation: score, de-duplicate, and tag images by face, apparent age, scene and garment, then filter on those tags with a reason recorded for every exclusion. Use it on its own to audit or moderate an image collection, or as the LoRA data-prep front-end of the Argus suite, captioned downstream by [argus-lens](https://github.com/smk762/argus-lens).

Two ways to use it:

- **Standalone moderation and dataset auditing.** Scan a folder or call the
  HTTP API, review the per-image report, and keep, drop or quarantine images
  on criteria you choose. Nothing has to be trained or captioned. See
  [Standalone: moderation and dataset auditing](#standalone-moderation-and-dataset-auditing).
- **The curation stage of the Argus suite**, described below.

`argus-curator` is the **curation stage** of the Argus suite. Upstream,
[argus-quarry](https://github.com/smk762/argus-quarry) acquires provenance-clean
images; curator decides *which images, of whom, at what quality* belong in a
LoRA training set; argus-lens then decides *what's in each image*. Curator and
lens share one `TargetProfile`, so a manifest written here is captioned
downstream with no remapping.

```
   argus-quarry               argus-curator (:8101)      argus-lens (:8100)      imogen / kohya
   ─ download   ─┐            ─ scan + score ─┐          ─ caption ─┐            ─ train ─
   ─ provenance ─┤  images    ─ face-cluster ─┤ manifest ─ buckets ─┤  dataset   ─ LoRA  ─
   ─ verify ─────┴─────────►  ─ select ───────┴────────► (identity/ ─┴────────►  ───────►
                              ─ export ───────►          wardrobe/…)
```

## Standalone: moderation and dataset auditing

Everything curator records about an image is usable without the rest of the
suite. A scan never modifies the source folder; it produces a per-image record
(`--json`, `--csv`, or `GET /scan/{scan_id}`), and an export applies filters
to it, giving every excluded image a `keep_reason`. That record doubles as an
audit trail: you can show which images were removed and on what grounds.

| Check | What is recorded | Filter on it |
|---|---|---|
| Technical quality | sharpness, resolution, aspect ratio, compression artifacts | `--min-score`, hard-filter `reject_reason` |
| Near-duplicates | pHash groups, best representative kept | `--keep-similar` to keep the rest |
| People | face count, identity clusters, head pose | `--face-clusters`, `--pose` |
| Apparent age | MiVOLO age in years and FairFace age bracket, per face | `--min-age` / `--max-age`, `--age` |
| Demographics | FairFace race and gender with confidences, pooled per identity | `--race`, `--gender`, `--min-attr-conf` |
| Setting | Places365 scene top-k, indoor/outdoor | `--scene`, `--environment` |
| Clothing | Fashionpedia garments: type, colour, cut off at the frame edge | `--garment`, `--exclude-garment`, `--uncropped` |
| Nudity | NudeNet detections per body-part class, exposed score (needs a [NudeNet server](#nudity-detection-nudenet)) | `--nsfw-threshold`, `--nsfw-label` |
| Policy | SingGuard verdict and the rules that fired: sexual content, crime, gore and self-harm, … (needs a [SingGuard server](#policy-check-singguard)) | `--guard-unsafe`, `--guard-rules` |

A typical moderation pass is scan, review, act:

```bash
# 1. Scan and report only: nothing is copied, moved or deleted.
argus-curator scan /data/inbox --mivolo --scenes --garments --csv report.csv --json scan.json

# 2. Review report.csv: one row per image with every tag and its confidence.

# 3. Keep only images whose primary face reads as clearly adult. --min-score 0,
#    --include-rejected and --keep-similar stop quality and duplicates from
#    removing anything, so age is the only criterion. Images where no age could
#    be estimated (no face found, or too blurry to pass the hard filters that
#    face detection runs behind) are excluded as `face-attributes-missing`: the
#    filter fails closed rather than letting unknowns through.
argus-curator scan /data/inbox --mivolo --min-age 25 \
    --min-score 0 --include-rejected --keep-similar \
    --copy-to /data/approved --csv decisions.csv
```

`decisions.csv` then lists every image with `keep` and its `keep_reason`
(`face-age-out-of-range`, `face-attributes-missing`, `garment-excluded`, …),
which is the list to send for human review or to quarantine. The same scans and
filters are available over HTTP (`POST /scan/folder`, `GET /scan/{scan_id}`,
`POST /export`), so curator can sit in front of an upload or moderation queue.

**Limits — read before relying on it for moderation:**

- **Content checks need their servers, and both miss things.** NudeNet
  detects exposed body parts only, and misses stylised, partial or unusual
  poses. SingGuard judges a whole policy (violence, gore, self-harm, …) but
  gives a verdict without a confidence, so nothing is "near the threshold" to
  review. Set NSFW thresholds conservatively and have a person review flagged
  and borderline images.
- **Age and demographics are model estimates from appearance**, not facts
  about anyone. Age estimates are typically off by a few years, more so from
  the face alone (MiVOLO runs face-only here), so filter with a margin above
  any legal threshold (`--min-age 25` rather than 18) and send borderline
  images to a person.
- Only the **primary** face of each image is filtered on; other faces are
  recorded per face (`faces[].attributes`) but do not exclude the image.
- Scene and garment models make mistakes too; see the measured accuracy in
  [Scene tags](#scene-tags-places365) and [Garments](#garments-fashionpedia).

Curator is a filter and an audit trail to put in front of human review, not a
replacement for it.

## Why not FiftyOne / fastdup / Immich?

| | Generic CV curation (FiftyOne, fastdup) | Consumer galleries (Immich, PhotoPrism) | **Argus Curator** |
|---|---|---|---|
| Quality scoring | Generic | None | **Training-suitability** (target-aware sharpness/res/artifact + face-count fit) |
| Dedup | pHash | Basic | pHash, keeps best representative, **reports** the rest |
| Faces | Plugin | Clustering for browsing | Clustering **for dataset filtering** (export by identity) |
| Dataset export | Manual | None | Structure-preserving copy/symlink/move + **manifest** |
| Captioner handoff | None | None | One shared `TargetProfile` → argus-lens |
| Moderation / audit | Manual queries | None | Apparent-age, scene and garment filters, with a `keep_reason` for every exclusion |

It's LoRA-native, identity-aware, and caption-integrated — the reason the suite exists.

## The shared TargetProfile (the moat)

Both services speak the same taxonomy. This single schema is the contract:

```python
from argus_curator import TargetProfile

TargetProfile(
    target_style="photo",        # "photo" | "anime"
    target_backend="sdxl",       # "sdxl" | "flux-dev-1" | ...
    checkpoint=None,
    target_category="identity",  # identity | wardrobe | pose_composition | setting
)
```

The curator uses it to weight scoring and label exports; argus-lens inherits it verbatim.

## Installation

```bash
pip install argus-curator                 # engine only (Pillow, numpy, ImageHash, pydantic)
pip install "argus-curator[server,cli]"   # FastAPI server + CLI
pip install "argus-curator[faces]"        # + InsightFace identity clustering (CPU onnxruntime)
pip install "argus-curator[gpu]"          # + onnxruntime-gpu for GPU face detection
pip install "argus-curator[all]"          # everything
```

System libraries for the face stack (Ubuntu/Debian):

```bash
sudo apt install -y libgl1 libglib2.0-0 libgomp1
```

## Usage

### Python

```python
from argus_curator import scan_folder, TargetProfile, ScanConfig, FaceConfig

summary = scan_folder(
    "/data/images",
    profile=TargetProfile(target_category="identity"),
    cfg=ScanConfig(min_short_side=512, blur_threshold=100.0, cluster_distance=10),
    faces_cfg=FaceConfig(enabled=True, model="buffalo_l", cluster_eps=0.5),
)

print(summary.passed, "passed,", summary.duplicates, "near-dupes")
for fc in summary.face_clusters:
    print(fc.cluster_id, fc.size, "faces, rep:", fc.representative_rel_path)
```

### CLI

```bash
# Report only — see the score distribution before committing to a threshold
argus-curator scan /data/images --csv report.csv

# Identity curation with face clustering, copy 0.65+ keepers (structure preserved)
argus-curator scan /data/images \
    --target-category identity --faces --device cuda \
    --min-score 0.65 --copy-to /data/curated

# Export only specific identities, capped to a diverse 200
argus-curator scan /data/images --faces \
    --face-clusters face_1,face_2 --max-keep 200 --copy-to /data/curated

# Pick a pose-balanced subset (head-on + 3/4 only, drop side profiles)
argus-curator scan /data/images --faces \
    --pose frontal,three_quarter --copy-to /data/curated

# Predict attributes per face (FairFace: race/gender/age bracket; MiVOLO: age in years/gender),
# then keep adults up to 35 with a confident race prediction
argus-curator scan /data/images --fairface --mivolo \
    --race east_asian --min-attr-conf 0.6 --min-age 18 --max-age 35 --copy-to /data/curated

# Which detectors are available?
argus-curator detectors
```

### Face attributes (FairFace + MiVOLO)

Two optional models read each detected face; either switch implies `--faces`.

| | [FairFace](https://github.com/dchen236/FairFace) (`--fairface`) | [MiVOLO v2](https://huggingface.co/iitolstykh/mivolo_v2) (`--mivolo`) |
|---|---|---|
| Predicts | race (7), gender, age bracket | age in **years**, gender |
| Confidence | softmax probability per label | gender probability; age is a regression (none) |
| Input | 224 px chip aligned on the 5 face keypoints | 384 px face box, letterboxed; face-only |
| Weights | CC BY 4.0, Google Drive | Apache-2.0, Hugging Face |

Race labels: white, black, latino_hispanic, east_asian, southeast_asian,
indian, middle_eastern. Age brackets: 0-2, 3-9, 10-19, …, 60-69, 70+.

With both on, MiVOLO wins gender and age (it is far stronger at both) and
FairFace alone supplies race; the bracket is then derived from MiVOLO's years,
so `--age` filters work with either model. `gender_source` / `age_source` say
which model produced each value. Results land per face (`faces[].attributes`),
on the primary face (`primary_face_attributes`, also in the CSV report), and
pooled per identity cluster (`face_clusters[].attributes`): probabilities are
averaged and ages take the median over every member face, so one odd frame
cannot flip or drag an identity's label.

MiVOLO is built to read a face *and* a person crop; there is no person
detector here, so it runs face-only (a supported mode, slightly less accurate
than MiVOLO's published face+body numbers).

These are a model's guess from appearance, not facts about anyone; expect
errors on ambiguous faces, mixed heritage, heavy makeup or poor lighting, and
check `*_conf` before relying on a label.

**Filters** (all on the primary face): `--race`, `--gender`, `--age`
(brackets), `--min-age` / `--max-age` (years, MiVOLO). `--min-attr-conf` /
`min_attribute_conf` additionally requires a filtered label's confidence to
reach the floor (a MiVOLO age has none, so it is not held to it). Excluded
images say why in `keep_reason`: `face-<attr>-excluded`,
`face-<attr>-low-confidence`, `face-<attr>-missing`,
`face-age-out-of-range`, `face-age-years-missing`, or
`face-attributes-missing`.

**Models** are not on PyPI; each is converted to ONNX once, after which
inference runs on the `[faces]` onnxruntime stack with no torch:

```bash
# MiVOLO: downloads the pinned checkpoint (commit + sha256 checked)
pip install "argus-curator[mivolo-convert]"
argus-curator mivolo-convert          # -> ~/.cache/argus_curator/models/mivolo_v2_d1_384.onnx

# FairFace: download fairface_alldata_20191111.pt from the FairFace repo first
pip install "argus-curator[fairface-convert]"
argus-curator fairface-convert fairface_alldata_20191111.pt
                                      # -> ~/.cache/argus_curator/models/fairface_res34_race7.onnx
```

Point elsewhere with `--fairface-model` / `--mivolo-model` or
`ARGUS_CURATOR_FAIRFACE_MODEL` / `ARGUS_CURATOR_MIVOLO_MODEL`. Model paths are
server configuration, never part of a request; a server without a requested
model refuses the scan with 400 before scoring anything, and `GET /detectors`
reports `fairface` / `mivolo`.

### Scene tags (Places365)

`--scenes` (or `"scenes": {"enabled": true}` on the scan request) runs a
[Places365](https://github.com/CSAILVision/places365)-CNN (Zhou et al., TPAMI
2017) on every passing image and records its top-k scene categories (365 of
them: `beach`, `bedroom`, `gas_station`, `forest_path`, …) with their softmax
probabilities, plus an indoor/outdoor call. `environment_conf` is the
probability mass on categories of that environment, not upstream's top-10
vote. Results land on `results[].scene` and in the CSV report.

Top-1 accuracy on Places365 is about 55% (top-5 about 85%), and neighbouring
categories overlap (`lagoon` / `beach` / `coast`), which is why filters match
any of the recorded top-k rather than only the first:

- `--scene beach,coast` / `scenes`: keep images with one of these among their
  top-k at probability ≥ `--min-scene-conf` / `min_scene_conf` (default 0.1, so
  a label lingering in the tail at 3% does not count). Unknown labels are
  rejected with 422 rather than silently matching nothing.
- `--environment outdoor` / `scene_environments`: indoor or outdoor, held to the
  same floor.
- `--diversify-scenes` / `diversify_scenes` (with `--max-keep`): the diversity
  cap also spreads the selection across backdrops, so an identity set is not
  two hundred bathroom selfies.

Excluded images say why: `scene-excluded`, `scene-low-confidence`,
`scene-environment-excluded`, `scene-environment-low-confidence`, or
`scene-missing`.

The weights are converted once to ONNX, like the face models:

```bash
pip install "argus-curator[places365-convert]"   # torch, only for this step
argus-curator places365-convert                  # resnet18 (default) or --arch resnet50
# -> ~/.cache/argus_curator/models/places365_resnet18.onnx
```

Upstream serves the checkpoints over plain HTTP, so each is pinned by sha256.
ResNet-50 is twice the size for about one point of top-1 (55.6% vs 54.5%).
Point elsewhere with `--places365-model` / `ARGUS_CURATOR_PLACES365_MODEL`.

### Garments (Fashionpedia)

`--garments` (or `"garments": {"enabled": true}`) runs a YOLOS-small detector
fine-tuned on [Fashionpedia](https://fashionpedia.github.io/home/)
([`valentinafevu/yolos-fashionpedia`](https://huggingface.co/valentinafevu/yolos-fashionpedia),
MIT; dataset CC BY 4.0) on every passing image. Each detection records its
garment type (27 classes: `dress`, `pants`, `shoe`, `bag_wallet`, `hat`, …),
probability, box, whether the box touches the frame edge (likely cut off), and
a dominant colour from the box's centre. Results land on `results[].garments`
(`[]` = ran, found nothing) and in the CSV as `label:conf:#colour`, with `*`
marking a cut-off garment.

Measured on 300 Fashionpedia validation images (CPU, ~0.3 s/image):

| | |
|---|---|
| Garments, AP50 / mAP | 0.327 / 0.193 |
| Strong (P / R at 0.5) | shoe 0.96 / 0.97 · pants 0.83 / 0.87 · dress 0.75 / 0.91 · hat 0.86 / 0.78 · jacket 0.86 / 0.58 |
| Weak | skirt 0.38 / 0.31 · coat 0.43 / 0.15 · shirt/blouse recall 0.31 |
| "Is garment X in the image?", all classes | P 0.64 / R 0.76 at 0.3 · **0.77 / 0.66 at 0.5** · 0.86 / 0.56 at 0.7 |

Fashionpedia's 19 garment *parts* (sleeve, pocket, zipper, …) scored AP50
0.105 and are not recorded; neither are its 294 fine-grained attributes, which
no published checkpoint predicts. Detections are recorded from 0.3 so reviewers
see the doubtful ones; filters count them from `--min-garment-conf` (0.5). A
floor below what the scan recorded cannot be honoured, so the server refuses it
with 400; the CLI instead records from the lower floor in the first place.

- `--garment dress,skirt` / `garments`: wearing at least one of these.
- `--exclude-garment bag_wallet,umbrella` / `exclude_garments`: none of these.
  Exclusion wants recall, so consider a lower `--min-garment-conf` for it.
- `--uncropped` / `garments_uncropped`: the matching garment is not cut off.
- `--diversify-outfits` / `diversify_outfits` (with `--max-keep`): the
  diversity cap also spreads the selection across outfits (garment types and
  their colours), so an identity LoRA does not learn one outfit as part of the
  person.

Exclusions say why: `garment-not-found`, `garment-low-confidence` (present,
below the floor), `garment-excluded`, `garment-cropped`, `garments-missing`.

```bash
pip install "argus-curator[garments-convert]"   # torch + transformers, only for this step
argus-curator garments-convert                  # pinned commit + sha256
# -> ~/.cache/argus_curator/models/yolos_fashionpedia.onnx
```

Point elsewhere with `--garments-model` / `ARGUS_CURATOR_GARMENTS_MODEL`.

### Nudity detection (NudeNet)

Curator can check every scanned image with [NudeNet](https://github.com/notAI-tech/NudeNet),
a detector for 18 body-part classes (`FEMALE_BREAST_EXPOSED`, `BUTTOCKS_COVERED`,
`FACE_FEMALE`, …). It talks to a **NudeNet server over HTTP**; curator never
imports NudeNet, so it gains no dependency and stays MIT. NudeNet itself is
AGPL-3.0, so running its server is a choice for whoever deploys it.

Configure the server in the environment (or `.env`), never per request:

```bash
ARGUS_CURATOR_NUDENET_URL=http://192.168.1.50:8080   # its origin; /infer is appended
ARGUS_CURATOR_NUDENET_TIMEOUT=30                     # seconds per image (default 30)
ARGUS_CURATOR_NUDENET_WORKERS=4                      # concurrent requests (default 4)
```

Any host running `ghcr.io/notai-tech/nudenet` works, including one elsewhere on
your LAN. `docker compose up` starts one alongside curator (pinned by digest)
and uses it unless `.env` says otherwise.

**It is opt-out.** When a URL is configured, every scan checks its images,
including ones that failed the quality filters, unless the scan opts out
(`--no-nsfw`, or `"nsfw": {"enabled": false}`). `--nsfw` / `"enabled": true`
makes the check mandatory: the scan is refused if no server is configured. Set
the URL empty to switch the check off for a deployment. The scan summary
records whether it ran (`nsfw_config.enabled`). Images are sent to that server,
so keep it on infrastructure you control.

Each image records every detection, the highest score per class
(`nsfw.labels`), and an **exposed score**: the highest score among the five
exposed-nudity classes (`BUTTOCKS_EXPOSED`, `FEMALE_BREAST_EXPOSED`,
`FEMALE_GENITALIA_EXPOSED`, `MALE_GENITALIA_EXPOSED`, `ANUS_EXPOSED`). Faces and
covered body parts are detected too but never count towards it. The CSV
report has `nsfw_exposed_score` and `nsfw_labels`.

Filters drop an image when a score is **at or above** its threshold:

```bash
# Total: drop anything whose exposed score is 0.5 or more
argus-curator scan /data/inbox --nsfw-threshold 0.5 --copy-to /data/clean

# Per class, alongside or instead of the total
argus-curator scan /data/inbox \
    --nsfw-label FEMALE_BREAST_EXPOSED=0.4,MALE_GENITALIA_EXPOSED=0.3,BELLY_EXPOSED=0.8 \
    --copy-to /data/clean
```

Over HTTP these are `nsfw_threshold`, `nsfw_label_thresholds` (`{"CLASS": score}`)
and `nsfw_exposed_labels` (which classes the total is taken over). The server
reports nothing under 0.2, so thresholds must be 0.2–1. An image that was never
checked, or whose check failed, is **excluded** whenever an NSFW filter is set
(`nsfw-unchecked`): the filter fails closed. Other reasons are `nsfw-exposed`
and `nsfw-<class>` (e.g. `nsfw-belly-exposed`). Exporting with NSFW thresholds
from a scan that did not run NudeNet is refused (400) rather than excluding
everything. `GET /detectors` reports `nudenet` when a URL is configured.

### Policy check (SingGuard)

Where NudeNet scores nudity per body part, a [SingGuard](https://huggingface.co/inclusionAI/SingGuard-4b-GGUF)
server judges each image against a whole moderation policy and answers with a
**verdict and the rule that fired**. Its default rules:

| Rule | Covers |
| --- | --- |
| A. Sexual Content Risk | sexual content |
| B. Real-World Crimes & Public Safety | violent crime, weapons, public-safety threats |
| C. Unethical Behavior | hate, harassment, self-harm, gore and shock content |
| D. Cybersecurity | data leaks, hacking, surveillance abuse |
| E. Agent Safety | attempts to expose model safeguards |
| F. Politically Sensitive | political advocacy, unrest, attacks on political figures |
| G. Animal Abuse | cruelty to animals |

It gives **no scores**, so it complements NudeNet rather than replacing it. It
runs a vision language model: about a second per image in `fast` mode, several
in `fast-slow` (which also writes why a rule fired). Configure it like NudeNet:

```bash
ARGUS_CURATOR_SINGGUARD_URL=http://192.168.1.198:9150   # its origin; /api/guard/check is appended
ARGUS_CURATOR_SINGGUARD_THINKING=fast                    # or fast-slow, for explanations
ARGUS_CURATOR_SINGGUARD_POLICY_FILE=/etc/curator/policy.txt  # optional: replaces the default rules
ARGUS_CURATOR_SINGGUARD_TIMEOUT=90                       # seconds per image (default 90)
ARGUS_CURATOR_SINGGUARD_WORKERS=2                        # concurrent requests (default 2)
```

It is opt-out in the same way (`--no-guard`, `"guard": {"enabled": false}`;
`--guard` makes it mandatory). The default rule A flags any sexual content,
adult nudity included. For an adult dataset, pick the built-in **`adult`**
policy per scan (`--guard-policy adult`, `"guard": {"policy": "adult"}`). Its
rule A covers only apparent minors, non-consent and sexual violence, and its
Safe rule names consensual adult nudity. Rules B to G cover the same areas
under the same letters, so `--guard-rules B,C` means the same thing under
both. They are written out as text, though, rather than left to SingGuard's
built-in wording, so verdicts can differ: a group photo of a poker game that
the default rule B flags passes under `adult`. `default` is the policy file when one is set, else SingGuard's own
rules. A request names a policy and never sends policy text. `GET /detectors`
lists the names in `singguard_policies`.

`adult-granular` (a letter per hard case: A minors, H non-consent, I sexual
violence) and `adult-cautious` (doubt about age or consent counts as rule A)
are under evaluation against `adult`; don't build on them yet.

To compare policies on your own images, judge one scan under several:
`"guard": {"policy": "adult", "variants": ["adult-granular", "adult-cautious"]}`
(CLI `--guard-variants`). `guard` stays the official verdict, and
`guard_variants` holds every policy's verdict by name, the official one
included. A missing name means that check failed. Each variant is one more
SingGuard pass per image. The filters read the official verdict unless
`guard_policies` lists others (CLI `--guard-policies`). An image is then
dropped if any listed verdict fails, or is missing, with the policy named in
the keep reason (`guard-unsafe:adult-granular`). The CSV report adds a
`guard_variants` column.
`scripts/guard_eval.py --url <singguard>` scores every policy on the text
probes in `eval/guard_probes.jsonl` and on an optional folder of images
sorted by expected verdict.

Each image records `guard.safe`, the rules that fired, the explanation, the
model and the policy: `default`, `adult`, or `custom-<sha256 prefix>` of the
file. The CSV report has `guard_safe` and `guard_rules`.

```bash
# Drop anything judged unsafe
argus-curator scan /data/inbox --guard-unsafe --copy-to /data/clean

# Drop only crime and unethical-behaviour verdicts, by letter or full name
argus-curator scan /data/inbox --guard-rules B,C --copy-to /data/clean
```

Over HTTP these are `guard_unsafe` and `guard_rules`. As with NudeNet, an
unchecked image is excluded whenever a SingGuard filter is set
(`guard-unchecked`); other reasons are `guard-unsafe` and `guard-rule-<rule>`
(e.g. `guard-rule-c`), or `guard-rule-unknown` for an unsafe verdict that names
no rule. `GET /detectors` reports `singguard`.

### HTTP server (:8101, peer to argus-lens)

```bash
pip install "argus-curator[server,faces]"
argus-curator serve --cors --port 8101 --source-root /data/images --export-root /data/out
```

Request-supplied paths are treated as untrusted: scan folders resolve under
`--source-root` (env: `ARGUS_CURATOR_SCAN_ROOT` / `CURATOR_SOURCE_PATH`) and
export destinations under `--export-root` (`ARGUS_CURATOR_EXPORT_ROOT` /
`CURATOR_EXPORT_PATH`). Traversal escapes are refused, and scan/export refuse
outright when their root is not configured. Both roots are needed to export:
the destination resolves under the export root, and the scan being exported
must itself lie under the source root — a `scan_id` is data from a store the
CLI also writes to, so its folder is re-checked rather than trusted (the same
check applies to `/thumb?scan_id=`). The destructive `mode: "move"` is
rejected unless the server is started with `--allow-move`
(`CURATOR_ALLOW_MOVE=1`). `--cors` allows only the localhost dev frontend
(:3000); allow-list other origins with `--cors-origin <origin>`
(`CURATOR_CORS_ORIGINS`), or use `--cors-any` for a credential-less wildcard
on public demos. A literal `*` in the allow-list is treated as `--cors-any`
(wildcard without credentials), never as origin reflection.

CORS is not a write boundary — `POST /upload` sends `multipart/form-data`, which
browsers deliver with no preflight — so state-changing requests are additionally
gated on `Origin`: it must be absent (curl, the CLI, server-to-server),
same-origin, or on the allow-list, else 403. `--cors-any` therefore permits
anonymous reads from anywhere but no cross-site writes; name an origin with
`--cors-origin` to let it write.

| Route | Description |
|---|---|
| `GET  /health` | Liveness |
| `GET  /detectors` | `{ torch, cuda, clip, insightface, onnxruntime, fairface, mivolo, places365, garments, nudenet }` |
| `GET  /folders?path=<rel>` | Browse Docker-mounted folders under the source root (for the UI picker) |
| `POST /scan/folder` | Scan + score + dedup + face-cluster → `ScanSummary` |
| `GET  /scan/{scan_id}` | Cached summary, paginated via `?offset=&limit=` |
| `GET  /thumb?path=<rel>&scan_id=<id>` | `image/webp` thumbnail from the mount |
| `GET  /image?path=<rel>&scan_id=<id>` | Original image bytes, same containment as `/thumb` |
| `POST /scan/{scan_id}/select` | Dry-run of the export filter → `{total, kept, rows: [{rel_path, abs_path, keep_reason}]}`. Writes nothing |
| `POST /upload` | Multipart image upload (`files` + `folder`) into a folder under the source root |
| `POST /export` | Structure-preserving transfer + manifest → `ExportResult` |
| `POST /scan/folder/stream` | Same as `/scan/folder`, streaming live progress over SSE |
| `POST /export/stream` | Same as `/export`, streaming per-file transfer progress over SSE |

The `*/stream` variants emit `event: progress` frames (`{phase, done, total}`)
while the work runs, then a single `event: complete` frame carrying the same
payload the non-streaming endpoint returns (or `event: error`).

`POST /scan/folder` body:

```jsonc
{
  "folder": "shoots/2026-07",         // relative to the server's --source-root ("" = the root)
  "target_profile": { "target_style": "photo", "target_category": "identity" },
  "config": { "min_short_side": 512, "max_aspect_ratio": 3.0, "blur_threshold": 100.0,
              "cluster_distance": 10, "max_workers": 4 },
  "faces":  { "enabled": true, "model": "buffalo_l", "min_det_score": 0.5, "cluster_eps": 0.5 }
}
```

`POST /export` body:

```jsonc
{
  "scan_id": "...",                 // or inline "selection": ["rel_path", ...]
  "dest": "training-set-v1",        // relative to the server's --export-root
  "mode": "copy",                   // "copy" | "symlink" | "move" (move needs --allow-move)
  "preserve_structure": true,
  "min_score": 0.6, "include_rejected": false, "keep_similar": false,
  "face_clusters": ["face_2"],      // optional: export only these identities
  "face_poses": ["frontal", "three_quarter"],  // optional: export only these head poses
  "face_races": null, "face_genders": null,    // optional: attribute filters on the primary face
  "face_ages": null,                // FairFace brackets ("20-29", ...)
  "min_age_years": 18, "max_age_years": 35,     // optional: MiVOLO age range
  "min_attribute_conf": 0.6,        // filtered attributes must be at least this confident
  "scenes": ["beach", "coast"],     // optional: Places365 scenes among the image's top-k
  "scene_environments": ["outdoor"],
  "min_scene_conf": 0.1, "diversify_scenes": false,
  "garments": ["dress"], "exclude_garments": ["bag_wallet"],  // optional: Fashionpedia
  "min_garment_conf": 0.5, "garments_uncropped": false, "diversify_outfits": true,
  "write_manifest": true,
  "caption_url": null               // optional: POST manifest to argus-lens
}
```

### Docker

```bash
docker compose up --build
```

Bind-mounts a dataset into `/data/images`, an output dir into `/data/out`, and
persists the scan cache + InsightFace model downloads across rebuilds. A
converted FairFace / MiVOLO model under `~/.cache/argus_curator/models/` is
picked up through the same cache mount.

## Handoff to argus-lens

Export writes a JSONL manifest (one row per **exported** image — rows exist
only for files whose transfer actually succeeded):

```jsonc
{ "manifest_version": "2.1", "rel_path": "...", "abs_path": "...",
  "exported_path": "...", "exported_abs_path": "...", "target_profile": { ... },
  "primary_face_cluster": "face_2", "primary_face_pose": "three_quarter",
  "score": 0.87, "similar_group": 3 }
```

The row shape is published in the wire schema as `ManifestRow`.
`exported_path` is the path actually written under the export root — consumers
must use it rather than re-deriving a location from `rel_path`. Flattened
exports (`preserve_structure: false`) de-collide duplicate basenames with a
short hash suffix, so the two can differ; collisions are detected
case-insensitively (and Unicode-normalised) so the result is safe on
case-insensitive destination filesystems, and the export fails loudly rather
than overwrite if a unique name cannot be generated. The same
`rel_path -> exported_path` mapping is returned as `exported_paths` on the
export response, so it is available even with `write_manifest: false`. Note
that each export call plans in isolation: re-exporting into the same
destination overwrites files (and rewrites the manifest).

`exported_abs_path` is that file's absolute location, and `exported_abs_paths`
the matching mapping on the export response. Prefer them: joining the export
root onto `exported_path` means reconstructing the server's layout rule
somewhere that does not own it, and the de-collision suffix above is not
reproducible by a client anyway. A manifest is also frequently read somewhere
other than the root it was written into — argus-lens receives it as an upload —
and nothing in a row names that root.

`abs_path` is where the image can be **read**, which is not always where it came
from: `mode: "move"` deletes the source during transfer, so for a moved export
`abs_path` equals `exported_abs_path`. Use `rel_path` for the original location
within the scan.

`GET /health` reports `manifest_version` alongside the package `version`, so a
consumer can check the contract it is about to speak — and refuse an unknown
major — rather than inferring it from which fields a response happens to carry.
The two versions move independently.

argus-lens batch-captions this manifest — categories are already shared, so no
remapping. Set `caption_url` on the export request to POST it straight to lens
for a one-click curate→caption run.

## How scoring works (per image)

1. **Hard filters** — min short side, max aspect ratio, blur (Laplacian-edge variance floor).
2. **Composite score** — target-aware weighted blend of sharpness / resolution / artifact, plus a small composition bonus that depends on `target_category` (e.g. identity rewards a single centred face; setting rewards wide framing).
3. **Face-count fit** — with `[faces]`, identity targets penalise 0 or 2+ faces; other categories are progressively more tolerant.
4. **Near-duplicate dedup** — pHash clustering keeps the highest-scoring representative and *reports* the rest (never silently dropped).
5. **Selection (at export)** — score threshold + optional diversity cap (`max_keep` / `diversity_weight`) + optional face-cluster filter. Every excluded image carries a `keep_reason`.

## State

Scans are cached on disk (keyed by `scan_id`, default `~/.cache/argus_curator/scans`,
override with `CURATOR_CACHE_DIR`). This is what makes paginated `GET /scan/{id}`
and export-by-id work without recomputing.

## Related projects

- [**argus-quarry**](https://github.com/smk762/argus-quarry) — provenance-first acquisition of public-domain / CC0 portraits (the suite's input stage).
- [**argus-lens**](https://github.com/smk762/argus-lens) — intent-aware, multi-model captioning (consumes the manifest this package exports).
- [**argus-studio**](https://github.com/smk762/argus-studio) — the suite's Next.js web UI (its `/curate` view drives this server).

## License

MIT — matches the rest of the Argus suite.
