Metadata-Version: 2.4
Name: caption-audit
Version: 1.1.1
Summary: Find the tokens that are fused with your LoRA trigger word, before you spend a training run.
Project-URL: Homepage, https://github.com/0xBeycan/caption-audit
Project-URL: Repository, https://github.com/0xBeycan/caption-audit
Project-URL: Issues, https://github.com/0xBeycan/caption-audit/issues
Author-email: beycan <halilbeycan0@gmail.com>
License: MIT
License-File: LICENSE
Keywords: captions,dataset,flux,lora,machine-learning,qwen-image,stable-diffusion,training,z-image
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Multimedia :: Graphics
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# caption-audit

[![CI](https://github.com/0xBeycan/caption-audit/actions/workflows/ci.yml/badge.svg)](https://github.com/0xBeycan/caption-audit/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/caption-audit.svg)](https://pypi.org/project/caption-audit/)
[![Python](https://img.shields.io/pypi/pyversions/caption-audit.svg)](https://pypi.org/project/caption-audit/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

**Find the tokens that are fused with your LoRA trigger word — before you spend a training run.**

```
pip install caption-audit
caption-audit ./dataset --trigger sks1
```

Works on any diffusion LoRA dataset: SD 1.5, SDXL, Flux, Qwen-Image, Z-Image, Wan, whatever comes next.
It reads captions, not model weights, so it does not care which trainer or base model you use.
Both caption dialects are handled — comma-separated Danbooru/booru tag lists **and** natural-language
sentences — and they are analysed side by side in the same run.

---

## The problem it detects

A token that appears in nearly every caption stops functioning as a describable attribute.

The model cannot tell it apart from the trigger word, so it bakes the concept into the identity
itself. You can no longer prompt it in, and you can no longer prompt it out.

That is the actual cause of symptoms people usually blame on the optimiser:

- every generation comes out with the same colour cast
- the background texture is forced no matter what you prompt
- lighting is locked to one look
- the character always wears the same thing

**It is a caption distribution problem, not a learning rate problem.** Lowering the LR will not fix
it. Training longer will not fix it. Nothing downstream of the dataset will fix it.

caption-audit measures the distribution and tells you which tokens are already fused, before you burn the
GPU hours.

---

## Install

```bash
pip install caption-audit          # pip
uv tool install caption-audit      # uv, installs the CLI globally
uvx caption-audit --trigger sks1   # uv, run without installing
```

Python 3.9+. **Zero dependencies** — standard library only.

---

## Quick start

```bash
# audit the current directory, guess the trigger
cd my_lora_dataset && caption-audit

# the normal case
caption-audit ./dataset --trigger sks1 --class-word woman

# captions and images in separate folders
caption-audit --dir captions --images-dir images --trigger sks1

# an attribute you WANT fused into the character (see --fuse below)
caption-audit ./dataset --trigger sks1 --fuse "red scarf"

# A/B two captioning styles over the same images
caption-audit --compare captions_tags captions_prose --trigger sks1

# machine-readable, for diffing between runs
caption-audit ./dataset --trigger sks1 --format json --out audit.json
```

Use it as a pre-training gate. Exit code is `1` if anything is CRITICAL:

```bash
caption-audit ./dataset --trigger sks1 && python train.py
```

---

## What it checks

### 1. Pairing integrity
Images with no caption file, captions with no image, empty captions, files that are not valid UTF-8,
stray byte-order marks. If your images live elsewhere, point `--images-dir` at them; with no images
present at all caption-audit runs in caption-only mode instead of screaming about it.

### 2. Trigger consistency
Coverage (is the trigger in 100% of captions?), position distribution (is it consistently first?),
and near-miss spelling variants found by edit distance — the `sksperson` / `sksperso` typo that
silently removes an image from your concept.

### 3. Document frequency — the primary check
For every word, 2-word phrase, 3-word phrase and whole comma segment, caption-audit computes **document
frequency**: in how many captions the term appears *at least once*.

Not raw occurrence count. A word repeated five times inside a single caption is not the same risk as
a word present in every caption, and raw counts cannot tell those apart.

| document frequency | severity | meaning |
|---|---|---|
| >= 85% | `CRITICAL` | fused with the trigger, unpromptable at inference |
| 60–85% | `WARNING` | strong bias, will bleed into unrelated prompts |
| 35–60% | `INFO` | worth watching |
| — | `INTENDED` | you declared it with `--fuse`; excluded from the exit code |
| — | `EXPECTED` | the trigger or a `--class-word`; supposed to be everywhere |

Thresholds are configurable with `--critical-threshold`, `--warn-threshold`, `--info-threshold`.

### 4. Trigger co-occurrence
For every flagged token, `P(token | trigger present)`. Tokens sitting at **1.00** are the dangerous
ones: they never appear independently of the trigger, so the loss has no example anywhere in the
dataset that separates the two concepts.

### 5. Caption redundancy
Byte-identical captions, and near-duplicates by token-set Jaccard similarity (default `>= 0.9`).
Identical captions on visually different images give the model no signal to disambiguate them, so it
averages them.

### 6. Distribution health
Caption length min/median/max/stdev with outlier detection, plus vocabulary diversity —
type-token ratio and hapax count. See [Reading the numbers](#reading-the-numbers).

---

## The interpretation caption-audit refuses to guess

Every CRITICAL and WARNING term has **two possible causes, and they need opposite fixes**:

**(a) Caption hygiene** — the concept genuinely varies across your images, but every caption mentions
it anyway. The variation exists in pixels and is invisible to the loss because the text never
distinguishes it.
→ *Fix: delete the word from the captions where it is not the point.*

**(b) Dataset composition** — the concept really is present in every image. Deleting the word changes
nothing; the model still learns it from pixels, it just loses the handle you would have used to name
it.
→ *Fix: shoot or collect contrast data.*

**caption-audit reads captions. It cannot see your images. So it will not guess.** For every flagged term
it prints the question you have to answer yourself, and the exact files to go and look at:

```
CRITICAL  red scarf           df 100%  P(t|trigger) 1.00  [bigram]
    - P(term | trigger) = 1.00 - this term never appears independently of the
      trigger. The model has zero signal to separate the two.
    - At 100% document frequency this is fused with the trigger and will be
      unpromptable at inference.
    ? Is 'red scarf' actually visually present in all 35 of those images?
      If yes -> dataset problem (b). If no -> caption problem (a).
    look at: img_001, img_002, img_004, img_005, ... (+31 more)
```

Doing the wrong one of those two fixes wastes an entire run. That is why it asks instead of telling.

---

## `--fuse`: when fusion is the goal

Sometimes you *want* an attribute welded to the trigger. A character's signature scarf is part of who they are;
a product's logo is not optional. Declaring it says so explicitly:

```bash
caption-audit ./dataset --trigger sks1 --fuse "red scarf"
```

The phrase and all of its fragments (`red`, `scarf`, `bright red silk scarf`) move to
`INTENDED`, stay visible in the table, and stop affecting the exit code — so the gate now enforces
*your* intent rather than a generic default.

It still reminds you what you signed up for: once trained, you will not be able to prompt that
attribute away or vary it.

---

## `--compare`: A/B your captioning

Same images, different caption sets. The only thing that moves between columns is the caption
distribution:

```bash
caption-audit --compare tags/ prose/ prose-no-hair/ --trigger sks1
```

```
                              tags             prose   prose-no-hair
captions                        35                35              35
trigger coverage              100%              100%            100%
trigger first token           100%              100%            100%
type-token ratio             0.284             0.265           0.310
CRITICAL / WARNING           6 / 0             4 / 5           0 / 0

TERM                          tags             prose   prose-no-hair
red scarf                    100%!             100%!               -
blurred                       89%!               43%               -
camera                           -              63%*             54%
```

`!` = CRITICAL, `*` = WARNING, `-` = below the info threshold in that set.

Fewest findings does not automatically mean best LoRA — it means the most spread caption
distribution. caption-audit says so in the verdict rather than letting you read it as a score.

---

## Reading the numbers

**Document frequency (DF)** — the fraction of captions containing a term at least once. This is the
whole idea. A word in 100% of captions carries no information: it cannot explain any difference
between two images, because it is identical in both. Information lives in variation.

**P(token | trigger)** — among the captions that contain the trigger, the fraction that also contain
this token. At `1.00` the two are inseparable in the data.

**Type-token ratio (TTR)** — unique words divided by total words. 1000 words of caption using 300
distinct ones gives `0.30`.

- **high TTR** (~0.5+) → you describe each image in its own words. The model sees many ways of
  phrasing things and generalises to phrasings it has not seen.
- **low TTR** (~0.2) → you reused the same sentence skeleton everywhere. The model over-fits the
  phrasing, and paraphrases at inference land off-distribution. Prompts get brittle: the exact
  wording from your captions works, anything else degrades.

There is no universal target — it depends on caption length and dataset size — but a big gap between
two caption sets over the *same images* is meaningful, which is what `--compare` is for.

**Hapax count** — words appearing exactly once in the whole dataset. High hapax with high TTR is
healthy specificity. High hapax with *low* TTR usually means a rigid template plus a few unique
nouns bolted on.

**N-grams and the segment view** — caption-audit analyses single words, pairs, triples, and whole comma
segments separately, because a phrase is a stronger signal than any of its words. `soft paper
texture` locked at 93% matters more than `soft` at 93%. N-grams never cross a comma, so tag lists
do not produce phantom phrases like `hair wearing`.

**Subsumption** — when `soft`, `paper`, `texture`, `soft paper` and `soft paper texture` all sit at
the same DF, that is one finding seen at five resolutions, not five findings. caption-audit attributes it
to the longest phrase and marks the fragments `(part of '...')`. They stay in the table; they do not
get their own interpretation block or inflate the count.

**Function words** are counted but kept out of the flag list, since `a`, `the` and `with` being
everywhere tells you nothing. `--no-stopwords` turns the filter off.

---

## Options

```
input
  DIR                       directory to audit (default: current directory)
  --dir DIR                 same thing, explicit
  --images-dir DIR          where the images live, if kept apart from captions
  --recursive               descend into subdirectories
  --compare DIR [DIR ...]   two or more caption sets, side by side

what the trigger is supposed to own
  --trigger TRIGGER         the trigger word; inferred and marked [INFERRED] if omitted
  --class-word WORD         e.g. 'woman', 'car'. Repeatable. Shown as EXPECTED, never flagged
  --fuse TERM               an attribute you WANT fused. Repeatable. Shown as INTENDED,
                            excluded from the exit code

thresholds
  --critical-threshold F    default 0.85
  --warn-threshold F        default 0.60
  --info-threshold F        default 0.35, below this nothing is reported
  --ngram-max N             longest phrase analysed, default 3
  --jaccard F               near-duplicate threshold, default 0.9
  --no-stopwords            stop filtering function words out of the flag list

output
  --format {term,md,json}   term (default), markdown, or JSON for diffing runs
  --out FILE                write to a file instead of stdout
  --list-files N            name up to N caption files per flagged term, default 10, 0 to hide
  --max-rows N              table rows, default 60
  --max-interpret N         terms expanded with full interpretation, default 12
  --no-color                disable ANSI colour (also honours NO_COLOR)
```

Supported image extensions: `.png` `.jpg` `.jpeg` `.webp`. Captions are `.txt` sidecars sharing the
image's basename.

**Exit codes:** `0` clean, `1` at least one CRITICAL finding, `2` bad input.

---

## Use in CI

```yaml
- name: Audit caption distribution
  run: |
    pip install caption-audit
    caption-audit dataset/ --trigger sks1 --fuse "red scarf" --format md --out audit.md
```

The run fails on CRITICAL findings, and `audit.md` is a readable artifact. With `--format json` you
can diff two runs and see exactly which term moved.

---

## Library use

```python
from caption_audit.loader import scan_dir
from caption_audit.checks import check_document_frequency, check_distribution

captions, images, orphans = scan_dir("dataset/")
stats = check_distribution(captions)
print(stats["type_token_ratio"], stats["hapax"])
```

Every check returns plain dicts. `caption_audit.cli.analyze(directory, args)` gives you the whole report
object, which is exactly what `--format json` serialises.

---

## Development

```bash
git clone https://github.com/0xBeycan/caption-audit
cd caption-audit

python -m unittest discover -s tests   # no dependencies at all
uv run --extra dev pytest -q           # same 17 tests, nicer output
uv run --extra dev ruff check .
uv build
```

CI runs the suite on Python 3.9–3.14, lints, then installs the built wheel into
a clean environment and re-runs the tests plus an end-to-end exit-code check
against the console script.

The test suite builds a synthetic fixture containing one of each defect — a fused adjective phrase at
100%, a duplicate caption pair, a near-duplicate pair, an orphan image, an empty caption, a non-UTF8
file and a trigger typo — and asserts that every check fires.

## License

MIT
