Metadata-Version: 2.4
Name: curu-caption
Version: 3.0.3
Requires-Dist: avdm>=0.2.2,<0.3
Requires-Dist: curu-vision>=0.8.1,<0.9
Requires-Dist: av>=17
Requires-Dist: click>=8
Requires-Dist: curu-infra>=0.66,<0.73
Requires-Dist: httpx>=0.27
Requires-Dist: huggingface-hub>=1.0
Requires-Dist: mutagen>=1.48.1
Requires-Dist: numpy>=2.0,<2.5
Requires-Dist: onnxruntime>=1.31,<1.32
Requires-Dist: pillow>=10
Requires-Dist: pydantic-ai>=2.0
Requires-Dist: pypng>=0.20220715.0
Requires-Dist: pyyaml>=6
Requires-Dist: rich>=13
Requires-Dist: scikit-learn>=1.3
Requires-Dist: torch>=2.2
Requires-Dist: torchvision
Requires-Dist: transformers>=5.19
Requires-Dist: watchdog>=4.0
Requires-Dist: librosa>=0.10 ; extra == 'audio'
Requires-Dist: soundfile>=0.12 ; extra == 'audio'
Requires-Dist: onnxruntime-gpu>=1.31,<1.32 ; sys_platform != 'darwin' and extra == 'gpu'
Provides-Extra: audio
Provides-Extra: gpu
Provides-Extra: remote
License-File: LICENSE
Summary: Local-first pipeline: media in, AI-Toolkit-ready LoRA training datasets out
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://github.com/darth-veitcher/curu-caption/blob/main/README.md
Project-URL: Repository, https://github.com/darth-veitcher/curu-caption

# curu-caption

## What is this

curu-caption makes LoRA training sets from videos and photographs, on your own computer. It
removes duplicate and blank frames, finds the items that you ask for, and captions only those
items. The result is media and `.txt` caption pairs, with the configuration that
[AI-Toolkit](https://github.com/ostris/ai-toolkit) trains from. It is for a person who trains
LoRAs and does not want to extract, sort and caption frames by hand. Each step runs locally by
default, and no step needs a cloud account.

![curu-caption --help](docs/assets/cli-help.svg)

## What you can do

| Task | Command | Section |
|---|---|---|
| Make a corpus from a video or a folder of videos | `curu-caption ingest --source episode.mp4 --out corpus/` | [Make a corpus from a video](#make-a-corpus-from-a-video) |
| Make a corpus from a folder of photographs | `curu-caption ingest --source photos/ --out photo-corpus/` | [Make a corpus from photographs](#make-a-corpus-from-photographs) |
| Index a corpus again | `curu-caption index --corpus corpus/` | [Index a corpus again](#index-a-corpus-again) |
| Find the items that match a description | `curu-caption cut --corpus corpus/ --out training/ --want "a wide shot" --dry-run` | [Find items by meaning](#find-items-by-meaning) |
| Make a training set for one model | `curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v --caption` | [Make a training set](#make-a-training-set) |
| Make a training set for MiniMax H3 | `curu-caption cut --corpus clips-corpus/ --out h3-training/ --projection h3 --caption` | [Make a training set for MiniMax H3](#make-a-training-set-for-minimax-h3) |
| Make an audio-visual bundle for amoral | `curu-caption cut --corpus clips-corpus/ --out bundle/ --trainer amoral-av --golden golden-v1/ --estate-root media/` | [Make a bundle for amoral](#make-a-bundle-for-amoral) |
| Look at a corpus or a cut, and correct it | `curu-caption review --corpus corpus/` | [Review a corpus or a cut](#review-a-corpus-or-a-cut) |
| Caption each file in a folder | `curu-caption caption photos/` | [Caption a folder](#caption-a-folder) |
| Find near-duplicate files in a folder | `curu-caption curate photos/` | [Find near-duplicates in a folder](#find-near-duplicates-in-a-folder) |
| Keep or reject each file against your criteria | `curu-caption judge photos/ --criteria "no blurry shots"` | [Judge a folder](#judge-a-folder) |
| Write `dataset.yaml` for a captioned folder | `curu-caption dataset-config photos/ --trigger-word ohwx` | [Write dataset.yaml for a folder](#write-datasetyaml-for-a-folder) |
| Read the ComfyUI workflow in a generated image | `curu-caption comfy inspect generated.png` | [Read a ComfyUI workflow](#read-a-comfyui-workflow) |
| Caption on a computer that has a GPU | `curu-caption caption photos/ --ollama-url http://gpu-box:11434` | [Caption on another computer](#caption-on-another-computer) |
| Work on a network share (NAS) | `curu-caption ingest --source videos/ --out corpus/ --stage-locally` | [Work on a network share](#work-on-a-network-share) |
| Set the caption model for each command | `export CURU_CAPTION_CAPTION_MODEL=saracen9/amoral-qwen3.8-27b` | [Set the caption model and other settings](#set-the-caption-model-and-other-settings) |
| Find out why a command stopped | `curu-caption cut --prune corpus/ \|\| echo "exit code $?"` | [Handle errors](#handle-errors) |

## Install

You need Python 3.12 or later, [uv](https://docs.astral.sh/uv/), a Rust toolchain
([rustup](https://rustup.rs/)), [ffmpeg](https://ffmpeg.org) and [Ollama](https://ollama.com).

To install the released package from PyPI, you need only Python 3.12 or later, uv, ffmpeg and
Ollama:

```bash
uv tool install "curu-caption[audio]"    # curu-caption and curu-caption-tui on the PATH
```

The wheels are for Linux x86_64 and macOS (arm64 and x86_64). They include the Rust extension
and the `review` program. On a CUDA computer, use `"curu-caption[audio,gpu]"`.

To develop, install from a clone of the repository. Then you also need the Rust toolchain, and
on Linux a C compiler:

```bash
git clone https://github.com/darth-veitcher/curu-caption.git
cd curu-caption
uv sync --extra audio
cargo build --release -p curu-tui && scripts/stage-tui.sh   # the review program
source .venv/bin/activate                                  # puts curu-caption on the PATH
```

- `uv sync` also compiles the Rust extension `curu_caption._core`.
- Give `--extra audio` each time that you run `uv sync`. A plain `uv sync` removes it. It makes
  the `sound`, `speech` and `music` facts. `ltx` needs the `sound` fact, and `h3` uses all three
  when it has them.
- The second line builds the `review` program. No other command needs it.
- No model downloads at install time. Each model downloads the first time that a command needs
  it (see [Technical notes](#technical-notes)).

Then get a caption model for Ollama. The
[amoral-qwen3.5-9B](https://github.com/darth-veitcher/amoral-qwen3.5-9B) project makes the three
caption models. The default model needs about 32 GB of memory (unified memory or VRAM):

```bash
ollama pull saracen9/amoral-qwen3.5-9b
```

| Model | Memory | Use it when |
|---|---|---|
| `saracen9/amoral-qwen3.5-9b` | about 32 GB | The default. It is correct for most datasets |
| `saracen9/amoral-qwen3.8-27b` | about 48 GB | You need fine detail or the subject matter is unusual |
| `saracen9/amoral-muse-glimmer-30b-abliterated` | about 64 GB | The smaller models do not describe the subject matter plainly |

The full setup, with each platform, cache and fix, is in [docs/setup.md](docs/setup.md).

## Quickstart

Make a training set for a text-to-video LoRA from one video, `episode.mp4`:

```bash
curu-caption ingest --source episode.mp4 --out corpus/
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v \
    --want "a person smiling" --limit 60 --dry-run
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v \
    --want "a person smiling" --limit 60 --caption --trigger ohwx
```

1. `ingest` extracts one frame each second, removes the duplicate and blank frames, and indexes
   the frames that it keeps. It does not caption. The result is the corpus, `corpus/`.
2. `cut --dry-run` shows the 60 frames that match "a person smiling", and the facts that each
   frame does not have yet. It writes nothing and it costs nothing.
3. `cut --caption` captions those 60 frames only, and writes them to `training/`.

`training/` then holds `frame_<ms>.png` and `frame_<ms>.txt` pairs and a `dataset.yaml`.
Each caption starts with the trigger word `ohwx`. Copy `training/` to the computer that trains,
and give `dataset.yaml` to AI-Toolkit.

Make the corpus one time, and cut it many times. A second LoRA from the same video is a
second `cut`, not a second `ingest`. Each command refuses an `--out` folder that is not empty,
so no run writes over an earlier result.

## Make a corpus from a video

`ingest` reads a video, or each video in a folder, and makes a corpus:

```bash
curu-caption ingest --source videos/ --out corpus/
```

It reads the videos in each subfolder too. At the end, it prints `corpus ready` with the
number of items, and the next command to run.

- **Frames.** By default, `ingest` extracts one frame each second (`--interval 1.0`). Each
  frame is a PNG named for its time in milliseconds, for example `frame_0000004000.png`.
- **The cull.** `ingest` removes near-duplicate frames and blank frames. A blank frame has a
  sharpness (Laplacian variance) below `--min-sharpness`, 1.0 by default. Of each group of
  near-duplicates, it keeps the sharpest frame.
- **The index.** `ingest` then indexes each frame that it keeps, so that `cut --want` can find
  it. It also records a quality score for each item (see
  [Index a corpus again](#index-a-corpus-again)).
- **Many sources.** In a corpus of many videos, each video has its own subfolder, with the
  path of the video under `--source`. In a cut, each file name starts with its source folder,
  so the `clip_000.mp4` of each episode stays.
- **Some videos of a folder.** Give `--source` more than once, for example
  `--source s01/ep01.mkv --source s01/ep02.mkv`. A video that you name directly gets a subfolder
  with its file name, without the extension, for example `ep01`. If two videos have the same
  subfolder name, `ingest` stops before it reads a video. Then give their parent folder as
  `--source`.

The corpus is the result that you keep. Each item holds its facts in the media file itself, in
PNG text chunks or MP4 metadata. Thus you can move or copy a corpus as one folder, and a single
item that you copy out keeps its facts.

**Clips for a video LoRA.** Add `--clips` to cut the video into MP4 clips, one for each shot,
with their audio. Use `--interval 1` or less with it: `ingest` finds each shot from the gaps
between the frames that it samples. With a wide interval, no clip is long enough, and `ingest`
prints this reason. The frames that `ingest` used to find the shots stay in the corpus, marked
`detection-only`, so a later detection costs nothing. A cut never selects them, and says
`N detection-only item(s) — kept as evidence, never selected`.

Before it finds the shots of a video, `ingest` prints `detecting shots in N frame(s)`. N is the
number of frames that it extracted from that video. The next line, with the clip count, comes
after it finds the shots and cuts the clips.

The audio of a clip is FLAC. It keeps each channel and the channel layout of the source, for
example 5.1.

| Option | Default | What a higher value does |
|---|---|---|
| `--interval` | 1.0 s | Fewer frames, and a faster ingest. With `--clips`, `ingest` finds fewer shots |
| `--phash-distance` | 8 | More frames count as near-duplicates, and the cull removes more |
| `--min-sharpness` | 1.0 | More frames count as blank. Keep it low: it measures detail, not focus |

`--phash-distance` is the number of bits, from 0 to 64, that differ between the perceptual
hashes of two frames. At 0, only frames that look the same are near-duplicates. On real TV
footage, the default 8 removed 37% of the frames, and each group was one static shot.

![curu-caption ingest --help](docs/assets/ingest-help.svg)

## Make a corpus from photographs

`ingest` also reads a folder of photographs, with its subfolders:

```bash
curu-caption ingest --source photos/ --out photo-corpus/
```

- Each photograph becomes one item. `ingest` converts it to PNG, because a PNG file can hold
  facts and a JPEG file cannot.
- The name of each item comes from its path under `--source`. Thus two photographs with the
  same file name do not write over each other.
- `ingest` ignores the files that are not media, for example `Thumbs.db`, `.DS_Store` and
  `.zip` files.
- `ingest` refuses `--clips` for photographs, because a photograph has no shots and no audio.
  It refuses before the ingest starts, not later at `cut` time.

| Path under `--source` | Item in the corpus |
|---|---|
| `man/man_0001.jpg` | `man_0001.png` |
| `2019/IMG_0001.jpg` | `2019_IMG_0001.png` |
| `2024/IMG_0001.jpg` | `2024_IMG_0001.png` |

`ingest` leaves out a folder name when the file name already starts with it, as for
`man_0001.jpg`.

## Index a corpus again

`index` runs the index step of `ingest` again on a corpus that exists. Use it after a change to
an index model, or on a corpus that you made with `--no-index` or `--no-quality`:

```bash
curu-caption ingest --source photos/ --out photo-corpus/ --no-quality
curu-caption index --corpus photo-corpus/                 # adds the quality scores
curu-caption index --corpus photo-corpus/ --tags          # also adds WD14 tags and rating
```

`index --no-quality` makes only the index. On a corpus of 11 items, of which `ingest` kept 7, a
second run of `index` prints:

```text
index 7 kept item(s) in photo-corpus (4 culled, excluded or detection-only, not indexed)
  indexing with EmbeddingGemma 2 (curu-vision)
  judging quality with LiquidAI/d1-3B — a review pre-sort, never a gate
  embedding: 0 extracted, 7 already current
  quality: 0 extracted, 7 already current

index ready — photo-corpus
```

`index` reads only the kept items. It does not make a fact again when the fact is current for
its model and parameters. Thus a second run does nothing, as in the example output.

Run `index` again on each corpus after a change to the index model. The embedding fact records the model and
the version of curu-vision. After you update curu-vision, each vector is stale. `cut --want` does
not use a stale vector, and `index` makes it again. curu-vision puts each transparent part of a
picture onto white before the embedding.

The **quality** fact holds two probabilities from 0 to 1. The model d1-3B gives them from three
frames of each clip, at 10%, 50% and 90%, or from the one picture of a still:

- `p_keep` is the probability that the item is good training data: sharp, one shot, and
  nothing broken.
- `p_person` is the probability that a person is visible.

A higher value is more likely to be true. The scores only sort items for review. No command
removes, culls or selects an item on these numbers. On 101 labelled clips, d1-3B put a good clip
above a rejected clip 72% of the time (AUROC 0.72; 0.5 is chance). That is useful to order a
review, and too weak to make the decision alone. A cut records the scores of each item that it
selects in `cut-selection.json`.

`--tags` adds two facts to each photograph, from one run of the WD14 tagger. A clip gets no
tags yet (see Limits):

- The **tags** fact holds the descriptive tags, the tag with the highest score first, for
  example `solo, 1girl, outdoors`. A tag for a thing in the picture needs a score above 0.35, and
  the name of a character needs a score above 0.70. It never holds a rating label.
- The **rating** fact holds the four WD14 rating scores, each from 0 to 1, for example
  `{"explicit": 0.002, "general": 0.736, "questionable": 0.011, "sensitive": 0.286}`.
  `general` is safe content, and `explicit` is sexual content. A higher score means that the
  model finds that rating more probable. Each score is independent of the other three, so the
  four do not add up to 1.

No command reads the rating yet. It is a fact in each media file, for your own sort or filter.
No command removes, culls or selects an item on it. WD14 learned from anime pictures, so its
rating of live-action footage is not accurate.

| Step | On a CUDA GPU | On a CPU or Apple silicon |
|---|---|---|
| Index (EmbeddingGemma 2) | not measured; it needs the `gpu` extra | about 5.4 s for each 1280x720 frame (Apple M1 CPU) |
| Quality (d1-3B) | about 0.25 s for each item (RTX 3090) | about 2 minutes for each item (16 GB Apple M1) |
| WD14 tags and rating (`--tags`) | — | about 4.5 s for each item (the `--tags` help) |

On a Mac, use `--no-quality` and run `index` later on a computer with a GPU.

## Find items by meaning

`cut --want` searches the index for the items that match a description in plain English.
With `--dry-run`, it shows the result and writes nothing:

```bash
curu-caption ingest --source videos/ --out corpus/
curu-caption cut --corpus corpus/ --out training/ --want "a wide shot" --limit 60 --dry-run
```

The command prints the number of matches, and the highest and lowest score that it kept. The
score of each item is the cosine similarity between the description and the item, from -1
to 1. A higher score is a closer match, but use the order, not the number. Over 377 clips, the
scores of each query were between 0.64 and 0.79. The score where good matches stop changes
with the query, so no fixed limit is correct for all queries.

- `--limit` keeps the best matches, 100 by default. `--limit 0` keeps each match. This is the
  control to use.
- `--threshold` removes each match with a score below the value. It is off by default.
- Without `--want`, the cut takes all the items of the corpus. With a `--limit` above 0, it
  takes the first `--limit` items, in name order.
- `cut-selection.json` records the description and the lowest score that the cut kept.

## Make a training set

`cut` selects items from a corpus and writes a training set (a cut) for one model. With
`--caption`, it captions the selected items only, not the full corpus:

```bash
curu-caption ingest --source episode.mp4 --out corpus/
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v --caption --trigger ohwx
```

```text
training/
├── frame_0000004000.png        the media, rendered for --projection
├── frame_0000004000.txt        "ohwx, a woman turns from the window as the door opens"
├── frame_0000004000.avdm.json  the scene document that the caption came from
├── dataset.yaml                the trainer config (job.yaml for h3 and h3-prose)
├── cut.json                    the projection, trainer, trigger and the source of each fact
└── cut-selection.json          the selected items, the lowest score kept and the quality scores
```

A projection is the form of caption that one model trains on. Select it for the model that you
train. The default is `ltx`.

| `--projection` | The caption | Frame count of a clip |
|---|---|---|
| `wan-t2v` | Text-to-video: the full scene and its motion, with no audio | aligned to `n % 4 == 1` |
| `wan-i2v` | Image-to-video: the change from the first frame. It needs a corpus of clips | aligned to `n % 4 == 1` |
| `ltx` | One paragraph of the picture and the sound | aligned to `n % 8 == 1` |
| `h3` | Three fields: the description with dialogue, the soundscape, and the music | 17k + 5 frames at 24 fps |
| `h3-prose` | Prose of the picture and the motion, with no audio | as `h3` |

- **Captions.** The caption model fills in a fixed form of claims about the middle frame of the
  item. Then d1-3B checks each claim with a yes or no question. The caption keeps only the
  checked claims, one plain sentence for each.
- **Missing facts.** `--dry-run` names each fact that the projection needs and the corpus does
  not have. `ltx` needs a `sound` fact, and `h3` uses one when it has it. `--caption` makes it
  when the `audio` extra is installed.
- **The trainer.** `--trainer ai-toolkit` (the default) writes `dataset.yaml`, or `job.yaml`
  for `h3` and `h3-prose`. `--trainer musubi` writes a `.wav` file beside each clip, and no
  AI-Toolkit file. An `ltx`, `wan-t2v` or `wan-i2v` cut copies each clip with its FLAC audio.
  For LTX-2, AI-Toolkit mixes 6 or 8 channels down to stereo. musubi-tuner mixes the audio down
  to mono or stereo. WAN 2.2 reads no audio.
- **Portable.** `folder_path` in the config is `.`, the cut folder itself. Thus the cut works
  on the computer that trains, and the corpus stays on yours.

The result table of `cut` counts the items:

| Row | Meaning |
|---|---|
| `renderable` | The item has each fact that the projection needs |
| `written` | The item is in the cut, with its caption |
| `unrenderable` | A fact is missing. The lines below the table name it. Add `--caption` to make it |

Each fact records its producer and parameters. A second cut does not make a fact again when its
producer and parameters did not change. A second cut of the same two items finished in less
than four seconds, after the first cut used minutes for the captions. The full reference is in
[docs/cut.md](docs/cut.md).

![curu-caption cut --help](docs/assets/cut-help.svg)

## Make a training set for MiniMax H3

MiniMax H3 trains on clips. Make a corpus of clips, then cut it with `--projection h3`:

```bash
curu-caption ingest --source episode.mp4 --out clips-corpus/ --clips --interval 1
curu-caption cut --corpus clips-corpus/ --out h3-training/ --projection h3 --caption --trigger ohwx
```

The cut encodes each clip again to 24 fps, to a frame count of 17k + 5 (5, 22, ... 345), and to
sides that are multiples of 32. It then measures each clip. A clip that does not agree with
all three rules goes in the report, not in the cut. The audio becomes stereo, because H3 trains
on stereo audio. The cut also writes a complete AI-Toolkit
`job.yaml`. To train, run `python <ai-toolkit>/run.py job.yaml` in `h3-training/`.

| Option | Default | Effect |
|---|---|---|
| `--resolution` | 512 | The short side in pixels, rounded down to a multiple of 32. A clip never gets larger |
| `--min-frames` | 73 (about 3 s) | A shorter clip goes in the report, before the cut captions anything |
| `--max-frames` | 345 (the H3 maximum) | The longest clip. Use a lower value to train with less VRAM |
| `--steps` | 3000 | The training steps in `job.yaml` |

These options apply to `h3` and `h3-prose` only. The other projections copy each clip as it is.

## Make a bundle for amoral

`--trainer amoral-av` writes a training bundle for the omni model of
[amoral](https://github.com/darth-veitcher/amoral), which hears and sees together. The bundle
has clips and derived clips, and no captions:

```bash
curu-caption cut --corpus clips-corpus/ --out bundle/ --trainer amoral-av \
    --golden golden-v1/ --estate-root media/ --want "two people talking" --limit 200 \
    --derive 1 --seed 0
```

- The corpus must come from `ingest --clips`, which records the source file of each clip.
- The cut writes nothing if a source file is also a source file of golden-v1, the evaluation
  set. It lists each such file. To keep these files out of the corpus, give `ingest` the
  option `--golden golden-v1/` too.
- `--derive N` makes N changed copies of each clip, for example with the audio of another clip.
- Open `review.html` in the bundle to look at and listen to each derived clip.

The bundle format, the five changes and the clips that the cut leaves out are in
[docs/amoral-av.md](docs/amoral-av.md).

## Review a corpus or a cut

`review` opens a corpus or a cut in the terminal. Look at each item, correct its caption, and
mark the items that you do not want:

```bash
curu-caption review --corpus corpus/
curu-caption review --corpus training/ --source corpus/
curu-caption cut --prune training/
```

```text
┌ Corpus (2/6) ⚠ 1 empty ────┐┌ Preview ────────────────────────────────────┐
│[!]   frame_0000001000      ││                                             │
│      frame_0000002000      ││        [ frame_0000002000.png ]             │
│[X]   frame_0000003000      ││                                             │
│[clip] clip_001             ││                                             │
└────────────────────────────┘└─────────────────────────────────────────────┘
                              ┌ Caption ────────────────────────────────────┐
                              │ a woman turns from the window as the door   │
                              │ opens                                       │
                              └─────────────────────────────────────────────┘
 j/k ↑↓ nav  e edit  V visual  R find-replace  P prefix  S set  x exclude  ? help  q quit
```

The title says **Corpus** or **Cut**. What you opened sets what `x` (exclude) means:

| You opened | `x` means | It applies to |
|---|---|---|
| a corpus | "poor quality: do not show me this again" | the item, and each future cut |
| a cut | "correct, but not for this training set" | this cut only |

- **Reject in the corpus from a cut.** Open the cut with `--source`, which names its corpus.
  Then `X` rejects the item in the corpus too.
- **Remove the rejected items.** The trainer config names the full cut folder, so a marked file
  that is still in the folder still trains. `cut --prune` deletes the marked files and writes
  the config again.
- **Safe to prune.** Each file in a cut is a copy, and `cut-selection.json` records the
  selection. `cut --prune` refuses a folder with no `cut.json`, because a corpus holds the only
  copy of its media.
- `⚠ N empty` in the title and `[!]` on a row show captions that are empty or duplicated.

Open a folder of corpora to work through each one in a queue. The keys, the batch editing and
the queue are in [docs/tui.md](docs/tui.md).

## Caption a folder

`caption` captions each image and video in a folder, or one file, without a corpus. It writes a
`.txt` file with the same name beside each file:

```bash
curu-caption caption photos/
curu-caption caption photos/ --dry-run --hint "a set of family birthday photos"
```

- `caption` reads the images (`.png`, `.jpg`, `.jpeg`) and videos (`.mp4`, `.mkv`, `.mov`,
  `.avi`, `.webm`) in the folder. It captions a video from one frame.
- It does not caption a file that has a caption of 10 characters or more. Use `--force` to
  caption it again.
- `--hint` gives context for each caption in the run. `--dry-run` prints the captions and writes
  no file.
- It shows the progress of each file. If one file fails, it continues. At the end, it exits
  with a code that is not 0.

Add WD14 tags, local and with no network, with `--tags`. With `--no-prose`, a caption has only
the tags, and no caption model is necessary:

```bash
curu-caption caption photos/ --no-prose --tags
```

WD14 tags are short Danbooru-style tags, for example `1girl, outdoors, smile`. With prose and
tags, the `.txt` file has the prose, an empty line, and then the tags.

## Find near-duplicates in a folder

`curate` reports the groups of near-duplicate images and clips in a folder. It does not change
or delete a file:

```bash
curu-caption curate photos/
```

```text
Found 1 near-duplicate group(s):
  Group 1: IMG_0001.jpg, IMG_0004.jpg
```

- Two files are near-duplicates when their perceptual hashes differ by 5 bits or fewer. Set a
  different value with `--threshold`. A higher value puts more files in each group.
- `curate` compares PNG and JPEG stills and clips in one pass. It compares a clip on its middle
  frame. A clip costs 150 to 200 times more than a still, so a few thousand clips take minutes.
- `curate` skips caption files and manifests, and does not report them.
- If all the files go in one group, the threshold is too high. Then `curate` says so, and
  reports no groups.

## Judge a folder

`judge` asks the caption model to keep or reject each image against your criteria. It does
not change a file:

```bash
curu-caption judge photos/ --criteria "no blurry shots, subject clearly visible"
```

```text
Judged 4 image(s): 3 keep, 1 reject.

  KEEP   IMG_0001.jpg: <the reason of the model>
  REJECT IMG_0004.jpg: <the reason of the model>
```

Each line gives the verdict and the reason of the model. If the model rejects all the images,
`judge` changes each verdict to keep and says so. Then the criteria are too strict or not
clear.
## Write dataset.yaml for a folder

`dataset-config` writes the `dataset.yaml` that AI-Toolkit trains from, for a folder of media
and `.txt` captions:

```bash
curu-caption dataset-config photos/ --trigger-word ohwx
```

It writes `photos/dataset.yaml`:

```yaml
datasets:
- folder_path: .
  caption_ext: txt
  num_frames: 1
  resolution:
  - 256
  - 256
  num_repeats: 1
  trigger_word: ohwx
  sample_prompts:
  - a test pattern, IMG_0001
```

`folder_path` is `.`, so run AI-Toolkit from the folder. `--num-repeats` sets the weight of the
dataset, 1 by default. `resolution` is the size of the first image, or 512 × 512 when there is
no image. `sample_prompts` holds up to five different captions from the folder.

## Read a ComfyUI workflow

`comfy inspect` prints each node of the ComfyUI workflow in a generated PNG image:

```bash
curu-caption comfy inspect generated.png
```

```text
Node 3 (KSampler)
  seed: 851616030078638
  steps: 20
  ...
Node 6 (CLIPTextEncode)
  text: masterpiece best quality girl
  clip: -> node 10 (output 1)
```

To use the text of a node as context for a caption, give its id to
`caption --comfy-node`.

## Caption on another computer

Captioning is the one step that needs a GPU. Keep the corpus on your computer, and send the
captions to Ollama on a computer that has a GPU:

```bash
curu-caption caption photos/ --ollama-url http://gpu-box:11434
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v --caption \
    --ollama-url http://gpu-box:11434 --model saracen9/amoral-qwen3.8-27b
curu-caption cut --corpus corpus/ --out training/ --projection wan-t2v --caption \
    --pod curu-caption-vlm
```

- `cut --caption` checks the endpoint before it selects or writes anything. If the computer
  does not have the model, the error names the models that it has.
- `--pod` names a RunPod pod that `curu-infra` made. The command gets its address and its
  token from `curu-infra`.
- For an Ollama behind a token, set `CURU_CAPTION_OLLAMA_TOKEN`. Do not put a token on the
  command line.

SSH tunnels, pods and the error messages are in
[docs/remote-captioning.md](docs/remote-captioning.md).

## Work on a network share

On a network share (NAS), add `--stage-locally` to `ingest` and `cut`. The command works on the
local disk and copies the result to the share in one pass at the end:

```bash
curu-caption ingest --source videos/ --out corpus/ --stage-locally
```

- A network share is slow for thousands of small reads and writes. Staging changes them into
  one copy.
- Both commands show `sync-back: <done>/<total>` while they copy. If the counter stops, the
  mount has stopped.
- `caption` stages its writes by default. `--no-stage-writes` stops this for a local folder.

The rules, the free disk space and the fix for a stopped macOS mount are in
[docs/nas.md](docs/nas.md).

## Set the caption model and other settings

Most settings are options. Set the caption model one time for each command:

```bash
export CURU_CAPTION_CAPTION_MODEL=saracen9/amoral-qwen3.8-27b
```

The command-line option comes first, then the environment variable, then the configuration file
`~/.config/curu-caption/config.toml`, then the default:

```toml
# ~/.config/curu-caption/config.toml
[caption]
model = "saracen9/amoral-qwen3.8-27b"

[ort]
providers = "cpu"
```

Each setting, environment variable and file on disk is in
[docs/configuration.md](docs/configuration.md).

## Handle errors

A command that cannot do its work stops before it writes, prints the reason, and exits with a
code that is not 0:

```bash
curu-caption ingest --source photos/ --out photo-corpus/ --no-index
curu-caption ingest --source photos/ --out photo-corpus/ --no-index || echo "exit code $?"
curu-caption cut --prune photo-corpus/ || echo "exit code $?"
```

```text
refusing: photo-corpus is not empty
exit code 1
Error: photo-corpus has no cut.json -- it is not a cut. Pruning a corpus would delete the only copy of its media, so this refuses rather than guesses.
exit code 1
```

- **Before the cost.** `cut --caption` checks the caption endpoint before it selects, copies
  or captions. With `--stage-locally`, a command checks the free disk space before it copies.
- **One bad file.** `caption`, `curate` and `judge` report a file that fails and continue.
  At the end, they exit with a code that is not 0.
- **Missing facts.** `cut` reports an item that it cannot caption. It does not write a caption
  that it cannot check.
- **A failed encode.** `cut --trainer amoral-av` writes each clip that it can encode. It shows
  the first 5 failures with the reason from ffmpeg, and the number of failures. Then it exits
  with 1.
- **Disk space.** `ingest` stops to wait when the disk of the corpus has less than 2 GB free.
  It writes a message on stderr when the wait starts, and again each 30 seconds. The message
  gives the free space and the minimum. After 10 minutes, it stops with an error.

## Limits

- **`--tags` does not work on clips.** On a corpus made with `--clips`, the `tags` and `rating`
  facts fail for each clip ([#170](https://github.com/darth-veitcher/curu-caption/issues/170)).
- **The `sound` fact is not heard.** The caption model writes it from the sound captioner's
  caption, the words, and one frame. On 40 test clips with the default 9B caption model, 9 of
  30 claims beyond "a voice speaks" were correct and 15 were false. Most false claims are
  sources that the captioner names wrongly, for example a siren, a horse or a cow. For `h3`,
  this text becomes the soundscape (`overall_soundscape`). Read it before you train, or train
  video-only with `h3-prose` ([#93](https://github.com/darth-veitcher/curu-caption/issues/93)).
- **Dialogue does not say who speaks.** Each line starts with `Someone says:`, so a clip with two
  speakers does not tell them apart. A fix needs speaker diarization in the captions.
- **The picture is checked, not proven.** On 40 test frames, 88 of the 898 claims that the check
  kept were false (precision 0.90, against 0.81 for the old prose captions). Most errors are
  upper garments, the direction that a person faces, and headwear. The check of helmets is poor,
  and an invented person keeps its claims.
- **`--provider openrouter` ignores the corpus prompts.** It does not get the claims form or the
  motion and sound prompts. Use Ollama for the captions of a cut. `cut --caption` has no cloud
  provider.
- **`review` and clips.** `x` on a clip in `review` does not stop `cut` from using the clip.
  `X` does not find a clip in a corpus of many sources
  ([#87](https://github.com/darth-veitcher/curu-caption/issues/87)).
- **`review` does not sort by quality**, because it cannot read the facts of a clip yet
  ([#87](https://github.com/darth-veitcher/curu-caption/issues/87)).
- **`review` has no mouse** ([#101](https://github.com/darth-veitcher/curu-caption/issues/101)).
- **d1-3B is slow without CUDA.** On a Mac, the quality scores and the claim check run on the
  CPU, and take minutes for each item.
- **It does not train.** curu-caption makes datasets. AI-Toolkit trains.

## Technical notes

- **Licence.** MIT ([LICENSE](LICENSE)).
- **Version.** 3.0.3, set in `pyproject.toml` (`project.version`) and in `Cargo.toml`
  (`workspace.package.version`). The changes are in [CHANGELOG.md](CHANGELOG.md).
- **Build.** `uv sync` builds `crates/curu-core` into `curu_caption._core` with maturin.
  `crates/curu-tui` is the `review` program, and `crates/curu-ondisk` writes PNG text chunks.
  After a change to the Rust code of the extension, run `uv sync --extra audio` again.
  `[tool.uv] cache-keys` in `pyproject.toml` lists the files that cause a new build. On Linux,
  the extension contains its own copy of liblzma, a compression library, so it needs no system
  copy.
- **Platforms.** macOS and Linux. On Apple silicon, the models run on the CPU: CoreML was 1.6 to
  5.6 times slower for WD14.
- **ffmpeg.** On x86_64, the Rust core downloads ffmpeg when it is not on the `PATH`. On other
  platforms, install it.
- **Models.** Each model downloads on first use, at a pinned revision. EmbeddingGemma 2 (1.8 GB)
  is the index, and d1-3B (6.2 GB) gives the quality scores and the claim check. WD14 (455 MB)
  gives the tags and the rating. EmbeddingGemma 2 and WD14 come from curu-vision, which checks
  their SHA-256.
- **Audio models.** The sound captioner (about 1 GB) and AST (346 MB) download into the cache of
  this project. AST is an audio tagger that gives a probability for each AudioSet sound
  label. Speech comes from curu-vision: MOSS (1.8 GB) on a computer with CUDA, which
  labels speakers, and Parakeet (2.4 GB) on other computers.
- **Tutorials.** The records of real training runs use the commands of version 2.0, which do
  not exist now. They are in
  [project-management/archive/tutorials-v2/](project-management/archive/tutorials-v2/README.md).
- **Where it fits.** curu-catalog finds media, curu-caption makes training data, curu generates,
  and curu-vision supplies shared models. See [docs/ecosystem.md](docs/ecosystem.md).
- **Decisions.** The decisions are in [project-management/ADRs/](project-management/ADRs/).
  The main ones:
  - Facts in the media file:
    [ADR-083](project-management/ADRs/ADR-083-the-media-file-is-the-fact-store.md)
  - The corpus and the cut:
    [ADR-084](project-management/ADRs/ADR-084-the-corpus-and-the-cut-as-built.md)
  - H3 through AI-Toolkit:
    [ADR-085](project-management/ADRs/ADR-085-h3-via-ai-toolkit-is-the-primary-target.md)
  - The cull thresholds:
    [ADR-080](project-management/ADRs/ADR-080-the-cull-thresholds-are-measured-not-inherited.md)
  - Search ranks and has no threshold:
    [ADR-081](project-management/ADRs/ADR-081-search-ranks-it-does-not-threshold.md)
  - AVDM, the scene document:
    [ADR-088](project-management/ADRs/ADR-088-avdm-is-the-canonical-scene-model.md)
  - EmbeddingGemma 2, the index:
    [ADR-089](project-management/ADRs/ADR-089-embeddinggemma-2-is-the-search-index.md)
  - The quality scores sort and do not decide:
    [ADR-090](project-management/ADRs/ADR-090-d1-pre-sorts-review-it-never-gates.md)
  - Audio facts of the trained window:
    [ADR-091](project-management/ADRs/ADR-091-audio-facts-describe-the-trained-window.md)
  - Captions from checked claims:
    [ADR-092](project-management/ADRs/ADR-092-captions-describe-the-picture-from-checked-claims.md)
  - The bundle for amoral:
    [ADR-093](project-management/ADRs/ADR-093-an-omni-model-extracts-the-facts-that-need-sound-and-picture-together.md)
  - Speech from curu-vision:
    [ADR-094](project-management/ADRs/ADR-094-speech-comes-from-curu-visions-transcription-port.md)
  - The user files under XDG:
    [ADR-064](project-management/ADRs/ADR-064-rename-the-on-disk-namespace.md)
  - The local quality gate, with no CI:
    [ADR-059](project-management/ADRs/ADR-059-local-quality-gate-instead-of-github-actions-ci.md)

## Develop

Run the quality gate before you push. There is no CI, so this local gate is the only gate:

```bash
uv tool install beacon-framework   # BEACON 1.18 or later: the docs check and beacon doctor
uv sync --extra audio
scripts/check.sh            # ruff, ty, cargo, the tests, the docs check and beacon doctor
scripts/check.sh --full     # also a build from a new clone, before a release or a merge
```

- `uv run pytest` runs the default tests, with no model and no endpoint.
  `uv run pytest -m model` and `uv run pytest -m vlm` run the tests that need weights or
  Ollama.
- Write `README.md`, `docs/` and `AGENTS.md` to the standard in
  [docs/WRITING.md](docs/WRITING.md). `beacon docs check` checks them.
- The workflow, the tests and the release are in [CONTRIBUTING.md](CONTRIBUTING.md). The rules
  for AI assistants are in [AGENTS.md](AGENTS.md).
- The project uses the [BEACON framework](https://github.com/JAMESVEITCH/beacon) and Spec Kit.
  The problem statement, the architecture, the decisions and the roadmap are in
  [project-management/](project-management/).

## Glossary

- **AAC**: a lossy audio format: the decoded audio is not the same as the audio before the
  encode. An MP4 file usually holds AAC audio.
- **AI-Toolkit**: the trainer that reads a `dataset.yaml` or a `job.yaml` (ostris/ai-toolkit).
- **AST**: an audio tagger that gives a probability for each AudioSet sound label. The `music`
  fact comes from it.
- **AUROC**: the probability that a score puts a good item above a bad item. 0.5 is chance, and
  1.0 is always correct.
- **AVDM**: the scene document that each caption comes from (the `avdm` package).
- **Channel layout**: the position of each audio channel. For example, 5.1 is front left, front
  right, centre, LFE (the low bass channel) and two rear channels. In 5.1(side), the last two
  channels are at the sides.
- **Claim check**: d1-3B's yes or no answer to a question about each claim of the caption model.
- **Clip**: an MP4 of one shot, with its audio as FLAC, made by `ingest --clips`.
- **Corpus**: the folder that `ingest` makes. Each item holds its facts.
- **Cosine similarity**: a measure of how close two embeddings are, from -1 to 1. A higher value
  is closer.
- **Cull**: the step of `ingest` that removes near-duplicate and blank frames.
- **Cut**: a training set that `cut` writes from a corpus. Each file in it is a copy.
- **d1-3B**: a LiquidAI model that gives the quality scores and the claim check.
- **dBFS**: the level of a sound sample in decibels (dB), where 0 dBFS is full scale. A lower
  (more negative) value is quieter.
- **Diarization**: a step that finds who speaks when, in an audio track.
- **Embedding**: a list of numbers for a picture or a text. Items with a similar meaning have
  similar embeddings.
- **Fact**: a value that a step measured or made for one item, with its provenance.
- **FLAC**: a lossless audio format. The decoded audio is the same as the audio before the
  encode.
- **Full scale**: the loudest sample value that a digital audio format can hold. The format
  holds a louder sample at full scale, and you hear distortion.
- **Item**: one frame, clip or photograph in a corpus.
- **Laplacian variance**: a measure of the detail in a picture. A blank frame is near 0.
- **LoRA**: a small set of extra weights that trains a model on one subject or style.
- **musubi-tuner**: the trainer that reads the files of `--trainer musubi` (kohya-ss/musubi-tuner).
- **Near-duplicate**: a frame or file that looks almost the same as another.
- **Ollama**: the local program that runs the caption model.
- **Perceptual hash (pHash)**: a 64-bit summary of how a picture looks. Similar pictures have
  hashes that differ in few bits.
- **Projection**: the form of caption and media that one model trains on.
- **Provenance**: the record of which producer made a fact, and with which parameters.
- **Staging**: work on the local disk, then one copy to the destination.
- **Trigger word**: the word at the start of each caption that calls the LoRA, for example
  `ohwx`.
- **VLM**: a vision-language model, here the caption model in Ollama.
- **WD14 rating**: four scores from the WD14 tagger, `general`, `sensitive`, `questionable` and
  `explicit`, each from 0 to 1. Each score is the probability that the picture has that content
  rating, from `general` (safe) to `explicit`.
- **WD14 tags**: short Danbooru-style tags from the local WD14 tagger.

