Metadata-Version: 2.4
Name: curu-vision
Version: 0.9.0
Requires-Dist: anyio>=4.4
Requires-Dist: httpx>=0.28,<1
Requires-Dist: huggingface-hub>=0.34
Requires-Dist: numpy>=2.0
Requires-Dist: onnxruntime>=1.31,<1.32
Requires-Dist: pillow>=11.0
Requires-Dist: psutil>=6.0
Requires-Dist: pydantic>=2.11,<3
Requires-Dist: tokenizers>=0.21
Requires-Dist: pydantic-ai-slim[anthropic,google,openai]==2.57.0 ; extra == 'cloud'
Requires-Dist: torch>=2.8 ; extra == 'gpu'
Requires-Dist: transformers==5.20.1 ; extra == 'gpu'
Requires-Dist: parakeet-mlx==0.5.3 ; platform_machine == 'arm64' and sys_platform == 'darwin' and extra == 'mlx'
Provides-Extra: cloud
Provides-Extra: gpu
Provides-Extra: mlx
License-File: LICENSE
Summary: Shared model adapters and media primitives for the curu ecosystem
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://github.com/darth-veitcher/curu-vision#readme
Project-URL: Repository, https://github.com/darth-veitcher/curu-vision

# curu-vision

## What is this

curu-vision is a Python library for programs that work with video, pictures and speech. It
gets frames from a video and finds the scenes in it. It finds copies of a picture and measures
blur, exposure and noise. It gives tags for pictures, makes vectors for search by meaning, makes
transcripts with timed words and asks a vision-language model about a picture. The other curu
packages use it, so each of these operations has one implementation. Only the model download, a
video URL and a vision-language model on a model server or a cloud provider use the network.

## What you can do

| Task | Code | Section |
|---|---|---|
| Get frames at given times | `media.frames_at(source, [12.0, 30.5])` | [Get frames at given times](#get-frames-at-given-times) |
| Find the scenes (shots) in a video | `media.shots(source)` | [Find the scenes in a video](#find-the-scenes-in-a-video) |
| Find copies of a picture | `hamming(a.hashes.phash, b.hashes.phash)` | [Find copies of a picture](#find-copies-of-a-picture) |
| Find blurred, dark, bright or noisy pictures | `signals.compute([ImageBytes(data)])` | [Find blurred, dark, bright or noisy pictures](#find-blurred-dark-bright-or-noisy-pictures) |
| Tag a picture or a frame | `tagger.tag([ImageBytes(data)])` | [Tag a picture or a frame](#tag-a-picture-or-a-frame) |
| Ask a vision-language model about a picture | `vlm.ask(VlmRequest(prompt="…", images=(picture,)))` | [Ask a vision-language model about a picture](#ask-a-vision-language-model-about-a-picture) |
| Search text and pictures by meaning (embeddings) | `embedder.embed([Query("cats on a couch"), ImageBytes(data)])` | [Search by meaning](#search-by-meaning) |
| Search video, sound and scenes by meaning | `embedder.embed([frames.to_video()])` | [Search video, sound and scenes by meaning](#search-video-sound-and-scenes-by-meaning) |
| Make a transcript | `transcriber.transcribe([Audio(samples, 16_000)])` | [Make a transcript](#make-a-transcript) |
| Read a video from bytes, a stream or a URL | `VideoSource(data=data)` | [Read a video from bytes, a stream or a URL](#read-a-video-from-bytes-a-stream-or-a-url) |
| Process many files in one call | `media.shots_many(sources)` | [Process many files in one call](#process-many-files-in-one-call) |
| Handle errors | `isinstance(result, ItemFailure)` | [Handle errors](#handle-errors) |
| Record what made a result | `result.provenance` | [Record what made a result](#record-what-made-a-result) |
| Download the models | `python -m curu_vision fetch embeddinggemma-2` | [Download the models](#download-the-models) |

## Install

You need these on the computer that installs the package:

- Python 3.12 or later.
- A Rust toolchain ([rustup](https://rustup.rs)). The package builds a Rust extension when
  uv installs it.
- `ffmpeg` and `ffprobe` 6.0 or later on the path, and 8.1 or later on arm64. Install them
  with `apt install ffmpeg` or `brew install ffmpeg`. On arm64, Ubuntu 24.04's ffmpeg 6.1 is
  too old, and Homebrew's `ffmpeg` is correct. curu-vision does not include or download a
  decoder.

Add the package to your project by its release tag:

```toml
# pyproject.toml
[project]
dependencies = ["curu-vision"]

[tool.uv.sources]
curu-vision = { git = "https://github.com/darth-veitcher/curu-vision", tag = "v0.9.0" }
```

Two optional extras add transcription engines. Use `curu-vision[mlx]` for Parakeet on Apple
silicon. Use `curu-vision[gpu]` for MOSS, which needs PyTorch and transformers. Your project
chooses the CUDA build of PyTorch from its own package index.

The extra `curu-vision[cloud]` adds cloud vision-language models through PydanticAI. You need
it only when you name a cloud model. A local model on Ollama or on a llama.cpp server needs no
extra.

### Download the models

The video module and the image signals module need no model. Tags, search by meaning and
transcripts need model files, which are not in the package. Download each model one time
before you use it:

```bash
python -m curu_vision fetch embeddinggemma-2      # search by meaning, about 3 GB
python -m curu_vision fetch embeddinggemma-2 --modality text --modality image  # 1.8 GB, no audio
python -m curu_vision fetch parakeet-tdt-0.6b-v3      # transcripts on the CPU, about 2.6 GB
python -m curu_vision fetch parakeet-tdt-0.6b-v3-mlx  # transcripts on Apple silicon, mlx extra
python -m curu_vision fetch moss-transcribe-diarize   # transcripts with speakers, gpu extra
python -m curu_vision fetch wd-v1-4-swinv2-tagger-v2  # tags, about 455 MB
```

The download is the only operation that goes to the network for curu-vision itself. Each
file has a pinned revision, size and SHA-256, and the command verifies each file. If a
download stops, run the same command again. The files go to `CURU_VISION_CACHE_DIR`, else
`$XDG_CACHE_HOME/curu-vision`, else `~/.cache/curu-vision`. When curu-vision loads a model, it
never downloads it: a missing model is an error (`ModelNotAvailable`).

A vision-language model is not one of these files. It runs on a model server, and you get it
with that server, for example `ollama pull qwen3-vl:2b-instruct`. curu-vision does not pull or
start a model.

## Quickstart

This example finds the shots in a video, and then measures the frame at the middle of each
shot. It uses the video module and the image signals module. It needs ffmpeg, and it needs no
model.

```python
from curu_vision.media import VideoSource, load_media
from curu_vision.signals import load_signals

media = load_media()  # finds ffmpeg and ffprobe and checks their versions, one time
signals = load_signals()  # needs no model
source = VideoSource(path="episode.mkv")

print(media.probe(source).duration)  # the length of the video, in seconds
shots = media.shots(source)
middles = media.frames_at(source, [(shot.start + shot.end) / 2 for shot in shots.shots])
for shot, result in zip(shots.shots, signals.compute(middles.frames)):
    detail, phash = result.measurements.detail, result.hashes.phash
    print(shot.index, shot.start, shot.end, round(detail), f"{phash:016x}")
```

The README tests run this example on a generated video of 36 s with three shots of 12 s. It
prints this:

```text
36.0
0 0.0 12.0 3402 b2bfd084314c1f8f
1 12.0 24.0 3093 93de15d12d245573
2 24.0 36.0 2152 9b932b369705b368
```

Your video gives different values. The generated frames are small blocks of colour with sharp
edges and a little noise. Thus their `detail` is usually higher than the 1,200 to 2,200 of the
sharp test photos in [Find blurred, dark, bright or noisy pictures](#find-blurred-dark-bright-or-noisy-pictures).

What the results mean:

- The first line is the length of the video in seconds.
- Each line after it is one shot: its index, its start and its end in seconds, then two
  values of the frame at its middle.
- The fourth value, `detail`, measures fine detail and sharp edges. A lower value usually means
  a more blurred frame. See
  [Find blurred, dark, bright or noisy pictures](#find-blurred-dark-bright-or-noisy-pictures).
- The last value is the pHash, a 64-bit number from the large shapes of the frame. Copies of
  one picture have almost the same pHash. See [Find copies of a picture](#find-copies-of-a-picture).
- `signals.compute` takes the frames from the video module as they are. These frames are
  already decoded, so they never give an `ItemFailure`.

The task sections that follow tell more about each step.

## Get frames at given times

```python
from curu_vision.media import VideoSource, load_media

media = load_media()  # finds ffmpeg and ffprobe and checks their versions, one time
source = VideoSource(path="episode.mkv")

info = media.probe(source)
print(info.duration, info.video[0].width, info.video[0].height)

frames = media.frames_at(source, [12.0, 30.5], size=(448, 252))
for frame in frames.frames:
    print(frame.time, frame.image.shape)  # the frame's own time, and (252, 448, 3)

every = media.frames_every(source, rate=1.0, start=10.0, end=20.0)
print([frame.time for frame in every.frames])  # one frame each second from 10 s to 19 s
```

What the results mean:

- `probe` reads the properties of the file: its streams, duration, frame rates, rotation, HDR
  and container. It does not decode the frames.
- `frames_at` gives one frame for each time, in the order of the times. Each frame is the
  frame on the screen at that time: the last frame that starts at or before it.
- `frame.time` is the frame's own presentation time in the source, in seconds. It can be a
  little before the time that you gave. This is also correct for a variable frame rate.
- `frame.image` is an RGB `uint8` array of shape (height, width, 3). It is upright, with the
  display aspect ratio. `size=(width, height)` scales it, and with no `size` it has the
  display size.
- `frames_every` gives the frames on the screen at `start`, `start + 1/rate` and so on, before
  `end`. With no `start` and `end`, it starts at the first frame and stops at the end of the
  video.
- A time outside the video raises `TimeOutOfRange`, and its message gives the valid range.

The [video guide](docs/video.md) gives the full rules.

## Find the scenes in a video

A shot is a continuous run of frames from one camera take. A video is a sequence of shots, and
a scene is one shot or a few shots together. `media.shots` finds the shots. curu-vision does
not join shots into scenes.

```python
from curu_vision.media import VideoSource, load_media

media = load_media()
source = VideoSource(path="episode.mkv")

shots = media.shots(source)
for shot in shots.shots:
    print(shot.index, shot.start, shot.end, shot.boundary.kind)  # "start", "cut" or "gradual"

picked = media.shot_frames(source, shots.shots[0], count=3, size=(448, 252))
print([frame.time for frame in picked.frames])  # three frames from the first shot
```

What the results mean:

- The shots cover the full video with no gap. Each shot ends where the next shot starts, and
  the last shot ends at the end of the last frame. Times are in seconds in the source.
- `shot.boundary.kind` tells how the shot starts. It is `"start"` for the first shot, `"cut"`
  for a sudden change and `"gradual"` for a fade or a dissolve.
- For a gradual boundary, `boundary.start` to `boundary.end` is the transition. The new shot
  starts at the first frame of the transition.
- `boundary.score` is the change of colours at the boundary, from 0 (the same colours) to 2
  (no colours in common). Most cuts have a score of 0.4 or more. A gradual boundary has a
  score of 0.6 or more. A high score does not always make a cut. Use the score to sort
  boundaries. It is not a probability.
- `shot_frames` gives frames from inside the shot only, with its first and last frame. Give
  `count` (2 or more, at equal time intervals) or `rate` (frames each second). With neither,
  you get every frame of the shot.

Accuracy, measured on public test sets, and speed:

- Cuts are accurate. On 11 BBC Planet Earth episodes, 97 % of the cuts found were real cuts,
  and it found 95 % of the real cuts.
- Gradual boundaries are less accurate. The RAI set is a public set of Italian television
  videos with labelled shot boundaries. On RAI videos 6 to 10, 39 % of the gradual
  boundaries found were real, and it found 85 % of the real ones. On all ten RAI videos, the
  values were 58 % and 72 %.
- It reads every frame. In a test on 2026-10-25 with a generated 720p video, one call ran
  about 29 times faster than real time, with 4 CPU threads.

## Find copies of a picture

A perceptual hash (pHash) is a 64-bit number that comes from the large shapes of a picture.
Two copies of one picture have almost the same bits. The Hamming distance is the number of
bits that are different, from 0 to 64.

```python
from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import hamming, load_signals

signals = load_signals()  # needs no model and reads no file
a, b = signals.compute(
    [ImageBytes(Path("photo.jpg").read_bytes()), ImageBytes(Path("copy.jpg").read_bytes())]
)
print(hamming(a.hashes.phash, b.hashes.phash))  # 0 to 64: small for a copy
print(f"{a.hashes.phash:016x}")  # the hash as 16 hexadecimal digits
```

What the distance means in practice:

| Distance | Usual meaning |
|---|---|
| 0 to 6 | A copy. A JPEG copy at quality 75 and half size was at most 6 bits away in the test pictures. |
| 7 to 19 | Not certain. Look at the two pictures. |
| 20 or more | Different pictures. All 703 pairs of unrelated test pictures were 20 bits or more apart. |

Frames of a video and of a copy at half size and lower quality were at most 2 bits apart. A
copy with 10 % cut from each side was 14 to 20 bits away, so a cropped copy can look
unrelated. A mirrored copy was 30 to 32 bits away.

Other values:

- `a.hashes.dhash` is a second 64-bit hash, the difference hash (dHash). Compare it with
  `hamming` too.
- `a.hashes.flat` is `True` for a picture of one colour. The hashes of all such pictures are
  at most 1 bit apart, so they all match each other. Their pHash is `8000000000000000`, or
  `0` for black.

pHash does not work well for all pictures. See [Limits](#limits).

## Find blurred, dark, bright or noisy pictures

```python
from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import load_signals

signals = load_signals()
(result,) = signals.compute([ImageBytes(Path("photo.jpg").read_bytes())])
values = result.measurements
print(values.detail, values.directionality, values.blur_direction)  # blur
print(values.dark_clipped, values.bright_clipped, values.contrast_spread)  # exposure, contrast
print(values.noise, values.colourfulness)
```

Each value is a measurement, not a decision. curu-vision gives no threshold, no verdict and
no combined score. Sort your pictures by a value. Or, find the limits that are correct for
your pictures.

The values in the table come from a test on 2026-10-27 with three sample photos (cats, a
tiger and a football match). The test applied a Gaussian blur, a motion blur of 15 px, and
changes of brightness and contrast. The [image signals guide](docs/image-signals.md#measured-values)
gives all the results.

| Value | What it measures | Range | What the value means in practice |
|---|---|---|---|
| `detail` | Fine detail and sharp edges | 0 and up | Lower usually means more blurred. Sharp test photos gave 1,200 to 2,200. A slight blur (1 px) gave 60 to 110, and a strong blur (4 px) gave less than 2. A plain area or fog also gives a low value, and noise increases it. |
| `directionality` | How much stronger the edges are in one direction | 0 to 1 | Motion blur gave 0.46 to 0.68. Sharp photos, and photos blurred equally in all directions, gave 0.03 to 0.15. |
| `blur_direction` | The direction of the weakest edges, in degrees counter-clockwise from horizontal | 0 to 180, or `None` | When `directionality` is 0.4 or more, this is the direction of the motion. It is `None` when `directionality` is 0. |
| `dark_clipped` | The part of the picture that is black (luma 5 or less) | 0 to 1 | 0.13 means 13 % of the pixels are black. Test photos made 80 levels darker gave 0.13 to 0.54. |
| `bright_clipped` | The part of the picture that is white (luma 250 or more) | 0 to 1 | Test photos made 80 levels brighter gave 0.05 to 0.13. |
| `contrast_spread` | The luma of the brightest 5 % of the pixels minus the luma of the darkest 5 % | 0 to 255 | Test photos gave 160 to 182. At 30 % of their contrast, they gave 48 to 55. |
| `noise` | Random variation from pixel to pixel, as a standard deviation in levels of 255 | 0 and up | Smooth pictures give less than 1. Fine texture, such as fur or grass, also counts as noise: the test photos gave 2.1 to 6.1 with no added noise. |
| `colourfulness` | How strong and varied the colours are | 0 and up | 0 is grey. On the scale of its authors, Hasler and Süsstrunk (2003), 15 is slightly, 33 moderately, 45 averagely, 59 quite, 82 highly and 109 extremely colourful. The test photos gave 41 to 96. |

The measurement of `detail`, `directionality`, `blur_direction` and `noise` uses a smaller copy
of the picture. Its longer side is at most 512 px (`SignalsConfig(analysis_side=512)`). Thus
one picture gives almost the same values at different resolutions. A smaller copy has less
noise, so compare `noise` only between pictures of similar size. For a copy smaller than 3 px
on a side, these four values are `None`.

The inputs can be `ImageBytes` (the bytes of a file), `ImageArray` (an RGB array) or a `Frame`
from the video module. The [image signals guide](docs/image-signals.md) gives the exact
definitions.

## Tag a picture or a frame

A tag is one thing that a picture shows, as the model writes it, such as `cat`, `outdoors`
or `blue_background`. The tagger is WD14, a model that gives a score from 0 to 1 for each of
9,083 tags. Download it first with `python -m curu_vision fetch wd-v1-4-swinv2-tagger-v2`.

```python
from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.tagging import load_tagger, select_tags

tagger = load_tagger()  # loads the model files that the fetch command downloaded
(tags,) = tagger.tag([ImageBytes(Path("cats.jpg").read_bytes())])
for tag in select_tags(tags, general=0.35, character=0.85):
    print(tag.name, tag.category, round(tag.score, 2))  # cat general 0.93, and more
```

`tags.scores` holds the score of each tag in `tags.labels`. `select_tags` gives the tags with a
score above your thresholds, highest first. You give each threshold, because curu-vision has
no default. The model's author uses 0.35 for general tags and 0.85 for characters.

What the values mean in practice:

| Value | What it is | Range | What the value means in practice |
|---|---|---|---|
| `score` | How sure the model is that the tag applies | 0 to 1 | Higher means more evidence. It is not a probability that a person checked. On three sample photos, the median score was about 0.00001 or less, and 11 to 13 general tags were above 0.35. |
| `category` `"general"` | A tag for content, such as `cat`, `soccer` or `outdoors` | 6,947 tags | The sample photos gave their highest general scores from 0.93 to 0.98. |
| `category` `"character"` | The name of a character from anime, manga or games | 2,132 tags | On the sample photos, no character score was above 0.002. |
| `category` `"rating"` | A content rating: `general`, `sensitive`, `questionable` or `explicit` | 4 tags | Each of the four has its own score. `select_tags` never returns a rating. |

A frame from the video module is an input too: `tagger.tag(frames.frames)`. Its time goes into
the provenance. A tag name is as the model gives it, with underscores. If no score is above
your thresholds, the result is an empty list. That is a result, not an error. The
[tagging guide](docs/tagging.md) gives the full rules and more measured values.

## Ask a vision-language model about a picture

A vision-language model (VLM) reads pictures and text, and writes text. Use it to describe a
picture, to answer a question about it, or to fill a form that you define. This example asks a
local Ollama server for the two main colours of a picture. First, start Ollama. Then run
`ollama pull qwen3-vl:2b-instruct`.

```python
from pathlib import Path

from pydantic import BaseModel

from curu_vision.embedding import ImageBytes
from curu_vision.vlm import BlockingVlm, OllamaConfig, VlmRequest


class Colours(BaseModel):  # the form that the reply must fill
    main_colour: str
    second_colour: str


config = OllamaConfig(model="qwen3-vl:2b-instruct", temperature=0.1, num_ctx=4096, attempts=3)
picture = ImageBytes(Path("photo.jpg").read_bytes())
with BlockingVlm(config) as vlm:  # reads the model's digest and capabilities from the server
    request = VlmRequest(prompt="Name the two main colours.", images=(picture,), output=Colours)
    result = vlm.ask(request)
print(result.value)  # main_colour='green' second_colour='blue', for example
print(result.images[0].source_size, result.images[0].sent_size)  # (640, 480) (640, 480)
print(result.usage.prompt_tokens)  # about 1,050 for one picture on Ollama 0.40.11
```

You give the temperature, because a server's default temperature can make a model add detail
that is not in the picture. Each request sends the picture again as a new JPEG, with its longer
side at most 1,024 px and no metadata. `output` is a pydantic class (a subclass of `BaseModel`).
The reply must fill it, else curu-vision sends the request again. It sends it at most `attempts`
times in total. Without `output`, `result.text` is the reply text and `result.value` is `None`.

What the values mean in practice:

| Value | What it is | What the value means in practice |
|---|---|---|
| `result.text` | The reply as the model wrote it | Never empty. A reply with no text is an error. |
| `result.value` | The reply as your form, for a request with `output` | It always matches the form. |
| `result.images` | Each picture as it was sent | `source_size` and `sent_size` are (width, height). Use them to map a position in the reply back to the picture. |
| `result.usage.prompt_tokens` | The size of the request in tokens | Ollama 0.40.11 gave 1,042 to 1,075 tokens for one picture. With `qwen3-vl:2b`, a picture of 2,688 × 3,840 px at full size needed about 10,100. A higher value makes the request slower and needs more memory. A value above the context gives `InputTooLong`, or on some runners a very slow request. |
| `result.reasoning` | The model's own reasoning text, if the server sent it | Usually `None`. Some models reason when you ask them not to. |
| `result.attempts` | Each attempt, with its reply and why it was rejected | One item when the first reply was correct. |

In async code, use `async with open_vlm(config) as vlm:` and `await vlm.ask(request)`. For a
llama.cpp server or vLLM, use `OpenAICompatibleConfig`. For a cloud model, use `CloudConfig`
with the `cloud` extra. The [VLM guide](docs/vlm.md) gives each setting, each server and each
error.

## Search by meaning

An embedding is a list of 768 numbers (a vector) that represents the meaning of a text or a
picture. Texts and pictures with similar meanings have vectors that are near each other. So
you can search pictures with a text, or texts with a picture.

```python
from pathlib import Path

from curu_vision.embedding import Document, ImageBytes, Query, load_embedder

embedder = load_embedder()  # one shared instance for each configuration, safe for threads
query, doc, image = embedder.embed(
    [
        Query("cats sleeping on a couch"),
        Document("Two cats asleep on a pink blanket.", title="Cats"),
        ImageBytes(Path("cats.jpg").read_bytes()),
    ]
)
print(query.vector @ doc.vector, query.vector @ image.vector)  # cosine similarities
print(image.provenance)  # the model, its revision and file hashes, the preprocessing, ...
```

What the results mean:

- Each vector has a length of 1, so `a.vector @ b.vector` is the cosine similarity. A higher
  value means a nearer meaning.
- Use the scores to sort results for one query. They are not probabilities. In a test with
  three photos, a text and the photo that it describes gave 0.70 to 0.75. A text and a
  different photo gave 0.52 to 0.61.
- Compare scores only between pairs of the same kind. Two photos of different animals gave
  0.71, which is near the score of a match between a text and a photo.
- Give a search text as a `Query` and a text to find as a `Document`, with an optional title.
  The model adds a different prompt to each, and the prompt is part of the vector space.
- Give a picture as `ImageBytes` (the bytes of a file) or `ImageArray` (a `uint8` array).
  curu-vision never reads a path string as a picture.

`EmbeddingConfig` sets how the model runs:

- `dimension`: 768, 512, 256 or 128 numbers. A shorter vector has a length of 1 again.
- `image_token_budget` and `video_token_budget`: 70, 140, 280, 560 or 1120 (280 and 140 are the
  defaults). A larger budget keeps more detail of each picture and takes more time. These two
  values have the largest effect on speed and quality.
- `providers`: the default is `("cpu",)`. `("cuda",)` uses an NVIDIA GPU, and `("coreml",)`
  uses Core ML on an Apple computer. A provider that is not there is an error.

## Search video, sound and scenes by meaning

The model also makes one vector for a video clip, a sound or a full scene. First, get the sound
track as 16 kHz audio with ffmpeg:

```bash
ffmpeg -i episode.mkv -vn -ac 1 -ar 16000 -c:a pcm_s16le episode.wav
```

```python
import wave

import numpy as np

from curu_vision.embedding import Audio, Combined, Query, load_embedder
from curu_vision.media import VideoSource, load_media

media = load_media()
embedder = load_embedder()
source = VideoSource(path="episode.mkv")
shot = media.shots(source).shots[0]
clip = media.shot_frames(source, shot, rate=1.0)  # one frame each second

with wave.open("episode.wav", "rb") as file:  # 16 kHz, 16-bit
    samples = np.frombuffer(file.readframes(file.getnframes()), np.int16)
first, last = int(shot.start * 16_000), int(shot.end * 16_000)
sound = Audio(samples[first:last], sample_rate=16_000, start=shot.start)

query, video, scene = embedder.embed(
    [
        Query("a door bell rings in a busy coffee shop"),
        clip.to_video(),
        Combined((clip.to_video(), *sound.split())),  # picture and sound together
    ]
)
print(query.vector @ video.vector, query.vector @ scene.vector)
print(video.provenance.video.frame_times)  # the source times of the frames that the model used
```

What the inputs and results mean:

- For a `Video`, the model uses one frame each second and at most 32 frames, spread over the
  clip. It selects frames by their own times.
- Give one shot or one scene as a clip. To cut a long video into clips, use
  [`media.shots`](#find-the-scenes-in-a-video).
- An `Audio` input must be 16 kHz and from 10 ms to 30 s long. Other rates and longer audio
  fail with a named error. curu-vision never resamples audio, and only `Audio.split()` cuts
  it.
- `Audio.split()` cuts audio into parts of 30 s or less, each with its start time.
- A `Video` takes (time, picture) pairs from any decoder. curu-vision reads a generator one
  time, so make a new `Video` each time that you use it.
- `Combined` makes one vector from its parts, in their order. This is the scene vector, which
  a catalogue can store. About 150 s of sound fits with 32 frames.

The [embedding guide](docs/embedding.md) gives all the input rules.

## Make a transcript

A transcript gives each spoken word with its start and end time. First, download a
transcription model (see [Download the models](#download-the-models)). Then get 16 kHz audio
with ffmpeg:

```bash
ffmpeg -i interview.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le speech.wav
```

```python
import wave

import numpy as np

from curu_vision.transcription import Audio, Transcript, TranscriptionConfig, load_transcriber

with wave.open("speech.wav", "rb") as file:  # 16 kHz, 16-bit
    samples = np.frombuffer(file.readframes(file.getnframes()), np.int16)

transcriber = load_transcriber()  # Parakeet on the CPU
# load_transcriber(TranscriptionConfig(engine="moss", device="cuda"))  # with speakers, on a GPU
(result,) = transcriber.transcribe([Audio(samples, 16_000, start=0.0)])
if isinstance(result, Transcript):
    print(result.text)
    for word in result.words:
        print(f"{word.start:8.2f} {word.end:8.2f} {word.text}")  # seconds in the source
```

What the results mean:

- Word times are in seconds in the source. `Audio(start=...)` is the time of the audio's
  first sample in the source.
- `result.segments` holds the segments: the words of one speaker turn. With MOSS, each
  segment has a speaker label, such as `S01`. With Parakeet, the speaker is `None`.
- `result.word_timing` is `"measured"` when the engine measured each word's time (Parakeet). It
  is `"spread"` when the engine spread the words over their segment (MOSS).
- A transcript with no words means that the engine found no speech. A failure is an
  `ItemFailure` with a named error, never an empty transcript.
- Audio of 0.08 s or more is correct, with no maximum length. The engine transcribes long
  audio in windows and joins them.

The engines:

| Engine | Configuration | Extra | Speakers |
|---|---|---|---|
| Parakeet on the CPU | `TranscriptionConfig()` | none | no |
| Parakeet on Apple silicon | `TranscriptionConfig(device="mlx")` | `mlx` | no |
| MOSS on an NVIDIA GPU | `TranscriptionConfig(engine="moss", device="cuda")` | `gpu` | yes |
| MOSS on the CPU | `TranscriptionConfig(engine="moss", device="cpu")` | `gpu` | yes |

A missing extra or device is an error when the engine loads. curu-vision never uses a
different engine instead. The [transcription guide](docs/transcription.md) tells how the
engine joins long audio and what the other fields mean.

## Read a video from bytes, a stream or a URL

A `VideoSource` is a local path, a URL, the bytes of a file or a stream:

```python
from pathlib import Path

from curu_vision.media import VideoSource, load_media

media = load_media()
data = Path("episode.mkv").read_bytes()
print(media.frames_at(VideoSource(data=data), [12.0]).frames[0].time)  # same as the path

with Path("episode.mkv").open("rb") as file:  # curu-vision reads a stream one time, from the start
    frames = media.frames_every(VideoSource(stream=file), rate=1.0, end=20.0)
print(len(frames.frames))
# VideoSource(url="https://example.com/clip.mp4") reads any URL that ffmpeg can open
```

- Bytes and streams never go to the disk. Bytes give the same results as their path.
- A stream cannot go back, so curu-vision reads it from the start, one time. Give `end` to
  `frames_every` for a stream.
- An MP4 or MOV file with its index after its frames cannot come from a stream. It fails as
  `NeedsSeekableInput`: give its bytes or its path instead.

## Process many files in one call

```python
from curu_vision.embedding import ItemFailure
from curu_vision.media import VideoSource, load_media

media = load_media()
results = media.shots_many([VideoSource(path="episode.mkv"), VideoSource(path="broken.mkv")])
for result in results:
    if isinstance(result, ItemFailure):
        print(result.index, type(result.error).__name__, result.error)  # 1 NotAVideo ...
    else:
        print(len(result.shots), "shots")
```

- A batch call gives one result for each input, in order. An input that fails gives an
  `ItemFailure`, and the other inputs continue.
- `probe_many` and `shots_many` use a pool of threads. Its size is `MediaConfig(workers=...)`,
  and the default is the CPU quota.
- `signals.compute`, `tagger.tag`, `embedder.embed` and `transcriber.transcribe` also take a
  list. In a test
  on 2026-10-26 on an x86-64 computer, 4 threads gave the image signals of about 300
  pictures of 1,024 × 1,024 px each second.
- The video and image signals calls release the Python lock (the GIL). Thus calls from
  different threads run at the same time.

## Handle errors

curu-vision never hides an error as an empty result. All errors are in `curu_vision.errors`.

```python
from curu_vision.errors import InputError, TransientError
from curu_vision.media import MediaConfig, VideoSource, load_media

media = load_media(MediaConfig(idle_timeout=120.0, probe_timeout=30.0, wall_timeout=None))
try:
    media.probe(VideoSource(path="broken.mkv"))
except InputError as error:  # the file is bad: correct it or skip it
    print(type(error).__name__, error)  # NotAVideo ...
except TransientError as error:  # the source or the computer failed: try again later
    print(type(error).__name__, error)
```

| Error | What it means | What to do |
|---|---|---|
| `InputError`, for example `UndecodableImage`, `NotAVideo`, `TimeOutOfRange` | One input is bad. A batch gives it as an `ItemFailure`. | Correct or skip that input. |
| `TransientError`: `SourceTimeout` or `SourceUnavailable` | The source or the computer failed while curu-vision read it, for example a network share that stopped. A batch of the video module gives it as an `ItemFailure`. | Try again later. |
| `TransientError`: `ServerUnavailable` or `ServerTimeout` | A model server did not answer, or it was silent for longer than the timeout. | Start the server, or try again later. |
| `ModelNotAvailable`, `WeightsVerificationFailed` | The model files are missing or changed, or a model server does not have the model. The message names the `fetch` command, or `ollama pull` for a model on Ollama. For another server, it gives the models that the server has. | Run the command that the message names, or name a model that the server has. |
| `InputTooLong` | A request is longer than the model's context. | Make the context larger, or send smaller pictures. |
| `InvalidModelOutput` | A model's reply has no text, stops at the output limit, or never fits your form. | Change the prompt, the form or the model. |
| `AuthenticationFailed`, `MissingCredential` | A server refused the token, or the variable of the token is not set. | Set the variable that the message names. |
| `CapabilityMissing` | The served model cannot read pictures, or cannot reason as the settings ask. | Use a different model, or change the setting. |
| `MediaToolMissing` | ffmpeg or ffprobe is missing or too old. | Install a correct version. |
| `RuntimeUnavailable` | An extra or a device is missing. | Install the extra, or use a different device. |
| `ImportError` | `open_vlm` got a `CloudConfig`, and the `cloud` extra is not installed. | Install `curu-vision[cloud]`. |
| Other `InfrastructureError` | The tools, the model or the computer failed. `TransientError` is one type of it. | Correct the setup. |

A failure of the model, the computer or the runtime stops the full call. There are two
exceptions. A batch of the video module gives a `TransientError` as an `ItemFailure`. In a
transcript batch, `InvalidModelOutput` or `InsufficientMemory` for one item gives an
`ItemFailure` for that item only. curu-vision never uses a different model or engine
instead.

A video that is cut off after some frames is not an error. You get the frames that decoded,
and `decoded_until` gives the time where they stop.

`MediaConfig` sets the time limits of each child process of ffmpeg or ffprobe. The example
shows the default values. The idle limit stops a child process that reads or writes no byte
for that time. Thus a long decode that continues to make progress does not stop. The wall
limit, if you set it, limits the full time of each call. The [video guide](docs/video.md)
lists each error.

## Record what made a result

Each result has a `provenance` field. It tells exactly what made the result, so you can store
it with the result. When a method changes, its version changes, and you can find old results.

```python
from pathlib import Path

from curu_vision.embedding import ImageBytes
from curu_vision.signals import load_signals

(result,) = load_signals().compute([ImageBytes(Path("photo.jpg").read_bytes())])
record = result.provenance
print(record.hash_method, record.hash_version, record.measure_version, record.package_version)
```

- An embedding names the model, its revision, the SHA-256 of each file, the package version,
  the preprocessing version and the configuration. For video and audio, it also gives the
  times of the frames and samples that the model used.
- A video result names the method and its version, each value in effect, the ffmpeg and
  ffprobe versions and the identity of the source.
- An image signals result names the hash method and version and the measurement version. It
  also gives each parameter, the decode version, the package version and the two picture sizes.
- A transcript names the engine, the runtime, the device, the model, its revision and the
  SHA-256 of each file. It also gives the package and preprocessing versions and the audio
  parts that the engine used.
- A tag result names the model, its revision, and the SHA-256 of the model file and of the
  tag list. It also gives the preprocessing, decode, package and ONNX Runtime versions, the
  picture size, and the time of a frame.
- A vision-language model result names the server, its version, the model and its digest, or
  says that the server gave no digest. It also gives the model name in the server's reply and
  each setting that was sent. It gives the version of the picture preparation, the package
  version and the time of a frame.

## Limits

- curu-vision is a library. It does not store results, run a pipeline or keep a queue. Its
  only command downloads models.
- It does not decode audio. Your program gives it 16 kHz samples, for example from ffmpeg.
- pHash does not find copies of smooth gradients or noise fields. In the tests, copies of these
  moved 4 to 32 bits.
- pHash does not find copies of pictures smaller than 32 px on a side. A 9 × 8 px picture
  moved 28 bits at half size.
- pHash does not find cropped or mirrored copies (see
  [Find copies of a picture](#find-copies-of-a-picture)).
- The shot finder can find false cuts inside slow dissolves and other gradual transitions.
- HDR video from a `stream` source fails as `UnsupportedSource`. Give it as bytes, a path or
  a URL.
- A call decodes one video stream: the first, or the stream that `video_stream` selects.
- curu-vision has no prompts and no rules for captions or judgements. Your program gives the
  prompt, the form of the reply and the check.
- curu-vision does not pull, start or download a vision-language model. A llama.cpp server or
  vLLM gives no digest, so a result from it cannot prove which model file it used.
- Some model servers ignore the context size: Ollama's MLX runner sent a request of 10,105
  tokens with a context of 4,096. Look at `result.usage.prompt_tokens`.
- The tagger knows only the 9,083 tags of its model. A tag that is not in that list has no
  score, not a score of 0.
- The tagger learned from illustrations (Danbooru tags). On photos it gives tags such as
  `realistic`, and some tags are wrong: a tiger photo got `signature`.
- A tag score is not calibrated: a score of 0.9 does not mean that the tag is correct 9 times
  in 10. Find the thresholds that are correct for your pictures.

## Technical notes

- **Build.** The package includes a Rust extension, `curu_vision._core`. maturin, a build tool
  for Rust extensions, builds it one time for Python 3.12 and later (abi3, the stable Python
  ABI). uv builds it from the release tag. Each video operation runs
  ffmpeg or ffprobe as a child process from the Rust core, with limits on its time.
- **Decoder.** curu-vision uses the system ffmpeg and ffprobe
  ([ADR-009](project-management/ADRs/ADR-009-decode-through-system-ffmpeg-from-rust.md)).
- **ffmpeg on arm64.** Releases before 8.1 convert frames to RGB with too much error on arm64,
  so `load_media()` requires 8.1 there
  ([ADR-010](project-management/ADRs/ADR-010-ffmpeg-8-1-minimum-on-arm64.md)).
- **Hashes.** The hashes are equal to imagehash 4.3.2's `phash` and `dhash` on Pillow 13.0.0
  with the x86-64 wheels of scipy 1.18.1. The installed versions of these packages have no
  effect. Rust code does Pillow's luma,
  Pillow 13's LANCZOS resize and scipy's DCT, bit for bit. `f"{h:016x}"` is imagehash's
  `str(hash)`.
- **Hashes on arm64.** The arm64 wheels of scipy fuse multiply-adds in this DCT. Thus imagehash
  on arm64 can give different bits for block graphics. curu-vision gives the x86-64 values on
  every platform.
- **Hashes and shots on a Mac.** The DCTs take their cosines and sines from glibc's
  constants, not from the maths library of the platform. Under `hash-1`, a Mac gave different
  hashes for some block graphics. Under `shots-2`, a Mac can give different shots for some
  videos. Compute each hash whose `hash_version` is `hash-1` again. Compute each shot list
  whose `method_version` is `shots-2` again.
- **Shots.** The shot method is `shots-3`. It finds cuts, fades and dissolves. Its name and all
  its parameters are in the provenance of each result.
- **Embedding.** The embedding model is
  [EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2), on ONNX Runtime, with no
  PyTorch when it runs ([ADR-002](project-management/ADRs/ADR-002-embeddinggemma-2-embedding-capability.md)).
- **Transcription engines.** Parakeet-TDT-0.6b-v3 runs on the CPU with ONNX Runtime, and on
  Apple silicon with MLX. MOSS-Transcribe-Diarize runs on NVIDIA GPUs
  ([ADR-006](project-management/ADRs/ADR-006-transcription-engine.md)). PyTorch and MLX are
  optional extras ([ADR-007](project-management/ADRs/ADR-007-audio-runtime-optional-extras.md)).
- **Tagging.** The tagger is
  [WD14 SwinV2 v2](https://huggingface.co/SmilingWolf/wd-v1-4-swinv2-tagger-v2) at a pinned
  revision, on ONNX Runtime, on the CPU by default. Its input and its scores are equal to its
  author's reference tagger code. The model reads BGR pixels from 0 to 255, padded to a white
  square of 448 px.
- **Vision-language models.** curu-vision reaches Ollama through Ollama's own `/api/chat`,
  because Ollama's OpenAI-compatible endpoint ignores `think` and `num_ctx`. Other servers use
  `/v1/chat/completions`, and cloud models use PydanticAI
  ([ADR-014](project-management/ADRs/ADR-014-vlm-port-transports-and-served-model-identity.md)).
  The tests use Ollama 0.40.13 with `qwen3-vl:2b-instruct` and `qwen3-vl:2b`.
- **Type check.** The type check runs in a temporary environment with the `reference` group and
  the `gpu` extra ([ADR-008](project-management/ADRs/ADR-008-type-check-in-an-isolated-venv.md)).
- **API contracts.** Each module has an API contract. It gives the exact types, rules and
  errors of each call:
  - [Embedding](specs/001-embedding-port/contracts/python-api.md)
  - [Video and audio inputs for the embedding](specs/002-video-and-audio-inputs/contracts/python-api.md)
  - [Transcription](specs/004-transcription-port-with-timed-words/contracts/python-api.md)
  - [Video probe, frames and shots](specs/007-video-probe-decode-shots/contracts/python-api.md)
  - [Image hashes and measurements](specs/008-perceptual-hashes-and-quality-metrics/contracts/python-api.md)
  - [Tags](specs/009-wd14-tagging-port/contracts/python-api.md)
  - [Vision-language models](specs/010-vlm-port/contracts/python-api.md)
- **Rules and decisions.** The rules for each change are in `.specify/memory/constitution.md`,
  and the decisions are in `project-management/ADRs/`.
- **The curu packages.** [How curu-vision fits with the other curu packages](docs/ecosystem.md).

## Develop

```bash
uv sync
uv tool install beacon-framework                  # one time: the beacon command
uv run ruff check && uv run ruff format --check
uv run --isolated --group reference --extra gpu ty check   # in a temporary environment
cargo fmt --check --manifest-path crates/curu-vision-core/Cargo.toml
# Python from uv: pyo3 builds against it (clippy, test); the test binary links uv's
# libpython and finds it at run time (on Linux the bare `cargo test` fails with `unable to
# find library -lpython3.14` or cannot load libpython3.14.so.1.0)
export PYO3_PYTHON="$(uv run python -c 'import sys; print(sys.executable)')"
LIBDIR="$("$PYO3_PYTHON" -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')"
cargo clippy --manifest-path crates/curu-vision-core/Cargo.toml -- -D warnings
RUSTFLAGS="-L $LIBDIR -C link-arg=-Wl,-rpath,$LIBDIR" \
  cargo test --manifest-path crates/curu-vision-core/Cargo.toml
uv run pytest                                     # hermetic: no network, no weights
uv sync --group reference                         # real weights vs the reference, one file a process
for f in tests/integration/test_*.py; do uv run pytest -m integration -s "$f"; done
uv run pytest -m apple                            # the MLX path, on Apple silicon
uv run pytest -m live                             # a running Ollama 0.40.13 (see below)
CURU_VISION_EVAL_SET=/path/to/set uv run pytest -m evaluation  # your labelled or golden set
CURU_VISION_EVAL_SET=/path/to/golden-v1 CURU_VISION_SHOT_SET=/path/to/rai \
  uv run pytest -m evaluation tests/evaluation/test_shots.py   # shot boundaries
beacon docs check                                 # the writing standard, docs/WRITING.md
```

The hermetic tier needs ffmpeg 6.0 or later, and 8.1 or later on arm64. It also needs the
`beacon` command on the path. Its HDR tests need the `zscale` filter (`libzimg`), and they
skip without it, as on Homebrew's ffmpeg 8 and 9. To run every gate on Linux, run
`tests/containers/matrix.sh` (Docker). It uses Ubuntu's ffmpeg 6.1.1 (not on arm64) and the
8.1.3 and 9.0.1 releases, each in its own container. See `tests/containers/README.md`.

The `live` tier needs Ollama 0.40.13 with `qwen3-vl:2b-instruct` and `qwen3-vl:2b` at the
digests that the tests name. `CURU_VISION_OLLAMA_URL` gives its address, else
`http://localhost:11434`. The README example test needs Ollama at `http://localhost:11434`.
A live test fails, and does not skip, when its server is not there.
`tests/live/test_vlm_openai.py` also needs a llama.cpp server. Start it in a second terminal.
Then run the test with its address:

```bash
llama-server -hf ggml-org/Qwen3-VL-2B-Instruct-GGUF:Q8_0 --ctx-size 4096 --jinja --port 8089
CURU_VISION_OPENAI_URL=http://localhost:8089 uv run pytest -m live tests/live/test_vlm_openai.py
```

To find whether each ffmpeg build gives the shots of a video the same frame numbers, use
`tools/shot_frames.py`. The section "Measure the shot frame numbers on each ffmpeg build" of
`tests/containers/README.md` gives the procedure.

Write `README.md`, `docs/` and `AGENTS.md` to the [writing standard](docs/WRITING.md).
`AGENTS.md` gives the rules for each change.

## Glossary

- **Analysis size**: the smaller copy of a picture that some measurements use. Its longer side
  is at most 512 px.
- **BGR**: the order blue, green, red of the three colour values of a pixel. Most files use RGB.
- **Boundary**: the place where one shot stops and the next shot starts. It is a cut or a
  gradual transition.
- **Category**: the kind of a tag: general, character or rating.
- **Clipped**: a pixel at the end of the brightness range, black or white, that has lost its
  detail.
- **Context**: the most tokens that a model reads and writes in one request. Ollama calls it
  `num_ctx`.
- **Cosine similarity**: a measure of how near two vectors are, from -1 to 1. Higher is nearer.
- **CPU quota**: the number of CPU threads that the container or the computer lets the process
  use.
- **Cut**: a sudden change from one shot to the next.
- **Danbooru**: a website of illustrations whose tags WD14 learned.
- **DCT** (discrete cosine transform): a calculation that changes a picture into frequencies.
  pHash uses it.
- **dHash** (difference hash): a 64-bit hash from the brightness differences of adjacent pixels.
- **Digest**: the SHA-256 of a model as a model server reports it.
- **Edge**: a line where the brightness of a picture changes quickly. Blur makes edges weaker.
- **Embedding**: a vector that represents the meaning of a text, a picture, a video or a sound.
- **Event loop**: the part of Python's `asyncio` that runs async code.
- **Extra**: an optional part of the package that adds dependencies, such as `gpu` or `mlx`.
- **Gaussian blur**: a blur that is equal in all directions. Its size σ is in pixels.
- **glibc**: the GNU C library, the usual C library on Linux. It gives the same cosines and
  sines on x86-64 and arm64.
- **GIL**: Python's global interpreter lock. A call that releases it lets other Python threads
  run at the same time.
- **Gradual transition**: a change from one shot to the next over a number of frames, such as
  a fade or a dissolve.
- **Hamming distance**: the number of bits that are different between two hashes.
- **HDR**: high dynamic range video. curu-vision converts it to standard RGB frames.
- **`ItemFailure`**: the result for one input that failed in a batch. It holds the input's
  `index` and the `error`.
- **LANCZOS**: a method to resize a picture.
- **JSON schema**: a description of the fields and types that a JSON value must have.
- **Label**: one tag of the model's list, with its category.
- **Laplacian**: a calculation that finds edges. `detail` is the variance of the Laplacian of
  the luma.
- **llama.cpp server**: a model server that gives an OpenAI-compatible endpoint.
- **Luma**: the brightness of a pixel, from 0 (black) to 255 (white).
- **Model server**: a program that runs a model and answers requests over HTTP, such as Ollama.
- **Ollama**: a model server for local models.
- **ONNX Runtime**: the library that runs a model file in the ONNX format.
- **OpenAI-compatible endpoint**: `/v1/chat/completions`, the request form that many model
  servers accept.
- **pHash** (perceptual hash): a 64-bit hash from the large shapes of a picture. Copies have
  near hashes.
- **PydanticAI**: a Python library that sends requests to cloud models.
- **Presentation time**: the time in the source at which a frame shows on the screen.
- **Provenance**: the record of what made a result: the method, its version, the model and the
  values in effect.
- **Runner**: the program inside Ollama that runs a model, such as llama.cpp or MLX.
- **Scene**: one shot or a few shots together.
- **Segment**: the words of one speaker turn in a transcript.
- **Shot**: a continuous run of frames from one camera take.
- **Tag**: one thing that a picture shows, as the model writes it, such as `cat`. A tag has a
  name, a category and a score.
- **Temperature**: how random a model's reply is. Lower gives replies that change less.
- **Threshold**: the score that a tag must be above to be selected. You choose it.
- **Token budget**: the number of tokens that the model uses for one picture. A token is one
  small part of the input. More tokens keep more detail.
- **Vector**: a list of numbers. An embedding is a vector.
- **vLLM**: a model server that gives an OpenAI-compatible endpoint.
- **VLM** (vision-language model): a model that reads pictures and text, and writes text.
- **WD14**: the family of tagging models that SmilingWolf trained on Danbooru tags.
- **Window**: one part of long audio that the engine transcribes in one call.

