Metadata-Version: 2.4
Name: vidwit
Version: 1.0.0
Summary: Multimodal video witness — local CLI that turns a video into an exhaustive markdown record.
Project-URL: Homepage, https://github.com/kalsan/vidwit
Project-URL: Repository, https://github.com/kalsan/vidwit
Project-URL: Issues, https://github.com/kalsan/vidwit/issues
Project-URL: Changelog, https://github.com/kalsan/vidwit/releases
Author-email: Sandro Kalbermatter <sandro@storabble.com>
License-Expression: LGPL-2.1-or-later
License-File: LICENSE
Keywords: accessibility,ffmpeg,llm,markdown,multimodal,transcription,video,whisper
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.11
Requires-Dist: faster-whisper>=1.0
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# vidwit

vidwit is a multimodal video witness. It reads a video and writes an
exhaustive Markdown record that combines verbatim transcript citations
with detailed visual descriptions, so that a reader of the Markdown
file knows everything a viewer of the video would have known.

This is mainly intended as a tool that allows LLMs to "watch" a video.
The idea is that you run vidwit first and feed the output to the LLM,
after which it should have most relevant information found in the video.

## What you get

For any input video, vidwit produces a single Markdown document that:

- tells the entire story of the video, in order, as a continuous
  narration;
- embeds transcript citations verbatim, with speaker labels when they
  are known;
- annotates each citation and scene with a description of what is
  visually shown at that moment, including subjects, actions, spatial
  relations, environment, and any on-screen text quoted verbatim;
- tags every block with a half-open timecode range
  `[MM:SS.mmm – MM:SS.mmm)` so that a reader can locate the
  corresponding slice in the source video;
- collects content warnings into an index at the top of the document,
  so a reader can decide which sections to skip — for instance graphic
  violence or NSFW imagery — without watching the video first.

The reader test for the output is straightforward: *if I read this
Markdown file, I should know every meaningful fact that a viewer of
the video would have learned.*

## Example output

```
### [02:14.300 – 02:31.800) — Tiger and cub at the riverbank [FOOTAGE]

> "The tiger has a young cub." — Narrator, 02:18.100

Visual: A large adult tiger sits on the left bank of a shallow river.
A small cub (about a quarter of the adult's size) sits to the tiger's
right and licks its own front paw. Background: dense green forest,
late afternoon light.

### [04:02.000 – 04:21.500) — Tiger hunts gazelle [FOOTAGE] [⚠ graphic]

> "Once it spots its prey…" — Narrator, 04:03.400

Visual: The adult tiger sprints across the savannah and takes down a
gazelle. A close-up shows the bite to the gazelle's neck; blood is
visible from 04:18. Skip 04:17 – 04:21 to avoid the blood.
```

## Installation

vidwit requires Python 3.11 or newer, along with `ffmpeg` and
`ffprobe` on the system `PATH`.

### From PyPI (recommended)

```bash
python3 -m venv .venv
.venv/bin/pip install vidwit
```

Or, with [pipx](https://pipx.pypa.io/) for an isolated user install:

```bash
pipx install vidwit
```

### From source (development)

```bash
git clone https://github.com/kalsan/vidwit.git
cd vidwit
python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
```

vidwit pulls `faster-whisper`, `ctranslate2`, and a few small
supporting libraries. PyTorch is **not** required, because
`faster-whisper` runs inference through `ctranslate2`. The Anthropic
and OpenAI Python SDKs are also not required, because vidwit uses the
standard library's `urllib` to call the HTTP endpoints directly.

### Invocation forms

After installation, the `vidwit` entry point lives at `.venv/bin/vidwit`
(or wherever pipx put it). You can invoke it in three equivalent ways:

```bash
.venv/bin/vidwit path/to/video.mp4       # explicit, no shell setup needed
source .venv/bin/activate                # then plain `vidwit ...` works
python3 -m vidwit path/to/video.mp4      # works without installing the script
```

The rest of this README uses the bare `vidwit` form for brevity. Prefix
with `.venv/bin/` or activate the venv first if the command is not on
your `PATH`.

The first run downloads the chosen whisper model into
`~/.cache/huggingface/hub/`. The default model is `small`, which is
roughly 480 MB on disk. For a smaller first download, choose `tiny`
(about 75 MB) or `base` (about 140 MB) via the `--whisper-model` flag
or the configuration file.

## Configuration

Drop a starter `vidwit.toml` into the current directory and paste in
your LLM API key:

```bash
vidwit init                # writes ./vidwit.toml
$EDITOR vidwit.toml
```

Or write it to `~/.config/vidwit/vidwit.toml` instead, so it applies
everywhere:

```bash
vidwit init --user
```

Pass `--prompt` to also drop a customisable system prompt
(`vidwit_prompt.md`) alongside the config. Pass `--force` to overwrite
existing files.

vidwit searches for its configuration file at `./vidwit.toml`, then at
`~/.config/vidwit/vidwit.toml`, or at an explicit path passed via
`--config <path>`. Settings are merged with the following precedence,
where higher entries win over lower ones: command-line flags,
environment variables, configuration file, and the built-in defaults.

Four LLM providers are supported out of the box:

| Provider    | Description                                                     |
|-------------|-----------------------------------------------------------------|
| `anthropic` | Anthropic Claude (Sonnet 4.6 by default, Opus 4.7 for highest fidelity). |
| `openai`    | OpenAI Chat Completions API.                                    |
| `lmstudio`  | Any OpenAI-compatible endpoint, such as LM Studio or vLLM.      |
| `dummy`     | A no-network placeholder useful for offline testing; it emits stub chunks instead of calling a real model. |

The starter templates ship inside the package at
`vidwit/templates/`; `vidwit init` copies them out for you. A populated
`./vidwit.toml` contains your API key and is excluded from version
control via `.gitignore`.

## Usage

```bash
vidwit path/to/video.mp4
vidwit path/to/video1.mp4 path/to/video2.mkv
vidwit path/to/dir/    # recurses, picks up files by extension
```

**If the `vidwit` command is not available, use the activation script or an alternative form (see above).**

For each input video `foo.mp4`, vidwit writes `foo.md` next to it in
the same directory and under the same basename. The default set of
video extensions is `.mp4`, `.mkv`, `.mov`, `.webm`, and `.avi`. You
can add further extensions with the `--ext` flag.

The most commonly used command-line flags are:

```
--fps FLOAT          frame sampling rate (default 1.0)
--window FLOAT       window length in seconds (default 10.0)
--overlap FLOAT      window overlap in seconds (default 1.0)
--overwrite          replace an existing .md output file
--no-resume          force re-run; ignore cached transcript/frames/chunks in scratch
--keep-scratch       keep the scratch directory after a successful run
--jobs N             threads for ffmpeg and whisper (default: nproc)
--paths home:DIR     write the final .md elsewhere (yt-dlp-style override)
--paths temp:DIR     place the scratch directory elsewhere
--default-speaker S  label transcribed words with the speaker name S
--prompt FILE        use a custom system prompt
--audio-language CODE   ISO language hint for whisper (e.g. "de"); skips auto-detect
--notes "TEXT"       free-text context forwarded to the LLM in every chunk
-o, --output PATH    explicit output file path (single-input only); relative or absolute
--frame-width N      downscale frames to fit within this width (default 256)
--frame-height N     downscale frames to fit within this height (default 144)
--max-tokens N       cumulative LLM token cap per video; abort + assemble what is done
--llm PROVIDER       anthropic | openai | lmstudio | dummy
--model NAME         model identifier
--base-url URL       OpenAI-compatible endpoint URL
--timeout SECONDS    HTTP timeout for each LLM call (default 600)
--extra-body K=JSON  extra field merged into the chat-completions payload;
                     repeatable. Example: --extra-body
                     'chat_template_kwargs={"enable_thinking":false}'
--whisper-model NAME tiny / base / small / medium / large-v3
```

## How it works

Because vision LLMs cannot see an arbitrarily long video in a single
call, vidwit iterates over fixed-length time windows and assembles the
results into one document. The core loop, written as pseudocode, is:

```
for window in windows(video, length=10s, overlap=1s):
    frames = ffmpeg frames within [window.start, window.end)
    words  = whisper word-level transcript within window
    chunk  = LLM(
        system  = vidwit_prompt,
        context = rolling summary + raw tail of recent chunks,
        input   = frames + words + window timecode + capture metadata,
    )
    write chunk to disk (resumable)
assemble chunks into final markdown (TOC + content warnings index)
write `<video>.md.part` and atomically rename it to `<video>.md`
```

The combination of fixed windows and a rolling context window is what
lets vidwit work on inputs that would never fit in a single LLM call.

### Why a rolling context is necessary

The rolling context allows the model to maintain continuity across
windows. Without it, the model would not remember introductions or
recurring entities. In particular:

- A narrator who refers to "the cub" several minutes after introducing
  it must still resolve correctly to the original animal.
- Re-appearing characters should be given the same name in every
  window where they appear.
- The model should not restate background information that has already
  been established in earlier windows.

### Capture metadata sent to the LLM

Every LLM call carries a `# Capture metadata` block that lists the
frame sampling rate, the window length, the overlap with neighbouring
windows, the source resolution, the whisper model used for
transcription, the total duration of the video, the language whisper
detected (with its confidence), and any user-supplied audio-language
hint or free-form notes. The model uses this metadata to calibrate
its description. In particular, visual events shorter than the frame
spacing may be missed in the supplied frames, and the model is
expected to infer those from the transcript instead.

### Unreliable transcripts and burned-in subtitles

Whisper is excellent for the languages it was trained on but can
produce gibberish for regional dialects or low-resource languages
(Swiss German is a common example). When the transcript reads as
gibberish, the affected timecodes are tagged with
`[⚠ transcript unreliable]` and the narration quote is skipped.

Burned-in subtitles are not promoted into the audio stream — they
are visual content. They stay in the visual description with a
source attribution, the same way any other on-screen text would
appear:

```
Visual: a man is speaking to camera; subtitle reads: "We are here."
```

Two flags help with these situations and can also be set under the
`[video]` table in `vidwit.toml`:

- `--audio-language CODE` forces whisper to decode in a specific
  language instead of auto-detecting; useful when auto-detect picks
  the wrong language for non-English speech.
- `--notes "TEXT"` accepts arbitrary free text — dialect, situation,
  burned-in subtitle presence — and forwards it verbatim to the LLM
  in every chunk's capture metadata.

### Resumability

Intermediate per-window outputs are stored under
`<video-dir>/.vidwit-tmp/<video-hash>/chunks/NNNN.md`. The whisper
transcript and the extracted frames are cached in the same scratch
directory. The final output is first written as `<video>.md.part`
and then atomically renamed to `<video>.md`.

**Resume is on by default.** If a run crashes, re-running the same
command picks up where it left off — cached transcript, frames, and
finished chunks are reused. To force a clean re-run from scratch, pass
`--no-resume`.

Scratch directories are cleaned up on success unless `--keep-scratch`
is passed.

Caveats:

- The scratch directory is keyed by video hash alone, not by model or
  by sampling settings. If you change `--fps`, `--window`,
  `--frame-width`, `--audio-language`, or the LLM model between runs,
  cached chunks and frames are no longer valid. Pass `--no-resume`
  for that run, or wipe the scratch directory first.

### Scratch and output paths

vidwit follows the yt-dlp convention for redirecting scratch and
output locations:

```
--paths home:/out          # write the final .md into /out
--paths temp:/fast-ssd     # place the scratch dir on /fast-ssd
```

By default, both the final output and the scratch directory live next
to the input video file.

## Project layout

```
vidwit/
  pyproject.toml
  vidwit/
    __init__.py
    __main__.py             # `python -m vidwit`
    cli.py                  # argument parsing, input expansion, per-file loop
    init_cmd.py             # `vidwit init` — copies starter config out of templates/
    config.py               # defaults + environment variables + TOML file merge
    pipeline.py             # per-video orchestration
    ffmpeg_io.py            # probe, audio extraction, frame extraction
    transcribe.py           # faster-whisper wrapper with word-level timestamps
    chunker.py              # window iteration with overlap
    llm.py                  # Provider protocol + Anthropic + OpenAI-compatible + dummy
    assembler.py            # chunk merge, table of contents, content warnings
    scratch.py              # paths, hashing, resume support
    templates/              # bundled starter files, dropped by `vidwit init`
      vidwit.toml
      vidwit_prompt.md
```

## Output format conventions

The system prompt enforces a strict block layout so that the output is
both readable and machine-parseable:

- Each block begins with a header of the form
  `### [MM:SS.mmm – MM:SS.mmm) — short title [TAG]`.
- Intervals are half-open (`[start, end)`) so that adjacent blocks
  never overlap.
- The tag indicates the kind of content in the block:
  - `[FOOTAGE]` for live-action camera shots,
  - `[ANIM]` for animation, infographics or title cards,
  - `[FOOTAGE + ANIM]` for live footage with overlaid text or graphics,
  - `[⚠ <reason>]` for content warnings, optionally followed by a
    recommended skip range underneath.
- On-screen text is quoted verbatim, with capitalisation preserved.
- Speech is rendered in `>` blockquotes, with a speaker label and a
  word-precise timecode taken from whisper.
- Non-speech audio is bracketed: `[lion roars]`, `[applause]`,
  `[wind]`.
- When speech crosses a window boundary, the quote ends with `…` and
  continues in the next block, starting with `…`.

## Roadmap

### v1 (current)

- Local-first CLI with no service and no database.
- ffmpeg, faster-whisper, and a vision LLM (Anthropic or any
  OpenAI-compatible endpoint).
- Iterative chunked loop with rolling context (sliding tail of the
  last couple of chunks).
- Resumable runs; yt-dlp-style scratch and output paths.
- Token-based cost ceiling via `--max-tokens N`: vidwit accumulates
  every LLM call's input + output tokens per video, and once the
  total exceeds the cap it stops issuing new calls, assembles
  whatever is already on disk, and writes the partial `<video>.md`.

### v2 — speaker awareness (if possible)

- Diarisation through `pyannote-audio`, merged with whisper's
  word-level timings.
- Face detection and re-identification through `insightface`. Faces
  are clustered across the video so that the same person receives a
  stable identifier throughout the record.

### v2+ — cross-video corpus state

The goal of corpus state is that when vidwit processes video B, it
already knows what happened in video A and can link, deduplicate, and
reuse identities.

```bash
vidwit --corpus ./my-docs/ ./my-docs/ep01.mp4
vidwit --corpus ./my-docs/ ./my-docs/         # process the whole directory
```

Per-corpus state lives in `<corpus_dir>/.vidwit/`:

```
.vidwit/
  entities.json   # stable id → {names, aliases, descriptors, embeddings}
  index.json      # video path → {hash, mtime, md_path, processed_at}
  faces/          # representative crops per face id
  voices/         # speaker embeddings per voice id
```

Concrete benefits of corpus state include:

- **Recurring entities.** The same person, place, product, or animal
  appearing in multiple files gets the same stable name. The record
  for video B can then say "same speaker as in
  `2025-04-12_interview.md` (Anna)".
- **Series continuity.** Episode 3 of a documentary can be analysed
  with episodes 1 and 2 in mind ("picks up the river-bank tiger from
  episode 1, 11:02").
- **Skip-link reuse.** If a content warning was justified in video A,
  the same scene reused in a recap in video B can reuse the warning.

The `--corpus` flag is opt-in. Without it, vidwit is stateless on a
per-video basis: identical input yields identical output, with no
hidden surprises.

The following are out of scope by design:

- Bi-directional editing. Renaming an entity does **not** rewrite past
  Markdown files; past Markdown is a frozen artefact.
- Cross-corpus federation. One corpus directory is one world.

## Notice of copyright issues with produced output files

vidwit produces verbatim transcripts and descriptive derivatives of
its input videos. The legality of using those outputs is your
responsibility — please make sure you have the rights to the inputs
you feed into the tool.

## License

See [LICENSE](LICENSE).
