Metadata-Version: 2.5
Name: cinematlas
Version: 0.3.1
Summary: Search inside video down to the second: hybrid keyframe + speech search on MongoDB Atlas ($rankFusion, $rerank, autoEmbed) and Voyage AI.
Project-URL: Homepage, https://github.com/ranfysvalle02/cinematlas
Project-URL: Issues, https://github.com/ranfysvalle02/cinematlas/issues
Author-email: Fabian Valle <fabian.valle-simmons@mongodb.com>
License-Expression: MIT
License-File: LICENSE
Keywords: atlas,embeddings,hybrid-search,mongodb,multimodal,rankfusion,rerank,search,vector-search,video,voyageai,whisper
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Multimedia :: Video
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: opencv-python-headless>=4.8
Requires-Dist: pillow>=10.0
Requires-Dist: pymongo>=4.7
Requires-Dist: scenedetect>=0.6.4
Requires-Dist: voyageai>=0.3
Requires-Dist: yt-dlp>=2024.1.1
Provides-Extra: all
Requires-Dist: boto3>=1.28; extra == 'all'
Requires-Dist: faster-whisper>=1.0; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: openai-whisper
Requires-Dist: openai-whisper>=20231117; extra == 'openai-whisper'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: whisper
Requires-Dist: faster-whisper>=1.0; extra == 'whisper'
Description-Content-Type: text/markdown

# Cinematlas

**Search inside video, down to the second: what was shown and what was said.**

Cinematlas turns videos (YouTube links, file URLs, uploads) into searchable scenes in MongoDB Atlas.
Ask a question and get back the scene that answers it, the exact sentence, and a deep link to the
second it's said.

```bash
pip install "cinematlas[whisper]"
cinematlas doctor                                   # what works, what doesn't, how to fix it
cinematlas setup
cinematlas ingest "www.b.com/keynote.mp4"
cinematlas search "when do they talk about pricing?"
```

```text
 1. vid_3f2a…#12 @    7:11  Pricing starts at ten dollars a seat.
    https://www.b.com/keynote.mp4#t=431
 2. vid_3f2a…#13 @    7:40  Enterprise plans include SSO and audit logs.
    https://www.b.com/keynote.mp4#t=460
```

Built on **PySceneDetect**, **yt-dlp**, **faster-whisper**, **Voyage AI** and **MongoDB Atlas**
(Vector Search, autoEmbed, Atlas Search, `$rankFusion`, `$rerank`). How it was built and measured:
[blog.md](blog.md) · [benchmark](bench/RESULTS.md) · [production considerations](considerations.md).

---

## How it works

```
 URL / YouTube / upload / stdin
            │
            ▼
   fetch + guard ─── SSRF guard · http(s) only · size cap · credential redaction
            │
   ┌────────┴──────────────────────────┐
   ▼                                   ▼
 PySceneDetect cuts (≤30s scenes)   ffmpeg → 16 kHz mono → faster-whisper
   │                                   (VAD + word timestamps)
   ▼                                   │ sentences assigned to scenes,
 middle-frame keyframes                │ split at cuts word by word
   │                                   ▼
   ├─ voyage-multimodal-3.5 ──────► visual_embedding   (keyframe)
   ├─ voyage-multimodal-3.5 ──────► scene_embedding    (keyframe + speech, interleaved)
   └─ voyage-4 (autoEmbed/client) ─► transcript vectors · BM25 full-text index
                                       │
                                       ▼
          MongoDB Atlas: one document per scene, timestamped sentences inside

 search() ─► $rankFusion(visual, scene, transcript, text) ─► sentence-level rerank
          ─► results with ranks, relevance, moment and moment_link
```

---

## Install

`ffmpeg` must be on your `PATH` (`brew install ffmpeg` / `apt-get install -y ffmpeg`).

```bash
pip install "cinematlas[whisper]"    # recommended: + local speech-to-text (faster-whisper, no PyTorch)
pip install cinematlas               # core only (transcribe via OPENAI_API_KEY + [openai])
pip install "cinematlas[all]"        # + S3 keyframe uploads + OpenAI Whisper API
```

```bash
export MONGODB_URI="mongodb+srv://<user>:<password>@cluster.mongodb.net/"   # MDB_URI also accepted
export VOYAGE_API_KEY="pa-..."
```

---

## Quickstart

```python
from cinematlas import Cinematlas

engine = Cinematlas()        # reads MONGODB_URI / MDB_URI and VOYAGE_API_KEY
engine.ensure_indexes()      # one time, idempotent

engine.ingest("https://www.youtube.com/watch?v=5NhYvbMdbBU")   # YouTube
engine.ingest("www.b.com/v.mp4")                               # any file URL (scheme optional)
engine.ingest("talk.mp4")                                      # local path, bytes, file object, or upload

results = engine.search("how loud is a sonic boom?")
print(results)               # readable table; renders as markdown in Jupyter
results.top.link             # 'https://…#t=34', the second it's said
results.top.text             # 'Sonic booms are about 110 decibels.'
results.top.explain()        # 'rerank #1, transcript #1, scene #2, text #3, visual #9, relevance 0.91, …'
```

One `ingest()` takes anything, and one `search()` answers most questions. Results are plain dicts
underneath (`json.dumps` works), with attribute shortcuts on top.

```python
result = engine.ingest("www.b.com/v.mp4", progress=lambda stage, info: print(stage, info))
# fetched {...} → scenes {'count': 14} → transcribed {...} → embedded {...} → stored {'scenes': 14}
print(result)   # Indexed 14 scenes (12 with speech) as 'vid_3f2a…' in 21.4s [autoembed]
```

---

## Doctor

```text
$ cinematlas doctor
 ✓ MongoDB                        connected · server 9.0.2 · cinematlas.scenes
 ✓ Transcript search              Atlas autoEmbed (voyage-4, embedded server-side)
 ! Index cinematlas_vector_index  outdated: visual_embedding.quantization: None -> 'scalar'
                                  → engine.ensure_indexes(update=True)  ·  cinematlas setup --update  (in place, no downtime)
 ✓ Index cinematlas_text_index    ready
 ✓ $rankFusion                    hybrid search runs as one native query
 ! $rerank                        $rerank is disabled for this Atlas project; using the Voyage rerank API (same model)
                                  → Atlas UI → Project Settings → enable Native Reranking. Nothing else to change.
 ✓ Voyage AI                      API key works (voyage-multimodal-3.5, voyage-4)
 ✓ Data                           65 scenes across 6 videos
 ✓ ffmpeg                         /opt/homebrew/bin/ffmpeg
 ✓ Speech-to-text                 faster-whisper (small), runs locally

0 problem(s), 2 warning(s). Cinematlas degrades gracefully on warnings.
```

`engine.doctor()` returns the same report as an object (`.ok`, `.problems`, `.to_dict()`), and
`cinematlas doctor --json` exits non-zero on failures, so it works as a deploy gate. It's cheap and
non-destructive, and it primes the engine: the first search skips capability discovery.

**Nothing is fatal that doesn't have to be.** If `$rankFusion` or `$rerank` isn't available, you get
one warning per process saying *why* (disabled in the project, server too old, not offered on this
deployment type) and *how to fix it*, and search keeps working on an equivalent path with the same
results.

**Indexes heal in place.** `ensure_indexes()` compares each index with the definition this version
recommends, ignoring defaults the server adds. With `update=True` (or `cinematlas setup --update`),
outdated ones are updated through `updateSearchIndex`; the old version keeps serving until the new one
is built. That's how a 0.1 collection picks up scalar quantization without downtime.

---

## Search

`search()` fuses up to five ranked lists with Reciprocal Rank Fusion. Every search takes `video_id=`
to scope results to one video, and each source is also available on its own (`search_transcript`,
`search_text`, `search_visual_vector`, `search_scene_vector`), all returning the same `SearchResults`.

| Source | What it matches | Why it's there |
| --- | --- | --- |
| `visual` | keyframe vectors (`voyage-multimodal-3.5`) | Finds silent scenes and "what did it look like" questions |
| `scene` | joint keyframe + speech vectors (interleaved input) | +20 points Hit@1 over keyframe-only on our benchmark |
| `transcript` | semantic speech vectors (`voyage-4`, via autoEmbed or client-side) | The strongest single source for spoken questions |
| `text` | Atlas Search BM25 over transcripts | Exact names, numbers and jargon ("X-59", "building 4826") |
| `rerank` | a Voyage reranker scoring every candidate **sentence** | Precision, and it picks the **moment** |

Default weights (`visual 0.25 · scene 1 · transcript 1 · text 1 · rerank 2`) were
[tuned on a labelled benchmark](bench/RESULTS.md). With equal weights, hybrid search was *worse* than
transcript search alone (Hit@1 0.70 vs 0.83), because keyframe vectors are noisy for questions about
speech. Override with `weights={...}` or narrow with `sources=(...)`.

Each result includes:

| Field | Meaning |
| --- | --- |
| `moment` | The best-matching sentence `{start, end, text}`; `None` for silent scenes |
| `moment_link` | Deep link to that second (YouTube `?t=431s`, direct files `#t=431`); `None` for uploads |
| `ranks` | Rank in each source that found it. Explains *why* it ranked |
| `relevance` | Reranker score of the moment (`None` if reranking is off or unavailable) |
| `score` | Fused RRF score |
| plus | `video_id`, `scene_id`, `timestamp_start/end`, `transcript`, `segments`, `video_url`, `filename`, … |

### Build your own `ask()`

Cinematlas doesn't pick an LLM for you. Results turn into citable context in one call, and you can
use any model:

```python
hits = engine.search("How loud is a sonic boom, and why is it banned over land?", top_k=5)
prompt = f"""Answer using only these video excerpts. Cite them like [1].

{hits.to_context()}

Question: How loud is a sonic boom, and why is it banned over land?"""

answer = my_llm(prompt)                                   # any provider, any SDK
links = {f"[{i}]": h.link for i, h in enumerate(hits, 1)}           # citations -> timestamps
```

`to_context()` (also `cinematlas.to_context(results)`) renders `[1] <video> @ 7:11 <link>` followed by
the sentence. From the shell:
`cinematlas search "…" --format context | your-llm-cli`.

---

## Ingestion sources

### Remote URLs

`ingest_video()` accepts YouTube links, direct file URLs (including presigned S3/GCS/Azure URLs) and
anything else yt-dlp supports. It's built to handle URLs your users paste in:

| Concern | Behaviour |
| --- | --- |
| Scheme-less input | `www.b.com/v.mp4` becomes `https://www.b.com/v.mp4`. A bare `clip.mp4` is reported as a missing file |
| **SSRF** | The host is resolved first. Private, loopback, link-local and reserved addresses are refused (e.g. `169.254.169.254`). Opt in with `allow_private_urls=True` for trusted internal hosts |
| Schemes / size | `http`/`https` only. `max_download_mb` defaults to 2048 |
| Credentials | Signatures and tokens (`X-Amz-*`, `sig`, `token`, `user:pass@`, …) are redacted before storage. A re-signed URL maps to the same `video_id` |
| Deep links | YouTube gets `?t=Ns`. Direct files get the Media Fragment `#t=N`, so signed query strings stay valid |

Redirects followed by yt-dlp aren't re-checked. For public-facing apps, also route downloads through
an egress proxy.

### Uploads

`ingest_file()` accepts a path, `bytes`, a binary file object, a FastAPI `UploadFile`, or a Flask
`FileStorage`. Uploads stream to disk in 1 MiB chunks. Without a `video_id`, the ID is the content
hash, so a re-upload replaces instead of duplicating. Empty or undecodable files raise
`IngestionError`.

```python
@app.post("/videos")                          # FastAPI (sync def runs in a threadpool)
def upload(file: UploadFile):
    return {"scenes": engine.ingest_file(file)}
```

### CLI

```bash
cinematlas doctor                                       # diagnose; --json for CI, exit 1 on failures
cinematlas setup                                        # create indexes; --update fixes outdated ones in place
cinematlas ingest www.b.com/v.mp4                       # URL (YouTube or file)
cinematlas ingest ./talk.mp4 --video-id talk-01         # local file
curl -sL https://b.com/v.mp4 | cinematlas ingest - --filename v.mp4   # stdin
cinematlas search "how loud is a sonic boom" -k 3       # table in a terminal, JSON lines when piped
cinematlas search "a person in a hangar" --by visual    # hybrid | transcript | visual | text
cinematlas search "…" --format context                  # citable text for an LLM
```

---

## MongoDB Atlas features used

| Feature | How Cinematlas uses it | Fallback |
| --- | --- | --- |
| **`$rankFusion`** (8.0+) | Hybrid search in **one round trip**. `scoreDetails` returns per-source ranks, so fusion is identical to the client-side path (24% faster on the benchmark) | Per-source queries fused client-side |
| **`$rerank`** (8.3+, Atlas) | Sentence-level reranking server-side: `$unwind` segments, then `$rerank` | Voyage rerank API (identical scoring model) |
| **Automated Embedding** (Preview) | Atlas embeds transcripts and query text with `voyage-4` | Client-side `voyage-4` vectors, e.g. on Atlas Local |
| **Atlas Search** | BM25 full-text over transcripts, filterable by `video_id` | — |
| **Scalar quantization** | Every vector index. ~75% less vector memory, **no measured recall loss** on the benchmark | `quantization=None` |
| **BSON float32 vectors** | Stored embeddings are 3.2× smaller than arrays of doubles | `bson_vectors=False` |
| **Vector pre-filters** | `video_id` filter on every index | — |

Native stages are detected automatically: the engine tries each once, remembers the answer, and falls
back with one explained warning. Force them with `native_fusion=` / `native_rerank=`. **`$rerank` must
be enabled in your Atlas project settings.** `cinematlas doctor` tells you whether it is.

---

## Reliability

* **Replacing a video never loses it.** Re-ingest inserts the new version, *then* deletes older ones
  (`ingest_id`). A failed re-ingest leaves the previous version searchable next to a `FAILED`
  tombstone.
* **Speech lands in the right scene.** VAD-trimmed word timestamps. A sentence clearly straddling a
  cut is split word by word; timestamp jitter never copies sentences across cuts.
* **Vectors stay aligned.** Scenes without a keyframe get `None`; every other vector stays on its own
  scene.
* **Degrades instead of failing.** Voyage and download calls retry with backoff. A failed search
  source is skipped. A reranker outage falls back to word-overlap moments. Missing audio means
  visual-only indexing.
* **Provenance.** `embedding_models` on every document records which models produced its vectors.

---

## Configuration

| Parameter | Default | Notes |
| --- | --- | --- |
| `voyage_model` | `voyage-multimodal-3.5` | Keyframe and joint scene vectors |
| `text_model` | `voyage-4` | Transcripts. `-lite`/`-large` share the same space |
| `rerank_model` | `rerank-2.5` | `None` disables reranking |
| `transcript_mode` | `auto` | `auto` \| `autoembed` \| `client` |
| `scene_embeddings` | `True` | Joint image+speech vectors (one extra embed call per spoken scene) |
| `whisper_model` | `small` | Local faster-whisper model. We measured `base` mishearing ordinary words |
| `max_scene_seconds` | `30` | Split longer spans; `None` disables |
| `native_fusion` / `native_rerank` | auto | `True`/`False` forces the Atlas-native stage on or off |
| `bson_vectors` | `True` | Store vectors as BSON float32 |
| `progress` | `None` | Default ingest progress callback `progress(stage, info)` |
| `allow_private_urls` | `False` | SSRF guard |
| `max_download_mb` | `2048` | Download size cap |

Every external client can be injected (`mongo_client=`, `voyage_client=`, `s3_client=`,
`openai_client=`). That's how the test suite runs offline.

---

## Development

```bash
uv sync
uv run pytest -m "not integration and not media"   # unit: offline, ~8s
uv run pytest -m media                              # real ffmpeg / yt-dlp path / Whisper on the fixture, offline
uv run pytest -m integration                        # live Atlas (autoEmbed) + Docker Atlas Local (client)
uv run python bench/ingest.py && uv run python bench/run.py   # retrieval benchmark
```

| Tier | What's real | What it proves |
| --- | --- | --- |
| **unit** | OpenCV, PySceneDetect, ffmpeg on generated video. Voyage and Mongo are fakes that record calls and encode identity | Alignment, fusion, moments, native/fallback parity, uploads, URL safety, gapless replace, CLI |
| **media** | ffmpeg, faster-whisper, and the download path over loopback HTTP, on a committed real-speech fixture | Known cuts are found, transcripts are verbatim, each topic lands in its own scene |
| **integration** | Live Atlas + Voyage *and* Docker `mongodb-atlas-local` | URL and upload ingest end to end; questions return the right *scene* and *moment*, with and without autoEmbed |

**No test contacts YouTube.** End-to-end runs use
[`tests/fixtures/x59_quiet_crew.mp4`](tests/fixtures/build_fixture.py) (847 KiB, NASA, public
domain). CI runs the unit and media tiers; the integration tier runs locally from `.env` (`MDB_URI` or
`MONGODB_URI`, `VOYAGE_API_KEY`). The Atlas Local
container starts and stops by itself. Release: bump `version`, then
`rm -rf dist && uv build && uv publish`.

## License

MIT. Test and benchmark media: NASA, public domain.
