Metadata-Version: 2.5
Name: cinematlas
Version: 0.2.0
Summary: Search inside video down to the second: hybrid keyframe + speech search on MongoDB Atlas ($rankFusion, $rerank, autoEmbed) and Voyage AI.
Project-URL: Homepage, https://github.com/ranfysvalle02/cinematlas
Project-URL: Issues, https://github.com/ranfysvalle02/cinematlas/issues
Author-email: Fabian Valle <fabian.valle-simmons@mongodb.com>
License-Expression: MIT
License-File: LICENSE
Keywords: atlas,embeddings,hybrid-search,mongodb,multimodal,rankfusion,rerank,search,vector-search,video,voyageai,whisper
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Multimedia :: Video
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: opencv-python-headless>=4.8
Requires-Dist: pillow>=10.0
Requires-Dist: pymongo>=4.7
Requires-Dist: scenedetect>=0.6.4
Requires-Dist: voyageai>=0.3
Requires-Dist: yt-dlp>=2024.1.1
Provides-Extra: all
Requires-Dist: boto3>=1.28; extra == 'all'
Requires-Dist: faster-whisper>=1.0; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: openai-whisper
Requires-Dist: openai-whisper>=20231117; extra == 'openai-whisper'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: whisper
Requires-Dist: faster-whisper>=1.0; extra == 'whisper'
Description-Content-Type: text/markdown

# Cinematlas

**Search inside video, down to the second: what was shown and what was said.**

Cinematlas turns videos (YouTube links, file URLs, uploads) into searchable scenes in MongoDB Atlas.
Ask a question and get back the scene that answers it, the exact sentence, and a deep link to the
second it's said.

```bash
pip install "cinematlas[whisper]"
cinematlas setup
cinematlas ingest "www.b.com/keynote.mp4"
cinematlas search "when do they talk about pricing?"
```

```jsonc
{"video_id": "vid_3f2a…", "scene_id": 12, "score": 0.047,
 "moment": {"start": 431.2, "end": 436.8, "text": "Pricing starts at ten dollars a seat."},
 "moment_link": "https://www.b.com/keynote.mp4#t=431",
 "ranks": {"scene": 1, "transcript": 1, "text": 2, "visual": 9, "rerank": 1}, "relevance": 0.91}
```

Built on **PySceneDetect**, **yt-dlp**, **faster-whisper**, **Voyage AI** and **MongoDB Atlas**
(Vector Search, autoEmbed, Atlas Search, `$rankFusion`, `$rerank`). How it was built and measured:
[blog.md](blog.md) · [benchmark](bench/RESULTS.md) · [production considerations](considerations.md).

---

## How it works

```
 URL / YouTube / upload / stdin
            │
            ▼
   fetch + guard ─── SSRF guard · http(s) only · size cap · credential redaction
            │
   ┌────────┴──────────────────────────┐
   ▼                                   ▼
 PySceneDetect cuts (≤30s scenes)   ffmpeg → 16 kHz mono → faster-whisper
   │                                   (VAD + word timestamps)
   ▼                                   │ sentences assigned to scenes,
 middle-frame keyframes                │ split at cuts word by word
   │                                   ▼
   ├─ voyage-multimodal-3.5 ──────► visual_embedding   (keyframe)
   ├─ voyage-multimodal-3.5 ──────► scene_embedding    (keyframe + speech, interleaved)
   └─ voyage-4 (autoEmbed/client) ─► transcript vectors · BM25 full-text index
                                       │
                                       ▼
          MongoDB Atlas: one document per scene, timestamped sentences inside

 search() ─► $rankFusion(visual, scene, transcript, text) ─► sentence-level rerank
          ─► results with ranks, relevance, moment and moment_link
```

---

## Install

`ffmpeg` must be on your `PATH` (`brew install ffmpeg` / `apt-get install -y ffmpeg`).

```bash
pip install "cinematlas[whisper]"    # recommended: + local speech-to-text (faster-whisper, no PyTorch)
pip install cinematlas               # core only (transcribe via OPENAI_API_KEY + [openai])
pip install "cinematlas[all]"        # + S3 keyframe uploads + OpenAI Whisper API
```

```bash
export MONGODB_URI="mongodb+srv://<user>:<password>@cluster.mongodb.net/"   # MDB_URI also accepted
export VOYAGE_API_KEY="pa-..."
```

---

## Quickstart

```python
from cinematlas import Cinematlas

engine = Cinematlas()        # reads MONGODB_URI / MDB_URI and VOYAGE_API_KEY
engine.ensure_indexes()      # one time, idempotent: vector, full-text and lookup indexes

engine.ingest_video("https://www.youtube.com/watch?v=5NhYvbMdbBU")   # YouTube
engine.ingest_video("www.b.com/v.mp4")                                # any file URL (scheme optional)
engine.ingest_file("talk.mp4")                                        # path, bytes, file object, upload

for hit in engine.search("how loud is a sonic boom?", top_k=3):
    print(hit["moment_link"], "-", hit["moment"]["text"] if hit["moment"] else hit["transcript"])
```

`search()` is the one call most apps need. The single-source searches are still there if you want
them: `search_transcript`, `search_text`, `search_visual_vector`, `search_scene_vector`.
Every search accepts `video_id=` to scope results to one video.

---

## Search

`search()` fuses up to five ranked lists with Reciprocal Rank Fusion:

| Source | What it matches | Why it's there |
| --- | --- | --- |
| `visual` | keyframe vectors (`voyage-multimodal-3.5`) | Finds silent scenes and "what did it look like" questions |
| `scene` | joint keyframe + speech vectors (interleaved input) | +20 points Hit@1 over keyframe-only on our benchmark |
| `transcript` | semantic speech vectors (`voyage-4`, via autoEmbed or client-side) | The strongest single source for spoken questions |
| `text` | Atlas Search BM25 over transcripts | Exact names, numbers and jargon ("X-59", "building 4826") |
| `rerank` | a Voyage reranker scoring every candidate **sentence** | Precision, and it picks the **moment** |

Default weights (`visual 0.25 · scene 1 · transcript 1 · text 1 · rerank 2`) were
[tuned on a labelled benchmark](bench/RESULTS.md). With equal weights, hybrid search was *worse* than
transcript search alone (Hit@1 0.67 vs 0.83), because keyframe vectors are noisy for questions about
speech. Override with `weights={...}` or narrow with `sources=(...)`.

Each result includes:

| Field | Meaning |
| --- | --- |
| `moment` | The best-matching sentence `{start, end, text}`; `None` for silent scenes |
| `moment_link` | Deep link to that second (YouTube `?t=431s`, direct files `#t=431`); `None` for uploads |
| `ranks` | Rank in each source that found it. Explains *why* it ranked |
| `relevance` | Reranker score of the moment (`None` if reranking is off or unavailable) |
| `score` | Fused RRF score |
| plus | `video_id`, `scene_id`, `timestamp_start/end`, `transcript`, `segments`, `video_url`, `filename`, … |

### Build your own `ask()`

Cinematlas doesn't pick an LLM for you. Results turn into citable context in one call, and you can
use any model:

```python
from cinematlas import to_context

hits = engine.search("How loud is a sonic boom, and why is it banned over land?", top_k=5)
prompt = f"""Answer using only these video excerpts. Cite them like [1].

{to_context(hits)}

Question: How loud is a sonic boom, and why is it banned over land?"""

answer = my_llm(prompt)                                   # any provider, any SDK
links = {f"[{i}]": h["moment_link"] for i, h in enumerate(hits, 1)}   # citations -> timestamps
```

`to_context` renders `[1] <video> @ 7:11 <link>` followed by the sentence. From the shell:
`cinematlas search "…" --format context | your-llm-cli`.

---

## Ingestion sources

### Remote URLs

`ingest_video()` accepts YouTube links, direct file URLs (including presigned S3/GCS/Azure URLs) and
anything else yt-dlp supports. It's built to handle URLs your users paste in:

| Concern | Behaviour |
| --- | --- |
| Scheme-less input | `www.b.com/v.mp4` becomes `https://www.b.com/v.mp4`. A bare `clip.mp4` is reported as a missing file |
| **SSRF** | The host is resolved first. Private, loopback, link-local and reserved addresses are refused (e.g. `169.254.169.254`). Opt in with `allow_private_urls=True` for trusted internal hosts |
| Schemes / size | `http`/`https` only. `max_download_mb` defaults to 2048 |
| Credentials | Signatures and tokens (`X-Amz-*`, `sig`, `token`, `user:pass@`, …) are redacted before storage. A re-signed URL maps to the same `video_id` |
| Deep links | YouTube gets `?t=Ns`. Direct files get the Media Fragment `#t=N`, so signed query strings stay valid |

Redirects followed by yt-dlp aren't re-checked. For public-facing apps, also route downloads through
an egress proxy.

### Uploads

`ingest_file()` accepts a path, `bytes`, a binary file object, a FastAPI `UploadFile`, or a Flask
`FileStorage`. Uploads stream to disk in 1 MiB chunks. Without a `video_id`, the ID is the content
hash, so a re-upload replaces instead of duplicating. Empty or undecodable files raise
`IngestionError`.

```python
@app.post("/videos")                          # FastAPI (sync def runs in a threadpool)
def upload(file: UploadFile):
    return {"scenes": engine.ingest_file(file)}
```

### CLI

```bash
cinematlas setup                                        # indexes -> {"transcript_mode": "autoembed"}
cinematlas ingest www.b.com/v.mp4                       # URL (YouTube or file)
cinematlas ingest ./talk.mp4 --video-id talk-01         # local file
curl -sL https://b.com/v.mp4 | cinematlas ingest - --filename v.mp4   # stdin
cinematlas search "how loud is a sonic boom" -k 3       # hybrid, JSON lines
cinematlas search "a person in a hangar" --by visual    # hybrid | transcript | visual | text
cinematlas search "…" --format context                  # citable text for an LLM
```

---

## MongoDB Atlas features used

| Feature | How Cinematlas uses it | Fallback |
| --- | --- | --- |
| **`$rankFusion`** (8.0+) | Hybrid search in **one round trip**. `scoreDetails` returns per-source ranks, so fusion is identical to the client-side path (24% faster on the benchmark) | Per-source queries fused client-side |
| **`$rerank`** (8.3+, Atlas) | Sentence-level reranking server-side: `$unwind` segments, then `$rerank` | Voyage rerank API (identical scoring model) |
| **Automated Embedding** (Preview) | Atlas embeds transcripts and query text with `voyage-4` | Client-side `voyage-4` vectors, e.g. on Atlas Local |
| **Atlas Search** | BM25 full-text over transcripts, filterable by `video_id` | — |
| **Scalar quantization** | Every vector index. ~75% less vector memory, **no measured recall loss** on the benchmark | `quantization=None` |
| **BSON float32 vectors** | Stored embeddings are 3.2× smaller than arrays of doubles | `bson_vectors=False` |
| **Vector pre-filters** | `video_id` filter on every index | — |

Native stages are detected automatically: the engine tries each once and remembers if the cluster
rejects it. Force them with `native_fusion=` / `native_rerank=`. **`$rerank` must be enabled in your
Atlas project settings.**

---

## Reliability

* **Replacing a video never loses it.** Re-ingest inserts the new version, *then* deletes older ones
  (`ingest_id`). A failed re-ingest leaves the previous version searchable next to a `FAILED`
  tombstone.
* **Speech lands in the right scene.** VAD-trimmed word timestamps. A sentence clearly straddling a
  cut is split word by word; timestamp jitter never copies sentences across cuts.
* **Vectors stay aligned.** Scenes without a keyframe get `None`; every other vector stays on its own
  scene.
* **Degrades instead of failing.** Voyage and download calls retry with backoff. A failed search
  source is skipped. A reranker outage falls back to word-overlap moments. Missing audio means
  visual-only indexing.
* **Provenance.** `embedding_models` on every document records which models produced its vectors.

---

## Configuration

| Parameter | Default | Notes |
| --- | --- | --- |
| `voyage_model` | `voyage-multimodal-3.5` | Keyframe and joint scene vectors |
| `text_model` | `voyage-4` | Transcripts. `-lite`/`-large` share the same space |
| `rerank_model` | `rerank-2.5` | `None` disables reranking |
| `transcript_mode` | `auto` | `auto` \| `autoembed` \| `client` |
| `scene_embeddings` | `True` | Joint image+speech vectors (one extra embed call per spoken scene) |
| `whisper_model` | `small` | Local faster-whisper model. We measured `base` mishearing ordinary words |
| `max_scene_seconds` | `30` | Split longer spans; `None` disables |
| `native_fusion` / `native_rerank` | auto | `True`/`False` forces the Atlas-native stage on or off |
| `bson_vectors` | `True` | Store vectors as BSON float32 |
| `allow_private_urls` | `False` | SSRF guard |
| `max_download_mb` | `2048` | Download size cap |

Every external client can be injected (`mongo_client=`, `voyage_client=`, `s3_client=`,
`openai_client=`). That's how the test suite runs offline.

---

## Development

```bash
uv sync
uv run pytest -m "not integration and not media"   # unit: offline, ~8s
uv run pytest -m media                              # real ffmpeg / yt-dlp path / Whisper on the fixture, offline
uv run pytest -m integration                        # live Atlas (autoEmbed) + Docker Atlas Local (client)
uv run python bench/ingest.py && uv run python bench/run.py   # retrieval benchmark
```

| Tier | What's real | What it proves |
| --- | --- | --- |
| **unit** | OpenCV, PySceneDetect, ffmpeg on generated video. Voyage and Mongo are fakes that record calls and encode identity | Alignment, fusion, moments, native/fallback parity, uploads, URL safety, gapless replace, CLI |
| **media** | ffmpeg, faster-whisper, and the download path over loopback HTTP, on a committed real-speech fixture | Known cuts are found, transcripts are verbatim, each topic lands in its own scene |
| **integration** | Live Atlas + Voyage *and* Docker `mongodb-atlas-local` | URL and upload ingest end to end; questions return the right *scene* and *moment*, with and without autoEmbed |

**No test contacts YouTube.** End-to-end runs use
[`tests/fixtures/x59_quiet_crew.mp4`](tests/fixtures/build_fixture.py) (847 KiB, NASA, public
domain). Integration env (`.env`): `MDB_URI` or `MONGODB_URI`, `VOYAGE_API_KEY`. The Atlas Local
container starts and stops by itself. Release: bump `version`, then
`rm -rf dist && uv build && uv publish`.

## License

MIT. Test and benchmark media: NASA, public domain.
