Metadata-Version: 2.5
Name: cinematlas
Version: 0.4.2
Summary: Search inside video down to the second: hybrid keyframe + speech search on MongoDB Atlas ($rankFusion, $rerank, autoEmbed) and Voyage AI.
Project-URL: Homepage, https://github.com/ranfysvalle02/cinematlas
Project-URL: Documentation, https://github.com/ranfysvalle02/cinematlas#readme
Project-URL: How it works, https://github.com/ranfysvalle02/cinematlas/blob/main/blog.md
Project-URL: Benchmark, https://github.com/ranfysvalle02/cinematlas/blob/main/bench/RESULTS.md
Project-URL: Limits & decisions, https://github.com/ranfysvalle02/cinematlas/blob/main/honest.md
Project-URL: Changelog, https://github.com/ranfysvalle02/cinematlas/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/ranfysvalle02/cinematlas/issues
Author-email: Fabian Valle <fabian.valle-simmons@mongodb.com>
License-Expression: MIT
License-File: LICENSE
Keywords: atlas,embeddings,hybrid-search,mongodb,multimodal,rankfusion,rerank,search,vector-search,video,voyageai,whisper
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Multimedia :: Video
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: opencv-python-headless>=4.8
Requires-Dist: pillow>=10.0
Requires-Dist: pymongo>=4.7
Requires-Dist: scenedetect>=0.6.4
Requires-Dist: voyageai>=0.3
Requires-Dist: yt-dlp>=2024.1.1
Provides-Extra: all
Requires-Dist: boto3>=1.28; extra == 'all'
Requires-Dist: faster-whisper>=1.0; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: whisper
Requires-Dist: faster-whisper>=1.0; extra == 'whisper'
Description-Content-Type: text/markdown

# Cinematlas

**Ask a question. Get the second in the video that answers it.**

Cinematlas turns videos into searchable moments in MongoDB Atlas. It listens to what's said and looks at
what's shown, and for each question it decides which to trust.

```bash
pip install "cinematlas[whisper]"
cinematlas doctor
cinematlas ingest "www.b.com/keynote.mp4"
cinematlas search "when do they announce pricing?"
```

```text
 1. vid_3f2a…#12 @    7:11  Pricing starts at ten dollars a seat.
    https://www.b.com/keynote.mp4#t=431
```

The story behind it: [blog.md](https://github.com/ranfysvalle02/cinematlas/blob/main/blog.md) · the numbers: [bench/RESULTS.md](https://github.com/ranfysvalle02/cinematlas/blob/main/bench/RESULTS.md) · the limits:
[honest.md](https://github.com/ranfysvalle02/cinematlas/blob/main/honest.md) · running it in production: [considerations.md](https://github.com/ranfysvalle02/cinematlas/blob/main/considerations.md)

---

## How it works

```
 video (URL, YouTube, upload)
   │
   ├─ scenes ────── PySceneDetect cuts, ≤30s each
   ├─ said ──────── faster-whisper → timestamped sentences, aligned to scenes word by word
   └─ shown ─────── a keyframe per scene
                      │
   Voyage AI ─────── keyframe vector · transcript vector · joint keyframe+speech vector
                      │
   MongoDB Atlas ─── one document per scene: vectors, sentences, deep links

 search(question)
   ├─ $rankFusion over keyframe, scene, transcript and full-text retrieval   (one query)
   ├─ $rerank over candidate sentences                                        (finds the moment)
   └─ route: is this about what was said or what was shown?                   (reranker confidence)
```

Questions about video come in two kinds. *"How many medals has his beer won?"* is about what was **said**;
*"the one with the girl on hay bales"* is about what was **shown**. Each needs a different specialist:

| Retrieval | Said | Shown |
| --- | --- | --- |
| transcript vectors + sentence reranker | **0.90** | 0.43 |
| joint keyframe + speech vectors | 0.73 | **0.93** |
| blending both with fixed weights | 0.80 | 0.50 |
| **routing by reranker confidence (default)** | 0.83 | 0.80 |

*Hit@1 on a 60-question benchmark ([details and caveats](https://github.com/ranfysvalle02/cinematlas/blob/main/bench/RESULTS.md)).*

The reranker scores how well any transcript sentence answers the question. High means "about what was
said", low means "about what was shown". `search()` turns that into a speech confidence and weights the
two specialists accordingly. The thresholds are calibrated per reranker; on one without a calibration,
search uses fixed fusion rather than guess.

---

## Quickstart

```bash
export MONGODB_URI="mongodb+srv://…"     # MDB_URI also works
export VOYAGE_API_KEY="pa-…"
```

```python
from cinematlas import Cinematlas

engine = Cinematlas()
engine.ensure_indexes()                                  # once; idempotent

engine.ingest("https://www.youtube.com/watch?v=5NhYvbMdbBU")
engine.ingest("www.b.com/talk.mp4")                      # any file URL, scheme optional
engine.ingest("lecture.mov")                             # or a path, bytes, file object, web upload

results = engine.search("how loud is a sonic boom?")
print(results)                  # a readable table (markdown in Jupyter)
results.top.link                # 'https://…#t=34': the second it's said
results.top.text                # 'Sonic booms are about 110 decibels.'
results.top.explain()           # why it ranked: rank per source, relevance, score
results.speech_confidence       # 1.0: routed as a question about what was said
```

Results are plain dicts underneath (`json.dumps` works) with attribute shortcuts on top.

### Build your own answer layer

Cinematlas doesn't choose an LLM for you. Results become numbered, citable context in one call:

```python
hits = engine.search("How loud is a sonic boom, and why is it banned over land?")
answer = my_llm(f"Answer from these excerpts, citing [n]:\n\n{hits.to_context()}")
sources = {f"[{i}]": h.link for i, h in enumerate(hits, 1)}
```

From the shell: `cinematlas search "…" --format context | your-llm-cli`.

---

## Search

| Source | Matches | Group |
| --- | --- | --- |
| `scene` | joint keyframe + speech vectors | shown |
| `visual` | keyframe vectors | shown |
| `transcript` | speech vectors (Atlas autoEmbed, or client-side `voyage-4`) | said |
| `text` | Atlas Search BM25 on transcripts (names, numbers, IDs) | said |
| `rerank` | Voyage reranker over candidate **sentences** (picks the moment) | said |

Every hit carries `moment` (the best sentence, `{start, end, text}`), `moment_link` (YouTube `?t=431s`,
direct files `#t=431`), `ranks` (position in each source), `relevance`, `score`, plus the scene's fields
(`video_id`, `scene_id`, `timestamp_start/end`, `transcript`, `segments`, `video_url`, …).
`results.speech_confidence` and `results.weights` show how the question was routed.

| Need | Call |
| --- | --- |
| Best overall (default) | `search(q)` |
| Fast, visual-first (~300 ms, scenes not moments) | `search(q, sources=("scene",), rerank=False)` |
| Speech only | `search(q, sources=("transcript", "text"))` |
| Your own blend | `search(q, weights={"scene": 2, "transcript": 1, "rerank": 1})` |
| One source | `search_transcript` · `search_text` · `search_visual_vector` · `search_scene_vector` |

All of them accept `video_id=` and return `SearchResults`. Query embeddings are cached per engine
(exact text, per model), so repeated searches skip the Voyage round trip; `engine.query_cache_info()`
shows hits and misses.

---

## Ingest

`engine.ingest(source)` accepts a URL (YouTube or any file link, scheme optional), a local path, `bytes`,
a binary file object, or a FastAPI `UploadFile` / Flask `FileStorage`. It returns an `IngestResult`
(video id, scene counts, per-stage timings); pass `progress=lambda stage, info: …` to follow along.

```python
@app.post("/videos")                     # FastAPI
def upload(file: UploadFile):
    return {"scenes": engine.ingest(file).scenes}
```

- **Uploads** stream to disk in 1 MiB chunks. Without a `video_id`, the ID is a content hash, so a
  re-upload replaces instead of duplicating.
- **Remote URLs** are treated as untrusted: private and metadata addresses are refused
  (`allow_private_urls=True` for trusted hosts), only `http(s)` is allowed, downloads are capped at
  `max_download_mb`, credentials in signed URLs are stripped before storage, and deep links use `#t=`
  fragments so signatures stay valid.
- **Re-ingesting** a video replaces it without a gap: the new version is written before the old one is
  removed.

---

## CLI

```bash
cinematlas doctor                               # health check with fixes; --json, exit 1 on failure
cinematlas setup [--update]                     # create indexes; --update upgrades outdated ones in place
cinematlas ingest <url|path|->                  # progress on stderr, IngestResult JSON on stdout
cinematlas search "<question>" [-k 5]           # table in a terminal, JSON lines when piped
    [--by hybrid|transcript|visual|text] [--format table|json|context] [--video-id ID]
```

Global options: `--uri`, `--db`, `--collection`, `--transcript-mode`, `-v`.

---

## Doctor

```text
$ cinematlas doctor
 ✓ MongoDB                        connected · server 9.0.2 · cinematlas.scenes
 ✓ Transcript search              Atlas autoEmbed (voyage-4, embedded server-side)
 ! Index cinematlas_vector_index  outdated: visual_embedding.quantization: None -> 'scalar'
                                  → engine.ensure_indexes(update=True)  ·  cinematlas setup --update  (in place, no downtime)
 ✓ $rankFusion                    hybrid search runs as one native query
 ✓ $rerank                        sentence reranking runs inside Atlas (rerank-2.5)
 ✓ Routing                        adaptive, calibrated for rerank-2.5 (0.45–0.55)
 ✓ Voyage AI                      API key works (voyage-multimodal-3.5, voyage-4)
 ✓ Data                           65 scenes across 6 videos
 ✓ ffmpeg                         /opt/homebrew/bin/ffmpeg
 ✓ Speech-to-text                 faster-whisper (small), runs locally
```

Also available as `engine.doctor()` (`.ok`, `.problems`, `.to_dict()`). When a native stage isn't
available, Cinematlas says why once per process (disabled, too old, or not offered on this deployment)
and keeps working on an equivalent path.

---

## MongoDB Atlas features

| Feature | Use | When unavailable |
| --- | --- | --- |
| `$rankFusion` (8.0+) | All retrieval in one query, with per-source ranks | Client-side fusion, same ranking |
| `$rerank` (8.3+, Atlas; enable in project settings) | Sentence reranking in the database | Voyage rerank API, same model |
| Automated Embedding (Preview) | Atlas embeds transcripts and queries with `voyage-4` | Client-side `voyage-4` vectors |
| Atlas Search | BM25 on transcripts | — |
| Scalar quantization | ~75% less vector-index memory | `quantization=None` |
| BSON float32 vectors | 3.2× smaller stored vectors | `bson_vectors=False` |
| `updateSearchIndex` | Upgrades outdated indexes in place | — |

---

## Configuration

| Parameter | Default | |
| --- | --- | --- |
| `voyage_model` | `voyage-multimodal-3.5` | keyframe and scene vectors |
| `text_model` | `voyage-4` | transcripts (`-lite` / `-large` share the space) |
| `rerank_model` | `rerank-2.5` | `None` disables reranking and routing |
| `routing_thresholds` | calibrated | `(lo, hi)` for a reranker without built-in calibration |
| `query_cache_size` | `256` | cached query embeddings (repeat searches ~38% faster); `0` disables |
| `transcript_mode` | `auto` | `autoembed`, `client`, or detect |
| `whisper_model` | `small` | local faster-whisper model |
| `scene_embeddings` | `True` | joint keyframe + speech vectors |
| `max_scene_seconds` | `30` | `None` to disable |
| `native_fusion` / `native_rerank` | auto | force the Atlas-native path on or off |
| `bson_vectors` | `True` | BSON float32 storage |
| `allow_private_urls` | `False` | SSRF guard |
| `max_download_mb` | `2048` | download cap |
| `progress` | `None` | default `progress(stage, info)` callback |

Every external client can be injected (`mongo_client=`, `voyage_client=`, `s3_client=`,
`openai_client=`).

---

## Development

```bash
uv sync
uv run pytest -m "not integration and not media"   # unit, offline (~9s)
uv run pytest -m media                              # real ffmpeg / Whisper / downloads, on a committed fixture
uv run pytest -m integration                        # live Atlas + Docker Atlas Local (reads .env)
uv run python bench/ingest.py && uv run python bench/run.py   # the benchmark
```

CI runs the unit and media tiers. No test contacts YouTube: end-to-end runs use an 847 KiB public-domain
NASA clip with known cuts and known speech ([build script](https://github.com/ranfysvalle02/cinematlas/blob/main/tests/fixtures/build_fixture.py)), served over
loopback HTTP. The integration tier runs locally from `.env`.

MIT license. Test and benchmark media: NASA, public domain.
