Metadata-Version: 2.5
Name: cinematlas
Version: 0.1.0
Summary: Multimodal video search engine: PySceneDetect + Whisper + Voyage AI embeddings + MongoDB Atlas Vector Search (incl. autoEmbed).
Project-URL: Homepage, https://github.com/ranfysvalle02/cinematlas
Project-URL: Issues, https://github.com/ranfysvalle02/cinematlas/issues
Author-email: Fabian Valle <fabian.valle-simmons@mongodb.com>
License-Expression: MIT
License-File: LICENSE
Keywords: atlas,embeddings,mongodb,multimodal,search,vector-search,video,voyageai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Multimedia :: Video
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: opencv-python-headless>=4.8
Requires-Dist: pillow>=10.0
Requires-Dist: pymongo>=4.7
Requires-Dist: scenedetect>=0.6.4
Requires-Dist: voyageai>=0.3
Requires-Dist: yt-dlp>=2024.1.1
Provides-Extra: all
Requires-Dist: boto3>=1.28; extra == 'all'
Requires-Dist: faster-whisper>=1.0; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: openai-whisper
Requires-Dist: openai-whisper>=20231117; extra == 'openai-whisper'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: whisper
Requires-Dist: faster-whisper>=1.0; extra == 'whisper'
Description-Content-Type: text/markdown

# Cinematlas

**Search inside video: the frames and the spoken words, not just titles and tags.**

Cinematlas turns a video (a YouTube link, a URL to a file, or an upload) into searchable scenes in
MongoDB Atlas. It detects scene cuts, picks a keyframe for each scene, transcribes the dialogue, and
embeds both with Voyage AI. You can then ask a question and get a deep link to the scene that answers it.

```bash
pip install "cinematlas[whisper]"
cinematlas setup
cinematlas ingest "www.b.com/keynote.mp4"
cinematlas search "when do they talk about pricing?"
```

Built on **PySceneDetect**, **yt-dlp**, **faster-whisper**, **Voyage AI** and **MongoDB Atlas Vector
Search**, with or without Atlas autoEmbed. The engineering story is in [blog.md](blog.md).

---

## How it works

```
 URL / youtube / upload / stdin
            │
            ▼
   ┌──────────────────┐   yt-dlp (H.264 + audio)  or  chunked upload → temp file
   │  fetch + guard   │   SSRF guard · http(s) only · size cap · credential redaction
   └────────┬─────────┘
            ▼
   ┌──────────────────┐        ┌───────────────────────────┐
   │  PySceneDetect   │        │ ffmpeg → 16 kHz mono MP3  │
   │  cuts + 30s cap  │        │ faster-whisper + VAD      │
   └────────┬─────────┘        └─────────────┬─────────────┘
            ▼                                │  overlap-aware
   middle-frame keyframes                    │  scene assignment
            ▼                                ▼
   voyage-multimodal-3.5            transcript per scene
   (image vectors)                  ├─ Atlas autoEmbed (voyage-4), or
            │                       └─ client-side voyage-4 vectors
            ▼                                ▼
   ┌──────────────────────────────────────────────────────┐
   │ MongoDB Atlas: one document per scene                 │
   │ $vectorSearch: visual_embedding · transcript          │
   └──────────────────────────────────────────────────────┘
```

| Data | Voyage model | Why |
| --- | --- | --- |
| Keyframes | `voyage-multimodal-3.5` | Images and text share one space, so a *text* query finds *frames* |
| Transcripts | `voyage-4` family | Better text retrieval. `-lite`, base and `-large` share one space, so you can index with one and query with another |

---

## Install

`ffmpeg` must be on your `PATH`.

```bash
brew install ffmpeg                  # macOS  (Ubuntu: sudo apt-get install -y ffmpeg)

pip install "cinematlas[whisper]"    # recommended: + local speech-to-text (faster-whisper, no PyTorch)
pip install cinematlas               # core only (use OPENAI_API_KEY for Whisper API transcription)
pip install "cinematlas[all]"        # + S3 keyframe uploads + OpenAI Whisper API
```

```bash
export MONGODB_URI="mongodb+srv://<user>:<password>@cluster.mongodb.net/"   # MDB_URI also accepted
export VOYAGE_API_KEY="pa-..."
export OPENAI_API_KEY="sk-..."                    # optional: Whisper API instead of local
export S3_BUCKET_NAME="my-cinematlas-keyframes"   # optional: keyframe thumbnails
```

---

## Quickstart

```python
from cinematlas import Cinematlas

engine = Cinematlas()           # reads MONGODB_URI / MDB_URI and VOYAGE_API_KEY
engine.ensure_indexes()         # one time; idempotent; returns "autoembed" or "client"

# From a URL: YouTube, a direct file link, or anything yt-dlp supports
engine.ingest_video("https://www.youtube.com/watch?v=5NhYvbMdbBU")
engine.ingest_video("www.b.com/v.mp4")          # scheme-less works; fetched over https

# From a file or an upload
engine.ingest_file("talk.mp4")

# Search what was said
for hit in engine.search_transcript("how loud is a sonic boom?", top_k=3):
    print(f"{hit['score']:.3f}  {hit['timestamp_start']}s  {hit['deep_link']}\n   {hit['transcript']}")

# Search what was shown (text -> keyframes)
engine.search_visual_vector("an aircraft on a runway", top_k=3)
```

Every search takes `video_id=` to scope results to one video.

---

## Ingestion sources

### Remote URLs

`ingest_video()` accepts YouTube links, direct file URLs (`https://cdn.example.com/v.mp4`, presigned
S3/GCS/Azure URLs) and anything else yt-dlp supports. It's designed so you can safely pass URLs
supplied by your users:

| Concern | Behaviour |
| --- | --- |
| Scheme-less input | `www.b.com/v.mp4` becomes `https://www.b.com/v.mp4`. `clip.mp4` is reported as a missing file, not treated as a host |
| **SSRF** | The host is resolved first, and private, loopback, link-local and reserved addresses are refused (for example `169.254.169.254`). Opt in with `allow_private_urls=True` for trusted internal sources |
| Schemes | `http` and `https` only |
| Size | `max_download_mb` (default 2048) is enforced by yt-dlp |
| Credentials | Signatures and tokens (`X-Amz-*`, `sig`, `token`, `user:pass@`, …) are **redacted before storage**. The raw URL is used only for the download. A re-signed URL maps to the same `video_id` |
| Deep links | YouTube gets `?t=42s`. Direct files get the W3C Media Fragment `#t=42`, which browsers seek to and which leaves signed query strings valid |
| Transient failures | Downloads retry with backoff |

Redirects followed by yt-dlp aren't re-checked. If you accept arbitrary URLs from the public
internet, also route downloads through an egress proxy.

### File uploads

`ingest_file()` accepts a path, `bytes`, a binary file object, a FastAPI `UploadFile`, or a
Flask/Werkzeug `FileStorage`.

```python
# FastAPI
@app.post("/videos")
def upload(file: UploadFile):
    return {"scenes": engine.ingest_file(file)}   # sync def: FastAPI runs it in a threadpool

# Flask
@app.post("/videos")
def upload():
    return {"scenes": engine.ingest_file(request.files["video"])}
```

* Uploads are streamed to disk in 1 MiB chunks, so memory stays flat for large files.
* Without a `video_id`, the ID is the content hash (`file_<sha256[:16]>`). Uploading the same file
  again replaces it instead of duplicating it.
* Empty or undecodable files are rejected with `IngestionError`.

### CLI

```bash
cinematlas setup                                     # {"transcript_mode": "autoembed"}
cinematlas ingest "https://www.youtube.com/watch?v=5NhYvbMdbBU"
cinematlas ingest www.b.com/v.mp4                    # remote file
cinematlas ingest ./talk.mp4 --video-id talk-01      # local file
curl -sL https://b.com/v.mp4 | cinematlas ingest - --filename v.mp4    # stdin upload
cinematlas search "how loud is a sonic boom" -k 3    # JSON lines
cinematlas search "an aircraft in the sky" --by visual
```

Options: `--uri`, `--db`, `--collection`, `--transcript-mode`, `-v`. Exit code 1 on errors, with a
message instead of a traceback.

---

## With or without Atlas autoEmbed

Transcripts can be searched in two ways, through one call (`search_transcript`):

* **`autoembed`**: Atlas [Automated Embedding](https://www.mongodb.com/docs/vector-search/crud-embeddings/automated-embedding/)
  embeds `transcript` and the query text itself with `voyage-4` (Atlas feature, currently in Preview).
* **`client`**: Cinematlas stores a `voyage-4` `transcript_embedding` and embeds queries itself. This
  works on any deployment with Vector Search, including `mongodb/mongodb-atlas-local`.

`ensure_indexes()` tries autoEmbed first and falls back to `client` if the cluster rejects it. If a
transcript index already exists, it keeps using that mode and never switches modes on existing data.
To force a mode, pass `transcript_mode="autoembed"` or `"client"`.

---

## Reliability

* **Replacing a video never loses it.** A re-ingest inserts the new scenes under a fresh `ingest_id`
  and only then deletes older versions. After a failure, the previous version stays searchable next to
  a `FAILED` tombstone.
* **Vectors stay aligned with scenes.** Scenes without a usable keyframe get `None`, and every other
  vector stays on its own scene. Short API responses are treated as failures, not shifted.
* **Speech goes to the right scene.** Timestamps come from voice-activity-trimmed Whisper output. A
  segment belongs to every scene it overlaps by ≥ 0.5s, and always to the scene it overlaps most, so
  normal timestamp drift doesn't copy sentences across cuts.
* **Scenes have a maximum length.** Footage without hard cuts is split into scenes of at most 30s
  (`max_scene_seconds`).
* **Things degrade instead of crashing.** Voyage and download calls retry with backoff. A missing
  audio track means visual-only indexing, and a failed transcription means an empty transcript.
* **Documents record provenance.** `embedding_models` stores which models produced the vectors, for
  future re-embedding migrations.

---

## Configuration

| Parameter | Default | Notes |
| --- | --- | --- |
| `voyage_model` | `voyage-multimodal-3.5` | Keyframe vectors |
| `text_model` | `voyage-4` | Transcripts (autoEmbed index model or client-side) |
| `transcript_mode` | `auto` | `auto` \| `autoembed` \| `client` |
| `whisper_model` | `small` | Local faster-whisper model. We measured `base` mishearing ordinary words |
| `max_scene_seconds` | `30` | `None` disables the scene-length cap |
| `allow_private_urls` | `False` | SSRF guard |
| `max_download_mb` | `2048` | Download size cap |
| `db_name` / `collection_name` | `cinematlas_enterprise` / `multimodal_scenes` | |

Every external client can be passed in (`mongo_client=`, `voyage_client=`, `s3_client=`,
`openai_client=`), which is how the test suite runs offline.

---

## Development & testing

```bash
uv sync
uv run pytest -m "not integration and not media"   # unit: offline, ~8s
uv run pytest -m media                              # real ffmpeg / yt-dlp / Whisper on the fixture, offline
uv run pytest -m integration                        # live Atlas (autoEmbed) + Docker Atlas Local (client)
uv run pytest                                       # everything
```

| Tier | What's real | What it proves |
| --- | --- | --- |
| **unit** | OpenCV, PySceneDetect, ffmpeg on generated video. Voyage and Mongo are fakes that record calls and encode identity | Alignment, retries, fallback, uploads, URL safety, gapless replace, CLI |
| **media** | ffmpeg, faster-whisper, and our download path over loopback HTTP, on a committed real-speech fixture | Known cuts are found, transcripts are verbatim, each topic lands in its own scene, timestamps skip silence |
| **integration** | Live Atlas + Voyage *and* `mongodb/mongodb-atlas-local` in Docker | URL and upload ingest end to end, and questions return the right *scene*, with and without autoEmbed |

**No test contacts YouTube.** End-to-end runs use
[`tests/fixtures/x59_quiet_crew.mp4`](tests/fixtures/build_fixture.py) (847 KiB, NASA, public
domain): three sentence-aligned topics joined by hard cuts, which gives the tests ground truth for
scene alignment. URL ingestion is served from a loopback HTTP server. The fixture is checked by
SHA-256 and rebuilt with `uv run python tests/fixtures/build_fixture.py`.

Integration env (`.env`): `MDB_URI` or `MONGODB_URI`, `VOYAGE_API_KEY`, and optionally `VOYAGE_MODEL`.
The Atlas Local container starts and stops by itself on port 27028; set `ATLAS_LOCAL_URI` to reuse
one. Tests write to `cinematlas_ci.scenes` under a unique `video_id` and clean up after themselves.

### Releasing

```bash
# bump version in pyproject.toml, then:
rm -rf dist && uv build && uv publish   # uses UV_PUBLISH_TOKEN
```

---

## Roadmap

* Async / queued ingestion (Celery, Temporal)
* OCR of on-screen text into scene documents
* Word-level timestamps for sentences that straddle a cut
* `AdaptiveDetector` option for dissolve-heavy footage

## License

MIT. Test fixture: NASA media, public domain (see [build_fixture.py](tests/fixtures/build_fixture.py)).
