Metadata-Version: 2.4
Name: wm-mcp
Version: 0.7.0
Summary: Run a world model locally, and let any LLM agent call it as a tool. V-JEPA 2 as an MCP server.
Project-URL: Homepage, https://github.com/rian-one/wm
Project-URL: Issues, https://github.com/rian-one/wm/issues
Author: Rian Herrmann
License-Expression: MIT
License-File: LICENSE
Keywords: agents,jepa,mcp,model-context-protocol,v-jepa,video,world-model
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: av>=12.0
Requires-Dist: fastapi>=0.110
Requires-Dist: huggingface-hub>=0.30
Requires-Dist: mcp>=1.2
Requires-Dist: numpy>=1.26
Requires-Dist: pillow>=10.0
Requires-Dist: torch>=2.4
Requires-Dist: torchvision>=0.19
Requires-Dist: transformers>=4.53
Requires-Dist: uvicorn>=0.30
Description-Content-Type: text/markdown

# wm

**Run a world model locally, and let any LLM agent call it as a tool.**

![An LLM agent consulting a video world model](https://raw.githubusercontent.com/rian-one/wm/main/demo.gif)

*Live and unedited: Claude calls the `surprise` tool; V-JEPA 2 watches the clip and
reports where its predictions broke (3.0–4.9s, z≈2 — the test video's splice is at
exactly 4.0s). Wait time hidden, output verbatim.*

LLMs can *describe* what happens when a cup gets knocked off a table. A world model
can *watch* it happen and tell you, frame by frame, where reality stopped matching
its prediction. `wm` puts Meta's [V-JEPA 2](https://huggingface.co/facebook/vjepa2-vitl-fpc64-256)
(MIT-licensed, 0.3B params) behind an [MCP](https://modelcontextprotocol.io) server,
so Claude, or any MCP client, can consult a video world model the way it consults a
database.

```
uvx wm-mcp serve
```

That's the whole install. First run downloads PyTorch and ~1.2GB of model weights —
**minutes of downloading, not a hang.**

## The demo

Ask your agent:

> "Where does this clip stop being predictable? /path/to/video.mp4"

The agent calls `surprise()`, the world model slides a window over the video,
predicts each window's second half from its first, and reports the segment where
its prediction failed hardest — z-scored per video, so scores mean something.

## The three tools

| Tool | What it does |
|---|---|
| `embed_video(source, start_s?, end_s?)` | Video, video slice, or still image → latent sequence, stored server-side under an opaque handle. Images become static clips — useful as goals. |
| `predict(handle, horizon_frames, target_handle?)` | Predict the trailing frames of a clip from its leading context (masked in-window prediction — horizon capped at clip length, this is not open-ended rollout). With `target_handle`, also returns cosine similarity between the prediction and a target embedding — "does this video end up looking like this photo?", computed server-side. |
| `surprise(source, window, stride, fps, start_s?, end_s?, max_windows)` | Streams over video of any length; per-segment "how wrong was the model" scores with timestamps in seconds + the most surprising segment. Long videos get an explicit compute budget (`max_windows`, default 120) and a `resume_s` cursor to continue — never a silent cap. |
| `compare(handle_a, handle_b)` | Cosine similarity between any two embeddings. "Are these two clips the same scene?", near-duplicate detection, frame-vs-reference matching. |

Latents never cross the wire: tools exchange handle IDs, tensors stay in the server.

## Setup with Claude Code

Do it in this order — step 1 downloads PyTorch and the model weights (~4GB total)
so your MCP client doesn't time out waiting on a first-run download:

```bash
uvx wm-mcp warmup                     # 1. one-time: pull deps + weights (minutes)
claude mcp add wm -- uvx wm-mcp serve # 2. register the server
```

Then: *"embed this video and tell me how surprising the ending is"*.

Also usable directly, no agent:

```bash
uvx wm-mcp surprise clip.mp4
uvx wm-mcp embed clip.mp4
uvx wm-mcp warmup     # pre-download weights so first real call is fast
uvx wm-mcp doctor     # environment diagnostic + timed inference — paste into bug reports
uvx wm-mcp info
```

## HTTP API (non-MCP consumers)

The same three primitives over localhost HTTP — for scripts, notebooks, or
anything that doesn't speak MCP:

```bash
uvx wm-mcp serve --http          # 127.0.0.1:8642
curl -s localhost:8642/health
curl -s -X POST localhost:8642/surprise -H 'content-type: application/json' \
     -d '{"source": "clip.mp4"}'
```

Endpoints: `GET /health`, `GET /info`, `POST /embed`, `POST /predict`,
`POST /surprise` — one-to-one with the MCP tools. Binds to localhost, no auth;
don't expose it to the internet.

## Things to ask your agent

- *"Where does this clip stop being predictable? ~/Downloads/dashcam.mp4"*
- *"Scan the last 10 minutes of warehouse.mp4 and tell me if anything unexpected happens."* (long footage scans in budgeted chunks — the agent resumes automatically via `resume_s`)
- *"Here's a photo of the finished assembly (goal.jpg). Does assembly-run-3.mp4 end up looking like it?"*
- *"Are clip_a.mp4 and clip_b.mp4 the same scene?"*
- *"Embed the first 30 seconds and the last 30 seconds of the match and tell me how similar they are."*

## Hardware & honest numbers

- Apple Silicon (MPS): ≥16GB unified memory. Measured on an M2 Pro / 16GB:
  `surprise` on a 6s clip = **~10-18s end to end** (fp16, 8fps sampling);
  `wm-mcp doctor` reports model load 9.3s, ~1.7s encode + ~1.1s predict per
  16-frame window. A full 64-frame `embed_video` is the heavyweight case —
  expect minutes, not seconds.
- NVIDIA (≥8GB VRAM): expected to work and to be faster — **currently untested
  by the author**. Run `wm-mcp doctor` and open an issue with the output; first
  confirmed CUDA report gets a thank-you in this README.
- CPU works; it's just slow.
- The model loads in fp16 on GPU devices (fp32 attention over 8k tokens will
  swap a 16GB machine — we measured it, you don't want it).
- Bigger machine? `WM_MODEL=facebook/vjepa2-vitg-fpc64-256 wm-mcp serve` swaps
  in a larger checkpoint (same V-JEPA 2 family only).

## Known limitations

- **Handles don't survive server restarts.** Embeddings live in process memory;
  a new MCP session starts empty. Re-embed (it's seconds) rather than storing
  handles long-term.
- **NVIDIA and Windows are untested.** Linux CPU is exercised in CI; macOS/MPS
  is the developed-on path.
- **`predict` is masked in-window prediction**, capped at clip length. It is not
  open-ended future simulation, and we won't pretend otherwise.
- **First run downloads ~4GB** (PyTorch + weights). `wm-mcp warmup` front-loads
  this; don't skip it before registering the MCP server.

## What this is (and isn't)

This is a **probe** — the smallest useful bridge between LLM agents and video world
models, shipped to find out whether anyone wants that bridge. It is not a robotics
stack, not a leaderboard, not a hosted service, and `predict` is honest about what
V-JEPA 2's predictor actually does (masked latent prediction within a clip), not a
marketing claim about simulating the future.

Public kill criteria, decided before launch: if by day 14 this has under 200 stars
AND under 10 substantive issues/PRs from strangers AND under 3 unprompted
integrations — it gets archived with a public retro. If it resonates, the next step
is a protocol, not a feature list.

## License

MIT. V-JEPA 2 weights are MIT-licensed by Meta FAIR.
