Metadata-Version: 2.4
Name: wm-mcp
Version: 0.3.0
Summary: Run a world model locally, and let any LLM agent call it as a tool. V-JEPA 2 as an MCP server.
Project-URL: Homepage, https://github.com/rian-one/wm
Project-URL: Issues, https://github.com/rian-one/wm/issues
Author: Rian Herrmann
License-Expression: MIT
License-File: LICENSE
Keywords: agents,jepa,mcp,model-context-protocol,v-jepa,video,world-model
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: av>=12.0
Requires-Dist: fastapi>=0.110
Requires-Dist: huggingface-hub>=0.30
Requires-Dist: mcp>=1.2
Requires-Dist: numpy>=1.26
Requires-Dist: pillow>=10.0
Requires-Dist: torch>=2.4
Requires-Dist: torchvision>=0.19
Requires-Dist: transformers>=4.53
Requires-Dist: uvicorn>=0.30
Description-Content-Type: text/markdown

# wm

**Run a world model locally, and let any LLM agent call it as a tool.**

![An LLM agent consulting a video world model](demo.gif)

*Live and unedited: Claude calls the `surprise` tool; V-JEPA 2 watches the clip and
reports where its predictions broke (z=3.1 at frames 56–71 — the exact splice point
of the test video). Wait time hidden, output verbatim.*

LLMs can *describe* what happens when a cup gets knocked off a table. A world model
can *watch* it happen and tell you, frame by frame, where reality stopped matching
its prediction. `wm` puts Meta's [V-JEPA 2](https://huggingface.co/facebook/vjepa2-vitl-fpc64-256)
(MIT-licensed, 0.3B params) behind an [MCP](https://modelcontextprotocol.io) server,
so Claude, or any MCP client, can consult a video world model the way it consults a
database.

```
uvx wm-mcp serve
```

That's the whole install. First run downloads PyTorch and ~1.2GB of model weights —
**minutes of downloading, not a hang.**

## The demo

Ask your agent:

> "Where does this clip stop being predictable? /path/to/video.mp4"

The agent calls `surprise()`, the world model slides a window over the video,
predicts each window's second half from its first, and reports the segment where
its prediction failed hardest — z-scored per video, so scores mean something.

## The three tools

| Tool | What it does |
|---|---|
| `embed_video(source)` | Video → latent sequence, stored server-side under an opaque handle. |
| `predict(handle, horizon_frames, target_handle?)` | Predict the trailing frames of a clip from its leading context (masked in-window prediction — horizon capped at clip length, this is not open-ended rollout). With `target_handle`, also returns cosine similarity between the prediction and a target embedding — goal-similarity scoring, computed server-side. |
| `surprise(source, window, stride)` | Per-segment "how wrong was the model" scores + the most surprising segment. Anomaly spotting, event detection, "did anything unexpected happen." |

Latents never cross the wire: tools exchange handle IDs, tensors stay in the server.

## Setup with Claude Code

```bash
claude mcp add wm -- uvx wm-mcp serve
```

Then: *"embed this video and tell me how surprising the ending is"*.

Also usable directly, no agent:

```bash
uvx wm-mcp surprise clip.mp4
uvx wm-mcp embed clip.mp4
uvx wm-mcp warmup     # pre-download weights so first real call is fast
uvx wm-mcp info
```

## HTTP API (non-MCP consumers)

The same three primitives over localhost HTTP — for scripts, notebooks, or
anything that doesn't speak MCP:

```bash
uvx wm-mcp serve --http          # 127.0.0.1:8642
curl -s localhost:8642/health
curl -s -X POST localhost:8642/surprise -H 'content-type: application/json' \
     -d '{"source": "clip.mp4"}'
```

Endpoints: `GET /health`, `GET /info`, `POST /embed`, `POST /predict`,
`POST /surprise` — one-to-one with the MCP tools. Binds to localhost, no auth;
don't expose it to the internet.

## Hardware & honest numbers

- Apple Silicon (MPS): ≥16GB unified memory. Measured on an M2 Pro / 16GB:
  `surprise` on a 6s clip = **~18s end to end** (11 windows, fp16). A full
  64-frame `embed_video` is the heavyweight case — expect minutes, not seconds.
- NVIDIA: ≥8GB VRAM, substantially faster.
- CPU works; it's just slow.
- The model loads in fp16 on GPU devices (fp32 attention over 8k tokens will
  swap a 16GB machine — we measured it, you don't want it).

## What this is (and isn't)

This is a **probe** — the smallest useful bridge between LLM agents and video world
models, shipped to find out whether anyone wants that bridge. It is not a robotics
stack, not a leaderboard, not a hosted service, and `predict` is honest about what
V-JEPA 2's predictor actually does (masked latent prediction within a clip), not a
marketing claim about simulating the future.

Public kill criteria, decided before launch: if by day 14 this has under 200 stars
AND under 10 substantive issues/PRs from strangers AND under 3 unprompted
integrations — it gets archived with a public retro. If it resonates, the next step
is a protocol, not a feature list.

## License

MIT. V-JEPA 2 weights are MIT-licensed by Meta FAIR.
