Metadata-Version: 2.4
Name: talkthrough-mcp
Version: 0.1.0
Summary: Local-first MCP server that turns narrated screen recordings into agent-ready structured data: timestamped transcript, scene keyframes, OCR text, and wall-clock anchoring.
Keywords: mcp,mcp server,model-context-protocol,screen-recording,video,transcription,whisper,ocr,feedback
Author: korovin-aa97
Author-email: korovin-aa97 <korovin.aa97@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Dist: faster-whisper>=1.2.1
Requires-Dist: mcp>=1.28.1
Requires-Dist: pillow>=12.3.0
Requires-Dist: rapidocr>=3.9.1
Requires-Dist: static-ffmpeg>=3.0
Requires-Python: >=3.11, <3.14
Project-URL: Homepage, https://github.com/korovin-aa97/talkthrough-mcp
Project-URL: Repository, https://github.com/korovin-aa97/talkthrough-mcp
Project-URL: Issues, https://github.com/korovin-aa97/talkthrough-mcp/issues
Project-URL: Changelog, https://github.com/korovin-aa97/talkthrough-mcp/blob/main/CHANGELOG.md
Description-Content-Type: text/markdown

# talkthrough-mcp

<!-- mcp-name: io.github.korovin-aa97/talkthrough-mcp -->

[![ci](https://github.com/korovin-aa97/talkthrough-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/korovin-aa97/talkthrough-mcp/actions/workflows/ci.yml)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![python](https://img.shields.io/badge/python-3.11%E2%80%933.13-blue.svg)](pyproject.toml)
<!-- uncomment at PyPI publish: [![PyPI](https://img.shields.io/pypi/v/talkthrough-mcp.svg)](https://pypi.org/project/talkthrough-mcp/) -->

[Quickstart](#quickstart) · [Tools](#tools) · [FAQ](#faq) ·
[Troubleshooting](docs/TROUBLESHOOTING.md) · [Changelog](CHANGELOG.md) ·
[Contributing](CONTRIBUTING.md)

**Feedback ingestion for AI agents.** Record your screen and talk; your agent
does the rest — files the bugs, writes the spec, builds the backlog.

![talkthrough demo: process a narrated recording, then query it lazily](assets/demo.gif)

`talkthrough-mcp` is a local-first MCP server that turns a narrated screen
recording (or any video/audio file) into agent-ready structured data:
timestamped transcript segments, scene-change keyframes, OCR'd on-screen text,
and wall-clock anchoring. Everything is served through lazy retrieval tools, so
a 30-minute recording never floods the model context — the agent pulls exactly
the transcript slice, moment bundle, or frame it needs.

There is no LLM inside the server and no cloud anywhere in the path: ffmpeg,
faster-whisper, and RapidOCR run on your machine, and the calling agent brings
the intelligence. What makes it different from screen-recorder SaaS and
video-analyzer MCPs: it works on arbitrary local files, it ships the agent
workflows (server prompts + example agents), and it anchors every timestamp to
**wall-clock time** — so "the moment I said the checkout hung" maps straight to
the right window of your server logs.

## Quickstart

One command, no system dependencies: ffmpeg falls back to a bundled build,
OCR is pip-only, and whisper models download themselves on first use.

<!-- gen:install — generated by scripts/gen_integrations.py; do not hand-edit -->

[![Install in Cursor](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/en/install-mcp?name=talkthrough&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ0YWxrdGhyb3VnaC1tY3AiXX0%3D)
[![Install in VS Code](https://img.shields.io/badge/VS_Code-Install_Server-0098FF?style=flat-square&logo=visualstudiocode&logoColor=white)](https://vscode.dev/redirect/mcp/install?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%2C%22type%22%3A%22stdio%22%7D)
[![Install in VS Code Insiders](https://img.shields.io/badge/VS_Code_Insiders-Install_Server-24bfa5?style=flat-square&logo=visualstudiocode&logoColor=white)](https://insiders.vscode.dev/redirect/mcp/install?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%2C%22type%22%3A%22stdio%22%7D&quality=insiders)
[![Add to LM Studio](https://files.lmstudio.ai/deeplink/mcp-install-light.svg)](https://lmstudio.ai/install-mcp?name=talkthrough&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ0YWxrdGhyb3VnaC1tY3AiXX0%3D)
[![Add to Kiro](https://kiro.dev/images/add-to-kiro.svg)](https://kiro.dev/launch/mcp/add?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%7D)
[![Install in Goose](https://block.github.io/goose/img/extension-install-dark.svg)](https://block.github.io/goose/extension?cmd=uvx&arg=talkthrough-mcp&id=talkthrough&name=Talkthrough&description=Turn%20narrated%20screen%20recordings%20into%20agent-ready%20data)

### Claude Code

```bash
claude mcp add -s user talkthrough -- uvx talkthrough-mcp
```

Or install the full plugin (server + the five workflow commands + the triage
agent + an agent skill):

```
/plugin marketplace add korovin-aa97/talkthrough-mcp
/plugin install talkthrough@talkthrough
```

### Every other MCP client

<details>
<summary><b>Claude Desktop</b></summary>

`claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}
```

More: [`integrations/claude-desktop/`](integrations/claude-desktop/)

</details>

<details>
<summary><b>Cursor</b></summary>

`~/.cursor/mcp.json (or project .cursor/mcp.json)`:

```json
{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}
```

More: [`integrations/cursor/`](integrations/cursor/)

</details>

<details>
<summary><b>OpenAI Codex CLI</b></summary>

`~/.codex/config.toml (or project-scoped .codex/config.toml in trusted projects)`:

```toml
[mcp_servers.talkthrough]
command = "uvx"
args = ["talkthrough-mcp"]
```

More: [`integrations/codex/`](integrations/codex/)

</details>

<details>
<summary><b>Gemini CLI</b></summary>

`~/.gemini/settings.json`:

```json
{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}
```

More: [`integrations/gemini-cli/`](integrations/gemini-cli/)

</details>

<details>
<summary><b>Cline / Roo Code</b></summary>

`cline_mcp_settings.json (via MCP Servers UI)`:

```json
{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}
```

More: [`integrations/cline/`](integrations/cline/)

</details>

<details>
<summary><b>OpenClaw</b></summary>

`~/.openclaw/openclaw.json`:

```json
{
  "mcp": {
    "servers": {
      "talkthrough": {
        "command": "uvx",
        "args": [
          "talkthrough-mcp"
        ]
      }
    }
  }
}
```

More: [`integrations/openclaw/`](integrations/openclaw/)

</details>

<details>
<summary><b>OpenCode</b></summary>

`opencode.json (project) or ~/.config/opencode/opencode.json`:

```json
{
  "mcp": {
    "talkthrough": {
      "type": "local",
      "command": [
        "uvx",
        "talkthrough-mcp"
      ],
      "enabled": true
    }
  }
}
```

More: [`integrations/opencode/`](integrations/opencode/)

</details>

<details>
<summary><b>Goose</b></summary>

`~/.config/goose/config.yaml`:

```yaml
extensions:
  talkthrough:
    enabled: true
    type: stdio
    cmd: uvx
    args: ["talkthrough-mcp"]
```

More: [`integrations/goose/`](integrations/goose/)

</details>

<details>
<summary><b>GitHub Copilot CLI</b></summary>

`~/.copilot/mcp-config.json`:

```json
{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}
```

More: [`integrations/copilot-cli/`](integrations/copilot-cli/)

</details>

<details>
<summary><b>Windsurf</b></summary>

`~/.codeium/windsurf/mcp_config.json`:

```json
{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}
```

More: [`integrations/windsurf/`](integrations/windsurf/)

</details>

<details>
<summary><b>Zed</b></summary>

`settings.json (Zed)`:

```json
{
  "context_servers": {
    "talkthrough": {
      "source": "custom",
      "command": {
        "path": "uvx",
        "args": [
          "talkthrough-mcp"
        ]
      }
    }
  }
}
```

More: [`integrations/zed/`](integrations/zed/)

</details>

Any other MCP stdio client uses the same server command: `uvx talkthrough-mcp`.
Per-engine folders with exactly these snippets plus verification steps live
in [`integrations/`](integrations/); agents can self-install via
[`llms-install.md`](llms-install.md).

<!-- /gen:install -->

### Local checkout (development)

```bash
git clone https://github.com/korovin-aa97/talkthrough-mcp
claude mcp add talkthrough -- uv run --directory /path/to/talkthrough-mcp talkthrough-mcp
```

Then, in your agent:

> Process `~/Desktop/recording.mov` and triage it — or just invoke the
> `triage-recording` server prompt.

## Tools

| Tool | What it does |
|---|---|
| `process_media(path, recorded_at?, vocabulary?, language?, model?, force?)` | Ingest a video/audio file: local STT, keyframes, OCR, wall-clock. Returns a compact summary. Idempotent by content hash — re-calls are instant. |
| `get_transcript(job_id, start_ms?, end_ms?, format?)` | Paginated transcript as `segments`, `text`, or `srt`; truncation returns `next_start_ms`. |
| `get_frames(job_id, at_ms? \| start_ms?+end_ms?, max_frames?, include_duplicates?)` | Keyframe images nearest a timestamp or evenly thinned across a range (unique frames by default, max 6/call). |
| `get_moment(job_id, start_ms, end_ms)` | The "one remark" bundle: transcript slice + up to 3 frames + their OCR text + wall-clock range. |
| `search(job_id, query)` | Substring search over the transcript AND on-screen OCR text; hits carry `t_ms`/`t_wall` and frame refs. |
| `extract_frame(job_id, at_ms, crop?)` | Exact-timestamp full-resolution re-extract from the source video (optional crop) when keyframes miss the instant. |
| `list_jobs()` | Recent processed recordings with durations, wall-clock starts, and counts. |

Every tool description ships 10+ usage examples, so agents pick the right tool
without extra prompting.

## Server prompts (slash commands in MCP clients)

| Prompt | Workflow |
|---|---|
| `triage-recording` | Narrated screencast → precise findings JSON (bug/feature/question routing, frame evidence) |
| `spec-from-workshop` | Recorded workshop → structured spec with quoted decisions and open questions |
| `backlog-from-demo` | Product demo → prioritized backlog with timestamped evidence |
| `meeting-actions` | Meeting audio → action items, decisions, open questions |
| `correlate-with-logs` | Recording remarks ↔ system logs via wall-clock windows |

The same prompts live as plain files in [`examples/prompts/`](examples/prompts/)
if your client doesn't surface MCP prompts. The findings contract used by
`triage-recording` is [`examples/output-contract.schema.json`](examples/output-contract.schema.json).

## Works as a skill too (no MCP required)

The same workflow ships as a cross-engine [Agent Skill](https://agentskills.io)
at [`.agents/skills/talkthrough/`](.agents/skills/talkthrough/) — Claude Code,
Codex CLI (`$talkthrough`), Cursor, Copilot, Gemini CLI, Goose and other
SKILL.md-compatible tools read it. Agents without MCP wiring can drive the
[CLI](#cli) directly: `talkthrough-mcp process recording.mov --json` prints the
same summary the MCP tool returns, and the job store is shared either way.

## Wall-clock anchoring

Every timestamped result carries both `t_ms` (video-relative) and `t_wall`
(ISO 8601 real time) once the recording start is known. Resolution ladder:

1. `recorded_at` parameter (agent/user override) → confidence `exact`
2. QuickTime `com.apple.quicktime.creationdate` tag, carries the local
   timezone (QuickTime Player recordings; ⌘⇧5 wrote it before macOS 26) → `high`
3. Container `creation_time` tag (UTC) → `medium` — macOS 26+ ⌘⇧5/ReplayKit
   screen recordings land here (no `creationdate` tag anymore); pass
   `recorded_at=` when local-tz `t_wall` matters
4. File mtime minus duration (recorders finalize files at recording END) → `low`
5. Nothing → tools still work with relative `t_ms` only

Why it matters: "the upload spinner froze *here*" becomes a ±30 s grep window
in your server logs.

## Privacy

Everything runs locally: your recordings never leave your machine, speech is
transcribed by a local whisper model, OCR is local ONNX inference, and there is
no telemetry. The only network access is one-time tool/model downloads (ffmpeg
build, whisper model, OCR models).

## Languages

Narration in any of Whisper's ~99 languages works: the language is
auto-detected per recording, and the summary reports both `language` and
`language_probability` so agents can tell a confident detection from a shaky
one (silence or music at the start can fool the detector — pin it with
`language="ru"` and `force=true` when that happens).

Pick the model for your languages — per call (`model=` parameter, agents do
this themselves when a transcript comes back garbled) or as the server
default (`TALKTHROUGH_WHISPER_MODEL`):

| Model | Size | Best for |
|---|---|---|
| `small` (default) | 464 MB | English and major-language narration on CPU |
| `large-v3-turbo` | ~1.5 GB | **recommended for non-English** — near-large quality at near-small speed |
| `medium` | ~1.5 GB | conservative alternative to turbo |
| `tiny` / `base` | 75–145 MB | quick drafts, CI |
| `*.en` variants | — | English-only, slightly faster/better for EN |

Tips that work in every language: pass product names via
`vocabulary="Term1, Term2"` (biases the decoder so jargon survives), and note
that the workflow prompts instruct agents to write digests in the
**narrator's language** while keeping quotes verbatim — the server never
translates (exact quotes are evidence; translation is the agent's job).

On-screen text (OCR) defaults to RapidOCR's Latin + Chinese models. For other
scripts set `TALKTHROUGH_OCR_LANG` to your language — `ru`/`uk` (→ the
`eslav` pack), `ja`, `ko`, `ar`, `hi`, `el`, `th`, or any RapidOCR pack name
like `cyrillic` — and reprocess with `force=true`; the matching recognition
model downloads once. Spoken-language support is unaffected either way.

## Configuration

| Env var | Default | Meaning |
|---|---|---|
| `TALKTHROUGH_WHISPER_MODEL` | `small` | default whisper model (`tiny`/`base`/`small`/`medium`/`large-v3`/`large-v3-turbo`); the `model` tool param overrides per call |
| `TALKTHROUGH_OCR` | `on` | set `off` to skip OCR |
| `TALKTHROUGH_OCR_LANG` | Latin+Chinese | recognition script for on-screen text: a language code (`ru`, `ja`, `ko`, `ar`, `hi`, …) or a RapidOCR pack name (`eslav`, `cyrillic`, `latin`, …); the model downloads once |
| `TALKTHROUGH_OCR_PARAMS` | — | advanced: JSON object of raw RapidOCR params merged over the derived ones, e.g. `{"Rec.lang_type": "cyrillic"}` |
| `TALKTHROUGH_MAX_SECONDS` | `7200` | max media duration |
| `TALKTHROUGH_MAX_FRAMES` | `600` | keyframe cap per job |
| `TALKTHROUGH_HOME` | `~/.talkthrough` | job store root |

## CLI

The pipeline is also a CLI — useful for pre-processing long recordings outside
an agent session (the store is content-addressed, so the agent then queries the
same job instantly):

```bash
talkthrough-mcp process ~/Videos/long-session.mov   # prints the summary
talkthrough-mcp process demo.mov --json             # machine-readable
talkthrough-mcp gc --keep-days 30                   # clean the job store
talkthrough-mcp serve                               # stdio MCP server (default)
```

First run notes: missing system ffmpeg triggers a one-time `static-ffmpeg`
download; the first transcription downloads the whisper model (~460 MB for
`small`); both are cached. After that, expect roughly 3× faster than real time
on an Apple-Silicon CPU with the default model, OCR included (a 2-minute clip
processes in ~40 s) — and instant re-runs on the same file. Progress streams
as MCP progress notifications, and the CLI prints stage lines. More:
[docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).

## Windows (best-effort)

CI runs lint, the unit suite, and a full CLI smoke on `windows-latest`
(static-ffmpeg Windows build, whisper `tiny` transcription, OCR, and the
instant idempotent re-run). Notes: the per-job lock is POSIX `fcntl` and
degrades to a no-op on Windows — fine for a single-user machine; quote paths
with spaces (`uv run talkthrough-mcp process "C:\Videos\Screen Recording.mp4"`).
Windows is not a release gate — if something breaks, please open an issue.

## Supported inputs

Video: `.mov` `.mp4` `.webm` `.mkv` — audio-only: `.m4a` `.mp3` `.wav` `.ogg`
`.flac` (transcript tools only; frame tools explain why they're unavailable).
Local files only.

## Limitations

Honest edges, so you can decide fast:

- **One speaker stream.** No diarization yet — "who said it" isn't tracked
  ([#4](https://github.com/korovin-aa97/talkthrough-mcp/issues/4)).
- **Local files only.** No URL/YouTube ingestion
  ([#5](https://github.com/korovin-aa97/talkthrough-mcp/issues/5)) — download
  first.
- **Keyframes + transcript, not motion analysis.** A glitch *between* scene
  changes can be invisible in the frame set; `extract_frame` re-checks any
  instant, but frame-by-frame motion reasoning is your multimodal model's job.
- **STT quality tracks the model you pick.** The default `small` favors speed;
  non-English narration wants `model="large-v3-turbo"` (see
  [Languages](#languages)).
- **OCR reads crisp UI text well;** tiny or low-contrast print is best-effort.
- **Wall-clock confidence depends on recorder metadata** — worst case pass
  `recorded_at=` (see the ladder above).
- **Windows is best-effort** (see above).

## How it compares

| | talkthrough | cloud recorder SaaS | meeting notetakers | typical video-analyzer MCPs |
|---|---|---|---|---|
| Runs fully locally | ✅ | ❌ | ❌ | varies |
| Any local video/audio file | ✅ | browser/app captures | meetings only | ✅ |
| Wall-clock anchoring (log correlation) | ✅ | ❌ | ❌ | ❌ |
| Ships agent workflows (prompts, skill, findings contract) | ✅ | ❌ | ❌ | ❌ |
| OCR of on-screen text, searchable | ✅ | some | ❌ | rare |

## FAQ

**Why not just upload the video to a multimodal model (e.g. Gemini)?**
For a short, non-sensitive clip — do that. The trade-offs appear with length
and sensitivity: an hour of screen recording costs on the order of a million
tokens *per question*, the file leaves your machine, and you still can't map a
remark to `14:32:07 UTC` to grep your server logs. talkthrough indexes once,
locally, then answers any number of follow-ups from the index.

**Why not screenpipe?**
Different job. screenpipe is an always-on recorder of *your* machine going
forward (commercial license). It can't open the `.mov` a teammate or customer
just sent you. talkthrough analyzes any file it's handed — the two compose
fine.

**There are agent skills that "watch" videos. Why a server with an index?**
Watch-style skills push a budgeted frame dump into the context window (and go
sparse on long videos), often call cloud STT for the audio, and keep nothing.
talkthrough builds a persistent local index — transcript + OCR, full-text
searchable — retrieves exact frames lazily, anchors everything to wall-clock
time, and answers the next question without reprocessing.

**I use Jam for bug reports — do I need this?**
Keep Jam for browser bugs: console+network captured at record time is great
evidence. talkthrough covers what a browser extension can't — desktop apps,
mobile screencasts, ops incidents, meetings, any file — with no account, and
correlates with *server-side* logs via wall-clock time.

**Can't I just script ffmpeg + whisper myself?**
Yes — that's exactly this pipeline. What you'd be rebuilding: scene-change
detection with perceptual dedup, OCR, transcript+OCR search, the wall-clock
ladder, MCP tools with embedded usage examples, five workflow prompts, and a
findings contract. One `uvx` command instead of an afternoon of glue.

**Is it really local? What leaves my machine?**
Nothing at runtime. The network is used only for one-time downloads (ffmpeg
build, whisper/OCR models). No telemetry. See [Privacy](#privacy) — and
[SECURITY.md](SECURITY.md) treats a violation of this promise as a
vulnerability.

## For agents & tooling

Machine-readable entry points, so AI agents can install and use this server
without a human reading docs:

- [`llms-install.md`](llms-install.md) — step-by-step install instructions for agents
- [`llms.txt`](llms.txt) — index of the documentation
- [`.agents/skills/talkthrough/SKILL.md`](.agents/skills/talkthrough/SKILL.md) — an [Agent Skill](https://agentskills.io) teaching the tool workflow; discovered automatically inside a checkout by Codex CLI (`$talkthrough`) and readable by Claude Code, Cursor, Copilot, Gemini CLI and other SKILL.md-compatible tools
- [`AGENTS.md`](AGENTS.md) — instructions for coding agents contributing to this repo
- [`server.json`](server.json) — MCP registry manifest
- [`integrations/`](integrations/) — per-engine adapters, all generated from one source of truth and drift-tested (incl. the Claude Code plugin under [`integrations/claude-code/`](integrations/claude-code/))

## Roadmap (not in v1)

URL/YouTube ingestion · speaker diarization · cloud STT · embeddings/semantic
search · hosted/remote mode · `.mcpb` bundle · whisper.cpp backend

## License

MIT
