Metadata-Version: 2.5
Name: miragea-dub
Version: 0.1.0
Summary: Open-source multi-speaker video dubbing: transcribe, find who speaks when, translate, and re-voice every speaker with their own cloned voice, with timing fit, ducked background, dubbed MP4, multi-language audio track and SRT/VTT. CLI + MCP server + Agent Skill, voiced by Miragea.
Project-URL: Homepage, https://codeberg.org/miragea-voice/miragea-dub
Project-URL: Documentation, https://codeberg.org/miragea-voice/miragea-dub#readme
Project-URL: Issues, https://codeberg.org/miragea-voice/miragea-dub/issues
Project-URL: Miragea API, https://miragea.ai
Author: miragea-dub contributors
License-Expression: MIT
License-File: LICENSE
Keywords: agent-skill,ai-dubbing,ai-voice-over,dubbing,elevenlabs-dubbing-alternative,heygen-alternative,localization,mcp,mcp-server,miragea,multi-speaker,speaker-diarization,srt,subtitles,text-to-speech,video-dubbing,video-translation,voice-cloning,youtube-multi-language-audio
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Classifier: Topic :: Multimedia :: Video
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: anyio>=4
Requires-Dist: httpx>=0.27
Requires-Dist: mcp>=1.9
Requires-Dist: numpy>=1.26
Requires-Dist: pydantic>=2
Provides-Extra: demucs
Requires-Dist: demucs>=4; extra == 'demucs'
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: pyannote
Requires-Dist: pyannote-audio>=3.1; extra == 'pyannote'
Provides-Extra: whisper
Requires-Dist: faster-whisper>=1.0; extra == 'whisper'
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://codeberg.org/miragea-voice/miragea-dub/raw/branch/main/docs/images/cover.png" alt="miragea-dub banner: two speakers' lanes in English above the same two lanes dubbed into Spanish, each speaker keeping their own voice" width="100%">
</p>

# miragea-dub: multi-speaker AI video dubbing, open source

**miragea-dub is an open-source command-line tool, MCP server and Agent Skill that dubs a video into another language while keeping a separate cloned voice for every speaker.** It transcribes the video, works out who speaks when, lets you (or your AI agent) translate each line, clones every speaker's voice from the video itself with the [Miragea](https://miragea.ai) voice API, fits each translated line into the original timing, ducks the original audio underneath, and exports a dubbed MP4, a stand-alone audio track for YouTube multi-language audio, and SRT/VTT subtitles. MIT licensed, Python 3.10+, works offline in `demo` mode.

[![license: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![python](https://img.shields.io/badge/python-3.10%2B-blue)](pyproject.toml) [![tests](https://img.shields.io/badge/tests-28%20passing-brightgreen)](tests)

## Demo: a real two-speaker dub

An English clip with two speakers (Ana and Ben) dubbed into Spanish. Both voices were cloned from the clip itself, nothing was pre-recorded:

- [Watch the dubbed MP4 (Spanish, burned-in subtitles, original English kept as 2nd audio track)](https://codeberg.org/miragea-voice/miragea-dub/raw/branch/main/examples/demo-dubbed-es.mp4)
- [Original English audio](https://codeberg.org/miragea-voice/miragea-dub/raw/branch/main/examples/real/original-audio.mp3)
- [Spanish subtitles](examples/output/two-voices.es.srt) and [timing report](examples/output/two-voices.es.report.json)

![Animated preview of the Spanish dub with burned-in subtitles and a waveform](https://codeberg.org/miragea-voice/miragea-dub/raw/branch/main/docs/images/demo.gif)

Round-trip check: transcribing the Spanish track again returns the translated script almost word for word, and the diarizer assigned all 6 lines to the right speaker.

## Quick start

```bash
pipx install miragea-dub          # or: uvx miragea-dub ...   (needs ffmpeg on PATH)
miragea-dub demo                  # offline end-to-end run with test tones, no key needed
export MIRAGEA_API_KEY=...        # create a key at https://miragea.ai/workspace/api
miragea-dub doctor                # checks ffmpeg, key, connectivity

# 1) transcribe + find speakers -> dub.json
miragea-dub prepare interview.mp4 --to es --speakers 2
# 2) translate: let your AI agent fill "translation" in dub.json, or use any OpenAI-compatible LLM
miragea-dub translate interview.dub/dub.json --translator openai
# 3) voice, fit, mix, export
miragea-dub render interview.dub/dub.json --model expressive
# or all in one:  miragea-dub dub interview.mp4 --to es --translator openai
```

Outputs next to the video: `interview.es.mp4` (dub track + original track + subtitles), `interview.es.m4a`, `.srt`, `.vtt` and `interview.es.report.json`.

## How it works

![Flow diagram: transcribe, diarize, translate, clone and voice, fit and mix, then publishable files](https://codeberg.org/miragea-voice/miragea-dub/raw/branch/main/docs/images/flow.png)

1. **Transcribe**: Miragea speech-to-text (no GPU), local `faster-whisper`, or your own SRT/VTT/JSON. Miragea returns subtitle cues cut at a fixed width, so cues are regrouped into whole sentences using the punctuated full text.
2. **Diarize**: a built-in, GPU-free speaker clustering on voice features (or `pyannote`, or an RTTM file). Speaker tags already in your transcript (`[Ana] ...`, `S2: ...`) are kept.
3. **Translate**: every line gets a length budget so the translation can be spoken inside the original time slot. Backends: `agent` (your AI agent edits `dub.json`), `openai` (any OpenAI-compatible endpoint), `file`, `copy`.
4. **Clone and voice**: an 8-20 second reference clip is cut per speaker from the video, hosted on a temporary public link, and each line is synthesized with that voice through the Miragea API. Requests are idempotent and cached, so a re-run never pays twice.
5. **Fit and mix**: lines are sped up (up to 1.35x by default) to fit, the original audio is ducked under speech (or removed, or replaced by your music stem, or separated with Demucs), then everything is muxed.

### Real timing report

![Timeline of the real run: for each speaker, outlined bars show the original slot and solid bars the dubbed Spanish line after fitting](https://codeberg.org/miragea-voice/miragea-dub/raw/branch/main/docs/images/timeline.png)

![Terminal capture of the real miragea-dub run: 6 lines, 2 speakers, 4 sped up, 0 trimmed](https://codeberg.org/miragea-voice/miragea-dub/raw/branch/main/docs/images/terminal-dub.png)

Lines that could not fit even at the speed limit are listed in `lines_to_shorten`; shorten them in `dub.json` and run `render` again (only changed lines are re-voiced).

## Use it from an AI agent

**MCP server** (Claude Desktop, Claude Code, Cursor, VS Code, Windsurf):

```json
{ "mcpServers": { "miragea-dub": { "command": "uvx", "args": ["--from", "miragea-dub", "miragea-dub-mcp"],
  "env": { "MIRAGEA_API_KEY": "your-key" } } } }
```

Tools: `prepare_dub`, `save_translations`, `show_project`, `estimate_dub`, `render_dub`, `list_voices`. The agent is the translator, so no second LLM key is needed.

**Agent Skill**: install `video-dubbing` from [miragea-skills](https://codeberg.org/miragea-voice/miragea-skills), or tell your agent: `Install https://miragea.ai/skills/miragea-audio/install.md`.

## How it compares

| | miragea-dub | [VideoLingo](https://github.com/Huanshere/VideoLingo) | [pyvideotrans](https://github.com/jianchang512/pyvideotrans) | Hosted dubbing apps |
|---|---|---|---|---|
| License | MIT | Apache-2.0 | GPL-3.0 | closed SaaS |
| Interface | CLI + MCP + Agent Skill | Streamlit app | desktop GUI + CLI | web app |
| Separate cloned voice per speaker | yes (from the video itself) | depends on TTS backend | depends on backend | yes |
| Runs without a GPU | yes (cloud ASR + voices) | local models recommended | optional | n/a |
| Your AI agent can drive it | yes (MCP, skill) | no | no | usually no |
| Editable project file between steps | `dub.json` | partly | partly | web editor |
| Speaker count / timing report | yes | no | no | varies |

Star counts at time of writing: VideoLingo 19.1k, pyvideotrans 19.7k. Those projects are bigger and have GUIs; miragea-dub is the small, scriptable, agent-friendly option.

## FAQ

**How do I dub a video into another language with AI?** Install miragea-dub, run `miragea-dub dub video.mp4 --to es --translator openai`, and you get a dubbed MP4 plus subtitles.

**Can I dub a video with two or more speakers and keep each voice?** Yes. Speakers are separated by diarization (or by tags in your own transcript) and each one is cloned from their own lines.

**Does it work for a YouTube video with multiple audio tracks?** The `.m4a` output is the stand-alone dubbed track you can upload as an additional language audio track; the `.mp4` already carries dub and original tracks.

**Is there a free open-source alternative to ElevenLabs or HeyGen dubbing?** The pipeline, diarization, timing fit and mixing are free and MIT; the voices come from the Miragea API, which needs a key.

**Can an AI agent (Claude, Cursor, Codex) dub videos?** Yes, through the MCP server or the `video-dubbing` skill. The agent writes the translations itself.

**How do I keep the dubbed speech from running longer than the original?** Every line gets a word or character budget before translation, long lines are sped up to the limit, and what still does not fit is reported for shortening.

**Does it keep background music?** By default the original audio is ducked under speech. Use `--background file` with a music stem, `demucs` to separate stems, or `remove`.

**Which languages?** Anything your translator and the Miragea voices handle; the repo's own example is English to Spanish.

## Troubleshooting

- `litterbox upload failed (HTTP 403)`: the default `--ref-host auto` already falls back to other free hosts; or use `--ref-host "cmd:your-upload-command {path}"`, or `prefix:` with `--ref-dir` for your own server.
- Wrong speaker count: pass `--speakers N`, or supply `--transcript` with `[Name]` tags.
- A line sounds rushed: shorten its `translation` in `dub.json` and run `render` again.
- `401`: create a key at https://miragea.ai/workspace/api and `export MIRAGEA_API_KEY`.

## Development

```bash
uv venv && uv pip install -e ".[dev]"
python -m pytest        # 28 tests, offline: Miragea API is mocked with httpx.MockTransport
python docs/make_images.py
```

Base URL defaults to `https://miragea.ai/v1` (override with `MIRAGEA_BASE_URL`). See [CHANGELOG](CHANGELOG.md) and [llms.txt](llms.txt). MIT, see [LICENSE](LICENSE).
