Metadata-Version: 2.4
Name: ffsubsync
Version: 0.5.1
Home-page: https://github.com/smacke/ffsubsync
Author: Stephen Macke
Author-email: stephen.macke@gmail.com
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: End Users/Desktop
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: auditok==0.1.5
Requires-Dist: chardet; python_version >= "3.7"
Requires-Dist: charset_normalizer
Requires-Dist: faust-cchardet
Requires-Dist: ffmpeg-python
Requires-Dist: numpy>=1.12.0
Requires-Dist: pysubs2; python_version < "3.7"
Requires-Dist: pysubs2>=1.2.0; python_version >= "3.7"
Requires-Dist: rich
Requires-Dist: srt>=3.0.0
Requires-Dist: tqdm
Requires-Dist: typing_extensions
Requires-Dist: webrtcvad-wheels
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist

FFsubsync
=======

[![CI Status](https://github.com/smacke/ffsubsync/actions/workflows/ci.yml/badge.svg)](https://github.com/smacke/ffsubsync/actions/workflows/ci.yml)
[![Support Ukraine](https://badgen.net/badge/support/UKRAINE/?color=0057B8&labelColor=FFD700)](https://github.com/vshymanskyy/StandWithUkraine/blob/main/docs/README.md)
[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![License: MIT](https://img.shields.io/badge/License-MIT-maroon.svg)](https://opensource.org/licenses/MIT)
[![Python Versions](https://img.shields.io/pypi/pyversions/ffsubsync.svg)](https://pypi.org/project/ffsubsync)
[![Documentation Status](https://readthedocs.org/projects/ffsubsync/badge/?version=latest)](https://ffsubsync.readthedocs.io/en/latest/?badge=latest)
[![PyPI Version](https://img.shields.io/pypi/v/ffsubsync.svg)](https://pypi.org/project/ffsubsync)


Language-agnostic automatic synchronization of subtitles with video, so that
subtitles are aligned to the correct starting point within the video.

Turn this:                       |  Into this:
:-------------------------------:|:-------------------------:
![](https://raw.githubusercontent.com/smacke/ffsubsync/master/resources/img/tearing-me-apart-wrong.gif)  |  ![](https://raw.githubusercontent.com/smacke/ffsubsync/master/resources/img/tearing-me-apart-correct.gif)

Try it in your browser (no install)
------------------------------------
You can now sync subtitles entirely in your browser — no Python, no `ffmpeg`, and
nothing to install. Your files never leave your machine (nothing is uploaded):

### 👉 https://smacke.github.io/ffsubsync

The browser version syncs against either a **correctly-synced reference subtitle** or a
**video / audio file** — audio is decoded in-browser with ffmpeg.wasm (large files are read
lazily, never uploaded). For bulk or scripted use, install the command-line tool below.

Helping Development
-------------------
Please consider [supporting Ukraine](https://github.com/vshymanskyy/StandWithUkraine/blob/main/docs/README.md)
rather than donating directly to this project. That said, at the request of
some, you can now help cover my coffee expenses using the Github Sponsors
button at the top, or using the below Paypal Donate button:

[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=XJC5ANLMYECJE)

Install
-------
First, make sure ffmpeg is installed. On MacOS, this looks like:
~~~
brew install ffmpeg
~~~
(Windows users: make sure `ffmpeg` is on your path and can be referenced
from the command line!)

Next, grab the package (compatible with Python >= 3.6):
~~~
pip install ffsubsync
~~~
If you want to live dangerously, you can grab the latest version as follows:
~~~
pip install git+https://github.com/smacke/ffsubsync@latest
~~~

Usage
-----
`ffs`, `subsync` and `ffsubsync` all work as entrypoints:
~~~
ffs video.mp4 -i unsynchronized.srt -o synchronized.srt
~~~

There may be occasions where you have a correctly synchronized srt file in a
language you are unfamiliar with, as well as an unsynchronized srt file in your
native language. In this case, you can use the correctly synchronized srt file
directly as a reference for synchronization, instead of using the video as the
reference:

~~~
ffsubsync reference.srt -i unsynchronized.srt -o synchronized.srt
~~~

`ffsubsync` uses the file extension to decide whether to perform voice activity
detection on the audio or to directly extract speech from an srt file.

If you omit `-i`, `ffsubsync` auto-detects input subtitles sitting next to the
reference that share its name, so you can simply run:
~~~
ffs video.mp4
~~~
This picks up files like `video.srt` and `video.en.srt` from the reference's
directory and writes the synced result for each to a `<name>.synced.srt`
alongside it (e.g. `video.synced.srt`), leaving the originals untouched.
Previously-synced `*.synced.srt` outputs are skipped, so re-running is safe. Add
`--overwrite-input` to overwrite the detected file(s) in place instead. Auto-
detection is skipped when subtitles are piped in on stdin.

The reference can also be a remote URL instead of a local file. Anything ffmpeg
can read directly works as a video / audio reference, and remote subtitle files
work as a reference too:
~~~
ffs "https://example.com/video.mp4" -i unsynchronized.srt -o synchronized.srt
ffs "https://example.com/reference.srt" -i unsynchronized.srt -o synchronized.srt
~~~
Supported protocols are `http(s)://`, `rtmp://`, `rtsp://`, and `ftp://`.
Processing streams the reference over the network, so it depends on connection
stability; for large or flaky sources, downloading first is more reliable.
Sibling-subtitle auto-detection (the no-`-i` form above) is local-only and is
skipped for remote references.

To speed up long references, `--max-duration-seconds N` processes only the
first `N` seconds (measured from `--start-seconds`). This is especially helpful
for remote references, since ffmpeg stops reading—and therefore downloading—once
that duration is reached:
~~~
ffs "https://example.com/video.mp4" -i unsynchronized.srt -o synchronized.srt --max-duration-seconds 600
~~~
On flaky connections, `--extract-audio-first` can be more reliable: instead of
holding a network stream open throughout speech detection, it first copies the
remote audio track to a local temp file (no re-encode) and runs detection on
that. It is ignored for local references and composes with
`--max-duration-seconds`:
~~~
ffs "https://example.com/video.mp4" -i unsynchronized.srt -o synchronized.srt --extract-audio-first
~~~

For long references where `--max-duration-seconds` would miss desync that only
shows up later, `--multi-segment-sync` instead samples several short segments
spread across the whole reference and runs speech detection on just those:
~~~
ffs "https://example.com/video.mp4" -i unsynchronized.srt -o synchronized.srt --multi-segment-sync
~~~
Only the sampled audio is extracted (and, for remote references, downloaded),
but because each segment keeps its true position on the timeline, the usual
framerate-ratio and offset search is unchanged—so a framerate mismatch is still
detected and corrected. Tune it with `--segment-count N` (default 8),
`--skip-intro-outro` (skip the first 30s / last 60s, which often lack dialogue),
and `--parallel-workers N` (overlap segment downloads, default 4). It applies to
video / audio references only.

Docker
------
Prebuilt images are published to the GitHub Container Registry. Pull the
latest release with:
~~~
docker pull ghcr.io/smacke/ffsubsync:latest
~~~
Run it by mounting the directory with your video and subtitles into `/video`:
~~~
docker run --rm -v "$PWD":/video ghcr.io/smacke/ffsubsync:latest \
  video.mp4 -i unsynchronized.srt -o synchronized.srt
~~~
You can also build the image yourself. The multi-stage Dockerfile defaults to
installing from the current working tree:
~~~
docker build -t ffsubsync .
~~~
To install a specific version from PyPI instead, set `FFSUBSYNC_VERSION`:
~~~
docker build -t ffsubsync --build-arg FFSUBSYNC_VERSION=0.4.31 .
~~~

Using as a Library
------------------
`ffsubsync` can be driven programmatically via `ffsubsync.run`, which accepts an
`argparse.Namespace` (build one with `make_parser`). To surface progress in your
own UI, pass a `progress_handler` callback; it is invoked repeatedly while the
reference's audio is being decoded with a `ProgressInfo` describing how far along
the extraction is:
~~~python
import ffsubsync
from ffsubsync.ffsubsync import make_parser

def on_progress(info: ffsubsync.ProgressInfo) -> None:
    # info.processed_seconds / info.total_seconds (total may be None);
    # info.fraction is a 0.0-1.0 ratio (None if the total is unknown).
    if info.fraction is not None:
        print(f"{info.fraction:.0%}")

args = make_parser().parse_args(["ref.mkv", "-i", "in.srt", "-o", "out.srt"])
result = ffsubsync.run(args, progress_handler=on_progress)
~~~
The handler is called only for the video / audio reference path (the dominant
cost of a sync). Exceptions it raises are logged and swallowed, so a buggy
handler can never abort syncing.

Character Encoding
------------------
Subtitle files in the wild come in a mess of legacy encodings (Windows-1251 for
Cyrillic, GBK / Big5 for Chinese, Latin-1, Shift-JIS, UTF-16 with a BOM, ...).
Robustly handling these is something `ffsubsync` does well compared to other
subtitle sync tools, and it happens automatically: `--encoding` defaults to
`infer`, which reads the input as raw bytes and auto-detects the encoding. Under
the hood it tries up to three detectors in order and takes the first that
answers — `cchardet`, then
[charset_normalizer](https://pypi.org/project/charset-normalizer/), then
[chardet](https://pypi.org/project/chardet/) — and decodes with
`errors="replace"` so a slightly-off guess degrades gracefully instead of
crashing. BOMs are handled for free (the raw bytes are what the detector sees).

If auto-detection guesses wrong, force it with e.g. `--encoding windows-1251`.
Output is written as UTF-8 by default; pass `--output-encoding same` to preserve
the input's encoding instead, or name any codec explicitly. When the reference is
itself a subtitle file, `--reference-encoding` controls it (also `infer` by
default).

One cross-version caveat worth knowing: the fastest / often most accurate
detector, `cchardet`, is supplied by the maintained
[faust-cchardet](https://pypi.org/project/faust-cchardet/) fork (which replaced
the unmaintained original `cchardet` and installs under the module name
`cchardet`). It is only declared as a dependency for **Python &lt; 3.13**
(`faust-cchardet;python_version<'3.13'` in `requirements.txt`). On **Python
3.13+** it isn't installed, so `import cchardet` fails quietly and detection falls
back to the pure-Python `charset_normalizer` and `chardet`. This is usually
indistinguishable in practice, but on an ambiguous legacy encoding the guess can
differ — in which case just pass `--encoding` explicitly, or run under Python 3.12
or earlier where the C detector is available. See the
[encoding docs](https://ffsubsync.readthedocs.io/en/latest/encoding.html) for the
full story.

Sync Issues
-----------
If the sync fails, the following recourses are available:
- Try to sync assuming identical video / subtitle framerates by passing
  `--no-fix-framerate`;
- Try passing `--gss` to use [golden-section search](https://en.wikipedia.org/wiki/Golden-section_search)
  to find the optimal ratio between video and subtitle framerates (by default,
  only a few common ratios are evaluated);
- Try a value of `--max-offset-seconds` greater than the default of 60, in the
  event that the subtitles are out of sync by more than 60 seconds (empirically
  unlikely in practice, but possible).
- If the subtitles start in sync but drift partway through — a commercial break
  was cut out, a scene was inserted or removed (a "director's cut"), or two discs
  were concatenated into one file — no single global offset can fix both sides.
  Try `--split-penalty`, which enables an
  [alass](https://github.com/kaegi/alass)-style piecewise alignment that lets the
  offset change across the timeline, introducing a break only where it genuinely
  improves alignment. Pass the flag with no value for a reasonable default, or a
  number (seconds of overlap; ~4–20 are typical) to set the cost of each split —
  lower splits more eagerly, higher stays closer to a single offset. See the
  [advanced options docs](https://ffsubsync.readthedocs.io/en/latest/advanced.html#mid-file-breaks-piecewise-sync)
  for the full story.
- Try `--vad=auditok` since [auditok](https://github.com/amsehili/auditok) can
  sometimes work better in the case of low-quality audio than WebRTC's VAD.
  Auditok does not specifically detect voice, but instead detects all audio;
  this property can yield suboptimal syncing behavior when a proper VAD can
  work well, but can be effective in some cases.
- Try `--vad=fused`, which combines WebRTC with the neural
  [silero](https://github.com/snakers4/silero-vad) VAD and can be more robust on
  noisy audio. The strategy can be tuned: `--vad=fused:intersection`
  (conservative -- speech only where both agree), `--vad=fused:union`
  (aggressive -- speech where either fires), or `--vad=fused:weighted` (the
  default). These options require the optional silero dependency, which itself
  requires PyTorch; install both with `pip install ffsubsync[torch]` (or just
  `pip install torch`). torch is not installed with `ffsubsync` by default.
- For a video with no subtitles to borrow from, try transcribing the audio with
  [whisper.cpp](https://github.com/ggml-org/whisper.cpp) and using the transcript
  as the reference: `--whisper-weights ~/whisper.cpp/models/ggml-base.en.bin`.
  This needs an `ffmpeg` (>= 8.0) built with `--enable-whisper`. `ffsubsync`
  expands `~` in the path, infers the language (English for `*.en.bin` models,
  else auto-detect; override with `--language`), and warns if the video already
  has embedded subtitles. Extra whisper filter options can be passed via
  `--whisper-args` (e.g. `--whisper-args queue=12`).

When syncing in bulk, a bad sync can be worse than none. Passing
`--skip-sync-on-low-quality` leaves the subtitles unmodified when the alignment
looks untrustworthy—an anti-correlated score (`--min-score`, default 0.0) or an
implausibly large offset (`--quality-max-offset-seconds`, default 30). There is
also a `--max-framerate-deviation` check (default 0.1, which permits every
framerate correction `ffsubsync` makes); tighten it only when you know the
framerate should not change.

If the sync still fails, consider trying one of the following similar tools:
- [sc0ty/subsync](https://github.com/sc0ty/subsync): does speech-to-text and looks for matching word morphemes
- [kaegi/alass](https://github.com/kaegi/alass): rust-based subtitle synchronizer with a fancy dynamic programming algorithm
- [tympanix/subsync](https://github.com/tympanix/subsync): neural net based approach that optimizes directly for alignment when performing speech detection
- [oseiskar/autosubsync](https://github.com/oseiskar/autosubsync): performs speech detection with bespoke spectrogram + logistic regression
- [pums974/srtsync](https://github.com/pums974/srtsync): similar approach to ffsubsync (WebRTC's VAD + FFT to maximize signal cross correlation)

Speed
-----
`ffsubsync` usually finishes in 20 to 30 seconds, depending on the length of
the video. The most expensive step is actually extraction of raw audio. If you
already have a correctly synchronized "reference" srt file (in which case audio
extraction can be skipped), `ffsubsync` typically runs in less than a second.

How It Works
------------
The synchronization algorithm operates in 3 steps:
1. Discretize both the video file's audio stream and the subtitles into 10ms
   windows.
2. For each 10ms window, determine whether that window contains speech.  This
   is trivial to do for subtitles (we just determine whether any subtitle is
   "on" during each time window); for the audio stream, use an off-the-shelf
   voice activity detector (VAD) like
   the one built into [webrtc](https://webrtc.org/).
3. Now we have two binary strings: one for the subtitles, and one for the
   video.  Try to align these strings by matching 0's with 0's and 1's with
   1's. We score these alignments as (# video 1's matched w/ subtitle 1's) - (#
   video 1's matched with subtitle 0's).

The best-scoring alignment from step 3 determines how to offset the subtitles
in time so that they are properly synced with the video. Because the binary
strings are fairly long (millions of digits for video longer than an hour), the
naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we
use the fact that "scoring all alignments" is a convolution operation and can
be implemented with the Fast Fourier Transform (FFT), bringing the complexity
down to O(n log n).

Limitations
-----------
In most cases, inconsistencies between video and subtitles occur when starting
or ending segments present in video are not present in subtitles, or vice versa.
This can occur, for example, when a TV episode recap in the subtitles was pruned
from video. FFsubsync typically works well in these cases, and in my experience
this covers >95% of use cases. Breaks and splits *outside* the beginning and
ending segments — a commercial break cut out of the middle, an inserted or removed
scene, or two discs concatenated into one file — are trickier, since no single
global offset can fix both sides. The experimental `--split-penalty` mode handles
many of these; see [Sync Issues](#sync-issues) above.

Future Work
-----------
Besides general stability and usability improvements, one line of work aims to
harden the [alass](https://github.com/kaegi/alass)-style `--split-penalty` mode
that handles splits / breaks in the middle of video not present in subtitles (or
vice versa). It works well in many cases today but is still considered
experimental. See [#10](https://github.com/smacke/ffsubsync/issues/10) for more
details.

History
-------
The implementation for this project was started during HackIllinois 2019, for
which it received an **_Honorable Mention_** (ranked in the top 5 projects,
excluding projects that won company-specific prizes).

Credits
-------
This project would not be possible without the following libraries:
- [ffmpeg](https://www.ffmpeg.org/) and the [ffmpeg-python](https://github.com/kkroening/ffmpeg-python) wrapper, for extracting raw audio from video
- VAD from [webrtc](https://webrtc.org/) and the [py-webrtcvad](https://github.com/wiseman/py-webrtcvad) wrapper, for speech detection
- [srt](https://pypi.org/project/srt/) for operating on [SRT files](https://en.wikipedia.org/wiki/SubRip#SubRip_text_file_format)
- [numpy](http://www.numpy.org/) and, indirectly, [FFTPACK](https://www.netlib.org/fftpack/), which powers the FFT-based algorithm for fast scoring of alignments between subtitles (or subtitles and video)
- Other excellent Python libraries like [argparse](https://docs.python.org/3/library/argparse.html), [rich](https://github.com/willmcgugan/rich), and [tqdm](https://tqdm.github.io/), not related to the core functionality, but which enable much better experiences for developers and users.

# License
Code in this project is [MIT licensed](https://opensource.org/licenses/MIT).
