Metadata-Version: 2.4
Name: scoreq-pytorch
Version: 0.1.0
Summary: Unofficial fairseq-free PyTorch implementation of SCOREQ
Author: Petr Grinberg
License-Expression: MIT
Project-URL: Homepage, https://github.com/Blinorot/scoreq-pytorch
Project-URL: Repository, https://github.com/Blinorot/scoreq-pytorch
Project-URL: Issues, https://github.com/Blinorot/scoreq-pytorch/issues
Project-URL: Model, https://huggingface.co/Blinorot/SCOREQ-PyTorch
Project-URL: Paper, https://arxiv.org/abs/2410.06675
Keywords: scoreq,speech-quality,audio,tts,neural-codec,pytorch
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.2.0
Requires-Dist: huggingface_hub>=0.20
Dynamic: license-file

<h1 align="center">
SCOREQ-PyTorch
</h1>

<p align="center">
  <a href="#about">About</a> •
  <a href="#model-types">Model Types</a> •
  <a href="#usage">Usage</a> •
  <a href="#how-to-reproduce">How To Reproduce</a> •
  <a href="#credits">Credits</a> •
  <a href="#license">License</a> •
  <a href="#citation">Citation</a>
</p>

<p align="center">
<a href="https://pypi.org/project/scoreq-pytorch/">
  <img src="https://img.shields.io/pypi/v/scoreq-pytorch.svg?logo=pypi&logoColor=white&cacheSeconds=300" alt="PyPI version">
</a>
<a href="https://pypi.org/project/scoreq-pytorch/">
  <img src="https://img.shields.io/pypi/pyversions/scoreq-pytorch.svg?logo=python&logoColor=white&color=blue&cacheSeconds=300" alt="Python versions">
</a>
<a href="https://huggingface.co/Blinorot/SCOREQ-PyTorch">
  <img src="https://img.shields.io/badge/HuggingFace-Model-yellow.svg?logo=huggingface&logoColor=white" alt="Hugging Face model">
</a>
<a href="https://github.com/Blinorot/scoreq-pytorch/blob/main/LICENSE">
  <img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="License: MIT">
</a>
<a href="https://arxiv.org/abs/2410.06675">
  <img src="https://img.shields.io/badge/Paper-arXiv%3A2410.06675-b31b1b.svg?logo=arxiv&logoColor=white" alt="SCOREQ paper">
</a>
</p>

## About

This is an unofficial `fairseq`-free implementation of the SCOREQ Speech Quality Assessment system proposed in [SCOREQ: Speech Quality Assessment with Contrastive Regression](https://arxiv.org/abs/2410.06675).

The [original implementation](https://github.com/alessandroragano/scoreq) provides a `fairseq`-based PyTorch model and an ONNX variant. In practice, the `fairseq` dependency can be difficult to install with recent Python, PyTorch, and dependency versions. The ONNX variant avoids `fairseq`, but it can be less convenient for PyTorch-based research workflows and may be difficult to run with GPU acceleration on `ARM/aarch64` systems.

[Recent study from ICASSP 2026](https://arxiv.org/abs/2509.24457) highlights the high correlation of SCOREQ with subjective listening scores for neural codecs. Therefore, modern neural audio codec and TTS research benefits from an easy-to-install SCOREQ implementation.

We provide a `fairseq`-free implementation written directly in `PyTorch` that matches the [original system](https://github.com/alessandroragano/scoreq) using converted weights and reimplemented modules.

We also provide a `TorchScript` variant that can be loaded with only PyTorch, without installing this package.

The PyTorch and TorchScript versions are validated against the original implementation and produce matching scores.

> [!NOTE]
> In contrast to the original implementation, we support batched audio assessment. However, we recommend running SCOREQ with **batch size 1** to avoid metric shifts caused by padding. Batching can be used for faster evaluation when small padding-related score differences are acceptable.

## Model Types

As in the [original system](https://github.com/alessandroragano/scoreq), we support 4 types of SCOREQ, i.e., 2 audio domains and 2 modes.

Data domain (what kind of audio is evaluated):

- `natural`: used for audio that was created from a genuine human speech (Audio Codecs, VoIP, Telephony, Speech Enhancement, Audio Restoration).
- `synthetic`: used for audio that was synthesized by a machine (Text-to-Speech (TTS), Voice Conversion (VC), Generative Speech Models).

Mode (whether there is a reference audio to compare with):

- `nr`: no-reference mode. Assesses the quality of audio, **the higher the better**, without relying on any reference.
- `ref`: reference mode. Calculate the distance between provided and reference audio embeddings, **the lower the better**.

We refer the user to the [original repository](https://github.com/alessandroragano/scoreq) and [paper](https://arxiv.org/abs/2410.06675) for more details on model types.

## Usage

You can install the repo as a package:

```bash
pip install scoreq-pytorch
```

Or from source:

```bash
git clone https://github.com/Blinorot/scoreq-pytorch.git
cd scoreq-pytorch
pip install -e .
```

The code requires:

| Package         | Version |
| --------------- | ------- |
| Python          | >=3.9   |
| PyTorch         | >=2.2.0 |
| HuggingFace Hub | >=0.20  |

The TorchScript checkpoint was scripted with `PyTorch 2.5.1`. We have tested that it works on `PyTorch 2.2.0`, however, `PyTorch >=2.5.1` is recommended for the
TorchScript variant.

Then, you can run the model as follows:

```python
import torchaudio
from scoreq_pytorch import SCOREQScoreTorch

device = "cpu" # set to "cuda" to use on GPU
data_domain = "natural" # or "synthetic"
mode = "nr" # or "ref"
scoreq = SCOREQScoreTorch(
  data_domain=data_domain,
  mode=mode,
  device=device
) # already in eval mode

# load an audio file, e.g. using torchaudio
test_audio_path = ... # path to an audio file
test_wav, sr = torchaudio.load(test_audio_path)

# convert to MONO 16 kHz
TARGET_SR = 16000
if test_wav.shape[0] != 1:
    test_wav = test_wav[0:1]
if sr != TARGET_SR:
    test_wav = torchaudio.functional.resample(test_wav, orig_freq=sr, new_freq=TARGET_SR)
# put on device
test_wav = test_wav.to(device)

# for "ref" mode, you need a reference audio
# same loading and pre-processing procedure
if mode == "ref":
    ref_wav = ...
else:
    ref_wav = None

# calculate the score
# accepts T, 1xT, Bx1xT
scoreq_score = scoreq.score(test_wav, ref_wav) # tensor of shape (batch_size,)
```

You can replace `SCOREQScoreTorch` with `SCOREQScoreScripted` to use the `TorchScript` variant instead. On first use, the package downloads converted SCOREQ weights from [Hugging Face Hub](https://huggingface.co/Blinorot/SCOREQ-PyTorch) and caches them locally using the Hugging Face cache.

For `TorchScript`, you can avoid downloading the package and use the model directly:

```python
import torch
import torchaudio
import wget

data_domain = "natural" # or "synthetic"
mode = "nr" # or "ref"

# download scripted checkpoint, e.g. using wget
checkpoint_url = f"https://huggingface.co/Blinorot/SCOREQ-PyTorch/resolve/main/scoreq_{data_domain}_{mode}_scripted.pt"
checkpoint_path = ... # path to saved checkpoint
wget.download(checkpoint_url, checkpoint_path)

# load directly with torch.jit
device = "cpu" # set to "cuda" to use on GPU
scoreq = torch.jit.load(checkpoint_path, map_location=device)
scoreq.eval()

# load an audio file, e.g. using torchaudio
test_audio_path = ... # path to an audio file
test_wav, sr = torchaudio.load(test_audio_path)

# convert to MONO 16 kHz
TARGET_SR = 16000
if test_wav.shape[0] != 1:
    test_wav = test_wav[0:1]
if sr != TARGET_SR:
    test_wav = torchaudio.functional.resample(test_wav, orig_freq=sr, new_freq=TARGET_SR)
# put on device
test_wav = test_wav.to(device)

# for "ref" mode, you need a reference audio
# same loading and pre-processing procedure
if mode == "ref":
    ref_wav = ...
else:
    ref_wav = None

# calculate the score
# accepts T, 1xT, Bx1xT
with torch.no_grad():
    scoreq_score = scoreq(test_wav, ref_wav) # tensor of shape (batch_size,)
```

### Notes

The model expects audio sampled at **16 kHz**.

Accepted tensor shapes:

| Shape       | Meaning                                          |
| ----------- | ------------------------------------------------ |
| `(T,)`      | single mono test_waveform                        |
| `(1, T)`    | single mono test_waveform with channel dimension |
| `(B, 1, T)` | batch of mono test_waveforms                     |

The input should be a floating point PyTorch tensor. Stereo audio should be converted to mono before scoring. `scoreq.score(test_wav)` returns a tensor of shape `(batch_size,)`, where each value is a predicted quality score.

For reference `ref` mode, a reference audio `ref_wav` must be provided: `scoreq.score(test_wav, ref_wav)`.

Note that `score()` and `forward()` return the same values. The only difference is that `score()` is decorated with `torch.no_grad()` for convenient inference. Since the raw TorchScript module exposes `forward()`, it is called directly as `scoreq(test_wav, ref_wav)` rather than through the package wrapper's `scoreq.score(test_wav, ref_wav)`.

**Batch size 1 is recommended to avoid padding-related score shifts.**

API classes:

| Class                 | Description                                     |
| --------------------- | ----------------------------------------------- |
| `SCOREQScoreTorch`    | PyTorch implementation using converted weights. |
| `SCOREQScoreScripted` | Wrapper around the TorchScript checkpoint.      |

## How To Reproduce

To reproduce PyTorch and Scripted checkpoints and validate them against the original SCOREQ module, follow the steps below.

First, install all required packages in a new environment:

```bash
# Optional
conda create -n scoreq python=3.9.7
conda activate scoreq

pip install pip==22.0
pip install -r requirements.txt
```

Then, you need to export weights from the original SCOREQ checkpoint:

```bash
# add --private to save privately
python extract_state_dict.py --repo-id USERNAME/REPO_NAME_ON_HUGGINGFACE
```

This will upload the state dict extracted from the original SCOREQ checkpoint (_for each data domain and mode pair_) to Hugging Face. The same state dict is used to load our `fairseq`-free PyTorch-only module.

To create a scripted version of the PyTorch model that allows to load SCOREQ without class definitions, run

```bash
# add --private to save privately
python create_scripted_model.py --repo-id USERNAME/REPO_NAME_ON_HUGGINGFACE
```

It will upload the scripted model to HuggingFace as well.

Finally, to test that all 3 variations (Original, PyTorch, Scripted) return the same scores, run

```bash
# set --device "cpu" to run on cpu
# set --batch-size to a value bigger than 1 to test batched version
python test.py --device "cuda" --batch-size 1
```

The models are tested on `test-clean` partition of [LibriSpeech](https://www.openslr.org/12) against [DAC](https://arxiv.org/abs/2306.06546)-regenerated codec audio for `natural` mode and [MMS-TTS](https://arxiv.org/abs/2305.13516)-synthesized TTS audio for `synthetic` mode.

| Data Domain | Mode  | SCOREQ Version | Score (LibriSpeech Test-Clean) |
| ----------- | ----- | -------------- | -----------------------------: |
| `natural`   | `nr`  | Original       |              4.184167324040683 |
| `natural`   | `nr`  | Torch          |              4.184167324040683 |
| `natural`   | `nr`  | Scripted       |              4.184167324040683 |
| `natural`   | `ref` | Original       |            0.10563717598792251 |
| `natural`   | `ref` | Torch          |            0.10563717685383922 |
| `natural`   | `ref` | Scripted       |            0.10563717685383922 |
| `synthetic` | `nr`  | Original       |              4.465621592707306 |
| `synthetic` | `nr`  | Torch          |              4.465621592707306 |
| `synthetic` | `nr`  | Scripted       |              4.465621592707306 |
| `synthetic` | `ref` | Original       |             0.5238628057009391 |
| `synthetic` | `ref` | Torch          |             0.5238628115362793 |
| `synthetic` | `ref` | Scripted       |             0.5238628115362793 |

## Credits

The code is based on the original [SCOREQ](https://github.com/alessandroragano/scoreq) and [fairseq](https://github.com/facebookresearch/fairseq) repositories.

## License

This project is released under the [MIT License](./LICENSE).

Parts of the implementation are adapted from the original SCOREQ and fairseq repositories, which are also MIT licensed. See [LICENSES](./LICENSES) for third-party license texts.

Converted checkpoints are derived from the original SCOREQ checkpoint. Original authors retain copyright over the original model and weights.

## Citation

If you use this package, please cite the original SCOREQ paper:

```bibtex
@article{ragano2024scoreq,
  title={SCOREQ: Speech quality assessment with contrastive regression},
  author={Ragano, Alessandro and Skoglund, Jan and Hines, Andrew},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={105702--105729},
  year={2024}
}
```
