Metadata-Version: 2.4
Name: OpenSTBench
Version: 1.3.0
Summary: Unified multidimensional evaluation toolkit for S2TT and S2ST systems in offline and streaming speech translation settings
Author-email: Yanjie An <691476922@qq.com>
License: MIT
Project-URL: Homepage, https://github.com/sjtuayj/OpenSTBench
Project-URL: Repository, https://github.com/sjtuayj/OpenSTBench
Project-URL: Documentation, https://github.com/sjtuayj/OpenSTBench#readme
Project-URL: Bug Tracker, https://github.com/sjtuayj/OpenSTBench/issues
Keywords: translation,evaluation,BLEU,COMET,BLEURT,MetricX,speech-translation,S2TT,S2ST,streaming,latency,temporal-consistency,UTMOS,WER,speaker-preservation,emotion,acoustic-events,BEATs,audio-events
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Python: <3.11,>=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: setuptools==80.9.0
Requires-Dist: torch<2.7,>=2.6
Requires-Dist: torchaudio<2.7,>=2.6
Requires-Dist: numpy
Requires-Dist: sacrebleu>=2.0.0
Requires-Dist: pandas
Requires-Dist: tqdm
Requires-Dist: scipy
Requires-Dist: soundfile
Requires-Dist: jiwer
Requires-Dist: textgrid
Requires-Dist: transformers
Requires-Dist: matplotlib
Requires-Dist: resemblyzer
Provides-Extra: comet
Requires-Dist: unbabel-comet>=2.2.2; extra == "comet"
Provides-Extra: whisper
Requires-Dist: openai-whisper; extra == "whisper"
Provides-Extra: tokenizer-ja
Requires-Dist: sacrebleu[ja]>=2.0.0; extra == "tokenizer-ja"
Provides-Extra: tokenizer-ko
Requires-Dist: sacrebleu[ko]>=2.0.0; extra == "tokenizer-ko"
Provides-Extra: speech-quality
Requires-Dist: jiwer; extra == "speech-quality"
Requires-Dist: opencc-python-reimplemented; extra == "speech-quality"
Provides-Extra: emotion
Requires-Dist: funasr; extra == "emotion"
Requires-Dist: modelscope; extra == "emotion"
Provides-Extra: acoustic-events
Requires-Dist: sed-scores-eval==0.0.4; extra == "acoustic-events"
Provides-Extra: metricx
Requires-Dist: transformers[torch]==4.30.2; extra == "metricx"
Requires-Dist: sentencepiece==0.1.99; extra == "metricx"
Requires-Dist: datasets==2.13.1; extra == "metricx"
Requires-Dist: protobuf==3.20.3; extra == "metricx"
Requires-Dist: accelerate>=0.26.0; extra == "metricx"
Provides-Extra: all
Requires-Dist: unbabel-comet>=2.2.2; extra == "all"
Requires-Dist: openai-whisper; extra == "all"
Requires-Dist: sacrebleu[ja,ko]>=2.0.0; extra == "all"
Requires-Dist: jiwer; extra == "all"
Requires-Dist: opencc-python-reimplemented; extra == "all"
Requires-Dist: funasr; extra == "all"
Requires-Dist: modelscope; extra == "all"
Requires-Dist: sed-scores-eval==0.0.4; extra == "all"
Requires-Dist: transformers[torch]==4.30.2; extra == "all"
Requires-Dist: sentencepiece==0.1.99; extra == "all"
Requires-Dist: datasets==2.13.1; extra == "all"
Requires-Dist: protobuf==3.20.3; extra == "all"
Requires-Dist: accelerate>=0.26.0; extra == "all"
Dynamic: license-file

# OpenSTBench

English | [中文](./README_zh.md)

[![arXiv](https://img.shields.io/badge/arXiv-2605.30792-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2605.30792)
[![PyPI](https://img.shields.io/pypi/v/OpenSTBench?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/OpenSTBench/)
[![Python](https://img.shields.io/badge/Python-3.9--3.10-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](LICENSE)
[![GitHub](https://img.shields.io/badge/GitHub-OpenSTBench-181717?style=for-the-badge&logo=github)](https://github.com/sjtuayj/OpenSTBench)
[![X-LANCE](https://img.shields.io/badge/X--LANCE-grey?labelColor=lightgrey&logo=leanpub&style=for-the-badge)](https://x-lance.sjtu.edu.cn/)

OpenSTBench is a multidimensional evaluation toolkit for speech translation. It is designed for heterogeneous systems, including speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline systems, and streaming systems.

The toolkit organizes evaluation into three dimensions:

- **Translation Quality**: whether the translated text preserves the source meaning.
- **Speech Quality**: whether generated speech is natural, text-consistent, speaker-preserving, emotion-preserving, and faithful to acoustic events.
- **Temporal Quality**: whether generated speech preserves duration structure and, for streaming systems, whether output is responsive.

## Installation

```bash
pip install OpenSTBench
```

For local development:

```bash
git clone https://github.com/sjtuayj/OpenSTBench.git
cd OpenSTBench
conda create -n openstbench python=3.10 -y
conda activate openstbench
pip install -e .
```

Optional extras:

```bash
pip install "OpenSTBench[comet]"
pip install "OpenSTBench[whisper]"
pip install "OpenSTBench[tokenizer-ja]"  # Japanese BLEU
pip install "OpenSTBench[tokenizer-ko]"  # Korean BLEU
pip install "OpenSTBench[speech_quality]"
pip install "OpenSTBench[emotion]"
pip install "OpenSTBench[acoustic-events]"
pip install "OpenSTBench[metricx]"
pip install "OpenSTBench[all]"
```

MetricX follows the official `google-research/metricx` runtime requirements. Installing
`OpenSTBench[metricx]` or `OpenSTBench[all]` pins the MetricX-compatible stack,
including `transformers[torch]==4.30.2`, `sentencepiece==0.1.99`,
`datasets==2.13.1`, `protobuf==3.20.3`, and `accelerate>=0.26.0`.

BLEURT is installed separately:

```bash
pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
```

## Package Names

- PyPI package: `OpenSTBench`
- Python import: `openstbench`

## Evaluation Dimensions

| Dimension | Evaluator | System type | Main outputs |
| :--- | :--- | :--- | :--- |
| Translation Quality | `TranslationEvaluator` | S2TT, S2ST text or generated speech | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT`, `MetricX`, `MetricX_QE`, and their `ASR_` variants |
| Speech Quality | `SpeechQualityEvaluator` | S2ST | `UTMOS`, `WER_Consistency`, `CER_Consistency` |
| Speech Quality | `SpeakerSimilarityEvaluator` | S2ST | `average_wavlm_large_similarity`, `average_resemblyzer_similarity` |
| Speech Quality | `EmotionEvaluator` | S2ST | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
| Speech Quality | `AcousticEventEvaluator` / `BEATsStrongEventDetector` | S2ST | `event_content_map` |
| Temporal Quality | `TemporalConsistencyEvaluator` | S2ST | `Duration_Consistency_SLC_0.2`, `Duration_Consistency_SLC_0.4` |
| Temporal Quality | `LatencyEvaluator` | Streaming S2TT/S2ST | `First_Audio_Delay_(StartOffset_ms)`, `Overall_Translation_Delay_(ATD_ms)`, `End_Action_Delay_(CustomATD_ms)`, `Real_Time_Factor_(RTF)` |

Offline and streaming are supported system settings, not separate metric dimensions. Use the evaluators that match the available outputs: text, generated speech, source/target audio pairs, event annotations, or streaming traces.

### Acoustic-event evaluation

Acoustic-event evaluation measures whether the target audio preserves the annotated event categories from the source audio. It does not evaluate when an event occurs. OpenSTBench runs the published `BEATs_strong_1` checkpoint without training or fine-tuning and computes threshold-free, clip-based macro Average Precision (`event_content_map`). Classes enter the macro average only when the evaluated cohort contains both positive and negative clips.

The public `AcousticEventEvaluator` and `compute_event_content_map` APIs return a Content mAP result directly, including the overall `event_content_map`, per-class AP, positive/negative counts, and class coverage. See [`examples/python/acoustic_event_eval.py`](examples/python/acoustic_event_eval.py) for a runnable example.

The `newtest` workflow writes `metrics_acoustic_events.json` with two metric views and separate coverage:

- `metrics.successful_only.event_content_map` is the official score over successfully generated target audio.
- `metrics.all_samples_zero_scored.event_content_map` is diagnostic only; failed target audio receives zero confidence for every event class before mAP is recomputed on the complete cohort.
- `coverage` reports the success rate, failure rate, and failure reasons.

Omit `--beats_model_path` in `newtest/eval/eval_acoustic_events.py` to download the fixed checkpoint into the OpenSTBench user cache, or provide an explicit checkpoint path. There is no training command or manual detection threshold.

## Experimental Overview

The radar plot below illustrates the multidimensional view produced by OpenSTBench for representative streaming and offline speech translation systems. It summarizes how systems can differ across translation quality, speech quality, and temporal quality: a system with strong translation quality may still show different behavior in speech realization, speaker or emotion preservation, acoustic-event fidelity, temporal consistency, and latency or efficiency.

![OpenSTBench experimental radar overview](./radar.png)

## Datasets

The paper uses the following datasets. Please follow the license and access terms of each original dataset.

| Dataset | Used for | Link |
| :--- | :--- | :--- |
| MSLT dev | Translation quality, speech quality, temporal consistency, latency | [Microsoft Speech Language Translation Corpus](https://www.microsoft.com/en-us/download/details.aspx?id=54689) |
| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set is available on [Hugging Face Datasets](https://huggingface.co/datasets/ayj111/openstbench-paired-set); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
| RAVDESS | Emotion preservation | [Audio_Speech_Actors_01-24.zip](https://zenodo.org/records/1188976) from the RAVDESS Zenodo record |
| MCAE-SPPS | Emotion preservation | [MCAE-SPPS on OSF](https://doi.org/10.17605/OSF.IO/9JYZC) |
| NonverbalTTS test | Acoustic-event content | [deepvk/NonverbalTTS](https://huggingface.co/datasets/deepvk/NonverbalTTS) |
| SynParaSpeech | Acoustic-event content | [shawnpi/SynParaSpeech](https://huggingface.co/datasets/shawnpi/SynParaSpeech) |

## Quick Start

```python
from openstbench import TranslationEvaluator

evaluator = TranslationEvaluator(
    use_bleu=True,
    use_chrf=True,
    use_comet=False,
    use_bleurt=False,
    use_metricx=True,
    device="cuda",
)

scores = evaluator.evaluate_all(
    reference=["我喜欢看电影。", "今天天气很好。"],
    target_text=["我喜欢看电影。", "今天天气很好。"],
    source=["I like watching movies.", "The weather is nice today."],
    target_lang="zh",
    asr_text=["我喜欢看电影。", "今天天气很好。"],
)

print(scores)
```

ASR translation-quality evaluation is enabled by default and runs only when
`target_audio` or precomputed `asr_text` is supplied. To transcribe generated
speech directly, pass a file, a list of files, or a directory as
`target_audio`; Whisper defaults to `medium` and can be changed with
`TranslationEvaluator(whisper_model="large-v3")`. Set `use_asr=False` to
disable every `ASR_` metric.

Route languages to different ASR models when needed:

```python
from openstbench import ASRRouter, TranslationEvaluator, WhisperASRBackend

router = ASRRouter({
    "default": WhisperASRBackend(model="medium"),
    "ja": WhisperASRBackend(model="large-v3"),
})
evaluator = TranslationEvaluator(asr_router=router)
```

Keep Cantonese as the independent route `"yue"`; if the selected Whisper
checkpoint does not support it, configure a Cantonese `ASRBackend` or provide
precomputed `asr_text`. Dzongkha (`"dz"`) likewise uses character BLEU/CER but
requires a custom ASR backend or `asr_text` with the default Whisper backend.
For language-aware latency tokenization, use
`--target-language ja --latency-unit auto`; omitting `auto` preserves the
existing explicit unit behavior.

### Multilingual quick start

`target_lang` is the language switch for translation quality and speech
consistency. Install the optional tokenizer before evaluating Japanese or
Korean BLEU:

```bash
pip install "OpenSTBench[tokenizer-ja]"  # Japanese
pip install "OpenSTBench[tokenizer-ko]"  # Korean
```

```python
scores = evaluator.evaluate_all(
    reference=["今日は天気がとても良いです。"],
    target_text=["今日は天気がとても良いです。"],
    asr_text=["今日は天気がとても良いです。"],
    target_lang="ja",
)
```

For streaming latency, pass the target language and let `auto` choose the
unit:

```bash
python -m openstbench.latency.cli \
  --source data/source.txt --target data/ref.txt \
  --task s2t --agent-script my_agent.py --agent-class MyAgent \
  --target-language ja --latency-unit auto
```

See [translation_eval.py](examples/python/translation_eval.py),
[speech_quality_eval.py](examples/python/speech_quality_eval.py), and
[latency_eval.py](examples/python/latency_eval.py) for complete parameter
templates. Cantonese and other languages unsupported by a selected Whisper
checkpoint require a custom `ASRBackend` or precomputed `asr_text`.

## Supported languages

OpenSTBench's built-in language policy is defined in
`src/openstbench/language_policy.py`. `target_lang` accepts normalized
ISO-style language codes and common aliases, including `eng -> en`,
`cmn/zho/chi -> zh`, `jpn -> ja`, and `kor -> ko`.

OpenSTBench does not reject language codes that are not listed below. For
unlisted languages, translation metrics use the default space-delimited policy:
SacreBLEU `13a`, `WER_Consistency`, and word-level latency tokenization. ASR
availability is backend-dependent: use a compatible `WhisperASRBackend`, a
custom `ASRBackend`, or precomputed `asr_text`.

| Language group | `target_lang` | BLEU tokenizer | Consistency metric | `latency_unit="auto"` | Notes |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Chinese | `zh` | `zh` | `CER_Consistency` | `char` | Aliases include `cmn`, `zho`, `chi` |
| Cantonese | `yue` | `zh` | `CER_Consistency` | `char` | Keep as a separate ASR route; may require a custom backend or `asr_text` |
| Japanese | `ja` | `ja-mecab` | `CER_Consistency` | `char` | Install `OpenSTBench[tokenizer-ja]` for BLEU |
| Korean | `ko` | `ko-mecab` | `CER_Consistency` | `char` | Install `OpenSTBench[tokenizer-ko]` for BLEU |
| Thai, Lao, Khmer, Burmese, Tibetan, Dzongkha | `th`, `lo`, `km`, `my`, `bo`, `dz` | `char` | `CER_Consistency` | `char` | ASR support depends on the configured backend |
| Space-delimited and default languages | `en`, `fr`, `de`, `es`, ... | `13a` | `WER_Consistency` | `word` | Unknown or unlisted language codes fall back to this policy |

## Examples

Complete parameter templates are kept in `examples/`. The README intentionally stays compact; use these files for configurable parameters, input formats, and output fields.

- `examples/python/translation_eval.py`
- `examples/python/speech_quality_eval.py`
- `examples/python/speaker_similarity_eval.py`
- `examples/python/emotion_eval.py`
- `examples/python/acoustic_event_eval.py`
- `examples/python/temporal_consistency_eval.py`
- `examples/python/latency_eval.py`
- `examples/bash/install_extras.sh`
- `examples/bash/run_latency_cli.sh`

Latency can also be run from the module CLI:

```bash
python -m openstbench.latency.cli --help
```

## Conventions

- Text inputs generally accept `list[str]`, one-sample-per-line `.txt` files, and `.json` files where supported by the evaluator.
- Audio inputs generally accept folders, `list[str]`, `.txt` path lists, and `.json` path lists where supported by the evaluator.
- Built-in language tokenization, speech-consistency, and `latency_unit="auto"` behavior are summarized in [Supported languages](#supported-languages).
- `ASRRouter` can route target languages to different `WhisperASRBackend` checkpoints or custom `ASRBackend` implementations. If a Whisper checkpoint does not expose the requested language, ASR returns an unavailable/empty transcript instead of passing an invalid language token to the model.
- Latency tokenization accepts `unit="auto"` (CLI: `--latency-unit auto`) together with a target language. Existing explicit `word` and `char` behavior is unchanged.
- Evaluators that accept pretrained model sources use a local-first rule. If the supplied local path exists, OpenSTBench uses it; otherwise it falls back to the configured remote model id.
- Optional dependencies are loaded only when the corresponding evaluator needs them.
- MetricX is enabled by default in `TranslationEvaluator`. It follows the official google-research/metricx README, uses text only, reports error scores in `[0, 25]` where lower is better, and can be disabled with `use_metricx=False`.
- `TranslationEvaluator` also enables ASR variants by default. With `target_audio`, Whisper transcribes the generated speech before the same enabled translation metrics are applied; with `asr_text`, the supplied transcript is used directly. Calls without either input preserve the text-only behavior.


## Acknowledgements

- We especially thank [SimulEval](https://github.com/facebookresearch/SimulEval), from which parts of OpenSTBench's latency evaluation components are adapted
- [sacreBLEU](https://github.com/mjpost/sacrebleu), [COMET](https://github.com/Unbabel/COMET), [MetricX](https://github.com/google-research/metricx), and [bleurt-pytorch](https://github.com/lucadiliello/bleurt-pytorch), a PyTorch port of [BLEURT](https://github.com/google-research/bleurt), for translation quality evaluation
- [Whisper](https://github.com/openai/whisper), [SpeechMOS/UTMOS](https://github.com/tarepan/SpeechMOS), [Resemblyzer](https://github.com/resemble-ai/Resemblyzer), and [WavLM](https://github.com/microsoft/unilm/tree/master/wavlm) for speech quality and speaker similarity evaluation
- [FunASR](https://github.com/modelscope/FunASR) and [Emotion2Vec](https://modelscope.cn/models/iic/emotion2vec_plus_large) for emotion preservation evaluation
- [PretrainedSED](https://github.com/fschmid56/PretrainedSED), BEATs, and `sed-scores-eval` for acoustic-event evaluation

## Citation
If you find our work useful, please cite as：

```bibtex
@misc{an2026openstbenchsemanticevaluationspeech,
      title={OpenSTBench: Beyond Semantic Evaluation for Speech Translation}, 
      author={Yanjie An and Yuxiang Zhao and Yichi Zhang and Qixi Zheng and Yujie Tu and Keqi Deng and Kai Yu and Xie Chen},
      year={2026},
      eprint={2605.30792},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2605.30792}, 
}
```

## License

OpenSTBench's original code is released under the MIT License. See [LICENSE](LICENSE).

Some latency evaluation components include code adapted from [SimulEval](https://github.com/facebookresearch/SimulEval), which is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). Those adapted portions are distributed under CC BY-SA 4.0. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for details.

The datasets referenced by OpenSTBench, including the datasets used in the paper, are not covered by the OpenSTBench code license. They are provided by their original authors or distributors under their own licenses and terms of use. Some datasets are restricted to research or non-commercial use.
