Metadata-Version: 2.4
Name: simpleder
Version: 0.0.6
Summary: A lightweight library to compute Diarization Error Rate (DER).
Home-page: https://github.com/wq2012/SimpleDER
Author: Quan Wang
Author-email: quanw@google.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: summary

# SimpleDER ![Python package](https://github.com/wq2012/SimpleDER/workflows/Python%20package/badge.svg) [![PyPI Version](https://img.shields.io/pypi/v/simpleder.svg)](https://pypi.python.org/pypi/simpleder) [![Python Versions](https://img.shields.io/pypi/pyversions/simpleder.svg)](https://pypi.org/project/simpleder) [![Downloads](https://static.pepy.tech/badge/simpleder)](https://pepy.tech/project/simpleder) [![codecov](https://codecov.io/gh/wq2012/SimpleDER/branch/master/graph/badge.svg)](https://codecov.io/gh/wq2012/SimpleDER) [![Documentation](https://img.shields.io/badge/api-documentation-blue.svg)](https://wq2012.github.io/SimpleDER)

## Table of Contents

* [Overview](#overview)
* [Diarization Error Rate](#diarization-error-rate)
* [Jaccard Error Rate](#jaccard-error-rate)
* [Implementation](#implementation)
* [Tutorial](#tutorial)
* [Citation](#citation)

## Overview

This is a single-file lightweight library to compute speaker diarization evaluation metrics:

* **Diarization Error Rate (DER)** with full component breakdown (`Miss`, `False Alarm`, `Confusion`) and optimal speaker mapping
* **Jaccard Error Rate (JER)** with per-speaker breakdown and optimal speaker mapping
* **Overlapped speech** support, plus optional `collar` tolerance and `skip_overlap` filtering
* **Corpus-level evaluation** (`corpus_der` and `corpus_jer`) across multiple recordings

To learn more about speaker diarization, here is a curated list of resources:
[awesome-diarization](https://github.com/wq2012/awesome-diarization).

## Diarization Error Rate

Diarization Error Rate (DER) is the most commonly used metric for
[speaker diarization](https://en.wikipedia.org/wiki/Speaker_diarisation):

```
       False Alarm + Miss + Confusion
DER = --------------------------------
              Reference Length
```

The definition of each term (all measured in *seconds*, accounting for overlapped speech):

* `Reference Length`: Total speaker-time duration in the reference (ground truth), $\int N_{\text{ref}}(t) \, dt$.
* `False Alarm`: Duration of hypothesis speakers in excess of reference speakers, $\int \max(0, N_{\text{hyp}}(t) - N_{\text{ref}}(t)) \, dt$.
* `Miss`: Duration of reference speakers missed by the hypothesis, $\int \max(0, N_{\text{ref}}(t) - N_{\text{hyp}}(t)) \, dt$.
* `Confusion`: Duration of active speakers assigned to the wrong speaker after applying an optimal 1-to-1 speaker mapping.

Note that DER can be larger than `1.0`.

References:

* [pyannote-metrics documentation](https://pyannote.github.io/pyannote-metrics/reference.html)
* [Xavier Anguera's thesis](http://www.xavieranguera.com/phdthesis/node108.html)

## Jaccard Error Rate

Jaccard Error Rate (JER), introduced for the DIHARD II challenge, evaluates diarization with equal weight for each reference speaker. Under an optimal 1-to-1 mapping between reference speakers $r \in S_{\text{ref}}$ and hypothesis speakers $h \in S_{\text{hyp}}$:

$$\text{JER} = \frac{1}{|S_{\text{ref}}|} \sum_{r \in S_{\text{ref}}} \left(1 - \frac{\text{duration}(r \cap h)}{\text{duration}(r \cup h)}\right)$$

Unmapped reference speakers receive a speaker error of `1.0`. Unlike DER, JER is always bounded in `[0.0, 1.0]`.

## Implementation

1. **Per-Speaker Canonicalization**: Overlapping or contiguous segments belonging to the same speaker are merged so each speaker's timeline consists of disjoint intervals.
2. **Pre-processing (`collar` and `skip_overlap`)**:
   * If `collar > 0`: For every reference boundary $t$, the interval $[t - \text{collar}, t + \text{collar}]$ (NIST `md-eval` convention) is excluded from both reference and hypothesis.
   * If `skip_overlap=True`: Regions where $N_{\text{ref}}(t) > 1$ are excluded from both reference and hypothesis.
3. **Optimal Mapping**: We solve the linear sum assignment problem via the Hungarian algorithm (`scipy.optimize.linear_sum_assignment`)—maximizing total overlap duration $\text{Match}$ for DER, or total Jaccard similarity for JER.
4. **Error & Load Calculation**: Let $\text{Load} = \int \max(N_{\text{ref}}(t), N_{\text{hyp}}(t)) \, dt$. Then:
   $$\text{Miss} + \text{False Alarm} + \text{Confusion} = \text{Load} - \text{Match}$$
   $$\text{DER} = \frac{\text{Load} - \text{Match}}{\text{Reference Length}}$$

## Tutorial

### Install

```bash
pip3 install simpleder
```

### API

#### Basic DER and JER

```python
import simpleder

# reference (ground truth)
ref = [("A", 0.0, 1.0),
       ("B", 1.0, 1.5),
       ("A", 1.6, 2.1)]

# hypothesis (diarization result from your algorithm)
hyp = [("1", 0.0, 0.8),
       ("2", 0.8, 1.4),
       ("3", 1.5, 1.8),
       ("1", 1.8, 2.0)]

der = simpleder.DER(ref, hyp)
jer = simpleder.JER(ref, hyp)
print(f"DER={der:.3f}, JER={jer:.3f}")
# DER=0.350, JER=0.381
```

#### Options: `collar`, `skip_overlap`, and `detailed` Breakdown

```python
details = simpleder.DER(ref, hyp, collar=0.0, skip_overlap=False, detailed=True)
# {
#   'der': 0.35,
#   'miss': 0.2,
#   'false_alarm': 0.1,
#   'confusion': 0.4,
#   'ref_length': 2.0,
#   'speaker_mapping': {'A': '1', 'B': '2'}
# }
```

#### Corpus-Level Evaluation (Multiple Recordings)

`corpus_der` computes the micro-averaged DER across recordings, and `corpus_jer` averages per-speaker JER across all reference speakers in the corpus (accepting either `dict` or `list` inputs):

```python
refs = {"rec1": ref, "rec2": [("A", 0.0, 2.0)]}
hyps = {"rec1": hyp, "rec2": [("1", 0.0, 2.0)]}

total_der = simpleder.corpus_der(refs, hyps)
total_jer = simpleder.corpus_jer(refs, hyps)
```

## Citation

We developed this package as part of the following work:

```
@inproceedings{wang2018speaker,
  title={{Speaker Diarization with LSTM}},
  author={Wang, Quan and Downey, Carlton and Wan, Li and Mansfield, Philip Andrew and Moreno, Ignacio Lopz},
  booktitle={2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={5239--5243},
  year={2018},
  organization={IEEE}
}

@inproceedings{xia2022turn,
  title={{Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection}},
  author={Wei Xia and Han Lu and Quan Wang and Anshuman Tripathi and Yiling Huang and Ignacio Lopez Moreno and Hasim Sak},
  booktitle={2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={8077--8081},
  year={2022},
  organization={IEEE}
}

@inproceedings{huang24d_interspeech,
  title={{On the Success and Limitations of Auxiliary Network Based Word-Level End-to-End Neural Speaker Diarization}},
  author={Yiling Huang and Weiran Wang and Guanlong Zhao and Hank Liao and Wei Xia and Quan Wang},
  year={2024},
  booktitle={Interspeech 2024},
  pages={32--36},
}
```
