Metadata-Version: 2.4
Name: evaluatio
Version: 0.5.1
Classifier: License :: OSI Approved :: MIT License
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: numpy>=1.19,<1.25 ; python_full_version < '3.9'
Requires-Dist: numpy>=1.25,<2.1 ; python_full_version >= '3.9' and python_full_version < '3.12'
Requires-Dist: numpy>=1.26,<2.1 ; python_full_version == '3.12.*'
Requires-Dist: numpy>=2.1 ; python_full_version >= '3.13'
License-File: LICENSE
Summary: High-performance NLP evaluation and significance testing library
Keywords: nlp,metrics,wer,cer,evaluation,word-error-rate,speech-recognition,rust-extensions
Author-email: Preben Vangberg <prv21fgt@bangor.ac.uk>
Maintainer-email: Preben Vangberg <prv21fgt@bangor.ac.uk>
License-Expression: MIT
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://prvinspace.github.io/evaluatio/
Project-URL: Homepage, https://codeberg.org/prvinspace/evaluatio
Project-URL: Issues, https://codeberg.org/prvinspace/evaluatio/issues
Project-URL: Repository, https://codeberg.org/prvinspace/evaluatio

# Evaluatio
![PyPI - Version](https://img.shields.io/pypi/v/evaluatio)
![License](https://img.shields.io/github/license/prvInSpace/evaluatio)
[![CI](https://github.com/prvInSpace/evaluatio/actions/workflows/CI.yml/badge.svg)](https://github.com/prvInSpace/evaluatio/actions/workflows/CI.yml)
[![codecov](https://codecov.io/github/prvInSpace/evaluatio/graph/badge.svg?token=63NBX8175Q)](https://codecov.io/github/prvInSpace/evaluatio)

***Statistically rigorous evaluation for NLP with fast metrics and built-in inference tools.***

## Why Evaluatio?
Most libraries make it easy to compute metrics.
Few make it easy to **evaluate models correctly**.

Evaluatio is designed to fix that.

It provides:
- **Fast metric implementations** (e.g. WER, CER)
- **Uncertainty estimation** (e.g. bootstrap confidence intervals)
- **Model comparison tools** (e.g. paired bootstrap tests)
- **Multiple testing correction** (e.g. Holm-Bonferroni)
- **High performance** (Rust-backed implementations)

## What's included

### Metrics
| Metric | Description |
|--------|-------------|
| WER / CER | Word and character error rate, including edit-distance variants |
| UED / UER | Type-agnostic edit distance/error rate. Works with any Python object supporting `__eq__` |
| PIER | Point of interest error rate, for evaluating punctuation, code-switching, capitalisation, etc. individually |

### Statistical inference
| Tool | Description |
|------|-------------|
| Bootstrap CI | Confidence intervals via bootstrap resampling (general, and corpus-correct variants for WER/CER) |
| Significance testing for model comparison |  Including paired bootstrap test, paired permutation test, and BLEU specific paired bootstrap test for MT. |
| Multiple testing correction | Bonferroni and Holm-Bonferroni |
| Cohen's d | Effect size estimation (independent and paired) |

Current task guides cover **ASR**. The inference tools are general-purpose and apply across tasks (especially for **machine translation**).

## Quick example
```python
import pandas as pd
from evaluatio.metrics.wer import (
    word_error_rate,
    word_error_rate_ci,
    word_error_rate_per_pair,
)
from evaluatio.inference.hypothesis import paired_bootstrap_test

df = pd.read_csv("inferences.csv")

# Corpus-level WER
wer = word_error_rate(df["references"], df["predictions"])

# Confidence interval
ci = word_error_rate_ci(df["references"], df["predictions"], 5000, 0.95)

# Model comparison
wer_a = word_error_rate_per_pair(df["references"], df["model_a"])
wer_b = word_error_rate_per_pair(df["references"], df["model_b"])

pvalue = paired_bootstrap_test(wer_a, wer_b, iterations=5000)

print(f"WER: {ci.mean:.3f} ± {ci.upper - ci.mean:.3f}")
print(f"P-value: {pvalue}")
```

This workflow reflects the recommended approach: compute a corpus-level metric, estimate uncertainty, and compare models using paired statistical tests.

## Key features

### Correct by default
- Explicit, strongly typed APIs that reduce common evaluation mistakes.
- Documentation is designed as a reference for statistically rigorous evaluation, not just API usage. 

### Built-in statistical inference
- Bootstrap confidence intervals
- Paired bootstrap significance testing
- Multiple testing correction

### High performance
- Rust-backed core for efficient computation
- Faster WER/CER than common alternatives

## Installation
```bash
pip install evaluatio
```

## Documentation
Full documentation (including evaluation guides and statistical background): [https://prvinspace.github.io/evaluatio/](https://prvinspace.github.io/evaluatio/).

## Status
ASR metrics (WER, CER) and the statistical inference tools are stable. APIs elsewhere may change.

## Contribute to the project
There is always room for improvements, new metrics, new functionality, etc. If you have any suggestions or requests please feel free to add an issue! The main repository for the project can be found at [codeberg.org/prvinspace/evaluatio](https://codeberg.org/prvinspace/evaluatio).

## Maintainer
The project is maintained by Preben Vangberg &lt;[prv21fgt@bangor.ac.uk](mailto:prv21fgt@bangor.ac.uk)&gt;.

## Etymology
The name `evaluatio` is a Latin noun and means "evaluation". It also doubles as the English verb "to evaluate" with the Welsh verbal derivational suffix `-io`, so it could also be Welsh slang for "to evaluate".

