Metadata-Version: 2.4
Name: pear-mt
Version: 1.0.1
Summary: Inference package for PEAR machine translation metrics.
Author: Lorenzo Proietti
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/prosho-97/pear
Project-URL: Repository, https://github.com/prosho-97/pear
Project-URL: Issues, https://github.com/prosho-97/pear/issues
Project-URL: Paper, https://aclanthology.org/2026.acl-long.1953/
Project-URL: Models, https://huggingface.co/collections/Prosho/pear
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <3.14,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: huggingface-hub>=0.23
Requires-Dist: numpy>=1.26
Requires-Dist: pytorch-lightning>=2.4
Requires-Dist: PyYAML>=6.0
Requires-Dist: torch>=2.5
Requires-Dist: transformers<6,>=4.40
Provides-Extra: dev
Requires-Dist: black>=24.0; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: twine>=5.1; extra == "dev"
Dynamic: license-file

<div align="center">

# 🍐 PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation

[![ACL 2026](https://img.shields.io/badge/ACL-2026-2f6fdd)](https://2026.aclweb.org/)
[![ACL Anthology](https://img.shields.io/badge/ACL%20Anthology-paper-b31b1b)](https://aclanthology.org/2026.acl-long.1953/)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-ffcc4d)](https://huggingface.co/collections/Prosho/pear)
[![PyPI](https://img.shields.io/pypi/v/pear-mt?cacheSeconds=300)](https://pypi.org/project/pear-mt/)
[![Python](https://img.shields.io/pypi/pyversions/pear-mt?cacheSeconds=300)](https://pypi.org/project/pear-mt/)
[![CI](https://github.com/prosho-97/pear/actions/workflows/ci.yml/badge.svg)](https://github.com/prosho-97/pear/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/License-Apache%202.0-green)](https://www.apache.org/licenses/LICENSE-2.0)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000)](https://github.com/psf/black)

</div>

This repository provides the inference toolkit for 🍐**PEAR**: **P**airwise **E**valuation for **A**utomatic **R**elative Scoring in Machine Translation.

PEAR reframes reference-free machine translation evaluation as a graded pairwise comparison: given a source segment and two candidate translations, it predicts both the **direction** and **magnitude** of their quality difference. The toolkit also supports reference-anchored PEAR scoring and PEAR-based minimum Bayes risk (MBR) decoding.

* 📄 Paper: [ACL Anthology](https://aclanthology.org/2026.acl-long.1953/)
* 🤗 Models and resources: [PEAR Hugging Face collection](https://huggingface.co/collections/Prosho/pear)

## Installation

PEAR supports Python 3.12 and 3.13. Install the published package from PyPI:

```bash
python -m pip install pear-mt
```

The PyPI distribution is named `pear-mt`; the Python import package is named
`pear`, and the primary command-line program is also `pear`:

```python
import pear
```

To install from a local clone instead:

```bash
git clone https://github.com/prosho-97/pear.git
cd pear
python -m pip install .
```

For local development, install the development extra too:

```bash
python -m pip install -e ".[dev]"
```

## Loading a model

```python
import pear

metric = pear.load_metric("pear")       # resolves to Prosho/pear on Hugging Face
metric_xl = pear.load_metric("pear-xl") # resolves to Prosho/pear-xl on Hugging Face
```

You can also pass a Hugging Face repository ID or a local PEAR model checkpoint path:

```python
metric = pear.load_metric("Prosho/pear")
metric = pear.load_metric("/path/to/checkpoints/model.ckpt")
```

### Reproducible model revisions

The official `pear` and `pear-xl` aliases use checkpoint and encoder revisions
pinned by this package. This keeps their results stable even when files on the
Hugging Face Hub change.

For another Hugging Face repository, PEAR follows that repository's default
branch unless you provide revisions explicitly. `revision` pins the PEAR
checkpoint repository; `encoder_revision` pins the underlying Transformers
encoder. A full commit hash gives the strongest reproducibility:

```python
metric = pear.load_metric(
    "organization/custom-pear",
    revision="<checkpoint-commit>",
    encoder_revision="<encoder-commit>",
)
```

The same controls are available from the CLI:

```bash
pear score \
    --hf-model organization/custom-pear \
    --revision <checkpoint-commit> \
    --encoder-revision <encoder-commit> \
    --input pairs.tsv \
    --output scored.tsv
```

Passing either option overrides the corresponding default. Local checkpoint
paths do not use a Hub checkpoint revision, although
`encoder_revision` can still pin an encoder that the checkpoint configuration
loads from the Hub.

## Pairwise QE scoring

```python
scores = pear.score_pairwise(
    metric,
    sources=["The cat is on the mat."],
    translations_a=["El gato está en la alfombra."],
    translations_b=["El gato está en el mapa."],
)
```

Positive scores mean `translations_a` is preferred over `translations_b`.

To score both candidate orders:

```python
scores = pear.score_pairwise(
    metric,
    sources=["The cat is on the mat."],
    translations_a=["El gato está en la alfombra."],
    translations_b=["El gato está en el mapa."],
    mode="both",
)
# {"forward": [...], "reverse": [...]}
```

## Reference-anchored scoring

Reference-anchored PEAR uses the human reference (or any other list of candidate translations) as the second candidate/anchor.

```python
scores = pear.score_reference_anchored(
    metric,
    sources=["The cat is on the mat."],
    translations=["El gato está en la alfombra."],
    references=["El gato está sobre la alfombra."],
)
```

As with pairwise QE scoring, set `mode="both"` to score both candidate orders for reference-anchored inputs.

## PEAR for MBR decoding

```python
from pear.mbr import pear_utility_matrix, select_mbr_hypothesis

hyps = ["This is a good translation.", "This is a very good translation.", "This is a bad translation."]
utility = pear_utility_matrix(metric, "Questa è una traduzione molto buona.", hyps, mode="half")
index, expected_utility = select_mbr_hypothesis(utility)
best = hyps[index]
# best -> "This is a very good translation."
```

Use `mode="full"` to score all off-diagonal ordered pairs (N<sup>2</sup> - N comparisons). Use `mode="half"` to score one triangular half and fill the opposite direction by PEAR antisymmetry.

## Batching, devices, and progress bars

Python and CLI inference are batched. Use `batch_size` / `--batch-size` to control the number of pairwise examples per PEAR forward batch.

By default, PEAR uses:

```python
gpus="auto"
```

This selects one available CUDA/MPS accelerator when present. Pass `gpus=0` or `--gpus 0` to force CPU. Progress bars are shown by default; in Python calls, control them with the `progress_bar` argument (`progress_bar=True` by default, or `progress_bar=False` to disable them).

## CLI

### Score a TSV file

Pairwise input TSV must contain `src`, `mt_0`, and `mt_1` columns:

```bash
pear score --model pear --input pairs.tsv --output scored.tsv --batch-size 16
pear score --hf-model Prosho/pear --input pairs.tsv --output scored.tsv --batch-size 16
pear score --checkpoint /path/to/model.ckpt --input pairs.tsv --output scored.tsv --batch-size 16
```

Reference-anchored input TSV must contain `src`, `mt`, and `ref` columns:

```bash
pear score --model pear --mode reference --input refs.tsv --output scored.tsv --batch-size 16
```

Add `--both` to score both candidate orders.

### Run MBR

Input JSONL rows must contain `src` and `hypotheses`:

```json
{"src": "source text", "hypotheses": ["hyp A", "hyp B"]}
```

Run half- or full-matrix PEAR MBR:

```bash
pear mbr --model pear --input nbest.jsonl --output selected.jsonl --utility half --batch-size 16
pear mbr --hf-model Prosho/pear-xl --input nbest.jsonl --output selected.jsonl --utility full --batch-size 16
pear mbr --checkpoint /path/to/model.ckpt --input nbest.jsonl --output selected.jsonl --batch-size 16
```

## Development

Install the development extra before running formatter checks:

```bash
python -m pip install -e ".[dev]"
```

Then run:

```bash
python -m black --check pear
python -m pytest
python -m compileall pear
```

Network tokenizer regressions use immutable encoder snapshots and run in CI
against both Transformers 4.40.2 and the latest supported 5.x release. The
multi-gigabyte checkpoint regression remains opt-in. Before creating a release
tag, record its baseline under Transformers 4.40.2:

```bash
python -m pytest \
    tests/test_regressions.py::test_released_checkpoints_match_440_scores_and_key_mismatches \
    --run-checkpoint-integration \
    --write-checkpoint-baseline=/tmp/pear-checkpoints-4.40.2.json
```

Then, on the same hardware and dtype under the latest Transformers 5.x, compare
against that file:

```bash
python -m pytest \
    tests/test_regressions.py::test_released_checkpoints_match_440_scores_and_key_mismatches \
    --run-checkpoint-integration \
    --checkpoint-baseline=/tmp/pear-checkpoints-4.40.2.json
```

The comparison requires the same checkpoint-key mismatch sets, finite scores,
preference directions and rankings, and scores within `rtol=1e-4` and
`atol=1e-4`.

## Citation

PEAR has been published at ACL 2026 (Main Conference). If you use it, please consider citing our paper as follows:

```bibtex
@inproceedings{proietti-etal-2026-pear,
    title = "{PEAR}: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation",
    author = "Proietti, Lorenzo  and
      Grundkiewicz, Roman  and
      Post, Matt",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1953/",
    doi = "10.18653/v1/2026.acl-long.1953",
    pages = "42189--42207",
    ISBN = "979-8-89176-390-6"
}
```

## License

This project is released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). See [`LICENSE`](LICENSE) for the full license text.
