Metadata-Version: 2.4
Name: benchleak
Version: 0.4.0
Summary: Detect benchmark contamination in large language models
Project-URL: Homepage, https://github.com/bouchnam/benchleak
Project-URL: Repository, https://github.com/bouchnam/benchleak
Author: Mouad Bouchnaf
License: Apache-2.0
License-File: LICENSE
Keywords: benchmark,contamination,evaluation,llm,memorization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: datasets>=2.18
Requires-Dist: numpy>=1.24
Requires-Dist: rich>=13.0
Requires-Dist: scipy>=1.11
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.40
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

# benchleak 🔍

[![CI](https://github.com/bouchnam/benchleak/actions/workflows/ci.yml/badge.svg)](https://github.com/bouchnam/benchleak/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/benchleak)](https://pypi.org/project/benchleak/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)

> **Did this model train on the test set? Find out in one command.**

When a model scores 90% on GSM8K or MATH, was it genuinely capable or did it
memorise the benchmark during training? Benchleak answers that with a
mathematical membership-inference test on the model's own token probabilities.
No LLM judges, no API calls, runs locally on any HuggingFace causal LM.

## Status

🚧 **Early alpha.** All three detectors are implemented and runnable end to end:
the **pre-training** detector (Min-K% Prob), the **fine-tuning** detector
(probabilistic variation, SPV-MIA), and the **RL-post-training** detector
(Self-Critique entropy).

## Install

```bash
pip install benchleak
```

This pulls in `torch`, `transformers`, and `datasets`. To work from a clone
instead, run `pip install -e .` in the repo root.

## Usage

```bash
benchleak --model Qwen/Qwen2.5-0.5B --benchmark gsm8k
```

The model can be any HuggingFace-format causal LM, given as a Hub id or a local
checkpoint directory. GGUF, llama.cpp, and Ollama formats are not supported.

### Private or gated models

For a repository that requires authentication, provide a HuggingFace token. Any
of these work:

```bash
huggingface-cli login                  # cached credential, picked up automatically
export HF_TOKEN=hf_xxx                  # environment variable
benchleak --model my/private-model --benchmark gsm8k --hf-token hf_xxx
```

### Speed

Scoring runs one forward pass per sample. On a CPU-only machine the default
`--limit 200` can take many minutes; use a smaller `--limit` for a quick look, or
`--device cuda` / `--device mps` to use a GPU.

```
benchleak: contamination report
====================================================
Model:       Qwen/Qwen2.5-0.5B
Benchmark:   gsm8k
Detector:    min-k% prob
Samples:     40 benchmark vs 40 reference
Reference:   bundled reference-math.txt

Separation (AUC):   0.580   [LOW]
Significance (p):   0.11
Flag thresholds:    AUC >= 0.6, p < 0.05

Verdict: NO STRONG EVIDENCE
```

The benchmark is compared against a reference set matched to its domain and
format — for GSM8K, original math word problems written for this project. That
matters: against a general-prose reference the same scan reports AUC 0.98 and
flags the model, but that separation is almost entirely *math vs prose*, not
*seen vs unseen*. A domain-matched reference isolates the memorisation signal.

Known benchmarks (`gsm8k`, `math`, `arc-challenge`, `truthfulqa`) work by name. For
any other Hub dataset, pass the path plus its text column(s):

```bash
benchleak --model my/model --benchmark some/dataset --field question --field answer
```

### Your own benchmark from a local file

Point `--benchmark` at a local file instead of a Hub id. Supported formats:

- `.txt`: one passage per line (no `--field` needed)
- `.jsonl` / `.json` / `.csv`: name the text column(s) with `--field`

```bash
benchleak --model my/model --benchmark ./my_benchmark.jsonl --field question --field answer
```

Local files are read with the standard library, so this path needs neither a
network connection nor the `datasets` package.

### Choosing a detector

`--detector pretrain` (the default) runs Min-K% Prob, which targets memorisation
from **pre-training**. `--detector sft` runs probabilistic variation (SPV-MIA),
which targets memorisation from **fine-tuning** and is the right choice when you
suspect a model was fine-tuned on a benchmark:

```bash
benchleak --model my/model --benchmark gsm8k --detector sft
```

The fine-tuning detector paraphrases each sample and compares log-likelihoods, so
it is roughly an order of magnitude slower than the pre-training detector. By
default it downloads a T5 paraphrasing model; pass `--perturber word` to avoid the
download (lower quality) and `--n-perturbations` to trade speed for stability. See
[docs/how-it-works-sft.md](docs/how-it-works-sft.md) for the method and caveats.

`--detector rl` runs Self-Critique, which targets memorisation from **RL
post-training** (RLVR/GRPO) — the phase where likelihood-based detectors fail. It
is the right choice for an RL/reasoning-tuned instruct model:

```bash
benchleak --model my/model --benchmark gsm8k --detector rl
```

Here each sample is treated as a *problem to solve*: the detector generates a
response, asks the model to redo it along a different reasoning path, and measures
how similar the two answers' entropy curves stay (a contaminated problem can't
deviate). Because it generates two responses per sample it is the slowest
detector — use a small `--limit` and tune `--max-new-tokens`. It needs a model
with a chat template. See [docs/how-it-works-rl.md](docs/how-it-works-rl.md).

## How it works

The benchmark is scored against a **reference set** of text the model is not
expected to have memorised. Min-K% Prob assigns each text the mean log-probability
of its least-likely *k%* of tokens. Memorised text has fewer surprising tokens
and scores higher. The tool then measures how strongly the benchmark's scores separate
from the reference's, reported as an AUC (`U / nm` from a Mann-Whitney test) with
a significance p-value. AUC ≈ 0.5 means the benchmark looks like fresh data; AUC
well above 0.5 is the memorisation signature of contamination.

Bundled reference sets ship with the tool so it runs out of the box, and the
right one is picked by the benchmark's domain: math benchmarks (`gsm8k`, `math`)
are compared against original GSM8K-style word problems written for this project
(never published, so no model trained on them — and their arithmetic is
machine-verified by the test suite); everything else uses a general-prose set.
For the cleanest signal, supply your own domain-matched reference data with
`--reference my_reference.txt` (one passage per line).

For the full reasoning (why a reference set is needed, the choice of test, and
where the method can mislead), see [docs/how-it-works-pretrain.md](docs/how-it-works-pretrain.md).

| Phase | Method | Status |
|-------|--------|--------|
| Pre-training | Min-K% probability (Shi et al. 2024) | ✅ implemented |
| SFT | Probabilistic variation / SPV-MIA (Fu et al. 2024) | ✅ implemented |
| RL post-training | Self-Critique entropy (Tao et al. 2025) | ✅ implemented |

## Caveats

- A verdict needs **≥ 5 samples per side**; the significance test cannot reach
  p < 0.05 below that.
- Math benchmarks are automatically compared against the bundled math reference.
  For any other narrow-domain benchmark the fallback is general-domain prose,
  which can confound *domain* with *memorisation*; prefer a domain-matched
  `--reference` for results you intend to publish.

## Troubleshooting

**`ModuleNotFoundError: No module named '_lzma'`** when loading a benchmark. The
`datasets` library needs Python's `lzma` module, which is absent from some Python
builds (commonly pyenv on macOS compiled without the `xz` library). Install `xz`
and rebuild Python:

```bash
brew install xz
LDFLAGS="-L$(brew --prefix xz)/lib" CPPFLAGS="-I$(brew --prefix xz)/include" \
  pyenv install -f <your-python-version>
```

Then reuse or recreate your virtual environment. benchleak detects this case and
prints the same guidance.

## Citation

Implements methods from:
- Shi et al., 2024. *Detecting Pretraining Data from Large Language Models* (arXiv:2310.16789)
- Fu et al., 2024. *Membership Inference via Self-Prompt Calibration*
- Tao et al., 2025. *Detecting Data Contamination from RL Post-training* (arXiv:2510.09259)

## License

Apache-2.0
