Metadata-Version: 2.4
Name: contamination-audit
Version: 0.1.2
Summary: Generate item-level eval contamination signals from lexical, canary, pattern, hash, and optional embedding checks.
Author: AuraOne
License-Expression: MIT
Project-URL: Homepage, https://github.com/auraoneai/contamination-audit
Project-URL: Documentation, https://github.com/auraoneai/contamination-audit#readme
Project-URL: Repository, https://github.com/auraoneai/contamination-audit.git
Project-URL: Issues, https://github.com/auraoneai/contamination-audit/issues
Project-URL: Changelog, https://github.com/auraoneai/contamination-audit/blob/main/CHANGELOG.md
Keywords: evaluation-data,data-contamination,benchmark-leakage,llm-evaluation,canary,embedding-similarity
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: embedding
Requires-Dist: sentence-transformers>=2.7; extra == "embedding"
Dynamic: license-file

# contamination-audit

Generate item-level signals that help reviewers investigate possible evaluation-data leakage.

`contamination-audit` is for benchmark maintainers and evaluation engineers who want a local, inspectable preflight before deeper contamination analysis. It combines n-gram overlap, answer-pattern checks, canary matching, synthetic registry-hash matching, lexical similarity, and an optional sentence-transformers backend.

## Inspectable Output

The CLI writes JSON with `item_count`, `finding_count`, a bounded aggregate risk score, and every detector-level finding. Findings include item and reference ids, detector name, matched corpus marker, canary, or similarity score where applicable.

The Python `run` API accepts separate `items`, `references`, `outputs`, and registry data. The CLI currently accepts only `--eval-data`; it uses those same rows as its comparison references and does not accept model outputs, so it is a smoke-test path rather than a held-out-corpus audit.

## Runtime Boundary

The default lexical path runs locally with no network dependency. The built-in `pile`, `c4`, and `hf-mmlu` registry entries are synthetic marker hashes; the package does not download or search those corpora.

The optional `sentence-transformers` backend loads the named model locally. Its underlying library may download model files when they are not already cached, so use a local model/cache when runtime network access is not allowed.

## Install

```bash
python -m pip install contamination-audit==0.1.2
```

Optional semantic embeddings:

```bash
python -m pip install "contamination-audit[embedding]==0.1.2"
```

For development from a clone:

```bash
python -m pip install -e .
```

## Quickstart

From a repository checkout, run the no-download path:

```bash
contamination-audit run \
  --eval-data examples/eval.jsonl \
  --skip-embedding \
  > contamination-report.json
```

See [`docs/methodology.md`](docs/methodology.md) for detector and data-boundary details.

## Release Status

Registry status verified July 13, 2026: version `0.1.2` is published on PyPI and tagged `v0.1.2` in the public repository. The project is alpha software. No claim of contamination detection coverage, clean data, or adoption is made.

## Limits

This package emits diagnostic signals, not proof that data is contaminated or clean. The aggregate risk score is a simple finding-count heuristic, the bundled corpus registry is synthetic, and reviewers must interpret evidence in context.

## Next Action

Run the no-download CLI path to inspect the report shape, then call the Python API with approved, separated references and model outputs before using any finding in a contamination review.
