Metadata-Version: 2.5
Name: noleak
Version: 0.1.0
Summary: Content-addressed fingerprints and train/eval leakage checks for AI/ML datasets.
Author-email: Atharv Jaju <atharv@jaju.in>
License-Expression: MIT
License-File: LICENSE
Keywords: ai,contamination,data-leakage,evaluation,fingerprint,llm,machine-learning
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# noleak

Content-addressed **dataset IDs** and **train/eval leakage checks** for AI/ML.

Published numbers are only as honest as the split behind them. `noleak` gives every corpus a stable fingerprint (like a Docker image ID) and measures how much of your eval set already appeared in train — exact copies, GPT-3-style word n-grams, and MinHash near-duplicates.

Zero runtime dependencies. Stdlib only.

```python
from noleak import check, fingerprint

train = ["the model was trained on Wikipedia dumps and licensed books"]
eval_set = ["The model was trained on Wikipedia dumps and licensed books"]

print(fingerprint(eval_set))
# noleak-fp-v1:a1b2c3d4e5f6

report = check(train, eval_set)
print(report)
# exact 1/1 (100.00%)  status FAIL
```

## Why this library

Influential AIML libraries are usually **one primitive done well** (`einops`, `safetensors`, `tqdm`). The primitive here:

1. **Identity** — an order-independent hash of unique normalized examples, so a paper can write `eval id noleak-fp-v1:…` and a reviewer can recompute it.
2. **Leakage** — exact overlap, 13-gram contamination (Brown et al., GPT-3), and character-shingle MinHash for paraphrases.

This does **not** replace scikit-learn `Pipeline` leakage tools such as [splitguard](https://pypi.org/project/splitguard/) (runtime `fit()` hooks). It answers a different question: *did this text eval set leak from the train corpus?*

## Install

```bash
pip install noleak
```

Requires Python 3.10+. No extra packages.

For local development from this repo:

```bash
python3 -m pip install -e ".[dev]"
```

## API

```python
from noleak import check, exact_overlap, fingerprint, near_duplicates, ngram_overlap

fp = fingerprint(eval_texts)          # Fingerprint
report = check(train_texts, eval_texts)  # CheckReport
report.contaminated                   # bool (CI-friendly)
report.to_dict()                      # JSON-serializable

exact_overlap(train, eval_set)        # identical after normalize
ngram_overlap(train, eval_set, n=13)  # shared word n-grams
near_duplicates(train, threshold=0.8) # pairs inside one corpus
```

Records may be strings or dicts. Dicts use the first present of `text`, `content`, `prompt`, `question`, `input`, `output`.

## CLI

```bash
noleak fingerprint eval.jsonl
noleak check --train train.jsonl --eval eval.jsonl
noleak check --train train.jsonl --eval eval.jsonl --json
echo $?   # 1 if contaminated, 0 if clean
```

JSONL, JSON lists, and plain text (one example per line) are supported. Use `--field name` when objects store the example under a custom key.

## Fingerprint scheme (`noleak-fp-v1`)

1. Normalize each example (`noleak-norm-v1`): Unicode NFKC, strip, collapse whitespace, casefold.
2. SHA-256 the UTF-8 bytes of that string.
3. Sort **unique** example hashes.
4. SHA-256 `noleak-fp-v1\n` plus those hashes, one per line.

Row order and duplicate rows do not change the digest; `n_examples` vs `n_unique` still records multiplicity.

## Leakage methods

| Method | Detects | Default gate |
| --- | --- | --- |
| `exact` | Same text after normalization | any hit fails |
| `ngram` | Shared word n-gram (default n=13) | rate ≥ 1% fails |
| `near` | MinHash Jaccard ≥ 0.8 on char 5-grams | rate ≥ 5% fails |

Short eval lines (fewer than `n` tokens) are hashed as a single gram so they are not skipped.

## What this will not do

- Catch **target leakage** (a feature derived from the label).
- Catch **preprocessing fitted on the full matrix** (use a `Pipeline` or a runtime hook).
- Prove a model never *saw* an eval item during pretraining of a closed corpus you cannot hash.
- Replace embedding-based semantic dedup (`sentence-transformers`). MinHash is deterministic and model-free; that is the point.

## Development

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[dev]"
pytest
python -m build
```

## License

MIT
