Metadata-Version: 2.5
Name: datasets-lint
Version: 0.2.0
Summary: ruff for training data: audit datasets for duplicates, PII, leakage and contamination
Project-URL: Homepage, https://github.com/horizonbymuneeb/datasets-lint
Project-URL: Issues, https://github.com/horizonbymuneeb/datasets-lint/issues
Author: datasets-lint contributors
License: MIT
License-File: LICENSE
Keywords: data-quality,datasets,deduplication,linter,llm,machine-learning,pii
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Requires-Dist: datasketch>=1.6
Requires-Dist: jinja2>=3.1
Requires-Dist: polars>=1.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Provides-Extra: dev
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: hf
Requires-Dist: datasets>=2.18; extra == 'hf'
Provides-Extra: ner
Requires-Dist: spacy>=3.7; extra == 'ner'
Description-Content-Type: text/markdown

# datasets-lint

**ruff for training data.** Audit a dataset for duplicates, PII, distribution problems and
eval-set contamination — offline, on CPU, in one command, with a CI-friendly exit code.

```console
$ datasets-lint run train.jsonl --report out.html
datasets-lint 0.1.0  ·  train.jsonl
  120,000 rows audited (of 120,000, seed 0)  ·  text=text  label=label

  ✓ stats      0.42s   120,000 rows · 0.00% empty · worst column 0.0% null
  ✗ dup       12.10s   18.20% near-duplicate (4,211 clusters), 12.00% exact
  ✗ pii        0.91s   0.41% of rows contain PII (email 487, phone 12, api_key 1)

  verdict: error — 2 errors, 0 warnings, 1 ok · 13.43s
  report → out.html
```

## Why

![datasets-lint HTML report](docs/report.png)

Every team that fine-tunes or publishes a dataset re-writes the same checks in a
throwaway notebook: duplicates, PII, contamination. It is slow, inconsistent, and never
runs in CI. `datasets-lint` packages it once:

- **fast** — samples 100k rows by default and finishes in seconds, not hours
- **offline** — no API keys, no cloud, no data leaves the machine
- **CI-native** — `exit 1` on threshold breach, JSON output, baseline diffing
- **readable** — one self-contained HTML report you can attach to a dataset card or a PR
- **extensible** — one check = one file; the plugin API is ~20 lines

## Install

```console
uvx datasets-lint --help        # run without installing
# or
pip install datasets-lint
```

## Usage

```console
datasets-lint run data.jsonl                       # text summary
datasets-lint run data.parquet --report out.html   # single-file HTML report
datasets-lint run train.jsonl test.jsonl --checks dup   # cross-split duplicate check
datasets-lint run data.csv --json-out report.json --fail-on warn
datasets-lint run data.jsonl --baseline prev.json  # did this PR make the data worse?
datasets-lint checks                               # list checks
```

Supported inputs: `.jsonl`, `.ndjson`, `.json`, `.csv`, `.parquet`.

Exit codes: `0` ok · `1` threshold breached · `2` usage/input error.

## Checks

| id | what it finds | default severity |
|---|---|---|
| `stats` | empty rows, nulls, length percentiles, label balance | `error` if >1% empty |
| `dup` | exact + near duplicates (MinHash/LSH), cross-file overlap | `error` if >2% near-dup |
| `pii` | emails, phones, IPs, IBANs, SSNs, API keys (redacted) | `error` on secrets |
| `contamination` | overlap with GSM8K / HumanEval or your own eval sets | `error` if >1% overlap |

### Contamination

A row is flagged when a large share of *its* n-grams already appears in an eval set
(containment, not symmetric Jaccard — training rows are often longer than the leaked
question). Bundled fingerprints: **GSM8K** (1,319 rows) and **HumanEval** (164), MIT
licensed, generated by `scripts/fetch_eval_sets.py`.

```console
$ datasets-lint run train.jsonl
  ✗ contamination   0.13s   4.00% of rows overlap an eval set (gsm8k 4.00%)
```

Add your own eval set (any `.jsonl` / `.csv` / `.parquet` / `.txt`):

```toml
[checks.contamination]
eval_sets = ["gsm8k", "humaneval"]
ngram = 8                 # words per shingle
min_containment = 0.5     # share of a row's n-grams that must be in the eval set
max_contaminated_pct = 1.0

[checks.contamination.custom_eval_sets]
my_bench = "evals/my_bench.jsonl"
```

## Configuration

`datasets-lint.toml` in the working directory (or `--config path`):

```toml
[defaults]
sample = 100000
seed = 0
fail_on = "error"          # none | warn | error

[checks.dup]
threshold = 0.9            # Jaccard threshold for near-duplicates
num_perm = 64
shingle_size = 5
max_near_dup_pct = 2.0

[checks.pii]
max_error_findings_pct = 0.1
max_warn_findings_pct = 5.0

[checks.stats]
max_empty_pct = 1.0
max_null_pct = 50.0
min_label_pct = 1.0
```

## Writing a check

```python
# src/datasets_lint/checks/short.py
from datasets_lint.checks.base import Check, Context
from datasets_lint.dataset import Dataset
from datasets_lint.result import CheckResult, Severity


class ShortRowsCheck(Check):
    id = "short"
    title = "Short rows"
    description = "Flags texts under N characters."

    def run(self, ds: Dataset, ctx: Context) -> CheckResult:
        limit = ctx.threshold(self.id, "min_chars", 20)
        bad = [i for i, t in ds.texts() if 0 < len(t) < limit]
        pct = round(100 * len(bad) / max(ds.audited_rows, 1), 2)
        return CheckResult(
            check_id=self.id,
            title=self.title,
            severity=Severity.WARN if bad else Severity.OK,
            summary=f"{pct}% of rows are shorter than {limit} chars",
            metrics={"short_pct": pct},
            findings=[],  # add Finding(...) entries for samples
        )
```

Register it in `src/datasets_lint/checks/__init__.py`. Add a fixture and a test — that's it.

## License

MIT. Bundled eval data: GSM8K and HumanEval, both MIT licensed — see
`src/datasets_lint/data/eval/manifest.json` for source URLs and checksums.
