Metadata-Version: 2.3
Name: cleanllm
Version: 4.0.0
Summary: Streaming JSONL cleaner for LLM fine-tuning datasets.
Author: cleanllm contributors
License: ﻿MIT License
        
        Copyright (c) 2026 cleanllm contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
Keywords: alpaca,audit,chatml,cleaning,dataset,deduplication,fine-tuning,jsonl,llm,sampling,sft,sharegpt
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Requires-Dist: orjson>=3.9.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: typer>=0.12.0
Provides-Extra: deita
Provides-Extra: dev
Requires-Dist: build>=1.0.0; extra == 'dev'
Requires-Dist: twine>=4.0.2; extra == 'dev'
Provides-Extra: gpu
Requires-Dist: torch>=2.0.0; extra == 'gpu'
Requires-Dist: transformers>=4.30.0; extra == 'gpu'
Provides-Extra: hf
Requires-Dist: datasets>=2.14.0; extra == 'hf'
Provides-Extra: pii
Requires-Dist: spacy>=3.7.0; extra == 'pii'
Provides-Extra: reward
Requires-Dist: torch>=2.0.0; extra == 'reward'
Requires-Dist: transformers>=4.30.0; extra == 'reward'
Provides-Extra: semantic
Requires-Dist: numpy>=1.24.0; extra == 'semantic'
Requires-Dist: sentence-transformers>=2.2.0; extra == 'semantic'
Provides-Extra: tiktoken
Requires-Dist: tiktoken>=0.5.0; extra == 'tiktoken'
Description-Content-Type: text/markdown

# cleanllm

**Streaming JSONL cleaner for LLM fine-tuning datasets.** Minimal dependencies, memory-safe, and fast — processes files line-by-line without loading them into memory.

[![PyPI](https://img.shields.io/pypi/v/cleanllm)](https://pypi.org/project/cleanllm/)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

---

## What it does

cleanllm gives you a pipeline for cleaning, validating, and profiling JSONL datasets before fine-tuning:

```
raw.jsonl → scan → fix → dedup → validate → stats → audit bundle → shards
```

Every step is streaming (no full-file load), resumable, and produces machine-readable JSON reports for CI gating.

---

## Install

```bash
pip install cleanllm
```

Or from source:

```bash
git clone https://github.com/verma8076/cleanllm
cd cleanllm
pip install -e .
```

For GPU-accelerated scoring, semantic deduplication, and PII redaction:

```bash
pip install cleanllm[gpu]              # IFD scoring (GPT-2 forward passes)
pip install cleanllm[reward]           # reward model scoring (DeBERTa RM)
pip install cleanllm[semantic]         # semantic dedup + decontaminate (sentence-transformers)
pip install cleanllm[pii]              # spaCy NER for redact-pii
pip install cleanllm[gpu,reward,semantic,pii]  # everything
```

---

## GPU Benchmark

Measured on **Google Colab Tesla T4** (15.6 GB VRAM), 500 records, gpt2 (117M params):

| Feature | Device | rec/sec |
|---|---|---|
| IFD scoring (`ifd-score`) | CPU | 2.9 |
| IFD scoring (`ifd-score`) | **T4 GPU** | **43.1** — 14.9× faster |
| IFD scoring via CLI file | CPU | 2.6 |
| IFD scoring via CLI file | **T4 GPU** | **32.1** — 12.3× faster |
| Semantic dedup | CPU | 25.3 |
| Semantic dedup | **T4 GPU** | **68.6** — 2.7× faster |

Semantic model: `all-MiniLM-L6-v2`. Run [`benchmark_colab.ipynb`](benchmark_colab.ipynb) to reproduce.

---

## Quickstart

```bash
# Scan for issues
cleanllm scan data.jsonl

# Fix: remove URLs, normalize whitespace, redact forbidden patterns
cleanllm fix data.jsonl -o data.cleaned.jsonl

# Deduplicate by prompt content
cleanllm dedup data.cleaned.jsonl -o data.dedup.jsonl --by prompt

# Profile the cleaned dataset
cleanllm stats data.dedup.jsonl --report-json stats.json

# Gate in CI: fail if invalid rows increased
cleanllm gate --compare compare.json --rules gate_rules.json
```

---

## CLI reference

### `scan`
Streaming scan for issues — invalid JSON, missing keys, URLs, forbidden patterns, language distribution, duplicate estimate.

```bash
cleanllm scan data.jsonl
cleanllm scan data.jsonl --report-json scan_report.json --dup-estimate
cleanllm scan data.jsonl --preset cp_portable
```

### `fix`
Remove URLs, normalize whitespace, redact or drop rows with forbidden patterns.

```bash
cleanllm fix data.jsonl -o cleaned.jsonl
cleanllm fix data.jsonl -o cleaned.jsonl --drop-on forbidden_pattern --drop-on invalid_json
cleanllm fix data.jsonl -o cleaned.jsonl --preset cpp17_clean --report-json fix_report.json
```

Drop rules: `invalid_json`, `missing_required_keys`, `forbidden_pattern`, `empty_assistant`, `placeholder`, `repetitive_response`, `bad_conversation`.

> **Note on `empty_assistant`:** By default this drops assistant responses shorter than 20 characters — calibrated for code datasets where very short responses are almost always errors. For text/chat datasets, set `--min-assistant-chars 1` to only drop truly blank responses.

### `validate`
Schema validation, line by line. Exit code `0` only if all rows pass.

```bash
cleanllm validate data.jsonl --schema basic_sft
cleanllm validate data.jsonl --schema cp_sft_v1
```

| Schema | Required fields |
|---|---|
| `basic_sft` | `id`, `messages` (list of `role`/`content` dicts) |
| `cp_sft_v1` | `id`, `source`, `problem_id`, `messages`, `tests` (non-empty, with `input`/`output`) |

### `dedup`
First-occurrence deduplication — by full record, prompt (system+user), or code (assistant).

```bash
cleanllm dedup data.jsonl -o deduped.jsonl --by record
cleanllm dedup data.jsonl -o deduped.jsonl --by prompt --normalized
cleanllm dedup data.jsonl -o deduped.jsonl --by code --report-json dedup_report.json
```

### `stats`
Single-pass profiler: distributions, structural stats, schema counts, response lengths, language distribution.

```bash
cleanllm stats data.jsonl
cleanllm stats data.jsonl --schema cp_sft_v1 --keys source,difficulty_bucket --top-k 20
cleanllm stats data.jsonl --report-json stats.json
```

### `compare`
Diff two stats reports to catch regressions between dataset versions.

```bash
cleanllm compare old_stats.json new_stats.json
cleanllm compare old_stats.json new_stats.json --report-json compare.json
cleanllm compare old.jsonl new.jsonl --from-jsonl --schema cp_sft_v1
```

### `gate`
CI-friendly quality gating. Nonzero exit on failures.

```bash
cleanllm gate --stats stats.json --rules gate_rules.json
cleanllm gate --compare compare.json --rules gate_rules.json --strict
cleanllm gate --compare compare.json --inline-rule "counts_diff.invalid_json_rows.delta<=0"
```

Gate rules JSON:

```json
{
  "version": 1,
  "mode": "compare",
  "rules": [
    {"name": "no_new_invalid", "metric": "counts_diff.invalid_json_rows.delta", "op": "<=", "value": 0},
    {"name": "enough_valid",   "metric": "counts_diff.valid_json_rows.new",    "op": ">=", "value": 1000}
  ]
}
```

Supported ops: `==`, `!=`, `<`, `<=`, `>`, `>=`. Severities: `error` (default), `warn`.

### `run`
Execute a JSON-defined multi-step pipeline with variable substitution.

```bash
cleanllm run --config pipeline.json
cleanllm run --config pipeline.json --set input_path=data.jsonl --set outdir=out/v2
cleanllm run --config pipeline.json --dry-run
```

Supported step types: `fix`, `validate`, `dedup`, `stats`, `audit`, `sample`, `shard`, `manifest`, `scan`, `compare`.

### `sample`
Reservoir sampling — random or stratified, deterministic with `--seed`.

```bash
cleanllm sample data.jsonl -o sample.jsonl -n 500 --seed 42
cleanllm sample data.jsonl -o sample.jsonl -n 500 --stratify source,difficulty_bucket
```

### `audit`
Build a reproducible audit bundle in one command: sampled JSONL + CSV review index (with original line numbers) + summary + manifest.

```bash
cleanllm audit data.jsonl --outdir audit_bundle -n 200 --seed 42
cleanllm audit data.jsonl --outdir audit_bundle -n 200 --stratify source --schema cp_sft_v1
```

Bundle contents: `audit_sample.jsonl`, `audit_index.csv`, `audit_summary.json`, `AUDIT_README.md`, `manifest.json`.

### `shard` / `manifest`

```bash
cleanllm shard data.jsonl --outdir shards --size 5000 --gzip
cleanllm manifest shards -o manifest.json
```

### `convert`
Convert a JSONL file between `sharegpt`, `alpaca`, and `chatml` formats.

```bash
cleanllm convert data.jsonl -o converted.jsonl --from sharegpt --to chatml
cleanllm convert data.jsonl -o converted.jsonl --from alpaca --to sharegpt
```

Supported formats: `sharegpt` (conversations list), `alpaca` (instruction/output), `chatml` (messages list).

### `merge`
Merge multiple JSONL files into one, with optional deduplication.

```bash
cleanllm merge a.jsonl b.jsonl c.jsonl -o merged.jsonl
cleanllm merge a.jsonl b.jsonl -o merged.jsonl --dedup
```

### `split`
Split a JSONL file into train and val sets.

```bash
cleanllm split data.jsonl --outdir splits/
cleanllm split data.jsonl --outdir splits/ --ratio 0.95 --seed 42 --no-shuffle
```

Outputs `<basename>_train.jsonl` and `<basename>_val.jsonl` in the output directory. Default ratio is 0.9 (90% train).

### `ifd-score`
Score each record with real Instruction-Following Difficulty (IFD) — PPL(response|instruction) / PPL(response alone). Higher = harder instruction = more valuable for SFT.

```bash
# Requires: pip install cleanllm[gpu]
cleanllm ifd-score data.jsonl -o scored.jsonl
cleanllm ifd-score data.jsonl -o scored.jsonl --device cuda --low-threshold 0.3
```

Stamps each record with `_ifd_score`. Use `cleanllm filter` to cut by threshold.

### `score-rm`
Score each record with a reward model. Uses `OpenAssistant/reward-model-deberta-v3-large-v2` (~180 MB) by default — works on CPU, faster on GPU.

```bash
# Requires: pip install cleanllm[reward]
cleanllm score-rm data.jsonl -o scored.jsonl
cleanllm score-rm data.jsonl -o scored.jsonl --device cuda --low-threshold 0.0
cleanllm score-rm data.jsonl -o scored.jsonl --model OpenAssistant/reward-model-deberta-v3-large-v2
```

Stamps each record with `_rm_score`. Combine with `cleanllm filter` to keep only high-reward examples.

### `filter`
Filter a pre-scored JSONL by any score field. Use after any scoring command.

```bash
cleanllm filter scored.jsonl -o filtered.jsonl --min-ifd 0.3
cleanllm filter scored.jsonl -o filtered.jsonl --min-rm 0.0
cleanllm filter scored.jsonl -o filtered.jsonl --min-ifd 0.3 --min-rm 0.0
cleanllm filter scored.jsonl -o filtered.jsonl --require-field _ifd_score --require-field _rm_score

# Verbosity filters (after score-verbosity)
cleanllm filter scored.jsonl -o filtered.jsonl --max-verbosity 5.0 --max-repetition 0.3 --max-filler 0.4

# DEITA / token noise filters (after score-deita / score-token)
cleanllm filter scored.jsonl -o filtered.jsonl --min-deita 3.0 --max-token-noise 0.2

# Multi-field expression
cleanllm filter scored.jsonl -o filtered.jsonl --score-expr "_ifd_score * _rm_score > 0.5"

# Audit trail — writes every dropped record with _rejection_reason, _rejected_by, _rejected_at
cleanllm filter scored.jsonl -o filtered.jsonl --min-ifd 0.3 --rejected-log dropped.jsonl
```

### `decontaminate`
Remove or flag training examples that overlap with standard evaluation benchmarks using 8-gram fingerprinting. Streaming, no GPU required.

```bash
cleanllm decontaminate data.jsonl -o clean.jsonl
cleanllm decontaminate data.jsonl -o clean.jsonl --benchmark mmlu --benchmark gsm8k
cleanllm decontaminate data.jsonl -o clean.jsonl --mode flag   # adds _contaminated field instead of dropping
cleanllm decontaminate data.jsonl -o clean.jsonl --n 10 --no-cache
# Custom benchmark:
cleanllm decontaminate data.jsonl -o clean.jsonl --benchmark-file mybench=eval.jsonl
# Add semantic similarity pass to catch paraphrase contamination (requires pip install cleanllm[semantic]):
cleanllm decontaminate data.jsonl -o clean.jsonl --semantic --semantic-threshold 0.85
```

Supported benchmarks: `mmlu`, `humaneval`, `gsm8k`, `arc` (all four checked by default). Benchmark n-gram indexes are cached to `~/.cleanllm/benchmarks/` after the first run. Custom benchmarks require a JSONL with a `text` field per row.

### `check-turns`
Multi-turn conversation quality checks. Flags role violations, length imbalance, empty turns, and structural issues without modifying the file.

```bash
cleanllm check-turns data.jsonl
cleanllm check-turns data.jsonl --report-json checks.json
```

### `score-token`
Token-level noise mask scoring. Writes `_token_noise_score` per record — the fraction of tokens flagged as high-noise. Compatible with TRL per-token loss weights.

```bash
# Requires: pip install cleanllm[gpu]
cleanllm score-token data.jsonl -o scored.jsonl
cleanllm score-token data.jsonl -o scored.jsonl --device cuda
```

### `score-deita`
DEITA-style complexity × quality scoring via a local LLM (Ollama). Writes `_deita_score`, `_complexity_score`, `_quality_score`.

```bash
# Requires: a running Ollama endpoint
cleanllm score-deita data.jsonl -o scored.jsonl
cleanllm score-deita data.jsonl -o scored.jsonl --model llama3.2 --ollama-url http://localhost:11434
```

### `score-verbosity`
CPU-only verbosity and repetition scoring. No model, no GPU — single streaming pass.

```bash
cleanllm score-verbosity data.jsonl -o scored.jsonl
```

Stamps three fields per record:

| Field | Meaning |
|---|---|
| `_verbosity_score` | response / instruction word ratio — high values = suspiciously long responses |
| `_repetition_score` | fraction of 4-grams in the response that appear more than once |
| `_filler_score` | fraction of sentences that match filler patterns ("I hope this helps", "In conclusion,", etc.) |

Filter downstream: `cleanllm filter scored.jsonl -o out.jsonl --max-verbosity 5.0 --max-repetition 0.3`

### `repair`
Auto-fix conversation structure. Streaming, no model.

```bash
cleanllm repair data.jsonl -o fixed.jsonl
cleanllm repair data.jsonl -o fixed.jsonl --no-strip-sycophancy --no-strip-filler
cleanllm repair data.jsonl -o fixed.jsonl --collapse-consecutive
```

Operations (all on by default):
- Strip sycophantic prefixes from assistant turns (`"Certainly! "`, `"Great question! "`, …)
- Strip trailing filler from assistant turns (`"I hope this helps!"`, `"Feel free to ask!"`, …)
- Drop trailing user turns so conversations end cleanly on an assistant message
- `--collapse-consecutive`: merge back-to-back same-role messages into one

### `select`
Score-weighted top-K data selection — the final step of the scoring pipeline.

```bash
cleanllm select scored.jsonl -o selected.jsonl --top 5000
cleanllm select scored.jsonl -o selected.jsonl --top 20%
cleanllm select scored.jsonl -o selected.jsonl --top 5000 --sort-by _rm_score
cleanllm select scored.jsonl -o selected.jsonl --top 5000 --score-expr "_ifd_score * _rm_score"
cleanllm select scored.jsonl -o selected.jsonl --top 5000 --diversity-weight 0.3
```

- `--top N` or `--top N%`: absolute count or percentage
- `--sort-by FIELD`: sort by a single score field (descending)
- `--score-expr EXPR`: safe Python arithmetic expression over record fields
- `--diversity-weight 0.0–1.0`: blend score ranking with embedding-based diversity (requires `[semantic]`)

### `mix`
Weighted dataset mixing — combine sources with per-file weights.

```bash
cleanllm mix source1.jsonl:0.6 source2.jsonl:0.3 source3.jsonl:0.1 -o mixed.jsonl --total 10000
cleanllm mix source1.jsonl source2.jsonl -o mixed.jsonl --strategy round-robin
```

- `file.jsonl:weight` — weight is a sampling probability (automatically normalized)
- `--total N`: target output record count
- `--strategy weighted|round-robin|interleave`
- `--seed N`: reproducible shuffle

### `redact-pii`
PII detection and redaction. Regex layer always active; spaCy NER is opt-in.

```bash
cleanllm redact-pii data.jsonl -o clean.jsonl --mode redact   # replace with [EMAIL], [PHONE], etc.
cleanllm redact-pii data.jsonl -o clean.jsonl --mode flag     # add _pii_detected field, keep text
cleanllm redact-pii data.jsonl -o clean.jsonl --mode drop     # remove rows that contain PII

# spaCy NER for PERSON, ORG, GPE, LOC (requires pip install cleanllm[pii] && python -m spacy download en_core_web_sm)
cleanllm redact-pii data.jsonl -o clean.jsonl --mode redact --use-spacy
```

Regex-detected types: `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `URL`. spaCy adds: `PERSON`, `ORG`, `GPE`, `LOC`.

### `preflight`
Framework-aware pre-flight validation before you start a training run. Catches schema mismatches, bad role sequences, and missing fields that would silently corrupt training.

```bash
cleanllm preflight data.jsonl --framework trl
cleanllm preflight data.jsonl --framework axolotl
cleanllm preflight data.jsonl --framework torchtune
cleanllm preflight data.jsonl --framework unsloth
cleanllm preflight data.jsonl --framework trl --report-json preflight.json
```

Supported frameworks and what they check:

| Framework | Format checked | Key rules |
|---|---|---|
| `trl` | `messages` list | roles: user/assistant/system/tool/ipython; no back-to-back same role; system must be first |
| `axolotl` | `messages` OR `conversations` OR alpaca | auto-detects format, delegates to appropriate validator |
| `unsloth` | same as axolotl | — |
| `torchtune` | `messages` list | roles: user/assistant/system/ipython |

Exit code `1` if any rows fail — pipe-safe for CI.

### `recipes`
Bootstrap pipelines and gate rules from built-in templates.

```bash
cleanllm recipes list
cleanllm recipes show cp_pipeline_cp_portable
cleanllm recipes write cp_bundle --outdir bootstrap/
```

Built-in recipes: `cp_pipeline_basic`, `cp_pipeline_cp_portable`, `cp_pipeline_fast_audit`, `gate_stats_basic`, `gate_compare_basic`, `gate_compare_strict`, `cp_bundle`.

---

## Python API

```python
from cleanllm import (
    scan_jsonl, fix_jsonl, FixRules,
    dedup_jsonl, validate_jsonl, stats_jsonl,
    sample_jsonl, audit_bundle,
    shard_jsonl, make_manifest,
    download_from_hub, detect_hf_schema,
)
from cleanllm.convert import convert_jsonl
from cleanllm.merge import merge_jsonl
from cleanllm.split import split_jsonl

# Scan
report = scan_jsonl("data.jsonl")

# Fix (code dataset)
rules = FixRules(
    drop_on={"forbidden_pattern", "empty_assistant"},
    max_tokens=4096,
    keep_language="python",
)
summary = fix_jsonl("data.jsonl", "cleaned.jsonl", rules)

# Fix (text/chat dataset — only drop truly blank responses)
rules = FixRules(drop_on={"empty_assistant"}, min_assistant_chars=1, forbidden_patterns=[])

# Dedup
result = dedup_jsonl("cleaned.jsonl", "deduped.jsonl", by="prompt", normalized=True)

# Stats
stats = stats_jsonl("deduped.jsonl", schema="cp_sft_v1", keys=["source", "difficulty_bucket"])

# Sample + audit
sample_jsonl("deduped.jsonl", "sample.jsonl", num_rows=200, seed=42)
audit_bundle("deduped.jsonl", "audit_bundle", num_rows=200, seed=42, stratify=["source"])

# Shard + manifest
shard_jsonl("deduped.jsonl", "shards", shard_size=5000, gzip_output=True)
make_manifest("shards", "manifest.json")

# Convert between formats
convert_jsonl("data.jsonl", "out.jsonl", from_fmt="sharegpt", to_fmt="chatml")

# Merge + split
merge_jsonl(["a.jsonl", "b.jsonl"], "merged.jsonl", dedup=True)
split_jsonl("merged.jsonl", "splits/", ratio=0.9, seed=42)

# Download from HuggingFace Hub (requires pip install cleanllm[hf])
result = download_from_hub("HuggingFaceH4/ultrachat_200k", "data.jsonl", split="train_sft")

# IFD scoring (requires pip install cleanllm[gpu])
from cleanllm.gpu import score_ifd_jsonl
result = score_ifd_jsonl("data.jsonl", "scored.jsonl", device="cuda", low_threshold=0.3)

# Reward model scoring (requires pip install cleanllm[reward])
from cleanllm.gpu import score_rm_jsonl
result = score_rm_jsonl("data.jsonl", "scored.jsonl", device="cuda", low_threshold=0.0)

# Benchmark decontamination
from cleanllm.decontaminate import decontaminate_jsonl
result = decontaminate_jsonl("data.jsonl", "clean.jsonl", benchmarks=["mmlu", "gsm8k"])

# Framework pre-flight validation
from cleanllm.validate import preflight_jsonl
result = preflight_jsonl("data.jsonl", framework="trl")

# Verbosity scoring (CPU-only, no deps)
from cleanllm.verbosity import score_verbosity_jsonl
result = score_verbosity_jsonl("data.jsonl", "scored.jsonl")

# Repair conversation structure
from cleanllm.repair import repair_jsonl
result = repair_jsonl("data.jsonl", "fixed.jsonl", strip_sycophancy=True, strip_filler=True)

# PII detection and redaction
from cleanllm.pii import redact_pii_jsonl
result = redact_pii_jsonl("data.jsonl", "clean.jsonl", mode="redact")
result = redact_pii_jsonl("data.jsonl", "clean.jsonl", mode="flag")

# Score-weighted selection
from cleanllm.select import select_jsonl
result = select_jsonl("scored.jsonl", "selected.jsonl", top=5000, sort_by="_rm_score")
result = select_jsonl("scored.jsonl", "selected.jsonl", top="20%", score_expr="_ifd_score * _rm_score")

# Weighted dataset mixing
from cleanllm.mix import mix_jsonl
result = mix_jsonl(
    [("source1.jsonl", 0.6), ("source2.jsonl", 0.4)],
    "mixed.jsonl",
    total=10000,
    strategy="weighted",
    seed=42,
)
```

---

## Presets

| Preset | Description |
|---|---|
| `general` | URL removal + whitespace normalization, no domain-specific forbidden patterns |
| `security_scan` | Redacts secrets: AWS keys, GitHub tokens, API keys, private keys |
| `pii_scan` | Redacts PII: emails, US phone numbers, SSNs, credit cards, IPv4 addresses |
| `cpp17_clean` | URL removal + whitespace normalization + redact C++ portability issues |
| `cp_portable` | Strict CP portability — drops rows with forbidden patterns |
| `deterministic_only` | Drops rows with non-deterministic APIs (`rand()`, `random_device`, etc.) |

---

## Defaults

- **Required keys:** `id`, `messages`
- **Forbidden patterns (default):** none — use `--preset cpp17_clean` or `--preset cp_portable` for CP datasets
- **`empty_assistant` threshold:** 20 characters (responses shorter than this are flagged as empty)

> **CP datasets:** To apply competitive-programming forbidden patterns (`freopen`, `ifstream`, `bits/extc++.h`, etc.) use a preset: `cleanllm fix data.jsonl -o out.jsonl --preset cp_portable`. In Python, pass `forbidden_patterns=list(DEFAULT_FORBIDDEN_PATTERNS)` explicitly.

---

## Data format

cleanllm expects JSONL where each line is a JSON object. The default schema (`cp_sft_v1`) requires:

```json
{
  "id": "unique-id",
  "messages": [
    {"role": "system",    "content": "..."},
    {"role": "user",      "content": "..."},
    {"role": "assistant", "content": "..."}
  ]
}
```

Optional fields: `source`, `difficulty_bucket`, `problem_id`, `tests`.

---

## Development

```bash
pip install -e .[dev]
pytest
python -m build
twine check dist/*
```

See `RELEASE_CHECKLIST.md` for the full release workflow.

---

## License

MIT
