Metadata-Version: 2.5
Name: dropoutt
Version: 1.2.0
Summary: Pre-flight checks for LLM training data. Find what is wrong before you spend a training run finding out.
Project-URL: Homepage, https://dropoutt.com
Project-URL: Documentation, https://github.com/ckproduces/dropoutt-cli/tree/v1.2.0/docs
Project-URL: Source, https://github.com/ckproduces/dropoutt-cli
Project-URL: Issues, https://github.com/ckproduces/dropoutt-cli/issues
Project-URL: Changelog, https://github.com/ckproduces/dropoutt-cli/blob/main/CHANGELOG.md
Author-email: dropoutt <cagrokan@gmail.com>
Maintainer-email: dropoutt <cagrokan@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: contamination,data-quality,dataset,dataset-curation,deduplication,fine-tuning,llm,sft,tokenizer,training-data
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Natural Language :: Turkish
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: OS Independent
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.23
Requires-Dist: jinja2>=3.1
Requires-Dist: numpy>=1.24
Requires-Dist: orjson>=3.9
Requires-Dist: py3langid>=0.3
Requires-Dist: pyarrow>=14.0
Requires-Dist: rich>=13.0
Requires-Dist: scipy>=1.10
Requires-Dist: tokenizers>=0.15
Requires-Dist: tomli>=2.0; python_version < '3.11'
Requires-Dist: typer>=0.12
Requires-Dist: zstandard>=0.22
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: scikit-learn>=1.3; extra == 'dev'
Description-Content-Type: text/markdown

# dropoutt

**Pre-flight checks for LLM training data.** Point it at a folder. It tells you
what is wrong before you burn a training run finding out.

```bash
pip install dropoutt
dropoutt scan ./data
```

No model, no config, no flags required. Add `--model` or `--target` to unlock
more checks and CI gating. Skipped checks always name the one flag that unlocks
them.

## What it is

A local CLI that:

1. **Scans** training datasets for structural bugs (empty loss masks, broken
   roles, truncation that kills the answer, contamination, PII, language damage).
2. **Fingerprints** the corpus so two datasets can be compared without shipping
   records.
3. **Maps** your data onto a frozen **atlas** — a latent coordinate system built
   from professional public datasets — so you see what you cover, what you miss,
   and what sits off the map.

It runs on your CPU. One install brings everything — there are no extras to
choose between, and nothing in the dependency list needs a compiler.

## Install

Requires Python 3.10+.

```bash
python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install dropoutt
```

That is the whole install. Every dependency publishes a wheel for every Python
and platform dropoutt supports, so nothing is ever compiled at install time —
which is what a user on Windows and CPython 3.14 hit when the language-detection
dependency had no wheel and pip fell through to needing Visual C++ Build Tools.

There used to be extras (`[atlas]`, `[parquet]`, `[lid]`, `[all]`). They are
gone. Each one was a way to end up with a dropoutt that silently could not read
your Parquet, and `pip install 'dropoutt[all]'` still works — pip warns about
the unknown extra and installs everything, which is what you wanted anyway.

If `dropoutt` is not on `PATH` (module systems, batch schedulers, some Windows
setups): `python -m dropoutt` does the same thing.

Supported inputs: JSON, JSONL/NDJSON, TXT, Markdown, CSV/TSV, Parquet, Arrow,
Feather, ORC, `.mds` (MosaicML Streaming) and `.tar` (WebDataset). Text formats
may be gzip / bzip2 / xz / zstd compressed.

Works on macOS, Linux, and Windows. Cache defaults to `~/.cache/dropoutt`, or
`%LOCALAPPDATA%\dropoutt` on Windows. Override with `DROPOUTT_CACHE`. See
[docs/portability.md](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/portability.md) for offline / HPC use.

## Quick start

```bash
dropoutt scan ./my-corpus
# writes .dropoutt/{report.html, report.md, report.json, findings.jsonl, fingerprint.json}

dropoutt scan ./my-corpus --model qwen3 --seq-len 4096 --target sft
# unlocks token/mask checks and exit code 10 on blocking findings

dropoutt checks                 # live catalog
dropoutt checks T0-MASK-001     # one check in detail
dropoutt fetch                  # pre-download everything --offline needs
```

The whole report prints in the terminal, and the same content is written to
`report.html`, `report.md` and `report.json` — the page, the pasteable version
and the machine-readable one all say the same things. `--brief` prints the
verdict and one line per finding instead; `--quiet` prints nothing and writes
the files.

The report is one self-contained file: no CDN, no web fonts, no network, opens
from `file://`. A scan opens it for you when there is a desktop to open it on,
and quietly does not when there is not — over SSH, in CI, under a batch
scheduler, or with output redirected. `--no-open` or `DROPOUTT_OPEN=0` turns
that off; `DROPOUTT_OPEN=1` forces it, which is what you want with X11
forwarding.

Anything above 24 MB is scanned across processes, and the pool is sized against
the machine this process can actually use — CPU affinity, any cgroup quota,
hyperthread topology and free memory, not `os.cpu_count()`. Measured: 480,000
records across eight datasets, 377 MB, in 38 seconds on a 14-core laptop. The
result does not depend on how many cores you have: same findings, same examples,
same fingerprint id on one core or on sixteen. Cap it with `-j` or
`DROPOUTT_WORKERS` if you are sharing a node.

**New here?** [docs/getting-started.md](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/getting-started.md).

## What it catches

Bugs that waste a whole training run without appearing in the logs:

- **Records that train nothing** — empty loss masks from role-name mismatches
  (`from: "gpt"` vs `role: "assistant"`).
- **Truncation that removes the answer** — including cases where the entire
  assistant span falls beyond `--seq-len`.
- **Benchmark contamination** — Tülu 3 rule against bundled hashed 8-gram indices.
- **Files that are not training data** — agent session logs and telemetry that
  look like chat.
- **Directional overlap** — a small set wholly contained in a large one.
- **Language damage** — e.g. Turkish that lost its diacritics (`degil mi`).
- **Atlas shape** — specialised vs broad coverage, missing subject areas, and
  regions of near-identical writing that shingle dedup cannot see.

## Atlas (coverage map)

The atlas is a **frozen topical map** compressed from high-quality public
datasets (258 regions, static multilingual embeddings, CPU-only). Every scan
with the `atlas` extra places a sample of your records on that map and reports:

| Section | What you learn |
| --- | --- |
| **What the map says** | A handful of sentences that clear both a size gate and a significance gate — a subject 8x denser here than the map is built for, an area the map spends a fifth of itself on that you barely reach. Nothing is shown for being true; it is shown for being large *and* true |
| **Where your data piles up** | The five crowded places, named by *your own record* nearest the centre of each, because that is the only description of a neighbourhood that is true by construction |
| **Where you have only a toehold** | The sparsest places you reach. Reaching a place is not covering it, and an occupancy count cannot tell the difference |
| **Shape** | Specialised or broad — right for a single-task set, wrong for a pretraining mixture, and the tool does not know which you are building |
| **Crowding** | One area holding half the corpus whose records are 0.98 alike is one template, not one topic — and shingle dedup cannot see it |
| **Same ground** | Datasets that occupy the same regions even when they share no wording, i.e. merging them adds volume and not coverage |
| **Off the map** | Records unlike the reference geography, with a diagnosis (often length or markup, not “bad data”) |

The atlas's own five-word captions for a region are shown as captions and never
as findings: they are frequency counts over reference records, roughly 40% of
that text is function words shared with other regions, and the subject-area
names were assigned per source dataset rather than per record. What the map is
trusted for is geometry. Details and the full list of what that costs:
[docs/atlas.md](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/atlas.md).

## Exit codes

| Code | Meaning |
| --- | --- |
| 0 | Completed (findings or not) |
| 1 | Internal error |
| 2 | Usage error |
| 10 | Blocking findings — only when `--target` was declared |

## Check catalog

Identifiers are `T{tier}-{GROUP}-{nnn}` and are **never renumbered**. Mute by id
in `dropoutt.toml`. Full narrative: [docs/checks.md](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/checks.md). Live list:
`dropoutt checks`.

### Tier 0 — structural (CPU)

| id | What it means |
| --- | --- |
| `T0-SCHEMA-001` | Files are not training data (logs / telemetry) |
| `T0-SCHEMA-002` | One folder mixes several record layouts |
| `T0-SCHEMA-003` | Records failed to parse |
| `T0-SCHEMA-004` | Message content was not a string |
| `T0-SCHEMA-005` | Content sits in keys the layout never reads |
| `T0-FORMAT-001` | Plain-text files are holding structured records |
| `T0-GEN-001` | Generator scaffolding outside the records |
| `T0-REASON-001` | Only some responses carry a reasoning trace |
| `T0-TRUNC-002` | Responses stop at a generation length cap |
| `T0-QUAL-001` | Documents whose lines mostly lack punctuation (corpus) |
| `T0-QUAL-002` | Documents built mostly from very short lines (corpus) |
| `T0-QUAL-003` | Documents repeating their own lines (corpus) |
| `T0-ROLE-001` | Conversation role structure is invalid |
| `T0-ROLE-002` | Role names are not the canonical vocabulary |
| `T0-TMPL-001` | Data is already formatted with a chat template |
| `T0-TMPL-002` | Records fail to render with the target chat template |
| `T0-MASK-001` | Records contribute zero trainable tokens |
| `T0-MASK-002` | Stop token is outside the trainable span |
| `T0-TRUNC-001` | Records exceed the sequence length |
| `T0-PACK-001` | Packing efficiency under concat-and-chunk |
| `T0-ENC-001` | Text encoding is damaged |
| `T0-DUP-001` | Exact and whitespace-identical duplicates |
| `T0-DEGEN-001` | Degenerate responses |

### Tier 1 — statistical

| id | What it means |
| --- | --- |
| `T1-NDUP-001` | Near-duplicate records (MinHash; reports, does not delete) |
| `T1-DUP-002` | Same prompt answered two different ways |
| `T1-OVERLAP-001` | Datasets overlap with each other (directional) |
| `T1-ATLAS-001` | Corpus sits in very few topical regions |
| `T1-ATLAS-002` | A crowded region holds near-identical records |
| `T1-CONTAM-001` | Training data overlaps evaluation benchmarks |
| `T1-LANG-001` | Language composition and detection confidence |
| `T1-LANG-002` | Records deviate from the dataset’s main language |
| `T1-LANG-003` | Script does not match the detected language |
| `T1-PII-001` | Personal data and credentials in training text |
| `T1-IDENT-001` | Assistant identity leakage and refusal boilerplate |
| `T1-STYLE-001` | Formulaic response openings |
| `T1-LIC-001` | Datasets have no recorded licence |

Every finding in this release is labelled `unverified`: no calibration corpus
yet links acting on a finding to a measured change in model quality.

## Progressive disclosure

| What you give | What it unlocks |
| --- | --- |
| nothing | inventory, schema, dedup, overlap, bundled contamination, language, PII, atlas (if installed) |
| `--model` | exact tokens, fertility, truncation, template, loss mask, stop token, packing |
| `--target` | pass-or-fail gating (exit 10) |
| the `atlas` extra | where the corpus sits on the map, and what it misses |

## Documentation

- [Getting started](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/getting-started.md)
- [CLI reference](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/cli.md)
- [Check catalog](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/checks.md)
- [Fingerprint](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/fingerprint.md)
- [Atlas](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/atlas.md)
- [Configuration](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/configuration.md)
- [Portability / offline](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/portability.md)
- [Limitations](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/limitations.md)
- [Design rules](https://github.com/ckproduces/dropoutt-cli/blob/v1.2.0/docs/design.md)

## What it will not do

- Fail a run whose purpose you never declared (`--target`).
- Tell you to delete data it has not measured for downstream effect.
- Write raw PII values into reports (matches are masked).
- Extract text from PDFs — point it at extracted text instead.

## Develop

```bash
pip install -e '.[dev]'
pytest -q
ruff check .
python -m build && twine check dist/*
```

Lint rules live in `pyproject.toml`, and every rule that is switched off says
why. Two conventions are worth knowing before reading the source:

- **Imports go inside functions** wherever the import is expensive or optional.
  `dropoutt --help` should not pay for numpy, tokenizers and the atlas.
- **Every fast path has a slow one beside it.** `tests/test_fastpaths.py`
  checks the vectorised implementations against the obvious ones they replaced.
  The contamination hashes in particular are frozen — the shipped `.idx` files
  are tables of exactly those numbers, and no benchmark text exists anywhere to
  recompute them from.

## Licence

dropoutt is Apache-2.0.
