Metadata-Version: 2.4
Name: autocurate
Version: 0.3.0
Summary: Automatic data curation and curation auto-research: signals as zero-copy columns, recipes as views, agents that search recipes against a fixed eval.
License: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pyarrow>=14
Requires-Dist: lancedb>=0.40
Requires-Dist: pylance>=0.30
Provides-Extra: torch
Requires-Dist: torch>=2.2; extra == "torch"
Provides-Extra: text
Requires-Dist: transformers>=4.45; extra == "text"
Requires-Dist: tiktoken; extra == "text"
Requires-Dist: fasttext-numpy2; extra == "text"
Provides-Extra: vision
Requires-Dist: transformers>=4.45; extra == "vision"
Requires-Dist: open_clip_torch; extra == "vision"
Requires-Dist: pillow; extra == "vision"
Requires-Dist: webdataset; extra == "vision"
Provides-Extra: research
Requires-Dist: optuna; extra == "research"
Requires-Dist: anthropic; extra == "research"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: scikit-learn>=1.2; extra == "test"
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.25; extra == "docs"
Provides-Extra: all
Requires-Dist: autocurate[research,text,torch,vision]; extra == "all"
Dynamic: license-file

# autocurate

Curation is a compute multiplier: train on the right subset of your data and you reach the same quality with less
compute. autocurate does it in one call. Point it at a table of training data. It detects what kind of data it is,
scores every row with the signals that matter for that kind, applies the strongest established public filter, and
hands you the subset to train on as a view: read straight from the table without copying it, in a globally shuffled
order. Seven kinds of data are routed from the schema: web text, noisy labels, image-text pairs, SFT chats,
preference pairs, RL prompts and robot episodes. Nothing in a default path calls an LLM. Give it your evaluation and
it tunes the recipe; add `agent=True` and an LLM agent surveys the literature and writes recipes. A tuned recipe is
applied only if it beats the default on your held-out metric. This is research code, and APIs may change.

## Install

```bash
pip install -e ".[all]"     # torch, transformers, open_clip, optuna (TPE search), anthropic (opt-in agent)
pip install fasttext        # for the web-text default (DCLM); without it, text falls back to FineWeb-Edu and says so
```

## One verb, three levels

```python
import autocurate as ac

ds = ac.load("data/*.parquet", name="pool", uri="./db")      # parquet, pandas, pyarrow, hf://... into a Lance table

def train_eval(recipe, ids):                                 # your proxy training; higher is better
    model = train(ds.view(ids, columns=["text"]))
    return {"score": evaluate(model), "heldout": evaluate(model)}   # use a held-out set the search never sees
```

```python
cur = ac.curate(ds)                                          # 1. default: domain recipe, no LLM, no training
cur = ac.curate(ds, eval=train_eval, budget=8)               # 2. tuned: TPE on your eval, guarded by the default
cur = ac.curate(ds, eval=train_eval, budget=8, agent=True)   # 3. agent: literature signals + LLM recipes, guarded
print(cur, cur.report["level"], cur.report["guard"])         # what ran, what was picked and why
loader = cur.to_torch(columns=["text"], batch_size=256)      # zero-copy, globally shuffled batches
```

`ac.diagnose(ds)` runs pre-flight checks before you spend compute. Level 2 is TPE by default (no LLM); level 3 is
opt-in and the only path that calls an LLM (a logged-in `claude` CLI, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or the
`codex` CLI). Without `eval=`, `agent=True` only runs signal discovery and applies the default. Every signal is a
column, cataloged with its citations and provenance. Tables live on local disk or an object store (S3).

## Defaults

| kind of data | detected from | `ac.curate(ds)` applies | vs no curation |
|---|---|---|---|
| web text | one long text column | DCLM fastText top 10% + SemDeDup | 1.8× (GPT-2-small, 4B tokens) |
| image-text pairs | image + caption | DFN top 30% + SemDeDup | 3.3× (DataComp-medium) |
| noisy labels | a class label column | lowest k-NN label disagreement per class | 1.0-6.0×, median 2.8× (13 tasks) |
| SFT chats | `messages` or prompt/response | reward-model top 50% + prompt SemDeDup | 2.2× (Tulu-3 → Qwen2.5-0.5B) |
| preference pairs | prompt/chosen/rejected | top 75% by reward-model margin | better held-out z; no curve |
| RL prompts | problem/answer | learnability: 0 < pass rate < 1, ranked by p(1-p) | ≈1× |
| robot episodes | episode/action columns | lowest k-NN task-label disagreement per task | n.s. on small sims |

Each is a compute multiplier against training on the whole pool at matched quality, read off measured curves and
never extrapolated (gross; DataComp-medium is the 80.2M-pair pool). An agent-searched recipe scored above DFN top 30%
there on all 3 seeds (3.90×, p = 0.051). Net of curation compute, and against the strongest public filter, see
[results](docs/results.md#headline-results). Fallbacks when a model cannot run are logged in `cur.report`.

## Docs

`pip install -e ".[docs]" && mkdocs serve`, or read them in [docs/](docs/index.md):
[quickstart](docs/quickstart.md) · [which level do I need?](docs/which_level.md) ·
[the guard](docs/guard.md) · [guides](docs/guides/lm_pretraining.md) · [agent mode](docs/agent.md) ·
[API](docs/api/index.md) · [results](docs/results.md) · [benchmarks](docs/benchmarks.md) ·
[contributing](docs/contributing.md).

## License

Apache-2.0.
