# Cureta

> Cureta is a Python framework for enriching text datasets at scale with a composable pipeline
> of feature-extraction *Taggers*. Install with `pip install "cureta[ray]"` (add `,llm` for the
> GPU model taggers). Requires Python >=3.10. This file is a concise, agent-oriented overview;
> each section links to the full docs. The complete docs inlined into one file: llms-full.txt.

Mental model: a **Document** (text + metadata + tags) flows through a **Pipeline** of
**Taggers**, each adding tags under its own namespace. CPU taggers run in `sdmi` mode (each data
shard passes through all CPU taggers, models kept warm); GPU taggers run in `simd` mode (the
model stays resident across the whole dataset). Built-in taggers are referenced by name; custom
taggers by a `tagger_class` dotted import path. Output is parquet or jsonl, optionally
checkpointed and resumable.

## Use it from Python
- Tag a single string: `from cureta import quick_tag` → `quick_tag(text, pipeline_config="p.yaml")`, or `quick_tag(text, taggers=[MyTagger()])`.
- Tag a dataset: `from cureta import run_pipeline` → `run_pipeline("p.yaml", dataset="data.jsonl", output_dir="out/", limit=1000)`.
- Also: `Pipeline` (construct/validate), `stream_tagged_batches` (iterate results), `read_manifest` (read run provenance). Full surface: [Python API](reference/python_api.md).

## Use it from the CLI
- `cureta run --pipeline NAME --dataset data.jsonl [--limit N] [--output-name v1] [--fresh|--resume]` — local run; resolves `pipelines/NAME/pipeline_config.yaml` in the cwd.
- `cureta launch --pipeline NAME` — provision a cluster, run, tear down (SkyPilot). Use `provision`/`submit`/`down` for a persistent cluster; `--keep` to leave it up.
- `cureta inspect PATH` — summarise a finished run's tags and manifest. Full reference: [CLI](reference/cli.md).

## Pipeline config (YAML)
```yaml
pipeline_name: "p"
dataset: "hf://datasets/org/name/"   # or a local .jsonl/.parquet path; or pass --dataset
tagger_stages:
  - tagger_name: "cureta_id"
  - tagger_name: "num_words"
    parameters: { lang: "en" }
  - tagger_name: "my_tagger"
    tagger_class: "my_module.MyTagger"   # required for custom (non-built-in) taggers
```
Other fields: `cluster`, `num_gpus`, `limit`, `output_name`, `output_format`,
`checkpoint_chunk_size`, `execution: {cpu_mode, gpu_mode}`. Full spec: [pipeline config](reference/pipeline_config.md).

## Taggers (45 built-in)
Reference any by its `tagger_name`. Categories: identity/statistics, quality
(`gopher_quality`, `c4_quality`, `fineweb_quality`, `perplexity`, `mmbert_quality`,
`fineweb_edu`, `fasttext_quality`), language ID (`indic_lid`, `glotlid`, `paragraph_langid`),
dedup (`exact_norm_hash`, `minhash`, `simhash`), safety (`pii_regex`, `pii_presidio`,
`detoxify`), structure, and metadata. Each tagger's output schema and parameters:
[Taggers reference](reference/taggers.md). Picking one: [Choose a tagger](how_to/choose_a_tagger.md).

## Write a custom tagger
Subclass `CPUTagger` or `GPUTagger`. Implement `process_document(self, document) -> Document`
(Tier 1, per-document) or `run(self, batch: pd.DataFrame) -> pd.DataFrame` (Tier 2, vectorised);
write tags with `document.add_tag(self.name, key, value)` and always `return document`. Reference
it via `tagger_class` with `PYTHONPATH=.`, or ship it as a package exposing a `cureta.taggers`
entry point. Guide: [Writing taggers](writing_taggers.md).

## Deploy on a cluster
Create `clusters/NAME.yaml` (cloud, `accelerators`, and a `setup:` block that installs cureta —
this is the single place installation happens), set `cluster: NAME` in the pipeline, then
`cureta launch --pipeline NAME`. Output goes to cluster persistent storage; checkpointed and
resumable (re-run to resume). Guide: [Deploying on cloud](tutorials/deploying_on_cloud.md);
[Configure SkyPilot](how_to/configure_skypilot.md).

## Optional
- [Execution model](concepts/execution_model.md): SIMD vs SDMI, the two-phase CPU/GPU model, Ray Data.
- [Tagger architecture](concepts/tagger_architecture.md) and [plugin system](concepts/plugin_system.md).
- [Resume a failed run](how_to/resume_a_failed_run.md) · [Tune chunk size](how_to/tune_chunk_size.md).
- [Full docs index](index.md).
