Metadata-Version: 2.5
Name: llm-distillery
Version: 0.1.0
Summary: Capture LLM traffic through a drop-in proxy, curate it into a dataset, and distill a small model that replaces the frontier one.
Project-URL: Homepage, https://github.com/volition-network/distillery
Project-URL: Documentation, https://github.com/volition-network/distillery#readme
Project-URL: Issues, https://github.com/volition-network/distillery/issues
Author: Shubham Kanodia
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: anthropic,distillation,fine-tuning,llm,openai,proxy,volition
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: datasketch>=1.6
Requires-Dist: fastapi>=0.115
Requires-Dist: httpx>=0.27
Requires-Dist: jsonschema>=4.22
Requires-Dist: pydantic-settings>=2.4
Requires-Dist: pydantic>=2.7
Requires-Dist: python-dotenv>=1.0
Requires-Dist: python-ulid>=2.7
Requires-Dist: rich>=13.7
Requires-Dist: structlog>=24.1
Requires-Dist: tiktoken>=0.7
Requires-Dist: tomli-w>=1.0
Requires-Dist: typer>=0.12
Requires-Dist: uvicorn[standard]>=0.30
Provides-Extra: lang
Requires-Dist: lingua-language-detector>=2.0; extra == 'lang'
Provides-Extra: presidio
Requires-Dist: presidio-analyzer>=2.2; extra == 'presidio'
Requires-Dist: spacy>=3.7; extra == 'presidio'
Provides-Extra: train
Requires-Dist: accelerate>=0.33; extra == 'train'
Requires-Dist: datasets>=2.20; extra == 'train'
Requires-Dist: peft>=0.12; extra == 'train'
Requires-Dist: torch>=2.3; extra == 'train'
Requires-Dist: transformers>=4.44; extra == 'train'
Requires-Dist: trl>=0.11; extra == 'train'
Description-Content-Type: text/markdown

# distillery

Capture your LLM traffic through a drop-in proxy, curate it into a clean dataset, and distill a small model that replaces the frontier model for that workload.

```
app ──▶ distillery proxy ──▶ OpenAI / Anthropic
              │
              ▼ (captures)
        SQLite store ──▶ scrub ──▶ curate ──▶ dataset.jsonl ──▶ train ──▶ eval report
```

- **One env var to adopt.** Point `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` at the proxy. Streaming, tool calls, and vision inputs pass through byte for byte.
- **Safe by default.** PII scrubbing is on, credentials are never written to disk, and nothing leaves your machine unless you choose a hosted training backend.
- **Every stage is a command.** JSONL in, JSONL out. Use the proxy alone, the scrubber alone, or the whole pipeline.
- **Backend-agnostic training.** Train locally with LoRA on one GPU, or submit the same dataset to [Volition](https://volition.network) with a cost estimate up front.

## 60-second quickstart

```bash
uv tool install llm-distillery        # or: pip install llm-distillery
distillery init                   # pick an upstream, paste your keys once

distillery serve                                              # terminal 1
OPENAI_BASE_URL=http://localhost:8000/v1 python your_app.py   # terminal 2, unmodified app
distillery stats
```

Your app keeps working exactly as before. Each request is stored with its response, latency, token usage, and any tags you attach.

`init` writes your choices to `~/.distillery/distillery.toml` and stores keys in `~/.distillery/credentials.toml` with mode 600. You can also add or replace one key at a time:

```bash
distillery login openai          # prompts, hides input, verifies against the provider
distillery login volition --api-key sk-...
distillery login upstream        # generic key for OpenRouter, vLLM, or any OpenAI-compatible host
distillery logout anthropic
distillery config set proxy.upstream_openai https://openrouter.ai/api
distillery config show
```

Environment variables (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `VOLITION_API_KEY`, `MODEL_API_KEY`) and a `.env` file still work and take precedence over stored credentials.

## From traffic to a small model

```bash
distillery export --scrub --out data/conversations.jsonl
distillery curate --in data/conversations.jsonl --out data/ --val-fraction 0.1
distillery train --data data/ --backend volition --base-model qwen3-4b
distillery eval --run runs/<name> --judge gpt-4.1-mini
```

`export` turns captures into vendor-neutral conversations with PII replaced by typed placeholders. `curate` filters errors, truncations, refusals, and duplicates, then writes `train.jsonl`, `val.jsonl`, a `DATASET_CARD.md`, and a `curation_report.json`. `train` fine-tunes a LoRA adapter. `eval` runs the held-out set through the fine-tuned model **and the untouched base model**, computes metrics for both against the reference, optionally asks a judge model to compare outputs pairwise, and writes `eval_report.md`. The base-model column is what tells you whether fine-tuning did anything.

### A real run

[`examples/json_extraction`](examples/json_extraction) captures receipt-extraction traffic and distills it into a 4B model. This is the report from 300 captured requests to `gpt-4o-mini` (via OpenRouter), curated into 270 train / 30 val rows, fine-tuned on Volition (`qwen3-4b`, LoRA, one $2.00 job), and evaluated on the 30 held-out prompts with `gpt-4.1-mini` as the pairwise judge. The baseline column is the same `qwen3-4b` before fine-tuning:

> Candidate matches the reference on 73% of held-out prompts, is schema-conformant on 100%, wins or ties 97% of pairwise judgments (win rate 52%, 95% CI 47%-58%). Before fine-tuning the base model matched on 63%, was schema-conformant on 100%, won or tied 87% of judgments.

| Metric | gpt-4o-mini (reference) | qwen3-4b (base) | ft:qwen3-4b |
| --- | --- | --- | --- |
| JSON valid | 100% | 100% | 100% |
| Schema conformant | 100% | 100% | 100% |
| Exact JSON match with reference | – | 63% | 73% |
| Pairwise judge wins / ties / losses | – | 2 / 24 / 4 | 2 / 27 / 1 |

Training loss went from 1.13 to 0.055 over 340 steps and validation loss fell monotonically from 0.088 to 0.068, so the adapter learned the task rather than memorising it. The gain over the untuned model is real but modest here: the base model already produced valid, schema-conformant JSON, and 270 examples are only enough to nudge conventions such as whether `payment_method` keeps the card's last four digits and how `DD/MM/YYYY` dates are read. Thirty held-out cases cannot separate 63% from 73% with confidence; treat it as directional and rerun with more traffic.
## The proxy

| Endpoint | Behaviour |
|---|---|
| `POST /v1/chat/completions` | forwarded to OpenAI, captured |
| `POST /v1/messages` | forwarded to Anthropic, captured |
| any other `/v1/*` | forwarded, not captured |
| `POST /distillery/feedback` | attach a score, label, or corrected response to a capture |

Request headers the proxy understands and strips before forwarding:

| Header | Effect |
|---|---|
| `X-Distillery-Tag-<name>: value` | stored as a tag, filterable in `export` and `curate` |
| `X-Distillery-Session: id` | groups requests; sessions never straddle the train/val split |
| `X-Distillery-Capture: off` | skip capture for this request |

Every captured response carries an `X-Distillery-Capture-Id` header. Post feedback against it:

```bash
curl -X POST localhost:8000/distillery/feedback \
  -d '{"capture_id": "01J...", "score": 1.0, "label": "accepted"}'
```

Feedback drives the `feedback` filter and the `preference` dataset format for DPO.

## Train on Volition

Volition is a hosted fine-tuning service with an OpenAI-compatible API. distillery talks to it directly.

```bash
export VOLITION_API_KEY=sk-...        # or put VOLITION_KEY=sk-... in a .env file
distillery train --data data/ --backend volition --base-model qwen3-4b --wait
```

The backend uploads your dataset, waits for validation, shows the estimated cost and your balance, asks for confirmation (`--yes` to skip), submits the job with an idempotency key, and streams training events until it finishes. The resulting `ft:` model id is written to `runs/<name>/run.json`.

```bash
distillery eval --run runs/<name> --judge gpt-4.1-mini
```

For a Volition run, `eval` resolves the model id, warms up serving, and calls `https://api.volition.network/v1/chat/completions` for you. The same model works with the OpenAI SDK by swapping `base_url`.

Useful commands: `distillery volition models`, `distillery volition jobs`, `distillery volition job <id>`, `distillery volition cancel <id>`.

## Train locally

```bash
pip install 'llm-distillery[train]'
distillery train --data data/ --backend local --base-model Qwen/Qwen2.5-1.5B-Instruct
distillery train --data data/ --dry-run        # sequence-length stats, no training
```

Defaults target a single 24 GB GPU with a 1–3B base model: LoRA r=16, bf16, gradient checkpointing, cosine schedule. Presets are chosen from dataset size (`tiny` under 500 rows, `small` under 5k, `medium` above) and can be overridden with `--preset` or individual flags. Serve the adapter with vLLM, Ollama, or llama.cpp and pass `--candidate-url` to `eval`.

## Privacy

- Only the two chat endpoints are captured, and only their JSON bodies plus metadata.
- `Authorization`, `x-api-key`, organization headers, and cookies are never persisted. A test greps the database file for the key to prove it.
- Scrubbing covers emails, phones, Luhn-valid cards, SSNs, IPs, IBANs, API keys, and JWTs, inside tool-call arguments and system prompts too. `deny_if_found` drops whole conversations instead of masking.
- Optional NER for names and locations: `pip install 'llm-distillery[presidio]'`.

Read [`docs/privacy.md`](docs/privacy.md) for exactly what is and is not caught.

## Configuration

Settings load from defaults, then `distillery.toml` in the working directory or `~/.distillery/`, then `DISTILLERY_*` environment variables (nested with `__`, for example `DISTILLERY_PROXY__PORT=9000`), then CLI flags. A `.env` file in the working directory is loaded too. See [`distillery.example.toml`](distillery.example.toml).

Upstream credentials come from the client's own `Authorization` or `x-api-key` header. When the client sends none, the proxy falls back to `OPENAI_API_KEY` / `ANTHROPIC_API_KEY`, then to a generic `MODEL_API_KEY`. The same lookup serves `eval` when it calls the reference or judge model.

### Using OpenRouter or another OpenAI-compatible provider

```bash
distillery serve --upstream-openai https://openrouter.ai/api
distillery eval --run runs/<name> --judge openai/gpt-4.1-mini   # judge goes through the same upstream
```

## Development

```bash
uv sync
uv run pytest
uv run ruff check . && uv run mypy
```

Integration tests spin up the proxy against a fake upstream and drive it with the official `openai` and `anthropic` SDKs, including streaming and tool-call round trips. See [`CONTRIBUTING.md`](CONTRIBUTING.md).

## License

Apache-2.0
