Metadata-Version: 2.4
Name: evaporator
Version: 0.2.0
Summary: Turn the predictable parts of LLM traffic into ordinary code, with an abstention boundary and a runtime guard.
License: Apache-2.0
Project-URL: Homepage, https://github.com/soummyaanon/evaporator
Project-URL: Repository, https://github.com/soummyaanon/evaporator
Project-URL: Issues, https://github.com/soummyaanon/evaporator/issues
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.12
Requires-Dist: rich>=13
Requires-Dist: textual>=8.2
Requires-Dist: anthropic>=1.0
Requires-Dist: openai>=1.0
Provides-Extra: anthropic
Provides-Extra: openai
Provides-Extra: presidio
Requires-Dist: presidio-analyzer>=2.2; extra == "presidio"
Provides-Extra: all
Requires-Dist: anthropic>=1.0; extra == "all"
Requires-Dist: openai>=1.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

<div align="center">

# 💧 Evaporator

### An AI application should need less AI over time.

Evaporator reads the logs of your high-volume LLM calls, writes ordinary code for the predictable part of that traffic, proves it on held-out data, and ships it to you as a pull request with a runtime guard. The model still handles everything the code isn't sure about.

[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-3776AB.svg?logo=python&logoColor=white)](pyproject.toml)
[![Generated code: Python 3.8+, zero deps](https://img.shields.io/badge/generated%20code-py3.8%2B%20%C2%B7%20zero%20deps-2ea44f.svg)](#-the-runtime-guard)
[![Status: pre-release](https://img.shields.io/badge/status-pre--release-orange.svg)](#-status)

**Routers pick a cheaper model. Caches reuse old answers. Distillers train a smaller model.**<br>
**Evaporator removes the model call.**

[Quickstart](#-quickstart) · [Connect](#-connect) · [How it works](#%EF%B8%8F-how-it-works) · [Safety model](#-safety-model) · [Runtime guard](#-the-runtime-guard) · [Control plane](#-control-plane) · [Coding agents](#-use-it-from-a-coding-agent) · [Status](#-status)

</div>

---

## ✨ The whole product in one interaction

```console
$ evaporator scan traces.jsonl --labels labels.csv
Loaded 20,000 calls from traces.jsonl across 1 call site(s); cost known for 100%, ground truth for 20,000.
Found 1 compilation candidate.

  classify_ticket()   70.1% validated offline

classify_ticket()
  Validated offline coverage         70.1%
  Held-out examples                  4,000
  Accuracy vs ground truth         100.00%
  LLM accuracy on same slice        98.61%
  Rare-class errors                      0
  Rules graduated                   4 of 6
  outcome preserved on held-out test

$ evaporator compile classify_ticket --target app/tickets.py:classify_ticket
✓ generated implementation      evaporated/classify_ticket.py
✓ generated abstention boundary evaporated/classify_ticket.envelope.json
✓ generated tests               tests/evaporated/test_classify_ticket.py (2,094 cases)
✓ 20,000 historical cases replayed: 14,052 answered by code (100.00% correct), 5,948 left to the LLM, 0 rare-class errors
✓ generated tests: 6 passed
✓ GitHub PR opened
```

<sub>Output from the bundled synthetic support-ticket dataset. Your numbers depend on your traffic, and finding them is the point of running <code>scan</code>.</sub>

What you get is a **pull request**: ordinary source code you review, own and merge. The diff at your call site is one line:

```python
from evaporated.classify_ticket import evaporate

@evaporate                      # 70% of calls answered by code, the rest go to the LLM exactly as before
def classify_ticket(text):
    return call_llm(text)
```

---

## 🧠 The core idea: compile + abstain

The code doesn't need to know everything. It needs to know two things very well: **what it can answer**, and **when it must abstain**. The second is the hard part, and it's what Evaporator is built around.

```mermaid
flowchart LR
    A[Input] --> B{Inside the validated<br/>envelope?}
    B -- yes --> C[Compiled code]
    B -- no --> D[Original LLM call]
    C --> E[Result]
    D --> E
    C -. 1% shadow sample .-> D
```

- **Partial coverage is success.** "68% runs as code, 32% still goes to the model" is the expected outcome, and abstaining isn't a failure.
- **Measure the real outcome, not agreement.** Code that copies the LLM's mistakes scores 99.99% agreement and proves nothing. When you supply ground truth, it's the bar. When you don't, the report says so, every time.
- **Report only what the evidence supports.** Rare or thinly observed regions stay on the LLM.
- **You own the code.** It's plain Python in your repo, with no black-box runtime.

---

## 🚀 Quickstart

```bash
uvx evaporator                  # try it without installing (needs uv: https://docs.astral.sh/uv/)
```

To keep it installed, pick one:

```bash
uv tool install evaporator      # recommended
pipx install evaporator
pip install evaporator          # into an existing Python 3.11+ environment
uv tool install "git+https://github.com/soummyaanon/evaporator"   # latest unreleased main
```

Run `evaporator` in your project folder. It opens a chat with the prompt ready, finds your call logs and your AI connection by itself and says in one line what it found. Then type:

```text
which of my AI calls can become code?
```

You get each call site with its calls, its spend and a plain label: *looks like it can become code (not proven yet)*, *stays on the AI (open-ended)* or *too few calls*. The chat then offers to prove the top candidate on held-out calls, naming the model it will use. A scan, a compile and a pull request each run only after you say yes.

Slash commands in the same prompt:

| Command | |
|---|---|
| `/model` | Change the AI connection or model |
| `/logs` | Pick which call logs to use, or type a path |
| `/agent` | Open Claude Code, Codex or fx full-screen with Evaporator's tools |
| `/dashboard` | Leave the chat for the call-site dashboard |
| `/help` | List these commands |
| `/quit` | Leave (ctrl+q works too) |

Prefer the commands, or working from a clone?

```bash
git clone https://github.com/soummyaanon/evaporator && cd evaporator
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[all]'

evaporator setup                                           # connect a model once (see Connect below)
evaporator scan your-export.jsonl --list                   # profile call sites; no model calls, no cost
evaporator scan your-export.jsonl --labels truth.csv       # synthesize + validate
evaporator compile <callsite> --target app/file.py:func    # branch, tests, PR
```

`evaporator setup` reruns the connection steps without opening the app. Without a terminal it prints how to configure through the environment instead.

In a terminal, `scan` and `report` draw the call site as a small isometric evaporator: the tank boils down to the share that still needs the LLM, and the steam that rises turns into code. Pipes and CI get plain text; set `EVAPORATOR_PLAIN=1` to force it.

**Supported log formats:** OpenAI and Anthropic request/response logs, Langfuse exports, Evaporator's native JSONL, and any JSONL or CSV through a small TOML field mapping:

```toml
[fields]
id = "request_id"
timestamp = "created_at"
input = "payload.text"
output = "result.label"
ground_truth = "reviewed_label"
```

---

## 🔌 Connect

Synthesis needs a model. The chat picks one automatically: the one you chose before, else a signed-in Claude Code, else a signed-in Codex, else a saved API key, else `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`. The header always names the connection in use; `/model` changes it (as does `evaporator setup`). If nothing is found, the chat asks you to connect one:

- **Claude Code.** Uses the `claude` you already signed in to. Evaporator runs the binary and never reads its token store.
- **Codex.** The same, through your `codex` binary.
- **API key.** Anthropic, OpenAI, or any OpenAI-compatible gateway.

Subscription usage counts against your Claude or ChatGPT plan. Evaporator writes no chat transcript, but when the chat runs on Claude Code or Codex those tools keep their own session history, as they do for any use.

**Key precedence:** `--api-key-env` / `EVAPORATOR_API_KEY_ENV` beats a saved credential, which beats `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`. Provider and model: flag, then `EVAPORATOR_*` environment, then your saved config, then the default.

**Where things live:**

| What | Where |
|---|---|
| Your provider and model | `~/.config/evaporator/config.toml` (honours `XDG_CONFIG_HOME`) |
| A saved API key | `~/.config/evaporator/credentials`, mode `600` |
| Per-project logs, labels and flags | `<project>/.evaporator/project.toml` (the project is the git root of where you run it) |

A key is never printed, logged or written anywhere else.

**Personal data.** Before the first scan in a project, and before Claude Code or Codex can otherwise reach it (Codex's first chat turn, `/agent`, `/dashboard`), the chat asks once whether the data contains patient or personal data, and saves the answer as `sensitive` in `project.toml`. Subscriptions are not BAA-covered, so on a sensitive project the chat warns once before it uses `claude-code` or `codex` ("Use anyway" or "Connect an API key"). "Use anyway" is saved as `allow_subscription = true` in `project.toml` and the header then shows "personal data · not BAA-covered"; "Connect an API key" ends that conversation and asks for a key on a BAA-covered endpoint. Until the warning is accepted (or you set `allow_subscription = true` yourself), the CLI, MCP and the dashboard refuse subscriptions on that project. (Changed on 2026-10-18, decided by the user: a sensitive project used to refuse subscriptions outright.)

**Nothing runs without your yes.** A scan runs only after you agree to it in the chat, with the model named. Writing a compile into a repository needs a yes naming the repository, and pushing or opening a pull request needs a second, separate yes.

---

## ⚙️ How it works

```mermaid
flowchart TD
    A[Exported LLM logs] --> B[Ingest + group by call site]
    B --> C{Bounded output?}
    C -- open-ended --> L[Stays on the LLM]
    C -- labels or JSON fields --> D[Time-ordered split<br/>60 / 20 / 20]
    D --> E[Synthesize rules<br/>from de-identified examples]
    E --> F[Graduate each rule<br/>on validation evidence]
    F -->|refine| E
    F --> G[Score once on<br/>untouched test split]
    G --> H[Compile: code + envelope<br/>+ guard + tests]
    H --> I[Pull request]
    I --> J[Runtime guard<br/>shadow · drift · revert]
    J --> K[Control plane]
```

| Step | What happens |
|---|---|
| **Ingest** | Groups calls by an explicit call-site tag or a fingerprint of the system prompt, and strips any prompt template every input shares. Cost comes from the logs or from token counts. |
| **Profile** | **Classification and routing** (at most 50 labels) and **extraction** (JSON with at most 30 fields) are candidates. Open-ended generation is out of scope. |
| **Synthesize** | Claude (default) or any OpenAI-compatible endpoint writes named `@rule("label")` functions or `@field("name")` extractors, refined over rounds of validation feedback. |
| **Graduate** | Each rule earns its place on its own: enough support, accuracy at the bar, a confidence lower bound, and zero rare-class errors. Rules that fail are dropped, and the rest are re-scored until the set is stable. |
| **Score once** | The surviving program is scored a single time on the time-ordered test split it has never seen. |
| **Compile** | Writes the code, envelope, a copy of the guard, and a replay test suite on a new branch through a temporary git worktree, so your checkout is never touched. It then opens the PR with the full evidence. |

---

## 🛡 Safety model

| Layer | Protects against |
|---|---|
| **Per-rule graduation.** At least `--min-support` calls, accuracy ≥ `--bar` (default 99.5%) or ≥ the LLM's own accuracy on that slice, and a 95% Wilson lower bound ≥ `--min-lower`. | Rules that look good on a handful of examples |
| **Rare-class rule.** Any error involving a label under 1% of traffic disqualifies the rule. | Fraud, escalations and rejections silently misrouted |
| **Out-of-distribution envelope.** Vocabulary, length and shape must match what was seen in training. | New input types, languages or document formats |
| **All-or-nothing extraction.** A call runs as code only if every field has a graduated answer. | Half-filled records |
| **Sandboxed synthesis.** Static validation, restricted builtins, an import allowlist, CPU/memory/file-descriptor limits, and an OS sandbox with no network or file writes. | LLM-written code doing anything but computing |
| **De-identification.** Regex covering names, DOB, MRN, IDs, contact details, addresses and dates, plus field-aware redaction and optional Presidio NER. Placeholders stay consistent across input and output. | PHI/PII reaching the synthesis model or your repo |
| **Runtime guard.** Shadow sampling, self-revert, drift alerts, and an instant revert switch. | Degradation after merge |

---

## 🪝 The runtime guard

The guard is stdlib-only and gets copied into your repo inside the PR, so there's nothing new to install.

- **Sync and `async def`**, including methods: `@evaporate(input_arg=1)`, with custom input extraction and output mapping.
- **Shadow sampling.** 1% of compiled answers are re-checked against the LLM in the background. Too many mismatches and the guard reverts itself.
- **Revert without a deploy.** `EVAPORATOR_REVERT=classify_ticket` (or `*`), a revert file, or one command through the control plane.
- **Drift detection.** It warns when the share of out-of-envelope inputs rises above what was seen offline.

---

## 📡 Control plane

```bash
evaporator serve --token "$TOKEN" --slack-webhook "$SLACK_URL"
export EVAPORATOR_TELEMETRY_URL=http://control:8787/v1/telemetry EVAPORATOR_TELEMETRY_TOKEN="$TOKEN"

evaporator status                                     # fleet view
evaporator revert classify_ticket --reason "incident" # every instance, at its next check-in
evaporator resume classify_ticket
```

```text
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┓
┃ call site       ┃ instances ┃ calls ┃ compiled ┃ shadow agree ┃ outside envelope ┃ saved ┃ state ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━┩
│ classify_ticket │ 1/1       │ 100   │ 60.0%    │ n/a          │ 20.0%            │ $0.05 │ ok    │
└─────────────────┴───────────┴───────┴──────────┴──────────────┴──────────────────┴───────┴───────┘
```

It's a stdlib HTTP server with SQLite storage.

- **Aggregation.** Combines telemetry from every instance and ignores out-of-order snapshots.
- **Alerts.** Sends drift and revert alerts to Slack or email, at most once an hour per call site and kind.
- **Auto-revert.** Reverts a call site fleet-wide when combined shadow mismatch since the last resume crosses the envelope's threshold.
- **New instances.** A new instance picks up an active revert at startup.

---

## 🤖 Use it from a coding agent

`evaporator mcp` is a stdio MCP server, so [fx](https://fx.sh), Claude Code, Cursor and other agents can ask Evaporator to do the work instead of hand-writing an unproven classifier. The agent gets four tools:

| Tool | What it does | Cost |
|---|---|---|
| `list_callsites` | Groups the logs by call site and says which ones can be compiled | Free, no model calls |
| `scan` | Synthesizes rules, graduates them and scores them once on held-out calls | Calls your synthesis model |
| `report` | Shows the evidence per rule and why rules were dropped | Free |
| `compile` | Writes code, envelope, guard and tests on a new branch; opens a PR only with `pr: true` | Free |

**Bring your own model.** Synthesis runs on your key, chosen when you start the server, never by the agent:

```bash
# Anthropic (default)
export ANTHROPIC_API_KEY=...

# OpenAI, or any OpenAI-compatible gateway
export OPENAI_API_KEY=...
export EVAPORATOR_PROVIDER=openai EVAPORATOR_MODEL=<model>   # EVAPORATOR_BASE_URL=<gateway> optional
```

The same `EVAPORATOR_*` variables work for `evaporator scan`. The agent can't change the model, the endpoint, the thresholds or de-identification, and it can't score its own rules against the test split. Those switches exist only on the CLI.

**fx:** add the server and the skill:

```text
/mcp add evaporator evaporator mcp
/skills install soummyaanon/evaporator --skill evaporator
```

A scan takes minutes, longer than fx's default 60-second MCP timeout, so raise it in `~/.fx/mcp.json`:

```json
{
  "mcp": {
    "evaporator": {
      "type": "local",
      "command": ["evaporator", "mcp"],
      "environment": { "EVAPORATOR_PROVIDER": "openai", "EVAPORATOR_MODEL": "<model>" },
      "operation_timeout_ms": 1800000
    }
  }
}
```

**Claude Code:** `claude mcp add evaporator -- evaporator mcp`, and copy `skills/evaporator` into `.claude/skills/`.

Then ask: *"Find the LLM calls in logs/traces.jsonl that should be code."*

---

## 🧭 Where it fits

| | What it does | After adopting it |
|---|---|---|
| Routers and gateways | Pick a cheaper model | Still inference |
| Semantic caches | Reuse answers to near-duplicate queries | Still inference on anything new |
| Distillation | Train a smaller specialist model | Still inference |
| **Evaporator** | **Replace a measured share of calls with reviewed code** | **Zero inference on that share** |

**In scope:** classification, routing, normalization, validation and structured extraction, meaning functions where a correct output can be stated and checked mechanically.<br>
**Out of scope:** open-ended generation, conversational replies, multi-step agents, and judgment-heavy decisions.

---

## 🔐 Configuration reference

<details>
<summary><b>scan</b></summary>

| Flag | Default | |
|---|---|---|
| `--labels FILE` | none | CSV or JSONL with `id,label` ground truth |
| `--list` | off | Profile call sites only; no model calls |
| `--program rules.py` | none | Evaluate an existing rules file; no model calls |
| `--provider` / `--model` / `--base-url` | `anthropic` / `claude-opus-5` | Any OpenAI-compatible endpoint, gateway or BAA deployment |
| `--bar` / `--min-support` / `--min-lower` | `0.995` / `30` / `0.95` | Graduation criteria |
| `--rare-share` | `0.01` | Labels under this share count as rare |
| `--deid-backend` | `regex` | `presidio` for NER: `pip install -e '.[presidio]' && python -m spacy download en_core_web_lg` |
| `--no-deid` | off | Send raw inputs to the synthesis model |

</details>

<details>
<summary><b>compile</b></summary>

| Flag | Default | |
|---|---|---|
| `--target file.py:func` | none | Add `@evaporate` to the call site (functions, methods, async) |
| `--pr / --no-pr` | `--pr` | Push the branch and open a GitHub PR through `gh` |
| `--no-commit` | off | Write files into the working tree only |
| `--include-abstained` | off | Also write inputs the code abstains on into the repo test file |
| `--raw-cases` | off | Skip redaction of the repo test cases |
| `--force` | off | Compile even if the held-out test failed |

</details>

<details>
<summary><b>Runtime environment variables</b></summary>

| Variable | |
|---|---|
| `EVAPORATOR_REVERT` | Comma-separated call sites (or `*`) to send back to the LLM |
| `EVAPORATOR_REVERT_FILE` | File listing call sites to revert |
| `EVAPORATOR_SHADOW_RATE` | Override the shadow sampling rate (default `0.01`) |
| `EVAPORATOR_TELEMETRY_URL` / `_TOKEN` / `_INTERVAL` | Control-plane reporting |

</details>

---

## 📍 Status

**Pre-release.** The CLI, guard and control plane are built and covered by 59 tests, including end-to-end scan → compile → run.

It **has not yet been validated on real production traffic**. The number that decides whether this is worth doing is *what share of real inference dollars can be removed without degrading the measured outcome*, and producing it is the first milestone:

- [x] Scan, compile, PR, runtime guard, control plane
- [x] Classification, routing and structured extraction
- [x] Sandboxed synthesis and PHI de-identification
- [ ] Phase 0: measured results on three real call sites
- [ ] First design-partner PR merged, savings verified over 30 days
- [ ] TypeScript guard, GitLab/Bitbucket

---

## 🛠 Development

```bash
pip install -e '.[all,dev]'
pytest
```

<div align="center">
<sub>Licensed under <a href="LICENSE">Apache-2.0</a>.</sub>
</div>
