Metadata-Version: 2.5
Name: touchstone-bench
Version: 0.1.3
Summary: Turn a company's running AI agent into a Harbor benchmark of its own work — capture, survey, review, train.
Author-email: Sam Larson <sam@pebbleml.com>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: fastapi>=0.110
Requires-Dist: httpx>=0.27
Requires-Dist: jsonschema>=4.21
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: tomli-w>=1.2.0
Requires-Dist: typer>=0.12
Requires-Dist: uvicorn>=0.29
Requires-Dist: websockets>=12
Provides-Extra: otel
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'otel'
Provides-Extra: whisper
Requires-Dist: faster-whisper>=1.0; extra == 'whisper'
Description-Content-Type: text/markdown

# Touchstone

A touchstone is the black stone assayers rubbed gold against to prove its purity before anyone paid
for it. Touchstone does that for AI agents: it turns a company's running agent into a
[Harbor](https://github.com/harbor-framework/harbor) benchmark of its own work, lets the people who
know the product review that benchmark by voice, and turns the results into training data.

Harbor is the middle of the product, unchanged. Touchstone is four extensions around it:

1. **Capture** — one line of code records what the production agent does.
2. **Survey** — a read-only agent reads the recordings and the code, copies the system into a Harbor
   environment, and writes Harbor tasks against it.
3. **Review** — a voice/text room where product people look at finished trials and correct the
   verifiers; Harbor regrades after every correction.
4. **Train** — turns finished jobs into distillation and RL datasets.

Everything Touchstone produces is a plain Harbor dataset the customer owns and can run without us.
Local-first: it runs on the customer's machine with their own Claude or Codex login — no Touchstone
account, no Touchstone servers. Open source. By Pebble ML.

Horizontal by design: nothing in `touchstone/harbor/` knows about any one customer, product, or
domain.

## The five commands

```
touchstone init                          # write touchstone.toml + the .touchstone/ db
touchstone demo                          # run the built-in agent and capture episodes (zero-key)
touchstone survey <repo>                 # read-only: map the code, simulate its services, score fidelity
touchstone bench -m <provider/model>     # run a model over the dataset; print pass rate per task
touchstone jobs                          # list Harbor job dirs with their pass rate per task
touchstone serve                         # the standalone local UI (overview, tasks, trials, train)
touchstone review                        # opens the browser and starts the voice review (server included)
touchstone train                         # turn finished jobs into distill + RL datasets
```

`touchstone survey` reads the recordings and the code with a read-only coding agent
(`[survey] provider`, default `claude-cli`; the customer's login pays), maps the tools and their
network boundaries, generates a SQLite-backed simulator per service, and replays every recorded call
through the real tools to score each simulator's fidelity. It is idempotent — outputs are reused
unless `--force` — and touches nothing outside `touchstone/` in the target repo.

`touchstone bench` also takes `--agent packaged|replica`, `--dataset <dir>`, and
`--against <job_dir>` (a per-task comparison against a previous run). `touchstone doctor` reports the
environment.

### Keys

Your agent under test uses its own provider key exactly as it always has — `OPENAI_API_KEY` or
`ANTHROPIC_API_KEY` in the environment you run it in; Touchstone reads nothing from it. Touchstone's
*own* OpenAI calls — the realtime voice in the review room, and an `openai:<model>` review provider —
read the same `OPENAI_API_KEY`, or a macOS Keychain item `<keychain_prefix>openai-api-key`. The
prefix defaults to `touchstone-` (so `touchstone-openai-api-key`); set `keychain_prefix` in
`touchstone.toml` to point at an item you already have. `touchstone doctor` prints which keys resolve.

### Capture

Add two lines to your own app — one to start recording, one to mark each conversation:

```python
import touchstone
touchstone.trace()                       # records every model + tool call to .touchstone/touchstone.db

with touchstone.episode("check-order-status"):   # one conversation = one episode
    reply = my_agent.run(user_message)
```

Then run your agent the way you normally do — a handful of real conversations is enough to start
(~10 gives the survey something to cluster), and more only sharpens it — and survey what was
captured:

```
touchstone survey .
```

`bench`, `gate` and `baseline` need a running Docker daemon — locally, or on a machine you name under
`[harbor]` in `touchstone.toml`.

Wrap any model call that is *not* the agent under test — a simulated user, a judge, an evaluator — in
`with touchstone.capture.paused():` so its model and tool spans are left unrecorded.

Run your real agent on a different model with one env var: set `TOUCHSTONE_MODEL` and capture rewrites
the `model=` keyword on every openai/anthropic/litellm call before it goes through — same SDK, same
code path. (A model passed positionally is left untouched.) This is how the packaged agent under test
is benched against a candidate model without editing the customer's code.

## The dataset

A survey produces a `touchstone/` directory in the customer's repo — a plain Harbor dataset they own
and version:

```
touchstone/
  dataset.toml      the Harbor dataset manifest (metadata; tasks run as the implicit tasks/ dataset)
  tasks/<name>/     one Harbor task per directory (instruction.md, task.toml, environment/,
                    solution/, tests/)
  environment/      the customer's system, copied to run in a sandbox
  agent/            the agent under test as a Harbor custom agent (agent.toml, tools.py, entry.py)
  simulators/       a small local service per network boundary
  baseline.json     what the current setup passes today (the first-five-minutes sentence)
  report.md         what was mapped, simulated, and left open
```

The agent under test runs in one of two modes, chosen automatically and recorded in `agent/agent.toml`:

- **packaged** (highest fidelity): the survey generates `agent/entry.py` — a `run(user_message)`
  that drives the customer's *real* agent loop for one message, importing their own modules. The
  Harbor agent runs `bash /app/agent/run.sh` in the sandbox with the model as a setting
  (`TOUCHSTONE_MODEL`) and imports the trajectory the customer's own capture wrote. An adapter check
  proves entry.py runs and calls a tool before this mode is chosen.
- **replica** (fallback when the code will not run): `touchstone.harbor.agent:TouchstoneAgent` runs
  an OpenAI-compatible tool-calling loop whose system prompt comes from `agent/agent.toml` and whose
  tool schemas + dispatch come from `agent/tools.py`.

Either way it records an ATIF `trajectory.json`, and `touchstone bench -m <candidate>` re-runs it on
any model without touching the customer's code.

## The review room

`touchstone review` is the whole thing: it starts the server if it isn't already running, opens the
browser on the room, and shows one big "▶ start" button — one click lets the browser use your
microphone, and from then on you just talk. The AI opens from the product's goal (the job labels and
the baseline pass count) and walks the finished trials in order — verifier unsure, models disagree,
never reviewed, then gate failures — reading each instruction, the trajectory in plain words, and
every criterion's score, and asking "do you agree it passed?". Agreement records trust (the share of
reviewed trials where the human agreed with the verifier, shown live). On a disagreement the agent
drafts a criterion change — edit/add/remove a rewardkit check, a dimension weight, a judge line, or
the instruction wording — reads it back, and on "yes" writes the `tests/` file and runs
`harbor job regrade` (the tasks are authored with a separate verifier so grading reruns from the
recorded artifacts, no agent), then reads out the new reward and any other trials that moved.
"Always" applies the same criterion to every task with the same job. Everything the room decides is
a row in `reviews` and a file change under `touchstone/` — nothing else.

## Training data

`touchstone train` reads the Harbor job directories (`trajectory.json` + `reward.json` per trial)
and writes, under `touchstone/train/`, `distill.jsonl` (full trajectories from trials that scored at
or above the threshold, for a student to copy), `rl_tasks.toml` (tasks in the learnability band —
pass rate strictly between 0 and 1 — with the verifier as the reward), and `manifest.json` (per-task
pass rate per model and where each task went). `touchstone train --teacher <spec>` runs the teacher
job first. Training itself stops at the exact GPU command; Touchstone produces the data, not the run.

## Storage

Only what is captured lives in SQLite (`.touchstone/touchstone.db`): `episodes`, `spans`, `rooms`,
`room_messages`, and `reviews` (what a review room decided about a task's trial). Harbor's own
`jobs/` directories are the run record — Touchstone reads them, it has no runs table.

## Running Harbor (remote Docker note)

Harbor bind-mounts local directories, so it must run where a Docker daemon lives. A laptop without a
daemon can offload to a host: set

```toml
[harbor]
host = "harbor-host.example"           # an SSH host with Docker + harbor
remote_root = "/srv/touchstone"
```

and `touchstone bench` (via `touchstone/harbor/run.py`) rsyncs the dataset to that host, runs Harbor
there over SSH, and rsyncs the job directory back. `scripts/harbor-mini.sh` is a thin wrapper for the
same path. With a local Docker daemon, everything runs locally and these settings are ignored.

## Install

```
uv tool install touchstone-bench
```

or, in a checkout, `uv run touchstone …`. Requires [Harbor](https://docs.harborframework.com)
(`uv tool install harbor`) and a Docker daemon to run benchmarks.

## Develop

```
uv run pytest -q
uv run ruff check .
```

Commits are small and sequential; see `docs/DESIGN.md` for the full v3 design.
