Metadata-Version: 2.5
Name: agent-crucible
Version: 0.4.2
Summary: Reproducible testing, benchmarking, and regression testing for Agent Skills.
License: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# Agent Crucible

Reproducible A/B testing for Agent Skills.

A `SKILL.md` is a prompt you ship to other people. This tool answers the only
question that matters about one: **does it actually make the agent better, or
does it just feel like it does?**

It runs the same task twice — once with a baseline agent, once with the skill
installed — repeats that N times, checks the results with deterministic
assertions, and reports the difference with a confidence interval attached.

**Real results are in [BENCHMARK.md](BENCHMARK.md)** — including a skill that only
becomes conclusive at N=10, two skills with no measurable effect, and an honest
surprise about what trigger detection can and cannot see.

```
                    Without       With
Task success            50%       100%
Assertion score        0.83       1.00
Tokens                8,700      7,100
Time                     55s        41s

IMPACT   (95% bootstrap CI on the delta)

  Task success        +50.0pp   [+20.0, +80.0]   p=0.033
~ Tokens              -23.0%    [-41.2%, +2.1%]

CONFIDENCE  MEDIUM  (N=5 per condition)
~ marks a delta not distinguishable from run-to-run noise.

PER TASK  (success rate)

task-001                  0%       100%
task-002                100%       100%
```

The per-task breakdown matters: an aggregate `+50pp` driven entirely by one
task is a different finding from one spread across all of them.

## Install

```bash
pip install agent-crucible
```

Or from a clone: `pip install -e .`

Requires Python 3.10+, `git`, and an authenticated [Claude Code](https://claude.com/claude-code)
CLI on `PATH`. Tested on Linux, macOS, and Windows (CI runs the suite on all
three across Python 3.10–3.12). Verify the agent side works before benchmarking
anything:

```bash
claude -p "reply with OK" --output-format json
```

If that returns `"Failed to authenticate"`, log in first — every run in this tool
shells out to that same command.

## Use

```bash
crucible init my-skill        # scaffold SKILL.md + evals/
crucible validate my-skill    # static checks, no tokens spent
crucible test my-skill        # the A/B benchmark
crucible report my-skill      # re-render the last run
crucible regression my-skill  # did the latest run drop below the baseline?
```

### 1. Validate

Runs before every benchmark, and refuses to spend tokens on a broken suite.

```
  [ok]   SKILL.md exists
  [ok]   valid YAML frontmatter
  [ok]   frontmatter has 'name'
  [ok]   frontmatter has 'description'
  [ok]   referenced files exist (2/2)
  [ok]   [task-001] prompt is customised, not the scaffold placeholder
  [warn] model 'sonnet' is a pinned version (aliases drift between releases)
```

### 2. Write eval tasks

```yaml
id: task-001
name: Make calc errors typed

prompt: |
  The test suite in tests/ fails. Fix src/calc.py so every test passes.
  Do not modify anything under tests/.

fixture: fixtures/task-001

assertions:
  - type: command          # did it solve the task?
    run: "python -m pytest -q"
    expect_exit: 0

  - type: file_contains    # did it follow the convention the skill teaches?
    path: src/calc.py
    pattern: "raise CalcError\\([^)]*code="

  - type: file_unchanged   # anti-gaming: editing the tests does not count
    path: tests/test_calc.py
```

Assertion types: `command`, `file_exists`, `file_absent`, `file_contains`,
`file_changed`, `file_unchanged`.

Before burning tokens, check two things — neither spends a cent:

```bash
crucible test my-skill --dry-run    # can the suite even measure work?
crucible test my-skill --estimate   # what will a real run cost?
```

`--dry-run` runs the assertions against the untouched fixture. Anything that
passes there is flagged `[TRIVIAL]` — it would pass even if the agent did
nothing. (Assertions that check for the *absence* of something — `file_absent`,
`file_unchanged`, or a `file_contains` with `expect: false` — are guards, and
correctly pass on the pristine fixture.)

`--estimate` projects cost and wall time as a range. With no history it is a
coarse prior; after one real run it uses your own recorded per-run cost. A real
`test` prints the estimate first, then runs.

### 3. Benchmark

```bash
crucible test my-skill -n 10
```

## How it works

Per run:

1. The task fixture is copied into a throwaway temp directory.
2. That directory is `git init`-ed and committed, which is what makes
   `file_changed` / `file_unchanged` assertions possible.
3. In the **with-skill** condition only, the skill is copied to
   `.claude/skills/<name>/` inside that workspace. The `evals/` directory is
   excluded — leaking it would hand the agent its own grading criteria.
4. `claude -p "<prompt>" --output-format json` runs with the workspace as cwd,
   with `--setting-sources project`. (Note: in practice this does not fully hide
   your personal `~/.claude` skills — see "Known gap" above. They appear in both
   conditions, so the delta stays attributable to the skill under test.)
5. Assertions run against the resulting workspace; tokens, cost and wall time
   come from the agent's own JSON result.
6. The workspace is deleted (`--keep-workspace` to inspect it).

Results land in `<skill>/.crucible/runs/*.json`, with one summary line per
invocation appended to `<skill>/.crucible/history.jsonl`.

## Design decisions worth knowing

**No LLM judge.** Scoring is deterministic: assertions pass or they don't. An
LLM grader adds a second noisy component on top of an already noisy agent, and
you can no longer tell which one moved. Quality rubrics are a later addition,
not the foundation.

**Every delta carries an uncertainty estimate.** Agent runs are
non-deterministic; the same prompt gives different token counts and sometimes
different outcomes. A single number invites false regression alarms. Anything
not separable from noise is printed with `~` and called inconclusive.

**Task success is judged by Fisher's exact test, not by the bootstrap.** This
was not the original design — it was forced by testing the tool against a
deliberately noisy stub agent with a weak 60%-vs-50% effect. The percentile
bootstrap called that a *conclusive regression* on the first attempt: five
Bernoulli samples per condition is exactly the regime where it is
anti-conservative. Fisher's exact test on the 2×2 outcome table has no such
problem at small N, and still detects a real effect at N=5 (0/5 vs 5/5 gives
p=0.008). Continuous metrics keep the bootstrap interval, at 95% to match the
same 0.05 threshold.

**Secondary metrics are not corrected for multiple comparisons.** Five metrics
per report at a 5% threshold means roughly one report in seven contains a
secondary claim that is not real. The report says so out loud when it makes one
below N=10, rather than letting you believe all five numbers equally.

**Confidence is labelled by sample size.** N<5 is `LOW` and says so in the
verdict. The tool will not pretend five runs are evidence — at N=1 it reports
the observed delta and then explicitly refuses to conclude from it.

**Pin your model.** `model: sonnet` is an alias that moves between releases. A
comparison against a run from last month is meaningless if the model changed
underneath it. `validate` warns about this.

**The baseline carries Claude Code's bundled skills.** They appear in both
conditions, so the delta stays attributable to the skill under test, and they
are identical across installs of the same version. `isolation: bare` removes
them for a clean-room baseline (needs `ANTHROPIC_API_KEY`).

**This is decision support, not an oracle.** `--strict` exists for CI, but
consider making the CI job a warning rather than a hard block, and gate it on
`SKILL.md` actually having changed — every invocation spends real tokens.
`N tasks × N repeats × 2 conditions` agent runs is not free the way a unit test
is free.

## Examples

Complete, runnable skills live in `examples/`, each mixing a "did it work"
command assertion with a "did it the right way" content assertion, and each
carrying trigger cases:

| Skill | Domain | The convention it checks |
|---|---|---|
| `error-handling` | code style | raises carry a machine-readable `code=` |
| `sql-parameterization` | security | queries use placeholders, never f-strings |
| `type-hints` | typing | functions get annotated params + return type |
| `negative-control` | — | **deliberately irrelevant** to task success |

```bash
crucible validate examples/sql-parameterization
crucible test examples/sql-parameterization --dry-run
crucible test examples/sql-parameterization --estimate
crucible test examples/sql-parameterization -n 5
```

Each positive example is built so the difference is visible: a baseline agent
usually makes the tests pass, but only the skilled agent reliably follows the
convention. "Tests pass" and "did it the right way" are different questions —
which is exactly why a single assertion is not enough.

`negative-control` is the opposite on purpose: its skill only changes docstring
wording, so it *cannot* affect whether the tests pass. Running it should report
**no conclusive effect** — a measurement instrument you can trust reads zero when
the effect is zero.

## CI

`examples/ci/skill-eval.yml` is a GitHub Actions template. It triggers only when
`SKILL.md` or `evals/` changed, and posts the report as a PR comment instead of
failing the build.

## Validated against a live agent

Run end-to-end against the real `claude` CLI, not just a stub. Two things the
example run demonstrated, both worth internalising before you trust any number:

- **A lucky small sample lies, and the tool catches it.** At N=3 the example
  skill scored a clean 3/3 vs 0/3 — an apparent +100pp. At N=5 the same skill,
  same prompt, scored 1/5 (+20pp, p=1.000). The effect is real but weak and
  inconsistent; three runs happened to catch a good streak. A tool that reports
  one number would have published "+100% success". This one reported "not
  distinguishable from noise" and asked for more runs. That is the whole point.

- **Availability is not invocation.** A skill in `.claude/skills/` is *discovered*
  by the agent but only *used* when its description matches the task. The prompt
  "make the tests pass" did not trigger an error-handling skill; "add proper
  error handling following this project's conventions" did. Your eval prompt is
  a real variable — the same skill looks useless or useful depending on it.

Baseline isolation, found in the same run: the headless agent always sees the
skills that ship with Claude Code (design, code-review, …). They are **not** your
personal `~/.claude` skills — they are bundled with the install, so they are the
same for anyone on the same Claude Code version, and they appear in *both*
conditions, cancelling out of the delta. For a true clean-room baseline set
`isolation: bare` in `evals.yaml` (runs the agent with `--bare`); it needs
`ANTHROPIC_API_KEY`, since bare mode never reads your OAuth login. The default
keeps the bundled skills and works with OAuth.

### 4. Regression

Each `test` writes a run record. `regression` compares the newest run against an
earlier baseline — by default the most recent run whose skill content differs, so
it answers "did my last edit make the skill worse?"

```bash
crucible regression my-skill
```

```
With-skill success:  86%  ->  62%   (-24pp, p=0.021)
--------------------------------------------------------
x  REGRESSION — with-skill success dropped beyond run-to-run noise.
```

Crucially, it is judged by the same Fisher exact test as the benchmark, not by
subtracting two scores. A drop from 100% to 80% on five runs prints
`~ PASS (inconclusive)` with `p=1.000`, because one flipped run is exactly what
noise produces — a naive point-vs-point check would have raised a false alarm.
It exits `1` **only** on a conclusive drop, so a CI gate blocks a genuine
regression without failing on the dice:

```yaml
- run: crucible test ./my-skill -n 10
- run: crucible regression ./my-skill   # exit 1 blocks the merge
```

### 5. Trigger evaluation

A skill can be excellent when used and still be worthless because the agent never
reaches for it — or noisy because it fires on unrelated prompts. `trigger`
measures that directly. In `evals/triggers.yaml` you label prompts:

```yaml
cases:
  - prompt: "Add error handling to src/calc.py."
    should_trigger: true
  - prompt: "What is 27 times 34?"
    should_trigger: false
```

```bash
crucible trigger my-skill
```

```
  Precision (when it fired, was it right): 100%   FP=0
  Recall (of prompts that should fire it):  75%   FN=1
  ! 1 prompt that should have used the skill did not — available but not reached for.
```

Detection is deterministic, not an LLM judging "was it probably used": the run
uses `--output-format stream-json`, and a trigger is a `Skill` tool-use event for
this skill in the stream.

**Scope, learned from real runs (see [BENCHMARK.md](BENCHMARK.md)):** this
measures *explicit `Skill`-tool invocation*. That is the right signal for skills
designed to be invoked as a procedure. But Claude Code activates a simple
*convention* skill by putting it in context, not by making the model call the
Skill tool — so such a skill can measurably change behaviour (proven by the A/B
test) while its explicit-invocation rate is zero. Read trigger recall as a lower
bound on influence, not as "the skill did nothing"; the A/B test is what measures
influence. The false-positive side is still meaningful: a skill firing on
unrelated prompts is a real defect this catches.

## Real results

Four skills were run against a live agent (Claude Code, `claude-sonnet-4-5`,
N=5). The full write-up with methodology and raw data is in
[BENCHMARK.md](BENCHMARK.md); the short version:

- **No skill reached statistical significance on task success at N=5.** Two
  showed promising but inconclusive swings (0→40%, 40→80%, both p≈0.5); the tool
  reported them as inconclusive rather than publishing the point estimate.
- **The negative control read exactly zero** (0.00 delta, p=1.000) — the
  instrument does not manufacture improvements.
- **The only conclusive deltas were token *costs*** — including a skill whose
  advice the baseline agent already followed, so it added tokens and nothing else.
- **Trigger recall was 0–50%**: the skills often were not invoked even when
  clearly relevant, which likely explains the weak outcome effects. Availability
  is not invocation.

This is the intended behaviour: run it on your own skill and it will tell you the
truth, including "no effect" or "costs more than it helps."

## Reproducibility

Every `test` writes a full run record to `.crucible/runs/*.json` — not just
the summary, but every per-run outcome, and an `environment` block capturing what
the numbers depend on: tool version, agent (`claude`) version, model, OS,
Python version, and separate fingerprints for the **skill** and the **eval
suite**. The suite fingerprint exists because a result can flip when you reword a
prompt without touching `SKILL.md`; comparing across a changed suite fingerprint
is comparing two different experiments. `crucible report` prints the
environment block so a reader knows the conditions.

## Limitations

Read these before quoting any number from this tool.

- **Results are model-, harness-, task-, and environment-dependent.** A correct
  claim is *"on this eval suite, under this model and Claude Code version, the
  skill raised task success by N percentage points"* — **not** *"this skill makes
  the AI N% better"*. The run record stores exactly those conditions so the
  scoped claim is the easy one to make.
- **A benchmark is only as good as its eval suite.** The tool measures whether
  the agent satisfied *your* assertions on *your* tasks. A weak or unrepresentative
  suite produces a confident, meaningless number. `--dry-run` guards against
  assertions that pass on the untouched fixture; nothing guards against tasks that
  don't represent real use.
- **Cost.** Each run is a real agent session spending real tokens. `N tasks × N
  repeats × 2 conditions` is not free the way a unit test is; `--estimate` prints
  the projected spend before you commit.
- **Small N is a smoke test, not evidence.** Below ~5 runs per condition the tool
  labels the result `LOW` confidence and refuses to call effects conclusive, on
  purpose.

## Status

Claude Code only: `init`, `validate`, `test` (with `--dry-run`, `--estimate`,
`--strict`), `report`, `regression`, `trigger`. Tested on Linux/macOS/Windows
across Python 3.10–3.12; the test suite stubs the agent, so CI needs no API key.

Deliberately not built yet: LLM-as-judge quality scoring, cross-agent
`benchmark` / `compare`, and `audit` for skill security. These are held back on
purpose — the differentiator is deterministic, statistically honest A/B testing,
not breadth.

## License

MIT
