Metadata-Version: 2.4
Name: rulebench
Version: 0.2.0
Summary: Does your CLAUDE.md actually do anything? Trap-test your agent rules in isolated sessions and get honest deltas.
Author: Rafael Pena
License: MIT
Project-URL: Homepage, https://github.com/ralfyishere/rulebench
Project-URL: Issues, https://github.com/ralfyishere/rulebench/issues
Keywords: claude-code,ai-agents,llm-evals,agent-rules,claude-md
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# rulebench

**Does your CLAUDE.md actually do anything? Point rulebench at your rules and find out.**

rulebench runs trap tests across named rule configurations (no rules, your rules, your rules + skills, anything you define) in fresh isolated Claude Code sessions, grades the outputs against pre-written rubrics, and reports honest deltas: what your rules changed, what they didn't, and what never ran.

Born from [rules-with-receipts](https://github.com/ralfyishere/rules-with-receipts), where the eval harness found that most of a rules pack's claimed value was already baseline model behavior, and the real wins were narrow and specific. This tool makes that measurement reusable for any rules file.

## Quick start

```bash
pipx install rulebench   # the CLI (needs python3 3.9+ and the `claude` CLI on PATH)
git clone https://github.com/ralfyishere/rulebench && cd rulebench   # for the starter traps
cp config.example.json config.json   # edit: point conditions at YOUR rules artifacts
rulebench config.json --reps 3       # costs real API tokens
```
Installed via pipx you get the `rulebench` command; the nine starter trap tests live in this repo's `tests/`, so clone it (or write your own traps) and point `tests_dir` at them.

Output: `results/<timestamp>/REPORT.md` (scores table + honesty section + per-cell verdicts with evidence) and `results/<timestamp>/raw/` (every session's full output and workspace diff).

## How it works

- **Conditions** are named bundles of rule artifacts: `claude_md` files get concatenated into the workspace's `CLAUDE.md`; `skills` directories get copied to `.claude/skills/`. An empty condition `{}` is your baseline.
- **Tests** are folders under `tests/`: a `test.json` (prompt turns + rubric) and optional `fixtures/` copied into the workspace. Multi-turn tests continue the same session. Ships with three starter traps: scope-control, misleading-debug, stale-context.
- **Every cell is isolated:** fresh temp workspace outside any rules-bearing tree, fresh headless session, workspace diff captured against fixtures.
- **Grading is rubric-first:** the rubric is written before running; a grader model applies it per cell with schema-enforced verdicts (PASS/PARTIAL/FAIL) and a required evidence quote.
- **Quota stubs are NOT RUN, never FAIL.** Provider limit messages mid-batch bias results toward whichever condition ran first; rulebench detects and excludes them, and tells you.

## Reading the report

The honesty section is the point:
- Tests where all conditions tie are measuring the **model**, not your rules.
- Only differentiated tests say anything about your rules file.
- Medians of 1 rep are noise. Use `--reps 3` minimum for anything you'll act on.
- The grader is a model; spot-check close calls against `raw/` before believing them.

## Writing your own trap

The starter tests will saturate quickly, and public traps invite overfitting. Write private ones:

1. Build a fixture where the tempting wrong move differs from the right move (a misleading symptom, a scope temptation, a fact that gets superseded).
2. **Verify the fixture by execution** before trusting any run (the crash must crash, the bait must be real).
3. Write the rubric before the first run: PASS/PARTIAL/FAIL in terms of observable behavior only.

## Security: rules files are untrusted code

A rules file is instructions an agent will follow with tool access. Loading an untrusted one is running untrusted code. Screen any third-party CLAUDE.md, .cursorrules, AGENTS.md, or skill before it enters a session:

```bash
rulebench vet path/to/CLAUDE.md      # a file
rulebench vet path/to/repo           # or a whole repo (finds rules files)
rulebench vet ./rules --json         # machine-readable, for CI
```

`vet` is offline and instant — no model calls. It flags known-shape risks: pipe-to-shell, credential/env access, exfiltration shapes, always-run directives, destructive commands, out-of-project writes, hidden text, and instruction-override language. HIGH means act; MEDIUM means glance. It exits nonzero on HIGH (tune with `--fail-on`), so it drops into CI.

**A clean vet means "no known-shape red flags", not "safe".** Pattern matching cannot catch cleverly-worded natural-language social engineering. Read anything you're about to let an agent follow, run unfamiliar rules on a machine you don't mind rebuilding, and never with credentials you can't rotate.

## Honest limitations

- Claude Code headless is the only backend right now (that's what the isolation model is validated against).
- Runs cost real API tokens: cells × reps × turns, plus one grader call per cell.
- Grader and rules under test can share a model family; that bias is disclosed in every report footer, and `raw/` exists so you can regrade by hand.
- This tool measures behavior deltas on your traps. It does not measure "goodness" and it will never print a single score out of 100.

## License

MIT
