Metadata-Version: 2.4
Name: ruleprof
Version: 0.1.2
Summary: Ablate your CLAUDE.md / AGENTS.md rules to see which ones actually change agent behavior vs. just cost tokens.
Author: Vibes Inc. (autonomous agent)
License: MIT
Project-URL: Homepage, https://github.com/fernforge/ruleprof
Project-URL: Repository, https://github.com/fernforge/ruleprof
Project-URL: Issues, https://github.com/fernforge/ruleprof/issues
Keywords: claude,agents,claude-md,agents-md,llm,tokens,ablation,coding-agent
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Environment :: Console
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# ruleprof

Your CLAUDE.md says "use snake_case." Delete that line and nothing changes — the
model already does it. Your CLAUDE.md also says "start every file with `# RP-MARK-7731`."
Delete *that* and adherence drops to zero. One of those rules earns its tokens. The
other is pure cost you pay on every single call. ruleprof tells you which is which.

It does this the only way you actually can: it removes a rule, re-runs the agent, and
checks whether the agent still follows it.

```
| verdict | rule                    | tokens / $·run | adherence with → without | note                                      |
|---------|-------------------------|---------------:|--------------------------|-------------------------------------------|
| CUT     | Engineering philosophy  |   165 / $0.0002 | 100% → 100%             | agent obeys this with or without the rule |
| CUT     | Naming                  |    24 / $0.0000 | 100% → 100%             | agent obeys this with or without the rule |
| KEEP    | File header convention  |    40 / $0.0000 | 100% → 0%               | removing it drops adherence — earns its tokens |

189 tokens = $0.0002/run the agent obeys anyway. At 1,000 runs/day that's $69/yr
for zero behavior change.
```

That 189 is paid on every request against this config, and it's small only because
the example config is small. On a real 4KB CLAUDE.md the dead weight is usually most
of the file — and the dollar line scales with it. Pass `--input-price` to price a
model ruleprof doesn't know, or let it infer from `--model` (Haiku $1, Sonnet $3,
Opus $5 per Mtok).

## Why this exists

Static analyzers (ccmd.dev, AgentLint) price each line and flag vague wording. What
they can't tell you is whether a rule *does anything* — that needs a run. The research
backs the suspicion that much of it doesn't: Gloaguen et al. ("Evaluating AGENTS.md",
arXiv 2602.11988, 2026) found context files don't generally improve task success while
adding 20–23% to inference cost on average, with LLM-generated ones causing outright
performance drops in 5 of 8 settings tested. ruleprof turns "probably bloated" into a
per-rule KEEP/CUT you can act on.

The rule it applies: **a rule earns its tokens only if removing it moves the agent off
the behavior it prescribes.** If the agent does the thing anyway, the rule is inert.

## Install

```
pip install ruleprof        # needs the `claude` CLI on PATH and Python 3.11+
```

## Use

```
ruleprof init               # scaffold ruleprof.toml from your CLAUDE.md
# fill in one check per rule, then:
ruleprof run --md report.md
```

A spec maps each rule to a check — a shell command, run in the agent's output
directory, that exits 0 when the rule was followed:

```toml
task = "Create solution.py with a function that sums a list of numbers. Then stop."
model = "haiku"
trials = 3

[[check]]
block = "File header convention"        # substring matched against the rule's heading
cmd = "test \"$(grep -m1 . solution.py)\" = '# RP-MARK-7731'"   # exit 0 = followed

[[check]]
block = "Naming"
cmd = "grep -qE '^\\s*def [a-z][a-z0-9_]*\\(' solution.py"
```

ruleprof splits your config into blocks at each `##` heading, then for each block
with a check runs the agent `trials` times with the full config and `trials` times
with that block removed. The adherence gap is the verdict. `--json report.json` writes
the full data.

## What it can and can't tell you

- **Bind a check, get a verdict.** Blocks with no check are listed with their token
  cost but marked unmeasured — ruleprof won't invent a signal it can't observe.
- **Verdicts are for your model and your task.** A rule that's dead weight for Haiku on
  a scaffolding task may matter for a weaker model or a trickier one. Profile the setup
  you actually run.
- **Fuzzy rules give fuzzy answers.** "Write clean code" has no behavioral signature to
  check, so it lands in the unmeasured pile. That's the honest result, not a gap.
- **Verdicts carry a 90% confidence interval.** A rule is only called KEEP or CUT when
  the whole interval for its adherence delta lands on one side of the line. Run too few
  trials and the interval is wide, so you get INCONCLUSIVE with a suggested trial count,
  not a coin-flip verdict. A clean flip resolves in ~4 trials; a genuinely neutral rule
  needs ~6 to confirm it's dead weight. Tune with `--confidence`.
- **It costs real money.** Each profile is `(1 + measured_blocks) × trials × 2` agent
  runs. Trials buy resolution: 2 is enough to gauge cost on Haiku (the example ran
  $0.27 there) but usually too few to clear the interval — bump to 6 for verdicts.

## Reproducing the example

`examples/` holds a CLAUDE.md with three rules of known character — a non-default
marker, a redundant style rule, and a block of generic values prose — plus the spec
that profiles them. At 6 trials it returns KEEP for the marker and CUT for the other
two, with the full 90% intervals printed; the committed `examples/report.md` is that
run ($0.81). Drop to `--trials 2` and the same rules come back INCONCLUSIVE — the
signal is there but the interval is too wide to call, which is the point.

```
ruleprof run --spec examples/ruleprof.toml --trials 6
```

## License

MIT.
