End-to-end benchmarks for Claude Code and Codex configuration packages. Every run is a real agent session on a real coding task — no mocks, no cherry-picking. All data collected by Rubin Johnson sole experimenter
15 tasks across feature, bugfix, refactor, test-writing, and sequential categories. Mix of self-contained fixture repos and real open-source repos (Flask 3.1, httpx 0.28). Single run per package — treat as directional signal, not statistically final.
Tasks passed means all success criteria for that task were met. Each task has 1-3 automated checks (file exists, command exits zero, output contains a string). This is the primary quality signal.
Cost is actual USD at Bedrock/OpenAI rates, including cache reads. Varies run-to-run based on cache state at execution time.
Tokens is input + output tokens summed across all tasks. Does not include cache hits. Lower is cheaper per run.
Baseline is vanilla agent with no configuration. Every other package is compared against it.
Single-run data has meaningful variance. One hard task that fails can swing pass rate significantly. Interpret trends, not individual numbers.
Vanilla Claude Code — no CLAUDE.md, no configuration. The reference all other packages are measured against.
Rubin's full personal config (~270 lines). Model selection, token efficiency, TDD, git workflow, communication style, parallel execution preference.
rubin + async-first: explicit instruction to batch all independent reads, greps, and shell commands before waiting for results.
rubin + opportunistic-lazy: never re-read a file already in context, batch edits to the same file, reference by filename+line instead of re-fetching.
Batch all independent reads, greps, and commands in one message. Targets wall time; token cost is secondary.
Session indexing on first call, no re-reads, prune unrelated context every 5 tool calls, write state file before topic switches.
State assumption about structure before searching, explore breadth-first, one deep dive per task.
Read full error before retrying, enumerate failure modes before implementing, run targeted verification after each change.
Identify the smallest change that satisfies the requirement and write only that — no extras, no side quests.
Never re-read a file already in the conversation. Batch all edits to one file before moving on.
Hard tool-call budgets per phase: 10 explore, 15 implement, 5 verify. Budget overrun = stop and re-plan.
Correctness over speed. Full file reads, exhaustive test coverage, explicit criterion verification. Inverse calibration baseline.
Maximum suppression: no articles, no filler, max 30 tool calls, no prose between tool calls, zero code comments.
Sentence-level constraints: 1 sentence max between tool calls, no markdown, no articles, no filler openers.
Multi-file .claude/ layout, max 50 tool calls per task, separate rule files for workflow and efficiency.
Think before acting, read before writing, skip files over 100KB, recommend /cost on long sessions.
Trimmed config (~40 lines). Behavior-changing rules only: lead-answer, grep-first, one-tool-per-step, TDD, challenge assumptions.
Aggressive context minimization: strict retrieval hierarchy (LSP, grep, line range, full file), subagent delegation, compaction policy.
Numbered plan before any code change, one step per file, run tests after each step, no exploratory reads.
Strict TDD: failing test first, minimum code to pass, refactor only after green. 100% coverage on new code.
Limit extended reasoning to 2-3 sentences. No enumeration of possibilities — pick the most likely and act.
Correctness-first: read all relevant files before writing, type hints and docstrings, cover all test cases, re-verify requirements before done.
Lead with the answer, one sentence between tool calls max, grep before read, batch independent calls, minimum code that satisfies requirements.
RTK (Rust Token Killer) — CLI proxy that compresses command outputs via a Claude Code PreToolUse hook. Requires the bin/rtk binary.
Persistent memory plugin — hooks capture tool usage, a worker compresses and stores observations across sessions. Requires the claude-mem worker service.
All data on this page is collected by me on a consistent setup — I don't accept self-reported numbers. Caylent colleagues may get direct access to run experiments; reach out if interested.
To submit a package for benchmarking:
benchmark request: <package-name>CLAUDE.md / AGENTS.md, optional settings.json)I'll benchmark packages that test a genuinely new strategy dimension. I reserve the right to decline submissions that duplicate existing ones or can't run cleanly in the standard environment.
New experiment ideas are just as valuable as new packages. If you think the benchmark is missing an important dimension, submit it.
What makes a good experiment proposal:
Open an issue titled experiment idea: <short description> with the experiment goal, what you'd measure, and ideally a draft task YAML. The task format is documented in the repo README.