The catch-the-cheating gate for AI-written code. This file explains, in plain terms, exactly how it works — what runs, in what order, why it's trustworthy, and how we ship it step by step.
The roast told us to change one thing, and it changes everything about how we build it:
An AI that grades another AI, which the suspect agent politely calls on itself. That's the weak, easy-to-copy part — and the big labs give it away free.
A cheat-catcher built on hard facts, run as a gate you control (a git hook / CI step), that the agent can't skip. Plus a signed record of every check.
The whole design follows one rule: trust the checks that have a right answer (a test runs or it doesn't; a file changed or it didn't) far more than checks that are a matter of opinion (an AI reading a diff). We climb this ladder top-down and only fall back to the weak rung when we must.
Small pieces, each doing one job. Green = free & open-source (the wedge that spreads). Purple = the paid, hosted parts (the moat).
Where it plugs in: a git pre-commit hook, a GitHub Action, or an MCP tool. The human wires it once, so the agent can't route around it. This alone kills the "fox guards the henhouse" problem.
Runs the project's tests in a clean, isolated process and reads the true result — not the agent's word for it. Records pass/fail, counts, and coverage.
The star. Diffs the change against git and parses the code (AST) to spot tampering — deleted tests, sys.exit(0), empty asserts, new skips, mocked-away logic. Zero AI, fully deterministic.
Only if turned on: an LLM compares the diff to the task and lists what may be missing. Advisory only. Bring-your-own-key so it costs us nothing.
An append-only, hash-chained record of every check: what ran, what passed, what cheating was caught, signed and time-stamped. This is what a team/manager pays for — proof, across repos, that can't be edited after the fact.
The roast said this is the one truly defensible piece. It's just careful code inspection — no AI, so it's fast, free, and always gives the same answer. The concrete checks:
sys.exit(0), os._exit(0), pytest.skip in the run path.assert True, removed assertions, assertTrue(1).@skip or @xfail added to make red go green.The roast's sharpest hit: "if the tests are on the agent's machine, the agent can read them." True. So we ship two modes and are upfront about each:
Even without hidden tests, the top two ladder rungs (real tests + tamper scan) already catch most real-world cheating. Hidden tests are the upgrade, not the base.
Same core, exposed four ways so it drops into any workflow with no friction — this is also the distribution: each front door is a place people discover it.
# run it by hand or in any script
pow check # checks the current diff
✓ tests: 142 passed
✓ no tampering found
⚠ AI judge: endpoint has no error test
VERDICT: PASS (1 warning)# .pre-commit-config.yaml
- repo: proof-of-work
hooks:
- id: pow-gate
# now every commit is checked
# cheating = commit blocked- uses: proof-of-work/gate@v1
with:
mode: hosted # secret tests
fail-on: tamper
# blocks the PR if the agent cheatedverify(task, changeset) → {
passed: false,
tampered: true,
unmet: ["delete-user path"],
attestation: "pow:9f2c…"
}Give away the part that spreads; charge for the part that proves. Same model LangChain used (free library, paid cloud).
→ Job: get installed everywhere, earn trust, collect stars.
→ Job: the number an eng manager pays for.
Build ONLY the deterministic cheat detector as a script. Run it on ~20 real agent-made PRs (yours + public). Measure how many cheats it catches. Post the number on r/ClaudeAI / HN. This tells you the whole idea is real — for two days of work.
Wrap the detector + real-test runner into a clean CLI and a pre-commit hook. Zero AI. One-command install.
Add the harder checks (weakened asserts, mocked-away logic, coverage drop). Depth here = the moat nobody else has.
Open-source repo, MCP tool, GitHub Action. List on mcp.so, Glama, Smithery, PulseMCP, awesome-mcp-servers. Show HN + Reddit post with the catch-rate proof.
Sign & hash-chain every run locally. This is the seed of the paid product.
Hosted secret-test bank + team dashboard + compliance export. Sell to the eng manager, not the IC.
| The risk (from the roast) | How the build answers it |
|---|---|
| Fox guards the henhouse — the agent chooses whether to call it | The gate is wired by the human as a hook/CI step. The agent can't opt out or reinterpret the result. |
| "Hidden tests" aren't hidden on the agent's machine | Two honest modes: local (stops casual cheating) and hosted secret bank (truly un-cheatable, and it's the paid tier). |
| The labs bundle self-verification for free | We don't sell generic self-verify. We own deterministic cheat detection + a cross-vendor audit trail — the part a lab won't build against its own agent. |
| AI-judging-a-diff is flaky and you can't trust it | Correct — so it's optional and advisory only, never the thing that decides pass/fail. The verdict rests on facts. |
| Agent has no wallet; the IC won't pay | Free for the IC forever. The paid product is a manager's compliance dashboard — a different, paying buyer. |
| The idea is in the air; someone else ships first | That's the real risk. Answer = the 48-hour test this week, then a fast public launch. Speed is the strategy. |