Proof-of-Work — Tech Stack & Technical Requirements

Decision 1 · the build stack

The finalized stack

One choice per layer, each verified against current (July 2026) versions. Bottom line: Python engine + tree-sitter (+ stdlib ast) + Docker-local / microVM-hosted + pytest·coverage.py / Vitest·v8 + git subprocess + official mcp SDK + uv/PyPI + SQLite.

Layer	Decision	Why	Watch out
Engine language	Python 3.12+	Native access to pytest/coverage internals, most mature MCP SDK, first-class tree-sitter, fastest solo dev loop.	No single binary — solved by uv/pipx below.
Code parsing (detector)	tree-sitter-language-pack + stdlib `ast`	One uniform parser across Python + JS/TS (306 grammars, actively maintained); stdlib `ast` for deep Python-only checks.	It's a syntax tree, not semantic — tamper checks are strong heuristics, not proofs. Pin versions (API churn).
Isolation (run tests)	Docker (local) · microVM (hosted)	A container is fine for your own local code; hosted mode running others' code needs a real guest kernel (E2B/Firecracker or microsandbox).	This is the one true architectural fork → hide behind a `Sandbox` interface. Reject WASM (can't run real test deps).
Tests + coverage	pytest + coverage.py 7.15 · Vitest 3 + v8	Machine-readable JSON output for both; Vitest v8 now matches Istanbul accuracy at 3–5× speed.	Also detect existing Jest projects. Coverage-drop needs a stored baseline — parse JSON, never scrape text.
Git diff access	shell out to `git` plumbing	Every repo already has git; output matches exactly, no build dependency.	Use `-z` NUL separators, disable color/pager, stick to plumbing commands for stable output.
MCP server	official `mcp` SDK 1.28.1 (FastMCP)	Current stable, production-recommended; expose one tool with `@mcp.tool()`.	A v2 rename lands ~late-July 2026 — wall all MCP code behind one thin module so migration is a one-file change.
Packaging / install	uv + PyPI (`uvx proof-of-work`)	10–100× faster than pipx, zero-install runs; PyPI publish keeps pipx users working too.	Skip single-binary until users demand offline. Ship a pre-commit hook + a composite GitHub Action.
Local storage	SQLite (stdlib), WAL mode	Zero-dependency single file, perfect for append-only run logs + coverage baselines.	Hook + CI can write at once → enable WAL and a `busy_timeout` to avoid locks.

Decision 2 · the detector

Two layers — only the deterministic one is trusted

Facts get signed; opinions stay advisory. This split is what makes the whole record reproducible and un-poisonable — the AI never decides the verdict.

🟢 Layer 1 — deterministic signatures

THE VERDICT · signed & logged · engine = git-diff + coverage delta + Semgrep + mutation testing

Test-integrity diff — tests deleted, renamed, or @skip/xfail added vs baseline.
Fake-pass — sys.exit(0), patched test runner, coverage disabled, function-under-test mocked away.
Coverage delta — tests still "pass" but cover much less code than the baseline.
Mutation testing (mutmut / Stryker) — the real fix for semantically gutted-but-present asserts that syntax checks miss: introduce a bug, confirm a test actually fails. Roast-added — the strongest signal.
Assert-weakening (heuristic) — assert True, removed assertions. Syntactic only; a helper, not a proof.

🟡 Layer 2 — LLM judge

ADVISORY ONLY · never signed · engine = DeepEval-style scorer

Reads the diff and asks "does this weaken verification / miss the task?"
Used to triage suspicious cases for review and to propose new rules — nothing more.
Its opinion is logged as metadata, never as the verdict.
Bring-your-own-key, so it costs the project nothing.

🔒 Why this matters: only Layer 1's deterministic output goes into the signed log. Because the record is deterministic, anyone can re-run it and get the same answer — and no one can poison the tool by gaming a non-deterministic AI judge.

Decision 3 · the self-improving loop ★ your request

How it gets sharper every run

🗓️ Roast correction — this is a v2, not v1. The council was unanimous: build the deterministic core first (Decisions 1–2), ship it, and let the corpus grow as a byproduct of real runs. The loop below is the plan for after the core has users — not week one. And name it honestly: it's a monotonically-growing net for known lazy-agent cheat patterns, not a measurable "catch-rate" (you can't measure cheats you never see). See the full verdict at the bottom.

The feature, when it comes, follows a published blueprint (Self-Harness, 2026): mine failures → propose a fix → gate it hard → only then promote. Golden rule: the loop can only ever ADD detection power — coverage never silently shrinks.

Capture — every run writes what happened

Verdict, which rules fired, a confirmed-clean run, a human override, or a cheat caught later (a bug that slipped past a "clean" verdict). Each signal is tagged by trust level and written to the tamper-proof log.

↓

Propose — mine the log for new cheat patterns

A batch job clusters confirmed cheats that no existing rule catches, and the LLM drafts a new Semgrep rule from those real examples. The AI only drafts — it has zero authority to ship it.

↓

🚦 Safety gate — the heart of "harnessed, not self-disabling"

The candidate is tested against two frozen, signed corpora before it can ship. Roast fix: dropped the vacuous "0% regression" clause — added Semgrep rules can't un-catch a positive, so that check never rejects anything. The real gate is FP-based:

catches ≥1 NEW cheat AND per-rule FP ≤1% (Wilson bound) AND AGGREGATE FP stays under budget

↓

Version & deploy — reversible, with garbage collection

The rule gets an id + semver + provenance hash, a signed git commit, and shadow-runs for N runs before it changes a real verdict. Rollback = git revert. Roast fix: "add-only" needs rule GC or FPs and runtime grow forever — so a new rule that subsumes an old one retires it automatically, but only when the frozen corpus confirms zero coverage is lost. Any change that would actually reduce coverage still needs a human two-key.

↓

Measure — honestly

The harness tracks precision + FP rate on the frozen corpora and coverage of known cheat classes per iteration. Roast fix: you cannot measure true "catch-rate" solo — the cheats you miss are unlabeled by definition. So don't claim recall on unknowns; claim "grows the net over known patterns, keeps FPs bounded." Ground truth must be human/synthetic-anchored, never LLM-labeled, or the judge leaks into the signed verdict.

🛡️ Anti-poisoning (an attacker teaching it to ignore a cheat): the loop may only add power — it can never auto-disable a check or add an exclusion. The regression corpus is frozen and signed, so even a fully poisoned input stream can't promote a rule that lets a known cheat through. Only deterministic "caught-later" and oracle signals create ground truth; human and AI labels are advisory and quarantined.

Decision 4 · logging every finding, provably

Tamper-evident record — one format, two tiers

Every run emits the same envelope — an in-toto attestation in a DSSE wrapper (the 2026 standard shared by Sigstore, SLSA, cosign). Because local and hosted records are byte-identical, upgrading tiers is a config change, not a rewrite.

DSSE ⟶ in-toto Statement { subject: sha256(changeset), predicate: { verdict, cheats_caught:[rule_ids], tool_version, ruleset_version, corpus_version, timestamp } }

🟢 Tier 1 — Local & free DEFAULT

hash-chained SQLite + Merkle head, signed

Each row links to the previous by SHA-256 hash chain — edit any past row and the chain breaks.
Hashes fold into a Merkle Mountain Range → one small signed "head" that commits to all history.
Head signed with minisign / cosign.
Honest limit: local key + local file is tamper-evident, not tamper-proof — so we periodically post just the signed head to an outside anchor (a gist / CI artifact / Rekor). Near-zero cost.

🟣 Tier 2 — Hosted & un-forgeable PAID

cosign keyless + Rekor v2 transparency log

cosign keyless signing (OIDC identity, short-lived cert) — no long-lived key to steal.
Each attestation pushed to Rekor v2 (GA, tile-backed) → public inclusion proof, third-party auditable, cross-repo.
This is the moat: a manager gets an un-forgeable proof that agent code was checked, across every repo and every vendor.
Self-hosted option: run Trillian Tessera to own the log on-prem.

Avoid: Amazon QLDB (shut down Jul 2025) and Trillian v1 (maintenance mode). Tessera / Rekor v2 are the live successors.

The spec

Technical requirements

What v1 must do (functional) and how it must behave (non-functional). These are the acceptance bar.

Functional

F-1Run the project's real tests in isolation and read the true result — never trust the agent's word.

F-2Detect the core cheats deterministically: deleted/skipped tests, weakened asserts, fake pass, coverage drop, mocked-away logic.

F-3Run a frozen held-out test bank the agent never saw and flag disagreement.

F-4Return a structured pass / fail + reasons the agent (or CI) can branch on.

F-5Ship as CLI, git hook, GitHub Action, and MCP tool — one engine behind all four.

F-6Support Python + JS/TS at v1; add languages via new grammars + rules.

F-7Self-improve each run through the gated loop (Decision 3).

F-8Write every finding to the tamper-evident log (Decision 4).

Non-functional

N-1Deterministic verdict — same input, same signed output, re-runnable by anyone.

N-2Human-wired gate — the agent can't skip it or reinterpret the result.

N-3Fast — the deterministic scan runs in seconds; the loop's learning is a background batch job, off the hot path.

N-4Reversible — every rule change is a signed commit; rollback is one command.

N-5Coverage never silently shrinks — the loop can only add checks; reducing them needs a human two-key.

N-6Zero-friction install — uvx proof-of-work, no account, works offline for the local tier.

N-7Isolation swappable — Docker/microVM/gVisor behind one Sandbox interface.

N-8MCP churn contained — all SDK calls behind one module.

How it's laid out

Component map & repo structure

front doors
CLI · hook · Action · MCP→ core engine
runner + detector→ Sandbox driver
(Docker / microVM)→ signed verdict
→ log ⟳ learn loop
(batch)

proof-of-work/ ├── core/ # the one engine everything shares │ ├── runner.py # run tests in isolation, read result │ ├── detector/ # Layer 1 signatures (Semgrep + git-diff) │ ├── oracle.py # frozen held-out randomized tests │ └── sandbox/ # Sandbox interface + Docker/microVM drivers ├── rules/ # versioned, signed Semgrep rules (auto-grows) ├── learn/ # mine → propose → gate → promote (batch) │ └── corpora/ # frozen regression + clean sets (signed) ├── log/ # DSSE envelopes, hash chain, Merkle head ├── judge/ # Layer 2 LLM judge (advisory, BYO-key) ├── interfaces/ # cli.py · precommit · action.yml · mcp_server.py └── eval/ # catch-rate harness, per-iteration metrics

The interfaces/ layer is thin — it just calls core/. The mcp_server.py is the single file walled off from the coming MCP v2 rename.

Straight talk

Hardest technical risks — and the answer

Risk	How the design handles it
Correctness can't be fully verified (the "verification horizon")	Never sign "this is correct." Only sign "these specific checks passed/failed." The detector is a strong filter, not an oracle.
AST checks are heuristic — a determined agent can evade them	Defense in depth: the authoritative signal is re-running tests + comparing coverage; the AST patterns are the extra net, not the whole net.
Local log is re-forgeable with disk + key access	Periodically anchor the signed head externally (gist / CI / Rekor). Tier 2 removes the local key entirely (keyless).
False-positive fatigue → humans disable the tool	The Wilson-bounded FP gate plus "loop can only add power" keep noise down; a bad rule can never ship silently.
Poisoning the learning signal	Frozen signed regression corpus + asymmetric authority (add-only) + label provenance tiers. Poisoning training can't beat an immutable test.
MCP SDK v2 breaking change (~late Jul 2026)	All MCP code behind `interfaces/mcp_server.py` — the migration is a one-file edit.

🎯 The through-line: Proof-of-Work applies its own thesis to itself — no change (even to its own rules) is trusted until it's gated against frozen ground truth. That's why it can safely get smarter every run without ever quietly getting weaker.

The council reviewed this architecture · 5 personas + Judge

🔥 The Roast — verdict on this spec

I put this finalized architecture through the /roast council. It caught real logic bugs, not just opinions — the corrections above (marked in gold) come straight from it.

RESHAPE → SCOPE DOWN

Confidence: HIGH

The stack choices are right and the moat is real — but as specified it's a cathedral a solo dev won't ship. Cut to the deterministic core; the loop and crypto are v2+.

Scorecard · avg 5.4 / 10

😈 Contrarian cathedral over a 200-line script

🔨 Builder core=8, the rest overruns a month

🧮 Logician vacuous gate + unmeasurable metric

🔎 Researcher choices spot-on; win on DX not novelty

🚀 Expansionist corpus + attestation = category moat

✅ SHIP in v1 (the 2-week core)

this is a solid 8–9/10 on its own

Deterministic detector: git-diff + coverage delta + sys.exit/skip checks + mutation testing
CLI + git pre-commit hook + GitHub Action (3 surfaces)
Local hash-chained SQLite log (prev-hash + signed head) — ~40 lines
Layer 2 LLM judge — advisory only, BYO-key

🕓 DEFER (v2+ / when there are users)

each is its own subproject — none has a week-one user

The self-improving loop (needs a real corpus first — grows as a byproduct)
MCP tool (add when asked; also the version-cliff surface)
Hosted microVM (that's infra + billing + on-call = a company)
Rekor / Trillian attestation (enterprise; local hash-chain is enough for v1)

🧠 The bugs it caught (now fixed above): ① the "0% regression" gate clause was logically vacuous (added rules can't un-catch). ② "add-only" makes false positives grow forever → needs rule GC. ③ you can't measure catch-rate solo (miss-rate is unlabeled) → reframed the claim. ④ syntactic checks miss semantic cheats → added mutation testing. ⑤ if ground truth is LLM-labeled, the judge leaks into the signed verdict → must be human/synthetic-anchored. ⑥ crypto proves integrity, not correctness — don't let it imply the verdict is right.

⚠️ Biggest risk

Scope. Built as written, you spend a month on corpus/eval/transparency-log plumbing and never ship the thing people wanted. And detection is an arms race vs adaptive agents — scope the claim to lazy, non-adversarial agents (today's real failure mode).

✅ Biggest upside

The moat isn't the detector — it's the corpus of how agents cheat + attestation as a standard. But the local-only spec forfeits it: to get the network effect the loop must become an opt-in federated corpus (users share caught cheats, everyone's net grows). That's the real 10x, later.

🧪 The 48-hour test (unchanged, and the council agreed): build ONLY the deterministic detector — git-diff for deleted/weakened tests + coverage delta + sys.exit/skip grep + one mutmut pass. Run it on ~20 real agent PRs. Publish the catch count. Don't write a line of the loop or the crypto until that number proves people care.

Verdict by the Judge over 5 independent council agents. Scores: Contrarian 3 · Builder 3 · Logician 5 · Researcher 8 · Expansionist 8. Not a KILL (the core is genuinely wanted and the stack is validated); not a GO (as specified it's a scope trap that won't ship solo).