Local · Nightly · Execution-verified
Whetstone
A model that trains itself overnight — and proves it didn't cheat.
Point it at your own repositories. Each night a local loop proposes candidate fixes, grades every one by re-executing the tests you hold in a sandbox — never by asking a model what it thinks — keeps only the wins, and LoRA-trains your local base on them.
uv tool install whetstonehq
CLI and import package are both whetstone. The nightly loop needs the MLX
extra: uv tool install "whetstonehq[mlx]". macOS on Apple Silicon, Python 3.10+, and uv.
01
The loop, in three movements
A night is a honing pass: a hard, honest edge, repeated strokes, and a record of what actually came off the blade. Nothing on the reward path asks a model for an opinion.
-
01 · Task
A broken commit, and tests that fail
A task is a repository at a known-broken commit, plus the tests that currently fail and the tests that must stay green. Point the miner at your own repository and it proves each candidate task live before keeping it.
The tests are the operator's artifact. The patch is the policy's. They never mix.
-
02 · Reward
The process exit status
The policy proposes a patch. The reward is whether the declared tests pass and the previously-passing ones still do — in a sandbox with no network, confined writes and a fixed seed. Held tests are restored from the golden copy after the patch lands.
No reference answer, no similarity score, no model opinion on the reward path.
-
03 · Train
Only the wins become examples
A night draws seeded attempts at every task under a frozen generation contract and keeps only the rollouts the strict verifier passed. Those are the training set for a LoRA run on your local base — verified by construction, so an unverified rollout can never become an example.
Everything stays on your machine; runs and checkpoints are gitignored.
In the morning
The verified delta, and what strictness caught
The report renders one night's sealed evidence into a page a person reads: the verified delta against the held-out set, what changed, and the reward-hacking count — rollouts a weaker check would have scored as wins.
A checkpoint is promoted only on a proven gain over a held-out verified set.
Unverified is never a win, and never rendered as PASS.
02
Why re-execution, not a judge
“One Token to Fool LLM-as-a-Judge” reports up to 35% false positives against a judge reward.
A guess about correctness is not a verification of it.
And a policy will find the guess. The usual patch — have a model grade the model — leaves the reward in the same place as the thing being measured.
Whetstone's reward is a process exit status. The verifier checks out the broken commit
into a sandbox, rejects any patch that touches an operator-held test path, restores
every held test from the golden copy, then runs the declared tests with a zero-skip
rule and an exact executed-set check — so -k, -m and
--deselect cannot quietly remove a test from the run.
The reward path is a guarded root: a scoped AST guard fails the build if an inference library is even importable from it. The classic failure mode is designed out rather than monitored for.
03
Status: honest
Read this before anything else. The machinery is built and the claim is not yet measured.
Built and tested
The reward, task ingestion, the nightly loop, the never-regress promotion gate, the baseline machinery, the honest-number report writer, the signed morning report, and the probe decision gate.
Not measured: everything the product claims
-
The nightly loop has run once. Night #1: 496 rollouts over 26.6
hours, with the control arm
INTACTon every one of them. -
It ended without a checkpoint. The training step raised inside the
pinned
mlx-lmbefore it wrote one. No trained model exists. - The promotion gate has never scored a real checkpoint.
- The morning report has never been rendered from a real night.
A training set and a yield figure now exist. A trained model does not. No performance number, gain, or benchmark figure appears on this page, because none has been measured.
04
Pre-registered
PREREGISTRATION.md was committed on 2026-07-29, in P1, before any number
about a model existed. It fixed what any future number may claim — the headline, the
metrics, the reporting rules, and the known limitations — while the outcome was still
invisible. That timing is the whole of its value.
When the first honest number exists, it will be published whether it is a gain, a zero, or a regression.
Publication is not gated on the result. There is no configuration of the numbers under which nothing is published.
Read PREREGISTRATION.md