See it catch a planted defect
falsepass demo audits a toy environment whose grader pays when the answer appears anywhere in the reply. The static scan points at the line, and three probes confirm it from different sides.
Grader testing for RL environments
falsepass calls an environment's own reward code on completions whose verdict is known in advance, and reports every disagreement: wrong answers that get paid, right answers that don't. No model, no rollouts, no GPU.
uvx falsepass demo| completion | got | want | |
|---|---|---|---|
| #### 42 | 1.0 | 1.0 | ok |
| The answer is definitely not 42. | 1.0 | 0 | false pass |
| \boxed{42 \text{ is wrong}} | 1.0 | 0 | false pass |
| A B C D E F G H I J | 1.0 | 0 | false pass |
| #### 42. | 0.0 | 1.0 | false fail |
| (empty) | 0.0 | 0 | ok |
A grader is an assertion. falsepass tests it the way you'd test any assertion you didn't write: feed it inputs whose right verdict you already know.
A Hub environment, a local folder, an Inspect AI task or a HUD environment. It is installed in its own throw-away virtualenv, pinned to the versions it was built for.
falsepass init owner/environment
The reference answer and harmless variations of it; empty, echoed, every-option, hedged and wrong answers; malformed output; repeats. A mutation search and a static scan of the reward source join in.
falsepass audit environment --search 120
Every failing probe becomes a stand-alone repro that needs only the environment's own packages. Patch the grader and rerun it to see exactly which rows change.
falsepass repro environment --probe P5
Run on a few dozen public RL environments, the same handful of grader defects turned up again and again, across different authors.
Free-form tasks scored by keywords. One generic text, sent for every task, beat the dataset's own reference answers.
return df # dropna fillna astype …Format and reasoning terms that never look at the answer pay a wrong answer about a third of the maximum.
<think>…</think> \boxed{B} (wrong)A trailing period scores zero; a denial with no answer marker at all scores full marks.
The answer is definitely not 42.Normalisation built for units drops the text after a number, and only one side of an equation is compared.
\boxed{13 = 12}A reward term that raises on every call is caught, logged and recorded as zero. Runs finish with plausible numbers.
NameError: name 're' is not definedA game keeps answering after it's over, and the reward re-reads every message, so whoever keeps talking scores highest.
That was a great game!falsepass demo audits a toy environment whose grader pays when the answer appears anywhere in the reply. The static scan points at the line, and three probes confirm it from different sides.
Published or still in a local folder. Each framework is driven through its own code path, so what gets tested is the grader as it really runs.
The environment's rubric or taskset, all three API generations, single- and multi-turn, with scripted tool-use policies.
The task's own solver chain with a scripted model in place of generate(), then its scorers.
The task template's grader, driven the way HUD's server drives it. Answer-free probes when no references exist.
The GitHub Action fails the build when a known-bad answer earns reward, and puts the probe table in the job summary.
# .github/workflows/falsepass.yml on: [push, pull_request] jobs: falsepass: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - uses: Super-Eric/falsepass@main with: path: . fail-on: "ceiling>0.1 errors>0"
Every command is non-interactive and has a machine-readable form, so a coding agent can run a whole audit and hand you leads it has already reproduced.
Scan, init, audit, results, repro and discover as tools, for any MCP client.
The full audit procedure, including how to tell a defect from a design choice.
A compact results digest sized for a context window, CI gates, and llms.txt for the docs.
$ claude mcp add falsepass -- \ uvx --from "falsepass[mcp]" falsepass mcp › audit the grader of owner/environment falsepass_init owner/environment falsepass_audit environment n=25 false_pass_ceiling 1.0 via P5 shotgun/all_in_one_slot falsepass_repro environment probe=["P5"] … runs the repro twice, reads the reward code, reports the line and a suggested fix
An illustrative session. Any MCP client works the same way: setup for other clients →
Leads, not verdicts. A failing probe is a reason to read the reward code. A finding needs a repro that runs twice with identical output, the line that causes it, and a check of how many real tasks it affects. Strictness the prompt asks for, and graders that are exploitable on purpose, aren't bugs.
Used on a few dozen public environments. Fixes go to the maintainers first, as small pull requests.