AI coding agents don't just write bugs. Told to “make the tests pass”, they delete the failing assert, skip the test, or import a module that doesn't exist. The suite goes green; the code is still broken. Vurnix is the deterministic gate that makes those shortcuts impossible to hide.
then vurnix gate ./myproject — compile + phantom-import + honest test count, one verdict.
$ vurnix integrity compare before.json tests/test_api.py WEAKENED: asserts dropped 17 → 3 WEAKENED: test function(s) deleted: test_redirect_302 (6 → 5) integrity: test file WEAKENED — do NOT count this round as fixed. exit 1BLOCK
Real output. A cloud model, brought in to rescue a failing build, rewrote a 17-assertion test file down to 3 to go green. The gate caught it — and named the deleted test.
A check that cannot run
is not a check that passed.
No model in the loop. No heuristics you have to trust — every check is plain code you can read, with a hard exit code. Missing a toolchain is a labelled SKIP, never a silent OK. Any failing check is a BLOCK and a non-zero exit. There is no “mostly green”.
Shortcuts a human reviewer rarely thinks to check for — and a deterministic checker always does.
The one line that proved the behaviour is simply removed. What's left still runs — it just tests nothing.
suite → green · code still brokenA @skip or xfail quietly excuses the case that was catching the bug.
assert TrueNew “tests” that assert nothing inflate the count so coverage floors look satisfied.
count → up · nothing verifiedfrom models import UrlRequest — no such module, anywhere. Valid syntax; fails cryptically later.
Each one turns the suite green without fixing the code. Vurnix makes every one of them a hard, named failure — before anyone claims the tests pass.
Six commands. Each is plain code with a hard exit code — usable standalone or as one composite gate.
| Command | Catches | Languages |
|---|---|---|
| vurnix integrity | tests being weakened to go green — deleted asserts, new skip/xfail, always-true asserts, deleted test functions | pyjsgojava |
| vurnix coverage | padded test counts — only distinct, non-trivial tests count; assert True and copy-paste duplicates don't |
pyjsgojava |
| vurnix mutation | weak tests — mutate the implementation; a mutant your tests don't kill is a coverage gap with a file:line name | py+ C-family |
| vurnix phantom | invented imports — a module referenced in code that exists nowhere: not stdlib, not local, not vendored | py |
| vurnix compile | code that doesn't even build, per file, before anyone claims “tests pass” | pyjsgojava |
| vurnix gate | composite — compile + phantom + coverage floor, resolved to one honest verdict | all |
Every checker earned its place catching a real incident in our 12B-class local-model benchmarks.
integrity, with the deleted test named.status == 200 scores ~15%; real behaviour ~70%.phantom before it failed cryptically at install.Deterministic orchestration is commoditised. An honest gate that never reports a green it can't defend has no counterpart — that's the whole point.