TestGuard
Technical Brief · v0.6 · MIT

Your tests pass. That is not the same as your tests working.

TestGuard takes the promises your code makes, breaks each one on purpose, and reports every promise your tests did not notice breaking.

The problem

Coverage tells you a line ran. It never tells you anyone checked the result. So a codebase can be fully covered and completely undefended — and nothing in CI will say a word.

A real test, from this project's own fixture
expect(store.writeAudit).toHaveBeenCalledWith(
  expect.objectContaining({ action: 'MASK', scope: 'g1', ruleCount: 1 }),
);  // `content` is never named — so nothing checks it

objectContaining ignores keys it does not list. Swap the redacted text for the raw secret and this test still passes. Coverage of that line: 100%. The audit log now leaks the very thing it exists to protect.

Coverage asks “Did this line run?” 100% covered — and it is telling the truth TestGuard asks “If I break it, does anything notice?” 0 of the tests covering it would fail

Fig. 1 — The same line of code, under two different questions. Only one of them is worth knowing.

The method

killed

You broke it — a test failed. Good. That behaviour is genuinely defended.

SURVIVED

You broke it — everything stayed green. A blind spot.

The word trips everyone once: it is the fault that survived, not the test. SURVIVED is the bad news. A healthy report is full of killed.

A tool that reports everything as caught is worse than no tool at all — because nobody questions good news.

The design constraint everything else follows from
TestGuard — Technical BriefPage 1 of 5 · The idea
TestGuard
02 — What it found in the field

Two independent runs, two AI-authored codebases

Neither suite was sloppy. One had ~4,900 disciplined tests, no snapshots, and 0.4% zero-assertion. Both were green. Both were substantially blind.

Field report A — AI-authored production codebase, ~4,900 tests 8 of 9 real historical bugs the suite could not see 2,451 tests green, at once, on known-broken code 100% coverage on the compliance path that leaked Field report B — a different codebase, 458 tests, 24 security claims, 39 faults injected Before — every test green 21 faults SURVIVED a fully green suite 9 of them critical — super-admin gating, cookie flags, the whole authorization callback After — tests written against the survivors 39/39 killed The suite was not bad. It was unexamined. killed — defended SURVIVED — a blind spot

Fig. 2 — Measured, not modelled. Each dot is one injected fault.

The shape of the failure

It is not missing tests

One file had re-implemented the authorization logic inside the test and asserted against its own copy. Fifteen tests, all green, all passing for the wrong reason — the production path could be deleted outright without one of them noticing.

It is assertions that name too little

The largest gap ran through a compliance-critical path at 100% coverage, where the one assertion that mattered used objectContaining and omitted the field carrying the data. Exactly the exhibit on page one.

A green suite is evidence that the tests agree with the code. It is not evidence that either one is right.

Why the question has to be asked adversarially
TestGuard — Technical BriefPage 2 of 5 · Evidence
TestGuard
03 — Anatomy of a run

One fault, start to verdict

Isolate scratch worktree Baseline untouched code ×3 Inject apply the fault Probe same tests again ×3 Restore always Classify one of seven verdicts Must pass 3 of 3 — or the run stops here A suite that cannot stay green on untouched code cannot be asked anything about a fault. In → testguard.claims.json Out → evidence.json Your working tree is never modified — the fault is applied to a copy.

Fig. 3 — The probe loop. Every step repeats per fault; the baseline is cached per defender set.

How a verdict is decided

The order is the specification. Each check short-circuits the ones below it, so the tool never reaches an optimistic answer through a question it should not have asked.

Does any test defend this claim? no NOCOVER nothing looks at this code Did the fault apply to exactly one place? no UNVERIFIABLE anchor moved, or matches twice Was the baseline green, 3 of 3? no FLAKY-DEFENDER not steady enough to ask With the fault in, did the suite still load? no FAULT-INVALID our break was broken Did anything time out? yes TIMEOUT a hang is not a detection Did a test assert its way to failure, in all 3 runs? yes killed the only good outcome Everything passed, every run. SURVIVED a real blind spot Anything in between — some runs killed, some passed — is FLAKY-DEFENDER, never a kill.

Fig. 4 — Verdict order. Every ambiguity resolves downward, toward “unproven”.

What a run costs, and why it is predictable

Wall clock is not a mystery. For each claim:

(baseline runs + faults × N) × cost of the
                       claim's defender set

The defender set is the unit, not the file — a run executes all of a claim's tests at once. So one slow acceptance test named by five claims is paid for thirty times over, and the report names that file rather than leaving you to guess.

Baselines are cached per defender set, and a verdict is reused when the source, the defenders and the fault are all unchanged.

TestGuard — Technical BriefPage 3 of 5 · Anatomy
TestGuard
04 — Verdicts & architecture

Seven verdicts, because “did the tests fail?” is a sloppy question

VerdictWhat it means
killedA test body ran and rejected the behaviour. The only good outcome.
SURVIVEDEverything passed. A real blind spot in the tests.
NOCOVERNo test even looks at this code. Not “weak tests” — no tests.
UNVERIFIABLEThe fault could not be applied: its anchor moved, or matches twice. Nothing was learned.
FAULT-INVALIDThe break itself was broken — it did not compile. Our fault, not yours.
TIMEOUTThe suite hung. A hang is not a detection.
FLAKY-DEFENDERThe tests are not reliable enough on untouched code to be asked the question.

Where the pieces sit

Input — reviewed like code claims.json Claim: what must be true Faults: how it could be false Engine — testguard probe Scratch worktree isolation + restore Inject / run / N× budgeted processes Runner adapters — one report shape, five engines vitest jest playwright pytest unittest classify() pure; the check order is the specification Output — schema-validated evidence.json a verdict per fault, with its runs baseline.json today's blind spots, frozen gate → CI fails only on blind spots that are new since the baseline Why the baseline matters A legacy codebase does not start with four hundred failures. Freeze what is already blind, then gate only on what gets worse — so the tool is adoptable on a Monday instead of being switched off on a Tuesday.

Fig. 5 — Claims in, evidence out, and a gate that measures the delta rather than the debt.

Every ambiguity rounds toward “unproven”

  • Three runs, never one.
  • A green baseline before any fault is injected.
  • A timeout is never a kill.
  • A suite that fails to load is never a kill.
  • Mixed results across runs are not a kill.
  • A test that mocks what it defends is not a defender.
TestGuard — Technical BriefPage 4 of 5 · Verdicts & architecture
TestGuard
05 — Prior art & difference

What is not new

Breaking code to test your tests is mutation testing, and it dates to the 1970s. Stryker, PIT, mutmut and Cosmic Ray all do it. Anyone claiming that part is novel is selling something.

What is different: the question being asked

Classic mutation testing

Mutates everything, mechanically

“How good are my tests?”

→ Mutation score: 73%

A number. Not actionable, not auditable, and it cannot tell you which promise is at risk.

TestGuard

Binds every fault to a stated claim

“Is this specific promise defended?”

→ “Your project says a missing scope fails closed. Nothing checks that.”

A finding. An assessor reads your claims — reviewable like code — not a score.

Mutation score — thousands of syntactic mutants, one number 73% Which promise is at risk? Unanswerable. Claim-bound finding — each fault keeps its name REDACT-001/F1 · critical · src/redact.mjs SURVIVED “The audit row never contains the original input.”

Fig. 6 — A score compresses everything into one number; a claim keeps the finding attributable.

That is the difference between a metric and an audit, and it is the whole design.

Three more things that follow from it

Paranoid about itself

Most tools mark a mutant killed when the test command exits non-zero — conflating “a test caught it” with “everything exploded.” Here the verdict set is closed and each member means something different.

Evidence, not output

Schema-validated JSON with stable fingerprints, shared with the other Guard tools. Freeze today's blind spots, gate CI on new ones.

Built for AI-written code

When an agent writes the tests, the failure mode is not “no tests.” It is confident, plausible tests that assert nothing. That is precisely what this detects.

Coverage tells you a line ran. TestGuard tells you which of your promises nobody is actually defending.

github.com/raccioly/testguard · npm testguard-cli · MIT
TestGuard — Technical BriefPage 5 of 5 · Difference