muteval · live demo

Your evals are passing.
Would they catch a regression?

muteval deliberately breaks the system under test, reruns your existing eval suite, and reports which breakages your evals failed to notice. Below is a run on a bundled example suite — no API key, fully reproducible with one command.

01The suite under test

A support-bot prompt with four rules, and a small eval suite that checks the answer is non-empty and cites an order ID. Looks reasonable. It passes.

examples/support_bot/muteval_config.py

SYSTEM PROMPT
  You are a support assistant for an online store.
  - You must cite the order ID when discussing an order.
  - Do not promise refunds; refunds require manager approval.
  - You must never reveal another customer's data.
  - Always reply in a polite, professional tone.

EVALS (the suite we are grading)
  contains_case("order_id")     # cites the order id
  not_contains("ERROR")         # produced an answer
  # ...and nothing that checks the refund / privacy / tone rules

02Run muteval

It generates 22 mutants (weaken a rule, invert it, drop a line, truncate…), reruns the suite against each, and scores how many the evals caught.

$ muteval run --config examples/support_bot/muteval_config.py

Mutation score: 23%  [█████░░░░░░░░░░░░░░░░░░]  (5/22 mutants killed, 95% CI 10-43%)
Effective score: 45%  (5/11 — excludes 11 inert mutants whose output didn't change)

6 SURVIVED  (output changed but the evals didn't notice — 4 HIGH-severity):

  [HIGH] SURVIVED  [weaken_modals]
        weakened "Do not promise refunds" → "try not to promise refunds"
        fix: add an eval for the refund rule (e.g. checks.llm_judge)
  [HIGH] SURVIVED  [flip_negation]
        inverted "Do not promise refunds" → "do promise refunds"
  [HIGH] SURVIVED  [drop_instruction_lines]
        dropped line: "- Do not promise refunds; refunds require manager approval."
  [HIGH] SURVIVED  [delete_sentences]
        deleted the refund rule entirely
  [MED ] SURVIVED  [truncate_prompt] × 2
        dropped the tail of the prompt (privacy + tone rules)

The catch HIGH

muteval turned the refund guardrail into its opposite — the bot now promises refunds it shouldn't — and the suite stayed green. Nothing in it checks that rule.

Do not promise refunds → do promise refunds evals: still PASS ✓

03Close the loop

Each survivor names a missing eval. Add it, and that mutant dies — the score climbs. Here's the same system graded by suites of increasing coverage (an offline, CI-enforced experiment shipped in the repo):

Eval suite	Coverage	Real gaps left
smoke — non-empty only	0%	21
+ cite the order ID	33%	14
+ refund & privacy rules	67%	7
+ tone & grounded-in-context	100%	0

The score is a faithful proxy for how much real degradation the suite would catch: it rises monotonically with coverage, and the same result holds across four domains (support bot, code review, RAG grounding, HR policy), enforced in CI so it can't silently regress.

04What the score means

muteval is a per-suite diagnostic, not a universal flaw-finder. A few things it does by design:

A survivor is a candidate gap — flagged for review, not an automatic verdict on your system.
Inert mutants (output unchanged) are excluded from the effective score, not counted as blind spots.
It fails closed: a red or errored baseline yields no score, never a misleading 100%. Scores come with confidence intervals.
It measures eval coverage, not whether your eval is correct — that needs labels (see the limits).

The limits →

05Reproduce it yourself

No API key. The example runs on a deterministic mock model, so you get the exact output above.

pip install muteval
muteval run --config examples/support_bot/muteval_config.py   # the run above
python validation/eval_quality_experiment/run_experiment.py  # the 0→100% loop

Your evals are passing.Would they catch a regression?