muteval · promptfoo example

Point muteval at a promptfoo suite.
Get two reports back.

Install, point it at your promptfooconfig.yaml, and read the two reports it writes. The example below runs against a public promptfoo suite (adelmuursepp/promptfoo-demo-evals); you'd point it at your own.

1 · install2 · the suite3 · run 4 · coverage report5 · quality report 6 · reading the report7 · your repo

01Install

Pure-Python core; the [promptfoo] extra only adds a YAML parser. No key needed to install.

pip install "muteval[promptfoo]"
export OPENAI_API_KEY=sk-...   # muteval runs your prompt on a model; set your provider's key

02The suite in this example

muteval reads any promptfooconfig.yaml directly — the prompt becomes the mutation target, each test a case, each assertion a graded check. This example uses a language-tutor feedback prompt; swap in your own config and the flow is identical.

The prompt (mutation target)promptfooconfig.yaml → prompts
You are a friendly language tutor.
Based on the following grading in {{language}}, provide constructive feedback to the
student in English so they can improve in learning the language.
If the mark is 10, just say the user did a great work. Do not use markdown for the
feedback message. The feedback should be concise and direct.
Return JSON with a single key 'feedback_message' containing the message.
The assertions (the graded suite)3 test cases
✓ is-json — output is valid JSON  (format)
✓ contains: feedback_message — the required key is present  (format)
✓ llm-rubric: "the feedback does not use markdown"  (one behavior)

Notice what's not asserted: friendly tone, answering in English, conciseness, the "mark = 10 → just say great work" shortcut. Hold that thought.

03Run it

muteval generates mutants of the prompt, reruns the promptfoo suite against each, and scores how many the assertions caught — then writes the two HTML reports.

muteval run    --promptfoo promptfooconfig.yaml
muteval report --html coverage.html
muteval probe  --promptfoo promptfooconfig.yaml --html quality.html

04Report 1 — coverage

Would the suite notice if the prompt silently degraded? For 11 of 15 injected regressions: no.

Mutation coveragecoverage.html
27% 4 / 15 mutants killed · 95% CI 11–52%

✓ Caught (4) — the assertions did their job

KILLEDMutations that broke the JSON output or dropped the feedback_message key — caught by is-json / contains.

Survived — uncovered behavior (7)

MEDdrop_instructiondropped "You are a friendly language tutor."
MEDdelete_sentencedeleted "…in English so they can improve…"
MEDdelete_sentencedeleted "If the mark is 10, just say the user did a great work."
MEDdelete_sentencedeleted "The feedback should be concise and direct."
MEDdrop_instructiondropped the conciseness rule  (+ 2 more sentence/line drops)
No assertion checks tone, language, conciseness, or the mark-10 shortcut — so degrading any of them passes clean. Each survivor prints the fix: add a checks.llm_judge for that rule, re-run, the mutant dies.

Survived — likely equivalent (3)

HIGHweaken_modalsweakened "Do not" → "try not to" use markdown
HIGHflip_negationinverted "Do not" → "do" use markdown
HIGHdelete_sentencedeleted the no-markdown rule
Flagged HIGH by severity heuristic, but the suite's no-markdown judge passed them — the model didn't emit markdown even with the rule weakened. Equivalent mutants: worth a glance, not a gap.

05Report 2 — eval quality

Coverage asks "would the suite catch a regression?" The quality card audits the suite itself, ranked by what each lens actually catches.

Eval-quality report cardquality.html
PASSjudge_reliabilitycore — 0/3 verdicts flipped across re-runs = 0% flaky (Krippendorff α 1.00). The no-markdown judge is stable.
PASSdiscriminationcore — not assessed (no good/bad exemplars). Add them to test whether the suite separates good answers from bad.
PASShuman_agreementvalidity — not assessed (no labels). muteval label emits a worksheet to enable it.
WARNstatistical_adequacyhygiene — 3/3 pass = 100% [95% CI 44–100%]. Only 3 cases → the number is too uncertain to trust; add ~9.
PASSredundancyhygiene — not assessed (needs ≥ 2 evals).
PASSthreshold_calibrationhygiene — not assessed (no good/bad exemplars).

No composite score on purpose — each lens is a separately-interpretable signal. core = catches an eval defect that matters · validity = needs labels · hygiene = sanity check.

06Reading the report

27% is a starting point, not a verdict.
  • The gaps that matter: the suite guards the output format but not the tutor's behavior — friendly, in English, concise, the mark-10 shortcut. Those are the survivors that count, and each one names the eval to add.
  • The markdown survivors are equivalent mutants. They passed the suite's own no-markdown judge because the model didn't emit markdown even with the rule weakened — flagged for a look, not a gap. Equivalent mutants are a normal part of a mutation report.
  • Read the interval, not the point. Three cases give a wide CI (11–52%); the adequacy lens flags it. More cases tighten the number.
  • Coverage, not correctness. Whether the suite is right needs labels — that's the validity lens.

muteval is a per-suite diagnostic: it shows where a suite has holes and names the fix. See the limits →

07Pointing it at your repo

Swap the example for your own suite. A few things to set, depending on how yours is built:

  • Your config path. Replace promptfooconfig.yaml with the path to yours. If it defines several prompts, muteval mutates the first — split them into separate configs to cover each.
  • Model & key. muteval runs the prompt through the model in your config's providers: block (any OpenAI-compatible endpoint), or the one you pass with --model / --base-url. Set the key your provider needs.
  • Which assertions carry over. contains/icontains (and -any/-all), not-contains, equals, not-equals, starts-with, regex, their not- forms, is-json (with a schema), and llm-rubric/model-graded become graded checks. javascript, python, similar and assert-set are skipped with a warning — add equivalent muteval checks if your suite relies on them. Output transforms and threshold/weight scoring aren't reproduced (reported in one warning).
  • Prompt variables. {{var}} placeholders in prompts and assertion values are filled from each test's vars (plus defaultTest vars and file:// refs; array vars run once per value).
  • Not on promptfoo? Point muteval at a Python config instead — same two reports, any pipeline. See Adopting muteval.