Metadata-Version: 2.5
Name: eval-builder
Version: 0.2.0
Summary: Turn real LLM app logs into an eval suite and measure which LLM judges you can trust.
Author: Abel Yagubyan
License-Expression: MIT
License-File: LICENSE
Keywords: deepeval,evals,inspect-ai,llm,llm-as-a-judge,mcp,promptfoo
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: mcp>=1.30
Requires-Dist: numpy>=1.24
Requires-Dist: pyyaml>=6.0
Requires-Dist: resvg-py>=0.2
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.10
Description-Content-Type: text/markdown

# eval-builder

**Is your LLM judge a coin flip?** One command tests it for position bias, flip-flopping
and padding bias, with no labels, then turns your real logs into an eval suite.

```sh
uvx eval-builder judge-check ollama:llama3.2:3b
```

`<judge>` is a model spec with no adapter to write: `ollama:<model>`, `openai:<model>`
(or any OpenAI-compatible base URL), `anthropic:<model>`, or your installed agent CLI
(`claude-code`, `codex`). With no logs it uses a bundled, human-labeled sample so it can
also report agreement with people. Local models run for free; paid APIs ask first.

It judges the same answer pairs repeatedly, with the order swapped and with an irrelevant
paragraph appended, and prints a verdict:

```
Coin flip risk: HIGH   (judge-check verdict: unstable)

Flips on 55% of identical repeats; follows the first-shown answer 64% of the time;
prefers the padded answer 0% of the time.

  Flips on identical repeats                55%  CI 34 to 74%
  Follows the first-shown answer            64%  CI 56 to 71%
  Prefers the padded answer                  0%  CI 0 to 16%
  Agreement with humans (80% acc)    kappa 0.60  CI 0.25 to 0.95

20 cases, 220 calls in 122.6s
card (svg): judgecheck/card.svg
```

…and a 1200×630 reliability card you can post:

<img src="docs/judge-card.svg" alt="Judge reliability card for ollama:llama3.2:3b: coin flip risk HIGH, flips on 55% of identical repeats, follows the first-shown answer 64% of the time" width="640">

### Three local judges on the bundled sample

Each judge graded the same 20 human-labeled MT-Bench answer pairs: 5 times per pair, 5
more with the two answers swapped, and 5 more with an irrelevant paragraph appended to
one answer, so 220 calls per judge (0 errors, 0 unparseable outputs). Run on this Mac
(Apple M4, 16 GB) at the default settings, one model at a time:

| judge | verdict | coin-flip risk | flips on repeats | follows the first-shown answer | survives answer swap | agreement: accuracy / kappa |
|---|---|---|---|---|---|---|
| llama3.2:3b | unstable | HIGH | 55% [34%, 74%] | 64% [56%, 71%] | 50% [30%, 70%] | 80% [58%, 92%] / 0.60 [0.25, 0.95] |
| qwen2.5:3b-instruct | biased | MEDIUM | 10% [3%, 30%] | 70% [63%, 77%] | 35% [18%, 57%] | 70% [48%, 85%] / 0.40 [-0.00, 0.80] |
| qwen2.5:7b-instruct | biased | MEDIUM | 10% [3%, 30%] | 66% [59%, 73%] | 60% [39%, 78%] | 60% [39%, 78%] / 0.20 [-0.23, 0.63] |

n = 20 pairs per judge; brackets are 95% intervals; padding moved no verdict toward the
padded answer (0%, interval up to 16%). Wall time 123 s, 138 s and 270 s. None of the
three is safe to grade with as is: llama3.2 changed its verdict on more than half of
identical repeats, and both qwen models lean toward whichever answer they saw first and
survive the swap on well under 80% of pairs. These numbers are about these judges, this
prompt and these 20 pairs; wider runs (`--pairs 80`) tighten the intervals.

The same command drives a hosted or agent-CLI judge. As a non-Ollama check,
`eval-builder judge-check claude-code` (Claude Haiku through the installed `claude` CLI)
on a tiny 6-pair run cleared every probe (no order preference, kappa 0.67 with the human
experts), 42 calls, 0 errors, in 235 s. Six pairs is far too few to conclude much, and
agent CLIs are slower than a local model, but it shows the `openai`/`anthropic`/
`claude-code`/`codex` path works end to end.

This builds on the owner's paper, [*The Coin Flip Judge? Reliability and Bias in
LLM-as-a-Judge Evaluation*](https://arxiv.org/abs/2606.13685) (Yagubyan, GroundLM @ EMNLP
2026): the position, repeat and padding probes follow it. Pass `--prompt judge.txt` to
test your own judge prompt instead of the built-in one.

## Turn your real logs into an eval suite

The other half: build a reusable suite from your own traces. Try it on 48 sample
conversations (no account, no API key, no model needed for these steps):

```sh
curl -sLO https://raw.githubusercontent.com/Abelo9996/eval-builder/main/examples/sample_logs.jsonl
uvx eval-builder ingest sample_logs.jsonl     # or your own logs: OpenAI, Anthropic, Langfuse, OpenTelemetry, JSONL
uvx eval-builder select -n 10 --stratify category && uvx eval-builder draft
```

![The quickstart running against eval-builder 0.1.1 from PyPI](docs/demo.gif)

What you get (real output of the commands above, eval-builder 0.1.1 from PyPI; the
curl and the three uvx calls took 35 s in total here; the very first uvx run also
downloads about 38 MiB, mostly numpy, scipy and scikit-learn, which took 6 s here):

```
ingested 48 traces into evalset/traces.jsonl
  sample_logs.jsonl: format=openai records=48 traces=48 skipped=0 sha256=2d5dd2295784988b
redactions: 0 {}
next: eval-builder select -n 30 (add --stratify <metadata keys> to cover them)
48 traces -> 16 unique (32 exact dupes, 0 near dupes) -> selected 10 (8 failures) across 4 clusters
  q121-llama-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
  q81-alpaca-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
  q102-alpaca-13b: covers category=reasoning (2 unique traces, 12%)
  q111-alpaca-13b: adds variety within cluster 1 (8 traces, 50%; triangle, response, person); least similar to cases already picked there
  ... (6 more picks)
next: run draft to turn the selection into cases.yaml
10 case(s) added, 10 total in evalset/cases.yaml
next: read the cases (list_cases, or cases.yaml), define criteria and judges (set_rubric, or rubric.yaml), fill expected_behavior and criteria per case and set status: ready (update_case), then run validate
```

`evalset/cases.yaml` now holds 10 real conversations with a TODO where the expected
behavior goes. Writing that, running judges and reading the statistics is the part your
coding agent does, through the MCP server (next section).

eval-builder is not another eval platform. It builds the suite and checks the judges,
then exports to the tools you already run: promptfoo, DeepEval, Inspect AI, or plain
JSONL. Only `judge-check <model>` calls a model (the judge you point it at); every
suite-building step runs locally and calls nothing. Your coding agent (Claude Code,
Codex, Cursor) does the thinking; eval-builder does the selection, the bookkeeping and
the statistics, and writes down the evidence.

## Use it with your agent

```sh
uvx eval-builder setup --yes    # registers the MCP server and the skill, then restart your agent
```

Then type something like this into Claude Code, from the folder that has your logs:

> Our assistant's chat logs from last month are in ./logs. Build a small eval suite from
> them (about 10 cases) that I can run in promptfoo, and tell me whether I can trust a
> local judge for grading it. I have Ollama running with qwen2.5:7b-instruct and
> llama3.2:3b. Keep the judge runs small.

This is from a real headless Claude Code session (October 2026, Claude Code 2.1.293)
given exactly that prompt and 48 conversations from [`examples/sample_logs.jsonl`](examples/sample_logs.jsonl).
It ran ingest, select, draft, list_cases, set_rubric, ten update_case calls, validate,
judge_plan, 140 judge calls to Ollama through a short script it wrote, judge_check,
export and report in 32 turns and about 10 minutes. From its final answer:

> I built a 10-case promptfoo suite, but **you can't trust either local judge yet.**
> Neither one has been checked against human labels, and each confidently got an
> obvious case wrong.
>
> | judge | flip rate | padding helped | verdict |
> |---|---|---|---|
> | qwen2.5:7b-instruct | 10% [2%, 40%] (n=10) | 0% [0%, 28%] | not_enough_data |
> | llama3.2:3b | 20% [6%, 51%] (n=10) | 0% [0%, 28%] | not_enough_data |
>
> **qwen7b failed a clear case.** It passed the Hawaii rewrite 5 out of 5 times, though
> only the first sentence starts with "A".
>
> **No grader is wired in.** Because no judge passed, the `llm-rubric` checks fall back to
> promptfoo's default grader, which needs an OpenAI key. To grade locally anyway, export
> again with the judge set explicitly (the `export` tool's `judge` option, e.g. `qwen7b`).
> Its verdicts will be unverified.

It wrote every expected behavior itself because nobody was there to confirm them, and
said so on each case. It did not invent human labels; it asked for them. That is where
this session and the earlier one both stopped, so 0.1.2 adds the missing step: `label`
writes a sheet the person labels in their browser, and `label import` brings the
labels back for judge-check ([Human labels, end to end](#human-labels-end-to-end)).

An earlier session against 0.1.0 is where the promptfoo context bug fixed in 0.1.1 came
from. The agent read the export and told the user: "Right now it only sends the final
question (`{{input}}`), so your app won't see what "it" refers to in "Can you
parallelize it?"."

## Example: real output

Everything below comes from runs on this machine (Apple M4, 16 GB) that are committed
in [`examples/mt-bench/`](examples/mt-bench/). The data is the LMSYS
[MT-Bench human judgments](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments)
dataset (CC BY 4.0): 80 two-turn questions answered by 6 models, plus pairwise verdicts
from expert human judges.

**Logs to suite.** `ingest` read 480 conversations (OpenAI chat format, redaction on,
0 matches). `select -n 24 --stratify category` removed 400 exact duplicates (each
question was asked to six models), clustered the 80 unique conversations into 9 topics
and picked 24 covering all 8 categories, 19 of them failures. From
[`suite/report.md`](examples/mt-bench/suite/report.md):

```
| trace           | cluster | represents | why it was picked |
| q81-alpaca-13b  | 0 | 6 | failure (negative user feedback); 62% of unique traces are failures and at least 30% of picks are reserved for them |
| q82-alpaca-13b  | 0 | 6 | adds variety within cluster 0 (11 traces, 14%; response, previous response, previous); least similar to cases already picked there |
| q156-alpaca-13b | 3 | 6 | covers category=humanities (10 unique traces, 12%) |
```

The agent wrote expected behavior for the 24 cases, and `export` produced files that
Inspect AI 0.3.277, DeepEval 4.2.8 and promptfoo 0.124.0 all load (see
[`verify_exports.sh`](examples/mt-bench/verify_exports.sh)).

**Which judges can you trust?** Five local LLM judge setups (four models through
Ollama, one of them also at temperature 0) and two controls that are not language models judged 80 answer pairs where human experts
picked a winner. Each judge ran 5 times per pair, 5 more with the answers swapped, and
5 more with an irrelevant paragraph appended to one answer: 8,400 calls, 0 errors.
`eval-builder judge-check`, from [`judges/report.md`](examples/mt-bench/judges/report.md):

| judge | verdict | flip rate | accuracy vs humans | kappa | survives answer swap | first-shown answer picked | padding helped |
|---|---|---|---|---|---|---|---|
| qwen2.5:7b-instruct, temp 0 | biased | 0% [0%, 5%] | 72% [62%, 81%] | 0.45 [0.26, 0.65] | 75% [65%, 83%] | 44% | 0% |
| qwen2.5:7b-instruct, temp 0.8 | biased | 12% [7%, 22%] | 70% [59%, 79%] | 0.40 [0.20, 0.60] | 75% [65%, 83%] | 42% | 1% |
| qwen2.5:3b-instruct | biased | 12% [7%, 22%] | 56% [45%, 67%] | 0.13 [-0.08, 0.35] | 44% [33%, 55%] | 35% | 9% |
| gemma2:2b | unstable | 59% [48%, 69%] | 56% [45%, 67%] | 0.14 [-0.07, 0.36] | 38% [28%, 48%] | 35% | 11% |
| llama3.2:3b | unstable | 56% [45%, 67%] | 51% [40%, 62%] | 0.04 [-0.17, 0.26] | 14% [8%, 23%] | 20% | 10% |
| control: longer answer wins | biased | 0% [0%, 5%] | 66% [55%, 76%] | 0.32 [0.12, 0.53] | 100% [95%, 100%] | 50% | 30% |
| control: coin flip | unstable | 95% [88%, 98%] | 51% [40%, 62%] | 0.03 [-0.19, 0.24] | 55% [44%, 65%] | 50% | 25% |

n = 80 pairs per judge; brackets are 95% intervals. No judge passed. The closest,
qwen2.5 7B, reaches kappa 0.40 to 0.45 with the human experts, but its verdict
flips on a quarter of the pairs when the answer order is swapped, and setting the
temperature to 0 removes the run-to-run flips without removing that order effect.
llama3.2 3B picked whichever answer it saw second 80% of the time. The longer-answer
rule never flips and agrees with humans 66% of the time, which is why stability and
accuracy alone are not enough: the padding probe catches it.

The same check on the suite's own pass/fail judge (qwen2.5 7B, 24 cases, no human
labels) gives `unstable`: its verdict changed across 5 identical calls on 7 of 24
cases (29%, interval 15% to 49%).

### Human labels, end to end

```sh
uvx eval-builder label                                   # picks 24 cases, writes label_sheet.html
uvx eval-builder label import ~/Downloads/labels.jsonl   # what the sheet's Export button saved
uvx eval-builder judge-check
```

![The labeling sheet: one case per screen, pass and fail buttons with keyboard shortcuts](docs/label-sheet.png)

Run on the 48 sample conversations with every model's answer kept as a case (47
cases), three local judges (1,128 calls, 0 errors), and 24 labels entered through the
sheet in headless Chrome. The labels were made on the developer's side (Claude Code
reading each case), not by an independent annotator, so read this as a demonstration
of the flow. Details and every file: [`examples/sample-labeling/`](examples/sample-labeling/).

- `label` split the 24 picks 12/12 between cases the judges called pass and fail; 15
  are cases where the judges disagree, flip or move under padding.
- The sheet made no network requests, survived a reload mid-way, and its download was
  byte-identical to the copy box. `label import` took 24 rows, rejected 0.
- `judge-check` against those labels (fail 18, pass 6):

| judge | verdict | flip rate | accuracy vs labels | kappa |
|---|---|---|---|---|
| qwen2.5:7b-instruct, temp 0 | misaligned | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] |
| qwen2.5:7b-instruct, temp 0.8 | misaligned | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] |
| llama3.2:3b | unstable | 47% [33%, 61%] | 54% [35%, 72%] | 0.12 [-0.26, 0.50] |

The qwen judges clear the kappa threshold (0.4), but their accuracy is no better than
always answering "fail" (75% of the labels), so since 0.1.3 judge-check calls them
`misaligned` instead of trustworthy (0.1.2 called them trustworthy; that was the bug).
Every miss went the same way: both passed a reply that said "Here is an allegorical
poem" and then wrote no poem. With 24 labels the kappa interval runs from about 0.1
to 0.8. Under 0.1.2 the DeepEval export wired the 0.8-temperature qwen judge in with
its exact prompt; DeepEval 4.2.8 ran three logged cases through it and it made the
same call on the missing poem. Now no judge is wired in unless you force one with
`export --judge <id>`, which warns.

## How it works

| step | what it does | what it uses |
|---|---|---|
| `judge-check <model>` | The one-command front door. Resolves a model spec (`ollama`, `openai`/OpenAI-compatible, `anthropic`, `claude-code`, `codex`), builds cases from a bundled human-labeled sample or your `--logs`, runs the judge over them with the repeat, swap and pad probes, and prints a coin-flip risk level plus a 1200×630 SVG reliability card. The only part of eval-builder that calls a model; local models are free, paid APIs need `--yes`. | your model; standard library |
| `ingest` | Reads OpenAI chat JSONL (fine-tuning style or request/response logs), Anthropic messages, Langfuse trace exports, OpenTelemetry GenAI spans (OTLP JSON, current `gen_ai.input.messages` and older `gen_ai.prompt.N` attributes), or generic input/output JSONL. Normalizes to one schema with input, output, earlier turns, tools called, error flag, user feedback, route and model. Redacts emails, API keys, Luhn-valid card numbers, phone numbers and SSN-shaped numbers by default and reports counts. | Python standard library |
| `select` | Removes exact duplicates (normalized hash of the user turns) and near-duplicates (character n-gram cosine), clusters the rest by topic, then picks: failures first (errors, negative feedback), at least one case per stratum (route, tool, feedback, any metadata key you name), one central case per cluster, and fills the rest by cluster size with the least similar remaining cases. Every pick records why it was picked. | scikit-learn (TF-IDF, k-means) |
| `draft`, `validate` | Writes `cases.yaml` and `rubric.yaml` with TODO markers. The agent fills in expected behavior and criteria with you. Validation refuses ready cases that still contain TODO or reference unknown criteria. | PyYAML |
| `judge-plan` | Lists every judge call to make: each case N times, plus probes that swap the answer order (pairwise judges) and pad an answer with an irrelevant paragraph. | |
| `judge-run` | Optional and off by default. Sends each request as a JSON line to a command you name (your script, your provider, your keys) and records the verdicts. eval-builder ships no API keys and no provider code. | your command |
| `label`, `label import` | Picks the ready cases a person should label (default 24): the budget is split across outcomes (the judges' consensus, or the logged failure flag before judges ran) and up to half of each share goes to cases where judges disagree, flip across repeats or move under padding. Writes `label_sheet.html`, one offline file (one case per screen, pass/fail or A/B buttons, a note, keyboard shortcuts, judge verdicts hidden, progress kept in the browser) whose Export button downloads `labels.jsonl` in the format judge-check reads, and `label_sheet.csv` for spreadsheet users. `label import` checks the file against `cases.yaml` and merges it into the workspace. | Python standard library; the sheet is plain HTML and JavaScript |
| `judge-check` | Per judge: flip rate across repeated calls (with a Wilson interval), self-agreement, majority-of-3 vote stability, accuracy and Cohen's kappa against your human labels (with intervals), position consistency and first-shown preference, and how often padding moved the verdict toward the padded answer. Verdict: `trustworthy`, `unstable`, `biased`, `misaligned`, or `not_enough_data`, with the numbers behind it. | |
| `export` | promptfoo `promptfooconfig.yaml` (the full conversation as chat messages, llm-rubric asserts, and a judge that passed judge-check wired in as the grader when its rubric entry names a promptfoo `provider`), DeepEval dataset plus a `deepeval test run` file (the same checked judge grades every case with its exact prompt; without one, GEval), Inspect AI dataset plus `task.py` (Inspect's default `model_graded_qa` grader), plain JSONL. The manifest lists file hashes and which judges passed. | |
| `report` | `report.md` and `report.json`: sources with sha256, counts, redactions, selection reasons, the judge table, and the limits. | |

The verdict thresholds are explicit flags with defaults: flip rate at most 20% of cases,
position consistency at least 80%, padding helps at most 10% of cases, kappa at least
0.4 on at least 20 human-labeled cases. A judge without human labels is never called
trustworthy. When at least 80% of the selected cases or of the human labels share one
outcome, `select` and `judge-check` say so with the real proportions, because a judge
that always gives that answer would look accurate on them.

## Setup for agents

```sh
uvx eval-builder setup          # shows what it would change
uvx eval-builder setup --yes    # applies it; restart your agent afterwards
```

`setup` registers the MCP server (`uvx eval-builder mcp`) with Claude Code (`claude mcp
add --scope user`), Codex (`[mcp_servers.eval-builder]` in `~/.codex/config.toml`) and
Cursor (`~/.cursor/mcp.json`), and copies the agent instructions to
`~/.claude/skills/eval-builder/` and `~/.codex/skills/eval-builder/`. It backs up any
file it edits and does nothing on a second run. The workflow the agent follows is in
[`skills/eval-builder/SKILL.md`](skills/eval-builder/SKILL.md).

MCP tools: `judge_check_now` (the one-command front door), `ingest`, `select`, `draft`,
`list_cases`, `update_case`, `set_rubric`, `validate`, `judge_plan`, `label`,
`label_import`, `judge_check`, `export`, `report`, `status`. The judge runner
is CLI only, because it executes a command.

## What it can't do

- It does not write expected behavior or human labels. The agent drafts expected
  behavior with you; labels must come from people. Without labels, judge-check can
  tell you a judge is unstable or biased, but not that it is right. `label` makes the
  labeling quick, but someone still has to read each case.
- Picking labels where judges disagree makes each label more informative, but those
  cases are harder than average, so accuracy measured on them leans pessimistic.
  `--uncertain-share 0` picks by outcome and topic only.
- The labeling sheet is a static page, so it cannot save files: the person has to
  click Export (or copy the text) and import it. Unexported progress lives only in
  that browser's local storage.
- Selection is lexical. Two requests that mean the same thing in different words can
  land in different clusters, and near-duplicate detection only catches close textual
  matches.
- Redaction is pattern-based. Names, street addresses and free-form secrets get
  through. Look at `traces.jsonl` before sharing a workspace.
- The bias probes cover answer order and irrelevant length only. Self-preference,
  style bias and rubric misreadings are not measured.
- Small samples give wide intervals. The report prints them; read them.
- It does not run your app, your judges or your eval. The agent (or a script you name
  with `judge-run`) calls the judge model; the exported files run the eval in promptfoo,
  DeepEval or Inspect AI.
- A checked judge is wired in as the grader in promptfoo and DeepEval, and only a
  pointwise one. promptfoo needs its prompt to answer in JSON (`{"pass": ...,
  "reason": ...}`), because llm-rubric cannot parse a bare "pass". The DeepEval test
  calls Ollama judges itself; for any other provider you fill in `call_judge`. The
  Inspect AI export still uses Inspect's default `model_graded_qa` grader, not your
  checked judge.

## Privacy and safety

The suite-building steps run locally and make no network calls. The exceptions are
explicit: `judge-check <model>` calls the judge model you name (a local Ollama server,
an OpenAI-compatible or Anthropic API with your key from the environment, or your
installed `claude`/`codex` CLI), and paid APIs do nothing until you pass `--yes`;
`judge-run --enable-judge-plugin` starts the judge command you name. eval-builder ships
no API keys. The labeling sheet is one HTML file with a content
security policy that blocks every network request; it keeps unexported labels in the
browser's local storage on that machine. Exported test files can call a model when
you run them (the DeepEval test calls the judge it was given). Redaction is on unless you pass `--no-redact`, and
the report lists what was replaced (counts and kinds, never the values).

## License

MIT. See [LICENSE](LICENSE).
