What a benchmark is
A benchmark is an exam for a prompt.
Take a pile of tasks with known correct answers. Run your prompt on a live model across all of them. Compare what the model said against the correct answer, and count the hits. Out comes a number.
The point is to replace "I think the prompt is better this way" with "0.83 against 0.71 on two hundred examples".
The exam has three parts:
The tasks themselves and their correct answers. One task per line of a file.
Small pieces of code that compare the model's answer to the correct one and score it from 0 to 1. Each task type has its own — arithmetic and translation cannot be checked the same way.
Which model, how many repeats, which settings. This is what decides whether the number is repeatable or accidental.
The datasets that ship here
Eleven sets are included. Four come from public research corpora; the rest were built here — partly hand-written, partly generated from fixed rules with a fixed random seed, so they rebuild byte for byte.
| Dataset | Examples | What it checks | Where from |
|---|---|---|---|
entity-extraction | 6 | Pull people and places out of a sentence into JSON. | A demo for clicking around |
entity-extraction-hard | 200 | The same, with traps: titles, negation, countries used as adjectives, duplicates, cases where the answer is empty. | 40 by hand, 160 generated |
multiconer-en | 200 | People and places in sentences picked for being ambiguous. | public MultiCoNER v2 |
few-nerd | 200 | People, places, organizations. | public Few-NERD |
gsm8k | 120 | Grade-school word problems needing several arithmetic steps. | public GSM8K |
mbpp | 80 | Write a Python function from a description. | public MBPP |
support-classification | 150 | Sort support tickets into four categories. 54 examples are deliberately borderline. | Generated |
summarization | 120 | Compress a document to 170 characters without losing a single name, number or date. | Generated |
translation | 120 | Translate English to Spanish while applying a binding glossary. | Generated |
grounded-qa | 120 | Answer strictly from the attached sources. In 30 examples the answer is not there — the model must say INSUFFICIENT_EVIDENCE instead of inventing one. | Generated |
agents | 120 | Tasks that require calling a tool: a calculator or a word counter. | Generated |
Do not draw conclusions from entity-extraction. Six examples, and the baseline prompt already scores 1.0 on them — there is no headroom, so every technique looks equally good. It exists to show you the interface. The sets with real headroom are entity-extraction-hard, multiconer-en and few-nerd.
What those public corpora are
A public corpus is a set of tasks that somebody else's research group collected and annotated, then released openly. The value is that nobody here tuned the data to produce a convenient result.
| Corpus | What it is, plainly | Licence |
|---|---|---|
| MultiCoNER v2 SemEval-2023, task 2 | A name-recognition competition assembled specifically from hard cases: names that look like ordinary words, rare names, ambiguous phrases. The closest public analogue of our entity-extraction-hard. | CC-BY-4.0 |
| Few-NERD | A large annotated set of sentences with people, places and organizations. | CC-BY-SA-4.0 |
| GSM8K OpenAI | Grade-school "the shop had this many, sold that many" problems. This is the set where "think step by step" was shown to help. It is here as a counterweight: every other dataset we ship is reading rather than counting, and on reading, reasoning usually gets in the way. | MIT |
| MBPP Google Research | Simple programming tasks, each shipping its own tests. Scored by running the code, not by comparing text. | CC-BY-4.0 |
Importing is done with prompt-playoff import-hf; the licence and the paper citation are printed on every import.
How the import keeps the data honest
- The gold answer is sliced out of the finished sentence. Otherwise the name in the answer and the name in the text could differ by a space, and the model would be marked wrong for somebody else's spacing.
- An entity type the schema does not ask for becomes an empty example, not a wrong one. If the only name in a sentence is a film title while the schema asks for people and places, the correct answer is empty rather than a penalty.
- A share of empty examples is kept on purpose. Drop every case where there is nothing to answer, and a prompt that simply guesses starts looking excellent: its extra answers have nowhere to show up.
- Corpus limitations are recorded, not hidden. In Few-NERD, for instance, two adjacent entities of the same type merge into one — that is the corpus's own property, and it is stated in the preset's notes.
How an answer gets checked
A grader takes the model's answer and the correct one and returns a number from 0 to 1. They come in two families.
Meaning — is the answer right
field_f1 | Partial credit over a list. Three entities out of four is roughly 0.86, not zero. Extras that were not in the gold answer are penalised too. This is the headline score for extraction. |
exact_match | Did the whole thing match, with no leniency. Almost always far below field_f1, which is normal. |
label_accuracy | Was the category right. For classification. |
numeric_close | Did the number match. Formatting is ignored — only the value counts. |
unit_tests | The code from the answer is actually run against the task's own tests. The score is the share that pass. Execution happens in a sandbox with a restricted set of modules. |
glossary_consistency | The share of required terms translated the way the glossary demanded. |
grounding_overlap | How much of the answer's wording actually occurs in the supplied sources. A blunt but working check against invention. |
contains_all | Did every required fact make it into the answer. |
coverage | Recall only: how much of what was wanted showed up. Extras are not penalised. |
tool_success | Did every tool call return a result rather than an error. |
agreement | How much several answers agree with each other, when the technique generates a batch. |
Form — is the answer shaped right
json_validity | Did the answer parse as JSON. |
json_schema | Did it match the schema completely. |
schema_shape | The share of required fields that are present. Partial credit on form. |
no_prose | Is there chatter wrapped around the answer. |
allowed_labels | Is the label from the allowed list rather than invented. |
length_limit | Did the answer fit the character limit. |
omission_check | Is the answer truncated or bloated — its length is compared against the source. |
deduplication | Are there repeats inside the lists. |
python_syntax | Does the code the model produced parse. |
regex_match | Did the answer match the required pattern. |
There is no LLM judge here. Every grader is ordinary code. The same answer always gets the same score, checking costs nothing and takes no time, and you can read the rule yourself.
The price is that "is it well written" cannot be scored. So summarization is measured as "are all the facts there and did it fit the limit", and translation as "was the glossary applied and did half the text go missing". About style, the benchmark says nothing.
How the graders become the two headline numbers
quality is one chosen meaning-grader, whichever fits the task: field_f1 for extraction, label_accuracy for classification, numeric_close for arithmetic, unit_tests for code. In the Graders table it is marked headline; the others are shown beside it for reference.
reliability is the share of correctly shaped answers multiplied by stability (whether the model gives the same answer to the same input).
What to do with those numbers is on the Help page.
Five rules, without which the number means nothing
1. A hundred examples, minimum. On six examples, a 5% difference is one example moving. On forty examples the threshold below which a difference is indistinguishable from noise is roughly 0.04. Anything smaller is not a result.
2. Repeats = 3. With a single pass, stability is 1.000 by definition — not because it is stable, but because there is nothing to compare against.
3. Empty examples are mandatory. Cases where the right answer is "nothing" keep a prompt honest. Without them, the guesser wins.
4. For optimization, read the held-out split only. The optimizer fits itself to the examples it saw. The gain only counts on the part it never saw.
5. A number belongs to one model-plus-dataset pair. Carry it to another model and it is a guess again. The tool says measured where it measured and prior only where it is assuming.
Which tools are involved
| What | What for | Install? |
|---|---|---|
| The built-in engine | Running examples, grading them, computing quality, reliability, stability, latency and tokens. No external dependencies. | Built in |
| Ollama | Local models on your own machine. The default route: free and no keys. | A separate program |
| OpenAI-compatible APIs | Cloud models: OpenAI, Anthropic, DeepSeek, Together, OpenRouter, Groq, Fireworks. Useful when the local model is too weak. | Needs a key in an environment variable |
| Hugging Face datasets | Importing public corpora (import-hf). Hugging Face is the main public warehouse of datasets and models. | Optional extra |
| DSPy | A library for automatic prompt search. Supplies three of the search algorithms behind the Optimize button. | Optional extra |
| promptfoo | Exports the prompt into another test runner, if your team already uses one. Only a technique's first stage is exported. | Optional extra |
| Langfuse / Phoenix | Tracing: they log every model call so you can look at exactly what went out and came back. | Optional extra |
The four search algorithms in Optimize
native | The built-in greedy loop: the model critiques its own prompt and rewrites it, and the best variant survives. Nothing to install. |
DSPy MIPROv2 | Bayesian search: it proposes instructions and example sets, and learns from the scores they got. In our measurements this is the only one that produced a gain. |
DSPy GEPA | Evolution: keeps a set of variants nothing has beaten outright and breeds from those. |
DSPy BootstrapFewShot | Leaves the instruction alone and only picks the examples that go into the prompt. |
What has already been measured
The finished reports live in docs/benchmarks/. In short:
- Thirteen techniques from the literature, on
few-nerd: eleven out of twelve lost to the plain baseline prompt. The ranking tracked token spend almost exactly, in the wrong direction: the more a technique "reasons", the worse it reads. Entity extraction is reading, not reasoning, and an intermediate restatement pulls the model away from the original text. - The built-in optimizer against MIPROv2 on
entity-extraction-hard, same data split for both: the greedy loop delivered exactly +0.000, MIPROv2 delivered +0.090 while using half as many model calls. - A large cloud model behaved the same way on reading: a separate reasoning stage cost 0.028 quality at 2.9× the tokens and 4.4× the time. On arithmetic both techniques hit the 1.000 ceiling, where no difference can show.
Your own data
The bundled sets are for finding your way around. Conclusions about your task can only come from your data. A .jsonl file, one task per line, uploaded straight from the interface:
{"id": "1", "input": "Mara entered Veyr with Captain Orin.",
"expected": {"people": ["Mara", "Captain Orin"], "places": ["Veyr"]}}Only id and input are required. Graders are picked from the shape of the expected answer; you can name them yourself in a graders field.
The one rule. The correct answer has to appear in the text word for word. If the text says "Captain Orin" and your gold says "Orin, captain", the model is marked wrong for not guessing your way of writing it. And keep the examples where the right answer is empty.
What the benchmark cannot do
- Score the quality of prose. There is no LLM judge — only checkable rules.
- Say anything meaningful on a tiny set. Six examples is not a measurement.
- Call arbitrary tools in agent tasks: only registered ones work, and a calculator is what ships.
- Export a multi-stage technique to promptfoo — only the first stage goes out.
- Carry a result over to a different model.
Per-dataset detail is in docs/datasets/; run reports are in docs/benchmarks/.