Cheat sheet: what goes where
| Field | Put this | Why |
|---|---|---|
| Task and constraints | Describe the task in your own words | The task type is detected from keywords, so plain English works best. |
| Model | qwen2.5:7b | The exact model name in Ollama. Fast, not a thinking model. |
| Model class | medium | Rough size. 3B → small, 7–13B → medium, bigger → large. Some techniques are ruled out for small ones. |
| Provider | ollama | Where the model runs. The other option is your own OpenAI-compatible endpoint. |
| Structured output | ✅ | The model can be forced to emit JSON matching a schema. Required for extraction. |
| System messages | ✅ | It understands a system instruction. Almost everything does. |
| Tool calling | ❌ | Only for agent tasks that call tools. |
| Reasoning control | ❌ | Only if the model is a thinking one and you want to speed it up by turning that off. |
| Dataset | multiconer-en | What to measure against. Use 100+ examples or the numbers mean nothing. |
| Repeats | 3 | How many times each example runs. At 1, stability reports a fake 1.000. |
| Optimizer rounds | 2 | Only used by the Optimize button. Ignored by Benchmark and Compare. |
| Optimizer search | DSPy MIPROv2 | Also Optimize only. Of the four, this is the one that actually improved anything. |
Button order: Create my prompt → Create with this technique → Benchmark this prompt → Compare all recommended. Optimize the prompt last, if you want to push further.
What this does
The usual way to write a prompt is to guess, then keep changing it until it feels better. There is nothing to check that feeling against.
This works differently: the tool picks a way to build the prompt, shows you the finished text, and runs it on a real model across dozens of examples. At the end you get numbers instead of impressions.
Four buttons, in order
Describe your task in your own words. You get the ranked methods with the reasoning for each, and the prompt for the top one is ready to copy.
On any other method in the list. Shows the finished prompt that one compiles to — the exact text that will be sent to the model.
Runs that prompt on a live model across every example in the dataset. This is the button that matters.
The same thing for all three techniques at once, so you can pick the best.
The two numbers that matter
quality — how correct the answers are
It gives partial credit. If the model found three of the four things it should have, that is roughly 0.85, not zero. So 0.78 means "mostly copes, but regularly drops one".
reliability — whether the answer is shaped correctly
This is about form, not meaning. The answer came back the way it should: valid JSON, all required fields, nothing extra around it.
Do not confuse the two. You will see reliability 1.00 next to quality 0.4. That means the model is neatly and correctly formatting the wrong answers.
The other numbers
mean latency | Seconds spent on one example. |
mean tokens | Tokens spent on one example. That is money, if the model is paid. |
mean calls | How many times the model is called per example. More calls means slower and pricier. |
failures | Examples that did not run at all. Should be 0. |
stability | Whether the model gives the same answer to the same question. |
Three ways to get burned
1. Set Repeats to 3. At Repeats = 1 stability always reads 1.000. That does not mean "stable", it means "nobody checked". The model can answer differently every time and you would never see it.
2. Few examples, worthless numbers. On 6 examples a 5% difference means nothing. Use a dataset of a hundred or more if you intend to decide anything from the result.
3. Numbers belong to one model. Measure on one model and it is a guess again for another. The tool says measured when it measured, and prior only when it is guessing.
What to do about it
| What you see | What to do |
|---|---|
| Low quality, high reliability | The shape is fine, the meaning is not. Look at Weakest examples at the bottom — the three worst cases. Low quality usually hides one repeated mistake. |
| Low reliability | The model is padding the answer with chatter. Tick Structured output — then the format is enforced by the runtime instead of politely requested. |
| failures above zero | The model is too slow to answer in time. Use a faster one — "thinking" models can spend 40 seconds per example. |
| Good but expensive | Compare against a simpler technique. Three model calls often do not pay for themselves in quality. |
| Numbers jump between runs | Too few examples, or Repeats = 1. Raise both. |
The Optimize button
It tries rewriting the prompt itself and checks whether that helped.
Look only at the Held-out validation table — that is the check on examples the optimizer never saw. If it shows zeros or a minus, nothing improved, whatever the table above it says.
This happens often. The optimizer easily fits itself to the examples it trained on and does no better on new ones. That is normal — just do not trust the numbers from the upper table.
Your own data
The bundled examples are for finding your way around. Draw conclusions from your own. One task per line:
{"id": "1", "input": "Mara entered Veyr with Captain Orin.",
"expected": {"people": ["Mara", "Captain Orin"], "places": ["Veyr"]}}Only id and input are required. The rest is worked out for you.
One rule. The correct answer has to appear in the text word for word. If the text says "Captain Orin" and your answer says "Orin, captain", the model is marked wrong for not guessing your way of writing it.
Details — if you need to dig deeper
Where quality comes from
It is the field_f1 metric: it counts both what the model missed and what it invented. In the Graders table it is marked headline. Next to it sits exact_match — did everything match outright, with no allowances. It is almost always far lower, and that is expected.
The format checks live there too: json_validity (did it parse), json_schema (did it match the schema fully), no_prose (no chatter around it). If json_validity is 1.00 while field_f1 is low, the problem is comprehension, not format.
Where reliability comes from
It is the share of correctly formatted answers multiplied by stability. Hence the catch with Repeats: on a single run stability is 1 by definition, so reliability looks inflated.
p95 latency
The time 95% of examples fit inside. If it is far above the mean, there are occasional very slow answers. For interactive use that matters more than the mean.
Measured versus Declared
Declared is the estimate written into the technique's config before anything was measured. Measured is what actually happened. A gap between them is the point, not a mistake — once measured, the config stops being used.
Comparing techniques
The Weighted column is the combined score under your weights. But if token cost carries any weight, a technique can win by being cheap while being worse at the job. Check the Quality column too.
The latency and token columns in a comparison are relative — computed against the best technique in that same run, so they do not transfer to other runs.
Pareto front in optimization
The variants that nothing beat on every measure at once. One of them is sometimes both more accurate and cheaper than the declared winner.
Prompt stages
Some techniques split the prompt into several stages, which means several model calls in a row. Labels like {previous} mark where the previous stage's output gets inserted at run time.
Empty examples in a dataset
Keep cases where the correct answer is empty. Without them, a prompt that guesses looks good, because there is nothing for its mistakes to show up on.
More depth in README.md and the docs/ folder.