Benchmark report / 2026-09-21-golden-three-engines / executed 2026-09-21
Three engines on one ladder — 1.0.0 vs 2.0.1 vs HEAD, and no score
{{ONE_PARAGRAPH: what this run was for, and what it is not. A benchmark rules no threshold.}}
Arms
fux-engine 1.0.0 → fux-engine 2.0.1 → HEAD 7a88a165
{{how each was installed}}
Corpora
the golden ladder, 8 rungs, 28 → 10 000 documents
{{tiers run, and tiers not run}}
Questions
374
{{judged / timing / planted unanswerable}}
Classification
informed
no threshold ruled — SR-WORK-BENCHMARK decision 6
How to read this report / every number carries its direction
Which way is good, stated on every metric
| marker | means | metrics it sits on in this run |
|---|---|---|
| ↑ higher is better | a bigger number is a better engine | hit@1 · hit@5 · hit@10 · hit@20 · hit@50 · declined‑when‑unanswerable |
| ↓ lower is better | a smaller number is a better engine | query p50 · query p95 · ingest · build · index bytes · bytes/document · fabricated |
| — neither | a change is a signal, not a score. Nothing here says which value is better — a person reads the rows | queries whose list moved · first differing rank · shard count · headroom · b and c |
“Better” here means better on THIS instrument, and nothing more. A direction marker says which way the metric points; it never says the difference is real, large enough to act on, or a regression. That is a person reading the rows — SR‑WORK‑BENCHMARK decision 6.
A capture with no number for this run keeps its section and says so. It is never dropped for being empty — a missing section reads as a capture that was not required.
The arms / evidence/ARMS.toml
Two engines, byte‑checked against one corpus
| arm A | arm B | |
|---|---|---|
| version | fux-engine 1.0.0 | fux-engine 2.0.1 → HEAD 7a88a165 |
| install | {{...}} | {{...}} |
| python | {{...}} | {{...}} |
| corpus sha256 | {{...}} | {{... identical, or the run is void}} |
| queries sha256 | {{...}} | {{...}} |
| enrichment | {{present|absent}} | {{present|absent}} |
{{Anything that makes this run non-reproducible -- an editable install of a dirty tree, an uncommitted change, a machine that was not quiet -- is stated HERE, not discovered later.}}
The null control / run first, as it always is
Arm A against itself
PASS — 374 rows compared, 0 moved. v1.0.0 against itself on rung-00500, all three sets, two independent builds and two independent runs. Nothing else in this run was measured until this number existed.
Why it is first
Ordering is deterministic. A difference between two runs of the same arm is a broken harness, not a finding — and it would be invisible in every table after this one.
CAP‑1 / the ranked lists each arm returned
CAP‑1 — the ranked lists
The ordered document ids and scores, per query, per arm, per corpus. It is filed whether or not anything moved, because it is the artefact the next run is compared against.
{{the one-line finding, or: no number exists for this run}}
| arm | version | lists filed— neither | queries— neither | results per list— neither | top‑1 score— neither |
|---|---|---|---|---|---|
| A | fux-engine 1.0.0 | {{}} | {{}} | {{}} | {{}} |
| B | fux-engine 2.0.1 → HEAD 7a88a165 | {{}} | {{}} | {{}} | {{}} |
Nothing on this slide says which ordering is better — scores from two engines are not comparable to each other. What moved is CAP‑2's; this is what each arm returned.
CAP‑2 / what moved, arm A → arm B
CAP‑2 — what moved between the arms
v2 → HEAD changes the ranked list on 125 of 125 queries at every rung — and the cause is one measured knob. v2 ships b = 0.75, HEAD ships b = 0.15 (W-144). Length normalisation enters every score, so it reorders everything. A changed list is not a better list.
CAP‑3 / hit@k against the planted key
CAP‑3 — hit@k at 1, 5, 10, 20, 50
No number exists for this run — CAP-3 — hit@k is left EMPTY on purpose. It needs relevant from the answer key, which is not open. This is W-204 phase D’s, after Arpit pastes it. The rows it will be computed from are already filed under this run’s evidence/ — 8 976 of them — so nothing is re-run to fill this.
CAP‑3 / headroom, stated in both directions
How much could have moved at all
One score answers neither question. Improvement headroom is the questions wrong in both arms — the most that could have been fixed. Regression headroom is the questions right in both — the most that could have broken.
No number exists for this run — Headroom is left EMPTY on purpose. Headroom is a property of a scored endpoint, and no endpoint here is scored. Capture 2 counts movement; movement has no direction without a key.
CAP‑4 / answered, declined, and the planted unanswerables
CAP‑4 — the answer layer
No number exists for this run — CAP-4 — the answer layer is left EMPTY on purpose. It needs answerable from the key. Same reason, same phase. ⚠ 0 declines were recorded on any arm: every call returned text, so an abstain_correct computed from a null answer would be 0 by construction and would look like a measurement.
CAP‑5 / the committed index
CAP‑5 — the committed index size
HEAD commits 1.16× v1 at 10 000 documents (21 444 → 24 876 KB) — and v1 is indexing four fewer seed documents, so part of that is corpus rather than format.
CAP‑6 / speed, arms interleaved A B A B
CAP‑6 — the speed
DESCRIPTIVE, not a benchmark measurement: the arms were NOT interleaved. ask p50 at 10 000 documents is 237 / 322 / 304 ms for v1 / v2 / HEAD. ⚠ v1 is fastest and is doing less — four fewer documents, no confidence band. Faster is not better when the arms are not doing the same work.
Reader parity / Node against Python, over ONE index
Do the two readers agree?
🔴 Read this the opposite way from every other table on this report. Everywhere else a difference between two arms is the finding. Here the two arms are two readers of one committed index claiming identical output — same ids, same order, same locators, same band — so 0 discordant is the expected result and anything else is a defect in one of them. SR‑WORK‑BENCHMARK decision 16.
{{the one-line finding, or: no number exists for this run}}
| pair | queries— neither | identical lists↑ higher is better | discordant↓ lower is better; 0 is the only clean value | earliest differing rank— neither | max |Δscore|↓ lower is better |
|---|---|---|---|---|---|
| B | B‑node | {{}} | {{}} | {{}} | {{}} | {{}} |
max |Δscore| is the number that had never been taken — two readers can agree on ORDER while their scores differ in the last places, which is what a log()/libm divergence looks like before it changes an ordering.
⚠ CAP‑5 and the ingest/build half of CAP‑6 have no Node column, by construction — the Node reader writes no index. The arms are A · B · B‑node: a benchmark measures what ships, and the graph tier ships on, so there is no tier‑off column.
Guard rails / what this run may never be used to say
What this run does not do
It rules no threshold
SR‑WORK‑BENCHMARK decision 6 — no pass/fail anywhere. A bar is a pre‑registration with its own id space and its own verdict.
{{scope limit}}
{{tiers not run, paths not run, arms not compared}}
{{classification limit}}
{{what informed costs this run, or what blind buys it}}
{{captures with no number}}
{{named here as well as in their own section, so the gaps are countable from one slide}}
CAP‑7 has no slide of its own: CAP‑7 is this report — SR‑WORK‑BENCHMARK decision 14. Where it came from is on the cover. The run is work/regression/2026-09-21-golden-three-engines/ — report.md, ANALYSIS.md, evidence/. This report carries no number that run does not; anything that disagrees with it is this file being wrong. What must be here at all: records/0053_WORK-benchmark.md.