> ARCHIVED 2026-07-09 — REJECTED; onnx-int8 2x slower than torch, int8-speedup hypothesis failed; ADR-0043 + backend removed ADR-0067

## RESULT: REJECTED / NO-GO (2026-07-04)

**Quality:** exact parity with torch — controlled 30-question A/B, byte-identical
results: recall@10 0.9667, nDCG 0.9544, MRR 0.95. The earlier "onnx 1.0" run was
noise from an uncontrolled single run (recall swings 0.0↔1.0 per question); controlled
parity is confirmed neutral.

**Speed:** WORSE. CE median torch 3917ms vs onnx-int8 7899ms on `--cpus 2` = **2.0×
slower**. Even the unconstrained datapoint (0.83×) is slower, not faster → int8-speedup
hypothesis FAILED.

Caveat: the 2× slowdown is partly inflated by onnxruntime not capping `intra_op`
threads — `--cpus 2` is a CFS quota, not affinity; inside the container
`os.cpu_count()` = 24 → ORT spawns 24 threads thrashing the 2-core budget. A
code-level `SessionOptions.intra_op_num_threads` fix could reduce wall time (OMP env
vars alone are ignored by ORT for its intra-op pool, confirmed by follow-up test:
`OMP_NUM_THREADS=2` gave 7791ms, statistically identical to uncapped 7899ms, while
container CPU dropped 157%→103%). But even after accounting for thread-thrash, the
unconstrained 0.83× datapoint is still slower than torch → no latency win expected
even with a perfect thread fix.

**Decision:** keep `GTE_RERANKER_BACKEND=torch`. Branch `feat/ce-onnx-int8-v2`
unpushed, revivable if ORT thread-cap is implemented and retested. See **ADR-0043**
(supersedes ADR-0038). Data in `/tmp/ce_speed_ab.json`.

---

# PLAN — CE Lever 3: Activate GTE-ModernBERT onnx-int8 Rerank (2026-07-03)

**Task:** CE Lever 3 — onnx-int8 GTE-ModernBERT rerank (prompt "task #13"; the
warm-profile doc tracks the same work as **task #92**; v5.98 PR #144 landed the
**dormant** Lever-3 guardrail — this plan **activates** it). Reconcile the task id
with the live tasklist before opening the PR (see §8).

**Status:** DRAFT — plan only, no code, no PR.

**Master at draft time:** core v5.104 (commit `8c9abbf1`), backend_version **5.10.0**
(bumped in v5.98 `b79e24d3` for the dormant Lever-3 wiring).

**Hard constraint (non-negotiable):** backend is pinned `--cpus 2` and STAYS. User
refuses more cores ("not everyone can spare cores for yadgar"). Every design choice
tunes for 2 CPUs. No GPU, no core bump.

**Co-equal goals:** speed AND quality. A fast CE that returns worse memories is a
regression, not a win. Every change is gated on LongMemEval recall@k; regression =
abort, flip back to torch.

---

## 0. TL;DR / BLUF

The prompt frames this as "build onnx-int8 from scratch and prove ModernBERT can
even export." **Observed state says most of that is already done.** The int8 ONNX
artifact for `Alibaba-NLP/gte-reranker-modernbert-base` **already exists on
HuggingFace and is in the local HF cache** (`onnx/model_int8.onnx`); the wiring
(`GTE_RERANKER_BACKEND`, `GTE_RERANKER_ONNX_FILE`, `OnnxRerankerUnavailableError`
loud guard, I25 registry rows) **already shipped dormant in v5.98**; and the host
CPU **has `avx512_vnni`**, so int8 VNNI kernels are available and the 2–4× speedup
is realistic on 2 cores, not a mirage.

**So this plan is NOT an export project. It is a verify-and-activate project:**
(1) prove `onnxruntime` loads and runs the GTE-int8 model **inside the built backend
image** (the one genuinely-unverified link — libgomp on `python:3.14-slim-trixie`);
(2) quality-gate int8 vs fp32 on LongMemEval; (3) measure the latency delta on
`--cpus 2`; (4) if quality holds and int8 wins, flip `GTE_RERANKER_BACKEND=torch →
onnx-int8` as the new default.

**Go/no-go lean: cautious GO to the verification spike.** The two historical
blockers are resolved (feasibility proven by the pre-exported artifact; VNNI
present). The one open risk is image-runtime (onnxruntime/libgomp in the slim
backend image) and the quality delta — both are cheaply falsifiable in a spike
before any default flip. If the spike shows onnxruntime won't load in-image OR
recall@k regresses OR int8 doesn't beat torch on 2 cores, **NO-GO** and the fallback
levers in §7 apply.

---

## 1. Context

### 1.1 CE is the recall wall — but which number?

Two CE latency figures appear across the source docs and they measure **different
paths**. The plan keeps them separate on purpose:

| figure | path | source | what it means |
|---|---|---|---|
| **8.8 s = 87%** of a 10 s recall | **cold / concurrent** `retrieval.rerank.cross_encoder` | real trace (prompt), historical `subagent_start` profile (memory 518692: 88–91% CE) | first-touch or 6-concurrent burst — CE dominates the wall |
| **~720 ms** CE inference | **warm-HIT** (`_rerank_rpc`, ~271 ms × ~2.7 calls) | `recall-warm-profile-2026-07-02.md` §Appendix | steady-state repeat query; caches hot |

**int8 is a multiplier on inference time**, so its value **concentrates on the
cold + concurrent 8.8 s path**, where CE is 87% of the wall. On the warm-HIT path
CE is ~720 ms of a ~2.74 s total and the **bigger warm lever is the fusion N+1
(~950 ms)**, not CE — that is a *separate* plan (warm-profile §3 #1) and is not this
plan's job. **This plan targets the cold/concurrent CE wall.** The target section
(§6) reports each number against its own path; do not average them.

### 1.2 The constraint

Backend container: `--cpus 2`, runs SurrealDB **and** the rerank service, RAM-
constrained. Core is `--cpus 1`. No GPU. This is fixed. Any lever that needs more
compute to help (bigger batch, more threads than cores) is bounded by these 2 CPUs.

### 1.3 What already exists (observed, not assumed)

- **int8 artifact is UPSTREAM-published (verified via HF API 2026-07-03):** the HF
  repo `Alibaba-NLP/gte-reranker-modernbert-base` `siblings` list includes
  `onnx/model_int8.onnx` (150.9 MB LFS) alongside a **full ONNX family** —
  `model.onnx`, `model_fp16.onnx`, `model_bnb4.onnx`, `model_q4.onnx`,
  `model_q4f16.onnx`, `model_quantized.onnx`, `model_uint8.onnx`. The local cache
  blob hash (`ecc6a0ae…`) **matches the HF LFS oid exactly** → it was **downloaded,
  not locally exported**. ModernBERT→ONNX+int8 export is therefore a **solved,
  upstream-distributed** problem: no local export spike, and runtime-download works
  like any ST model (image stays flat). Provenance confirmed — this is the "flip a
  default," not "export + distribute a model," branch.
- **Wiring shipped dormant (v5.98, `b79e24d3` / PR #144):**
  - `GTE_RERANKER_BACKEND: str = "torch"` (config.py:277) — values `torch` | `onnx-int8`.
  - `GTE_RERANKER_ONNX_FILE: str = "onnx/model_int8.onnx"` (config.py:278).
  - Load: `LocalMLClient._load_gte_reranker()` (ml_client.py:356–407); `onnx-int8`
    branch loads `STCrossEncoder(model_id, backend="onnx",
    model_kwargs={"file_name": onnx_file})` (ml_client.py:381–386).
  - **Loud guard:** `OnnxRerankerUnavailableError` (ml_client.py:42–46) raised
    when `GTE_RERANKER_BACKEND=onnx-int8` AND the ONNX load fails (ml_client.py:
    396–401) — deliberately does NOT silently fall back to fp32/FlashRank/zeros, so
    a broken int8 flip fails loudly instead of silently degrading quality.
  - I25 three-way registry rows present: `config_registry.py:183–184`
    (`YADGAR_GTE_RERANKER_BACKEND`, `YADGAR_GTE_RERANKER_ONNX_FILE`).
- **CPU supports int8 fast-path:** host is AMD Ryzen AI 9 HX PRO 375; `/proc/cpuinfo`
  shows `avx512_vnni`, `avx512f`, `avx2`. VNNI int8 kernels are available →
  quantization speedup is real on this box.
- **Reranker fallback chain (ml_client.py:524–544):** GTE-reranker (primary) →
  FlashRank (ONNX) → sentence-transformers MiniLM CrossEncoder (fp32, terminal).
  `GTE_RERANKER_ENABLED=true` (config.py:273) means GTE is primary and **preempts**
  the ST MiniLM path — this is the root of the prior failure (§2).

---

## 2. Root-cause of the prior onnx "failure" + how this plan fixes it

There were **two distinct prior failures**. The plan must not conflate them (the
warm doc partly does).

### Failure #1 — WIRING: the v5.85 int8 flip was a no-op (wrong knob)

v5.85 (`426768ca`) shipped int8 for the **ST MiniLM CrossEncoder** via
`CROSS_ENCODER_BACKEND=onnx-int8`. But `GTE_RERANKER_ENABLED=true` (default) makes
GTE-ModernBERT the **primary** reranker; the ST MiniLM path is the **3rd-priority
fallback that GTE preempts**. So flipping `CROSS_ENCODER_BACKEND` quantized a model
that never runs on the hot path → **zero effect**. (warm-profile-2026-07-02 header,
lines 5–9, states this explicitly.)

**Fix:** this plan targets the **correct knob** — `GTE_RERANKER_BACKEND` (the GTE
primary), added in v5.98 precisely to address this. Not `CROSS_ENCODER_BACKEND`.

### Failure #2 — RUNTIME: "onnxruntime broken" was a PROFILER-VENV artifact, not prod

warm-profile §3 #3 says onnx-int8 is "not runnable on this box" because
`import onnxruntime` fails (`ImportError: import numpy failed`). **That failure was
in the standalone py3.14 profiler venv (`.venv/bin/python`), NOT the backend
container.** Evidence it is a venv artifact, not a prod blocker:

- FlashRank (an ONNX reranker) **already runs in the backend image** — proof
  onnxruntime works there.
- `onnxruntime` reaches the backend via `sentence-transformers[onnx]>=2.2.0`
  (pyproject.toml:160), installed in `Dockerfile.backend` (`pip install "/app[ml]"`).

**Fix / honest caveat:** the plan does NOT assume prod is fine — it **verifies**
(§4 Step 1). The genuinely-open question is **not** "does onnxruntime import" (it
does, FlashRank proves it) but "does onnxruntime load and run **this specific
ModernBERT-int8 graph** inside the slim-trixie image without a missing shared lib
(e.g. `libgomp.so.1`) or opset mismatch." v5.98 left Lever 3 dormant for exactly
this reason (per the `b79e24d3` message: "onnxruntime + onnx CE load unverified in
built backend image; libgomp.so.1 risk on slim base"). **Step 1 closes that gap
empirically before anything else.**

### Why this plan won't repeat the traps

- **Wrong-knob trap:** targets `GTE_RERANKER_BACKEND`, verified as the primary path.
- **Profiler-venv trap:** all verification runs **in the built backend image**, never
  the host venv.
- **Silent-degradation trap:** the `OnnxRerankerUnavailableError` guard is already in
  place — a broken flip fails loudly, so a bad activation cannot masquerade as
  "working but slightly worse."

---

## 3. Design

### 3.1 Model + export

**No export step.** Use the upstream pre-exported `onnx/model_int8.onnx` from the
`Alibaba-NLP/gte-reranker-modernbert-base` HF repo (already in cache; downloaded at
runtime like all ST models — not bundled in the image, keeping image size flat).
This sidesteps the two ModernBERT-export hazards entirely (no local
`optimum.exporters` run, no `attn_implementation="eager"` tracing dance): the
upstream artifact already resolved them. **If** the upstream int8 file is ever found
inadequate (quality §5), a local static/dynamic re-quantization is a *fallback*, not
the plan (see §7 audit).

### 3.2 Quantization method (upstream-provided; specifics unconfirmed)

The artifact is **almost certainly int8 dynamic** quantization of the ModernBERT-base
cross-encoder (the standard `optimum`/`onnxruntime` dynamic-quant path for encoder
classifiers; weights int8, activations quantized per-op at runtime — the family also
ships `model_quantized.onnx` / `model_uint8.onnx`, consistent with the standard
optimum quantizer output set). On a VNNI CPU this uses `avxvnni`/`avx512_vnni` int8
GEMM kernels. **Method NOT metadata-confirmed** — treat "dynamic, no calibration" as
*likely, not verified*; confirm from the ONNX graph metadata or the upstream export
script/README before relying on the no-calibration claim. Either way we do NOT
re-quantize: we **consume and verify** the upstream artifact (score-parity + recall@k,
§5). The upstream repo shipping it is what removes the calibration-dataset burden —
verify that assumption holds rather than asserting it.

### 3.3 Wiring (already present — this plan flips a default, not adds code)

The load path exists (§1.3). The **only code change** is potentially flipping the
**default** `GTE_RERANKER_BACKEND` from `"torch"` to `"onnx-int8"` in config.py:277
+ config_registry.py:183 — and **only after** Steps 1–3 pass. Until then the flip is
**runtime-only** (`YADGAR_GTE_RERANKER_BACKEND=onnx-int8` env) for measurement.

Decision to make at implementation time: **default-flip vs opt-in-env.** Prefer
keeping the default `torch` and shipping onnx-int8 as opt-in *if* the quality/latency
win is marginal or box-dependent (VNNI-only); flip the default *only* if the win is
robust and quality is neutral-or-better. State the choice in the PR with the numbers.

### 3.4 Backend image deps

- `onnxruntime` currently arrives **transitively** via `sentence-transformers[onnx]`
  — **unpinned**. If Step 1 shows it loads and runs GTE-int8 cleanly, **add an
  explicit pin** (`onnxruntime==<verified-version>`) to `pyproject.toml` `[ml]`
  extra so the working version can't drift out from under a shipped default. If Step 1
  needs a system lib (e.g. `libgomp1`), add it to the `apt-get install` line in
  `Dockerfile.backend` (currently only `curl`).
- `optimum` is **not** required at runtime for *consuming* a pre-exported ONNX via
  the sentence-transformers `backend="onnx"` path — do NOT add it unless Step 1
  proves the ST onnx backend needs it. (Keeps image lean; backend cap ≤2.0 GB,
  currently ~1.63 GB.)
- **Image-size guard:** onnxruntime CPU wheel is ~5–15 MB; the int8 model (~150 MB)
  is **runtime-downloaded, not imaged**. Net image growth expected small; verify
  against `scripts/check_image_size.py` (≤2.0 GB ratchet) in CI regardless.

---

## 4. Step-by-step

Ordered so the cheapest falsifier runs first. STOP at the first hard failure and
apply §7 rollback/alternatives.

**Step 0 — Reconcile task id + branch.** Confirm live task number (#13 vs #92) in the
tasklist; branch off latest `master` (`feat/ce-onnx-int8-lever3` or per tasklist).
Run the Check-Branch-State gates.

**Step 1 — Prove onnxruntime runs GTE-int8 INSIDE the built backend image (the crux).**
- Build (or reuse latest) `yadgar-backend` image.
- In a throwaway container off that image (read-only, no infra mutation):
  `python -c "import onnxruntime; print(onnxruntime.__version__,
  onnxruntime.get_available_providers())"` — expect `CPUExecutionProvider`, no
  `ImportError`.
- Load the CE via the real path with `GTE_RERANKER_BACKEND=onnx-int8` and score one
  `[query, doc]` pair; assert a float score comes back and
  `OnnxRerankerUnavailableError` is NOT raised.
- If it raises: capture the exact error. `libgomp.so.1 missing` → add `libgomp1` to
  Dockerfile apt line, rebuild, retry (this is the expected, fixable failure).
  Opset/arch error → NO-GO on this artifact (see §7).
- **Gate:** onnxruntime loads + scores in-image. FAIL twice on the same cause → STOP,
  report, do not proceed.

**Step 2 — Score-parity (int8 vs fp32) on a fixed pair set.**
- Same container, same candidate pairs, run both `torch` and `onnx-int8`; compare
  raw CE scores. Report Spearman rank correlation + max abs score delta. int8 dynamic
  typically holds rank order tightly (>0.98 Spearman); a *rank* flip in the top-k is
  the thing that hurts recall, not a small magnitude shift.
- **Gate:** Spearman ≥ 0.98 on the pair set AND no top-k order inversion on the
  sampled queries. Below that → quality-risk, proceed to Step 3 but weight the
  LongMemEval result heavily.

**Step 3 — LongMemEval quality gate (THE gate — see §5).**
- `make longmemeval Q=<N>` twice: baseline (`torch`) then candidate (`onnx-int8`).
- Compare recall@k. **No regression → eligible to flip. Regression beyond tolerance
  → ABORT, flip stays off.**

**Step 4 — Latency measure on `--cpus 2` (see §6).**
- MCP-tool `recall` on the live backend + Tempo `retrieval.rerank.cross_encoder` span,
  cold and concurrent-6, quiet host. Report before/after.

**Step 5 — Decide + wire the default (only if Steps 1–4 pass).**
- If robust win + neutral quality: flip default `torch → onnx-int8` (config.py:277 +
  config_registry.py:183), pin onnxruntime, update FIELD_META/registry in lockstep,
  keep `test_config_three_way_sync` green.
- If marginal/box-dependent: keep default `torch`, document `onnx-int8` as opt-in in
  MIGRATION_NOTES + CAPABILITY_REGISTRY.
- Add/keep the loud-guard test (`test_gte_onnx_int8_load_failure_raises_loud_not_silent`).

**Step 6 — Loop-until-clean.** Run backend rerank tests, `test_config_three_way_sync`,
recall e2e BC suite, image-size check. Surface pre-existing failures; do not silence.

---

## 5. QUALITY GATE (co-equal with speed — the abort trigger)

**Harness:** `make longmemeval Q=<N>` (Makefile:307–315) →
`benchmarks/run_longmemeval.py --unified --retrieval-only --variant s
--stratify-per-type`. Routes through the **prod unified MCP recall path**
(retrieval-only, no LLM reader/judge), emits **recall@k**. This is the sanctioned
gate (ce-perf-options §B TDD: "A/B the int8 vs fp32 CE on the LongMemEval recall
set").

**Q sizing:** run `Q=30` (stratified, fast) for the go/no-go signal during iteration;
confirm the flip decision with `Q=100` (or `Q=0` = all 500 if the Q=30 delta is
within noise and the decision is close). Report the exact Q and seed in the PR.

**Two-layer quality check:**
1. **Score-parity (Step 2):** Spearman rank-corr int8-vs-fp32 CE scores ≥ **0.98**,
   no top-k inversion. Cheap early signal.
2. **Recall@k (Step 3):** the real gate. **Threshold: no regression.**
   **First establish whether the harness is deterministic:** run `torch` twice on the
   same Q/seed/corpus. Retrieval-only recall@k with a fixed model+corpus is likely
   **deterministic** (identical ranking → identical recall@k, spread ≈ 0).
   - **If deterministic** (torch-vs-torch spread ≈ 0): the rule is
     `recall@k(onnx-int8) >= recall@k(torch)` on the aggregate — aggregate must not
     drop; per-question ranking churn is expected (int8 will reorder some tails) and
     is fine as long as the aggregate holds.
   - **If stochastic** (nonzero spread — e.g. sampling/tie-breaking in the harness):
     `recall@k(onnx-int8) >= recall@k(torch) - <tolerance>`, tolerance = the measured
     torch-vs-torch spread (NOT an arbitrary 2%).
   Either way, aggregate recall@k dropping below torch → **regression → ABORT**,
   `GTE_RERANKER_BACKEND` stays `torch`.

**Decision rule:** quality is co-equal. A latency win does NOT buy back a recall@k
regression outside the noise band. Regression = abort, no exceptions.

---

## 6. LATENCY MEASURE + TARGET (`--cpus 2`, quiet host)

**Method (must match warm-profile discipline):** `YADGAR_OTLP_ENDPOINT` set to the
real Tempo for span capture (do NOT use the standalone-profiler empty-OTLP mode when
reading real spans; but if using the off-cgroup profiler, empty OTLP to avoid the
5–180 s retry-pollution artifact documented in warm-profile §Measurement caveat).
Measure via the **MCP `recall` tool** + Tempo `retrieval.rerank.cross_encoder` span.
Sequential single-query for cold/warm; the concurrent-6 burst **only if health-kill
risk is mitigated** (warm-profile method note). Quiet host, models pre-warmed,
median of ≥10 runs per variant.

**Targets (per path — do not average; each row is a DIFFERENT measurement):**

| path | measurable? | CE span now | int8 target (VNNI, 2 cores) | basis |
|---|---|---|---|---|
| **cold, single-query** (PRIMARY) | **YES — safe** | **~8.8 s (87% of a 10 s wall)** | **~2.5–4.5 s** (÷2–4×) | the headline win; ~51 candidates, CE dominates the wall (memory 518692: `subagent_start` cold-single = 88–91% CE) |
| concurrent-6 burst (secondary) | only if health-kill mitigated | worse than cold-single | proportional | risk-gated observation, NOT the headline — warm-profile method note forbids the burst without mitigation |
| warm-HIT inference | YES | ~720 ms | ~240–360 ms | warm-profile §3 #3 estimate (2–4×) |

The **cold single-query 8.8 s → ~2.5–4.5 s** is the headline and it is **safely
measurable** (sequential, no health-kill risk) — the 8.8 s figure IS a cold-single
number (`subagent_start`, ~51 candidates), not the burst. The concurrent-6 burst is a
**secondary, risk-gated** observation only; do not put it in the headline target. The
2–4× range is upstream-typical for int8-dynamic on a VNNI CPU; the actual multiplier is
what Step 4 measures on the cold-single path. If the measured multiplier on 2 cores is
<~1.5×, the win is thin (see §7 go/no-go).

**Note on `--cpus 2` scaling:** with only 2 cores shared with SurrealDB, int8's
throughput gain may be *less* than a benchmark on an idle many-core box. This is
precisely why Step 4 measures on the real constrained box, not a synthetic bench.

---

## 7. AUDIT (adversarial — brutal)

### 7.1 Risks

1. **onnxruntime won't load GTE-int8 in the slim image (libgomp/opset).** The one
   genuinely-unverified link (§2 Failure #2). *Likelihood: moderate-low* (FlashRank
   proves onnxruntime runs; the risk is the specific ModernBERT graph). *Mitigation:*
   Step 1 is the first, cheapest falsifier; `libgomp1` is a one-line Dockerfile fix.
   *If unfixable* (opset/arch): NO-GO on the upstream artifact.
2. **int8 accuracy loss regresses recall@k.** ModernBERT int8-dynamic usually holds
   rank order, but the whole point of the CE is fine-grained top-k ordering — the
   exact thing quantization noise can perturb. *Mitigation:* the two-layer gate (§5).
   *This is the co-equal-quality abort trigger.*
3. **int8 doesn't beat torch on 2 shared cores.** VNNI is present (good), but on 2
   cores shared with SurrealDB, memory-bandwidth and kernel-dispatch overhead can eat
   the int8 arithmetic win; the 2–4× is a many-core-idle number. *Mitigation:* Step 4
   measures on the real box; if <~1.5×, the change is not worth the added dep surface.
4. **Image bloat / dep drift.** onnxruntime is currently unpinned-transitive — pinning
   it (needed for a shipped default) adds maintenance surface, and a future
   sentence-transformers bump could break the pinned pair. *Mitigation:* pin +
   image-size CI check; keep `optimum` OUT unless proven needed.
5. **RAM footprint on the constrained box.** onnxruntime + a second model graph in
   memory alongside torch (if both load) on a RAM-constrained backend. *Mitigation:*
   verify only ONE backend loads (torch OR onnx, not both) and measure RSS in Step 1.
6. **The prior-failure trap (re-armed).** If an implementer "helpfully" flips
   `CROSS_ENCODER_BACKEND` (the MiniLM knob) instead of `GTE_RERANKER_BACKEND`,
   they'll reproduce the v5.85 no-op and report a false negative. *Mitigation:* §2 +
   §3.3 name the exact knob; the loud guard makes a real GTE-int8 failure visible.
7. **Score-parity looks fine but top-k inverts on the tail.** Aggregate Spearman can
   be 0.99 while a handful of adversarial queries flip their #1 result. *Mitigation:*
   Step 2 checks top-k inversion explicitly, not just aggregate correlation; Step 3
   (recall@k on the real set) is the backstop.

### 7.2 What would make this NOT worth it

- Step 4 shows <~1.5× on `--cpus 2` (int8 arithmetic win eaten by 2-core/bandwidth
  limits) — then the added onnxruntime pin + second code path buys little.
- Step 3 shows any recall@k drop outside the noise band — quality veto, full stop.
- Step 1 needs `optimum` + heavy deps to make the ST onnx backend work → image bloat
  toward the 2.0 GB cap for a marginal latency win.

### 7.3 Rollback

Trivial and instant: `GTE_RERANKER_BACKEND=torch` (env or config default). The torch
path is the shipped, production-tested default and is untouched. No data migration, no
model re-download blocking. The loud guard means a bad flip is caught at load, not in
silent quality drift. This is the **cheapest possible rollback** — a single env var.

### 7.4 Go / No-Go criteria

**GO to activate (flip default or ship opt-in) iff ALL:**
- Step 1: onnxruntime loads + scores GTE-int8 in the built backend image.
- Step 3: recall@k neutral-or-better (within/above torch noise band) at Q≥100.
- Step 4: measured CE speedup ≥ ~1.5× on the cold/concurrent path on `--cpus 2`.
- Image stays ≤2.0 GB; RAM footprint fits the constrained box.

**Flip DEFAULT to onnx-int8 iff:** all GO criteria **and** speedup is robust (≥2×
cold path) **and** quality strictly neutral-or-better. Otherwise ship as **opt-in
env**, default stays `torch`.

**NO-GO (keep torch, pursue §7.5 alternatives) if:** Step 1 unfixable, OR recall@k
regresses, OR speedup <~1.5× on 2 cores, OR RAM/image blows the cap.

### 7.5 Alternatives if onnx-int8 is NOT worth it on 2 cores

Ranked by likely value on this constrained box:
1. **torch thread tuning.** `torch.set_num_threads(2)` / `OMP_NUM_THREADS=2` matched
   to `--cpus 2` — a wrong thread count (default may spawn >2 threads that thrash on 2
   cores) can *lose* time. Cheapest possible experiment; measure first, may recover a
   chunk of the fp32 cost with zero new deps. **Try this regardless — it's free and
   orthogonal.**
2. **torch dynamic quantization** (`torch.quantization.quantize_dynamic` on the
   linear layers) — int8 speedup without the onnxruntime dep surface; stays in the
   torch path. Less mature for ModernBERT than the ONNX route but no new runtime.
3. **fp16 CE.** On CPU, fp16 is usually *not* faster (x86 CPU fp16 is often emulated) —
   **likely a non-starter on this box**; note and skip unless the CPU has native fp16.
4. **Reduce CE candidate count** (warm-profile §3 #4) — trades recall breadth for
   latency linearly; MEDIUM quality risk; only if the above fail.
5. **Skip CE on high retrieval_confidence** (ce-perf-options §G) — orthogonal lever,
   separate plan; removes CE cost entirely on unambiguous queries.

Honest bottom line: **the pre-exported artifact + VNNI make onnx-int8 the most
promising single lever**, but the 2-core constraint means its real multiplier is
uncertain until Step 4. If Step 4 disappoints, **torch thread-tuning (#1) is the
free consolation** and should be measured in the same session.

---

## 8. PR scope

**One PR** (backend/ changes → I25 + image + backend_version):

- **Contents:** verified onnxruntime pin (pyproject `[ml]`); any `Dockerfile.backend`
  apt addition (e.g. `libgomp1`) if Step 1 needs it; the default-flip OR opt-in
  documentation for `GTE_RERANKER_BACKEND`; FIELD_META/registry lockstep if any knob
  metadata changes; the loud-guard test; MIGRATION_NOTES + CAPABILITY_REGISTRY entry;
  the measured numbers (recall@k Q, score-parity, latency before/after) in the PR body.
- **backend_version:** already `5.10.0` (bumped in v5.98 for the dormant wiring). This
  PR changes backend image deps (onnxruntime pin ± apt lib) and/or the runtime
  default → **bump backend_version** (→ 5.11.0 or per the versioning convention) so
  deploys pull the new image. Confirm against `docs` versioning convention wiki. This
  is the **task #83/#4 backend-version-bump** obligation the prompt flags.
- **Task id:** reconcile #13 (prompt) / #92 (warm doc) / Lever 3 (#144 lineage) in the
  tasklist before opening; reference the correct one in the PR title.
- **PR template:** search `find <repo> -maxdepth 4 -iname "pull_request_template*"` +
  org fallback before writing the body; extend template sections; verify via
  `gh pr view <N> --json body` after create. (Codeberg = bot identity; branch-first;
  no `--no-verify`.)

---

## Appendix — key file:line references (observed)

- CE load + backend switch: `yadgar/backend/ml_client.py:356–407`
  (`_load_gte_reranker`), `:381–386` (onnx-int8 branch), `:409–446`
  (`_try_gte_reranker`).
- Loud guard: `ml_client.py:42–46` (`OnnxRerankerUnavailableError`), `:396–401` (raise).
- Fallback chain: `ml_client.py:524–544` (GTE → FlashRank → ST-MiniLM).
- Config: `config.py:273–278` (GTE_RERANKER_* incl. `GTE_RERANKER_BACKEND` default
  `torch` :277, `GTE_RERANKER_ONNX_FILE` :278); `config_registry.py:183–184`.
- Image: `Dockerfile.backend` (base `python:3.14-slim-trixie`; `pip install "/app[ml]"`);
  `pyproject.toml:160` (`sentence-transformers[onnx]>=2.2.0`, onnxruntime transitive,
  unpinned).
- Harness: `Makefile:307–315` (`longmemeval`), `benchmarks/run_longmemeval.py`.
- Version: `yadgar/__init__.py:21` (backend_version 5.10.0).
- Artifact: `~/.cache/huggingface/hub/models--Alibaba-NLP--gte-reranker-modernbert-base/snapshots/f7481e60…/onnx/model_int8.onnx`.
- CPU: AMD Ryzen AI 9 HX PRO 375, `avx512_vnni` present (`/proc/cpuinfo`).
- Prior-failure narrative: `docs/plans/recall-warm-profile-2026-07-02.md` (header
  lines 5–9, §3 #3); `docs/plans/ce-perf-options.md` (option B, §v5.85).
