EVALARC / EVIDENCE LABDownload offline review

ACTUAL LOCAL MODEL GENERATIONS / SEPTEMBER 2026

Upgrade a model.
Review every changed check.

Compare Qwen3-8B in BF16 with Qwen3.8-27B in FP8 on the same eight support-planning cases. Read all three fresh generations per case, inspect the native pytest checks, and recompute the comparison without calling a model.

24 generations per configuration. These are selected, public development cases; plans were not executed. Model size, quantization and architecture differ, so this comparison does not isolate the cause of a difference or estimate general model quality.

BASELINE · Qwen3-8B · BF16complete plans match the explicit contract
CURRENT · Qwen3.8-27B · FP8complete plans match the explicit contract

Open all changed checks   Comparison JSON

Prompt, policy and generation settings
Method and limitations
Both configurations receive the same policy and case text. Seeds are 17, 29 and 43, temperature 0.6, top-p 0.9, top-k 20, thinking disabled and a 512-token output budget. Each output is a new generation, not a regrade of a frozen answer. Repeated outcomes describe this small run only.
Five named checks per case assess JSON shape, routing, notes and retry keys, closure, and the complete ordered plan. Native pytest JUnit suites preserve all three attempts; EvalArc reports any check that loses passes or coverage. No tool calls or customer tickets are executed. Full plan equality is a narrow contract check, not a measure of useful support work.
The FP8 recorder applies a documented in-memory skip-list correction for Transformers 5.17.0 and rejects missing or unexpected checkpoint keys. Original model weights are unchanged. Model identities, runtime adjustments and failed initialization notes are in the download.
python examples/model-upgrade/regrade.py --output runs/model-review-001

evalarc diff baseline.xml current.xml --output runs/check-diff-001
Reproduction guide · Frozen protocol · Baseline identity · Current identity