Post #4 · Scale study

What happens when you give the same database a model 10× bigger?

We froze everything — the store (rebuilt byte-identically from its event log), the task suite (SHA-pinned before any measurement), the prompts, the temperature — and changed exactly one thing: the model. Qwen2.5 at 7B, 14B, 32B, and 72B, each driving the same operator toolbox and the same retrieval baselines, on H100 and Ada GPUs.

The operator-backed advantage grew 4.6× as the model scaled. The baselines didn't move.
0.00.2 0.40.6 7B14B 32B72B·AWQ static-graph RAG text-to-Cypher TGMS 0.628 · probes 1.000 0.138 0.340 0.511 AWQ, not scale
Exact match on the frozen CollegeMsg test split (94 tasks) as only the model changes. TGMS climbs 0.138 → 0.340 → 0.628; the baselines never exceed 0.28 at any size. The 72B point is AWQ-quantized — and drops.

The interface is the multiplier

This is the cleanest evidence yet for the claim that started this project: the interface is what converts model capability into answer quality. A stronger planner writes better operator DAGs, so every increment of model quality flows straight into accuracy — 0.138 at 7B, 0.340 at 14B, 0.628 at 32B. Meanwhile the same stronger models gained the baselines almost nothing, because no amount of reasoning fixes an input that doesn't contain the answer's structure: serialized edges still don't compose, and generated Cypher still can't express belief-state time travel.

The belief-state result is the sharpest version: correction probes — "as of transaction time T, what did we believe?" — went 0.38 → 0.77 → 1.000. A perfect score at 32B means the bi-temporal machinery isn't aspirational headroom; a good enough planner uses it flawlessly, today, with open weights on one GPU.

TGMS, 7B → 32B
0.138 → 0.628
4.6× — the advantage grows with the model.
Belief-state probes at 32B
1.000
Perfect. The baselines' best at any scale: 0.385.

Surprise #1: the 72B model got worse

The biggest model in the sweep — 72B, five times the parameters of our 32B point — scored lower: 0.511 overall, and a striking 0.31 on probes. The asterisk explains it: 72B only fits a single 80 GB GPU with AWQ 4-bit quantization, and every other point in the curve ran full-precision. Quantization, not scale, is the planning bottleneck: the compressed model writes visibly worse structured plans even though its general benchmark scores barely move. That matters for deployment, because AWQ-class quantization is exactly what small-GPU serving uses. If your agent plans against tools, benchmark your quantized model — the degradation concentrates precisely in structured planning.

Surprise #2: we un-hobbled RAG, and it got worse

Since our first writeup, the fairest criticism has been: "your vector-RAG baseline only retrieved one chunk because your GPU was small." True — a 24 GB card's serving window forced k=1. So on an H100, at the model's full native 32k context, we ran the intended k=20 retrieval breadth. The result:

vector-RAG configcorpus exposuretokens/taskexact match
k=1 × 256-event chunks (24 GB card)~0.4%~19k0.106
k=20 × 24-event chunks (H100, 32k)~0.8% × 20 sites32,4750.021
TGMS, same run, same model6,5210.362

More context made it worse — 17× behind TGMS at 5× the token cost. Retrieval breadth was never the bottleneck; representation was. Spreading attention across twenty chunks of serialized edges dilutes exactly the signal the questions quantify over. You cannot prompt your way to a temporal join.

The number that never moved

Across every cell of this study — four scales, two quantizations, plus two other model families (Llama-3.1-8B and Phi-4-mini) driving the same toolbox — the emitted unsupported-claim rate was 0.000. Weaker models answered fewer questions; they never answered more of them wrongly, because claims that don't verify against the execution trace never ship. That's the architecture's contract: competence lives outside the trust boundary and scales with the model; safety lives inside it and doesn't have to.

Buy a better model, get better answers — but only through an interface that converts capability into computation instead of context.
📎 All runs on frozen, SHA-pinned test splits against a byte-identically replayed store (tgms replay), on the University of Memphis iTiger GPU cluster; receipts and per-class tables in the technical report §8.2c and the v0.3.0 release. The scale figure appears in the updated paper.