The first time we pointed a live model at our operator toolbox, we were ready for it to fail in interesting ways. It failed in a boring one: after 22 tasks, the execution success rate across the run was exactly 0.00 — and every single plan had passed validation.
What the validator saw
TGMS makes an LLM express its query as a small JSON DAG of operator calls, and a static verifier checks that plan before anything executes. The checks are not shallow: JSON Schema on every argument, DAG acyclicity, reference scope, temporal sanity on every window, a grounding rule (no identifier the task didn’t supply or an operator didn’t resolve), and a cost estimate. Qwen2.5-7B cleared all of it. The plans were well-formed, well-typed, well-grounded, affordable — and doomed.
Here is the failure, minimally reproduced. The model planned a reachability query, then tried to read the result:
// step s2: temporal_reachability — fine, passes every input check {"id": "s3", "op": "compute", "args": {"op": "identity", "value": {"$ref": "s2.count"}}} // what s2 actually returns: {"rows": [...], "rows_total": 343, "truncated": false}
There is no s2.count. The operator emits
rows_total. At runtime this is a dangling reference into a
result that will never contain the field — but at validation time, nothing
knew that. Input schemas constrain what goes into a tool. No schema
constrained what the model was allowed to read back out.
Why models invent output paths
Once you see it, it’s obvious why this is the failure mode. The model
has read a million JSON APIs where a counting endpoint returns
count. Absent evidence, it emits the modal name — a perfectly
reasonable guess that happens to be wrong for this toolbox. And nothing in
the standard tool-calling stack pushes back: tool definitions ship an input
schema, the call succeeds, and the invention only surfaces later as a
dangling reference, an empty value, or a silently wrong answer. Bigger
models guess field names better; they do not stop guessing.
This is worth stating generally, because it applies to any agent given tools: an operator manual that documents arguments but not results is half a contract, and the model will improvise the missing half.
The fix: output contracts
The repair was almost embarrassingly small. Every operator in the TGMS
registry now declares its output fields alongside its input schema — the
same registry that generates the tool definitions the model sees, so the
manual and the enforcement can’t drift apart. The static verifier resolves
every $ref in a plan and rejects any path whose first field the
producing operator does not emit. The rejection is a structured repair
payload that carries the real field list:
E_SCHEMA: temporal_reachability outputs no field 'count' (available: rows, rows_total, truncated, cursor)
The planner gets that message and, on the next attempt, reads
rows_total like it should have the first time. One check, one
useful error message.
With the check in place, the 7B run stopped being a wall of zeros and became an actual measurement — first-attempt validity 0.17 on single-operator tasks, lifted to 0.50 execution success by repairs, and 0.67 exact match on the correction probes that motivated the system in the first place. The bottleneck moved from “can the model address a result” to “can the model plan” — which is the bottleneck you want, because it’s the one that model scaling actually improves (first-attempt validity roughly triples from 7B to 14B on our dev split).
Manuals must be checkable, not just readable
We could have “fixed” this with prompting — paste every operator’s result shape into the manual and hope. But the model already had a readable manual; what it lacked was a checkable one. The distinction is the whole design philosophy of TGMS in miniature: every property of the system that matters is either enforced by a machine or it will eventually be violated by a model. Documentation is a prompt; a contract is a gate.
If you build tools for agents — MCP servers included — the transferable lesson is one sentence: result schemas deserve the same rigor as argument schemas, and small models are where you find out whether you meant it. Input validation tells you the model called your tool correctly. Only an output contract tells you the model is allowed to believe what it thinks your tool said.