Project Helix / Incident D-036 / 2026-07-15
A model reviewing a bill of materials produced two false statements in one run. They were wrong in different ways, and only one of them was catchable by checking values against source data. That difference is the entire argument for how this system is built.
An eighteen-line bill of materials went through the full pipeline: deterministic checks first, then a local language model writing the narrative recommendation a client reads. Two of those lines matter here.
| Part | Price | Lead time |
|---|---|---|
| ESP32-S3 module | $3.20 | 21 days |
| BME280 sensor | $2.40 | 112 days |
The BME280's lead time crosses the ninety-day risk threshold, so the tool looked up substitutes and handed them to the model: a Bosch BME680 at $3.10 shipping in 14 days, and a Sensirion SHT31-DIS-B at $1.95 shipping in 21 days.
Everything the model needed was in front of it, correct. Here is what it wrote.
Model output, verbatim
For the BME280 sensor, the Bosch BME680 at $3.10 is slightly cheaper than your current part at $2.40 and ships far sooner.
$3.10 is not cheaper than $2.40. It is seventy cents more expensive, and the lookup data the model was handed said "higher cost" in plain words.
The grounding check passes this sentence, and it is right to. Every number in it is real: $3.10 and $2.40 both appear in the source data, exactly as written. What is false is the word cheaper — a claim about the relationship between two values, not about either value.
No amount of value-checking catches that. A library claiming otherwise would be misrepresenting its own scope, which is a worse failure than the one it is trying to prevent.
Defence — arithmetic moved into code
The cost delta is now computed in Python before the prompt is assembled. The model receives the finished comparison — "$0.70 MORE expensive than the original" — and is asked only to phrase it. It is never asked to work out which of two numbers is larger, because that is a thing it demonstrably gets wrong and a thing a computer has never once got wrong.
This is not a prompt fix. Prompting a model to be more careful with arithmetic is a request; removing the arithmetic is a guarantee.
Model output, verbatim
The ESP32-S3 module at $3.40 has a lead time concern; consider the SHT31-DIS-B at $1.95 as a second source.
The ESP32-S3 costs $3.20. There is no $3.40 anywhere in the submitted data. The model also attached a temperature sensor to a compute module as though they were interchangeable, and invented a lead-time concern for a part that had none.
This one is caught, and caught cheaply. The value is in the allowed set or it is not — no second model is consulted, no confidence score is produced, nothing that can itself hallucinate is involved in the decision.
Defence — deterministic grounding check
Every currency amount, measurement, identifier, quantity, percentage and date is extracted from the generated text and compared against values computed from the source. Text that fails is regenerated with the specific invented value named. After the retry budget is exhausted, the narrative is withheld entirely and the deterministic findings are delivered alone.
The correction fed back to the model is specific rather than a scolding, because a blind re-roll reproduces the same error at roughly the same rate:
YOUR PREVIOUS ATTEMPT CONTAINED VALUES THAT DO NOT APPEAR IN
THE SOURCE DATA...
- $3.40 (currency) in: "The ESP32-S3 module at $3.40 has a
lead time concern; consider the SHT31-DIS-B at $1.95..."
State only values that appear explicitly in the data provided
above. If you need a value you were not given, omit the claim
entirely rather than estimating it.
The temptation after an incident like this is to pick one defence and call it the answer. Both errors came from the same model in the same run, and neither defence would have stopped both.
| Failure | Shape | Stopped by | When |
|---|---|---|---|
| "slightly cheaper" | Wrong relation between two real values | Arithmetic in code | Never generated |
| "$3.40" | Value absent from source data | Grounding check | Rejected before delivery |
Neither defence is a model checking a model. Both cost less than an inference call, and neither can itself fabricate.
The general rule this incident produced, recorded at the time and applied since: arithmetic and multi-entity bookkeeping belong in deterministic code. The model's job is to phrase something already correct, never to compute something and then be trusted about it. That rule was not designed in advance — it was learned from watching a $3.10 part get called cheaper than a $2.40 one.
Both errors above are reproducible from the repository. The script asserts the behaviour this page describes — that the backwards comparison passes the grounding check and the fabricated price fails it — and exits non-zero if either ever stops being true.
python scripts/reproduce_d036.py
It is also part of the test suite, so the claims on this page cannot quietly stop being accurate while the page keeps making them.