Most memory-system posts are announcements: a new benchmark, a new number, a launch. This one isn't. It's the record of chasing a bug for several sessions, closing it with a negative result, and writing that down instead of quietly moving the threshold and hoping nobody checks. Measuring before believing — and saying the limits out loud — is the one rule this project doesn't bend on, so this had to be a post, not a line buried in a changelog.
The setup
hc_recall can say nothing. Instead of always returning its best guess,
hipercampo's recall gate can abstain — return
an empty result rather than a confident wrong one. On our own benchmarks (small,
single-topic corpora) that works: abstention accuracy of 0.833 against unrelated queries.
Run the same gate against LongMemEval — real dense, single-person conversation histories, dozens of sessions per haystack — and abstention accuracy collapses to 0.000. Recall stayed healthy (0.830 at k=5, chance is ~0.10), so this wasn't a retrieval failure. The memory found the right things. It just never said "I don't know" when it should have.
Attempt 1 — raise the floor
The gate's simplest defense is an absolute floor: if the best match doesn't clear a score, stay quiet. Swept the whole grid against real LongMemEval haystacks instead of synthetic filler text:
| floor | recall@k | abstention |
|---|---|---|
| 0.19 (production) | 0.833 | 0.00 |
| 0.24 | 0.667 | 0.43 |
| 0.33 | 0.333 | 0.71 |
| 0.42+ | 0.167 | 0.86 (never 1.0) |
No floor buys abstention without paying for it in recall at close to 1:1. Worse: raised as high as we swept it, abstention plateaus at 0.86 — it never reaches a clean 1.0, while recall keeps falling. Whatever this is, it isn't a knee to pick.
Attempt 2 — the margin instead of the score
Next hypothesis: maybe the raw score is the wrong axis. A real answer might stand out clearly from its own nearest competitor even when its absolute score is diluted by a large candidate pool; a near-miss that only shares vocabulary might have several similarly mediocre competitors nearby. So: measure the gap between the best match and the second-best, instead of the best alone.
There was a real trend — median margin was about 3× higher on questions with a true answer (0.035) than on ones without (0.011). But overlap between the two groups stayed at 0.43, and three of the seven no-answer cases had a margin as large as several of the answerable ones. A trend that overlaps that much isn't a gate.
Attempt 3 — looking for why, and finding a proof instead of a bigger sample
Thirteen observations is a small sample, and the honest next step would normally be "get more data." Instead we looked at which two points, across the whole set, sat closest to each other in (score, margin) space — one that should have triggered an answer, one that should have triggered silence:
gpt4_70e84552_abs should ABSTAIN → best=0.207 margin=0.002 6a1eabeb should ANSWER → best=0.198 margin=0.001
Distance between them: ≈0.009. Effectively the same point, answerable one way and not the other. That's not a sample-size problem — it's a ceiling. No monotonic recalibration of this signal, however sophisticated (a hand-tuned threshold, a learned margin, conformal or isotonic calibration), can ever separate two observations that collide in feature space. More data would only find more collisions like it.
What actually would fix it
Similarity tells you how much a candidate sounds like the question. It doesn't tell you whether the candidate contains the fact being asked for — a number, a name, the specific thing that answers it. That's a different kind of check: closer to content verification than to threshold-tuning, and doing it without a heavyweight model call sitting in the hot recall path is a real, unsolved design question — not a tuning knob we haven't turned far enough yet.
We haven't built that. It's written down as an open problem in the roadmap, not as a solved one. LongMemEval's full run can proceed reporting this honestly, as a documented limitation of dense, single-person corpora — which is more useful to anyone evaluating this than a threshold quietly tuned to look better on one benchmark and worse everywhere else.
Reproduce it
$ python scripts/calibrate.py --n 20,100,500 --longmemeval 60
Needs the official LongMemEval dataset (not shipped in this repo; ~170s CPU per instance). Everything above — the floor sweep, the margin analysis, the nearest cross-class pair — comes out of that one run. The commits are eba84ae, c546810, and 514dfcc.