Groundlens: a proofreader for RAG answers

It marks the words your sources don’t back — and shows you what each one should have said.

QUESTION    What is the invoice total?
SOURCE      ...the total amount due is 10,000 dollars, payable within 30 days...
ANSWER      The invoice total is 1,000 dollars, due in 30 days.

GROUNDLENS  1,000   nothing supports this.   Closest in invoice.pdf#p1: '10,000'

It never tells you the answer is wrong. It tells you which word to look at, and which document to open. Thirty seconds of human attention instead of five minutes.

pip install "groundlens[encoder]"

from groundlens import proofread, SentenceTransformerEncoder

marks = proofread(answer, [("policy.pdf#p3", passage)],
                  encoder=SentenceTransformerEncoder())
print(marks.report())

Ten is not a hundred

A retrieved document says the total due is 10,000 dollars. The answer says 1,000 dollars. A human catches that instantly, without a finance degree.

Embedding similarity does not. Cosine between the correct answer and the wrong one is about 0.99 — the error dissolves into the vector the way a drop of ink dissolves in a pool. An LLM judge does not either: it reads for plausibility, and “the total is 1,000 dollars” is a perfectly plausible sentence about an invoice. Sentence encoders organise text by vocabulary, topic and structure. Never by truth.

So groundlens uses two tests for two kinds of content. Words are anchored by meaning — the best cosine similarity a word reaches against any word of the sources. Numbers are anchored by arithmetic — parsed to a value with formatting normalised, so 10,000 and 10000 and $10,000 and 10.000 are one number, then checked for presence. Support is exactly 1.0 or exactly 0.0. Similarity is not allowed to vote.

And we report the floor, not the average. Every token-similarity metric aggregates by the mean, and the mean is where single-token errors go to die. On the invoice above the mean support of the wrong answer is 0.79, which looks fine. The weakest anchor is 0.00, which is an alarm.

There is no threshold

We measured nine detectors across five public benchmarks — two published encoder models, an NLI cross-encoder, an LLM judge, and this metric — at the operating point production actually runs at: false-positive rate at 95 % hallucination recall.

Forty-five cells across the full grid. The best is 0.65. A random detector sits at 0.95. One method ranks best of all by AUROC and flags 99 % of correct answers at the operating point. Nobody is in the usable corner, including us.

So the library ships no default cut and the result carries no decision field. Fit a threshold on your own labelled traffic with calibrate(), which hands back the measured false-positive rate next to it and refuses to run on fewer than 200 examples.

Reproducibility

The numeral channel is exact: Decimal comparison, fixed arithmetic context, locale from an argument and never from the environment. Byte-for-byte identical everywhere, proven in CI across ten OS × Python combinations under a randomised hash seed and a Turkish locale.

The lexical channel is a float32 cosine from a pinned encoder revision — not a model name, because a silent re-upload would change every number you ever published. Reproducible to 1e-6 across platforms, with stable ordering. Not bit-identical between x86 and Apple Silicon, and we don’t claim it is.

Scope

It verifies stated values against a retrieved source. Like a human reviewer, it can only check the document in front of it. It cannot verify computed values — “revenue tripled” against a source saying “5M to 15M”. It cannot check reasoning. It inherits your retrieval. No training, no labels, no API call, no network, and zero runtime dependencies.

v3 is a rewrite. Performance figures published before it could not be regenerated from committed code and have been withdrawn. RETRACTIONS.md says exactly what was wrong. Read it before citing anything from this project.