Metadata-Version: 2.4
Name: protforge-groundcheck
Version: 0.1.0
Summary: Cheap hallucination detectors for RAG, benchmarked on RAGTruth
Author-email: ProtForge <hello@protforge.com>
License-Expression: MIT
Project-URL: Homepage, https://protforge.com/research/hallucination-detectors
Project-URL: Repository, https://github.com/protforge-ai/groundcheck
Project-URL: Issues, https://github.com/protforge-ai/groundcheck/issues
Keywords: hallucination,hallucination-detection,rag,llm,evaluation,faithfulness,nli,ragtruth
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: nli
Requires-Dist: torch; extra == "nli"
Requires-Dist: transformers; extra == "nli"
Provides-Extra: judge
Requires-Dist: anthropic; extra == "judge"
Requires-Dist: numpy; extra == "judge"
Requires-Dist: pandas; extra == "judge"
Requires-Dist: scikit-learn; extra == "judge"
Provides-Extra: bench
Requires-Dist: numpy; extra == "bench"
Requires-Dist: pandas; extra == "bench"
Requires-Dist: scikit-learn; extra == "bench"
Requires-Dist: torch; extra == "bench"
Requires-Dist: transformers; extra == "bench"
Provides-Extra: all
Requires-Dist: anthropic; extra == "all"
Requires-Dist: numpy; extra == "all"
Requires-Dist: pandas; extra == "all"
Requires-Dist: scikit-learn; extra == "all"
Requires-Dist: torch; extra == "all"
Requires-Dist: transformers; extra == "all"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# groundcheck

[![PyPI](https://img.shields.io/pypi/v/protforge-groundcheck)](https://pypi.org/project/protforge-groundcheck/)
[![Python](https://img.shields.io/pypi/pyversions/protforge-groundcheck)](https://pypi.org/project/protforge-groundcheck/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
![benchmark: RAGTruth](https://img.shields.io/badge/benchmark-RAGTruth-lightgrey)

Cheap hallucination detectors for retrieval-augmented generation (RAG), benchmarked on the public
[RAGTruth](https://github.com/ParticleMedia/RAGTruth) corpus. Built by ProtForge.
Write-up: https://protforge.com/research/hallucination-detectors

A plain lexical-overlap score (share of response tokens never seen in the source) beat a small NLI model on
RAGTruth, and a logistic-regression ensemble of both did not beat the lexical score alone. All of this is
modest: precision stays below 0.5 at the F1-tuned thresholds.

| detector | AUROC (95% CI) | precision | recall | F1 |
|---|---|---|---|---|
| `lexical` | 0.778 (0.746-0.806) | 0.476 | 0.858 | 0.612 |
| `nli` | 0.716 (0.681-0.747) | 0.476 | 0.836 | 0.607 |
| `numeric` | 0.535 (0.503-0.566) | 0.353 | 1.000 | 0.522 |
| `ensemble` | 0.766 (0.737-0.797) | 0.450 | 0.934 | 0.607 |

900 of the 2,700 RAGTruth test responses (stratified subsample, seed 0); thresholds and ensemble fit on 299
train responses. Full details, per-task numbers and caveats under "Results" below.

Caveats up front: the test set is a subsample; thresholds and the ensemble were tuned on only 299 responses;
precision is poor (the all-positive baseline gives F1 0.522); labels are response-level; the LLM judge is
implemented but **not evaluated yet**; English only, 2023-era generators.

Given a source text and a model response, each detector returns a score in [0, 1] (higher = more likely
hallucinated):

| detector | idea | cost |
|---|---|---|
| `lexical` | share of content tokens in the response (lowercased, stopwords and punctuation removed, numbers kept) that never appear in the source | microseconds |
| `numeric` | share of numbers in the response not found in the source after normalising `1,000` / `1000` / `1k`; 0 when the response has no numbers | microseconds |
| `nli` | SummaC-style. Response split into sentences (first 12 scored; later ones fall back to the lexical per-sentence score); source split into sentences; for each response sentence, NLI (`cross-encoder/nli-deberta-v3-small`) against each of the top-3 source sentences by content-token overlap and against their concatenation (each premise capped at 200 tokens); sentence entailment = max over those 4 pairs; score = 1 - the weakest sentence's entailment | about 7 s per response on a loaded 4-thread CPU |
| `llm_judge` | Claude as judge via the `anthropic` SDK (`groundcheck.detectors.LLMJudge`): structured JSON verdict, score = confidence if hallucinated else 1 - confidence. **Not evaluated yet**; see "LLM judge" below | API cost |
| `ensemble` | logistic regression on the `lexical`, `nli` and `numeric` scores, fit on the train sample only | - |

## Install

```bash
pip install protforge-groundcheck           # core: lexical + numeric detectors, no dependencies
pip install "protforge-groundcheck[nli]"    # adds torch + transformers for the NLI detector
pip install "protforge-groundcheck[judge]"  # adds anthropic (+ numpy, pandas, scikit-learn) for the LLM judge
pip install "protforge-groundcheck[bench]"  # everything needed to reproduce the benchmark
pip install "protforge-groundcheck[all]"
```

The PyPI distribution is `protforge-groundcheck`; the import name and CLI are `groundcheck`. Note that another,
unrelated PyPI project called `groundcheck` also installs an importable `groundcheck` package, so do not install
both in the same environment.

## Python quickstart

```python
from groundcheck import Lexical, Numeric

source = "The company reported revenue of 1,200 units in 2023 and hired 40 engineers."
good = "The company reported revenue of 1,200 units in 2023."
bad = "The company reported revenue of 3,400 units and opened an office in Berlin."

print(Lexical().score(source, good), Lexical().score(source, bad))  # 0.0 0.5
print(Numeric().score(source, good), Numeric().score(source, bad))  # 0.0 1.0
```

Scores are in [0, 1]; higher means more of the response is unsupported by the source. The `lexical` threshold of 0.26
in `results/metrics.json` is the RAGTruth F1-optimal value, not a default to trust: tune on your own data.
`from groundcheck import NLI` gives the NLI detector (`pip install "protforge-groundcheck[nli]"`).

From the command line (two text files):

```bash
groundcheck score --source source.txt --response response.txt --detector lexical
```

`--detector nli` downloads `cross-encoder/nli-deberta-v3-small` on first use and is slow on CPU.

From a source checkout: `pip install -e ".[test]"` (pytest) or `pip install -e ".[all,test]"`.

## CLI

```bash
groundcheck score --source source.txt --response response.txt --detector nli
```

Prints JSON with the response score and a per-sentence verdict (`--threshold`, default 0.5, is not tuned;
tune it on your own data). `--detector` is one of `lexical`, `numeric`, `nli`.

```bash
groundcheck bench [--data-dir data] [--out-dir results] [--seed 0] [--train-n 300] [--test-n 900] [--threads 4]
```

Downloads `response.jsonl` and `source_info.jsonl` from the RAGTruth repository into `data/`, scores a
stratified train sample (by task type x label, default 300) and a stratified test subsample (task type x model, default 900 of the 2,700 test responses), tunes each detector's threshold to maximise
F1 on the train sample, and writes to `results/`: `metrics.json`, `by_task.csv`, `by_model.csv`,
`test_scores.csv` (ids and scores only, no source text) and `examples.json`. NLI results are cached in
`data/nli_cache.jsonl`, so an interrupted run resumes. If the NLI speed measured on the train sample projects the
test run past 2.5 hours, the test subsample drops to 600 and `metrics.json` records it (not triggered in the
published run). `lexical` and `numeric` are also scored on the full 2,700-response test split for reference. A CUDA device is used automatically when available.

## Reproduce

```bash
pip install -e ".[bench,test]"
python -m pytest
groundcheck bench --seed 0
```

## Data, license, citation

RAGTruth is released by Particle Media under the MIT License (Copyright (c) 2023 Particle Media; text in the
repository's `LICENSE`). Only per-response scores, ids and short snippets are published here, not the corpus.

> Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, Tong Zhang.
> *RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models.*
> ACL 2024. arXiv:2401.00396.

```bibtex
@inproceedings{niu2024ragtruth,
  title     = {RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models},
  author    = {Niu, Cheng and Wu, Yuanhao and Zhu, Juno and Xu, Siliang and Shum, Kashun and Zhong, Randy and Song, Juntong and Zhang, Tong},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)},
  year      = {2024},
  eprint    = {2401.00396},
  archivePrefix = {arXiv}
}
```

Label: a response is hallucinated if it has at least one annotated span. Split: the corpus's own `train` /
`test` field.

## Results

Run: seed 0, threshold and ensemble fit on a stratified train sample of 299 responses, evaluation on a
stratified test subsample of **900 of the 2,700** test responses (300 per task, stratified by task type x
model). Hallucination base rate in the subsample is 35.3% (35.0% on the full test split). Thresholds maximise F1
on the train sample. 95% intervals are bootstrap intervals over the 900 responses. Total runtime 2 h 33 min
(NLI scoring of 299 + 900 responses, 4 CPU threads, machine shared with other workloads).

| detector | AUROC (95% CI) | precision | recall | F1 |
|---|---|---|---|---|
| `lexical` | 0.778 (0.746-0.806) | 0.476 | 0.858 | 0.612 |
| `nli` | 0.716 (0.681-0.747) | 0.476 | 0.836 | 0.607 |
| `numeric` | 0.535 (0.503-0.566) | 0.353 | 1.000 | 0.522 |
| `ensemble` (LR on the three scores) | 0.766 (0.737-0.797) | 0.450 | 0.934 | 0.607 |
| `llm_judge` | not evaluated | - | - | - |

Predicting "hallucinated" for everything gives precision 0.353, recall 1.0, F1 0.522, so F1 near 0.6 is a modest
gain; AUROC is the better yardstick. `numeric`'s F1-optimal train threshold is 0.0 (flag any response with a
number-like token that is missing from the source), so its precision/recall row is the all-positive baseline.

`lexical` and `numeric` on the full 2,700-response test split (same thresholds), for comparison with the
subsample: `lexical` AUROC 0.785, P 0.477, R 0.876, F1 0.617; `numeric` AUROC 0.551, P 0.349, R 1.000, F1 0.518.

By task type (test subsample, AUROC / F1; base rate in brackets):

| task | n | base rate | lexical | nli | numeric | ensemble |
|---|---|---|---|---|---|---|
| QA | 300 | 0.20 | 0.708 / 0.426 | 0.706 / 0.424 | 0.683 / 0.333 | 0.740 / 0.468 |
| Summary | 300 | 0.20 | 0.616 / 0.332 | 0.627 / 0.360 | 0.478 / 0.333 | 0.582 / 0.361 |
| Data2txt | 300 | 0.66 | 0.661 / 0.797 | 0.496 / 0.784 | 0.514 / 0.795 | 0.663 / 0.795 |

Full test split, `lexical` / `numeric` AUROC: QA 0.713 / 0.658, Summary 0.673 / 0.545, Data2txt 0.697 / 0.526.

Reading the numbers honestly:

- A bag-of-tokens overlap score beats the small NLI model on this corpus, and the logistic-regression ensemble
  (fit on 299 responses) is not better than `lexical` alone overall. The ensemble does best on QA only.
- `nli` is at chance on Data2txt (AUROC 0.50): the source there is flattened JSON and sentence-level NLI against
  key-value lines is a poor fit. It is competitive with `lexical` on QA and Summary.
- `nli` probabilities are saturated near 0 and 1 (the tuned threshold is 0.999), so precision/recall trade off
  sharply around that value.
- Per-model breakdowns are in `results/by_model.csv`, per-response scores in `results/test_scores.csv`, and three
  illustrative cases (two where `nli` catches a hallucination `lexical` misses, one the reverse) in
  `results/examples.json`.

## Ablation and operating points

`groundcheck ablation` (code in `groundcheck/ablation.py`) reuses the cached scores: `results/test_scores.csv` for the
900 test responses and `data/nli_cache.jsonl` to rebuild the 299 train responses, so the main NLI scores are not
recomputed. It refits the ensemble on the train sample and checks it reproduces the published ensemble scores.
Outputs: `results/ablation.json`, `results/operating_points.csv`, `results/cost_ratio_table.csv`,
`results/data2txt_verbalised_scores.csv`.
Bootstrap: 1,000 resamples, seed 0, responses resampled within task and paired across detectors, so differences
have their own interval.

**Headline.** To catch 80% of hallucinations (overall test subsample, threshold chosen on that same subsample), the
best detector, `lexical`, also flags 40% of the correct answers (false-block rate 0.40; precision 0.53). The
ensemble matches it (0.40) and `lexical+numeric` is slightly worse (0.41). Operating points below are in-sample.

The marginal overall AUROC intervals (`lexical` 0.778, `ensemble` 0.766, `lexical+numeric` 0.762) overlap, so
their ranking cannot be resolved from the marginals; only paired differences can be. The paired result for
`lexical+numeric` minus `lexical` is **-0.016 (95% CI -0.027 to -0.005)** overall: adding the numeric detector to
`lexical` via logistic regression does not help and probably costs a little. Per task: QA +0.024 (+0.005 to +0.048),
Summary -0.038 (-0.071 to -0.005), Data2txt +0.002 (-0.023 to +0.026). The overall loss is driven by Summary, where
the fitted combination generalises worse than `lexical` alone (the weights come from a 299-response train sample).

**1. Does NLI add anything to the ensemble?** `ensemble_lex_num` is the same logistic regression on `lexical` and
`numeric` only (no NLI).

| split | n | lexical | nli | full ensemble | lexical+numeric | full minus lex+num |
|---|---|---|---|---|---|---|
| QA | 300 | 0.708 (0.646-0.769) | 0.706 (0.638-0.774) | 0.740 (0.680-0.803) | 0.733 (0.674-0.791) | +0.007 (-0.015 to +0.028) |
| Summary | 300 | 0.616 (0.538-0.694) | 0.627 (0.538-0.707) | 0.582 (0.510-0.660) | 0.579 (0.503-0.653) | +0.003 (-0.020 to +0.024) |
| Data2txt | 300 | 0.661 (0.594-0.720) | 0.496 (0.422-0.564) | 0.663 (0.594-0.725) | 0.663 (0.594-0.725) | -0.001 (-0.002 to +0.001) |
| overall | 900 | 0.778 (0.749-0.808) | 0.716 (0.682-0.750) | 0.766 (0.739-0.797) | 0.762 (0.734-0.793) | +0.005 (-0.001 to +0.011) |

AUROC (95% bootstrap CI). Within noise, NLI adds nothing to the ensemble: every difference interval includes zero.
The overall interval only just does (-0.001 to +0.011), so a gain of about half a point cannot be excluded, but it
is small and the ensemble is still not better than `lexical` alone. (The "overall" figures differ slightly from the
table above because the interval here resamples within task.)

**2. Operating points.** Threshold chosen on the test subsample itself so recall reaches 0.8 (or 0.5); this shows
the best trade-off available, not a deployable threshold. False-block rate = correct answers flagged / all correct
answers. Full table for every detector (including `numeric`, whose scores are mostly zero so it cannot reach
recall 0.5-0.8 without flagging everything) in `results/operating_points.csv`.

| task (base rate) | detector | precision @ R 0.8 | false-block @ R 0.8 | precision @ R 0.5 | false-block @ R 0.5 |
|---|---|---|---|---|---|
| QA (0.20) | lexical | 0.29 | 0.49 | 0.33 | 0.25 |
| | nli | 0.31 | 0.45 | 0.33 | 0.25 |
| | ensemble | 0.34 | 0.38 | 0.35 | 0.23 |
| | lexical+numeric | 0.31 | 0.45 | 0.34 | 0.24 |
| Summary (0.20) | lexical | 0.25 | 0.62 | 0.27 | 0.34 |
| | nli | 0.25 | 0.61 | 0.32 | 0.26 |
| | ensemble | 0.24 | 0.62 | 0.25 | 0.38 |
| | lexical+numeric | 0.24 | 0.64 | 0.25 | 0.38 |
| Data2txt (0.66) | lexical | 0.72 | 0.60 | 0.77 | 0.28 |
| | nli | 0.66 | 0.81 | 0.65 | 0.53 |
| | ensemble | 0.73 | 0.59 | 0.79 | 0.27 |
| | lexical+numeric | 0.73 | 0.57 | 0.79 | 0.27 |
| overall (0.35) | lexical | 0.53 | 0.40 | 0.67 | 0.14 |
| | nli | 0.49 | 0.46 | 0.55 | 0.22 |
| | ensemble | 0.52 | 0.40 | 0.66 | 0.14 |
| | lexical+numeric | 0.52 | 0.41 | 0.65 | 0.15 |

In plain terms: at 80% recall a detector blocks roughly 40-60% of the correct answers per task, and precision is
only 0.24-0.34 on QA and Summary (base rate 0.20). On Data2txt, precision at recall 0.8 is 0.72 against a
0.66 base rate, so the detector is only slightly better than flagging everything. At recall 0.5 the false-block rate
drops to 25-35% on QA and Summary.

**3. Cost-ratio view.** A false block (flagging a correct answer) costs C, a missed hallucination costs 1. For each
C the threshold minimises C x false blocks + misses on the overall test subsample (n=900, 35% hallucinated).
Intervals are bootstrap 95% percentile CIs where the threshold is re-selected inside every resample, so they include
threshold selection; thresholds are still chosen in-sample, so the point estimates are optimistic. False-block rate =
correct answers flagged / all correct answers. Full table, including `numeric`: `results/cost_ratio_table.csv`.

| false-block : miss | detector | threshold | precision | recall | false-block rate | resamples flagging nothing |
|---|---|---|---|---|---|---|
| 1:1 | lexical | 0.49 (0.41-0.53) | 0.67 (0.61-0.75) | 0.53 (0.40-0.71) | 0.14 (0.08-0.24) | 0% |
| 1:1 | nli | 1.00 (1.00-1.00) | 0.56 (0.53-0.64) | 0.54 (0.15-0.66) | 0.24 (0.05-0.30) | 0% |
| 1:1 | ensemble | 0.54 (0.52-0.57) | 0.66 (0.60-0.72) | 0.50 (0.44-0.64) | 0.14 (0.11-0.21) | 0% |
| 1:1 | lexical+numeric | 0.54 (0.51-0.54) | 0.66 (0.60-0.71) | 0.49 (0.45-0.64) | 0.14 (0.12-0.22) | 0% |
| 5:1 | lexical | flags nothing | - | 0 | 0 | 98% |
| 5:1 | nli | flags nothing | - | 0 | 0 | 82% |
| 5:1 | ensemble | flags nothing | - | 0 | 0 | 100% |
| 5:1 | lexical+numeric | flags nothing | - | 0 | 0 | 100% |
| 20:1 | lexical | flags nothing | - | 0 | 0 | 100% |
| 20:1 | nli | flags nothing | - | 0 | 0 | 96% |
| 20:1 | ensemble | flags nothing | - | 0 | 0 | 100% |
| 20:1 | lexical+numeric | flags nothing | - | 0 | 0 | 100% |

Flagging an answer only pays when precision exceeds C/(1+C): 0.50 at 1:1, 0.83 at 5:1, 0.95 at 20:1. At 1:1 the
detectors can be used (about half the hallucinations caught, 14% false blocks for `lexical`, `ensemble` and
`lexical+numeric`; `nli` is worse). At 5:1 and 20:1 no detector reaches that precision at any threshold on this
data, so the cost-minimising policy is to flag nothing; the threshold intervals for those rows cover only the few
resamples that did flag something (none for `lexical` at 20:1). The 1:1 `nli` threshold is the top of its score
range (scores saturate near 1.0), so its recall interval is wide.

**Is the lexical threshold stable across corpora?** No. Threshold tuned on one task for recall 0.8, applied to the
others (recall / false-block rate):

| tuned on (threshold) | applied to QA | applied to Summary | applied to Data2txt |
|---|---|---|---|
| QA (0.23) | 0.80 / 0.49 | 0.68 / 0.51 | 1.00 / 1.00 |
| Summary (0.18) | 0.95 / 0.58 | 0.82 / 0.62 | 1.00 / 1.00 |
| Data2txt (0.48) | 0.12 / 0.10 | 0.07 / 0.02 | 0.80 / 0.60 |

QA and Summary thresholds are close (0.18-0.23) and transfer reasonably (recall within 0.15). The Data2txt threshold
is more than twice as high, because even faithful data-to-text answers contain many tokens absent from the flattened
JSON: a QA/Summary threshold flags every Data2txt answer, and the Data2txt threshold misses almost everything in
QA and Summary. The per-task sweep (`lexical_threshold_sweep` in `ablation.json`, steps of 0.05) shows the same:
recall falls from 0.88 to 0.33 between thresholds 0.20 and 0.40 on QA, while Data2txt stays at recall 1.0 up to
about 0.30. Thresholds here are tuned and evaluated on the same 300 responses per task, so the diagonal is
optimistic. Tune the threshold per corpus.

**3. Data2txt verbalisation.** The structured JSON source is turned into sentences of the form
"The <name> has <field> <value>." (nested keys joined, camelCase split, null and false dropped, true gives the
field name only, review text kept as its own sentences) before the same NLI pipeline. NLI was re-run on the 300
Data2txt test responses only (the train sample, thresholds and other tasks are unchanged). That run took about
49 minutes on 12 CPU threads, but two copies of the script ran concurrently by mistake and shared the CPU, so
this is an upper bound; no clean timing was taken.

| Data2txt (n=300) | AUROC (95% CI) |
|---|---|
| `lexical` | 0.661 (0.593-0.727) |
| `nli`, flattened JSON (published) | 0.496 (0.422-0.561) |
| `nli`, verbalised source | 0.467 (0.398-0.538) |

Verbalisation did not help: `nli` on the verbalised source is 0.029 AUROC below raw-JSON `nli` (paired interval
-0.084 to +0.025, so no detectable change) and still at chance, while `lexical` is ahead of it by 0.194 (paired
interval +0.100 to +0.295). The failure on Data2txt is therefore not explained by the key-value format alone.
We have not tested why; plausible causes are that the label depends on numbers and attributes that a small NLI
model does not compare well, and that the top-3 sentence retrieval step picks poor premises for this task.

Caveats: single seed and a 900-response test subsample; AUROC intervals for the ablation and the verbalisation
are from the same fixed resamples, not independent experiments; operating points are in-sample on the test
subsample.

## LLM judge (implemented, not evaluated yet)

`groundcheck judge` scores the exact benchmark sample (the 299 train and 900 test responses behind `results/`)
with one or more Claude models and caches every answer, so interrupted runs resume.

```bash
groundcheck judge --dry-run --models claude-haiku-5-5 claude-sonnet-5-5 claude-opus-5-5 --budget-usd 50
groundcheck judge --models claude-haiku-5-5 claude-sonnet-5-5 claude-opus-5-5 --budget-usd 50
groundcheck bench --with-judge
```

- **Key:** the client is built only from the `PROTFORGE_ANTHROPIC_KEY` environment variable, or, when that is
  unset, from a `.protforge_key` file next to `pyproject.toml` (whitespace stripped; the file is git-ignored and
  excluded from source distributions). No other credential source is read, and the key is never printed. Without
  a key the command exits with a message; `--dry-run` needs none and makes no network calls.
- **Request:** static instructions in a cached system prompt; the user turn holds the source and response;
  structured output `{"hallucinated": bool, "unsupported_claims": [str], "confidence": 0-1}`; `effort` low; no
  temperature/top_p; `max_tokens` 4000 (thinking is always on and counts as output). Refusals and truncated
  answers are cached with their cost but not scored.
- **Cost control:** models run cheapest first, 8 requests in flight with SDK retries. Cost comes from
  `response.usage` (cache writes billed at 1.25x and reads at 0.1x of input price) at $/1M tokens input/output:
  haiku-5-5 0.10/0.50, sonnet-5-5 2/10, opus-5-5 4/20. Before each call the worst case of that call (plus the
  worst case of calls in flight) is checked against `--budget-usd`; if it could exceed the budget the run stops
  and can be resumed with a larger budget. Spend already in the caches counts toward the budget.
- **Report:** `groundcheck bench --with-judge` adds one `llm_judge_<model>` row per cached model to
  `metrics.json`, `by_task.csv`, `by_model.csv` and `test_scores.csv` (threshold fit on train, like the other
  detectors; responses the judge refused to score are excluded from that model's metrics and counted in
  `metrics.json` under `llm_judge`), plus cost per 1,000 answers and median latency.

## Limitations

- English only; RAGTruth covers three tasks (QA, summarisation, data-to-text) and a fixed set of generator
  models from 2023. Scores will not transfer to other domains or newer models without re-tuning.
- Labels are response-level (any annotated span). The detectors, especially `nli`, work per sentence, and
  span-level quality is not measured.
- Thresholds are tuned on a train sample of 300 responses to maximise F1, and the ensemble is fit on
  the same sample; both are small.
- NLI on CPU is slow, so the NLI comparison uses a 900-response stratified test subsample and a 299-response
  train sample; confidence intervals on AUROC are bootstrap intervals over that sample. `nli` scores only the
  top-3 retrieved source sentences per response sentence (not the whole source) and the first 12 response
  sentences; later sentences use the lexical score.
- `lexical` and `numeric` use exact token matching with no stemming, number words or unit conversion.
  Structured (JSON) sources in the data-to-text task are flattened into `key: value` lines before scoring.
- `llm_judge` is implemented but not evaluated yet; its results are not in this report.

## Files in `results/`

- `metrics.json`: all metrics, thresholds, sampling and run settings.
- `by_task.csv`, `by_model.csv`: metrics split by task type and by generator model.
- `test_scores.csv`: per-response detector scores for the test subsample (ids and scores only, no source text).
- `ablation.json`, `operating_points.csv`, `cost_ratio_table.csv`, `data2txt_verbalised_scores.csv`: output of `groundcheck ablation` (see "Ablation and operating points").
- `examples.json`: three illustrative cases with short excerpts (RAGTruth is MIT-licensed).

## Citing and contact

```bibtex
@software{groundcheck,
  title  = {groundcheck: cheap hallucination detectors for RAG, benchmarked on RAGTruth},
  author = {{ProtForge}},
  year   = {2026},
  url    = {https://protforge.com/research/hallucination-detectors}
}
```

Issues and pull requests: use GitHub issues. Other contact: hello@protforge.com.
