Metadata-Version: 2.5
Name: legal-rag-verifier
Version: 0.2.0
Summary: Sentence-level claim and NLI verification with KV-cache rollback during generation, for legal RAG
Project-URL: Homepage, https://github.com/azterizm/legal-rag-verifier
Project-URL: Repository, https://github.com/azterizm/legal-rag-verifier
Project-URL: Issues, https://github.com/azterizm/legal-rag-verifier/issues
Project-URL: Changelog, https://github.com/azterizm/legal-rag-verifier/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/azterizm/legal-rag-verifier/blob/main/docs/API.md
Project-URL: Verdict, https://github.com/azterizm/legal-rag-verifier/blob/main/docs/VERDICT.md
Project-URL: Consultancy, https://memonsystems.com/engagements
Author-email: Abdullah Memon <abdullah@memonsystems.com>
Maintainer-email: Memon Systems Ltd <abdullah@memonsystems.com>
License-Expression: AGPL-3.0-only
License-File: LICENSE
Keywords: hallucination,kv-cache,legal,nli,rag,verification
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Legal Industry
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.11
Provides-Extra: hf
Requires-Dist: accelerate>=1.0; extra == 'hf'
Requires-Dist: bitsandbytes>=0.46.1; (platform_system != 'Windows') and extra == 'hf'
Requires-Dist: torch>=2.4; extra == 'hf'
Requires-Dist: transformers>=4.56; extra == 'hf'
Provides-Extra: nli
Requires-Dist: torch>=2.4; extra == 'nli'
Requires-Dist: transformers>=4.56; extra == 'nli'
Provides-Extra: sglang
Requires-Dist: transformers>=4.56; extra == 'sglang'
Description-Content-Type: text/markdown

# legal-rag-verifier

**Sentence-level verification for legal RAG, while the model generates.**

[![PyPI](https://img.shields.io/pypi/v/legal-rag-verifier)](https://pypi.org/project/legal-rag-verifier/)
[![CI](https://github.com/azterizm/legal-rag-verifier/actions/workflows/ci.yml/badge.svg)](https://github.com/azterizm/legal-rag-verifier/actions/workflows/ci.yml)
[![Python 3.11–3.13](https://img.shields.io/badge/python-3.11%E2%80%933.13-blue)](https://github.com/azterizm/legal-rag-verifier/blob/main/pyproject.toml)
[![Licence: AGPL-3.0-only](https://img.shields.io/badge/licence-AGPL--3.0--only-blue)](https://github.com/azterizm/legal-rag-verifier/blob/main/LICENSE)
![Runtime dependencies: none](https://img.shields.io/badge/runtime%20dependencies-none-brightgreen)

`legal-rag-verifier` checks each sentence of an answer against the statutory provisions it was
given, as the sentence is written. A deterministic claim check covers figures, dates, citations,
instrument titles, modal verbs and qualifiers; an NLI head (DeBERTa-v3) covers the prose. A
rejected sentence is rolled back and regenerated on the inference server's prefix cache, so only
that sentence is thrown away. The retry can insert the provision the sentence was checked against
at the cut point, which makes a small model's retry pass the check far more often.

**Status: proof of concept.** The mechanism holds on a production inference server. With the
DeBERTa head as the auditor the loop did not improve answers. With an LLM judge in its place it
did, on the tests run so far. The package ships the DeBERTa head; the LLM judge is in `scripts/`.
The [verdict](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/VERDICT.md) states what holds, what does not and
what is required first.

Built by [Abdullah Memon](https://memonsystems.com) at [Memon Systems Ltd](https://memonsystems.com).
Layer 4 of the Memon Systems hosted reference architecture, after
[`legal-rag-router`](https://github.com/azterizm/legal-rag-router), which can serve as its
abstention gate.

## Install

```bash
pip install legal-rag-verifier            # claim check only: no runtime dependencies
pip install "legal-rag-verifier[nli]"     # + DeBERTa-v3 NLI head (torch, transformers)
pip install "legal-rag-verifier[sglang]"  # + client for an SGLang server (tokenizer only)
pip install "legal-rag-verifier[hf]"      # + in-process Hugging Face generation
```

Python 3.11 or later. English and United Kingdom legislation in 0.1.0.

## Check a sentence

```python
from legal_rag_verifier.premise import Premise
from legal_rag_verifier.verifier import Verifier

premise = Premise.from_text(
    "The limit of a compensatory award is the lower of £123,543 and 52 multiplied by a week's pay.",
    titles=("Employment Rights Act 1996",),
    citations=("s124",),
)
verifier = Verifier()  # Verifier(SentenceNLIVerifier("cross-encoder/nli-deberta-v3-base")) adds NLI

verdict = verifier.check_sentence(premise, "The compensatory award is capped at £85,000.")
verdict.verdict  # Verdict.ROLLBACK
verdict.reasons  # (Reason.UNGROUNDED_FIGURE,)
verdict.ungrounded  # ['£85,000']
```

`check_text` runs the same checks over a whole answer, for post-hoc use.

## Generate with rollback

```python
from legal_rag_verifier.backends.sglang import SGLangBackend
from legal_rag_verifier.engine import InFlightGenerator
from legal_rag_verifier.nli import SentenceNLIVerifier

backend = SGLangBackend.connect("http://127.0.0.1:30000", "Qwen/Qwen2.5-7B-Instruct")
generator = InFlightGenerator(
    backend,
    Verifier(SentenceNLIVerifier("cross-encoder/nli-deberta-v3-base")),
    steering="inject",
)
answer = generator.generate_verified("What is the cap on the compensatory award?", premise)
answer.text  # the verified answer
answer.trace  # per sentence: verdicts, retries, what was inserted, prefix-cache hits, timings
```

The engine sends one request per sentence. A rejected sentence is not resubmitted, so the next
request reuses the verified prefix from the server's cache (SGLang's RadixAttention). Each sentence
gets at most three retries and each answer at most eight; after that the engine writes a fixed
refusal for that point and moves on. Retry modes:

- `allow`: constrain a rejected figure to the grounded values of the same type, then `ban`;
- `ban`: restart the sentence with its rejected first token banned;
- `inject`: insert the provision the sentence was checked against at the cut point (a new user
  turn; `inject_template`, which may restate the question with `{query}`) and regenerate. The
  provision stays in the context and never enters the answer.

The SGLang server needs `--enable-custom-logit-processor` for `allow` and `ban`. Full API:
[`docs/API.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/API.md).

## Evidence

Every figure below is measured. Figures: [`docs/measurements.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/measurements.md).
Methods, fixed before each run: [`docs/GRID.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/GRID.md). Raw results:
[`results/`](https://github.com/azterizm/legal-rag-verifier/tree/main/results).

**Detector, sealed held-out battery** (346 rows, sealed before it ran once; claim check +
`nli-deberta-v3-base` + enrichment): 80.0 % of planted errors caught (124/155), 7.9 % of correct
sentences rolled back (15/191), 89.2 % rollback precision. Weakest: dropped qualifiers (6/18).

**Rollback on Qwen 2.5 7B, SGLang, one NVIDIA L4:** every retry reused the verified prefix from the
cache. A retry request costs 117 ms in bf16 (69 ms in FP8), against 270 ms (170 ms) to re-prefill.
Checking sentence by sentence released the first verified sentence after 3.8 s, against 8.1 s when
the whole answer was checked first.

**Inserting the provision at the cut point** (48 rejected sentences retried from the same state):
the retry passed the auditor 39 times with the provision inserted, against 19 with decode-level
steering (exact McNemar p = 1.1e-5). The same text at the top of the prompt passed 17 of 46; a new
turn without the provision, 14. With the provision inserted, the regenerated sentence became more
likely in 46 of 46 states, the rejected sentence less likely in 45 of 46, and attention on
provision text rose in 46 of 46.

**Live output, DeBERTa head as the auditor:** its rejections were mostly wrong: 1 of 12 by hand on
software questions, 18 of 46 by a frontier model's labels on statutes. Graded by hand, the loop's
answers were no better than one plain answer from the same model. Inserting the provision makes
the retry pass the auditor; it does not make it better.

**An LLM judge in the auditor's place** (Qwen 27B at 4 bits as generator and judge on one SGLang
server; the judge is the entailment scorer, in `scripts/live_loop.py`): right on 93.8 % and 90.1 %
of the sentences of two hand-graded tests (354 and 151 sentences), against 68.7 % and 58.3 % for
the DeBERTa head. Run live on 40 questions with the source in the prompt, 48 of 49 sentences were
committed at the first try, none wrong, at 1.9 s per answer at the median. When the answer started
from memory, 36 of 65 sentences were wrong without the loop and 3 of 50 with it, all three
refusals.

**Other judges on the same two tests:** a 1B model trained on the 27B's labels, 89.0 % and 82.1 %
at 0.06 s per call ([adapters](https://huggingface.co/azterizm/minicpm5-1b-clause-judge)); Jev
1.13, 90.7 % and 88.1 %; the OpenAI Decisions API, 89.0 % and 86.8 %. None is within 3 points of
the 27B on both tests.

See [`docs/VERDICT.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/VERDICT.md) for the proof-of-concept verdict and its limits.

## Reproduce the evidence

The repository, not the package, holds everything behind the figures: the batteries and their
seals (`batteries/`), the per-provision enrichment (`enrichment/`), every answer, trace and
judgement (`results/`) and the scripts that produced them (`scripts/`). The rollback-point tables
and the placement and attention report re-derive from the committed results, with no GPU or API
key:

```bash
git clone https://github.com/azterizm/legal-rag-verifier && cd legal-rag-verifier && uv sync
uv run python scripts/grid_inject.py --cells 4B 4B-inject 4B-inject-v2
uv run python scripts/replay_report.py
```

Re-scoring the grid (`scripts/grid_score.py`) and re-running the batteries also need the statute
corpus, built with [`legal-rag-router`](https://github.com/azterizm/legal-rag-router) and copied
to `data/`; it is not redistributed here. Generation runs used Modal (`scripts/modal_m4.py`).

## Development

```bash
uv sync
uv run ruff check && uv run ruff format --check && uv run mypy && uv run pytest
```

Progress and decisions: [`docs/ROADMAP.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/ROADMAP.md). Contributing:
[`CONTRIBUTING.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/CONTRIBUTING.md). Security: [`SECURITY.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/SECURITY.md).

## Citation

If you use the verifier or its evaluation, cite it with [`CITATION.cff`](https://github.com/azterizm/legal-rag-verifier/blob/main/CITATION.cff)
("Cite this repository" on GitHub).

## Consultancy

`legal-rag-verifier` is built and maintained by **Abdullah Memon** at
**[Memon Systems Ltd](https://memonsystems.com)**, which designs and audits retrieval systems for
legal and other regulated domains. For help putting verification into production or testing your
own legal RAG, see [engagements](https://memonsystems.com/engagements) or write to
abdullah@memonsystems.com.

## Licence

AGPL-3.0-only (see [`LICENSE`](https://github.com/azterizm/legal-rag-verifier/blob/main/LICENSE)). Contains public sector information licensed under the
Open Government Licence v3.0 (see [`NOTICE`](https://github.com/azterizm/legal-rag-verifier/blob/main/NOTICE)).
