Metadata-Version: 2.4
Name: sourced-evidence
Version: 0.4.1
Summary: Per-claim grounded/contradicted/unverified checking of LLM output against its own source context -- never a blended trust score.
License-Expression: MIT
Project-URL: Homepage, https://github.com/MaXiMo000/sourced
Project-URL: Source, https://github.com/MaXiMo000/sourced
Project-URL: Issues, https://github.com/MaXiMo000/sourced/issues
Project-URL: Changelog, https://github.com/MaXiMo000/sourced/releases
Keywords: verification,grounding,llm,rag,evidence,provenance
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: judge
Requires-Dist: anthropic>=1.8; extra == "judge"
Dynamic: license-file

# sourced

[![ci](https://github.com/MaXiMo000/sourced/actions/workflows/ci.yml/badge.svg)](https://github.com/MaXiMo000/sourced/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/sourced-evidence)](https://pypi.org/project/sourced-evidence/)

**Checks an LLM's output against its own source context, claim by claim --
`grounded`, `contradicted`, or `unverified`, never one blended trust score.**

<p align="center"><img src="https://raw.githubusercontent.com/MaXiMo000/sourced/main/docs/demo.svg" alt="sourced marking two claims grounded, one contradicted by the source's 8%, and one unverified" width="100%"></p>

A RAG pipeline retrieves the right documents and still generates a sentence
those documents don't support -- a wrong number, an entity the source never
mentions, a stat close enough to sound plausible. Most RAG evaluation
scores *retrieval quality*: did the right chunks come back. Almost nothing
checks the *output* sentence by sentence against what was actually
retrieved. `sourced` does that second, narrower thing.

```
$ sourced check output.txt source.txt
[OK] Microsoft reported revenue of $56 billion.
[XX] Microsoft grew 40% year over year.
       claims 40% near 'Microsoft', but the source's own text near that
       entity says ['56 billion', '8%']
[??] The outlook remains uncertain.
       no checkable numbers, quoted text, or capitalized entities in this claim

1 grounded, 1 contradicted, 1 unverified
```

(That transcript is real output, not illustrative -- `source.txt` says growth
was 8%; note the second claim repeats "Microsoft" by name rather than
saying "it," because pronoun coreference isn't resolved -- see "What this
does NOT do.")

## Install

```
pip install sourced-evidence   # the command it installs is `sourced`
```

(`sourced` was already taken on PyPI, so the package is `sourced-evidence`; the command
is still `sourced`.)

## Use

```
sourced check <output-file> <source-file> [source-file ...] [--json]
```

`output-file` is the LLM's generated text. `source-file`(s) are the
context it was supposed to be grounded in -- the retrieved chunks, the
document it summarized, the transcript it's answering questions about.
Multiple source files are concatenated before checking.

Exit code is `1` only if any claim is `contradicted` -- same convention as
[`receipt`](https://github.com/MaXiMo000/receipt) and
[`invariant`](https://github.com/MaXiMo000/invariant): `unverified` is a
legitimate "can't tell," not a failure.

## How a claim gets checked

1. **Split into claims.** Every sentence in the output is one claim
   candidate -- no attempt to tell a factual assertion from an opinion or
   a hedge (`sourced/claims.py`).
2. **Extract signals.** Numbers, quoted substrings, and capitalized
   entity-shaped phrases (`sourced/signals.py`) -- the concrete, matchable
   facts a source text either does or doesn't contain. Numbers are compared
   by value, not as text: `1,000` = `1000`, `$56 billion` = `56B` = `56bn`,
   `8%` = `8 percent`, and `5` is never "found" inside a source's `56`.
3. **Classify against the source** (`sourced/check.py`):
   - **`grounded`** -- every number, quote and name in the claim appears
     **in one passage** of the source (three consecutive sentences),
     together with at least 60% of the claim's own content words, and the
     source's closest sentence is not negated where the claim isn't (or
     the other way round). At least one signal must be a real anchor: a
     claim whose only match is its capitalized first word ("Operating
     costs rose...") stays `unverified`.
   - **`contradicted`** -- a passage about the same thing (every name in
     the claim, most of its words) states a different number, and not the
     claim's. The report shows the numbers that passage gives near those
     names.
   - **`unverified`** -- everything else, with the reason: which signals
     never appear together, a negation mismatch, or names that co-occur
     while most of what the claim says about them doesn't.

**Why local.** The first version grounded a claim when each signal
appeared *anywhere* in the source. Checked against the 73 KB Wikipedia
article on PostgreSQL, it called three false sentences grounded:

```
Stonebraker returned to Berkeley in 1991 ...   (the source: 1985; 1991 is elsewhere)
POSTGRES reused most of the Ingres code.        (the source: "but not its code")
Berkeley released POSTGRES under a GPL license. (the source: an MIT License variant)
```

Now: the first is `contradicted` (the passage says 1982 and 1985), the
other two `unverified`, and the three true sentences in the same summary
stay `grounded`.

## Measured on RAGTruth

[RAGTruth](https://github.com/ParticleMedia/RAGTruth) labels real answers
from GPT-4, GPT-3.5, Llama-2 and Mistral span by span for hallucination.
`bench/ragtruth.py` scores sourced on its 2,700-answer test split, one
sentence at a time (a sentence is hallucinated when a labelled span
overlaps it). String matching only, no `--judge`:

| task | sentences | hallucinated | `grounded` | of which hallucinated | `contradicted` | precision |
|---|---|---|---|---|---|---|
| summarization | 5,083 | 263 | 1,540 | 2.1% | 23 | 17% |
| QA | 6,942 | 469 | 1,146 | 1.6% | 9 | 22% |
| data-to-text | 8,236 | 936 | 393 | 2.8% | 23 | 61% |
| **all** | **20,261** | **1,668 (8.2%)** | **3,079** | **2.0%** | **55** | **36%** |

How to read it:

- **`grounded` is the signal to rely on.** 2.0% of grounded sentences
  were hallucinated, against an 8.2% base rate, and 96% of all hallucinated
  sentences were *not* called grounded.
- **`contradicted` is a lead, not a verdict.** 36% precision overall
  (4.4x the base rate), 61% on structured data, under 25% on free prose,
  where a different number near the same names is often a different fact.
- **Most sentences come back `unverified`** -- by design, but it is
  what `--judge` is for.

The first run found the false-contradiction classes that are fixed now,
each in real output: a preamble line ("Here's the summary within 66
words:") read as part of the first claim; list numbering and `(Passage 1)`
citations read as figures; a capitalized first word ("To", "Rinse")
anchoring a contradiction; `$1.6b` not matching "1.6 billion"; and a number
stated in another passage about the same subject. Contradiction precision
went from 14% to 36% and false-grounded from 2.9% to 2.0%. One tried fix
was reverted: ignoring clock times removed false contradictions but let
wrong opening hours read as grounded.

## In code, and in bulk

```python
from sourced import check, assert_grounded

report = check(answer, [doc1, doc2])
assert_grounded(answer, [doc1, doc2])                          # contradictions fail
assert_grounded(answer, [doc1, doc2], allow_unverified=False)  # so does anything unconfirmed
```

```
$ sourced batch results.jsonl          # one {"output", "sources"} per line
2 record(s): 1 grounded, 1 contradicted, 0 unverified claims; 1 record(s) with a contradiction
```

`--json` prints one result per record. A malformed line is reported and
counted, never skipped silently, and it fails the run.

## `--judge`: let Claude decide what string matching can't

```
pip install 'sourced-evidence[judge]'
sourced check output.txt source.txt --judge
```

String matching can't see paraphrase ("costs went down a lot" vs
"operating costs fell sharply") or a flipped verb, so those claims come
back `unverified`. `--judge` sends **only those claims** -- never ones
already decided -- to Claude (`claude-opus-5` by default, `--judge-model`
to change it) with the source, and applies its grounded / contradicted
verdicts.

The judge is held to the same standard as the rest of the tool: every
verdict must quote its evidence **verbatim from the source**, and sourced
checks that quote is really there. A verdict backed by a quote that isn't
in the source is discarded and the claim stays `unverified` -- the model
can't talk a claim into `grounded`. Its reasons show up in the report:

```
[OK] Costs went down a lot after the reorganization.
[XX] Operating costs rose after the restructuring.
       judged contradicted by claude-opus-5: the source says "Operating costs fell sharply"
```

(Shape of the output, not a recorded run.) One request per check: the
source goes in a cached system prompt, so re-checking against the same
source -- the usual CI loop -- reads it at cache prices. A safety refusal
is re-run server-side on Anthropic's recommended fallback model
(`fallbacks: "default"`); if the chain still refuses, the claims stay
`unverified` and the report says so. Credentials are whatever the
anthropic SDK finds (`ANTHROPIC_API_KEY`, or `ant auth login`). Without
`--judge`, sourced never makes a network call.

## What this does NOT do

This is the part worth reading before trusting a result.

- **No semantic understanding, without `--judge`.** "Revenue was $56
  billion" and "the company made fifty-six billion dollars" are the same
  fact, and the string pass will not see it that way -- it matches strings
  and numbers, not meaning. `--judge` (above) is the optional
  meaning-level pass; nothing requires it.
- **No real named-entity recognition.** `signals.proper_nouns()` is a
  capitalization heuristic, not a trained model. "Bank of America" splits
  into `Bank` and `America` because a lowercase joiner breaks the
  capitalized-word chain. Documented in `signals.py`, not hidden.
- **No real sentence tokenizer.** `claims.split_claims()` is a regex.
  "Dr. Smith signed the report." reads as two sentences. Ordinary prose
  splits correctly; abbreviation-heavy text won't always.
- **No pronoun or coreference resolution.** "Microsoft reported $56B. It
  grew 40%." -- the second sentence's "It" is correctly excluded as an
  entity (it's a pronoun, not a name), so there's no anchor to check that
  claim's number against, and it comes back `unverified` rather than
  `contradicted`. Each claim is checked entirely on its own; a real fix
  needs coreference resolution, real NLP work of its own, not attempted
  here.
- **Contradiction detection is deliberately conservative**, and that cuts
  both ways: a real contradiction with no shared entity to anchor on comes
  back `unverified`, not `contradicted` -- this will under-report
  contradictions before it will over-report them. That's the intended
  trade for a tool whose whole point is not manufacturing false
  accusations against a source.

## Tests

```
pip install -e .
python tests/test_claims.py     # sentence splitting
python tests/test_signals.py    # number/quote/entity extraction
python tests/test_check.py      # the grounded/contradicted/unverified decision
python tests/test_cli.py        # the real CLI entry point, real files, real argv
python tests/test_judge.py      # --judge, against a stand-in client (no network)
```

Several tests exist specifically because a first pass got something
wrong and testing against ordinary prose (not synthetic numbers-only
input) caught it -- e.g. a number regex that swallowed a following
sentence period (`"42."` parsed as the number `42.`), a sentence-
initial entity ("Microsoft" opening a sentence) needing case-insensitive
matching against a source that uses the same word lowercase mid-sentence,
without that leniency being used for anything else, and American-style
closing punctuation putting the
period *inside* a quote (`"four hours."` not `"four hours".`) silently
merging two sentences into one claim -- the sentence-end regex only ever
checked for `[.!?]` immediately before the split point, never a quote
mark sitting in front of it.

MIT licensed.
