# TextGraph Retrieval Benchmarks

Quality *and* cost for every run (G7 — no number without its cost). Generated by
`python -m benchmarks.retrieval --write`; deterministic given the same corpus.

| benchmark | queries | k | recall@k | MRR | mean tokens/query | p50 ms | p95 ms |
| --- | --- | --- | --- | --- | --- | --- | --- |
| fixture:docs (text: BM25+PPR+RRF) | 5 | 5 | 0.700 | 0.800 | 131 | 1.04 | 5.20 |
| fixture:docs (vision: MaxSim/hash) | 5 | 5 | 0.800 | 0.600 | 196 | 26.63 | 42.51 |

## Method

- **Retrieval (text):** hybrid BM25 (chunk passages) + Personalized PageRank over
  the dual-node entity+chunk graph, fused with Reciprocal Rank Fusion (L8).
- **Retrieval (vision, Phase 8):** late-interaction **MaxSim** over multi-vector
  *pages* (documents). The default embedder is deterministic (hash tokens, CI-safe);
  the `[vision]` extra swaps in a ColPali/ColQwen model over rendered page images.
  Its recall@k counts whether a retrieved page names a gold entity (page-level).
- **recall@k (text):** fraction of a query's gold entities appearing in the top-k hits.
- **MRR:** mean reciprocal rank of the first gold entity.
- **cost:** estimated tokens returned per query (~4 chars/token) and wall-clock
  latency percentiles over the query set.

## Reproduce

```bash
python -m benchmarks.retrieval          # fixture benchmark, zero downloads
python -m benchmarks.retrieval --write   # regenerate this file
python -m benchmarks.retrieval --data DIR  # external set (DIR/queries.json)
python -m benchmarks.security            # FGAC overhead + no-bleed (Phase 9)
python -m benchmarks.reasoning           # Graph-of-Thoughts adaptive vs static (Phase 10)
python -m benchmarks.quality             # extraction quality vs a gold set (Sprint 3.1)
```

## Graph-of-Thoughts (Phase 10) - adaptive vs static reasoning cost

The reasoner answers a query by building a graph of thoughts, each bound to real graph
evidence (§4.2). Cost is the number of graph tool calls (search/neighbors/path/why/gql) it
makes — a deterministic, machine-independent count. Adaptive gates topology expansion on
runtime complexity; static always expands the full topology (the DGoT/AGoT baseline).

| Question | complexity | adaptive calls | static calls | grounded |
|---|---|---|---|---|
| who is John Doe | 1 | 4 | 16 | True |
| which bank was involved | 1 | 4 | 16 | True |
| how is Acme Corp connected to Delta Trust | 2 | 6 | 16 | True |
| what is the link between Beta Ltd and Gamma Holdings | 2 | 6 | 16 | True |
| who controls Gamma Holdings | 1 | 4 | 16 | True |

- **Total: adaptive 24 vs static 80 tool calls -> 70% cheaper**, and adaptive is never more
  expensive than static on any query. Reproduce with `python -m benchmarks.reasoning`.
- **Every reasoning step cites real graph spans (ESCARGOT): True** — a thought that gathers
  no graph evidence is pruned, so the finished chain is verifiable end to end.
- Adaptive spends where difficulty is: a single-entity question runs a cheap linear chain;
  a connection question spawns the Aggregation/Refinement branches.

## Enterprise FGAC (Phase 9) - security-aware retrieval

Cost of enforcing access control *inside* traversal (§3.2 — an unauthorized node's
transition probability drops to 0), measured on `tests/fixtures/corpora/secure`
(2 docs, 5 queries; a batch of all queries timed per rep, 40 reps):

| Configuration | p50 (ms) | p95 (ms) |
|---|---|---|
| baseline (no policy) | 7.19 | 10.13 |
| security-aware (full access) | 8.18 | 10.69 |

- **Overhead ~ +14% p50** for the security-aware walk under a full-access policy
  (the guard's per-query cost, isolated from any restriction). Absolute numbers are
  machine-dependent; reproduce with `python -m benchmarks.security`.
- **Transparency:** with everything authorized, secured hits are **byte-identical** to
  the unsecured result — the guard adds nothing but cost when nothing is restricted.
- **No-bleed:** under a one-document policy, the number of unauthorized hits that leak
  through PPR/paths/summaries is **0** — proven by the red-team suite
  `tests/integration/test_security_redteam.py` (which also asserts the leak is *real*
  without a policy, so the result can't be vacuous).

LoCoMo / LongMemEval-S: place the labelled set at `DIR/queries.json` (each item
`{"query": ..., "gold": [...]}`) and a corpus at `DIR/corpus/`. Absent data is
skipped, so CI never downloads.

## Extraction quality (Sprint 3.1) - vs a hand-labelled gold set

Fixture `tests/fixtures/corpora/docs`, deterministic rules extractor (default install, zero LLM). Every correctness number is deterministic; reproduce with `python -m benchmarks.quality`.

| Metric | Precision | Recall | F1 |
|---|---|---|---|
| Entities | 1.000 | 1.000 | 1.000 |
| Asserted relations | 0.833 | 0.833 | 0.833 |

- **False / hallucinated edge rate: 0.167** (1 of 6 asserted edges). The one false edge is a coreference error (`Beta Ltd → Gamma Holdings` where the source's "the company" refers to Acme Corp) — reported honestly rather than hidden. Negated relations ("did not transfer") are correctly tagged `polarity=neg` and excluded from asserted edges, so the extractor is credited for *not* asserting them.
- **Citation coverage: 1.000** — every asserted relation edge carries a source span, and byte-level re-verification of those spans is gated at **100%** by `tests/integration/test_edge_provenance.py` (G3). This is the number most GraphRAG systems can't report at all.

## Honest peer comparison

A fair *head-to-head* on a shared benchmark harness against Graphiti / LightRAG / HippoRAG / Neo4j GraphRAG Python is **not yet feasible** here — those systems are LLM-driven and evaluated on QA datasets (e.g. LoCoMo, MultiHop-RAG) with different tasks, and running them requires API keys and non-deterministic pipelines. Rather than fabricate numbers, this is a **capability comparison** on the axes that actually differ; the retrieval numbers above are ours, measured.

| Capability | TextGraph | Typical LLM GraphRAG peer |
|---|---|---|
| Byte-level provenance on every edge (re-verifiable span) | ✅ 100% (gated) | ✗ usually chunk-level or none |
| Deterministic build (byte-identical `graph.json`) | ✅ gated in CI | ✗ LLM extraction is non-deterministic |
| Works with **zero LLM calls** by default | ✅ | ✗ LLM required to build the graph |
| Hallucinated-edge rate is *measured and published* | ✅ 0.167, honest | rarely reported |
| Bi-temporal claim versioning (`[t_valid, t_invalid)`) | ✅ | partial (Graphiti) / none |
| Local-first, no service required | ✅ DuckDB default | often needs a graph DB service |
| Recall ceiling on hard multi-hop QA | lower (rules default) | higher (LLM extraction) |

**Honest weaknesses:** the deterministic default trades peak recall for determinism + provenance — an LLM extractor will find edges our rules miss (see the coref error above), which is exactly why the `[ie]` GLiNER and opt-in L4 LLM paths exist. Where we win is *trust*: reproducibility and re-verifiable citations, which the peers largely do not offer. A shared-harness QA comparison is tracked as future work and will be added here only when it can be run fairly.


## Zero-config defaults (PDF + entity resolution)

A default install (`pip install textgraph-kg`, no flags, no extras) already ingests PDFs and resolves entity aliases. Proof:

- **PDF ingestion, default:** a generated `.pdf` ingests as `format=pdf` (OK) -- `pypdf` is a core dependency, no `[ingest]` extra required.

- **Entity resolution, default** (rules backend) -- before/after on a fixed 3-doc corpus where one org appears as *Acme Corporation / Acme Corp / ACME*. Resolution is non-destructive (aliases are kept and linked, not deleted), so the win is *unified identity*, not fewer nodes:

| | Acme surface forms | SAME_AS links | canonical identities |
|---|---|---|---|
| ER off (`resolve_entities=False`) | 2 | 0 | 0 |
| **ER on (default)** | 2 | 2 | 1 |

With ER on, the 2 surface forms are unified under 1 canonical identity via 2 cited `SAME_AS` link(s) -- so a query for `ACME` reaches facts filed under `Acme Corporation`, and PageRank/paths treat them as one node. Zero configuration.

- **Latency** (machine-dependent): build 38 ms (ER on) vs 29 ms (ER off); a hybrid `search` over the resolved graph in 1.0 ms.
