# TextGraph — PLAN.md

> Written before any pipeline code, per Section 10 of the master build prompt. Phase 0 is the repo + CI/CD foundation.

## 1. The core translation problem (in my own words)

Graphify gets its guarantees — determinism, locality, provenance, cost linearity — *for free*, because source code is defined by a formal grammar. A tree-sitter parser is a pure, total function from bytes to a concrete syntax tree: the same file always yields the same tree, every node has an exact byte span, and a one-line edit only re-parses locally. Natural language hands you none of this. There is no free, deterministic, total parser for English prose; the "grammar" is statistical, ambiguous, and context-dependent, and the tools that approximate it (NER, coreference, relation extraction, LLMs) are heavier, fuzzier, and — if used naively — non-deterministic and unbounded in cost. The central engineering problem of TextGraph is therefore: **how do you recover Graphify's four guarantees for a domain that does not give them to you for free?**

The answer this architecture commits to is *stratification*. Rather than throwing the whole corpus at one probabilistic model, extraction is split into layers ordered by how much trust and non-determinism each introduces: a fully deterministic structural spine (L0/L1, zero models) is built first and is always sufficient to ship a usable graph; encoder-based information extraction (L3) runs next with pinned weights and fixed seeds so it is *reproducibly* non-trivial rather than random; an LLM pass (L4) is quarantined behind an explicit opt-in flag, hard-budgeted, cached, and tagged `GENERATED` so its output can never be mistaken for ground truth. Every layer is a pure function of the layer below it plus a pinned config hash, every edge carries a re-verifiable byte-range citation and one of four confidence tags, and contradictions are invalidated rather than deleted. In effect, the non-determinism that natural language forces on you is *isolated, pinned, tagged, and made auditable* — never allowed to leak into the parts of the system that promise determinism.

## 2. Confirmed tech-stack decisions

| Decision | Choice | Notes / deviations |
|---|---|---|
| Core language | Python 3.11+ | Matches encoder/NLP ecosystem. |
| Package manager | `uv` (primary) | Installed at a non-PATH location on the build machine (`0.9.18`); `pip`/`pipx` documented as fallbacks. **No deviation** from the spec's intent. |
| Content addressing | `blake3` | Prebuilt Windows wheels available; core dependency. |
| Storage (v0.1–v1.0) | NetworkX in-memory + DuckDB/Parquet on disk, behind a `GraphStore` interface | Nothing above L6 touches NetworkX directly. |
| Serialization | Canonical JSON (sorted keys, compact separators, UTF-8, trailing newline) | Underpins G1. |
| Static graph viewer | `sigma.js` + `graphology`, server-precomputed ForceAtlas2, single self-contained file | Phase 1+. |
| Pipeline console | Separate local web app (`textgraph console`) | Phase 6. |
| CI | GitHub Actions | Confirmed — repo is on GitHub (`krishddd/Wiki_textgraph`). |
| Heavy deps | Behind extras: `[full]`, `[ocr]`, `[vision]`, `[neo4j]`, `[mcp]`, `[security]` | Lean default install. |

**Deviation flagged:** the build machine is Windows 10 / Python 3.11.1. `uv` is installed but not on `PATH`; the build uses its absolute path. Mypy/ruff targets are unaffected. No spec goals change.

## 3. Roadmap (Phase 0–10) with effort estimates

| Phase | Title | Est. effort | Notes |
|---|---|---|---|
| **0** | Repo & CI/CD foundation | 1 unit | **In progress.** No pipeline code; scaffold + CI + core primitives. |
| 1 | Structural spine (L0+L1) | 3 units | First usable `graph.json`/`graph.html`/`REPORT.md`, zero models. |
| 2 | Encoder IE (L2+L3) | 4 units | Coref is highest-ROI; pin weights, keep determinism green. |
| 3 | Entity resolution (L5) | 3 units | Blocking → Splink scoring → non-destructive clustering. |
| **4** | Retrieval (L6+L7+L8) | 4 units | **Done.** Claims + analytics + dual-node hybrid retrieval + MCP; `BENCHMARKS.md`. |
| **5** | Temporal + incremental | 3 units | **Done.** Bi-temporal invalidation (L6), DuckDB persistent store, incremental cache + `watch`, CLI parity. |
| **6** | Optional LLM + UI polish + packaging | 3 units | **Done — v1.0.0 shipped.** L4 opt-in LLM ✅, `textgraph console` web UI ✅, wheel packaging ✅, version bump + release ✅. |
| **v1.1** | Interactive console + retrieval quality | 3.5 units | **Done.** Layout, canvas renderer, community sidebar, inspect, search, tag-filter, path mode, temporal slider, shared offline `graph.html` viewer, reranker (recall/MRR up), whole-package coverage gate. |
| **7** | GQL / standards layer | 2 units | **Done.** Pure-Python ISO-GQL/Cypher subset (`textgraph/gql/`) — pattern matching + quantified paths + WHERE/RETURN/aggregation; `textgraph gql` CLI; pattern-tools round-trip. |
| **8** | Vision-native multimodal ingestion | 2 units | **Done.** MaxSim late-interaction page retrieval (`l8_retrieval/vision/`); deterministic hash embedder default, ColPali behind `[vision]`; `textgraph vision` CLI; benchmarked. |
| **9** | Enterprise fine-grained access control | 3 units | **Done.** ReBAC+ABAC policy engine (`textgraph/security/`); security-aware PPR/path/traversal (unauthorized transition probability → 0); red-team no-bleed suite; `textgraph secure` CLI; OpenFGA behind `[security]`. |
| **10** | Dynamic agent reasoning (GoT) | 2 units | **Done.** Complexity-gated Graph-of-Thoughts reasoner (`textgraph/got/`); four operators bound to real `neighbors`/`path`/`why`/`gql` evidence; every step cited; `textgraph reason` CLI; adaptive ~70% cheaper than static baseline. |

No reordering proposed. Phases 0–6 are strictly sequential and bottom-up (L0→L1→L2→L3→L5→L6→L7→L8→L9, with L4 last among core layers). v1.1 (interactive UI) and Phases 7–10 are post-v1.0.

## Post-v1.0 — v1.1 UI (in progress)

- [x] Deterministic force-directed layout (`l7_analytics/layout.py`), hash-seeded + fixed
      iterations, coords baked onto entity nodes; `graph.json` stays byte-identical
- [x] Interactive canvas console: community-coloured/PageRank-sized nodes, pan/zoom/hover,
      Communities sidebar with toggles, confidence-tag filter, search highlight,
      click-to-inspect (cited claims + validity windows), path mode
- [x] Temporal slider — drag a date, superseded claims fade out (per-edge `[t_valid,
      t_invalid)` window on `/api/graph`; label reddens on supersession)
- [x] Shared canvas renderer (`console/renderer.py`): the live console and the offline
      `graph.html` render identically via a `TG` adapter (API vs embedded data +
      client-side path/search). Retires the concentric-ring SVG stub.
- [x] Retrieval-quality debt: second-stage reranker (`rerank.py`) interleaves answer
      entities with evidence passages — recall@5 0.60→0.70, MRR 0.29→0.80; cross-encoder
      behind `[rerank]`. Coverage gate widened from `core/` to whole-package (88% floor).

## 4. Open questions / assumptions

1. **Target OS / deployment.** Assuming cross-platform (dev on Windows 10, CI on Linux via GitHub Actions ubuntu-latest + a Windows matrix leg where cheap). CI matrix: Python 3.11 + 3.12.
2. **Corpus scale for v1.0.** Assuming up to ~10k docs / low-millions of edges — the range where NetworkX-in-memory + DuckDB-on-disk is comfortable and `sigma.js`/WebGL still renders. Larger scale is a post-v1.0 storage-engine swap (Phase 7 groundwork).
3. **CI target.** Assumed GitHub Actions (confirmed by repo host).
4. **Hosted/always-on product.** Assumed **out of scope** for v1.0 — TextGraph ships as a local CLI + MCP server + skill. A hosted service is future/optional.

*These are working assumptions; flag any you'd like changed.*

## 5. Repo scaffold

See Section 5.2 of the build prompt — generated in Phase 0 exactly as specified (`textgraph/` package with `core/`, `l0_ingest/`…`l9_artifacts/`, `store/`, `mcp/`, `cli.py`; `ui/`, `schema/`, `tests/{unit,integration,determinism,fixtures}`, `benchmarks/`, `docs/`, `.github/workflows/`, root docs).

---

## Phase 6 — Definition of Done (in progress)

- [x] **L4 optional LLM (`l4_llm_optional/`)**: opt-in (`build --llm`), off by default;
      GENERATED community summaries (`Summary` nodes + `SUMMARIZES` edges), quarantined by
      tag so they never masquerade as cited facts; dependency-free OpenAI-compatible
      client with the API key read from the environment only (never hashed/persisted);
      hard-budgeted + response-cached; determinism holds (byte-identical with LLM off).
      Verified live against a vLLM Nemotron endpoint; 8 mock-client tests in CI.
- [x] **UI / `textgraph console`**: dependency-free local web app over the QueryEngine —
      all 8 typed tools in the browser, cited, self-contained page (no CDN, G2), pure
      `route()` unit-tested without a socket
- [x] **Packaging**: wheel builds and bundles the new packages (console, l4_llm_optional,
      duckdb_store, watch) with the `textgraph` console-script entry point
- [x] **v1.0.0 shipped** (2026-08-04): version bumped, CHANGELOG cut, tagged `v1.0.0`;
      the release workflow re-verifies determinism on the built wheel before publishing

## Phase 10 — Definition of Done (met)

- [x] **Graph-of-Thoughts reasoner** (`textgraph/got/`), pure-Python & deterministic: thought
      vertices with roles (Plan/SubProblem/Hypothesis/VerificationStep/DistilledSummary) and
      the four operators — Generation/Aggregation/Refinement/Distillation (§4.1)
- [x] **Every operator binds a real graph query as evidence** (ESCARGOT): Generation=`neighbors`,
      Aggregation=`path`, Refinement=`why`+`gql` triple check; every substantive thought cites
      re-verifiable `[doc:span]` bytes, and an un-grounded thought is pruned
- [x] **Complexity-gated / adaptive** (DGoT/AGoT): runtime complexity (entities the query names)
      gates topology expansion; a `static` full-topology mode is the baseline; tool-call budget
      hard-bounded (G7)
- [x] **Adaptive cost < static baseline, shown empirically**: `benchmarks/reasoning.py` → ~70%
      fewer tool calls (24 vs 80) at equal grounding; `BENCHMARKS.md`
- [x] **Read-only**: reasoning is query-time over the assembled graph — `graph.json` byte-identical
      (G1); `textgraph reason` CLI
- [x] +11 tests (model, four operators end-to-end, gating, adaptive<static, grounding invariant,
      determinism, CLI); whole-package coverage ≥ 88%; ruff + strict mypy green

## Phase 9 — Definition of Done (met)

- [x] **ReBAC + ABAC policy engine** (`textgraph/security/`), pure-Python & deterministic:
      Zanzibar/OpenFGA relation tuples (`owner`/`viewer`/`member`/`parent` + usersets,
      transitive group/folder policy paths) and Cedar-style attribute conditions
      (`MinClearance`/`IpAllowlist`/`TimeWindow`); depth-bounded, cycle-safe (G1/G7)
- [x] **Security-aware traversal, not a post-filter** (§3.2): PPR runs on a graph masked to
      the principal's authorized nodes so an unauthorized node's transition probability is
      `0`; `path` prunes restricted nodes+edges in Dijkstra/Yen; `neighbors`/`why`/
      `timeline`/`contradictions`/`communities`/`vision_search` are all context-aware, and
      an edge is hidden unless its own source document is authorized
- [x] **Default install unaffected**: with no policy/context every tool is byte-identical to
      the un-secured engine — access control is query-time only, `graph.json` untouched (G1/G2)
- [x] **OpenFGA behind `[security]`**, import-guarded with a clean fallback to the built-in
      `rebac`; `textgraph secure … --policy … --principal …` CLI
- [x] **Red-team proves zero context-bleed** through PPR/paths/neighbors/summaries/vision
      (`tests/integration/test_security_redteam.py`, incl. a real-leak control);
      **overhead measured** (~+14% p50, full access) in `benchmarks/security.py` → `BENCHMARKS.md`
- [x] +33 tests; whole-package coverage ≥ 88%; ruff + strict mypy green

## Phase 8 — Definition of Done (met)

- [x] Late-interaction **MaxSim** operator + multi-vector representation
      (`l8_retrieval/vision/maxsim.py`), pure-Python and deterministic
- [x] Deterministic **hash embedder** default (SHAKE-256, CI-safe, zero-GPU) so the whole
      vision pipeline runs/tests reproducibly; **ColPali/ColQwen behind `[vision]`**,
      import-guarded with clean fallback to `hash`
- [x] `QueryEngine.vision_search()` ranks documents-as-pages by MaxSim; `textgraph vision`
      CLI (`--backend hash|colpali`); pages seed PPR via existing entity mentions
- [x] **Default install unaffected**: embeddings are query-time only, so `graph.json`
      stays byte-identical (verified) — G1/G2/G3 untouched
- [x] **Value shown with a benchmarked number** (`BENCHMARKS.md`): vision (MaxSim/hash)
      recall@5 0.80 vs text 0.70, with its token/latency cost
- [x] +7 tests; whole-package coverage ≥ 88% (89%); ruff + strict mypy green

## Phase 7 — Definition of Done (met)

- [x] Pure-Python, deterministic GQL (ISO/IEC 39075 / Cypher) subset over the property
      graph: tokenizer + recursive-descent parser + executor (`textgraph/gql/`)
- [x] Property-graph pattern matching in all directions + **quantified/variable-length
      relationships** (`*min..max`, loop-free + depth-capped, G7)
- [x] `WHERE` (comparisons, string ops, `IN`, `AND`/`OR`/`NOT`), `RETURN` with
      `type()`/`labels()`/`id()` + `count(*)` aggregation, `DISTINCT`, `ORDER BY`,
      `SKIP`, `LIMIT`; stably-ordered deterministic rows
- [x] `textgraph gql <corpus|.duckdb> "<query>"` CLI (also queries a persisted snapshot)
- [x] Read-only surface over the same graph the L8 tools use — G1/G2/G3 untouched;
      pattern-based tools (`neighbors`/`path`/`contradictions`) **round-trip** to GQL
- [x] 17 tests; whole-package coverage stays ≥ 88% (89%); ruff + strict mypy green

## Phase 5 — Definition of Done (met)

- [x] **Bi-temporal claims (L6)**: later-dated opposite-polarity claims supersede
      earlier ones — `t_invalid` closed + cited `SUPERSEDES` edges (invalidation, not
      deletion). Ordering from in-text dates only (no wall-clock, G1); undated conflicts
      left open and surfaced via `CONTRADICTS`. `timeline`/`why`/`contradictions` +
      `explain` show the `[t_valid, t_invalid)` window. `corpora/temporal` fixture.
- [x] Persistent `DuckDBGraphStore` behind the `GraphStore` ABC (`[graph]`/duckdb extra),
      import-guarded; exact Node/Edge round-trip; `build --store` + `.duckdb` query paths
      load without a rebuild.
- [x] Incremental rebuild: per-document IE cache keyed by `(doc_id, config_hash)`;
      byte-identical to a full build; `textgraph watch <dir>` (content-addressed change
      detection, refuses nested output dir).
- [x] CLI parity: `neighbors` / `timeline` / `contradictions` / `communities` / `stats`
      verbs over the same `QueryEngine`; 193 tests; determinism holds.

## Phase 4 — Definition of Done (met)

- [x] L6 claim reification: relation edges → citable `Claim` nodes with `t_valid`
      grounded to the nearest in-sentence `Date` (`t_invalid` deferred to Phase 5);
      direct edges preserved; reified edges re-verify against raw bytes
- [x] L7 analytics (pure-Python, deterministic): weighted PageRank + Brandes
      betweenness + label-propagation communities (c-TF-IDF labels); god nodes,
      bridges, orphans, contradictions; folded into the graph as node properties +
      `CONTRADICTS` edges. Leiden behind `[graph]`
- [x] L8 dual-node retrieval graph (entity + `Chunk` nodes) + eight typed tools:
      `search` (BM25 + Personalized PageRank + RRF, local/global routing), `neighbors`,
      `path` (maximum-likelihood k-shortest), `why`, `timeline`, `contradictions`,
      `communities`, `stats` — every result bounded (token budget) and `[doc:span]`-cited
- [x] MCP surface (`textgraph.mcp` specs + dispatcher, CI-tested; stdio server behind
      `[mcp]`) + CLI `query` / `path` / `explain` over the *same* `QueryEngine`
- [x] First retrieval benchmark with cost: recall@k / MRR + tokens/query + p50/p95
      latency → `BENCHMARKS.md`; LoCoMo/LongMemEval-S guarded by local data presence
- [x] Agent-session integration test: answers via tool calls only, every citation
      re-verified against raw bytes; determinism CI still byte-identical (166 tests)

## Phase 3 — Definition of Done (met)

- [x] Blocking (deterministic keys, type-gated) + scoring (Jaro-Winkler + relational
      shared-neighbour) + non-destructive complete-linkage clustering
- [x] `SAME_AS` lattice to canonical nodes; original entities preserved (reversible)
- [x] `textgraph er audit` command; B-cubed metrics gated in CI (F1 ≥ 0.9 floor)
- [x] Blocking recall ≥ 0.99 on the ER fixture
- [x] god-node diagnostic flags an intentionally-injected over-merge
- [x] "Acme Corp" / "Acme Corporation" / "ACME" collapse; "Alpha Bank" stays separate
- [x] Splink behind `[er]`; determinism holds with L5; 133 tests, ~97% coverage

## Phase 2 — Definition of Done (met)

- [x] L2 substrate (sentences, coref-lite, negation/modality) + L3 entity/relation
      extraction; default deterministic rule backend, GLiNER behind `[ie]`
- [x] `textgraph build` produces entities (Org/Person/Money/Account/Date/Email) and
      typed relations (TRANSFERRED/CONTROLS/DIRECTOR_OF/…), tagged EXTRACTED/INFERRED
- [x] four-tier confidence taxonomy wired through the graph; negation/modality kept
- [x] Determinism CI still byte-identical with IE in the loop; models pinned/seeded
- [x] coref coverage reported in `manifest.json`; `schema.yaml` records observed types
- [x] ablation harness: L1-only vs. +encoder IE (edge/entity/relation delta)
- [x] CPU-only, model-free default path (zero GPU); 112 tests green, ~97% coverage

## Phase 1 — Definition of Done (met)

- [x] L0 ingestion across md/txt/html/docx/odt/rtf/epub/json/yaml/toml/log/chat
      (PDF behind `[ingest]`); offset fidelity + hierarchical chunking
- [x] L1 zero-model structure parse incl. Rationale/Requirement nodes; every edge
      `STRUCTURAL` with a re-verifiable span
- [x] L9 artifacts: graph.json + GRAPH_REPORT.md (10 questions) + graph.html +
      schema.yaml + manifest.json; `textgraph build`
- [x] Determinism CI byte-identical across three corpus shapes
- [x] Edge-level provenance re-verification (100%) across every fixture corpus
- [x] `GRAPH_REPORT.md` 10 grounded questions on docs / ADR / chat corpora
- [x] 91 tests green; ruff + strict mypy + ~97% core coverage

## Phase 0 — Definition of Done (tracking)

- [x] Repo scaffold (Section 5.2)
- [x] `pyproject.toml` + extras + lockfile
- [x] `CanonicalDoc` / offset-map / content-addressing primitives, unit-tested
- [x] `GraphStore` interface stubbed
- [x] Canonical-JSON serialization
- [x] Trivial deterministic pipeline + `graph.json`
- [x] CI workflows: lint-and-typecheck, test (matrix), determinism, pre-commit, benchmark, release
- [x] `.pre-commit-config.yaml`
- [x] Docs stubs: ARCHITECTURE.md, SKILL.md, docs/how-it-works.md, SECURITY.md, CHANGELOG.md, BENCHMARKS.md
- [x] `uv sync && uv run pytest` green
