Metadata-Version: 2.4
Name: treehash-rag
Version: 1.0.1
Summary: Vectorless RAG: tree + hash addressing with structural query resolution and a BM25 fallback, instead of one embedding per chunk. Zero required dependencies. Domain packs for medical, finance, and education.
Author: Harshit Rai
License: MIT
Keywords: rag,retrieval-augmented-generation,vectorless,vector-database,llm,nlp,bm25,information-retrieval,token-optimization,medical-nlp,finance-nlp,education
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.25; extra == "anthropic"
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: llm
Requires-Dist: anthropic>=0.25; extra == "llm"
Requires-Dist: openai>=1.0; extra == "llm"
Provides-Extra: docx
Requires-Dist: python-docx>=1.0; extra == "docx"
Provides-Extra: pdf
Requires-Dist: pdfplumber>=0.10; extra == "pdf"
Provides-Extra: formats
Requires-Dist: python-docx>=1.0; extra == "formats"
Requires-Dist: pdfplumber>=0.10; extra == "formats"
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == "langchain"
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.11; extra == "llamaindex"
Provides-Extra: server
Requires-Dist: fastapi>=0.100; extra == "server"
Requires-Dist: uvicorn>=0.23; extra == "server"
Provides-Extra: redis
Requires-Dist: redis>=4.0; extra == "redis"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == "postgres"
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == "mcp"
Provides-Extra: all
Requires-Dist: anthropic>=0.25; extra == "all"
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: python-docx>=1.0; extra == "all"
Requires-Dist: pdfplumber>=0.10; extra == "all"
Requires-Dist: langchain-core>=0.3; extra == "all"
Requires-Dist: llama-index-core>=0.11; extra == "all"
Requires-Dist: fastapi>=0.100; extra == "all"
Requires-Dist: uvicorn>=0.23; extra == "all"
Requires-Dist: redis>=4.0; extra == "all"
Requires-Dist: psycopg2-binary>=2.9; extra == "all"
Requires-Dist: mcp>=1.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# treehash

[![CI](https://github.com/Harshitraiii2005/treehasg_rag/actions/workflows/ci.yml/badge.svg)](https://github.com/Harshitraiii2005/treehasg_rag/actions/workflows/ci.yml)
![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)
![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)
![Dependencies: zero required](https://img.shields.io/badge/dependencies-zero%20required-brightgreen.svg)

**[Project site & benchmarks →](https://harshitraiii2005.github.io/treehasg_rag/)**

**A vectorless RAG engine.** Instead of embedding every chunk into a
vector database, `treehash` parses your documents into a tree, addresses
every node with a content+location hash, and answers most queries by
*resolving an address* — no embeddings, no vector index, average-case
O(1) lookup. When a query genuinely has no structural anchor, it falls
back to BM25 (the same statistical-ranking family behind
Elasticsearch/Lucene) computed only at section/document granularity —
never one score per paragraph.

Zero required dependencies. Install it and it runs — no model downloads,
no API key needed to use the core engine, no `sklearn`/`torch`/`numpy` to
even get started.

```bash
pip install treehash-rag
```

Published on PyPI as `treehash-rag` (`treehash` itself is already an
unrelated package there, checked directly, not assumed) — `import
treehash` either way. Verified directly: a brand-new virtualenv with
nothing else installed, `pip install treehash-rag`, `pip freeze` shows
nothing but the package itself, and the quickstart below runs
unmodified. To work from source instead:

```bash
git clone https://github.com/Harshitraiii2005/treehasg_rag && cd treehasg_rag
pip install -e .
```

```python
from treehash import KnowledgeBase

kb = KnowledgeBase(domain="finance")   # or "medical", "education", or None
kb.add_document("10k_2025", filing_text)

result = kb.query("What was revenue in Q2 2025?")
print(result.method)       # "structural" -- resolved by address, not search
print(result.hits[0]["content"])   # "6.1M"

ctx = kb.context_for_llm("What was revenue in Q2 2025?")
print(ctx["estimated_tokens"])     # a handful of tokens, not a whole page
```

Feed `context_for_llm()`'s output to your own LLM call, or let
`kb.ask(query, api_key=..., provider="anthropic")` do it and hand back the
**provider-reported** input/output token counts — so the reduction shows
up as a real number on your bill, not an estimate.

**Contents:** [Why vectorless](#why-vectorless) ·
[Persistence](#persistence) · [Domain packs](#domain-packs) ·
[The honest benchmark](#the-honest-benchmark-not-a-marketing-number) ·
[Postgres and sharding](#postgres-and-sharding-enterprise-scale) ·
[Semantic fallback](#semantic-fallback-optional-third-tier) ·
[Query log and live metrics](#query-log-and-live-metrics) ·
[Bring your own API key](#bring-your-own-api-key) ·
[Auditable answer trail](#auditable-answer-trail-finance-legal-compliance) ·
[Document formats](#document-formats) ·
[Capacity](#capacity-large-documents-multiple-documents) ·
[Server mode](#server-mode) ·
[MCP server](#mcp-server-for-agents) ·
[Framework integrations](#framework-integrations) ·
[What's implemented](#whats-implemented) ·
[What this does NOT prove yet](#what-this-does-not-prove-yet) ·
[Project layout](#project-layout) · [Run it](#run-it)

## Why "vectorless"

A conventional RAG pipeline chunks every document, embeds every chunk,
and stores one vector per chunk — a corpus of a million paragraphs means
a million vectors, and every query pays a similarity search over all of
them. `treehash` instead:

1. Parses documents into a **universal element tree** (sections,
   paragraphs, tables, table cells, lists, links — not just flattened
   text blobs), and hashes every node's address (WHERE) separately from
   its content (WHAT, enabling content-addressed dedup across documents).
2. Tries to **resolve the query to an address directly** — "revenue in
   Q1 2025", "the third chapter", "section 4" — via a rule-based resolver
   that either matches with a stated reason and confidence, or returns
   nothing so the caller can fall back honestly instead of guessing.
3. Only when that fails, ranks **section/document-level** text with BM25
   over an inverted index. No per-paragraph vectors are ever built, and
   no vectors are built at all — BM25 scores query terms against postings
   lists directly.
4. Only when *that* also fails — the query shares literally zero terms
   with anything in the corpus — an **optional** embedding-based tier
   gets one last try, scored over the same small, already-pruned
   section/document candidate set from step 3, never the whole corpus.
   Off by default; see [Semantic fallback](#semantic-fallback-optional-third-tier).

Both the resolver and the fallback ranker are pure Python
(`treehash/mathlib.py`) — the underlying math (TF, IDF, BM25's
saturation and length-normalization terms), not a library import.

## Persistence

```python
kb = KnowledgeBase(domain="finance", store="filings.db")
kb.add_document("q2_2025_10q", filing_text)
kb.close()

# ... later, a different process, no re-parsing ...
kb = KnowledgeBase(domain="finance", store="filings.db")
kb.query("revenue in Q2 2025")   # answers immediately, nothing re-ingested
```

`store=None` (the default) keeps everything in memory -- gone when the
process exits, zero setup, what every example above this section uses.
`store="path/to/file.db"` (or `store=":memory:"` for a SQLite-backed
instance that never touches disk) persists the tree, the content store,
and the fallback index to a SQLite file via a `StorageBackend`
abstraction (`treehash/storage.py`) that `TreeIndex` and `ContentStore`
depend on instead of raw dicts. `sqlite3` is Python stdlib, so this adds
no new required dependency. `store="postgresql://...?schema=<name>"`
does the same against Postgres instead of a local file -- see
[Postgres and sharding](#postgres-and-sharding-enterprise-scale) below
for when that actually matters.

Two things worth knowing:

- **Structural resolution stays a primary-key lookup** on the SQLite
  backend too (`SELECT ... WHERE address_hash = ?`, plus an indexed EAV
  table for metadata-filtered lookups like "section 4") -- opening a
  persisted knowledge base doesn't trade away the O(1)-average claim for
  addressed queries.
- **The fallback ranker changes with the backend.** `MemoryBackend` still
  uses the pure-Python `mathlib.BM25Index` described above. `SQLiteBackend`
  uses SQLite's own FTS5 extension and its native `bm25()` ranking
  function instead of re-deriving the same math in Python -- same
  section/document-only granularity, same "no per-chunk index" property,
  different engine underneath. `fallback="tfidf"` is ignored when the
  backend supports FTS5.
- **Opening an incompatible on-disk schema fails loudly**, not silently:
  a version stamp is written to the database, and a future incompatible
  schema raises `SchemaVersionError` rather than risk misreading old data.

Tests: `python3 -m unittest discover -s tests -v` runs the same test
bodies against both backends (`tests/test_storage_backends.py`), plus a
cross-backend parity check and SQLite-specific persistence/schema-version
tests. No dependency needed to run them (pytest also works if installed,
as an optional `dev` extra).

## Postgres and sharding (enterprise scale)

A SQLite file needs a shared filesystem and is fundamentally single-
machine. `PostgresBackend` implements the exact same `StorageBackend`
abstraction reachable from multiple app-server processes or machines at
once instead:

```python
kb = KnowledgeBase(store="postgresql://user:pw@host:5432/db?schema=finance1")
```

One Postgres **schema** per kb (not one database per kb), so many
tenants share one Postgres database cleanly -- `?schema=` is treehash's
own convention, stripped before connecting. FTS uses Postgres's own
`tsvector`/GIN index and `ts_rank()`, the same role SQLite's FTS5 plays.
Needs `pip install treehash[postgres]` (psycopg2); lazy-imported, so the
base package and every other backend are unaffected.

**Sharding**: `treehash/sharding.py`'s `ShardRouter` routes a kb name to
one of several configured backends (directories and/or Postgres DSNs,
freely mixed) via a stable hash -- the same per-kb namespace boundary
that already gave each kb its own SQLite file or Postgres schema, just
picked from N configured backends instead of always one. In server mode,
set `TREEHASH_SHARDS` (comma-separated); unset, behavior is unchanged
(the single `TREEHASH_DATA_DIR` from Server mode below).

```bash
TREEHASH_SHARDS="postgresql://host1/db,postgresql://host2/db,/data/shard2"
```

This is a routing layer, not a migration tool: changing the shard list
changes where a kb's namespace routes to next; it does not move existing
data there.

**What this closes, measured, not just designed**: `BENCHMARKS.md`'s
"Enterprise-shaped scale" section found and fixed two real O(n_documents)
bugs in the structural resolver (document scoping, and the section-number
index not accounting for per-document scope) that only showed up once a
corpus had thousands of SEPARATE documents rather than one document with
many sections -- confirmed flat `structural_exact` latency afterward from
100 to 20,000 documents (860,000 elements). It also found and fixed a
real Postgres-specific concurrency bug (`CREATE SCHEMA IF NOT EXISTS` is
not atomic against a race, fixed with an advisory lock) and two real
round-trip-reduction fixes (`descendants_bulk`, `get_content_bulk`) that
matter specifically because every `StorageBackend` call crosses a network
boundary on Postgres, unlike a local file. Postgres's own latency/scaling
numbers there are reported as directional (single shared container, not
dedicated hardware), not a tight bound -- read that section before
sizing a deployment around them.

## Domain packs

`domain="medical"`, `"finance"`, or `"education"` layer curated
vocabulary onto the resolver and fallback scorer, with zero change to the
core engine:

- **Section aliases** — a finance filing's "Item 7", a textbook's
  "Chapter 3", a clinical note's "Assessment" all resolve like "section N".
- **Table aliases** — "statement", "panel", "rubric", "schedule" resolve
  like "table".
- **Glossary synonyms** — a query for "sales" matches a table column
  headed "Revenue"; "dx" expands to "diagnosis"/"condition"; "objective"
  expands to "goal"/"outcome". Expansion is symmetric (applied to both the
  query and the candidate text) and purely lexical — curated vocabulary,
  not a trained model, so it contributes nothing rather than guessing
  when a domain pack doesn't recognize a term.

```python
from treehash import KnowledgeBase, list_domains
print(list_domains())   # ['education', 'finance', 'medical']

kb = KnowledgeBase(domain="medical")
kb.add_document("note1", clinical_note_text)
kb.query("what is the value for blood pressure")
```

Bring your own domain with a custom `DomainProfile`
(`treehash.domains.base.DomainProfile`) if these three don't fit —
it's four lists and a dict, not a training pipeline. Malformed shape (a
non-lowercase alias, an empty glossary synonym list) is rejected
immediately at construction, not discovered later inside `resolver.py`.
Registering a new pack in `treehash/domains/__init__.py` gets it
automatically covered by `tests/test_domain_pack_contract.py` — the same
structural sanity checks (every alias resolves like the word it extends,
glossary expansion is symmetric, loading the domain never regresses
plain queries) run against every domain, present or future, with zero
extra test-writing. See `CONTRIBUTING.md`'s "Adding a domain pack" for
the full checklist — this is what "open the domain-pack format up to
contribution" actually means here: a shared guardrail, not just a
request to copy `finance.py` carefully.

**Scope note:** these packs are vocabulary for *structural text
retrieval*. The medical pack is not a coding system and makes no clinical
claims; the finance pack is not financial advice; nothing here should be
used to make a clinical or investment decision.

## The honest benchmark (not a marketing number)

The methodology below is written out explicitly, as a checklist with
cited examples, in [`DISCLOSURE_STANDARD.md`](DISCLOSURE_STANDARD.md) —
reproducible numbers, disclosed bad runs, estimates labeled as
estimates, real-subprocess concurrency proof, stated scale limits. Not a
claim that anyone else has adopted it; a name for the practice this
project holds itself to, offered for anyone to check a claim against or
borrow from.

Run `python3 benchmark.py`. It measures accuracy, latency, and token
usage across four corpora (a generic doc pair plus one realistic-shaped
document per domain — several paragraphs per section, tables, lists) for
`treehash` against a conventional one-vector-per-paragraph baseline
(`treehash.baseline.NaiveVectorRAG`, same hand-rolled TF-IDF+cosine math,
so any gap comes from retrieval *granularity*, not one side using better
math). Latest measured run, 48 labeled test cases across all four corpora:

For pure scale behavior (does structural resolution stay O(1)-average as
the corpus grows, does ingest stay linear, does persistence hold up) —
run `python3 benchmark.py --scale 100000 --backend sqlite` and see
`BENCHMARKS.md`, which also documents five real performance bugs found
and fixed while producing those numbers -- including one that an earlier
revision of that same page argued (wrongly) was not a bug, and one
(quadratic ingest on a single very-large document) only exposed by
testing a full order of magnitude past what the table used to cover.

For what ingest itself costs — not querying, the one-time cost of
getting a corpus in — see `benchmarks/ingest_cost/`: treehash's
add_document() never calls an embedding model, verified by making the
network call itself raise if attempted, not just read off the source.
Compared against what a chunk-and-embed pipeline would spend at
published OpenAI embedding pricing (an estimate, clearly labeled as
one — no embedding API is actually called producing that comparison).

For real (not synthetic) documents and a real LLM — see
`benchmarks/financebench/`: treehash vs a conventional chunk baseline on
[FinanceBench](https://github.com/patronus-ai/financebench) (real SEC
filings, real gpt-4o-mini calls, real provider-reported tokens). The
honest result there is *not* a repeat of the win above — it's a smaller,
still-real one. The first run found treehash using MORE tokens than the
baseline, traced to a real gap in the PDF parser (one 121-page filing
detected as a single section); fixing that (and two other real bugs found
along the way) flipped it to treehash using fewer tokens, but nowhere
near the 98–99% figure — real filings stay messier than the structured
synthetic corpora above. Accuracy on FinanceBench's harder computed-
metric questions is a separate, still-open gap for both sides. Read
`benchmarks/financebench/README.md` for the full before/after — two
rounds, not one, kept on purpose.

| Metric | treehash | naive per-chunk baseline |
| --- | --- | --- |
| Accuracy (hit rate) | **48/48 (100%)** | 16/48 (33%) |
| Indexed units | **36** (section/doc-level) | 65 (per-paragraph) |
| Latency, structural queries | **< 0.1ms** (median) | 0.006–0.01ms |

**Token reduction is real, but it is not a flat number, and reporting it
as one would be exactly the kind of unverified claim this project rejects
by design.** Broken out by what the query actually asks for:

- **Exact/targeted lookups** (a table cell, a numbered/titled section, an
  addressed fact — "revenue in Q1 2025", "the value for LDL", "points for
  correctness"): **98–99% fewer tokens**, measured, consistently, across
  every domain corpus. This is the common case for a lot of real RAG
  traffic — "what's the Q2 number", "what's the patient's blood
  pressure" — and it's where structural resolution wins decisively: the
  answer *is* a hash lookup, so the context is a couple of tokens instead
  of a whole retrieved chunk.
- **"Summarize this whole section" queries**: roughly break-even, and
  sometimes slightly *negative* versus the baseline. This isn't a bug —
  if the question genuinely needs the whole section, no retrieval
  strategy makes that content smaller. The overall blended number across
  all 48 cases in the current run is an 11.0% token reduction, dragged
  down by these broad queries; look at the per-category breakdown in
  `benchmark.py`'s output, not the blended average, to see where the win
  actually is. (This blended figure moves release to release — it was
  5.7%, then 4.2% after an accuracy fix traded some of it away
  (`benchmarks/financebench/README.md`'s Round 3), then 11.0% after a
  real-world-testing pass on retrieval itself (Round 4: a thin stopword
  list and a redundant-ancestor bug in section ranking were both
  measurably hurting which content got selected, not just how much of
  it). Accuracy held at 48/48 through all of it. The 98–99% figure
  above, which is what actually matters for targeted queries, is
  unaffected throughout.)

If your workload is mostly targeted-fact queries over structured
documents (filings, clinical notes, structured coursework), expect
numbers close to the 98–99% figure. If it's mostly "explain this whole
section to me," expect closer to break-even on tokens — but you still get
the accuracy and latency wins from resolving by address instead of
similarity search.

**48/48 is real, but earned on well-phrased queries — a separate
adversarial set measures how that degrades under realistic messiness.**
Run `python3 benchmark.py --adversarial`: the same 4 corpora's same
underlying facts, under paraphrase, typos, multi-hop questions, and
vague phrasing, reported entirely separately (never blended into the
48/48 above — see `accuracy_cases/README_ADVERSARIAL.md` for why
blending would hide the exact thing this set exists to find). Current
result: **28/48 (58.3%)**, a real, materially-below-clean gap. The
resolver's honesty property held throughout — every miss degraded to
BM25 fallback or declined outright, never a wrong structural answer
asserted with false confidence.

48 cases is enough to see a large gap, not enough to trust any one
category's exact percentage — `accuracy_cases/expand_bank.py`
mechanically expands the same underlying facts into a >=1,000-case,
per-category-tagged bank (every generated case's answer traces back to
a hand-verified fact, never invented) that `--adversarial` now also
runs. At that scale, after three iterations of "find the highest-leverage
real gap, fix exactly that, verify no regression": **92.0% blended**
(up from an 84.4% baseline), with every generated variant type —
`case_punct` (100%), `paraphrase` (97.7%), `synonym` (89.6%), `typo`
(87.0%, up from 66.5% via a narrowly-scoped edit-distance-1 anchor-word
correction — never a general fuzzy-match pass), and `multi_hop` (81.0%,
up from 49.0% via two separate resolver fixes for two different
multi-reference mechanisms) — clearing a 0.75 per-category bar.
`vague` (58.3%, n=12, hand-verified only, not yet a generated category)
is the sole remaining gap; the old 48-case set's "100% (4/4) on
multihop" was flagged at the time as likely small-sample luck, and
n=100 confirmed it was. All three fixes verified with zero case-level
regressions, not just an unchanged aggregate. Full numbers,
methodology, and the iteration log in
`accuracy_cases/ITERATION_LOG.md`.

## Semantic fallback (optional third tier)

The one real objection to "structural resolve, then BM25" as a purist
stance: what happens when a query has no structural anchor AND shares
zero words with anything in the corpus? Before this, the honest answer
was "it falls back to BM25 forever," and BM25 has nothing to offer a
query with zero term overlap — the result is silence (`method="none"`).

```python
kb = KnowledgeBase(
    semantic_fallback_api_key="sk-...",       # bring your own key, same as kb.ask()
    semantic_fallback_provider="openai",      # default; text-embedding-3-small
)
kb.add_document("handbook", employee_handbook_text)
result = kb.query("holiday allowance")   # corpus says "vacation", "paid time off" -- zero literal overlap
print(result.method)   # "semantic_fallback", instead of "none"
```

Now the answer is "it degrades gracefully": structural → BM25/FTS5 →
(only when BOTH of those find nothing) embeddings, scored over the SAME
small, already-pruned section/document-level candidate set BM25 already
operates over — never the whole corpus, never per-paragraph, and nothing
is stored or persisted; the vectors are computed for one query and
thrown away. This tier is **off by default** (`semantic_fallback_api_key`
and `semantic_fallback_embed_fn` both default to `None`), so "zero
required dependencies" and "vectorless by default" stay true whether or
not you turn it on. Turning it on needs `pip install treehash[openai]`
(or pass your own `semantic_fallback_embed_fn=` for a local model or a
different vendor) and costs money on your key, same as `kb.ask()`.
Server mode: set `TREEHASH_SEMANTIC_FALLBACK_API_KEY`.

## Query log and live metrics

Every query that does NOT resolve structurally is recorded to
`kb.query_log` (optionally also to a JSONL file via
`query_log_path=`) — not for automatic pattern learning, the structural
resolver (`resolver.py`) stays exactly as curated and rule-based as
before. What it enables is `kb.query_log_report()`: real frequency
analysis surfacing which queries keep missing structural resolution
(repeat misses are the strongest signal a new resolver pattern is worth
writing) and which words are disproportionately common in the queries
that fall through (a concrete vocabulary gap, the same shape of fix as
an existing domain-pack alias). A person still decides whether either
signal is worth a resolver.py change — what's different is that the
decision is now backed by real traffic instead of guesswork.

`kb.metrics()` reports the live split of query volume across all three
tiers (structural / BM25-or-FTS5 / semantic / unresolved) plus an
estimated token-savings figure — computed from queries this KB has
actually served, not asserted from `benchmark.py`. In server mode:
`GET /kb/{name}/metrics` and `GET /kb/{name}/query-log-report`.

**The data moat, made concrete**: `treehash.querylog.aggregate_query_logs()`
merges query-log JSONL files from multiple deployments (different
customers, different time periods, or both) into one combined view —
the actual mechanism behind "every customer's real traffic makes the
pattern library better," not just a description of it. A query shape
that only appears once in any single deployment's log can appear
disproportionately often once several deployments' logs are combined.
`treehash.querylog.compare_reports()` diffs two `frequency_report()`
snapshots (e.g. before/after adding a batch of resolver patterns) and
names exactly which previously-repeated fallthrough queries no longer
appear — the artifact that actually demonstrates the pattern library
broadened, rather than asserting it did. Deliberately **not** exposed as
a server endpoint across tenants: aggregating query CONTENT across kbs
would need its own authorization separate from the per-KB keys in
[Server mode](#server-mode) below, to avoid leaking one tenant's queries
to another — left as an operator-run offline tool against exported logs,
not a live multi-tenant API.

## Bring your own API key

```python
answer = kb.ask(
    "What was revenue in Q2 2025?",
    api_key="sk-...",
    provider="anthropic",   # or "openai"
)
print(answer["usage"])   # {"input_tokens": ..., "output_tokens": ...} -- real, provider-reported
```

`kb.ask()` calls `context_for_llm()` internally, sends only that minimal
context plus your query, and returns the **actual billed token counts**
from the provider's API response — not an estimate — so the reduction
this library gives you is something you can verify on your own account,
for your own documents and queries, rather than take on faith. Requires
`pip install treehash[anthropic]` or `treehash[openai]`; both are lazy
imports, so the base package needs neither. A bring-your-own-client
escape hatch (`treehash.llm.ask_with_client`) covers any other provider
or a local model.

## Auditable answer trail (finance, legal, compliance)

The one thing a vector-DB-based RAG structurally cannot offer as
cleanly: a cosine-similarity hit names a nearest neighbor, not an
address. Every `kb.ask()` call here traces to a literal tree address, a
resolution method, and a confidence — inspectable, not just scored.

```python
kb.ask("What was revenue in Q2 2025?", api_key="sk-...")   # recorded automatically

ok, bad_index = kb.audit_log.verify()   # hash-chain integrity check
kb.audit_log.export_csv("audit-trail.csv")     # spreadsheet-friendly, for a human reviewer
kb.audit_log.export_jsonl("audit-trail.jsonl")  # full fidelity, for re-verification
```

Every `ask()` call is appended to `kb.audit_log` (`treehash/audit.py`):
the query, the resolution method/confidence, the literal source
addresses, the answer, and the real provider-billed token usage.
**Hash-chained**, the same content-addressing idea this whole project
already applies to tree nodes (`hashing.py`), applied here to log
entries instead — each record's hash covers its own fields plus the
previous record's hash, so `verify()` can detect an edited or reordered
past record, not just implicitly trust "it's in a log file." (This is
not encryption or an external notarization service — someone with write
access to the log file could still rewrite it entirely from scratch;
what it actually guarantees is that a normally-appended log cannot have
an earlier record edited without that edit being detectable.)

`context_for_llm()` exposes the same `sources` list even if you run your
own LLM call outside `kb.ask()`. Server mode: `GET
/kb/{name}/audit-log/verify` and `GET /kb/{name}/audit-log/export?format=jsonl|csv`.

**Cost savings, verified, not asserted**: `kb.cost_savings_report()`
turns the real, provider-billed token counts from every `ask()` call
into a dollar figure against your own rate:

```python
report = kb.cost_savings_report(input_cost_per_1k=0.15, output_cost_per_1k=0.60)
# {"ask_calls": ..., "real_input_tokens": ..., "actual_cost_usd": ...,
#  "naive_baseline_cost_usd_estimate": ..., "estimated_savings_usd": ..., ...}
```

Stated plainly because the asymmetry is real and unavoidable:
`actual_cost_usd` is computed from real, provider-billed tokens.
`naive_baseline_cost_usd_estimate` is an ESTIMATE of what the same calls
would have cost sending the whole corpus as context instead — that path
was never actually run against a real API, so it can't be provider-
verified; it's the same honestly-labeled "vs full corpus" comparison
`metrics()` already reports as a percentage, here as a dollar amount.
Server mode: `GET /kb/{name}/cost-savings?input_cost_per_1k=...`.

## Document formats

```python
kb.add_document("q3_10q", markdown_text)                       # default, no extra needed
kb.add_document("q3_10q", html_text, format="html")            # stdlib only, no extra needed
kb.add_document("q3_10q", open("filing.docx", "rb").read(), format="docx")  # pip install treehash[docx]
kb.add_document("q3_10q", open("filing.pdf", "rb").read(), format="pdf")    # pip install treehash[pdf]
```

All four formats parse into the exact same universal Element tree
(`treehash/elements.py`) — the resolver, tree/hash index, and fallback
ranker can't tell which parser produced a given node, and don't need to.
Verified directly: `tests/test_parsers.py` runs the same 7 labeled
queries (`accuracy_cases/documents.py`) against identical content
ingested via all four parsers and checks for identical resolution,
including a table-cell lookup ("revenue in Q3 2025" → "7.4M") that
resolves the same way — same method, same confidence formula — regardless
of source format.

- **HTML** (`treehash/parsers/html_parser.py`) uses only
  `html.parser.HTMLParser` from the standard library — no extra install.
  `<h1>`–`<h6>` → section/subsection, `<table>` → table/row/cell,
  `<ul>`/`<ol>` → list, `<a href>` → link.
- **DOCX** (`pip install treehash[docx]`, python-docx) reads Word's own
  heading styles ("Heading 1", "Heading 2", …) for sections and native
  tables directly — no font-size guessing needed, since DOCX already
  carries structural metadata. Detection is style-name based, so a
  document that fakes headings with bold runs instead of using Word's
  heading styles won't be recognized as having headings — a stated
  limitation, not a silent gap.
- **PDF** (`pip install treehash[pdf]`, pdfplumber) has no structural
  metadata to read at all — a PDF is positioned text. Three independent
  signals are combined, because real-world testing found that any one
  alone fails badly on real filings: **font size** relative to body text
  (the original approach — it alone found exactly 1 section in a real
  121-page 10-K, because that filing's headings are typeset at the *same*
  size as body text); **boldness** relative to the body's dominant style,
  which catches same-size bold headings size alone misses; and **pattern
  matching** for SEC filings' standardized `PART I` / `ITEM 7.` markers,
  independent of styling entirely. Repeating running headers/footers
  (e.g. a page header printed on every page) are detected and dropped.
  On that same real 10-K, this combination found 9 real sections and 497
  correctly-nested subsections instead of 1. Still a heuristic, still
  stated as one: single-column layouts only, no OCR (a scanned image PDF
  yields no elements), and a PDF whose headings are distinguished some
  *other* way (color alone, indentation alone) won't be recognized.
  Measured before/after on real filings in
  `benchmarks/financebench/README.md`, not asserted.

  PDF-sourced elements also carry two metadata fields the other parsers
  don't need: `page` (the PDF page number a paragraph/table/heading came
  from) and, on section/subsection nodes, `heading_source`
  (`"pattern"`/`"boldness"`/`"size"` — which signal actually fired), so a
  resolved hit's provenance and the confidence basis behind it are
  inspectable, not just asserted. `Element.metadata` is an open dict
  (`elements.py`), so this didn't need a schema migration — parsers are
  free to attach whatever provenance is meaningful for their format.

**Tables and images.** Both PDF and DOCX detect embedded images
(`page.images`/`w:drawing`) and map them to `image` elements; a nearby
line matching a "Figure N" / "Table N" / "Exhibit N" pattern (or, in
DOCX, Word's built-in "Caption" style) is attached as a child `caption`
element instead of being indexed twice. Tiny images (under ~20pt in
either dimension) are filtered out as bullet/icon glyphs, not real
figures — verified directly against real documents (Apple's 10-K: 4
real figures found; a 13.6MB WHO report: 74; a 31MB OECD-style DOCX
report: 1,061 real embedded charts/graphics, cross-checked against the
document's own XML rather than assumed). A PDF/DOCX table strategy using
text-position heuristics instead of the default line/rect detection was
tried and rejected: tested against several of this repo's real documents
(not just one), it mostly re-found tables the default strategy already
caught, and its few genuinely extra "tables" were shredded paragraph
prose or word-wrapped diagram labels almost everywhere it was tried — a
real, measured result, so only the default line-based table detector
ships. Stated limitations, not silently scoped away: no OCR (a scanned
page's images/tables yield no text), and charts drawn with PDF vector
operators (lines/curves) rather than embedded as a raster image aren't
captured — recovering their underlying data would need chart-specific
reconstruction, not general parsing.

## Capacity: large documents, multiple documents

Tested against real files, not synthetic stand-ins: ten actual PDFs
(SEC filings, a WHO report, a NIST report, academic papers, 0.1–13MB
each) ingested together into one `KnowledgeBase`, and a single ~110MB
PDF built by concatenating real documents together (pypdf) for a
deliberate over-100MB stress test.

| Test | Result |
|---|---|
| 10 real documents (0.1–13MB each, ~38MB total), one corpus | All 10 ingested successfully, 56s total, 349MB peak RSS, cross-document queries correct |
| 1 document, ~110MB (2,581 real pages) | Ingested successfully, 170s, 542MB peak RSS, 131,369 elements, still queries in low milliseconds |

**A real memory bug was found and fixed getting there, and it was
severe.** The first attempt at that 110MB file used **1.1GB peak RSS**
before a fix — and the *same* 13MB WHO report alone used **1.5GB**,
disproportionate to its size, before the underlying cause was found:
`pdfplumber` caches every parsed object (characters, rects, curves,
table-detection edges) **per page** and never releases it on its own.
Holding the whole PDF open across a full page loop — which
`extract_words()`/`find_tables()` alone give no reason to suspect — let
every page's cache accumulate simultaneously across the whole document.
Closing each page immediately after pulling its words/tables out (both
already plain dicts by that point, not live references into the page's
cache) dropped the 13MB file from 1.5GB to 69MB — a ~22x reduction,
confirmed by measuring both, not assumed. A second, smaller fix (not
storing per-word data that was only ever used to compute derived line
summaries) cut the 110MB-file number further, from 1.1GB to 542MB.
Covered by a fast, CI-friendly regression test
(`tests/test_parsers.py::test_page_cache_is_released_per_page`) that
reproduces the failure mode on a small-but-many-page synthetic fixture
rather than requiring a slow multi-hundred-MB test in every CI run.

What this does and doesn't establish: ingestion is I/O- and CPU-bound
(roughly 0.6–0.7MB/s for real, image-and-table-heavy PDFs; a plain-text
markdown/HTML document of the same size would be much faster, since PDF
parsing is what's expensive here, not `treehash`'s own indexing), and
memory now scales roughly proportionally with content rather than
catastrophically — but "roughly proportionally" is still real memory
(~5x the input file size, from this measurement), not free. A single
process ingesting several 100MB+ documents *concurrently* would add up;
size accordingly for your deployment. The server's
`TREEHASH_MAX_UPLOAD_MB` (default 150 — see Server mode) is sized around
this 100MB target specifically, accounting for base64's ~33% overhead
on the `content_base64` upload path.

## Server mode

For a non-Python stack (Go, Rust, Node, …), run treehash as an HTTP
sidecar instead of a library:

```bash
docker build -t treehash-server .
docker run -p 8000:8000 -v treehash-data:/data treehash-server
```

```bash
curl -X POST http://localhost:8000/kb/finance1/documents \
  -H "Content-Type: application/json" \
  -d '{"document_id": "q3_10q", "domain": "finance", "content": "# Item 7\n\nRevenue grew.\n\n| Metric | Q3 2025 |\n| --- | --- |\n| Revenue | 7.4M |\n"}'

curl -X POST http://localhost:8000/kb/finance1/query \
  -H "Content-Type: application/json" \
  -d '{"query": "revenue in Q3 2025"}'
# -> {"method": "structural", "hits": [{"content": "7.4M", ...}], ...}

curl http://localhost:8000/kb/finance1/stats
```

Or run it directly without Docker: `pip install treehash[server]` then
`uvicorn treehash.server:app --host 0.0.0.0 --port 8000`.

Each `{name}` in `/kb/{name}/...` is its own namespace — its own
`SQLiteBackend`-persisted file under `TREEHASH_DATA_DIR` (default
`./treehash_data`, `/data` inside the container), created on first use.
Endpoints: `POST /kb/{name}/documents` (`content` for text formats,
`content_base64` for docx/pdf), `POST /kb/{name}/query`,
`POST /kb/{name}/ask` (pass your own `api_key`, same as `kb.ask()`),
`GET /kb/{name}/stats`, `GET /kb/{name}/metrics` (live per-tier
resolution split + token savings), `GET /kb/{name}/query-log-report`
(fallthrough-query frequency analysis) — see
[Query log and live metrics](#query-log-and-live-metrics).

**Auth** is per-KB, so a multi-tenant deployment doesn't have to trust
one token for every tenant's data:
- `TREEHASH_API_KEYS`: a JSON object mapping kb name to key or `[keys]`,
  e.g. `{"finance1": "key-abc", "legal1": ["key-1", "key-2"]}`. A kb name
  listed here only accepts one of ITS keys — not any other tenant's key,
  and not the fallback token below.
- `TREEHASH_API_TOKEN`: a single shared bearer token, checked only for kb
  names with no entry in `TREEHASH_API_KEYS` — the original v1 auth
  story, now a fallback rather than the whole story.
- Neither set for a given kb name → that kb is open — fine for local
  development, not for anything network-exposed. Whichever applies, every
  request needs `Authorization: Bearer <token>`; comparisons use
  `hmac.compare_digest`, not `==`.

**Production hardening**, added after testing the realistic failure modes
directly rather than guessing at them:
- **Concurrent writers.** Real-world testing with actual separate
  *processes* (not just threads — the shape `uvicorn --workers N` or
  multiple server instances actually produces) writing to the same
  `{name}`'s SQLite file found it reproducibly lost writes: 2 of 5
  concurrent writers failed outright with "database is locked." Fixed via
  WAL mode, a busy timeout, and `BEGIN IMMEDIATE` instead of a deferred
  transaction — reran the same test dozens of times afterward with zero
  losses.
- **Crash recovery.** A process SIGKILLed partway through an uncommitted,
  batched write leaves the database openable AND fully rolls back —
  verified by a test that actually kills a real subprocess mid-transaction,
  not assumed from WAL mode's general reputation.
- **Request size limit** (`TREEHASH_MAX_UPLOAD_MB`, default 150) rejects
  an oversized body before it's read into memory.
- **Rate limiting** (`TREEHASH_RATE_LIMIT_PER_MINUTE`, default 60, `0`
  disables it): per client IP, backed by an in-memory sliding window by
  default — single-process only. Set `TREEHASH_REDIS_URL` to back it with
  Redis instead (fixed-window counter, `pip install treehash[redis]`), so
  the limit is enforced across every worker process/server instance
  sharing that Redis, not just one.
- **Structured request logging** (`treehash.server` logger, one line per
  request: method, path, kb name, status, latency) — composes with
  whatever log aggregation your deployment already has.

Optional extras: `pip install treehash[server]` (fastapi + uvicorn) and,
only if you set `TREEHASH_REDIS_URL`, `treehash[redis]`; both lazy
imported, so the base package is unaffected. Tests: `tests/test_server.py`
(in-process, via FastAPI's TestClient — covers the full request/response
cycle, per-KB and global-token auth, path-traversal-safe name validation,
persistence across a simulated restart, and the hardening above),
`tests/test_ratelimit.py` (the in-memory and Redis-backed limiters, the
latter proven to share state across instances without needing a real
Redis server), plus real-subprocess concurrency and crash-recovery tests
in `tests/test_storage_backends.py`. The `docker build`/`docker run` step
was verified in CI (all jobs, including Docker, pass — see the badge/
Actions tab), though not run locally in this development environment.

## MCP server (for agents)

Deterministic address resolution is a better fit for a tool-calling
agent than for a human typing a question — an agent that already knows
a document id and section number benefits from an address it can get
right every time, not a natural-language query that might or might not
match one of `resolver.py`'s patterns. `treehash/mcp_server.py` wraps a
KnowledgeBase as an [MCP](https://modelcontextprotocol.io) server so any
agent framework that speaks MCP (Claude, LangGraph, or anything else)
can plug it in as a tool directly — the same near-zero-integration-cost
bet the LangChain/LlamaIndex adapters below already made for those
frameworks.

```bash
pip install treehash[mcp]
python3 -m treehash.mcp_server   # stdio transport, the standard for a local MCP server
```

Point an MCP host at it with `examples/mcp_config.json`'s shape (Claude
Desktop's `claude_desktop_config.json`, Claude Code's `.mcp.json`, or
any other host that reads the same `mcpServers` shape).

**Structured tools, not just natural-language `query`** — the primary
surface exposes `KnowledgeBase`'s deterministic query methods directly:

```python
kb.get_section("10k_2025", 7)                                    # exact address, no NL parsing
kb.get_table_cell(header="Q3 2025", row_key="Revenue")            # exact match, not fuzzy overlap
kb.get_by_address("10k_2025/S07/T02R14C03")                       # from an address a prior call returned
kb.list_documents()                                               # discovery
kb.list_sections("10k_2025")                                      # discovery
```

Each bypasses `resolver.py`'s regex matching entirely, calling the
tree/hash index directly — deterministic by construction, not by a query
happening to match a pattern. `query` (natural language, resolver → BM25/
FTS5 → optional semantic tier) is still exposed as a fallback for
exploratory use with no known address.

`ask()` is **not** exposed as an MCP tool: the calling agent already IS
the LLM in an MCP setup, so it doesn't need treehash to call another LLM
on its own behalf — and accepting an API key as a tool argument would be
a real credential-handling smell for no benefit.

Reads the same `TREEHASH_DATA_DIR`/`TREEHASH_SHARDS`/
`TREEHASH_SEMANTIC_FALLBACK_*`/`TREEHASH_AUDIT_LOG_DIR` env vars Server
mode does, so an MCP client and the HTTP server can point at the same
data directory and see the same kbs. Trust model: this targets the
standard local stdio transport (the host process trusts a subprocess it
spawned itself) — there is no per-KB auth here the way `TREEHASH_API_KEYS`
provides for the HTTP server; running over an HTTP-based MCP transport
would need that same thinking applied, which this module doesn't attempt.

Tests: `tests/test_mcp_server.py`, in-process against the real `mcp`
package (no subprocess/stdio transport needed to test), skipped rather
than failed when `mcp` isn't installed.

## Framework integrations

Most developers will meet a retriever through whatever framework they
already use, not as a standalone library call — treehash plugs into both
major ones as a drop-in retriever:

```python
# LangChain (pip install treehash[langchain])
from treehash.integrations.langchain_retriever import TreehashRetriever
retriever = TreehashRetriever(kb, top_k=3)
retriever.invoke("What was revenue in Q3 2025?")   # -> list[Document]

# LlamaIndex (pip install treehash[llamaindex])
from treehash.integrations.llamaindex_retriever import TreehashRetriever
retriever = TreehashRetriever(kb, top_k=3)
retriever.retrieve("What was revenue in Q3 2025?")  # -> list[NodeWithScore]
```

Each returned document/node's text is exactly what `context_for_llm()`
would assemble for that hit — not a dump of an entire matched subtree —
so the token-reduction behavior carries through into whatever chain or
query engine consumes the retriever, not just direct `kb.query()` calls.
The LlamaIndex adapter additionally excludes its own metadata (address,
method, confidence) from the LLM-visible content so it stays available
for your own inspection without inflating the prompt.

Runnable, end-to-end examples: `examples/langchain_retrievalqa.py` builds
a real LCEL retrieval chain (retriever → prompt → LLM) and
`examples/llamaindex_query_engine.py` builds a `RetrieverQueryEngine`,
both using a canned/mock LLM so they run standalone with no API key.
(Note: the original target for the LangChain example was the
`RetrievalQA` chain class — as of LangChain 1.x, `langchain.chains` no
longer exists; that class was removed in favor of composing chains with
LCEL directly. The example uses the current idiomatic pattern instead.
The actual integration point, `langchain_core.retrievers.BaseRetriever`,
is unaffected by that change.)

**A vertical starter kit**: `examples/tenk_analysis_audit_trail.py` --
10-K/10-Q analysis end to end (structural retrieval → `kb.ask()` →
audit-log verify/export → cost-savings report), runnable standalone with
no API key (a clearly-labeled demo stand-in for the LLM call; set
`ANTHROPIC_API_KEY`/`OPENAI_API_KEY` to see a real answer instead,
nothing else in the script changes). This is the adoption-cost-near-zero
path for the exact buyer the audit trail and cost-savings features above
are built for.

## What's implemented

- **Universal element model** (`elements.py`) — sections, paragraphs,
  tables/rows/cells, lists, links as typed nodes, not flattened text.
- **Multi-format parsing** (`parser.py` for markdown, `parsers/` for
  html/docx/pdf) — see Document formats above. Nothing downstream cares
  which parser produced a tree.
- **Two-hash design** (`hashing.py`): `address_hash` (WHERE) separate
  from `content_hash` (WHAT) — enables content-addressed dedup across
  documents (`content_store.py`).
- **Tree + hash index** (`tree_index.py`): hash-mapped children at every
  node, O(1)-average lookup per level, secondary indexes for
  type/metadata queries (no full-tree scans).
- **Storage backends** (`storage.py`): `MemoryBackend` (default, in-process
  dicts), `SQLiteBackend` (persists to a file, FTS5-backed fallback), and
  `PostgresBackend` (schema-per-kb, tsvector-backed fallback, reachable
  from multiple machines) — see Persistence and Postgres and sharding
  below.
- **Shard routing** (`sharding.py`): a kb name to one of several
  configured backends via a stable hash — see Postgres and sharding
  below.
- **Rule-based structural resolver** (`resolver.py`): section-by-number,
  by-ordinal, by-title-keyword; table-cell lookup by header+row-key;
  bare table/figure references. Returns `None` — not a guess — when it
  doesn't recognize a pattern.
- **Pure-Python retrieval math** (`mathlib.py`): BM25 (vectorless
  fallback, the default) and TF-IDF+cosine (opt-in `fallback="tfidf"`,
  and what the baseline uses) — both hand-rolled, zero dependencies.
- **Domain packs** (`domains/`): medical, finance, education, plus a
  `DomainProfile` base for your own.
- **CRUD**: `add_document`, `query`, `get`, `update_content`,
  `delete_subtree` (with content-store garbage collection).
- **`context_for_llm()` / `ask()`**: minimal-context assembly, ranked by
  keyword overlap when a matched subtree has more pieces than needed, and
  an optional direct bridge to Anthropic/OpenAI with real usage reporting.
- **`baseline.py`**: a naive one-vector-per-paragraph RAG implementation
  for honest, apples-to-apples comparison, not a strawman.
- **Framework adapters** (`integrations/`): LangChain and LlamaIndex
  retrievers — see Framework integrations above.
- **HTTP server** (`server.py`): FastAPI app over `SQLiteBackend`, plus a
  `Dockerfile` — see Server mode above.
- **Optional semantic fallback tier** (`embeddings.py`): reached only
  when structural resolution AND the lexical fallback both find nothing;
  off by default — see Semantic fallback above.
- **Query log and live metrics** (`querylog.py`,
  `KnowledgeBase.metrics()`): every non-structural query recorded for
  frequency analysis, plus a live per-tier resolution/token-savings
  report — see Query log and live metrics above.
- **Hash-chained audit trail and cost-savings report** (`audit.py`,
  `KnowledgeBase.cost_savings_report()`): every `ask()` call recorded
  with its literal source addresses, tamper-evident, exportable as
  evidence — see Auditable answer trail above.
- **Structured, deterministic query API** (`get_section`,
  `get_table_cell`, `get_by_address`, `list_documents`, `list_sections`
  on `KnowledgeBase`): bypasses `resolver.py`'s natural-language
  matching entirely for a caller that already knows what it wants.
- **MCP server** (`mcp_server.py`): the structured query API exposed as
  MCP tools, so any agent framework can plug a kb in directly — see MCP
  server above.

## What this does NOT prove yet

Carried forward from the original prototype, because it's still true:

1. **The resolver is curated patterns, not a trained model.** It handles
   the query shapes it was written for — including three domains' worth
   of alias/synonym vocabulary now — and falls back honestly on anything
   else. Genuinely novel phrasing still lands on BM25 (or, if configured,
   the optional semantic tier), not a learned-reasoning layer. What's
   changed: `kb.query_log_report()` now surfaces exactly which queries
   keep missing structural resolution and which words are
   disproportionately common in them, from real traffic — so growing
   resolver coverage release to release is backed by measured gaps
   instead of guesswork. A human still has to write the new pattern; this
   does not make the resolver a trained model, it makes the *decision of
   what to add* data-driven instead of speculative.
2. **Domain packs are vocabulary, not validated domain knowledge.** They
   improve *retrieval* over text that already exists in your documents;
   they don't verify medical or financial facts.
3. **Table-cell lookup needs signal on both axes, and that has two
   separate consequences worth knowing.** It requires some word overlap
   on both the column header and the row key (any single shared word is
   enough — that's what lets a domain glossary match "sales" to a
   "Revenue" header) to identify one cell unambiguously. First: tables
   shaped like (attribute → value) pairs rather than (metric × period)
   grids don't give it that two-axis signal at all, so they don't resolve
   structurally — see the notes in `accuracy_cases/medical.py` and
   `accuracy_cases/education.py` for a concrete, measured example where
   this falls back to BM25 and still gets the right answer, just not via
   a hash lookup. Second, at scale: the correct minimal candidate set for
   "any word overlaps" grows with how many rows share a query's terms, so
   a corpus that reuses a small label vocabulary across many rows costs
   more than one with mostly distinct labels — the same property any
   overlap-based/inverted-index search has for common terms, not specific
   to this library. Measured version of both in `BENCHMARKS.md`.
4. **Accuracy/latency/token numbers in the benchmark table above are at
   toy-to-moderate scale** (single documents, tens to fifties of elements
   per corpus). Separately, `BENCHMARKS.md` tests pure structural-
   resolution and ingest behavior up to 100,000 elements in one document,
   and up to 860,000 elements across 20,000 SEPARATE documents (the
   enterprise-shaped test this claim actually needs — see Postgres and
   sharding above) — address lookups stay flat in both shapes, and
   ingest stays linear — but even 860K is still short of the
   10⁴–10⁸-element range a durable enterprise-scale claim would
   ultimately need, and PostgresBackend's own latency numbers there are
   reported as directional (shared dev hardware), not a tight bound.

## Project layout

```
treehash/                  the package: import treehash to get all of this
  domains/                 medical/finance/education vocabulary packs
  parsers/                 html/docx/pdf -> universal Element tree
  integrations/            LangChain / LlamaIndex retriever adapters
  storage.py, server.py    persistence backends (Memory/SQLite/Postgres), the FastAPI server
  sharding.py, ratelimit.py  kb-name shard routing, in-memory/Redis rate limiting
  embeddings.py, querylog.py  optional semantic fallback tier, fallthrough-query logging
  audit.py                 hash-chained audit trail (KnowledgeBase.ask() call log)
  mcp_server.py             MCP tool wrapper -- structured query API for agents
samples/                   example documents used by demo.py/benchmark.py/tests
  binary/                  generated .docx/.pdf fixtures (Phase 3 cross-format parity)
accuracy_cases/            hand-verified (query, expected_substring) test cases, one module per corpus
tests/                     unittest suite -- zero dependency needed to run it
examples/                  runnable end-to-end scripts for the framework adapters
benchmark.py, demo.py      the two documented entry points (kept at the repo root on purpose)
frontend/                  the project site (static HTML/CSS, no build step) -- see below
vercel.json                zero-config Vercel deploy of frontend/ (outputDirectory, no build step)
BENCHMARKS.md              scale-testing methodology and results, reproducible commands
CHANGELOG.md, CONTRIBUTING.md   what changed and how to contribute
```

**`frontend/`** is the project site linked at the top of this README --
`index.html` + `styles.css`, no framework, no build step. Open
`frontend/index.html` directly in a browser, or serve the directory with
anything static (`python3 -m http.server`, GitHub Pages, Netlify, ...).
Not wired to any CI/Pages config in this repo; point your host's
static-site source at this directory to publish it.

**Vercel**: `vercel.json` at the repo root points `outputDirectory` at
`frontend/` with no build/install step (there's nothing to build), so
importing this repo into Vercel as-is deploys the site correctly with
zero dashboard configuration. `frontend/vercel.json` covers the
alternative setup too -- if you instead set the Vercel project's Root
Directory to `frontend/`, that's the config Vercel reads instead. Either
way: `cleanUrls` on, no trailing slash, no framework detection to fight.

## Run it

```bash
python3 demo.py                                     # ingestion, queries, CRUD, dedup, a domain-pack example
python3 benchmark.py                                 # accuracy / latency / tokens, proposed vs baseline, 4 corpora
python3 benchmark.py --scale 100000 --backend sqlite     # single-document scale test -- see BENCHMARKS.md
python3 benchmark.py --scale-docs 20000 --backend sqlite  # many-documents (enterprise-shaped) scale test
python3 -m unittest discover -s tests -v              # full test suite, zero dependency required
TREEHASH_TEST_POSTGRES_DSN=postgresql://... python3 -m unittest tests.test_postgres_backend -v
                                                       # needs: pip install treehash[postgres] + a real Postgres
python3 examples/langchain_retrievalqa.py             # needs: pip install treehash[langchain]
python3 examples/llamaindex_query_engine.py           # needs: pip install treehash[llamaindex]
python3 examples/tenk_analysis_audit_trail.py         # audit trail + cost savings, no API key needed
python3 -m treehash.mcp_server                        # needs: pip install treehash[mcp]
uvicorn treehash.server:app --reload                  # needs: pip install treehash[server]
```

See `CONTRIBUTING.md` for the project's non-negotiables (no fabricated
benchmark numbers, no required vector/embedding dependency, ever) before
sending a PR, and `CHANGELOG.md` for what's changed release to release.
