Metadata-Version: 2.5
Name: promptev-context-engine
Version: 0.0.6
Summary: Promptev Context Engine — language-agnostic ingestion + hybrid retrieval
Project-URL: Homepage, https://promptev.ai/documentation/context-engine/
Project-URL: Documentation, https://promptev.ai/documentation/context-engine/
Project-URL: Repository, https://github.com/promptev/context-engine
Project-URL: Issues, https://github.com/promptev/context-engine/issues
Author: Promptev Inc
License: Apache-2.0
License-File: LICENSE.md
License-File: NOTICE
Keywords: embeddings,graph-rag,hybrid-search,llm,mcp,pii-redaction,rag,retrieval,vector-search
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: alembic>=1.13
Requires-Dist: anthropic<1,>=0.77
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: cryptography>=42
Requires-Dist: httpx>=0.27
Requires-Dist: json-repair>=0.60
Requires-Dist: jsonschema>=4.20
Requires-Dist: lingua-language-detector>=2.1
Requires-Dist: openai>=1.40
Requires-Dist: openpyxl>=3.1
Requires-Dist: pgvector>=0.4
Requires-Dist: pydantic-settings>=2.4
Requires-Dist: pydantic>=2.7
Requires-Dist: pypdf>=5.0
Requires-Dist: pypdfium2>=4.30
Requires-Dist: python-docx>=1.1
Requires-Dist: python-pptx>=1.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: tiktoken>=0.7
Provides-Extra: compute
Requires-Dist: pandas>=2.0; extra == 'compute'
Provides-Extra: django
Requires-Dist: django>=4.2; extra == 'django'
Provides-Extra: fastapi
Requires-Dist: fastapi>=0.115; extra == 'fastapi'
Requires-Dist: python-multipart>=0.0.32; extra == 'fastapi'
Provides-Extra: flask
Requires-Dist: flask[async]>=3.0; extra == 'flask'
Provides-Extra: gemini
Requires-Dist: google-genai>=1.24; extra == 'gemini'
Provides-Extra: graph
Requires-Dist: neo4j>=5.20; extra == 'graph'
Requires-Dist: networkx>=3.2; extra == 'graph'
Provides-Extra: mcp
Requires-Dist: aiohttp>=3.9; extra == 'mcp'
Requires-Dist: mcp>=1.2; extra == 'mcp'
Provides-Extra: ocr
Requires-Dist: pillow>=10; extra == 'ocr'
Requires-Dist: pytesseract>=0.3; extra == 'ocr'
Provides-Extra: pdf
Requires-Dist: pdf-inspector>=0.2.6; extra == 'pdf'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Provides-Extra: presidio
Requires-Dist: presidio-analyzer>=2.2; extra == 'presidio'
Provides-Extra: reranker
Provides-Extra: vision
Requires-Dist: pillow>=10; extra == 'vision'
Description-Content-Type: text/markdown

<!-- Absolute URL, not a relative path: this README is also the PyPI
     long-description and PyPI does not resolve relative image paths.
     A copy of the mark lives at assets/promptev-logo.png. -->
<p align="center">
  <img src="https://promptev.ai/favicon.png" alt="Promptev" width="88" height="88">
</p>

<h1 align="center">context-engine</h1>

<p align="center">
  <b>Retrieval that enforces access control and PII redaction inside the query,<br>
  as a library, on your own Postgres.</b>
</p>

<p align="center">
  <a href="https://pypi.org/project/promptev-context-engine/"><img src="https://img.shields.io/pypi/v/promptev-context-engine?label=PyPI&color=3775A9" alt="PyPI"></a>
  <a href="https://www.npmjs.com/package/@promptev/context-engine"><img src="https://img.shields.io/npm/v/@promptev/context-engine?label=npm&color=CB3837" alt="npm"></a>
  <a href="https://github.com/promptev/context-engine/actions/workflows/publish.yml"><img src="https://github.com/promptev/context-engine/actions/workflows/publish.yml/badge.svg" alt="CI"></a>
  <img src="https://img.shields.io/badge/python-3.12%2B-3776AB" alt="Python 3.12+">
  <a href="LICENSE.md"><img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="Apache-2.0"></a>
</p>

> **Built by [Promptev](https://promptev.ai?utm_source=github&utm_medium=readme) — the governed AI agent platform.**
> This library is the retrieval engine behind Promptev's Context Packs,
> open-sourced (Apache-2.0). Use it standalone, or
> [see what the platform adds](#when-you-outgrow-the-library).

Most retrieval stacks make permissions your problem: you filter results after
the fact, in application code, and hope the ordering is right. This one pushes
the ACL into the SQL predicate every search leg shares, so a document a caller
may not see is never fetched, never ranked, and never reaches a reranker. PII
redaction sits at the same depth — masked *before* hits are POSTed to a hosted
reranker and *before* the tool-call audit row is written.

It is a **library, not a server**. No UI, no orchestration, no workflow engine,
no opinion about your framework — `pip install`, point it at a Postgres you
already run, call `engine.search()`. **BYO everything: your database, your
embedding provider, your LLM.** It never calls a hosted service you didn't
configure, and never implements auth, billing, or credential storage — it
exposes hooks (`on_usage`, `on_error`, `on_progress`) and injection points (`auth`,
`principals`) for your app to wire up.

On top of that it does the ordinary work well: structure-aware ingestion for
PDF/DOCX/PPTX/XLSX/HTML/images, hybrid retrieval (full-text + trigram + vector,
RRF-fused) with an optional graph leg, and a governed tool-execution path with
approval gates and audit.

> Not overclaiming: it does not prevent hallucination or guarantee grounding,
> and its `compute()` action (LLM-generated Python) is **off by default** —
> only enable it behind real OS-level isolation.

📖 **Full documentation:** https://promptev.ai/documentation/context-engine/

In this repo: **[access control](docs/access-control.md)** (the trichotomy,
enforcement, and the vector-index interaction) ·
**[the ACL-recall benchmark](benchmarks/acl_recall/)** ·
**[cookbook](cookbook/)** ·
**[TypeScript client](js/)** (`@promptev/context-engine`)

---

## Why you'd choose this

**1. Access control that survives the vector index.** Filtering an approximate
index with a `WHERE` clause is a *post-filter* — the index picks candidates
first, your ACL throws them away second, and the query returns short. We
measured it on 200k public documents at production embedding width, and found
something the literature misses: **because real permissions are topically
clustered rather than random, the loss is about twice what a conventional
benchmark reports** — recall 0.473 against 0.980 at identical visibility, a
quarter of queries empty, no error raised. Tuning does not fully close it, so
this library decides between an exact and an approximate scan per query
instead. [How access control works](docs/access-control.md) ·
[benchmark and method](benchmarks/acl_recall/).

Suspect your current stack? Prove it on your own data — one read-only
command, no embedding provider called:

```bash
context-engine check-acl-exposure --database-url postgresql://...
```

**2. Redaction that runs in the right place.** Not more PII detectors than
everyone else — better *placement*. Hits are masked **before** the reranker
POSTs them to a hosted provider and **before** the audit row is written, so
the masked value is the only one that ever leaves the process. Wiring a
redaction library into a pipeline by hand is exactly where that ordering goes
wrong. (Bring [Presidio](https://github.com/microsoft/presidio) for breadth —
its entities become ordinary rules.)

**3. It's a library, so the data never moves.** No server to run, no corpus to
hand over, no orchestration framework to adopt. If you're subject to rules
where *you* stay accountable for what your AI did — the EU AI Act's Article 12
logging duties fall on the deployer, not the vendor — then "runs in my
process, on my Postgres" is the answer you need, and a hosted service can't
give it to you.

**4. Governed tool execution in the same place as retrieval.** ACL-scoped
tools, encrypted credentials, a fail-closed approval gate with single-use
atomic resolution, and an audit row per call. Most stacks buy this separately
from a gateway.

<details>
<summary><b>How this compares to the alternatives</b></summary>

|  | Shape | Permissions |
|---|---|---|
| **context-engine** | Library, your Postgres | Enforced in the SQL predicate of every retrieval leg |
| pgai | Library, your Postgres | None — bring your own |
| R2R, RAGFlow | Server you deploy | Own user/collection model |
| Onyx, Glean | End-user search app | Synced from source-system ACLs, at query time |
| Azure AI Search | Managed cloud service | Query-time, tied to Entra |
| LlamaIndex, LangChain | Framework you build with | Your application's problem |

The closest thing to this package's *shape* is Timescale's `pgai` — same "a
library over the Postgres you already have" posture, without the governance.
The closest on *permissions* are Onyx and Glean, which are applications rather
than something you build on.

</details>

---

## When you outgrow the library

The library is the engine. [Promptev](https://promptev.ai?utm_source=github&utm_medium=readme)
is the platform it was extracted from. It is the same engine either way; what
differs is how much of the surrounding plumbing you run yourself.

**Your database, your model keys and your connectors stay your choice either
way** — run them yourself, or have Promptev run them. That is a deployment
decision, not a difference between the two:

| | Self-managed | Promptev-managed |
|---|---|---|
| Database | Your Postgres — your backups, tuning and upgrades | Hosted for you |
| Embeddings & LLM | Your keys, in your environment | Your keys still (BYOK), stored encrypted at rest and shared across agents |
| Connectors | You write the sync | Managed by Promptev, re-indexed on change |

What the platform adds on top of the engine:

| | The library | Promptev |
|---|---|---|
| Building agents over the data | A Python API | No-code builder, playground, deploys to web/Slack/WhatsApp |
| Approvals a human can operate | The state machine — the UI is yours | Approval gates, traces and usage dashboards included |
| RBAC and compliance posture | Hooks (`on_usage`, `on_error`, `on_progress`) | Included; SOC 2 Type II |

Self-hosting the engine is a first-class path and stays one — it is
Apache-2.0 and never calls home. The platform is for when the plumbing around
the engine becomes the job.

---

## Install

```bash
pip install "promptev-context-engine[postgres]"
# or with uv:
uv add "promptev-context-engine[postgres]"

# before it's on PyPI, install from git:
pip install "promptev-context-engine @ git+https://github.com/promptev/context-engine"
uv add "promptev-context-engine @ git+https://github.com/promptev/context-engine"
```

TypeScript lives in [`js/`](js/) as `@promptev/context-engine` — same Postgres
schema, so a corpus ingested by either client is searchable by the other:

```bash
npm install @promptev/context-engine pg
```

Needs **Node 22+**.

Needs **Python 3.12+** and **Postgres** with the `vector`, `pg_trgm`, and
`unaccent` extensions (auto-created by `context-engine migrate` if the role has
`CREATE EXTENSION`).

**The core engine is web-framework-agnostic** — it depends on no web framework.
Pick the router adapter that matches YOUR project (FastAPI, Flask, or Django)
via an extra, or use none and call `engine.*` directly.

Optional extras — install only what you use:

| Extra | Needed for |
|---|---|
| `postgres` | **the database driver — you almost certainly want this.** Separate from core because psycopg2 is LGPL and everything else is MIT/BSD/Apache, so a bare install pulls no copyleft. Any SQLAlchemy Postgres driver can be used instead, but this is the one that is tested (pg8000, measured: runs migrations and the vector and full-text legs, then fails the trigram leg — its paramstyle cannot express pg_trgm's `%` operator). |
| `fastapi` | `create_router()` — mount in a FastAPI app |
| `flask` | `create_flask_blueprint()` — register in a Flask app |
| `django` | `create_django_urlpatterns()` — include in Django `urls.py` |
| `graph` | `mode="graph"` (entity + community graph; Neo4j optional) |
| `vision` | scanned pages & images — transcribed by **whichever LLM you configure** |
| `gemini` | only if that LLM (or your embedder) is Gemini — covers both the `gemini` and `vertex_ai` providers |
| `ocr` | Tesseract OCR fallback |
| `mcp` | `create_mcp_app()` / `context-engine mcp` |
| `compute` | `engine.compute()` dataframe support |
| `presidio` | Microsoft Presidio entities as redaction detectors |
| `pdf` | structure-preserving PDF text extraction (tables stay tables) |

```bash
pip install "promptev-context-engine[fastapi,graph]"   # or [flask] / [django]
uv add "promptev-context-engine[fastapi,graph]"
```

### Teach your coding agent the access-control rules

Most integration code is now written with an agent's help, and the highest-cost
mistake in this library is one an agent makes readily: `principals=None` means
*trusted caller, no filtering*, so treating it as "nobody is logged in" returns
the entire corpus.

That spelling is being retired. `None` is falsy, so it is what every accessor
degrades to — `user.groups or None`, a missing key, `if user else None` — which
made the unauthenticated path the maximum-privilege path. Say it explicitly
instead:

```python
from context_engine import TRUSTED

await engine.search(q, principals=TRUSTED)      # skip ACL, deliberately
await engine.search(q, principals=[])           # anonymous
await engine.search(q, principals=user.groups)  # scoped
```

`None` still works and warns; it raises in 1.0. The package also ships a skill
stating this and the other invariants worth not guessing.

```bash
context-engine install-skill            # -> ./.claude/skills/context-engine/
context-engine install-skill --global   # -> ~/.claude/skills/
context-engine install-skill --print    # stdout, for agents that read other formats
```

It is an explicit command, never an install hook — the file becomes instructions
inside your agent, so it should be your decision. An edited skill is never
overwritten without `--force`.

## Configure

`ContextEngineConfig` is the one settings object (or build it from `CE_`-prefixed
env vars, nested with `__`, e.g. `CE_EMBEDDING__PROVIDER`).

```python
from context_engine import ContextEngineConfig, EmbeddingConfig, LLMConfig

config = ContextEngineConfig(
    database_url="postgresql://user:pass@localhost:5432/mydb",
    embedding=EmbeddingConfig(provider="openai", model="text-embedding-3-small", api_key="sk-..."),
    # llm is optional — only for structured extraction, compute, and the graph:
    llm=LLMConfig(provider="openai", model="gpt-4o-mini", api_key="sk-..."),
)
```

Embeddings: `openai` · `azure_openai` · `gemini` · `vertex_ai` · `voyage` ·
`cohere` · `custom` (any OpenAI-compatible `/embeddings` endpoint via
`base_url`). LLMs add `anthropic` · `bedrock`. (Anthropic has no embeddings
API — pick another provider for `EmbeddingConfig`.)

`anthropic` now goes through the official `anthropic` SDK rather than a
hand-rolled HTTP call, which is what lets it retry a 429 or a 529 like every
other SDK-backed provider. It is a core dependency, pinned `>=0.77,<1`: the
1.x line is built on `httpx2` and cannot accept the guarded `httpx` client the
engine injects, and keeping a host-supplied or engine-guarded client on every
outbound call matters more than the major version. The request body is
unchanged and the SDK's telemetry headers are suppressed.

Three host-visible consequences. `base_url` on an `anthropic` `LLMConfig` is
now **honoured** — it used to be ignored and the endpoint hardcoded, so a
deployment that set it was talking to the public API without being told. An
empty or unset `api_key` now **fails at the call** — the SDK raises
`TypeError("Could not resolve authentication method. Expected one of api_key,
auth_token, or credentials to be set…")` — instead of sending an empty
`x-api-key` and collecting a 401. Separately, the errors a live call raises are
the SDK's typed ones: an unrelated 400 is
`anthropic.BadRequestError` rather than `httpx.HTTPStatusError`, and a
connection failure or timeout is `APIConnectionError` with the httpx error as
`__cause__`. Neither `ANTHROPIC_API_KEY` nor `ANTHROPIC_BASE_URL` is consulted
— both are always passed explicitly, so the config decides the credential and
the endpoint; `ANTHROPIC_CUSTOM_HEADERS` still injects the headers it names.

`vertex_ai` is the same Gemini models served from Vertex AI, and the only
difference is how you authenticate: Application Default Credentials against a
GCP project instead of an API key.

```python
llm=LLMConfig(provider="vertex_ai", model="gemini-2.5-pro",
              project="my-project", location="us-central1"),
```

`project` and `location` are optional — leave them out and the Google SDK
resolves them from `GOOGLE_CLOUD_PROJECT` / `GOOGLE_CLOUD_LOCATION`, so an
existing GCP deployment needs no new configuration. The Python SDK will also
discover the project from ADC when neither is set; **the TypeScript client's
SDK will not** — measured, not assumed — so for `@promptev/context-engine`
name the project in config or in the environment.

Credentials themselves come from ADC in every form:
`GOOGLE_APPLICATION_CREDENTIALS`, workload identity, or
`gcloud auth application-default login`. There is no credentials field to set.
`api_key` is ignored when a project is present (the SDK rejects the pair
outright); on its own it selects Vertex express mode.

**Retries.** `max_retries` on `LLMConfig` and `EmbeddingConfig` is the number of
retries *on top of* the first attempt — default `2`, `0` disables retrying
entirely, and it reads from the environment like everything else
(`CE_LLM__MAX_RETRIES`, `CE_EMBEDDING__MAX_RETRIES`,
`CE_VISION_LLM__MAX_RETRIES`, `CE_GRAPH__EXTRACTION_LLM__MAX_RETRIES`). The
retrying itself is the provider SDK's, because each one already classifies its
own retryable errors and already honours `Retry-After`: `openai` /
`azure_openai` / `custom`, `gemini` / `vertex_ai`, `bedrock` and `anthropic`
all honour the field. The reranker and the `voyage` / `cohere` embedding paths
do not — they are raw HTTP with no SDK to delegate to — and batch submission
never retries, because `batches.create` is not idempotent and a retry after a
5xx that actually landed is a second paid job.

**The call timeout is per ATTEMPT**, so a call's ceiling is
`(max_retries + 1) x timeout + backoff` — about **12 minutes** at the defaults
(240s x 3) against an endpoint that accepts connections and never answers. It
layers: `extract_structured_data(max_reasks=1)` re-asks the model after an
unusable reply, so one document is about 24 minutes, and vision's three batch
attempts multiply again. Set `max_retries=0` to get back the single-attempt
behaviour, which is what a latency-sensitive interactive path wants.
`extract_structured_data`'s re-ask parameter used to be called `max_retries`
and is now `max_reasks`, so the two names cannot be confused; passing the old
one by keyword is an error.

Every outbound call the engine makes — LLM, embeddings, reranker, tool HTTP,
MCP OAuth — can run on a client the host supplies instead of one the engine
builds: `ContextEngine(config, http_client_factory=lambda purpose: ...,
mcp_resolver=...)`. `http_client_factory` is asked for an `httpx.AsyncClient`
per purpose (`llm` · `embeddings` · `reranker` · `tool_http` · `mcp_oauth`,
`context_engine.HttpClientFactory` / `HttpClientPurpose`); returning `None`
keeps the engine-built client. It must be a **long-lived, host-owned**
client — the engine never closes it, and it is called far more than once
(per LLM call, per extraction attempt, per tool call). `mcp_resolver` is the
resolver MCP transports resolve through. `tool_http`/`mcp_oauth` host clients
still get the per-hop egress check and redirect refusal on every call;
`google-genai` (`gemini`/`vertex_ai`) and `boto3` (`bedrock`) manage their own
transports and are out of scope. TypeScript: `new ContextEngine(config, {
fetchFactory, mcpLookup })`, handing back a `fetch` for the same five
purposes.

`StorageConfig` holds the deployment knobs: `ann_exact_threshold` (the vector
leg's exact/approximate crossover — hardware- and dimension-specific, see
[docs/access-control.md](docs/access-control.md)) and the connection pool,
passed through to SQLAlchemy — `pool_size` (5), `max_overflow` (10),
`pool_timeout` (30s), `pool_recycle` (1800s, keep it under the idle timeout of
any pooler in front of the database) and `pool_pre_ping` (on: without it every
connection left stale by a database restart or failover fails one query
first). All reachable as env vars, e.g. `CE_STORAGE__POOL_SIZE`.

## Provision the database

```bash
context-engine migrate --database-url postgresql://user:pass@localhost:5432/mydb --dim 1536
# add --graph to also create entity/relationship/community tables
```

`--dim` must match your embedding model's width (1536 for
`text-embedding-3-small`) and is locked in on first migrate.

**`migrate` does not write-lock your tables.** Every index it adds after the
initial schema is built with `CREATE INDEX CONCURRENTLY`, so ingest keeps
writing while the build runs. The build is slower than a plain one and it
waits for transactions that were already open when it started, which is the
trade for not stopping writes. `migrate` prints one line to stderr per index
it builds, naming it, so a long wait is visibly a wait and not a hang;
`SELECT * FROM pg_stat_progress_create_index` shows how far along it is. An
index that is already present and valid is neither rebuilt nor announced.

**A failed `migrate` is safe to re-run, and re-running is the repair.** It is
not one transaction: a failure part-way leaves the revisions before it
applied, every statement it runs is idempotent, and running it again picks up
where it stopped. An interrupted build (a cancelled statement, a killed
process) leaves Postgres holding an unusable index; the failing run drops that
partial copy when it can, and the next `migrate` drops and rebuilds whatever
remains. The error says which of the two happened. If `migrate` stops and
names an index, the cause is in the error reported with it: for
`ux_context_engine_tool_approvals_pending_scope` that means duplicate pending
approval rows to clear first.

A `lock_timeout` or `statement_timeout` on the connection you run `migrate`
with applies to these builds, and to the cleanup that follows a failed one.
`migrate` will not override it, so on a database with long-lived write
transactions either raise it for the migration or run when those transactions
are short.

**Graph mode runs on Postgres alone.** `graph.enabled` needs only
`extraction_llm`; `neo4j_uri` is optional, and leaving it unset is a supported
deployment shape rather than a degraded one. Entity and relationship
extraction, the whole Postgres mirror, Louvain community detection and its
summaries, and all four navigation actions (`get_neighbors`, `traverse`,
`find_related`, `community_summary`) are Postgres-side and unchanged. What a
Neo4j-less deployment gives up is **multi-hop expansion**: the graph leg's
candidate set stops at the ACL-scoped vector seeds instead of widening to
chunks two hops away through shared entities, so "what else is connected to
this" narrows toward what the vector leg would have found anyway. Nothing is
re-weighted — the connectivity signal falls back to one constant for every
candidate, which cannot reorder a sort, and the legs are fused on rank.
Neo4j stays all-or-nothing inside graph mode: a URI without a password is an
error, and **a password without a URI is now an error too**, because it is
otherwise indistinguishable from a dropped URI silently downgrading a
Neo4j deployment. A Postgres-only deployment still needs the `[graph]` extra
for `networkx` (community detection is not optional); the `neo4j` driver is
imported only when a URI is set.

**Deleting a document repairs the graph it contributed to.** With
`graph.enabled`, `delete_document` now also, after the document and its chunks
are gone: recomputes `frequency` for the entities that document mentioned,
deletes the ones nothing mentions any more (their relationships follow through
the endpoint cascade), and deletes the communities that touched any of those
entities. `frequency` therefore changes meaning — it is now *the number of
distinct chunks that mention the entity*, which is read by the
community-summary prompt (retrieval counts its own and never read the column).
A deleted community is rebuilt by the next `ingest(mode="graph")`, which
re-detects corpus-wide; until then a community that merely *shared* an entity
with the deleted document is gone corpus-wide too, so the community leg has
fewer summaries to draw on. When Neo4j is configured its chunk nodes are
removed after the Postgres side commits, and a Neo4j failure never fails the
delete — it is logged and reported to `on_error` with
`stage="delete_document_graph"`. Without `graph.enabled` a delete does no graph
work at all.

## Ingest

```python
from context_engine import ContextEngine

engine = ContextEngine(config)

# Plain text
report = await engine.ingest(
    text="Annual leave accrues at two days per month for every band-three employee.",
    name="leave-policy",
    source_id="hr-handbook",       # a namespace to scope search/stats
    acl=["group:hr"],              # omit for an unrestricted document (visible to everyone)
)

# A file — PDF/DOCX/PPTX/XLSX/HTML/EML/images/plain text, auto-detected
with open("policy.pdf", "rb") as f:
    report = await engine.ingest(file=f, name="policy.pdf", source_id="hr-handbook")

print(report.totals)               # {"files": 1, "failed": 0, "units": 3, ...}
print(report.documents[0].status)  # "completed" | "failed" | "batch_pending"
```

Construction is cheap and lazy — no connection is opened until the first call
that needs one. Release the pool (and the embedding client, and the Neo4j
driver) with `await engine.aclose()`, or use the engine as an async context
manager; a process-lifetime singleton can skip it, anything building engines
per job or per test should not.

- `mode="hybrid"` (default) or `mode="graph"` (needs the `[graph]` extra).
- Scanned pages / images are transcribed to **structured Markdown** (tables and
  headings preserved) when `vision_llm` is configured — otherwise embedded text,
  with Tesseract as a fallback if `[ocr]` is installed. **Vision is
  provider-agnostic**: it goes through the same `call_llm` seam as everything
  else, so Claude, GPT-4o, Gemini and Bedrock all work — whatever your
  `LLMConfig` names. The `[vision]` extra only adds page rasterization.
- Born-digital PDFs keep their structure too, with the `[pdf]` extra: the text
  layer is converted to Markdown locally — no OCR, no LLM, no network — so a
  table survives as a table into the chunker instead of being flattened into a
  run of words. Without the extra you get today's plain-text extraction.
- **A re-ingest only pays for what changed.** Identical content is still
  deduplicated by hash and skipped outright; when the content did change, every
  chunk whose text did not keeps the vector the previous version already paid
  for — copied inside the database, never read into the process — and only the
  changed chunks go to the provider. A chunk matches on a hash of its
  whitespace-normalised text, and only when the same embedding
  provider/model/dimension produced the stored vector. Synchronous ingest only:
  `batch=True` submits the whole chunk set and re-embeds all of it, because a
  batch result is paired with the document's rows by position. Two callers
  racing the same document are settled by the database, not by hope: the loser
  of the identity race reports `skipped` with `skip_reason="in_progress"` and
  writes nothing while the winner is still running — including the ACL and
  attribute updates that request carried, so re-offer the document once the
  winner is done.
- **Chunks are budgeted to `min(2000, embedding.max_input_tokens)`.** A chunk is
  one embedding input, so it is held to the provider's per-input cap — 512
  tokens on cohere, where a constant 2,000 would fail the document before the
  first request. Changing provider or `embedding.max_input_tokens` therefore
  re-chunks each document on its next ingest, and the vectors that would have
  been reused are paid for once more.
- **Two intake caps, both refusing rather than trimming.**
  `ingest.max_file_bytes` (256 MiB) is answered without reading the document at
  all — `os.stat` for a path, `Content-Length` for an HTTP upload.
  `ingest.max_rows` (unset) needs the bytes, and it is EXTENSION-SCOPED: only
  `.csv`, `.tsv` and `.xlsx` are counted, because the filename extension is
  also what routes a file to a tabular reader at all — a spreadsheet saved as
  `.txt`, or with no extension, is ingested as text and never counted. The
  count at the door is a pre-check; the streaming reader re-counts as it reads
  and THAT count is the authoritative one — it is what settles a newline inside
  a quoted field and an XLSX sheet's declared row count, and it refuses at the
  row that passes the limit instead of after parsing the whole file. Over
  either raises
  `IngestTooLarge`, which the routers map to **413** with a message naming the
  knob. Both take `None` to lift the limit, and neither ever samples or
  truncates: a silently shortened document is a corpus that answers wrongly and
  looks healthy doing it. `routing_core.upload_limit_bytes(engine)` hands the
  byte cap to your own upload middleware.
- `ingest()` never raises on a bad document — check `report.documents[i].status`.

### How a large document is embedded

Chunks are **stored before they are embedded**, and the embedding requests are
packed by tokens as well as by count. What that buys you:

- **A batch is closed by whichever cap it reaches first** — items or tokens —
  against a per-provider table, written items / tokens per request / tokens per
  input: openai-family 256 / 250k / 8,192; gemini and vertex_ai 100 / — /
  2,048; voyage 128 / 120k / 32k; cohere 96 / — / 512.
  Override any of them with `embedding.max_batch_items`,
  `embedding.max_batch_tokens` and `embedding.max_input_tokens` (all unset =
  the table). A spreadsheet is what makes this necessary: 256 rows of a wide
  sheet is a legal batch size and 450,000 tokens in one request is not.
- **Tokens are counted, not guessed, when they can be.**
  `embedding.tokenizer="auto"` (the default) uses cl100k through `tiktoken`
  **only if its encoding file is already in the cache** and estimates
  otherwise — it never downloads, because the ingest path may not fetch
  anything unasked. `"exact"` is the explicit opt-in that may fetch;
  `"estimate"` never imports it. The estimator is UTF-8 bytes ÷
  `embedding.token_bytes_ratio` (3.0), divided again by
  `embedding.safety_margin` (0.85) for headroom; the margin applies to the
  batch budget only, never to `max_input_tokens`.
- **A single chunk over `max_input_tokens` raises `InputTooLarge`** instead of
  being split — half a chunk's vector on a row that stores the whole chunk is a
  silent wrong-answer bug. Cohere requests carry `truncate: "NONE"` for the
  same reason.
- **A provider that rejects a batch for size halves the cap and retries**, for
  that run only; if it still refuses a one-item batch you get
  `EmbeddingBatchRejected` naming both suspects. 429s honour `Retry-After`
  within `embedding.max_retries` (2); other failures back off with jitter.
- **`embedding.max_concurrency` (4)** requests are in flight per document —
  enough to overlap network latency, not enough to earn the deployment a 429.
- **A failure keeps every batch you already paid for.** The document sits at
  the new non-terminal status **`embedding`** with
  `meta_data.embedding_progress = {done, total, tokens}`, and
  `await engine.resume_embeddings(document_id)` embeds only the rows still
  missing a vector — no re-extract, no re-chunk, no second bill for the
  finished batches. Called with no id it sweeps every resumable document,
  fenced by `ingest.resume_stale_seconds` (600) so a cron job cannot take over
  a run that is still moving; naming an id is never fenced. The trade is
  stated plainly: a mid-embed failure no longer leaves the PREVIOUS version
  indexed, because the new body was written when the run claimed the row. A
  partly embedded document is still findable — the text legs see every chunk,
  the vector leg only the embedded ones.
- **Progress is visible.** `on_progress` gains a third state `"progress"` on
  the `embed` stage, one per completed batch, carrying `{done, total, tokens}`;
  `started`/`done` are unchanged and `ingest.progress_batches` (on) is the
  escape hatch for a consumer that cannot ignore an unknown state. The
  `embed:` log line reports `chunks reused embedded requests tokens counter
  concurrency halvings retries` for one document. A custom `StorageBackend`
  joins in by implementing `set_chunk_embeddings(document_id, ids, vectors, *,
  progress=None, embedding_key=None)` — the engine always passes the key, an
  opaque `provider\tmodel\tdim` string to store verbatim beside the vectors,
  because a later re-ingest reuses a vector only when it matches; drop it and
  every row of that backend reads as "unknown" and is never reused. A Python
  backend that implements neither method still works, it just writes the
  vectors once at the end. `EmbedBatchFailed` carries how far the run got.
- **A file that could not be read raises `ExtractionFailed`**, with whatever
  the extraction library threw chained as `__cause__`. Two things are NOT
  wrapped, because both are caught by type upstream: `ImportError` (a missing
  optional extra, which carries an install hint) and `IngestTooLarge` — the
  streaming tabular readers apply `ingest.max_rows` as they go, theirs is the
  authoritative count, and every router maps that type to 413 with the knob,
  the limit and the actual row count in the body.

### Why a document failed

A failed document carries two answers, on purpose, because two different
readers need different things.

- `error` is the raw `str(exc)` of whatever stopped the run. It is an
  OPERATOR's copy: an embedding provider's error is a body the provider
  wrote, and those have carried an API key, a model name, a request id and a
  quoted fragment of the document itself. `engine.get_document`,
  `engine.list_documents` and `DocumentReport.error` all keep it.
- `failure_reason` is one code from a fixed vocabulary, classified from the
  exception rather than from its text: `input_too_large`,
  `provider_rejected`, `provider_unavailable`, `extraction_failed`,
  `chunk_set_changed`, `unknown`. `failure_message` is the one fixed sentence
  for that code — never interpolated with anything the provider said.

The model-facing doors return the code and the sentence and never `error`:
the knowledge tool's `get_doc`, `get_docs` and `list` (and therefore the MCP
tool, which is the same function), plus `GET /documents/{id}` on every
router. A router mounted with `principals=lambda: TRUSTED` is an operator's
own surface and still gets `error`; an anonymous or principal-scoped caller
does not. `list` carries the code only — a census of fifty documents does not
need fifty copies of one of six sentences.

A document that later succeeds clears both fields.

### What an ingest usage event says

`hooks.on_usage` is the metering seam — this package never touches billing —
and every `UsageEvent(kind="ingest")` carries the same two mode fields in its
`detail`, on every path that emits one (a completed ingest, a resumed one, a
completed batch, a mode upgrade, a structured-extraction backfill):

- `mode` — the mode the document was actually PROCESSED at, never the one
  that was asked for. On the resume and batch paths that is the mode stored on
  the row, because those runs decide nothing: they finish what an earlier run
  started.
- `mode_reason` — why that is not the requested mode, as one short fixed
  string (currently only `"tabular documents stay hybrid"`), or `None` when
  the two agree. The same value `DocumentReport.mode_reason` carries, and it
  is present on every event, `None` included: a key that only appears on the
  downgrade path is a key a consumer cannot switch on.

A spreadsheet ingested with `mode="graph"` therefore bills as `hybrid` with
that reason, on the first run and on every later one.

A **DOCX is billed by the page count the file reports**, not by a guess: `pages`
comes from the word processor's own `<Pages>` (`docProps/app.xml`), else from
the page breaks the last render left behind, else from hard and section breaks,
and only falls back to the ~250-words-a-page estimate when the file says
nothing. `pages_source` — `"app_xml"`, `"rendered_breaks"`, `"page_breaks"` or
`"estimate"` — says which, and rides beside `pages` everywhere it already goes:
`Extracted`, `DocumentReport`, the document's `meta_data`, the `extract`
progress event and this event's `detail`. It is `None` for every other format.
The declared count is believed only as far as the body could
hold it — blocks (paragraphs and table rows) plus breaks, plus one — so a
one-kilobyte upload declaring a hundred thousand pages is not a hundred
thousand units; the cost of that rule is that a document laid out over many
pages from ONE enormous paragraph falls back to the estimate.
It stops a bare declared number and nothing more: padding a
document with empty paragraphs satisfies the bound as cheaply, so page-based
metering stays uploader-controlled — bound the cost with
`ingest.max_file_bytes` and your own quotas on the usage events.

### Spreadsheets and CSVs

**CSV/XLSX documents are indexed in hybrid mode with one schema chunk per
sheet; aggregate questions go to `compute`.** The detail:

- Rows are emitted as **real RFC 4180 CSV** (quoted commas, doubled quotes,
  embedded newlines survive), streamed rather than materialised — a 22 MB CSV
  chunks in well under a quarter of a gigabyte of peak memory instead of
  failing. Empty unnamed columns are dropped; a named-but-empty column is kept,
  because the name is a fact.
- Each sheet gets a leading chunk with `meta.chunk_kind == "schema"` naming the
  columns, their inferred types and a few sample values, and pointing the model
  at `compute` for arithmetic. It is an ordinary chunk — hashed, embedded,
  retrieved and redacted like any other.
- **A tabular document stays hybrid even in a graph-mode corpus** and says so:
  `report.documents[i].mode_used == "hybrid"` with a fixed
  `mode_reason`. Its rows are a frame to compute over, not a graph to walk, and
  the per-chunk extraction LLM would be spent on thousands of near-identical
  lines.
- `ingest.max_document_text_chars` (unset) caps the stored `documents.text`.
  Leaving it unset is load-bearing: `compute` rebuilds its dataframes from that
  exact column, so a cut body is a spreadsheet that quietly answers with fewer
  rows than it has. Setting it is RECORDED — `meta_data.text_truncated` is
  written on every ingest, `true` or `false`, so raising the knob heals it —
  and both `compute` and `discover` refuse a truncated document by name rather
  than computing a wrong answer.

## Search

```python
result = await engine.search(
    "how much annual leave do I get",
    source_ids=["hr-handbook"],
    document_ids=None,         # optionally pin to specific documents WITHIN the sources
    principals=["group:hr"],   # TRUSTED = trusted/internal (skips ACL); [] = anonymous
    top_k=10,
)

for hit in result.hits:
    print(hit.document_name, hit.score, hit.chunk_text[:80])
```

Fuses full-text + trigram + vector search with Reciprocal Rank Fusion;
`mode="graph"` adds a graph leg. `compress_to_tokens=N` trims weak hits to fit a
prompt budget.

### Thresholds follow the query's shape

A fixed threshold is wrong for some query whatever value it takes, so
`pg_trgm`'s similarity threshold moves with the query
(`context_engine.query_shape`, the same rules in both ports). A two-character
query needs a strict threshold or `%` degenerates into noise; a twelve-word one
needs a loose one or it matches nothing: 0.45 down to 0.15, measured in
characters for spaceless scripts (Chinese, Japanese, Thai, Khmer, Burmese, Lao,
Tibetan) and in words for everything else. This is what the trigram leg has
always done.

**There is a matching floor for the vector leg, and it is OFF by default.** An
embedding model returns the *nearest* chunks, never "nothing is close enough",
so a minimum cosine similarity chosen from the query (0.45 for one or two
words, 0.40 for a code or an acronym, down to 0.25 for a sentence) drops the
rest from the vector leg before fusion — those rows can still arrive through
full-text or trigram, and a floor that empties the vector leg leaves the search
running on the other legs.

**The two exact-lookup shapes are checked FIRST and end it there.** A query
that asks for an exact string (`query_type` reads it as `exact_match`) or that
contains an acronym floors at 0.40 before the word-count ladder is consulted at
all, so a two-word acronym query such as `VAT rules` is 0.40, not the 0.45 its
length alone would give it. Both are looking for a token, and the vector leg's
answer to a rare token is the chunks least far away from it rather than the
ones that hold it.

```python
config = ContextEngineConfig(
    database_url=...,
    embedding=...,
    search=SearchConfig(vector_floor="adaptive"),   # opt in
)
# or, on a config you already have:
config.search.vector_floor = "adaptive"            # a typo here RAISES
```

**With the floor on, the vector leg can return fewer candidates than `limit`,
and nothing backfills them.** That is the point — a dropped row was weak — but
it means fusion sees a shorter list from that leg, and on a corpus where every
chunk is far from the query the leg can come back empty. The search still runs
on full-text and trigram.

**Measure before you turn it on.** Those numbers came from one embedding model.
Cosine similarity bands differ between models — some score every pair of texts
above 0.7, some cluster everything near 0.2 — so an absolute floor can empty the
vector leg for short queries on a model it was not measured against, and a
search that returns nothing is a worse failure than one that returns a weak row.

| setting | default | what it does |
| --- | --- | --- |
| `search.vector_floor` | `None` | `"adaptive"` turns on the floor above. Unset, retrieval behaves exactly as it did before the floor existed |
| `search.trgm_limit` | `None` | pins `pg_trgm`'s threshold to one value for every query, independently of `vector_floor`. `None` keeps the query-shaped one |

### The graph leg is chosen by the QUERY

Leaving `mode` out (the default) no longer means "this corpus has a graph
document somewhere, so use graph mode for everything". The leg runs when the
**query itself** resolves to an entity the graph knows and this caller is
allowed to see — otherwise one graph document would re-rank every search in the
corpus by its own vocabulary, which is exactly how six chunks of an unrelated
document came to fill an answer. Naming `mode="graph"` bypasses that gate but
not the entity set: a query that names nothing has no graph question to answer
and gets the hybrid answer.

The result says which of those happened, additively — every existing `usage`
key kept its place:

```python
result.usage["mode"]       # "graph"  — what you asked for
result.usage["mode_used"]  # "hybrid" — what actually ran
result.usage["graph"]      # {"applied": False,
                           #  "reason": "no_entities_in_query",
                           #  "entities": []}

# ... and for a query that does name one — entity names come back in the
# NORMALISED form they are stored under, not as the query spelled them:
#   {"applied": True, "reason": "requested", "entities": ["apple inc."]}
```

`reason` is one short fixed string from `context_engine.GRAPH_REASONS` —
`not_requested`, `graph_disabled`, `graph_unreachable`, `no_graph_documents`,
`no_entities_in_query`, `graph_detection_failed`, `no_visible_graph_chunks`,
`auto`, `requested` — so a host can switch on it instead of parsing prose.
`graph_detection_failed` is the mode-less path saying it could not work the
mode out at all and fell back to the hybrid legs; an explicit `mode="graph"`
never reports it, because that caller gets the error instead. One failure is
never degraded on either path: a missing `[graph]` extra raises its install
hint, because that is a configuration error and a deployment that believes it
has a graph must be told it has not — install the extra, or set
`graph.enabled=False`, which is answered before any import is attempted and
reports `graph_disabled`. `entities` lists
only the entities the query itself named, never the neighbours the leg reached
through them, in their normalised (lower-cased, whitespace-collapsed) stored
form. The usage event counts graph units only when the leg actually
contributed.

Four knobs shape the leg, all on `GraphConfig`:

| knob | default | what it does |
| --- | --- | --- |
| `rerank_weights` | `vector_score` 0.3, `entity_match` 0.3, `relationship_relevance` 0.2, `community_match` 0.1, `graph_connectivity` 0.1 | per-signal weights for the leg's ranking; a partial dict overrides only the keys it names, and an unknown key raises rather than doing nothing |
| `max_query_entities` | `30` | ceiling on the leg's entity set (matched entities plus one hop); a matched entity is never evicted by a hop |
| `entity_match_threshold` | `0.6` | `word_similarity` floor for the fuzzy fallback, which runs only when the exact pass found nothing. `None` switches that fallback off |
| `entity_max_name_words` / `entity_min_name_chars` | `5` / `3` | how wide and how short a query n-gram may be when probing for stored names |

The leg is bounded by the same per-leg candidate limit as the other three, and
only chunks with an actual graph signal reach fusion — a bare vector seed is
already the vector leg's, at its proper rank, and counting it twice is what
crowded better matches out. Entity resolution is index-backed (`migrate
--graph` creates the trigram and expression indexes it needs); the fuzzy pass
covers the first ~400 query tokens, so an entity named only in the tail of a
very long paste may be missed.

## Compute over tables

`engine.compute()` turns the in-scope CSV/XLSX documents into dataframes and has
the LLM write code against them. `compute_over_frames()` is the same path for
frames you already hold — an uploaded workbook, a connector's sheet, a query
result — with no document to point at:

```python
from context_engine import compute_over_frames

out = await compute_over_frames(
    {"expenses": df},              # {sheet name: DataFrame}
    "total the amount column",
    config=engine.config,
)
out["result"]
```

Both are **off by default** (`enable_code_execution=True` turns them on — only
behind real OS-level isolation) and both apply `config.redaction` before the LLM
sees anything: every string cell *and* every column header is masked, and the
returned dict — generated code included — is swept on the way out.

The prompt states what the sandbox refuses, and code refused by the static
check buys ONE rewrite: the model gets its own code and the refusal back and
answers again, and the second attempt goes through the same check. A frame key
that does not exist buys the same one rewrite (see below); no other runtime
error is retried. `out["attempts"]` is `1` or `2` either way, and both
calls' tokens are summed into `out["provider_tokens"]`, so a retried run costs
two code-generation calls. `on_error` fires once per FAILED attempt, with the
attempt number in its context (`{"stage": ..., "attempt": 1}`): a retry that
succeeds fires one event with `attempt == 1` while `attempts == 2`, and a run
that ends in failure fires one whose `attempt` equals `attempts`. `attempts`
counts calls ATTEMPTED, so counting "how often the model hits the sandbox
limits" from it alone over-counts by the provider-failure rate — the
`attempt` plus `stage: "compute_codegen"` pair on the error event is how to
tell a provider failure from a refusal.

**What the frames are called.** Over ONE document in scope the key is the
SHEET name (`dfs['sheet1']`), unchanged. Over several, a document with exactly
one sheet is keyed by its document NAME (`dfs['revenue.xlsx']`) and a document
with several keeps `<name>:<sheet>` (`dfs['book.xlsx:Q1']`); two documents that
share a name both fall back to `<name>:<sheet>`, and a key that still collides
after that gets `#2`, `#3` appended, because collapsing two frames into one
would drop a document from the answer silently. The keys depend on the
document SET and never on the order it arrives in — the suffixes are assigned
in document-id order, so `discover` (which lists caller-first) and `compute`
(which loads newest-first) give the same document the same key. Every sheet is
a frame, a header-only one included: it has columns and no rows. The prompt
lists the keys and says to use them exactly as listed, and a run that asks for
a key that is not there fails with `no frame named 'revenue.xlsx'; available:
['revenue.xlsx:Q1', 'revenue.xlsx:Q2', 'costs.csv']; did you mean
'revenue.xlsx:Q1'?` — handed to the model for its one rewrite, and reported as
`out["error"]` if the rewrite is wrong too.

The in-process sandbox behind `enable_code_execution` is defence in depth, not
a security boundary. To run the generated code somewhere actually isolated —
a subprocess, a container, a gVisor/WASM sandbox — pass `code_runner=` to
`ContextEngine(config, code_runner=...)` or directly to `compute()`/
`compute_over_frames()`: a callable that runs the same `(code, context,
timeout)` shape out-of-process (`context_engine.CodeRunner` names the shape).
That hook is the real isolation boundary; its result is trusted only as far
as JSON. Code bound for a host runner is refused only for the base reasons
(unsafe imports, dunder names/attributes, `open`/`eval`/`exec` and the other
dangerous builtins, `.system`/`.popen`/`.exec`/`.eval`); the in-process
sandbox's extra refusals — file-I/O names such as `to_string`/`read_csv`, the
same names string-dispatched through `agg`/`apply`/`query`, attribute
assignment, `yield`, `functools.update_wrapper`/`wraps`, and any
`.format()`/`.format_map()` that is not directly on a string literal — apply
only when the code runs in-process. Two limits are not static refusals at all:
in-process the sandbox has no class machinery, so defining a class or calling
`type()` with three arguments fails when the code RUNS, not in the checker
(and `isinstance(x, type)` raises with it — `type` inside the sandbox is a
one-argument function). The code-generation prompt states whichever set will
actually apply.

When the code does run in-process, `out["result"]` is plain data: the sandbox
converts it to `None`/`bool`/`int`/`float`/`str`/`list`/`dict` before
returning, so a DataFrame arrives as a list of row dicts (with a non-default
index moved into leading columns, so `groupby`/`describe` keep their labels),
a Series as a dict, a numpy scalar as a plain `int`/`float`, a date or
timestamp as an ISO-8601 string, and anything else with no plain-data form as
its `repr`. That conversion happens inside the sandbox's runtime guard and
uses type-level calls only, so nothing the generated code left attached to the
result can run afterwards — in the redaction sweep, in your own code, or in a
JSON encoder. Ask for a number, a string, a list or a dict (which is what the
prompt already asks the model for) and nothing is lost.

Two shapes are worth knowing before you parse the result. A DataFrame whose
column labels are not unique comes back as `{"columns": [...], "data": [[...],
...]}` and a Series with repeated index labels as `{"index": [...], "values":
[...]}`, because row dicts would silently drop every duplicate but the last —
and masked headers make duplicates real (two e-mail columns both become
`[EMAIL]`). A result past the call's output cap (`max_output_length`) or
nested deeper than 50 levels is cut with a marker string rather than raising;
the marker is an ordinary string, indistinguishable from identical text the
data itself contained. The generated code is handed COPIES of the frames and a
deep copy of the documents, so it cannot reach the objects you went on using,
and `out["documents_used"]` is built from ids captured BEFORE the run — as
`str(...)` of each id, so `UUID` ids come back as strings.

## Schema-enforced replies

Every place the engine asks a model for JSON — community summaries, entity
extraction, vision page transcription, structured extraction — sends a JSON
Schema the **provider** enforces instead of hoping the reply parses. Nothing is
configured; it applies to whatever `LLMConfig` names, with a fallback for a
model or endpoint that predates native support:

| provider | enforced with | fallback |
|---|---|---|
| `openai` · `azure_openai` · `custom` | `response_format` json_schema, `strict` | `json_object` + the schema in the prompt |
| `anthropic` | `output_config.format` | forced strict tool use |
| `gemini` · `vertex_ai` | `response_json_schema` | the schema in the prompt |
| `bedrock` | Converse `outputConfig` | forced tool use — strict only where the SDK can send it |

A downgrade fires only on an error that names the schema *parameter* — an
"invalid schema" error is a bug to fix, not something to work around — and
every genuine one is logged at WARNING naming the provider. Once a fallback has
actually worked the endpoint is remembered for the process, keyed by provider,
base URL and model, so the refusal is paid once rather than per call. That memo
has no expiry: a long-lived host that upgrades an SDK or a model calls
`reset_native_schema_support()` (`context_engine.providers.llm`) to let the
engine rediscover native support.

Bedrock's native path needs **botocore 1.43.0 or newer** — the release that
added Converse's `outputConfig`, and `strict` on a tool spec with it. boto3 is
not a declared extra here (a bedrock deployment brings its own), so that floor
is prose rather than a resolvable constraint. An older SDK is detected on its
own parameter validation, and because `toolSpec.strict` landed in the same
release the fallback tool cannot carry `strict` either: the model is *asked* to
fill the tool's schema rather than held to it, so that call honestly reports
`mode="prompt"` and its reply is validated locally. A service-side refusal on a
capable SDK is the case that does get a strict tool, and `mode="tool"` with it.

**`mode` is the guarantee, and it is on every reply.** `native` and `tool` mean
the provider enforced the schema, so a reply that still breaks it is refused
rather than re-asked — paying twice for the same bug fixes nothing. `prompt`
means the shape was only asked for in words: the reply is validated locally
before anything trusts it, and one failure buys exactly one retry that carries
the validation error and the model's own answer back. A second failure is loud.

```python
from context_engine.providers.structured import structured_call

result = await structured_call(
    config.llm,
    system="Extract the invoice fields.",
    user=text,
    schema={"type": "object",
            "properties": {"total": {"type": ["number", "null"]}}},
    schema_name="invoice",
)
result.value       # validated against the same bytes the provider was sent
result.mode        # "native" | "tool" | "prompt"
result.attempts    # 2 means the prompt path needed its one retry
result.tokens      # summed over every attempt
```

`structured_call` returns a `StructuredResult` or raises `StructuredCallError`
(`StructuredRefusal` when the provider refused in the field it reserves for
saying so). Both errors carry `mode`, `attempts`, `tokens` — a failed attempt
was billed like a successful one — and `raw`, the full final reply, for a call
site that wants to run its own tolerant reader over it. `raw` is deliberately
kept out of logs: read the attribute, don't dump the exception.
`call_llm(..., response_schema=, schema_name=)` is the layer below, if you want
the mechanism without the validation.

`map_reduce` is the one JSON call that sends no schema — its shape is the
caller's, not the engine's — so it stays on JSON mode and reports
`mode: "prompt"` in its `map_reduce_parse` error context.
`structured_repair_count()` is the number that says whether any of this is
being honoured: how many enforced-mode replies needed the prose helper this
process. It should stay at zero; a non-zero value means some provider accepted
a response schema and then answered around it.

Schemas travel through the **portable subset** — the intersection of every
provider's strict mode. Objects, arrays, strings, numbers, integers, booleans,
nulls and `$ref` into a root `$defs`; nullable is a type array
(`["string", "null"]`); every object gets `additionalProperties: false` and all
its properties required; enums hold scalars only. No `minItems`/`maxItems`, no
`pattern`/`format`, no `anyOf`/`oneOf`/`allOf`, no recursion. A schema outside
it raises `SchemaError` before any request goes out — `strict_portable()` from
`context_engine.llm_schemas` is the linter, and it is what normalises the schema
that is both sent and validated against. For a shape the subset cannot describe
at all — an object whose keys come from the document rather than from you —
pass `enforce=False`: no schema goes on the wire, JSON mode does, and the reply
is validated here against the schema as written (so that schema must state its
own `required` and `additionalProperties` — nothing fills them in). It relaxes
what may be described, never what is checked: the schema still has to stay
inside the keyword vocabulary both ports' validators implement — `format` is
not in it — and one that strays raises `SchemaError` before any paid call.

Host-visible effects, per call site:

- **Structured extraction with `field_hints` returns EXACTLY the hinted keys.**
  The schema is built from the hints and enforced, so each value is the hint's
  scalar type or null and extra keys are no longer possible. Without hints the
  payload stays free-form (`enforce=False`, envelope validated).
- **A document with no readable quality signal is now `quality="failed"`.** The
  envelope's `readability` is an enum and a missing or invalid one is a
  validation failure; it no longer falls back to `"medium"`.
- **New `on_error` stages:** `structured_call_attempt` and `structured_call`
  from the wrapper (per failed attempt and terminal, with `mode` and `attempt`),
  `structured_call_repair` when a provider that said it would enforce the schema
  returned something that is not JSON, plus `entity_extraction` and
  `structured_extraction` at the call sites — a failed extraction fires exactly
  one `structured_extraction` event, carrying `document_id`, `mode` and
  `attempts`. A host sees both the wrapper's
  per-attempt events and the site's outcome event — attempt telemetry and
  outcome are different questions — so a batch rescued on its retry still
  produced wrapper error events.
- **`hooks=` is now accepted by `extract_text_from_images` and
  `extract_pdf_pages`**, so vision failures reach `on_error` like everything
  else. A vision batch keeps its own three attempts around the wrapper's one
  retry, so the prompt path can cost up to six paid calls per batch.

## Serve

The same document/search/stats surface ships as a ready adapter for **FastAPI,
Flask, or Django** — pick the one matching your project. All three expose
identical routes (`POST/GET/DELETE/PATCH /documents`, `POST /search`,
`GET /stats`) with identical behavior (they share one framework-neutral core).
`auth` and `principals` are **required** arguments — the package implements no
auth, and resolving who the caller is has no safe default. To run a mount
open on purpose, say so: `principals=lambda: []` (anonymous — sees only
documents with no ACL). Returning `TRUSTED` (import it from `context_engine`)
means a fully trusted caller and skips ACL filtering entirely; returning
`None` is rejected with a 500 so a missing resolver can never open a mount by
accident.

A caller may only file a document under an ACL it already holds; `POST
/documents` and `PATCH /documents/{id}` answer `403` otherwise, so an
untrusted request can't push content into another group's retrieval scope.

**FastAPI** (`[fastapi]` extra):

```python
from context_engine import create_router

router = create_router(engine, auth=my_auth, principals=my_principals)
app.include_router(router, prefix="/context")
```

**Flask** (`[flask]` extra) — `auth(request)` / `principals(request)`, async views:

```python
from context_engine import create_flask_blueprint

bp = create_flask_blueprint(engine, auth=my_auth, principals=my_principals)
app.register_blueprint(bp, url_prefix="/context")
```

**Django** (`[django]` extra) — include the urlpatterns in your `urls.py`:

```python
from django.urls import include, path
from context_engine import create_django_urlpatterns

urlpatterns = [
    path("context/", include(create_django_urlpatterns(engine, auth=my_auth, principals=my_principals))),
]
```

Not on any of them? Just call `engine.ingest()` / `engine.search()` directly
from your own views — the engine needs no web framework at all.

Or serve the same engine as an **MCP** tool surface (`[mcp]` extra):

```bash
context-engine mcp --database-url postgresql://... --port 8080
```
```python
from context_engine import create_mcp_app, UNSCOPED
app.mount("/mcp", create_mcp_app(engine, principals=my_principals, scope=UNSCOPED))
```

`scope` is the ceiling of source ids this mounted tool may ever reach, and is
REQUIRED. It is resolved per call like `principals` and is never a tool
argument: a model can ask for any source id it likes, and a tool that believed
it would let one caller read another's documents. Map your own word onto it —
a project, a matter, a customer — and pass `scope=lambda: ["src-a", "src-b"]`,
or `Scope(source_ids=(...), document_ids=(...))` to hand an agent specific
documents. A host with one shared corpus writes `scope=UNSCOPED` on purpose.
The caller can narrow within that ceiling and never widen past it.

Registers ONE knowledge tool, `search_knowledge_base`, whose `action` picks
what it does:

| Action | What it is for |
|---|---|
| `discover` | what documents there are and what is inside each one — sheets and columns, sections and pages, or JSON keys — plus the extracted fields by document type, a census of the corpus, and what each action can do. Call it first |
| `search` | passages by meaning or keywords |
| `get_doc` / `get_docs` | one whole document by id, or several at once |
| `get_chunks` | walk one long document in order, a piece at a time. Start at a search hit's `chunk_idx` to read around that passage |
| `list` | browse the documents without searching |
| `query_meta` | filter documents by their structured fields |
| `compute` | a figure derived from the spreadsheets, over every row |
| `map_reduce` | the same question asked of every document in scope |
| `get_neighbors` / `traverse` | what is one step, or a few steps, from a named thing |
| `find_related` | connections of a kind across the corpus |
| `community_summary` | the themes the corpus groups into |

Every graph answer carries the entity `id` beside the name — `id` on the start
entity and on each neighbour, `target_id` on a traversal path,
`source_id`/`target_id` on a `find_related` relationship — and `entity`
accepts one. That is the handle that survives an output redaction policy:
where a policy masks the names, the name in an answer is a mask token and
matches nothing, so navigation would otherwise stop after one hop. The id
carries no document text, so it is exempt from that masking, and it is a
handle rather than a capability: it resolves behind the same visibility
predicate a name does, so an id the caller cannot see gives the same `entity
not found` an unknown name gives. It is stable while the entity exists, not a
permanent handle — rebuilding the graph mints new ids. A name still works
wherever it was not masked — the question, a search hit, the documents. With no
unmasked name to start from, `find_related` is the way in: it takes a category
rather than an entity, and its answer hands back the `source_id`/`target_id` to
navigate from.

Combining the tools into sub-actions of one is better for the model that calls
them: one description carries the decision rules once, and there is one name
to route through.

Every search hit carries `chunk_idx`, the position of the chunk it matched
inside its document, beside `document_id`, `document_name`, `chunk_text`,
`score` and `source_id`. It is the argument `get_chunks` takes as `start`, so
reading around a passage is one call rather than paging a long document from
the beginning. A hit found by file name carries it too, pointing at the first
chunk it showed, and a document with no chunks at all reports `null`.

`discover` answers with the SCHEMA, not an inventory, so one call is enough to
write one correct `compute` or `query_meta` call instead of three exploratory
ones. Every listed document carries `document_type`, `mode`, and a `structure`
whose shape follows the type — `sheets` (name, column headers, row count, and
`frame_key`: what `compute` will call that sheet's dataframe, over the
documents THIS call listed) for a workbook, `sections` and `last_page` for a
document with headings, `keys` for JSON, and `chunks` as the floor for
anything else. Beside them,
`fields_by_type` lists the extracted structured field names with each field's
data type and how many in-scope documents carry it, and `document_types` is a
census of the corpus. Both are keyed by `(kind, type)`: `kind` comes from the
mime and is always known, while `document_type` is free text an LLM wrote and
exists only where `extract_structured` was enabled — so a PDF invoice and a
spreadsheet of invoice rows stay separate rather than merging under one label.

Lists inside a `structure` are capped, with the remainder reported as
`more_columns` / `more_sections` / `more_keys` / `more_sheets`, because
`discover` is the first call of a conversation and an unbounded list lands in
the model's context before it has asked anything. **The cap is for the call
that did not name its documents:** `discover` and `list` take `document_ids`,
and a `discover` scoped to specific documents answers their structure whole
and narrows the field grouping and the census to them too. While the remainder
is small enough to be worth fetching, the truncated structure carries that
call already filled in; past that it says what to do instead, because a sheet
with hundreds of columns is one to compute over rather than read back.

Omit `mode` on a search and it is worked out from the documents in scope — the
graph leg is added when the deployment has a graph and something in scope was
ingested that way, and it runs beside the other legs rather than ahead of
them. Naming `"hybrid"` or `"graph"` forces one.

An AGGREGATE question — a total, an average, a count, a ranking, a group-by, a
numeric comparison — is REFUSED when every document in scope is a spreadsheet.
`search` and `query_meta` answer `{"success": false, "error": …,
"next_action": {"action": "compute", "query": …, "suggestion": {"frame_key":
…, "measure": …, "group_by": …}}}` instead of returning rows. `query` is the
question handed straight back, so the redirect is a call to make rather than
advice to translate; `suggestion` beside it is the vocabulary, not an
argument, and it is plain DATA rather than code: `frame_key` is the name
`compute` will really give that sheet's frame (the same rule `discover`
reports, computed over the documents in this call's scope), and `measure` /
`group_by` are the newest sheet's first numeric and first text column. Any of
the three can be null.
Search returns EXCERPTS, so totalling what it returned is totalling a sample
and reporting it as the figure. A MIXED scope still searches and only carries
an additive `hint`; a scope with no spreadsheets in it, or with nothing
visible to the caller, is untouched. `map_reduce` is refused the same way for
ANY instruction, aggregate-shaped or not — it reads a SAMPLE of each document,
so the answers it merges over sheets are partial totals — and the refusal lands
before any model call and before a host-supplied runner; there the question is
asked of the documents that CALL would read (the same ordering and the same
`limit or 25` cap `map_reduce` itself selects by, resolved first as ids and then
READ as ids, so the decision and the fan-out cannot be about different
documents), so a scope of thirty whose newest twenty-five are sheets is refused
rather than sampled, and a mixed target set carries a `hint` with its own
reason. The decision is one grouped query over
the documents the call would read, under the caller's ACL and the host's
ceiling. Set `redirect_aggregates_to_compute=False` on the config to turn it
off.

A question that NAMES A FILE — "get detail about INV-EG.pdf", "what is in
report_2024.xlsx" — searches document NAMES instead of chunk text, and answers
one row per document with `matched_by: "filename"` beside the ordinary `hits`
(`document_name` is the name, `chunk_text` the document's first chunk and
`chunk_idx` that chunk's position). Chunk
text is the wrong haystack for it: the file name appears inside the document
only by accident, so the one document the caller asked for is the one thing the
text legs cannot reliably find. The name match is a trigram similarity on the
lowercased name, under the same scope predicate and ACL as every other document
read, with an exact name always first and the document id as the final
tiebreak. When nothing is called that — including when the only document with
that name is one the caller may not see — the ordinary search runs on the
original query, so the branch can only add an answer, never remove one, and the
call is metered once rather than twice. The aggregate refusal above is asked
FIRST, so "what is the total in sales.csv" still goes to `compute`. **With
`redirect_aggregates_to_compute=False` and this branch left on, that same
question is answered from the named file's FIRST chunk, which for a spreadsheet
is the schema chunk: a sample of the columns, never a total.** Turning the
redirect off is the operator's decision, and so is that answer. Three
things it does not do: `mode` is ignored when it answers (there is no text leg
to rank), `top_k` is the row cap with the same floor of 1 and no ceiling
`search` applies, and detection is ASCII-only — a document named
`rapport-café.pdf` never triggers the branch and is answered by the ordinary
search. Its `usage` carries `legs: []` and `leg_hits: {}` — present and empty,
because no leg ran — and omits `candidates`, `reranked`, `compressed` and
`provider_tokens`, which describe work this path does not do. The rule is
`query_shape.filename_in_query`, a pure function on the module (not exported
from the package root). Set `filename_search=False` on the config to turn it
off.

`query_meta` and `map_reduce` need an LLM, `compute` needs code execution
explicitly enabled, and the graph actions need a configured graph — `discover`
marks them off otherwise, and an off action called anyway answers a plain
`{"success": false, "error": …}` result instead of hiding. `discover` and
`list` page (`has_more` + `next_cursor`, sent back as `cursor`), and a
truncated answer carries the `next_page` call that reads the rest. The
connector-tool gateway (`search_tools` + `execute_tool`) is registered beside
it — `search_tools` takes the same `kind` / `name_contains` /
`requires_approval` filters and `cursor` as the method, and answers
`{"tools": [...], "next_cursor": ...}`.

`knowledge_tool_definition()` exports the tool as data — name, description and
input schema as JSON Schema — so a host wiring it into its own agent loop never
hand-writes the schema. `create_mcp_app` also takes `redaction` (a policy
applied per call), and `compute` / `map_reduce` callables that REPLACE the
built-in ones, which is where a host applies its own rules for permission,
billing and approval.

The same tool is a plain method, so an application driving its own agent loop
needs no MCP at all — and cannot get a different answer, because both doors
call one implementation:

```python
answer = await engine.search_knowledge_base(
    action="search", query="annual leave", principals=caller, scope=["hr"]
)
```

## Tools

Let an LLM/agent call `http` / `db` / `mcp` (external server) and `function`
(a Python callable) tools through one governed path — ACL-checked, config
AES-256-GCM-encrypted at rest, every call audited.

```python
from context_engine import ToolConfig

# HTTP tool — the LLM fills `city` at call time
await engine.register_tool(ToolConfig(
    name="get_weather", kind="http",
    description="Fetch the current weather for a city",
    config={
        "method": "GET",
        "url": "https://api.example.com/weather",
        "headers": {"Authorization": "Bearer super-secret-token"},
        "llmQueryParameters": {"properties": {"city": {"type": "string"}}, "required": ["city"]},
    },
))  # -> call_name "http_get_weather"

# Function tool — no config, no persistence
def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b
engine.register_function_tool(add)  # -> call_name "fn_add"

out = await engine.execute_tool("http_get_weather", {"city": "Lahore"})
# -> {"result": {"status_code": 200, "data": {...}}, "usage": {"kind": "tool", "units": 1}}
```

**Two tools with one name.** Nothing makes a tool's name unique, so two
registered tools can share a call name. Every tool that `list_tools` and
`search_tools` return carries its `tool_id`; hand it back to say which one you
mean. Without it a name that matches more than one tool the caller can see
raises `AmbiguousToolError` (409 on `POST /tools/execute`) instead of running
whichever row loaded first. The name is still required, and a `tool_id` the
caller cannot see reads as "tool not found":

```python
hit = (await engine.search_tools("weather", principals=["user:a"]))[0]
out = await engine.execute_tool(hit["name"], {"city": "Lahore"},
                                principals=["user:a"], tool_id=hit["tool_id"])
```

**Discovery** — `search_tools` ranks on the query and filters on everything
else. `kind` (one of `http`/`db`/`mcp`/`function`, or a list), `name_contains`
(case-insensitive substring of the call name) and `requires_approval` combine
with AND, so an empty query with filters lists the filtered set, and an unknown
kind is an empty result rather than an error:

```python
page = await engine.search_tools(
    "", principals=["user:a"], kind="db", requires_approval=False, limit=20
)
if page.next_cursor:  # only present when more matches remain
    rest = await engine.search_tools(
        "", principals=["user:a"], kind="db", requires_approval=False, limit=20,
        cursor=page.next_cursor,   # same query + filters, or it is refused
    )
```

The page is the list it has always been, so a caller that ignores paging is
unaffected. The walk is a live offset rather than a snapshot: a tool registered
or deleted mid-walk shifts the tail.

A non-empty `query` also EXCLUDES: a tool none of its terms match is not in the
answer, so `search_tools("nonsense", kind="db")` returns nothing rather than
every db tool. The query ranks *within* what the filters selected — it does not
replace them. To list without hiding anything, pass an empty query with the
filters you want, or call `list_tools`, which never hides a tool.

**Approval gate** — a tool with `requires_approval=True` returns an
`approval_required` payload instead of running, until resolved:

```python
from context_engine import resolve_approval

out = await engine.execute_tool(
    "http_get_weather", {"city": "Lahore"},
    principals=["user:a"],
    approval_scope="run-42",   # opaque, chosen by you — a run id is the usual shape
)
# -> {"approval_required": {"approval_id": "...", "expires_at": "...", ...}}   (not executed yet)
await resolve_approval(engine, approval_id, "approved", approver="boss")
# the same call, same approval_scope, runs for real — exactly once
```

`approval_scope` is the claim scope: an approved record is consumed only by
the same call in the same scope (and, when the record carries principals, by a
caller who overlaps them). Within a scope, identical pending requests share one
record, so a retrying agent gets the same `approval_id` back. Resolve the scope
server-side — on the routers and the MCP gateway it is a dependency beside
`principals`, never a request field. Calling without it still works this
release, unscoped and with a `DeprecationWarning`; the next release refuses a
gated call with no scope.

An approval is for one tool, not for a name. The record carries the `tool_id`
it was opened for (`None` for a function tool), the `approval_required` payload
and `ApprovalRecord` both show it, and an identical call to another tool that
shares the name opens its own record instead of claiming this one. `migrate`
adds the column and swaps the pending unique index for one that includes it;
a record opened before that carries no `tool_id`, so it is no longer claimable
by a registered tool and the call simply asks again.

Getting the request in front of a human (Slack, in-app) is the consumer's job —
the engine owns only the single-use, atomic approval state machine. Park until
`expires_at`; after that the record can no longer be approved.

**Result budget** — the returned `result` is cut to 8,000 characters and 100
rows unless you say otherwise. Size it to your model's window, or pass `None`
to lift a limit. `response_mode="tsv"` works for **any** tool: every array of
objects in the result — a db tool's `rows`, an HTTP API's `data.items`, a
function's returned list — comes back as a TSV string under the same key, and
the budget cuts at whole rows:

```python
out = await engine.execute_tool(
    "db_sales", {"query": "SELECT * FROM orders"},
    principals=["user:a"],
    result_max_chars=200_000,  # None = no character limit
    result_max_rows=None,      # per table; a db tool's own max_rows (default 1000) still applies
    response_mode="tsv",       # -> {"success": True, "columns": [...], "rows": "id\tname\n1\t..."}
)
# a cut reports where: {"_result_shaping": {"rows": {"rows_returned": 812, "rows_omitted": 188}}}
```

In TSV, `\N` is NULL, nested values are compact JSON cells, and tab, newline,
CR and backslash inside a value are backslash-escaped. Parts of a result that
aren't arrays of objects stay JSON; a result that *is* one becomes the string
(wrapped as `{"result": ..., "_result_shaping": ...}` when cut). The audit row
always keeps the original under its own 50KB cap, whatever you pass.

An **http tool** can set its format once, in its config — for every method
(`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `QUERY`). Omitted, it is `json`; a
caller's explicit `response_mode=` still wins:

```python
await engine.register_tool(ToolConfig(
    name="orders", kind="http",
    config={"method": "QUERY", "url": "https://api.example.com/orders",
            "response_mode": "tsv"},   # "json" (default) | "tsv"
    acl=["group:ops"],
))
```

The same conversion is a plain function for anything else:
`format_result(result, "tsv")` (and `rows_to_tsv(rows)`) from `context_engine` —
`"json"`, the default, returns the input unchanged.

**A failed call's error text is stripped, then masked**, and the caller, the
audit row (`context_engine_tool_calls.error_message`) and the `on_tool_call`
hook all get the same reduced text. A db driver's message loses what carries
row data first: where the driver offers structured fields the message is built
from those, otherwise the first signal line is kept and the
`DETAIL`/`HINT`/`CONTEXT`/`LINE n`/caret lines, the `[SQL: ...]`/`[parameters:
...]` echo and a statement echoed back verbatim are removed, and every
single-quoted literal is blanked to `'?'` (double-quoted identifiers — the
constraint and column names that say what broke — are kept). What survives
then goes through the call's
redaction policy, principals and key, the same ones the success path uses, so
a per-tenant policy applies to failures too. If the masking itself fails — a
`hash` rule with no key, discovered only once the tool has already run — all
three surfaces get `error text withheld: the redaction policy could not be
applied` instead of the unmasked text, and the real reason is logged for the
operator. The exception type and the failure shape are unchanged; only the
text is.

**Connecting an `mcp` tool that needs OAuth** ends in a popup on your origin
that `postMessage`s the result to `window.opener`. On failure that message is
now `{type: "mcp-oauth-error", code, message, correlation_id}`. `code` is the
stable name to branch on, one of `idp_denied`, `malformed_callback`,
`state_invalid`, `token_endpoint_unreachable`, `token_exchange_failed` and
`internal_error`; `message` keeps its key but is one of six fixed sentences
written for the person looking at the popup, **not the underlying error text**
— so a host that string-matched on it must read `code` instead.
`correlation_id` is `mcpoauth_<32hex>`, shown on the page as `Reference: …` and
logged server-side beside the detail that is deliberately not rendered or
posted; the success payload carries it too. The provider's `error_description`,
a token endpoint's response body and any exception text reach the log and
nothing else.

The six sentences, so a host building its own fallback UI can match or replace
them — identical in both clients:

| `code` | `message` |
|---|---|
| `idp_denied` | The provider declined the connection. Ask your administrator to approve this app, then try again. |
| `malformed_callback` | The provider's response was incomplete. Please start the connection again. |
| `state_invalid` | This connection attempt expired. Please start again. |
| `token_endpoint_unreachable` | Could not reach the provider's servers. Please try again in a moment. |
| `token_exchange_failed` | The provider rejected the connection. Please try again. |
| `internal_error` | Something went wrong completing the connection. |

They are the wording, not the contract — branch on `code`, and treat a sentence
as something that can be reworded in any release.

## Redaction

Span-level masking of PII (or anything you can describe with a regex or a
function) as content is ingested, retrieved, or returned. It **composes with
ACL** (ACL = *who* may see a document; redaction = *what's masked* inside it)
and is a **pure transform** — no model, no network. The default policy is a
guaranteed no-op.

```python
from context_engine import ContextEngineConfig, RedactionPolicy, RedactionRule

config = ContextEngineConfig(
    database_url="postgresql://...",
    embedding=...,
    redaction=RedactionPolicy(rules=[
        RedactionRule(name="emails", detector="email"),                  # -> [EMAILS]
        RedactionRule(name="order_id", pattern=r"\bORD-\d{6}\b"),         # -> [ORDER_ID]
        RedactionRule(name="acct", pattern=r"Account:\s*(?P<value>\d+)"), # -> "Account: [ACCT]"
    ]),
)
```

A `pattern` rule whose regex names a group `value` (either `(?P<value>...)`
or `(?<value>...)` — both are accepted, normalised on load) masks only that
group's span instead of the whole match; a match where `value` didn't
participate contributes no masking at all.

Built-in detectors: `email` · `phone` · `ssn` · `credit_card` · `iban` ·
`api_key`. The two `phone` forms have different rules. Without a country
code it needs a `-` or a `.` between the groups — the same one in both
gaps — or the `(415) 555-0123` form, optionally behind a `1`/`+1`:
`(415) 555-0123`, `415-555-0123`, `415.555.0123`, `1-415-555-0123` and
`+1 (415) 555-0123` all match, while `415 555 0123`, `415-555.0123` and a
bare `4155550123` do not. Behind a leading `+` the rule is looser — a `+`
then a digit run that may be split by spaces or dashes — so `+14155550123`
and `+1 415 555 0123` both match, though a dotted `+1.415.555.0123` does
not.
Actions: `mask` · `hash` (keyed, joinable pseudonym — needs
`redaction_secret_key`, which falls back to `secret_key`) · `remove`.
`apply_at`: `output` (default, masks on every read, can be
principal-conditional via `unless`) · `ingest` (scrub before storage) · `both`.

`hash` normalises the matched text per detector before HMACing it, so one
real-world value written two ways gives one token: `phone` keeps the decimal
digits only, folded to ASCII (a non-ASCII decimal digit — Arabic-Indic, say —
counts as its ASCII value, since the detectors match those too), and a
10-digit match is then hashed as if it carried a `+1` — that is a
HASHING rule, not a detection one, and a bare `4155550123` in text is still
not a phone number at all; `email` trims and lower-cases; `ssn` and
`credit_card` fold digits the same way;
`iban` strips whitespace and upper-cases. `api_key` and every `pattern` rule
hash the raw match — case is significant for an opaque token and a custom
pattern has no detector to normalise by. **A `hash` token stored by an earlier
version for a `phone`, `email`, `ssn`, `credit_card` or `iban` rule no longer
matches the token this version produces for the same value and key.**

One engine serving many tenants can give each its own policy and hash key
without a second connection pool:

```python
tenant_engine = engine.with_redaction(tenant_policy, secret_key=tenant_hash_key)
```

The view shares the pool, providers and tools, applies `tenant_policy` on every
method (ingest rules included), and hashes with `tenant_hash_key`, so tokens do
not join across tenants. Stored tool configs still decrypt with the engine's
`secret_key`. It is a Python call only — no HTTP or MCP argument reaches it —
and closing the view closes nothing. Called with no `secret_key` while
`redaction_secret_key` is unset and the policy has a `hash` rule, the view
falls back to `secret_key` and warns (`UserWarning`), because tokens then join
across every tenant sharing that fallback — pass a key, set
`redaction_secret_key`, or filter the warning if that is what you meant. With
no key at all it raises instead: a `hash` rule with no effective key is
refused when the view is built.

A mounted MCP app takes the same pair per call instead of a view:
`create_mcp_app(engine, principals=..., scope=..., redaction=policy,
secret_key=tenant_hash_key)`. Each is a value or a callable resolved fresh on
every call — so one mounted app can serve many tenants — and both are host-set,
never a tool argument. The connector-tool gateway takes the same pair
(`register_tool_gateway(..., redaction=..., secret_key=...)`). It is also a
per-call argument on every read path
that already took `redaction=` — `search`, `get_document`, `get_documents`,
`get_chunks`, `list_documents`, `spreadsheet_schema`, `document_structure`,
`document_types`, `field_summary`, `query_structured`, `map_reduce`,
`compute`, `search_knowledge_base` and `execute_tool`. Omitted, the engine's
effective redaction key applies.

**The detectors are deliberately a small set — the placement is the point.**
Redaction runs *inside* retrieval: hits are masked before `_maybe_rerank` POSTs
their text to Cohere/Voyage/Jina, and tool results are masked before the audit
row is written, so the masked value is the only one that leaves the process or
lands in retention. That ordering is what usually goes wrong when a redaction
library is bolted onto a pipeline by hand.

For broader detection, bring [Microsoft Presidio](https://github.com/microsoft/presidio)
(`[presidio]` extra) — its entities become ordinary rules and inherit the same
placement guarantees:

```python
from context_engine import presidio_detectors

RedactionPolicy(
    rules=[RedactionRule(name="person", detector="PERSON")],
    custom_detectors=presidio_detectors(["PERSON", "LOCATION"]),
)
```

Any `Callable[[str], list[tuple[int, int]]]` works as a detector, so an
in-house classifier drops in the same way.

---

## Benchmarks

**Does ACL filtering cost you retrieval quality?** Yes — badly — unless the
vector leg is tuned for it, and the failure is silent.

Any system that applies an access-control predicate as a `WHERE` clause on top
of an approximate vector index is doing a *post-filter*: the index walk is
ordered by distance to the query, rows the caller may not see are discarded as
they are met, and the walk stops when its candidate budget runs out. If the
caller can see only a slice of the corpus, that budget can be spent entirely
on rows they aren't allowed to have.

And the size of the visible slice is not the variable that matters most.
**Every ACL benchmark we know of assigns permissions at random. Real
permissions are clustered** — `group:hr` covers HR documents, and those are
semantically adjacent by construction, so a group's slice is a *region* of the
embedding space rather than a uniform sample.

200k documents from BEIR `nq`, real 1536-dim embeddings, exact brute-force
oracle over eligible rows. Same corpus, same index, same queries, same number
of visible rows — only the arrangement differs:

| permissions | visible | post-filter recall | empty | with iterative scan |
|---|---:|---:|---:|---:|
| random | 10% | 0.980 | 0.0% | 0.980 |
| **clustered** | 10% | **0.473** | **25.0%** | 0.928 |
| random | 1% | 0.175 | 20.8% | 0.966 |
| **clustered** | 1% | **0.087** | **75.0%** | 0.883 |

Clustering roughly doubles the recall loss. HNSW descends greedily from an
entry point toward the query: scattered eligible rows are met along the walk,
concentrated ones are not, so a query pointing away from the visible region
spends its whole budget on rows the caller may not see.

**Iterative scan does not fully close it** — 0.883 clustered against 0.966
random, same setting. That residual is why this library *decides* between an
exact and an approximate scan per query rather than tuning a parameter and
hoping.

**You may not be able to reproduce the failure, and that is also a result.**
With a GIN index on `acl`, all twelve default-planner rows chose an exact
`bitmap+sort` instead of the vector index — recall 1.000, no failure, at
44–107 ms p50. Which plan Postgres picks is a cost estimate that moves with
dimension, corpus size and statistics freshness, and `&&` selectivity
estimation on GIN arrays is unreliable enough to flip it without warning. The
table above therefore forces the index. Whether *you* are exposed is decided
per query, by a cost model — which is the actual argument for deciding it
yourself.

**Measure it on your own corpus** rather than trusting ours — offline,
read-only, no embedding provider called:

```bash
context-engine check-acl-exposure --database-url postgresql://...
```

It reports, for the ACL values that actually occur in your data, what a bare
scoped query returns versus what this library returns, both scored against an
exact oracle computed inside Postgres.

Written up in full: **[Your ACL Benchmark Is Measuring the Easy Case](https://promptev.ai/acl-benchmarks-measure-the-easy-case/)**.
Method, how `clustered` is constructed, the limitations — including a metric
that behaved as a control rather than a result, and a retracted earlier table
— are in [`benchmarks/acl_recall/`](benchmarks/acl_recall/), along with the
harness and raw CSVs. Requires pgvector ≥ 0.8.

### Checking your database

Managed Postgres varies in ways that matter here. To check a provider before
committing to it:

```bash
CE_COMPAT_DATABASE_URL='postgresql+psycopg2://user:pass@host/db' \
    uv run pytest tests/test_managed_postgres_compat.py -v -s
```

Point it at a **scratch database** — it runs the real migrations. It verifies
pgvector is new enough for iterative scan, that `pg_trgm`/`unaccent` are
installable, and — the one that catches people — that their functions are
actually reachable on the role's `search_path`. Supabase installs extensions
into an `extensions` schema, so `CREATE EXTENSION` can succeed while
`similarity()` and `%` stay unresolvable and the trigram leg breaks at query
time rather than at migrate time.

## Deep dives

This README covers the basics. Full documentation is at
**https://promptev.ai/documentation/context-engine/**. The exhaustive design
notes — redaction apply points, `hash` joinability, overlapping-rule merge
semantics, ACL edge cases, graph internals, cost model, and known limitations —
also live in the module and config docstrings (`config.py`, `redaction.py`,
`engine.py`).
