Metadata-Version: 2.5
Name: docpipe-maestro
Version: 0.4.1
Summary: A minimal, provider-agnostic, structure-aware RAG pipeline library.
Project-URL: Homepage, https://github.com/ivnmoreno/docpipe-maestro
Project-URL: Repository, https://github.com/ivnmoreno/docpipe-maestro
Project-URL: Issues, https://github.com/ivnmoreno/docpipe-maestro/issues
Project-URL: Changelog, https://github.com/ivnmoreno/docpipe-maestro/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/ivnmoreno/docpipe-maestro/blob/main/docs/README.md
Author-email: ivnmoreno <ivnmoreno@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: chromadb,chunking,embeddings,llm,nlp,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: chromadb>=1.5.9
Requires-Dist: langchain-text-splitters>=0.3.0
Requires-Dist: litellm>=1.108.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: tomli>=2.0.0; python_version < '3.11'
Provides-Extra: all
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'all'
Requires-Dist: docutils>=0.20; extra == 'all'
Requires-Dist: ebooklib>=0.18; extra == 'all'
Requires-Dist: openpyxl>=3.1.0; extra == 'all'
Requires-Dist: pypdf>=6.21.3; extra == 'all'
Requires-Dist: python-docx>=1.1.0; extra == 'all'
Requires-Dist: python-pptx>=1.0.0; extra == 'all'
Provides-Extra: docx
Requires-Dist: python-docx>=1.1.0; extra == 'docx'
Provides-Extra: epub
Requires-Dist: ebooklib>=0.18; extra == 'epub'
Provides-Extra: html
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'html'
Provides-Extra: pdf
Requires-Dist: pypdf>=6.21.3; extra == 'pdf'
Provides-Extra: pii
Requires-Dist: presidio-analyzer>=2.2.0; extra == 'pii'
Requires-Dist: presidio-anonymizer>=2.2.0; extra == 'pii'
Provides-Extra: pptx
Requires-Dist: python-pptx>=1.0.0; extra == 'pptx'
Provides-Extra: rerank
Requires-Dist: sentence-transformers>=3.0.0; extra == 'rerank'
Provides-Extra: rst
Requires-Dist: docutils>=0.20; extra == 'rst'
Provides-Extra: xlsx
Requires-Dist: openpyxl>=3.1.0; extra == 'xlsx'
Description-Content-Type: text/markdown

# docpipe-maestro

[![PyPI](https://img.shields.io/pypi/v/docpipe-maestro.svg)](https://pypi.org/project/docpipe-maestro/)
[![CI](https://github.com/ivnmoreno/docpipe-maestro/actions/workflows/ci.yml/badge.svg)](https://github.com/ivnmoreno/docpipe-maestro/actions/workflows/ci.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![Type Checked: mypy](https://img.shields.io/badge/type%20checked-mypy-blue.svg)](https://mypy-lang.org/)

> **"docpipe-maestro is a minimal RAG pipeline: it ingests any document, chunks it correctly, embeds it, stores it, retrieves the right sections, and optionally generates a grounded answer. It is the engine — not the app, not the framework."**

`docpipe-maestro` is a provider-agnostic, structure-aware Python RAG (Retrieval-Augmented Generation) pipeline library and `docpipe-maestro` CLI. It solves the data-preparation and retrieval failure modes common in standard fixed-size chunkers: preserving document hierarchy, refusing to split legal or policy clauses mid-sentence, resolving intra-document cross-references, and guaranteeing that policy exceptions always stay attached to their rules.

---

## Table of Contents

- [Why docpipe-maestro?](#why-docpipe-maestro)
- [60-Second Quickstart](#60-second-quickstart)
- [Key Architecture & Core Concepts](#key-architecture--core-concepts)
- [Supported Formats & Limitations](#supported-formats--limitations)
- [Installation Guide](#installation-guide)
  - [Core Install](#core-install)
  - [Optional Extras](#optional-extras)
  - [Development Setup](#development-setup)
- [Configuration Guide](#configuration-guide)
  - [1. Local Execution (Default: Zero Cloud Accounts Required)](#1-local-execution-default-zero-cloud-accounts-required)
  - [2. Cloud Provider API Keys (`.env` or Shell Environment)](#2-cloud-provider-api-keys-env-or-shell-environment)
  - [3. Local Persistence Storage Directory](#3-local-persistence-storage-directory)
  - [4. CLI Configuration File (`docpipe.toml`)](#4-cli-configuration-file-docpipetoml)
  - [5. Chunking Profiles (`DEFAULT`, `POLICY`, and Custom Profiles)](#5-chunking-profiles-default-policy-and-custom-profiles)
- [Quickstart: CLI Usage Guide](#quickstart-cli-usage-guide)
  - [1. Ingest Documents (`docpipe-maestro ingest`)](#1-ingest-documents-docpipe-maestro-ingest)
  - [2. Keep Collections in Sync (`docpipe-maestro sync`)](#2-keep-collections-in-sync-docpipe-maestro-sync)
  - [3. Inspect Collection Status (`docpipe-maestro status`)](#3-inspect-collection-status-docpipe-maestro-status)
  - [4. Remove a Source Document (`docpipe-maestro remove`)](#4-remove-a-source-document-docpipe-maestro-remove)
  - [5. Migrate Embedding Models (`docpipe-maestro reindex`)](#5-migrate-embedding-models-docpipe-maestro-reindex)
  - [6. Semantic Search Without an LLM (`docpipe-maestro query`)](#6-semantic-search-without-an-llm-docpipe-maestro-query)
  - [7. Grounded Question Answering (`docpipe-maestro ask`)](#7-grounded-question-answering-docpipe-maestro-ask)
  - [8. Offline Retrieval Evaluation (`docpipe-maestro eval`)](#8-offline-retrieval-evaluation-docpipe-maestro-eval)
- [Python Library API Guide](#python-library-api-guide)
  - [Phase 1: Ingestion & Lifecycle Workflow](#phase-1-ingestion--lifecycle-workflow)
  - [Phase 2: Semantic Retrieval & Grounded RAG Workflow](#phase-2-semantic-retrieval--grounded-rag-workflow)
  - [Retrieval Evaluation (`docpipe_maestro.eval`)](#retrieval-evaluation-docpipe_maestroeval)
  - [Opt-In Structured Logging (`docpipe_maestro.log`)](#opt-in-structured-logging-docpipe_maestrolog)
  - [Core Data Shapes & Exceptions](#core-data-shapes--exceptions)
- [Security, Privacy & Grounding Notes](#security-privacy--grounding-notes)
- [Safety & Verification](#safety--verification)
- [Repository Structure](#repository-structure)
- [Contributing](#contributing)
- [Documentation](#documentation)
- [License](#license)

---

## Why docpipe-maestro?

| Common RAG Failure Mode | How `docpipe-maestro` Solves It |
|---|---|
| **Orphaned exceptions** (*"20 days leave"* retrieved without *"except probationary hires"*) | **Parent-child retrieval + `POLICY` clause preservation** searches fine-grained child chunks and returns deduplicated full parent sections, enforced by the mandatory Reverse Safety Gate (`tests/test_reverse_safety.py`). |
| **Broken intra-document references** (*"subject to Section 4.2"* without knowing what Section 4.2 says) | **`POLICY` `CrossrefsEnricher`** (`enrichers=("crossrefs",)`) appends cycle-safe `[Crossref ...]` excerpts directly onto referring sections during Phase 1 ingestion (toggleable at query time via `--no-crossrefs` / `follow_crossrefs=False`). |
| **Exact identifiers & ranking misses in dense-only search** (`SEC-101`, `IR-15M`) | **Hybrid search + child-first reranking** pairs collocated SQLite FTS5 BM25 + dense vector RRF (`fts5_hybrid`) with optional local cross-encoder reranking (`--rerank` / `rerank=True`). |
| **Cross-corpus noise across files, sections, profiles, or dates** | **Unified `where` metadata & manifest filtering** compiles validated filter operators (`$eq`, `$ne`, `$in`, `$nin`, `$gt`, `$gte`, `$lt`, `$lte`, `$and`, `$or`) over chunk fields and manifest attributes (`profile`, `ingested_after`, `ingested_before`) with ChromaDB and FTS5 parity and zero re-ingestion. |
| **Orphaned vectors when files shrink or change** | **Transactional `doc_version` staging** writes new chunks, flips the JSON manifest atomically, and deletes stale versions afterward (with startup `sweep_uncommitted()` recovery). Failed runs leave the prior version intact. |
| **Silent embedding vector-space corruption** | **Manifest model lock** pins each collection to one `embed_model` and raises `ModelMismatchError` unless migrated explicitly via `reindex`. |
| **Hallucinations when nothing relevant matches** | **Deterministic empty-context refusal** returns `"Not found in the provided context."` without calling the LLM (`ask` checks retrieved hits automatically; `answer()`/`answer_detailed()` short-circuit when `has_context=False`). |
| **Sensitive PII stored or sent to remote LLMs** | **Opt-in pre-chunking `PiiEnricher` + outbound `guard_egress()`** redact, HMAC-tokenize, or block entities cleanly (`PiiBlockedError`), log zero-raw-PII `.audit.jsonl` records (`0o700`/`0o600`), and enforce `--local-only` pre-network across embeddings and generation. |
| **Hidden HTML/XLSX/EPUB injections & unsafe archives** | **Hardened intake** strips hidden DOM/EPUB elements (`display:none`, `hidden`, `aria-hidden`, `<template>`), skips `hidden`/`veryHidden` Excel sheets, disables RST file-inclusion/raw directives, enforces single-pass ZIP uncompressed size caps (`256 MiB`) across `.docx`/`.xlsx`/`.pptx`/`.epub`, rejects legacy `.doc`/`.xls`/`.ppt` binaries, and skips directory symlinks. |
| **Blind retrieval regressions, vendor lock-in & library bloat** | **Offline `eval` harness, Strategy registries & Library Purity** deliver reproducible `MRR` / `Precision@k` / `Recall@k` checks, 9 `@register_*` extension points, `NullHandler`-only `docpipe_maestro.log` (`--log-json` `stderr` telemetry), zero config-file reads in library code, and LiteLLM routing with zero hardcoded provider registry. |

---

## 60-Second Quickstart

Run an entirely local pipeline using [Ollama](https://ollama.com/) (`nomic-embed-text` + `llama3.2`):

```bash
# 1. Install from PyPI with all format parsers
pip install "docpipe-maestro[all]"

# 2. Pull local Ollama models (one-time)
ollama pull nomic-embed-text
ollama pull llama3.2

# 3. Ingest a folder of policies and ask a grounded question
docpipe-maestro ingest ./policies --profile POLICY --collection handbook
docpipe-maestro ask "When are employees not entitled to overtime?" --collection handbook --threshold 0.35
```

*(Running from a cloned source repository? Prefix CLI commands with `uv run docpipe-maestro ...` or `uv run python -m docpipe_maestro ...`.)*

---

## Key Architecture & Core Concepts

```mermaid
flowchart LR
    subgraph P1["Phase 1: Offline Ingestion (Atomic & Transactional)"]
        A[Source Files] --> B[Intake & Format Detect]
        B --> C[Clean & Boilerplate Strip]
        C --> D["Enrichers (PII -> Crossrefs)"]
        D --> E[Structure-Aware Chunker]
        E --> F["local_only Check + Batched Embeddings"]
        F --> G[(ChromaDB + FTS5 + Manifest + .audit.jsonl)]
    end

    subgraph P2["Phase 2: Online Query (Read-Only)"]
        Q[Question] --> QE["local_only Check + Query Embed"]
        QE --> S["Child Search (Dense or fts5_hybrid RRF)"]
        S --> RR["Optional Rerank (Cross-Encoder Sigmoid)"]
        RR --> PE["Parent Expansion & max(score) Dedupe"]
        PE --> BP["Pure build_prompt() / build_chat_messages()"]
        BP -->|results > 0| EG["Egress Guard (local_only + PII Policy)"]
        EG --> LLM["Optional answer() via LiteLLM"]
        BP -->|empty results| REF["Deterministic Refusal (No LLM Call)"]
    end
```

1. **Two-Phase Separation**:
   - **Phase 1 (Offline Ingestion)**: Files $\to$ Parse $\to$ Clean $\to$ Enrich (`pii` $\to$ `crossrefs`) $\to$ Structure-Aware Chunking $\to$ Embeddings $\to$ Vector Database + `.audit.jsonl`. Correctness and crash safety matter.
   - **Phase 2 (Online Query)**: Question $\to$ Embed $\to$ Retrieve (Match Children, Return Deduplicated Parents) $\to$ Build Prompt $\to$ Egress Guard $\to$ Optional Grounded LLM Generation. Read-only and low-latency.
2. **Search Small, Return Big (Parent-Child Retrieval)**:
   - Small **child chunks** (individual clauses/paragraphs prefixed with `[Source: ... | Section: ...]`) are embedded and searched for high precision.
   - Large **parent sections** (complete contextual sections) are stored with zero-vector placeholders and returned to the prompt.
3. **Exception Preservation & Reverse Safety Gate**:
   - In policy and legal documents, a rule without its exception is actively misleading (e.g., *"Employees receive 20 days leave"* without *"unless probationary"*). In `POLICY` profile, clauses are kept intact even when exceeding target chunk size, and parent sections ensure exceptions always accompany their rules.
4. **Crash-Safe Transactional Synchronization**:
   - Ingestion maintains an atomic database-collocated manifest (`.docpipe/manifests/<collection>.json`) tracking `sha256` content hashes, `mtime`, and `doc_version` prefixes.
   - Modifying a document writes new versioned chunks, commits the manifest atomically, and deletes stale versions afterward, preventing orphaned clauses when documents shrink.
5. **Embedding Model Lock**:
   - The manifest locks the collection to the embedding model used during ingestion. Attempting to ingest or query with a different model raises `ModelMismatchError` rather than silently mixing incompatible vector spaces. Model migration is handled cleanly via `reindex`.

---

## Supported Formats & Limitations

| Format | Extensions | Structural & Content Extraction | Extra Required |
|---|---|---|---|
| **Markdown** | `.md`, `.markdown` | ATX headings (`#`–`######`), code fence awareness (` ``` ` / `~~~` ignores `# comments`) | Core install |
| **Plain Text** | `.txt` | Heuristic `Section` / `Article` / `Chapter` / `Part` and numbered headings | Core install |
| **CSV / TSV** | `.csv`, `.tsv` | Strict UTF-8 / BOM normalization, trimmed empty trailing columns, ragged-row tolerance, 25-row bounded `Header: Value` sections | Core install |
| **HTML** | `.html`, `.htm` | `<h1–h6>` headings, paragraphs, lists, tables (` \| `), strips `<script>`/`<style>`/`<nav>`/`<footer>`/`<noscript>` & hidden subtrees | `[html]` (`beautifulsoup4`) |
| **Word** | `.docx` | Word `Heading 1`–`Heading 6` styles, paragraphs, tables (` \| `) | `[docx]` (`python-docx`) |
| **PDF** | `.pdf` | Per-page text, 1-based page numbers, multi-page active heading carryover, boilerplate header/footer stripping | `[pdf]` (`pypdf`) |
| **Excel** | `.xlsx` | Visible sheets only (skips `hidden`/`veryHidden`), evaluated cached values, 25-row bounded `Header: Value` sections | `[xlsx]` (`openpyxl`) |
| **PowerPoint** | `.pptx` | Per-slide titles & 1-based page numbers, body text, grouped shapes, tables (` \| `), speaker notes | `[pptx]` (`python-pptx`) |
| **EPUB** | `.epub` | Ordered spine chapters with `<h1–h6>` section headings, offline hidden-DOM/boilerplate stripping | `[epub]` (`ebooklib`) |
| **reStructuredText** | `.rst` | Section/heading hierarchy (`heading_level`), paragraphs, lists, tables (` \| `), admonitions; local file/raw directives disabled | `[rst]` (`docutils`) |

- **Extensionless files**: `detect_format()` inspects the first 4 KiB for `%PDF-`, legacy OLE2 binaries (rejected cleanly), single-pass `PK\x03\x04` ZIP containers (`word/document.xml` → `docx`, `xl/workbook.xml` → `xlsx`, `ppt/presentation.xml` → `pptx`, `META-INF/container.xml` → `epub`), HTML root tags (`<!doctype html` / `<html`), Markdown headings, or valid UTF-8 text.
- **Current scope & limitations**:
  - Image-only / scanned PDFs without an embedded text layer yield empty text (OCR is out of scope).
  - Legacy binary `.doc`, `.xls`, and `.ppt` files are not supported (modern `.docx`, `.xlsx`, `.pptx`, or `.csv`/`.tsv` only).
  - `docpipe-maestro` is a library and CLI engine — it deliberately excludes web servers, auth layers, streaming UIs, and multi-tenant isolation.

---

## Installation Guide

`docpipe-maestro` requires **Python 3.10+**.

### Core Install

The core installation includes Markdown (`.md`, `.markdown`), plain text (`.txt`), CSV (`.csv`), and TSV (`.tsv`) parsing, embedding generation, ChromaDB vector storage, and semantic retrieval:

```bash
# Install from PyPI
pip install docpipe-maestro

# Or install the latest development head from GitHub:
# pip install "git+https://github.com/ivnmoreno/docpipe-maestro.git"
```

### Optional Extras

Generation (`answer()`, `ask`) is available in the base install — no extra needed. Install optional extras for additional document format parsers:

| Extra | Purpose | Dependencies Installed |
|---|---|---|
| `pdf` | Extract text, page numbers, and heuristic section headings (`Section X.Y`) from PDF files | `pypdf>=6.21.3` (BSD-3-Clause) |
| `docx` | Extract text, style-based headings (`Heading 1`–`Heading 6`), and tables from Word documents | `python-docx>=1.1.0` |
| `html` | Strip boilerplate and parse `<h1–h6>` headings from web pages / HTML exports | `beautifulsoup4>=4.12.0` |
| `xlsx` | Extract visible Excel worksheets into header-paired 25-row sections | `openpyxl>=3.1.0` |
| `pptx` | Extract numbered slides, body/tables, grouped shapes, and speaker notes from PowerPoint decks | `python-pptx>=1.0.0` |
| `epub` | Extract ordered spine chapters and headings from EPUB books safely offline | `ebooklib>=0.18` |
| `rst` | Extract structured sections and tables from reStructuredText offline (`.. include::` disabled) | `docutils>=0.20` |
| `all` | Installs **all** optional document format parsers (`pdf`, `docx`, `html`, `xlsx`, `pptx`, `epub`, `rst`) | `pypdf`, `python-docx`, `beautifulsoup4`, `openpyxl`, `python-pptx`, `ebooklib`, `docutils` |
| `rerank` | Second-stage local neural cross-encoder reranking (`BAAI/bge-reranker-base` default) | `sentence-transformers>=3.0.0` |
| `pii` | Optional multi-language NER PII recognition alongside deterministic recognizers | `presidio-analyzer>=2.2.0`, `presidio-anonymizer>=2.2.0` |

> **Note on Structure Extraction**: Markdown, plain text, CSV/TSV, HTML, DOCX, PDF, XLSX, PPTX, EPUB, and RST extract headings, clauses, tables/rows, and slide/page metadata cleanly without external SaaS parsers. Note that `[all]` installs document parsers only; include `[rerank]` and/or `[pii]` alongside it (`[all,rerank,pii]`) when enabling reranking or Presidio NER. For offline `[pii]` usage, pre-install the required local spaCy models (e.g. `en_core_web_sm`, `es_core_news_sm`).

Install specific extras:
```bash
# PDF support only
pip install "docpipe-maestro[pdf]"

# PDF, DOCX, and XLSX support
pip install "docpipe-maestro[pdf,docx,xlsx]"

# Full installation with all document format parsers
pip install "docpipe-maestro[all]"

# Parsers + local cross-encoder reranking + optional NER PII detection
pip install "docpipe-maestro[all,rerank,pii]"
```

### Development Setup

To contribute or run tests locally (uses PEP 735 `[dependency-groups]`):

```bash
# Clone the repository
git clone https://github.com/ivnmoreno/docpipe-maestro.git
cd docpipe-maestro

# Option A: Using uv (recommended)
uv sync --all-extras --group dev

# Option B: Using pip 24.1+ (PEP 735 dependency groups)
pip install --group dev -e ".[all]"
pre-commit install --hook-type commit-msg
```

> **Note on Versioning & Commitizen**: Package versions are derived dynamically from Git tags via `hatch-vcs`. Use `uv run cz check` (to validate Conventional Commits) and `uv run cz changelog` (to update `CHANGELOG.md`), rather than `cz bump`.

---

## Configuration Guide

`docpipe-maestro` enforces strict **Library Purity**: **no library module reads external configuration files**. Configuration is passed explicitly via function arguments or provider environment variables. Only the `docpipe-maestro` CLI optionally reads a `--config` TOML file.

### 1. Local Execution (Default: Zero Cloud Accounts Required)

Out of the box, `docpipe-maestro` defaults to local execution with [Ollama](https://ollama.com/):
- **Default Embeddings Model**: `ollama/nomic-embed-text`
- **Default Generation Model**: `ollama/llama3.2`

Make sure Ollama is installed and running (`http://localhost:11434`), then pull the default models:

```bash
ollama pull nomic-embed-text
ollama pull llama3.2
```

> **Tip — Choosing a local generation model**: For multi-section policy synthesis (especially on CPU or limited RAM), `ollama/qwen2.5:1.5b-instruct` or `ollama/qwen2.5:3b` follows retrieved context more reliably than `ollama/llama3.2` (which can over-refuse paraphrased queries) and `<1B` models (which are too small for grounded synthesis). Override the default with `--model ollama/qwen2.5:1.5b-instruct` or set `model = "ollama/qwen2.5:1.5b-instruct"` in `docpipe.toml`.

> **Air-gapped / strict zero-egress environments**: By default, `litellm` attempts to download its public model-cost JSON map from GitHub when first imported before falling back to its bundled copy. Set `export LITELLM_LOCAL_MODEL_COST_MAP=true` in your shell to prevent any startup HTTP call on import.

### 2. Cloud Provider API Keys (`.env` or Shell Environment)

Copy the tracked template `.env.example` to `.env` (git-ignored) and restrict permissions:

```bash
cp .env.example .env && chmod 600 .env
```

Or export provider environment variables directly in your shell (shell variables always take precedence over `.env`):

```bash
# Groq serverless inference
export GROQ_API_KEY="your-groq-api-key"

# Together.ai serverless inference (TOGETHER_API_KEY alias is also recognized by LiteLLM)
export TOGETHERAI_API_KEY="your-together-api-key"

# OpenAI / Anthropic / Google Gemini
export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
export GEMINI_API_KEY="your-gemini-api-key"

# Air-gapped / offline startup cost-map safety
export LITELLM_LOCAL_MODEL_COST_MAP="true"
```

You can then pass any `provider/model` string directly to `--embed-model` or `--model` (e.g., `groq/llama-3.3-70b-versatile`, `together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo`, `openai/gpt-4o-mini`, `gemini/gemini-2.5-flash`). Note that the CLI loads `.env` strictly from the `--config` directory or current working directory (never walking parent directories), while direct Python API callers configure `os.environ`.

### 3. Local Persistence Storage Directory

By default, data is stored in `./.docpipe`:
```text
.docpipe/
├── chroma.sqlite3                    # ChromaDB embedded vector database
├── <uuid>/                           # ChromaDB vector segments and HNSW index files
├── lexical/
│   └── <collection>.db               # Collocated SQLite FTS5 BM25 lexical index (`fts5_hybrid`)
└── manifests/
    ├── <collection>.json             # Atomic JSON manifest tracking file hashes, versions, and locked model
    └── <collection>.audit.jsonl      # Append-only zero-raw-PII audit log (written when PII policy is active)
```
Customize the persistence directory with `--persist-dir` on the CLI or `persist_dir=` in Python. Never commit `.docpipe/` to version control if your corpus is private.

### 4. CLI Configuration File (`docpipe.toml`)

When `--config` is omitted (and `--no-config` is not passed), the CLI automatically loads `./docpipe.toml` from the current working directory if present (non-recursively; never walking parent directories). Create an optional TOML configuration file to set default CLI options (never store API keys or secrets here — explicit CLI flags override `--config` / `./docpipe.toml`, which override built-in defaults):

```toml
# docpipe.toml
collection = "my_kb"
persist_dir = "./data/docpipe_index"
embed_model = "ollama/nomic-embed-text"
model = "ollama/llama3.2"
profile = "POLICY"
top_k = 5
threshold = 0.4
searcher = "default"               # "default" (dense vector) or "fts5_hybrid" (dense + SQLite FTS5 RRF)
# Optional second-stage cross-encoder reranking (requires `[rerank]` extra):
# rerank = false
# reranker = "cross_encoder"
# rerank_top_n = 20
# Optional online/gateway defaults for `docpipe-maestro ask` (CLI flags override these):
# api_base = "https://api.groq.com/openai/v1"
# temperature = 0.0
# max_tokens = 512                 # Keep omitted (or >= 4096) for reasoning models
# timeout = 60.0
# fail_on_truncation = false
# Privacy & PII defaults (JSON/TOML path string or `[pii_policy]` table applied in CLI only):
# local_only = false
# pii_policy = "./pii-policy.json"
```

Pass an explicit config path with `--config`, or ignore `./docpipe.toml` with `--no-config`:
```bash
docpipe-maestro query "What is the probation period?" --config docpipe.toml
docpipe-maestro ask "What is the probation period?" --no-config
```

#### Optional PII Policy File (`--pii-policy`)

When enabling `--pii-policy ./pii-policy.json` (or `.toml`) on `ingest`, `sync`, or `ask`, configure actions (`"redact"`, `"tokenize"`, `"block"`) per entity type (`EMAIL_ADDRESS`, `IBAN_CODE`, `CREDIT_CARD`, `US_SSN`, `IP_ADDRESS`, `PHONE_NUMBER`, `ES_DNI`, plus optional `[pii]` Presidio `ner: true`):

```json
{
  "types": {
    "EMAIL_ADDRESS": "tokenize",
    "IBAN_CODE": "redact",
    "US_SSN": "block"
  },
  "token_salt_env": "DOCPIPE_PII_SALT",
  "egress": "redact"
}
```
*(In `docpipe.toml`, set `pii_policy = "./pii-policy.json"` or define an inline `[pii_policy]` table. Use `token_salt_env` outside the repository for deterministic cross-run tokens.)*

### 5. Chunking Profiles (`DEFAULT`, `POLICY`, and Custom Profiles)

Built-in profiles are immutable presets (`docpipe_maestro.config`) calibrated for specific document types:

| Parameter | `DEFAULT` Profile | `POLICY` Profile |
|---|---|---|
| **Intended For** | Notes, general prose, articles, documentation | Handbooks, compliance policies, HR manuals, contracts |
| **Target Chunk Size** | ~512 tokens | ~1000 tokens |
| **Chunk Overlap** | ~64 tokens (scoped within section) | ~80 tokens (scoped within section) |
| **Clause Preservation** | Disabled (splits on sentences/paragraphs) | **Enabled** (never splits inside detected numbered clauses; PDF clause detection is heuristic) |
| **Cross-Reference Resolution** | Disabled | **Enabled** (resolves `§2.1`, `Section 4`, `Clause 3.2`) |
| **Enrichers (`enrichers`)** | `()` (skipped) | `("crossrefs",)` |
| **Parent-Child Expansion** | Enabled | **Enabled** |

> **Note on PDF Clause Detection**: Markdown, DOCX, and HTML headings/numbering provide explicit structural boundaries, whereas PDF clause detection uses extracted-text heuristics (`1.1`, `Clause 2`, `(a)` layout cues) and depends on clean PDF text extraction order.

In Python, you can also pass a custom [`ChunkingProfile`](docs/api-reference.md) instance directly to `ingest(..., profile=custom_profile)` or `sync(...)`, and configure ordered post-clean enrichers via `profile.enrichers` and `@register_enricher`.

---

## Quickstart: CLI Usage Guide

Run `docpipe-maestro --help` to see all available subcommands (global flags such as `--config`, `--no-config`, `--collection`, `--persist-dir`, `--local-only`, `--log-json`, and `--log-level` work either **before or after** any subcommand):

```text
usage: docpipe-maestro [-h] [--version] [--config CONFIG] [--no-config]
                       [--collection COLLECTION] [--persist-dir PERSIST_DIR]
                       [--local-only | --no-local-only]
                       [--pii-policy PII_POLICY] [--log-json]
                       [--log-level {DEBUG,INFO,WARNING,ERROR}]
                       {ingest,sync,status,remove,reindex,query,ask,eval} ...
```

See the full [CLI Reference](docs/cli-reference.md) for all flags, TOML keys, exit codes, and output schemas.

### 1. Ingest Documents (`docpipe-maestro ingest`)

Ingest a single file or an entire directory tree atomically (if any file fails validation, no records are committed):

```bash
# Ingest with default profile
docpipe-maestro ingest ./docs --collection knowledge_base

# Ingest policy documents with clause-preservation and cross-reference resolution
docpipe-maestro ingest ./company_policies --profile POLICY --collection hr_policy

# Ingest strictly locally with pre-chunking PII scrubbing/blocking and .audit.jsonl output
docpipe-maestro ingest ./company_policies --local-only --pii-policy ./pii-policy.json --collection hr_policy

# Force re-ingestion even if files have not changed on disk
docpipe-maestro ingest ./docs --force --collection knowledge_base

# Ingest policies and export a self-contained offline HTML integrity & chunk audit report
docpipe-maestro ingest ./company_policies --profile POLICY --collection hr_policy --html-report ./audit.html
```

### 2. Keep Collections in Sync (`docpipe-maestro sync`)

Reconcile a target directory against the manifest (adds new files, skips unchanged files via `mtime`/`sha256`, updates modified files, and purges deleted files), or preview changes and offline token/cost estimates safely first with `--dry-run` (plus `--json` for CI gating):

```bash
# Preview deltas, chunks, tokens, and estimated cost without mutating disk or calling embedders
docpipe-maestro sync ./company_policies --profile POLICY --collection hr_policy --dry-run

# Apply reconciliation
docpipe-maestro sync ./company_policies --profile POLICY --collection hr_policy
```
**Example output**:
```text
Synced 'hr_policy': added=1, updated=2, skipped=10, removed=1.
```

### 3. Inspect Collection Status (`docpipe-maestro status`)

View indexed files, active `doc_version` hashes, chunk counts, and locked embedding model:

```bash
docpipe-maestro status --collection hr_policy
```
**Example output**:
```text
Collection: hr_policy | Model: ollama/nomic-embed-text (dim=768) | Sources: 2 | Children: 18 | Parents: 6
  - hr_leave_policy.md [4f1a29b01c3d] (10 children, 3 parents)
  - travel_expenses_policy.md [a8c2194ef001] (8 children, 3 parents)
```

Use `--json` for machine-readable output or `--html-report` to export a self-contained single-file HTML Document Integrity & Chunk Audit Report:
```bash
docpipe-maestro status --collection hr_policy --json
docpipe-maestro status --collection hr_policy --html-report ./audit.html
```

### 4. Remove a Source Document (`docpipe-maestro remove`)

Purge all child chunks, parent sections, and manifest entries for a specific file:

```bash
docpipe-maestro remove hr_leave_policy.md --collection hr_policy
```

### 5. Migrate Embedding Models (`docpipe-maestro reindex`)

Re-embed all existing child chunks stored in the database under a new embedding model, update parent vector dimensions, and re-lock the collection (or pass `--dry-run [--json]` to estimate active child chunks, tokens, and offline cost before making API calls):

```bash
# Preview active chunks and estimated embedding cost before migrating
docpipe-maestro reindex --model openai/text-embedding-3-small --collection hr_policy --dry-run

# Migrate collection to OpenAI embeddings
export OPENAI_API_KEY="sk-..."
docpipe-maestro reindex --model openai/text-embedding-3-small --collection hr_policy
```

### 6. Semantic Search Without an LLM (`docpipe-maestro query`)

Perform Phase 2 semantic retrieval to inspect scored passages and provenance without invoking a language model:

```bash
# Basic semantic query
docpipe-maestro query "How many vacation days do full-time employees accrue?" --collection hr_policy

# Restrict to top 3 sections with a minimum similarity threshold (0.0 to 1.0)
docpipe-maestro query "What is the flight reimbursement policy?" --top-k 3 --threshold 0.5 --collection hr_policy

# Combine dense vector similarity with SQLite FTS5 BM25 via Reciprocal Rank Fusion
# (ideal for exact error codes, identifiers, or clauses like "SEV-1" or "AES-256")
docpipe-maestro query "SEV-1 escalation window" --searcher fts5_hybrid --collection hr_policy

# Apply optional second-stage cross-encoder reranking over top candidate child chunks
# (requires `docpipe-maestro[rerank]`)
docpipe-maestro query "SEV-1 escalation window" --searcher fts5_hybrid --rerank --rerank-top-n 20 --collection hr_policy

# Return raw child chunks instead of expanded parent sections
docpipe-maestro query "annual leave" --no-expand-parent --collection hr_policy

# Scope search by document metadata or manifest profile/ingestion window (without re-ingestion)
docpipe-maestro query "annual leave" --where profile=POLICY --where page=2 --collection hr_policy
docpipe-maestro query "annual leave" --where ingested_after=2026-01-01 --collection hr_policy

# Emit structured JSON (for evaluation fixtures and scripts) or add --log-json telemetry on stderr
docpipe-maestro query "annual leave" --json --collection hr_policy
docpipe-maestro --log-json --log-level INFO query "annual leave" --collection hr_policy 2>trace.jsonl
```

**Example output**:
```text
[1] score=0.8421 | Source: hr_leave_policy.md | Section: 3. Annual Leave
3.1 Full-time employees accrue 20 days of paid annual leave per calendar year.
3.2 Employees with less than one year of tenure accrue leave on a pro-rata basis.
------------------------------------------------------------
```

### 7. Grounded Question Answering (`docpipe-maestro ask`)

Retrieve relevant context, build a grounded prompt, and generate a citation-backed answer:

```bash
# Using local Ollama (default)
docpipe-maestro ask "How many days of leave does a first-year employee get?" --collection hr_policy

# Using a custom Ollama tag or cloud model / OpenAI-compatible base URL
docpipe-maestro ask "Summarize travel meal limits." --model ollama/qwen2.5:1.5b-instruct --collection hr_policy
export OPENAI_API_KEY="sk-..."
docpipe-maestro ask "Summarize travel meal allowance limits." --model openai/gpt-4o-mini --collection hr_policy
docpipe-maestro ask "Summarize travel policy." --model openai/my-local-model --api-base http://localhost:8000/v1 --collection hr_policy

# Safe thresholding: if no retrieved content matches threshold, short-circuits with a refusal
docpipe-maestro ask "What is the recipe for lasagna?" --threshold 0.7 --collection hr_policy

# Inspect retrieved context on stderr, request JSON telemetry on stdout, stream tokens live, or fail if truncated at max_tokens
docpipe-maestro ask "Summarize travel limits." --show-context --collection hr_policy
docpipe-maestro ask "What is the SEV-1 escalation window?" --searcher fts5_hybrid --rerank --collection hr_policy
docpipe-maestro ask "How many days of annual leave do employees get?" --stream --collection hr_policy
docpipe-maestro ask "Summarize travel limits." --json --fail-on-truncation --collection hr_policy

# Enforce local-only execution pre-network or scrub outbound remote prompts via --pii-policy
docpipe-maestro ask "Summarize travel limits." --local-only --collection hr_policy
docpipe-maestro ask "Summarize travel limits." --where source=travel_policy.md --collection hr_policy
docpipe-maestro ask "Summarize employee contacts." --model openai/gpt-4o-mini --pii-policy ./pii-policy.json --collection hr_policy
```
*(By default, if a provider hits `--max-tokens` and sets `finish_reason == "length"`, `ask` prints a `Warning:` on `stderr`; pass `--fail-on-truncation` to fail non-zero instead.)*

### 8. Offline Retrieval Evaluation (`docpipe-maestro eval`)

Benchmark Phase 2 retrieval (`Precision@K`, `Recall@K`, `MRR`) against a labeled JSON fixture without mutating persisted manifests or indexes (see [Retrieval Evaluation Baselines](docs/eval-baseline.md)):

```bash
# Run curated 24-query policy benchmark with hybrid lexical + dense search
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --searcher fts5_hybrid

# Benchmark with local cross-encoder reranking enabled
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --searcher fts5_hybrid --rerank

# Customize cutoff values, filter subset by metadata, inspect raw child chunks, or emit JSON
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --where profile=POLICY --k 1 3 5 --no-expand-parent --json
```

*(Target collection resolves in order: explicit `--collection` > fixture `collection` > `docpipe.toml` `collection` > `"default"`.)*

---

## Python Library API Guide

See the complete [Python API Reference](docs/api-reference.md) for every signature, parameter default, config constant, and exception class.

### Phase 1: Ingestion & Lifecycle Workflow

```python
from pathlib import Path
from docpipe_maestro import PiiBlockedError, PiiPolicy, ingest, reindex, remove, status, sync

pii_policy: PiiPolicy = {
    "types": {
        "EMAIL_ADDRESS": "tokenize",
        "IBAN_CODE": "redact",
        "US_SSN": "block",
    },
    "token_salt_env": "DOCPIPE_PII_SALT",  # omit to use process-random fallback salt + warning
    "egress": "redact",
}

# 1. Ingest a document or directory with pre-chunking PII safeguards
try:
    result = ingest(
        Path("./policies"),
        profile="POLICY",  # "DEFAULT", "POLICY", or custom ChunkingProfile
        collection="handbook",
        persist_dir=".docpipe",
        embed_model="ollama/nomic-embed-text",
        pii_policy=pii_policy,
        local_only=True,
    )
    print(
        f"Ingested {len(result.ingested)} files "
        f"({result.chunks_written} children, {result.parents_written} parents)."
    )
except PiiBlockedError as exc:
    # Exception reports source & blocked entity type(s) without printing raw PII values
    print(f"Aborted cleanly on {exc.source}: {exc.blocked_entities}")

# 2. Synchronize folder after changes (adds, edits, deletions)
sync_result = sync(
    Path("./policies"),
    profile="POLICY",
    collection="handbook",
)
print(
    f"Added: {len(sync_result.added)}, Updated: {len(sync_result.updated)}, "
    f"Skipped: {len(sync_result.skipped)}, Removed: {len(sync_result.removed)}"
)

# 3. Check collection metadata and tracked documents
coll_status = status(collection="handbook")
print(
    f"Locked embed model: {coll_status.embed_model}, "
    f"Tracked files: {coll_status.total_sources}, "
    f"Children: {coll_status.total_children}"
)

# 4. Remove a document from the index
removed = remove("outdated_policy.md", collection="handbook")

# 5. Reindex collection with a new embedding model
reindexed_count = reindex(
    embed_model="openai/text-embedding-3-small",
    collection="handbook",
)
```

### Phase 2: Semantic Retrieval & Grounded RAG Workflow

```python
from docpipe_maestro import answer, build_chat_messages, retrieve

question = "What is the travel meal reimbursement rate?"

# 1. Retrieve deduplicated parent sections (pass searcher="fts5_hybrid" for dense + BM25 RRF)
retrieved_sections = retrieve(
    question=question,
    collection="handbook",
    top_k=5,
    threshold=0.35,  # Normalized similarity / RRF score [0.0, 1.0]
    expand_parent=True,  # Expand child matches to full parent section
    searcher="default",  # "default" or "fts5_hybrid"
    rerank=False,  # Set True (with `[rerank]` extra installed) for cross-encoder rescoring
    where={"profile": "POLICY"},  # Optional chunk/manifest metadata filter
)

for section in retrieved_sections:
    print(f"[{section.score:.2f}] {section.metadata['source']} - {section.metadata['section']}")
    print(section.text)

# 2. Build role-separated ChatML messages (Pure function, zero network calls)
#    (build_prompt() also remains available for single-string prompts.)
messages = build_chat_messages(question=question, sections=retrieved_sections)

# 3. Generate a grounded answer (pass stream=True for synchronous Iterator[str] chunks,
#    or use answer_detailed() for finish_reason/token telemetry)
response = answer(
    prompt=messages,
    model="ollama/llama3.2",
    temperature=0.0,  # Omit (None) for reasoning models that reject temperature
    max_tokens=512,  # Omit (None) or pass model_kwargs={"reasoning_effort": "low"} for reasoning models
    timeout=60.0,
    has_context=bool(retrieved_sections),  # Short-circuits with refusal if context is empty
)
print("Answer:\n", response)

# Incremental streaming with identical preflight/egress safety before the first pull:
for delta in answer(
    messages, model="ollama/llama3.2", has_context=bool(retrieved_sections), stream=True
):
    print(delta, end="", flush=True)
print()
```

### Retrieval Evaluation (`docpipe_maestro.eval`)

```python
from docpipe_maestro.eval import evaluate, normalize_k_values, precision_at_k, recall_at_k

report = evaluate(
    "tests/fixtures/eval_policy_corpus.json",
    collection="eval_policy",
    persist_dir=".docpipe",
    k_values=normalize_k_values([10, 3, 3, 1]),  # -> (1, 3, 10)
    searcher="fts5_hybrid",
    where={"profile": "POLICY"},  # Optional chunk/manifest metadata filter
)
print(report["aggregate"]["mrr"], report["aggregate"]["recall_at_k"])
```

### Opt-In Structured Logging (`docpipe_maestro.log`)

Library calls stay silent by default (`NullHandler` only). Attach a formatter on `"docpipe_maestro"` when integrating with your application logs:

```python
import logging
from docpipe_maestro.log import JsonFormatter, correlation_id, get_logger

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger = get_logger()
logger.addHandler(handler)
logger.setLevel(logging.INFO)

with correlation_id("req-42"):
    # All internal stage.end telemetry events share correlation_id="req-42"
    ...
```

### Core Data Shapes & Exceptions

Exported from `docpipe_maestro` (`docpipe_maestro.types`):

- `Chunk`: Base unit (`id`, `text`, `parent_id`, `metadata: ChunkMetadata`).
- `EmbeddedChunk`: `Chunk` with dense embedding (`vector: list[float]`).
- `RetrievedChunk`: `EmbeddedChunk` with normalized similarity or RRF score (`score: float` in `[0.0, 1.0]`).
- `GenerationResult`: Detailed LLM completion result (`text`, `model`, `finish_reason`, `truncated`, `prompt_tokens`, `completion_tokens`, `reasoning_tokens`, `total_tokens`, `elapsed_seconds`).
- `IngestResult`: Summary (`collection`, `embed_model`, `ingested`, `skipped`, `chunks_written`, `parents_written`).
- `SyncResult`: Summary (`collection`, `embed_model`, `added`, `updated`, `skipped`, `removed`, `dry_run`, `estimated_children`, `estimated_parents`, `estimated_tokens`, `estimated_cost_usd`).
- `CollectionStatus`: Manifest overview (`collection`, `embed_model`, `embed_dim`, `total_sources`, `total_children`, `total_parents`, `sources`).
- `SourceRecord`: Per-file manifest tracking record (`source`, `content_hash`, `doc_version`, `size`, `mtime`, `ingested_at`, `embed_model`, `profile`, `chunk_ids`, `parent_ids`).
- Exceptions: `DocpipeError` (base) with `UnsupportedFormatError`, `MissingExtraError`, `ModelMismatchError`, `MissingCredentialError`, `PiiBlockedError`, and `RegistryError`.
- Privacy & Extensibility: `PiiPolicy` (`TypedDict`) passed programmatically to `ingest()`, `sync()`, `answer()`, and `answer_detailed()`, plus lazy root `@register_pii_recognizer` (`docpipe_maestro.types.PiiRecognizer`) for custom entity patterns.

---

## Security, Privacy & Grounding Notes

- **Trusted local corpus assumption**: Like most local ingestion tools, `docpipe-maestro ingest` processes files inside the target directory you specify. Only ingest trusted documents and avoid pointing `ingest`/`sync` at directories containing symlinks to sensitive files or untrusted HTML/PDF inputs.
- **Credentials & transport security**: API keys are never accepted in `docpipe.toml`; remote custom endpoints require `https://` (`http://` is permitted only for loopback hosts), and error handlers scrub regex key patterns as well as loaded environment secrets (`from None`) so credentials never leak in exceptions.
- **`--local-only` & outbound PII egress guard**: Pass `--local-only` (`local_only=True`) to reject non-local models and remote `--api-base` URLs prior to network requests. When calling remote LLMs with `pii_policy`, `guard_egress()` scrubs or blocks outbound prompt strings and role-separated ChatML messages before dispatch, and Phase 1 writes zero-raw-PII counts/metadata to `<persist_dir>/manifests/<collection>.audit.jsonl` under `0o700`/`0o600` permissions.
- **Opt-in structured telemetry (`docpipe_maestro.log` / `--log-json`)**: Library calls stay silent by default (`NullHandler` only on `"docpipe_maestro"`, root logger and `stdout` untouched). CLI flags `--log-json` and `--log-level {DEBUG,INFO,WARNING,ERROR}` (CLI-only; ignored in `docpipe.toml`) stream correlated `stage.end` events exclusively to `stderr` with traceback suppression and zero raw documents, questions, prompts, answers, PII, or credentials at `INFO` or `DEBUG`.
- **Honest scope & accuracy limitations**: Built-in pattern/checksum recognizers (`EMAIL_ADDRESS`, `IBAN_CODE`, `CREDIT_CARD`, `US_SSN`, `IP_ADDRESS`, `PHONE_NUMBER`, `ES_DNI`) and optional `[pii]` Presidio NER heuristics can experience both false positives and false negatives depending on language, formatting, custom identifiers, and OCR/PDF text noise. These safeguards reduce risk but are technical controls—not a guarantee of exhaustive redaction or regulatory certification. See [SECURITY.md](SECURITY.md) for full threat-model notes.
- **Reduce hallucinations with `--threshold`**: Set a similarity `--threshold` (e.g. `0.35`–`0.6` depending on embedding model) on `retrieve()` / `docpipe-maestro ask` so off-topic questions return zero matches and short-circuit deterministically (`"Not found in the provided context."`).
- **Protect your `.docpipe/` store on shared hosts**: Embedded ChromaDB files store plaintext document sections unless scrubbed beforehand; keep `persist_dir` on owner-restricted paths (`chmod 700`) and excluded from version control.

---

## Safety & Verification

`docpipe-maestro` includes two mandatory automated release gates:
1. **Reverse Safety Test Gate** (`tests/test_reverse_safety.py`, `@pytest.mark.reverse`): Enforces that every clause containing an exception marker (`except`, `unless`, `not entitled`, `subject to`, `provided that`) always retrieves a parent context containing the exception alongside the general rule.
2. **PII Safety Test Gate** (`tests/test_pii_safety.py`, `@pytest.mark.pii`): Verifies offline on synthetic fixtures that blocked PII halts ingestion before database commit, redacted/tokenized spans never persist to chunks or outbound remote payloads, `--local-only` blocks remote endpoints pre-network, and `.audit.jsonl` logs zero raw PII.

Run the test suite and quality gates:

```bash
# Run linters, security rules, and formatting checks
uv run ruff check .
uv run ruff check --select S src/
uv run ruff format --check .

# Run static type checking (strict mode)
uv run mypy src tests

# Run all unit, integration, and safety tests with branch coverage offline
LITELLM_LOCAL_MODEL_COST_MAP=true uv run pytest -v --cov

# Verify reverse and PII safety gates individually
uv run pytest -m reverse -v
uv run pytest -m pii -v
uv run pytest -m integration -v
```

---

## Repository Structure

```text
docpipe-maestro/
├── .github/                 # CI (ci.yml), release publishing (release.yml) & issue/PR templates
├── .specify/
│   └── memory/
│       └── constitution.md  # Non-negotiable architectural & safety principles
├── src/docpipe_maestro/
│   ├── __init__.py          # Public API exports & lazy pipeline/answer imports
│   ├── __main__.py          # Enables `python -m docpipe_maestro` CLI execution
│   ├── py.typed             # PEP 561 typed package marker
│   ├── cli.py               # `docpipe-maestro` CLI + TOML config loader
│   ├── config.py            # Default constants & immutable chunking profiles
│   ├── _crossrefs.py        # Shared phase-neutral cross-reference matching & excerpt helpers
│   ├── _filters.py          # Shared phase-neutral metadata filter validation, manifest resolution & SQL compiler
│   ├── _pii.py              # Shared phase-neutral PII recognizers, HMAC tokenizer & lazy NER
│   ├── _registry.py         # Lazy Strategy + Registry core (`register_*` decorators)
│   ├── _resolve.py          # Format-to-parser key resolution helper
│   ├── _stages.py           # Top-level stage re-exports and protocols
│   ├── log.py               # Opt-in structured JSON/text telemetry, NullHandler logger & ContextVar correlation IDs
│   ├── report.py            # Offline single-file HTML Document Integrity & Chunk Audit Report exporter
│   ├── embedder/            # Batched LiteLLM embedding adapter, locality checks & registry
│   ├── manifest.py          # Atomic sha256/mtime manifest, model lock & .audit.jsonl logger
│   ├── store/               # ChromaDB vector store (`chroma.py`) & SQLite FTS5 (`lexical.py`)
│   ├── types.py             # Dataclasses, TypedDicts & DocpipeError hierarchy
│   ├── ingest/              # Phase 1: intake/, clean/, enricher/ (`pii`, `crossrefs`), chunker/, pipeline
│   ├── query/               # Phase 2: searcher/, reranker/, egress.py, retriever, prompts, answer
│   └── eval/                # Phase 2: offline metrics (`metrics.py`) & read-only runner (`harness.py`)
├── tests/                   # Unit, integration, security, reverse & PII-safety suites
├── documents/               # Synthetic sample policy corpus for local demos
├── docs/                    # Architecture, API, CLI, baselines, dev guide & roadmap
├── specs/                   # Feature specification, plan & contracts
├── .env.example             # Template for local/cloud provider environment variables
├── .gitattributes           # Cross-OS LF line-ending enforcement (keeps binary PDFs/DOCX untouched)
├── .pre-commit-config.yaml  # Commitizen Conventional Commits & Ruff hook configuration
├── CHANGELOG.md             # Release history
├── CONTRIBUTING.md          # Contribution workflow & quality gate checklist
├── LICENSE                  # MIT License
├── PRODUCT.md               # Product specification & requirements baseline
├── README.md                # Project overview, quickstart & guides
├── SECURITY.md              # Vulnerability reporting & privacy model
└── pyproject.toml           # PEP 621 / 735 packaging, ruff, mypy & pytest config
```

---

## Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) and the [Development & Testing Guide](docs/development.md) before opening a pull request. In brief:

1. Follow **Conventional Commits** (`feat:`, `fix:`, `docs:`, `test:`, `chore:`) verified with `uv run cz check`.
2. Pass all CI quality gates (`ruff format --check .`, `ruff check .`, `ruff check --select S src/`, `mypy src tests`, `pytest -m reverse -v`, `pytest -m pii -v`, and `pytest -v --cov` with branch coverage `>= 80%`).
3. Keep all Phase 1 and Phase 2 library modules free of config/policy-file reading and external UI/server concerns.

For security vulnerabilities, see [SECURITY.md](SECURITY.md).

---

## Documentation

- **[Documentation Index](docs/README.md)** — Full guide overview
- **[Architecture & Internals](docs/architecture.md)** — Two-phase pipeline flow, intake, crossref resolution, chunking, and transactional staging
- **[Python API Reference](docs/api-reference.md)** — Signatures, profiles, data shapes, and error handling
- **[CLI Reference](docs/cli-reference.md)** — `docpipe-maestro` commands, TOML config, outputs, and troubleshooting
- **[Retrieval Evaluation Baselines](docs/eval-baseline.md)** — Offline dense vs `fts5_hybrid` comparison metrics ([JSON](docs/eval-baseline.json))
- **[Development & Testing Guide](docs/development.md)** — Setup, quality gates, offline test doubles, and Ollama live runs
- **[Future Features Backlog](docs/future-features.md)** — Implementation status and post-v0.2 roadmap
- **[Changelog](CHANGELOG.md)** — Release history

---

## License

This project is licensed under the [MIT License](LICENSE).
