Metadata-Version: 2.4
Name: phantom-documents
Version: 0.1.0
Summary: Local-first document anonymization using local LLMs via Ollama
Author-email: Esteban Ponce de León <pdlr.estebanfelipe@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/estebanpdl/phantom-docs
Project-URL: Repository, https://github.com/estebanpdl/phantom-docs
Project-URL: Issues, https://github.com/estebanpdl/phantom-docs/issues
Keywords: anonymization,pii,privacy,local-llm,ollama,redaction,document-processing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Legal Industry
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Natural Language :: Spanish
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: ollama>=0.4
Requires-Dist: pydantic>=2.7
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: httpx>=0.27
Requires-Dist: typer>=0.12
Requires-Dist: tqdm>=4.66
Requires-Dist: pdfplumber>=0.11
Requires-Dist: pymupdf>=1.24
Requires-Dist: jinja2>=3.1
Requires-Dist: python-docx>=1.1
Provides-Extra: ui
Requires-Dist: streamlit>=1.49; extra == "ui"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# phantom-docs

[![tests](https://github.com/estebanpdl/phantom-docs/actions/workflows/tests.yml/badge.svg)](https://github.com/estebanpdl/phantom-docs/actions/workflows/tests.yml)
[![python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)](https://github.com/estebanpdl/phantom-docs)
[![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)

Local-first document anonymization using local LLMs. Detects and replaces personally identifiable information — names, organizations, dates, emails, phone numbers, addresses, national ID numbers — in PDF, DOCX, Markdown and plain-text documents.

**Everything runs on your machine.** No API keys, no cloud calls, no telemetry. Powered by [Ollama](https://ollama.com).

---

## Why

Journalists, researchers and legal teams routinely need to share documents that contain information about real people. Commercial redaction tools send those documents to a server — which is precisely the thing you cannot do when the document is a leaked dossier, a medical record, or a source's testimony.

phantom-docs does the work locally. The document is read from disk, processed by a model running on your own hardware, and written back to disk. Nothing crosses the network.

---

## What it does

- **Reads real documents.** PDF (including scanned pages via OCR), DOCX, Markdown and plain text.
- **Understands context.** An LLM does the detection, not a regex list — so it catches names and addresses that pattern matching misses, and leaves ordinary text alone.
- **Stays consistent across a whole document.** The same person gets the same placeholder from page 1 to page 300, even when the text calls them "John Smith" in one place and "J. Smith" in another — 87.5% of the time on the evaluation corpus, with the residue erring towards keeping people separate rather than merging them wrongly.
- **Handles long documents.** Text that exceeds the model's context window is split into overlapping windows, processed independently, then reassembled seamlessly.
- **Converts PDFs to clean Markdown.** Heading hierarchy from bookmarks or a visual table of contents, tables as GFM tables, reading order preserved — useful on its own, and a much better input for the anonymizer than raw PDF text.
- **Lets you narrow the scope.** Anonymize only names and emails, or everything.
- **Processes directories.** Point it at a folder; one bad file never stops the run.
- **Is as reproducible as local inference allows.** Temperature 0 and a fixed seed remove sampling randomness. Runs are not bit-identical, though — GPU floating-point reductions vary, and one flipped token cascades — so expect small run-to-run differences.

---

## Entity types

`PERSON` · `ORG` · `DATE` · `EMAIL` · `PHONE` · `ADDRESS` · `SSN` · `OTHER`

Replacements are sequential per type and globally consistent across the document: `[PERSON_1]`, `[EMAIL_2]`, and so on.

Variants of the same entity collapse into one placeholder, so `Maren Dolvic`, `M. Dolvic` and `Dolvic` all read as the same person. Matching is lexical first — shorter forms, initials, dropped honorifics, folded accents — with embedding similarity as the fallback for cases that need semantics. Where a form fits two different people, the tool keeps them separate rather than guessing: telling a reader that two individuals are one is a worse error than leaving a link unmade.

---

## How it works

```
PDF / DOCX / MD / TXT
  │
  ├── Ingestion            format dispatcher → text
  │     └── PDF pipeline   structure → extraction → OCR → Markdown assembly
  │
  └── Anonymization pipeline
        ├── Chunker        overlapping word windows
        ├── Engine         local LLM → structured PII detection + replacement
        ├── Deduplicator   embeddings cluster entity variants across chunks
        ├── Assembler      global renumbering + overlap trimming
        └── Writer         <name>_anonymized.md
```

For PDFs specifically: heading structure is read from bookmarks (falling back to visual TOC detection, then font-size heuristics); text and tables are extracted with `pdfplumber`; pages are classified as `text`, `image` or `mixed` by image coverage; image and mixed pages are rendered to PNG and passed to a vision model for OCR.

For DOCX: body paragraphs and tables are read **in document order**, so a table stays between the paragraphs that surround it. Headings become Markdown headings, tables become GFM tables (the same renderer the PDF pipeline uses), merged cells are collapsed rather than repeated, and nested tables are followed. Headers, footers and review comments are extracted too — letterheads, classification banners and comment threads routinely carry names and contact details. Footnotes, endnotes and text-box content cannot currently be reached; when a document contains them you get an explicit warning rather than silence.

---

## PDF processing

Three libraries do three different jobs. Two are installed automatically; the third is optional and most people will never need it.

| Library | Job | How it is installed |
|---|---|---|
| **pdfplumber** | Extracts text and tables, with the character positions and font sizes used to detect headings | pip, automatic |
| **PyMuPDF** (`fitz`) | Reads the bookmark outline, and renders pages to PNG for OCR | pip, automatic |
| **Poppler** (`pdftoppm`) | Alternative page renderer | System package, **optional** |

### Why two PDF libraries

They are good at different things, and the pipeline uses each for what it does well.

`pdfplumber` gives per-character coordinates and font metrics, which is what makes heading detection possible at all — a heading is recognised by being larger than the page's median font size, not by any marker in the file. It also finds table structure. What it does not do is render.

`PyMuPDF` reads the embedded bookmark outline, which is a far more reliable source of document structure than font sizes when a PDF has one, and it rasterises pages. Both are pure pip installs with no system dependencies, which matters for a tool whose users may not have admin rights on the machine holding their sensitive documents.

### When rendering happens at all

A page is only rasterised in two situations:

- **OCR** — a page classified as `image` or `mixed` is rendered and sent to a vision model
- **`convert --validate`** — renders page 1 so you can eyeball whether extraction is sane

If you leave `PHANTOM_DOCS_VISION_MODEL` empty and never pass `--validate`, no page is ever rendered and the renderer choice is irrelevant.

### Poppler is opt-in and usually unnecessary

PyMuPDF is the default and needs nothing installed. Poppler exists as an alternative for people who already have it, or who hit a PDF that PyMuPDF renders badly — malformed files do occasionally favour one rasteriser over the other.

```bash
brew install poppler          # macOS
apt install poppler-utils     # Debian / Ubuntu

phantom-docs convert scan.pdf --renderer poppler
```

Selecting a renderer that is not installed fails immediately with an explanation rather than silently falling back, so a run cannot quietly use a different tool than you asked for:

```
poppler is not installed — pdftoppm not found on PATH.
Install with: brew install poppler  (macOS)
```

### Graceful degradation

Every optional part of the PDF path has a fallback, so a missing piece narrows the result rather than stopping it:

| Missing | Consequence |
|---|---|
| Bookmark outline | Falls back to scanning early pages for a visual table of contents |
| Visual TOC too | Falls back to font-size ranking for heading levels |
| Vision model | OCR skipped; image-only pages contribute no text — **and a warning says so** |
| Poppler | PyMuPDF is used instead (unless you explicitly asked for Poppler) |

The vision-model case is the one to watch. A scanned PDF with no OCR configured produces a document that anonymizes cleanly because there was nothing to anonymize — which is why extraction emits a warning when a page yields neither text nor OCR, and why the UI shows it rather than logging it.

---

## Requirements

**Python 3.11+** and a running [Ollama](https://ollama.com) server.

Start Ollama:

```bash
OLLAMA_FLASH_ATTENTION=0 OLLAMA_NO_CLOUD=1 ollama serve
```

> **Gemma models and Flash Attention.** Gemma's hybrid attention architecture (sliding-window + global layers) is incompatible with Ollama's Flash Attention implementation — with the default `OLLAMA_FLASH_ATTENTION=1` the server hangs on prompts longer than roughly 500 tokens. Set `OLLAMA_FLASH_ATTENTION=0`. This does not apply if you configure a different generation model.

Install:

```bash
pip install phantom-documents            # core
pip install "phantom-documents[ui]"      # + web interface
```

Or from a clone, for development:

```bash
git clone https://github.com/estebanpdl/phantom-docs
cd phantom-docs
pip install -e ".[dev,ui]"
```

Pull the models:

```bash
phantom-docs models pull
```

or manually:

```bash
ollama pull gemma4:26b            # generation — PII detection
ollama pull embeddinggemma        # embeddings — entity deduplication
ollama pull deepseek-ocr:latest   # vision — OCR for scanned pages (optional)
```

Any Ollama model works; these are the defaults. A smaller generation model trades accuracy for speed. Leave `PHANTOM_DOCS_VISION_MODEL` empty to disable OCR entirely.

**Nothing else is required.** PDF handling needs no system packages — see [PDF processing](#pdf-processing) if you want to know what is doing the work, or if you would rather use Poppler.

---

## Web interface

For interactive use, or for anyone who would rather not work in a terminal:

```bash
pip install "phantom-documents[ui]"
phantom-docs app
```

Opens on `http://localhost:8501`. `--port` moves it; `--no-browser` skips opening a window. A `--host` given globally reaches the interface too, so `phantom-docs --host http://gpu-box:11434 app` points it at another Ollama server.

Upload a document, review what was extracted, anonymize, download. The UI calls the same library the CLI does — no logic lives in the interface layer, so the two cannot drift apart in behaviour.

It surfaces the same integrity signals as the CLI, and in the same order of prominence: a document with passages that could not be anonymized shows a red banner naming them **before** the output, before the entity table, before anything congratulatory. Audit files sit behind a deliberate expander with the de-anonymization warning attached, rather than beside the download button.

---

## Command line

```bash
phantom-docs --help
```

Global options: `--host` (Ollama URL), `--verbose` / `-v`, `--quiet` / `-q`.

| Command | Purpose |
|---|---|
| `check` | Verify Ollama and print the active configuration |
| `anonymize` | Anonymize a single document |
| `batch` | Anonymize every supported document in a directory |
| `convert` | PDF to Markdown, without anonymizing |
| `models` | List or pull the models phantom-docs uses |
| `app` | Launch the web interface |

### `check`

Verify Ollama is reachable and print the active configuration.

```bash
phantom-docs check
```

Exits `1` when the health check fails, so it can gate a shell script.

### `anonymize`

Anonymize a single document.

```bash
phantom-docs anonymize dossier.pdf -o output/
phantom-docs anonymize notes.md --entity-types PERSON --entity-types EMAIL
phantom-docs anonymize report.docx --max-words 400 --overlap 60
```

| Flag | Effect |
|---|---|
| `-o`, `--output-dir` | Destination directory (default `output/`) |
| `--entity-types` | Restrict detection to these types; repeat per type |
| `--model` | Override the generation model |
| `--embed-model` | Override the deduplication embedding model |
| `--max-words` | Words per chunk (default 500) |
| `--overlap` | Words shared between consecutive chunks (default 50) |
| `--no-ocr` | Skip OCR for PDF input |
| `--renderer` | `pymupdf` or `poppler` — only matters when a PDF needs OCR |
| `--no-warmup` | Skip model pre-loading (useful for a short one-off run) |
| `--keep-warm` | Leave models loaded after the run instead of releasing them |
| `--entity-map PATH` | Write a JSON record of every replacement — **a de-anonymization key**, see below |
| `--report PATH` | Write a human-readable summary for checking the result |

Prints the number of chunks and a per-type entity breakdown, and writes `<name>_anonymized.md`.

### `batch`

Anonymize every supported document in a directory.

```bash
phantom-docs batch ./documents -o output/ --recursive
```

Accepts the same model, chunking, entity-type, `--renderer` and `--no-warmup` flags as `anonymize`. `--audit-dir DIR` writes a per-file entity map and report into `DIR`. Per-file status, entity count and elapsed time are printed as the run proceeds. A failing file is reported and skipped without aborting the rest; the command exits `1` if any file failed. With `--recursive`, the subdirectory layout is mirrored in the output directory.

### `convert`

Convert a PDF to Markdown without anonymizing it.

```bash
phantom-docs convert contract.pdf -o output/
phantom-docs convert scan.pdf --no-ocr
phantom-docs convert scan.pdf --renderer poppler --validate
```

| Flag | Effect |
|---|---|
| `-o`, `--output-dir` | Destination directory (default `output/`) |
| `--no-ocr` | Skip OCR even when a vision model is configured |
| `--renderer` | `pymupdf` or `poppler` |
| `--validate` | Render page 1 to `tmp/pdfs/` for a visual spot-check |

Reports pages by type, word count, tables found and pages OCR'd. Warns when a page yields neither extractable text nor OCR output.

### `models`

```bash
phantom-docs models list   # installed models, tagged [generation] [embedding] [vision]
phantom-docs models pull   # pull everything phantom-docs is configured to use
```

`list` warns about configured models that are not installed; `pull` streams progress and continues past individual failures.

---

## Output integrity

A local model can occasionally return output phantom-docs cannot parse. When that happens the tool never drops the passage from your document — but it also never pretends the passage was anonymized.

Every chunk of a document ends in one of three states, and the two that are not clean are always reported, on stderr, even under `--quiet`:

| State | What it means | Exit code |
|---|---|---|
| **Clean** | The model returned valid structured output. Text is anonymized, entities recorded. | `0` |
| **Degraded** | Parsing failed but the anonymized text was recovered. The text is anonymized, but the entities behind it could not be recorded, so that chunk's placeholders are namespaced separately — see below. | `0` |
| **Not anonymized** | Parsing and recovery both failed. **The original text was written to the output and still contains PII.** | `1` |

A failed chunk is reported with its position so you can find the passage:

```
  NOT ANONYMIZED — 1 chunk(s) could not be processed: chunk 3 (words 1000–1500)
  The text of those passages was written to the output unchanged and still
  contains PII. Review before sharing.
```

### Namespaced placeholders

Placeholders are normally global: `[PERSON_1]` means the same person everywhere in the document. A degraded chunk breaks that guarantee, because the entities behind its text were lost and cannot be matched against the rest of the document.

Rather than emit a placeholder that *looks* global but isn't, phantom-docs qualifies those with their chunk number:

```
[PERSON_1]        global — the same person throughout the document
[PERSON_C3_1]     local to chunk 3 — identity not established
```

A namespaced placeholder can never be confused with a global one. The cost is a real loss of information: `[PERSON_C3_1]` might be the same person as `[PERSON_1]`, and the tool cannot tell you. That is the honest outcome — the alternative is a document that silently claims two different people are the same, which for a document about people is worse than an admitted gap.

`batch` counts degraded and review-needed files separately from hard errors, lists the affected filenames at the end, and exits `1` if any file needs review. One bad file never stops the run.

**Scripting:** treat a non-zero exit from `anonymize` or `batch` as "do not publish this output yet".

---

## Concurrency

**There isn't any, deliberately.** Chunks are processed one at a time.

We measured whether concurrency was worth building, dispatching the same 6-chunk workload through a thread pool at three levels:

| Workers | Wall time | Speedup | Peak VRAM |
|---|---:|---:|---:|
| **1** | **225.5s** | **1.00×** | 15.4 GB |
| 2 | 229.0s | 0.98× | 15.4 GB |
| 3 | 228.0s | 0.99× | 15.4 GB |

Concurrency was marginally *slower*, with memory unchanged to the byte. Unchanged memory is the tell: had Ollama opened parallel slots, each would have allocated its own KV cache. It served all three client threads through one slot and queued the rest, so the only measurable effect was queueing overhead.

The cause is server-side policy. `OLLAMA_NUM_PARALLEL` was unset, so Ollama chose — and for an 18 GB model it chose 1. Client-side threads cannot overrule that.

Two caveats: this is one model on one machine, and a smaller model where Ollama *would* open parallel slots could behave differently. If you want to explore it, raise `OLLAMA_NUM_PARALLEL` on the server and measure — but note that each slot costs its own KV cache, on hardware where running out of GPU memory is already a live failure mode.

### Model lifecycle

Models are pre-loaded before a run and released after it, so a finished command doesn't hold ~15 GB indefinitely. `--no-warmup` skips the pre-load; `--keep-warm` skips the release, which is worth it for back-to-back commands.

---

## Checking the result

`--report` writes a Markdown summary built for a person about to publish something. It leads with whether the output is safe to share, then lists every replacement so you can check the list against what you know is in the document — an entity missing from it was not detected.

`--entity-map` writes the same information as JSON, plus run metadata and the integrity data, for tooling.

### These files are de-anonymization keys

Both artifacts name the values that were removed and pair them with the placeholders that replaced them. Anyone holding one and the anonymized document can reconstruct the original.

That is unavoidable rather than an oversight: answering "did it catch my source's name?" requires the name to be in the file. A record that omitted the originals would be safe and useless.

So they are never written unless you ask. When you do, phantom-docs warns, the JSON leads with a `_WARNING` key explaining itself to anyone who opens it cold, and writing one into the same directory as the anonymized output is flagged separately — shipping the key alongside the lock is the easy mistake to make in a hurry.

```bash
phantom-docs anonymize dossier.pdf -o output/ --entity-map keys/dossier.json
```

**phantom-docs does not de-anonymize.** There is no `restore` command and there will not be one. Exporting a map serves the audit need at the moment of use; shipping a reverser would make round-tripping a supported workflow, and supported workflows end up with keys stored next to outputs permanently, by design. Anyone with the map can of course reverse it by hand — the file says so — but the tool will not encourage it.

---

## Python API

The CLI is a thin wrapper — every capability is available as a library call.

### Anonymize a document

```python
from pathlib import Path

from phantom_docs.config import cfg
from phantom_docs.ingestion import ingest
from phantom_docs.ingestion.writer import write_anonymized
from phantom_docs.pipeline.runner import run_pipeline

source = Path("dossier.pdf")

text = ingest(source, cfg)                 # PDF/DOCX/MD/TXT → text
result = run_pipeline(text, cfg, max_words=500, overlap=50)

result.chunk_count       # 7
result.entities          # canonical, globally renumbered PIIEntity list
result.replacement_map   # {"Ana Rivera": "[PERSON_1]", "A. Rivera": "[PERSON_1]", ...}

write_anonymized(source, result.anonymized_text, Path("output/"))
```

### Anonymize a string

```python
from phantom_docs.anonymizer import anonymize
from phantom_docs.config import cfg

result = anonymize("Contact Ana Rivera at ana@example.org.", cfg)

result.anonymized_text   # "Contact [PERSON_1] at [EMAIL_1]."
result.entities          # [PIIEntity(original='Ana Rivera', entity_type='PERSON', ...), ...]
```

Restrict the types considered:

```python
anonymize(text, cfg, entity_types=["PERSON", "EMAIL"])
```

### Convert a PDF

```python
from pathlib import Path

from phantom_docs.config import cfg
from phantom_docs.pdf import apply_ocr, assemble, extract, extract_structure
from phantom_docs.pdf.writer import write_markdown

pdf = Path("contract.pdf")

doc = extract(pdf)                     # text blocks, tables, page classification
doc = apply_ocr(doc, cfg)              # no-op when no vision model is configured
structure = extract_structure(pdf)     # bookmarks or visual TOC → heading levels
markdown = assemble(doc, structure)

out_path, stats = write_markdown(doc, markdown, Path("output/"))
print(stats)
# contract.pdf — 12 pages (9 text, 2 image, 1 mixed) — 3,847 words — 4 tables — 2 OCR pages
```

---

## Configuration

Every setting is read from an environment variable or a `.env` file at the project root.

| Variable | Default | Description |
|---|---|---|
| `OLLAMA_HOST` | `http://localhost:11434` | Ollama server URL (or use `--host`) |
| `OLLAMA_KEEP_ALIVE` | `-1` | How long models stay resident during a run |
| `PHANTOM_DOCS_MODEL` | `gemma4:26b` | Generation model used for PII detection |
| `PHANTOM_DOCS_EMBED_MODEL` | `embeddinggemma:latest` | Embedding model used for deduplication |
| `PHANTOM_DOCS_VISION_MODEL` | `deepseek-ocr:latest` | Vision model for OCR; empty string disables OCR |
| `PHANTOM_DOCS_RENDERER` | `pymupdf` | PDF page renderer: `pymupdf` or `poppler` |
| `PHANTOM_DOCS_NUM_CTX` | `4096` | Context window in tokens |
| `PHANTOM_DOCS_SEED` | `42` | Sampling seed. Removes sampling randomness; does not make runs bit-identical |
| `PHANTOM_DOCS_STRUCTURED_OUTPUT` | `schema` | `schema` grammar-constrains output so a missing field is impossible; `json` asks only for valid JSON |
| `PHANTOM_DOCS_THINK` | `false` | Reasoning tokens before answering. **Leave off** — see below |
| `PHANTOM_DOCS_BACKEND` | `ollama` | Model server. Only `ollama` is implemented; selecting another fails loudly rather than silently falling back |

Sampling temperature is fixed at `0.0` and is intentionally not configurable: anonymization must be deterministic.

**Set these on the Ollama server, not on phantom-docs.** They configure the server process, which a client cannot influence:

| Variable | Why |
|---|---|
| `OLLAMA_FLASH_ATTENTION=0` | Required for Gemma models — see above |
| `OLLAMA_NO_CLOUD=1` | Disables telemetry. phantom-docs cannot enforce this; check it yourself if local-only operation matters to you. |
| `OLLAMA_NUM_PARALLEL` | Controls how many requests the server runs in parallel. Left unset, Ollama chooses — for an 18 GB model on Apple Silicon it chooses 1. See [Concurrency](#concurrency). |

> **`PHANTOM_DOCS_THINK` is off by default, and you should measure before turning it on.** With `gemma4:26b` on the evaluation corpus, reasoning tokens caused **every chunk to fail** both validation and recovery, so the original text was returned untouched — a 100% leak rate, every entity readable in the output. It failed identically under both `json` and `schema`, so output constraint was not the cause.
>
> The flag remains available because that result is one model's behaviour, not a law: a model with a well-behaved reasoning phase may do fine. Enabling it logs a warning. Run the accuracy benchmark on your own model before trusting it in either direction.

Example `.env`:

```
OLLAMA_HOST=http://localhost:11434
PHANTOM_DOCS_MODEL=gemma4:26b
PHANTOM_DOCS_EMBED_MODEL=embeddinggemma:latest
PHANTOM_DOCS_VISION_MODEL=deepseek-ocr:latest
```

---

## Package layout

```
phantom_docs/
├── config.py             Configuration — environment variables and defaults
├── health.py             Ollama connectivity guard
├── cli.py                Command-line interface
├── markdown.py           Shared table rendering (PDF and DOCX)
├── prompts/              Jinja2 system-prompt templates
├── backends/
│   ├── protocol.py       BackendProtocol — what the pipeline needs of a server
│   ├── ollama_backend.py Ollama implementation, bound to the configured host
│   └── factory.py        Backend selection
├── pdf/
│   ├── extractor.py      pdfplumber → text blocks, tables, page classification
│   ├── assembler.py      extracted document → Markdown
│   ├── renderer.py       PDF page → PNG (PyMuPDF or poppler)
│   ├── ocr.py            PNG → vision model → recovered text
│   ├── structure.py      bookmarks / visual TOC → heading hierarchy
│   └── writer.py         Markdown → file, with extraction statistics
├── anonymizer/
│   ├── models.py         PIIEntity, AnonymizationResult
│   ├── prompts.py        system prompt construction
│   ├── engine.py         LLM call, validation, fallback recovery
│   └── deduplicator.py   embedding-based entity clustering
├── ingestion/
│   ├── dispatcher.py     format detection and routing
│   └── writer.py         anonymized output writer
└── pipeline/
    ├── chunker.py        word-based overlap chunker
    ├── processor.py      per-chunk anonymization
    ├── assembler.py      global renumbering and reassembly
    └── runner.py         end-to-end pipeline
```

Each module has a single responsibility; the CLI and any future front-end call the same library functions rather than duplicating logic.

---

## Design principles

- **Local only.** No data leaves the machine. `OLLAMA_NO_CLOUD=1` is required.
- **Deterministic.** Fixed temperature and seed on every call — the same input yields the same output.
- **Never lose a document, never fake success.** Every optional component degrades gracefully: no bookmarks falls back to font-size heuristics, no vision model skips OCR, no Poppler uses PyMuPDF, and a malformed model response falls back to regex recovery and then to the untouched original. Nothing is dropped — and any passage that did not anonymize cleanly is reported and reflected in the exit code. See [Output integrity](#output-integrity).
- **Structured output.** Model responses are validated against Pydantic models before use.
- **Backend-agnostic core.** The anonymization logic is independent of the model server, so alternative local runtimes can be supported without rewriting the pipeline.

---

## Testing

```bash
pytest -q
```

The unit suite mocks all model calls, so it runs without a live Ollama instance or a GPU.

**Integration tests** talk to a real server and are opt-in:

```bash
pytest --integration
```

They skip rather than fail when Ollama is unreachable or the configured model isn't installed — a red suite for a missing optional dependency teaches people to ignore red suites. Point them at a smaller model with `PHANTOM_DOCS_TEST_MODEL=llama3.2:3b`.

**Accuracy benchmark** — a labelled corpus of 20 synthetic documents (English and Spanish, five document types, 248 labelled entities) with ground truth generated by construction rather than hand-annotated:

```bash
python tests/fixtures/eval/harness.py            # full run
python tests/fixtures/eval/harness.py --limit 3  # quick check
```

It reports **leak rate** as the headline number — how many labelled entities are still readable in the output — with detection recall, type accuracy and variant unification alongside. Leak rate is what matters: an entity can appear in the entity list and still survive in the text, and a regex-recovered chunk anonymizes text while reporting no entities at all. The full report lands in `assessments/accuracy-baseline.md`.

Regenerate the corpus with `python tests/fixtures/eval/generate.py` (deterministic — same seed, byte-identical documents).

> **`temperature=0` does not make runs identical.** It makes sampling deterministic given identical logits, but GPU logits are not bit-identical between runs — floating-point addition is not associative, so parallel reductions can order differently, and a single flipped argmax cascades through the rest of the output. Use `--repeat N` and compare ranges; a difference of a couple of points between single runs is not evidence of anything.

> **Long benchmark runs on Apple Silicon:** we have repeatedly seen Ollama exhaust GPU memory partway through a run with a large model, after which every request fails with `Insufficient Memory (kIOGPUCommandBufferCallbackErrorOutOfMemory)` — including after unloading every model. It appears that GPU allocations are not fully released across repeated model loads. Restarting the Ollama server sometimes clears it; a reboot reliably does. Partial results survive: `--resume` continues from the last completed document rather than starting over.

### Measured results

`gemma4:26b` · `embeddinggemma` · `num_ctx` 4096 · seed 42 · schema-constrained output · thinking off · 20 documents, 7,598 words, 248 labelled entities, averaged over 2 runs with no degraded or failed chunks.

| | Overall | English | Spanish |
|---|---:|---:|---:|
| **Leak rate** — labelled entities still readable | **0.4%** | 0.0% | 0.8% |
| Detection recall | 98.8% | 98.4% | 99.2% |
| Entity-type accuracy | 95.1% | 96.7% | 93.5% |
| Variant unification | 87.5% | 85.0% | 90.0% |

Throughput was roughly 41s per document (~9 words/second) on the author's machine.

**Read these numbers with four caveats.**

*They describe one model.* Everything above is `gemma4:26b`. A different model may score very differently — the same benchmark showed `format` and reasoning-token settings swinging leak rate from 0.4% to 100% on this model alone. If you run something else, run the harness on it.

*They describe synthetic documents.* The corpus is generated, which makes the labels exact but also makes the prose more regular than real-world material. Treat the figures as a floor for comparison between configurations, not as a prediction for your documents.

*Small differences are not evidence.* `temperature=0` fixes sampling, not the underlying logits, so repeated runs vary. Leak rate and recall each moved 2.0 points between the two runs above. Treat anything under a few points as noise, and use `--repeat` when comparing.

*Variant unification is the weakest number, and it is the one to watch.* 87.5% of people appearing under several forms — `Maren Dolvic`, `M. Dolvic`, `Dolvic` — collapse to a single placeholder. The remainder carry more than one, so a reader cannot always tell that two passages refer to the same individual. Some of that residue is deliberate: where a bare surname fits two different people, the deduplicator declines to guess rather than merging them, because asserting that two people are one is a worse error than leaving them separate.

---

## Caveats

Anonymization quality depends on the model you run — see [Measured results](#measured-results) for what that means in practice, and note that those figures are for one model on synthetic documents. Detection is very good but not perfect: unusual name forms, PII embedded in images, and heavily domain-specific identifiers can be missed, and the same person may currently receive more than one placeholder when their name appears in several forms.

**Review the output before publishing or sharing a document.** phantom-docs is a strong first pass, not a substitute for human verification when the stakes are high.

---

## License

MIT
