Metadata-Version: 2.4
Name: token-distiller
Version: 0.1.0
Summary: Distills PDFs, photos, and repos into token-efficient context for LLM agents. Local OCR reads embedded diagrams, not just text layers; content-hash caching collapses re-reads; BM25 retrieval bounds huge documents. Nothing is discarded — anything shortened expands back in full. 33x on a 25-page PDF, 97% on pytest output.
Project-URL: Homepage, https://github.com/priyanchen/token-distiller
Project-URL: Repository, https://github.com/priyanchen/token-distiller
Project-URL: Issues, https://github.com/priyanchen/token-distiller/issues
Author: Sri PriYa N. Chen
License-Expression: PolyForm-Noncommercial-1.0.0
License-File: LICENSE
Keywords: anthropic,claude,context,llm,ocr,pdf,rag,retrieval,tokens
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: General
Requires-Python: >=3.11
Requires-Dist: anthropic>=0.40
Requires-Dist: pathspec>=0.12
Requires-Dist: pdf2image>=1.17
Requires-Dist: pdfplumber>=0.11
Requires-Dist: pillow-heif>=0.15
Requires-Dist: pillow>=10.0
Requires-Dist: pypdf>=4.0
Requires-Dist: pytesseract>=0.3.10
Requires-Dist: rank-bm25>=0.2
Provides-Extra: dev
Requires-Dist: pytest-mock; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: rag-semantic
Requires-Dist: voyageai>=0.3; extra == 'rag-semantic'
Description-Content-Type: text/markdown

# Token Distiller

Distills PDFs and photos into token-efficient text before they enter an LLM's context, packs repos Repomix-style, indexes ingested content for retrieval instead of raw dumping, and tracks session activity mode to bias what a context audit flags.

Pure Python. No shell scripts, no Node/TypeScript.

## Setup

### From PyPI

```bash
brew install poppler   # required by pdf2image for PDF page rasterization
pip install token-distiller
```

### As a Claude Code plugin

```
/plugin marketplace add priyanchen/token-distiller
/plugin install token-distiller@token-distiller
```

This wires the hook configuration only. Claude Code has no mechanism to run setup
commands after installing a plugin, so it can't provision a Python environment for you —
separately run `pip install token-distiller` so the `distill` binary the hook invokes is
on `PATH`. Skipping that step doesn't break anything: the hook fails open, so `Read` on a
PDF passes through unchanged rather than erroring. It just means nothing gets distilled
until the binary is actually installed.

### From source

```bash
brew install poppler   # required by pdf2image for PDF page rasterization
python3 -m venv .venv
source .venv/bin/activate
pip install .
```

Use a regular install, not `pip install -e .`. On this machine the editable install's
`_editable_impl_token_distiller.pth` was silently ignored by `site.py` (the file was
readable and its contents correct, and a byte-identical copy under a different filename
*was* honored — root cause unresolved), leaving `token_distiller` unimportable. A regular
install copies the package into `site-packages` and avoids the `.pth` indirection
entirely. Re-run `pip install .` after editing source.

Optional: set **`TOKEN_DISTILLER_ANTHROPIC_API_KEY`** to enable vision-model fallback for
figures and pages OCR can't read. Prefer that name over plain `ANTHROPIC_API_KEY` — a host
agent (Claude Code included) may also read `ANTHROPIC_API_KEY` and switch from subscription
authentication to per-token API billing if it finds one. The scoped name is checked first,
so this tool gets a key without changing anyone else's auth.

Set `VOYAGE_API_KEY` and `pip install ".[rag-semantic]"` to enable semantic (embedding)
retrieval on top of the default BM25 keyword index.

## CLI

Placeholders are uppercase; substitute your own path, directory, or question.

```bash
distill file PATH             # distill one PDF/photo
distill scan DIR              # batch distill a directory
distill repo DIR              # Repomix-style repo pack
distill index DIR             # build a retrieval index
distill query "QUESTION"      # query the index
distill mode                  # current session activity mode
distill audit PATH            # CLAUDE.md/MEMORY.md structural audit (defaults to .)
distill report                # cumulative token/savings report
distill expand HANDLE         # full distilled text for a handle (--list to browse)
distill compress              # compress verbose command output read from stdin
distill install-hook          # wire the PreToolUse Read-interception hook into a project
```

Every command that distills a PDF accepts `--no-figures` to skip reading embedded figures,
and `--no-vision` to stay on local OCR only. `distill file` additionally accepts
`--accurate-tokens`, which calls Anthropic's `count_tokens` endpoint to report the real
tokenizer count for the distilled output next to the chars/4 estimate. It's opt-in because
it needs an API key (`TOKEN_DISTILLER_ANTHROPIC_API_KEY` or `ANTHROPIC_API_KEY`) and a
network round trip — the call itself is free and doesn't count against message-creation
rate limits, but the estimate is what every other command uses by default.

## Compressing command output

Verbose CLI output is the one context cost the document pipeline doesn't touch. Pipe it in:

```bash
pytest -q | distill compress --stats
git status | distill compress
npm install | distill compress
```

Measured on this repo's own output: a 169-test `pytest` run goes **289 → 8 tokens (97%)**,
keeping the failure list, the first assertion detail, and the summary line while dropping
the wall of dots. Plain `git status` goes **153 → 58 tokens (62%)**, grouped by state.

Two properties worth knowing:

- **It never inflates.** `git status --porcelain` is already denser than any per-state
  summary of it, so when compression would produce more text than it consumed, the original
  is returned unchanged.
- **It never executes anything.** `distill compress` reads stdin and writes stdout. The
  alternative — a hook that rewrites your Bash command to route it through a wrapper —
  means building shell strings out of model-supplied input, which is exactly where command
  injection lives. Piping output that you already ran has no such surface. Automatic
  interception is deliberately not implemented for that reason.

## Verified in a live session

The `Read` interception was checked against a real Claude Code session, not only against
synthetic hook payloads. A one-page PDF holding a canary string was read through the hook:
the session reported `2,572 → 38 tokens` and the model quoted the canary correctly,
confirming the substituted text is what actually reaches it. A second run asking for one
specific page of a four-page PDF passed straight through to the native ranged read with no
hook note, confirming a page range is never answered with whole-document content.

## How it avoids losing anything

Every distillation is stored whole, keyed by a SHA-256 of the file's bytes, before any
shortening happens. Anything the hook shortens carries a handle, and `distill expand
<handle>` returns the complete text. Concretely:

- **Re-reads collapse.** Reading the same unchanged file twice in a session returns a
  one-line pointer the second time instead of the whole document again (7,634 → 287
  chars measured). Edit the file and the hash changes, so it is re-distilled in full —
  a stale cache can never be served.
- **Repeated page boilerplate is restated once.** A line must appear on ≥80% of pages to
  qualify, so a running copyright footer (25/25 pages) collapses while a structural
  marker like `Example:` (15/25) is left alone. Collapsed lines are listed at the top of
  the output.
- **Large documents defer rather than truncate.** Past `TOKEN_DISTILLER_LARGE_DOC_TOKENS`
  (default 8000) the hook returns a head plus retrieval instructions. Nothing is
  discarded — `distill expand` or `distill index` + `distill query` reach the rest.
- **Embedded figures are read, not skipped.** Native-text extraction sees a page's text
  layer only, so a diagram sitting beside that text would otherwise go unread. Each
  embedded figure is cropped out by its bounding box and put through the same OCR →
  vision chain used for scanned pages, then written into the output labelled
  `[figure N on page M]`. Cropping to the figure matters: the surrounding prose is
  already captured losslessly, so including it would pay vision tokens to re-read text
  we already have. Hairline rules and background strips are skipped via
  `TOKEN_DISTILLER_FIGURE_MIN_SIDE_PT` (default 48pt). `--no-figures` turns it off.
- **A weak OCR pass is retried on a preprocessed copy.** Figures cropped from a PDF are
  often below the ~300 DPI Tesseract expects and sit on a tinted panel, which is exactly
  when raw OCR returns nothing. A second attempt greyscales, stretches contrast, upscales
  small crops, and binarizes with an Otsu threshold. It is a retry rather than the default
  because binarizing can destroy anti-aliased text that read fine raw, so the preprocessed
  pass has to win on word count and confidence to be used. Measured on a real 765-page
  book: of 13 figures that raw OCR could not read at all, **10 were recovered** — one at
  confidence 96, transcribing `Market Research / Competitive Analysis / SWOT Analysis /
  Goal Setting / Resource Allocation`. That book ends at 60 of 63 figures read, with no
  API key.
- **Figures that still can't be read are flagged, never dropped silently.** A purely
  graphical diagram with no legible labels yields nothing from OCR, and without
  `ANTHROPIC_API_KEY` there's no vision fallback to describe it. Those pages stay in
  `pages_with_uncaptured_images()` and surface as one compact note — never one line per
  page — in `distill file`, `--json`, and the hook-read response. Set `ANTHROPIC_API_KEY`
  to close the remainder.

## What leaves your machine

With no API keys set, nothing does. OCR runs locally through Tesseract and retrieval runs
locally through BM25, so the default configuration makes **zero network calls**.

Two features are opt-in, and they send different things to different companies:

| Enabled by | Goes to | What is sent |
|---|---|---|
| `TOKEN_DISTILLER_ANTHROPIC_API_KEY` (or `ANTHROPIC_API_KEY`) | Anthropic | A PNG of a **single cropped figure**, plus a fixed prompt. No source code, no file paths, no surrounding page text. |
| `VOYAGE_API_KEY` + `pip install ".[rag-semantic]"` | Voyage AI | **Chunk text** from whatever you indexed. If you indexed a repo pack, that includes your source code. |

The Voyage path is the one to think hardest about — it is a separate company under separate
terms, and it is the only path that can transmit code. It is off unless you both install the
extra and set the key; BM25 retrieval works without it.

On the Anthropic path, [their commercial terms](https://www.anthropic.com/legal/commercial-terms)
state that the customer "retains all rights to its Inputs", "owns its Outputs", that
Anthropic "disclaims any rights it receives to the Customer Content", and that Anthropic
"may not train models on Customer Content from Services". Read them yourself rather than
relying on this summary; this is not legal advice.

Turn figure reading off entirely with `--no-figures` or
`TOKEN_DISTILLER_DESCRIBE_FIGURES=0`, and it will never reach for the vision model.

Toggle any of it off: `TOKEN_DISTILLER_CACHE=0`, `TOKEN_DISTILLER_REREAD_COLLAPSE=0`,
`TOKEN_DISTILLER_BOILERPLATE=0`.

## A note on the numbers

`raw_tokens_est` models what the **host** pays to ingest the file, not what its text
alone would cost. Reading a PDF natively renders each page to an image and bills those
pixels on top of the text, so a 25-page text PDF costs ~60,000 tokens to read raw but
~1,800 distilled (33x). Scoring it as text-only would have reported a meaningless 1.0x.

Both figures are estimates (chars/4 for text, Anthropic's published pixel formula for
images) — good enough to compare methods, not exact. `distill file --accurate-tokens`
swaps the distilled-side estimate for a real count from Anthropic's tokenizer, to check
the estimate rather than trust it.

That 33x describes a *sparse* page, where a fixed per-page rendering cost dominates a
small amount of actual text — it is not a document-size-independent multiplier. A
densely-written page compresses by far less through this mechanism alone, because the
distilled side scales with real content: measured on a 765-page, prose-dense book, the
whole-document ratio was 4.77x (1,567,889 raw → 328,589 distilled), not 33x. That is
expected, not a regression — a page's text cannot be compressed below its own token
count by an extraction step that isn't lossy.

For a document that size, the number that actually matters is not the whole-document
ratio anyway. 328,589 distilled tokens is well past `TOKEN_DISTILLER_LARGE_DOC_TOKENS`
(default 8000), so the hook's large-document deferral fires: a live session reading that
765-page file through the hook receives a head plus a `distill index` / `distill query`
pointer, measured at ~1,675 tokens — roughly **940x** against the 1,567,889 raw cost.
(The payload embeds the file's absolute path, so the exact token count shifts a little
with where the file lives.)
For large documents, the deferral-and-retrieval path is where the real savings come
from; the raw-vs-native-text mechanism the 33x figure describes matters most for
documents small enough to be read in full.

## License

**PolyForm Noncommercial 1.0.0.** Free for personal, private, and noncommercial use —
no license purchase needed. Commercial use requires contacting Sri PriYa N. Chen
(p.chen@NeoclassicalPopArt.com) for a commercial license.
