Metadata-Version: 2.4
Name: distlk
Version: 0.1.0rc1
Summary: Compiles source documents into a structured, navigable Knowledge Package.
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: docling-core==2.90.0
Requires-Dist: docling==2.118.0
Requires-Dist: google-genai==2.16.0
Requires-Dist: openai==2.53.0
Requires-Dist: pydantic-settings==2.14.2
Requires-Dist: pydantic==2.13.4
Requires-Dist: rapidfuzz==3.14.5
Requires-Dist: ruamel-yaml==0.19.1
Requires-Dist: tenacity==9.1.4
Requires-Dist: typer==0.26.8
Provides-Extra: dev
Requires-Dist: pytest-asyncio==1.4.0; extra == 'dev'
Requires-Dist: pytest-cov==7.0.0; extra == 'dev'
Requires-Dist: pytest==9.1.1; extra == 'dev'
Requires-Dist: ruff==0.16.1; extra == 'dev'
Description-Content-Type: text/markdown

# distl

**Compile a PDF or DOCX document into individually addressable, versioned Knowledge Objects — so a coding agent can fetch one fact instead of reading the whole file.**

[![CI](https://github.com/ruriazz/distl/actions/workflows/ci.yml/badge.svg)](https://github.com/ruriazz/distl/actions/workflows/ci.yml)
![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)
![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue)
![Version: 0.1.0rc1](https://img.shields.io/badge/version-0.1.0rc1-orange)
![Tests: 214 passing](https://img.shields.io/badge/tests-214%20passing-brightgreen)
![Coverage: 94%](https://img.shields.io/badge/coverage-94%25-brightgreen)

> **Read this before adopting it:** this README documents what is actually implemented and
> measured today — not a roadmap, not a pitch. The [Not built yet](#not-built-yet) section is as
> important as the feature list above it.

## Project status

First release `v0.1.0rc1` (pre-release) published to PyPI as `distlk`; single maintainer, actively developed. The core pipeline (ingest →
chunk → extract → validate → write, plus incremental update) is built and tested; what's not
built yet is listed explicitly below, not left for you to discover the hard way.

214 tests, 94% statement coverage (`pytest --cov=src/distl`) — the lowest-covered module is
`cli.py` at 54%, since most of its logic is thin argument wiring already exercised indirectly
through the pipeline tests, not untested business logic. CI (`.github/workflows/ci.yml`) runs
lint + the full test suite with coverage on every push/PR to `master`; pushing a `v*` tag runs
`.github/workflows/release.yml`, which gates on the same checks, builds the sdist + wheel,
publishes to PyPI as `distlk`, and files a GitHub Release.

---

## Table of contents

- [Project status](#project-status)
- [The problem](#the-problem)
- [What distl does](#what-distl-does)
- [Quick start](#quick-start)
- [How it works](#how-it-works)
- [CLI reference](#cli-reference)
- [Configuration](#configuration)
- [Output layout](#output-layout)
- [Design decisions, briefly](#design-decisions-briefly)
- [Measured, not claimed](#measured-not-claimed)
- [Not built yet](#not-built-yet)
- [Development](#development)
- [Contributing](#contributing)
- [License](#license)

---

## The problem

Coding agents are usually handed one of two bad options when a task depends on a spec, an RFC, or
a requirements doc: dump the entire PDF into context (expensive, unfocused, drowns the one
relevant paragraph in fifty irrelevant ones), or rely on the agent to grep/skim it (unreliable,
easy to miss a page-12 constraint that contradicts a page-40 assumption).

`distl` exists to produce a third option: compile the document once into small, individually
addressable, versioned units — a requirement, a business rule, a workflow step — each one citing
exactly where it came from, so an agent (or a human) can pull the three relevant objects instead
of the whole 140-page PDF.

## What distl does

- **Ingests PDF/DOCX** via [Docling](https://github.com/docling-project/docling), preserving
  reading order, heading hierarchy, page/section anchors, and table content (rendered as
  Markdown). Pictures are detected and their image bytes preserved on disk regardless.
- **Reads pictures (opt-in)**: with `DISTL_VISION_ENABLED=true`, a picture's content (a chart,
  diagram, or screenshot) is read by a vision-capable LLM (cheap tier first, escalating to a
  stronger model only on low self-reported confidence) and turned into real Knowledge Object(s) —
  the same structural validation and id-dedup guarantees every text-derived object gets, tagged
  `extraction_method: vision-picture-extract-v1`. Default **off**; a picture that isn't read (or
  fails to be) still produces today's visible gap, image preserved on disk regardless.
- **Chunks** at heading boundaries under a configurable token budget — a chunk never silently
  merges two unrelated sections into one citation.
- **Extracts** Knowledge Objects per chunk via an LLM router (Gemini primary, OpenRouter
  fallback) with retry/backoff, automatic provider fallback, and a hard per-run cost ceiling that
  halts cleanly and resumes later without re-processing anything already written.
- **Validates structurally** before anything is written — a malformed object becomes a visible
  gap in `validation/gaps.yaml`, never a silent drop and never a file on disk.
- **Updates incrementally**: `distl update` re-diffs a changed source, re-extracts only the
  sections that actually changed, bumps versions only on real content changes, fuzzy-matches
  objects across revisions, and routes anything conflicting with a human edit to a merge-conflict
  file instead of overwriting it.
- **Governs tags** through a reviewed, curated vocabulary instead of letting the LLM invent an
  ever-growing, inconsistent tag soup.
- **Shards large manifests** automatically once a package crosses a token budget, so the index
  stays cheap to read even as a package grows.

### A concrete example

*(Illustrative — the schema and pipeline behavior below are real; this specific document is a
constructed example, not a real customer document.)*

Given a source paragraph like:

> **2.3 Password Reset.** A user who has forgotten their password may request a reset link via
> the "Forgot password" control on the sign-in screen. The link expires after 30 minutes. After a
> successful reset, all other active sessions for that account are terminated.

`distl compile` produces `requirements/REQ-7.md`:

```markdown
---
id: REQ-7
title: Password reset link expires after 30 minutes
status: active
description: >
  A user who forgot their password can request a reset link from the "Forgot password"
  control on the sign-in screen. The link expires 30 minutes after issuance. On a
  successful reset, all other active sessions for the account are terminated.
confidence: 0.93
source_location:
  document: requirements.pdf
  page: 4
  section: "2.3 Password Reset"
extraction_method: chunk-summarize-v1
version: 1
extraction_provider: google-ai-studio
extraction_model: gemini-3.5-flash
tags: [authentication, session-management]
---
```

An agent asking "what happens to other sessions on password reset?" reads one ~15-line file with
an exact page citation — not a 140-page PDF.

## Quick start

Requires Python 3.12+.

```bash
pip install distlk                        # published release
pip install -e ".[dev]"                   # or: development install from this repo

cp .env.example .env   # then fill in DISTL_GEMINI_API_KEY at minimum
distl compile requirements.pdf --output ./knowledge-package
```

Sample output (illustrative — object ids/titles/timings vary per document and run; the line
format itself is real, from `cli.py`'s progress reporter):

```
Parsing requirements.pdf...
[1/6] extracted REQ-1 'User accounts require email verification' (confidence 0.95, page 1) via google-ai-studio, 2.1s
[2/6] extracted REQ-2 'Password minimum length is 12 characters' (confidence 0.91, page 2) via google-ai-studio, 1.8s
...
Compiled 6 knowledge object(s) to knowledge-package
```

### A more realistic workflow

Compiling once is the easy case. What actually happens across a document's lifetime
(illustrative counts/comments below — the commands and CLI output format are real, the specific
numbers are made up for the example):

```bash
# Initial compile hits the cost ceiling partway through a large document
distl compile spec-v1.pdf --output ./kp
# Halted: fallback cost ceiling exceeded -- see validation/cost-ceiling-alert.md

# Resume later without re-processing what's already written
distl compile ./kp --resume

# The source doc gets revised -- re-extract only what changed
distl update ./kp spec-v2.pdf
# 3 updated, 1 new, 1 deprecated, 0 merge conflict(s)

# A reviewer had hand-edited one object; the update flagged a real conflict
distl merge-conflicts ./kp

# New tags the LLM proposed need a human decision before they're trusted
distl tags review ./kp

# Spot-check a sample for citation accuracy before trusting the package
distl audit ./kp --sample 0.2
```

More, including a script for the sequence above and a programmatic (non-CLI) query example: see
[`examples/`](examples/).

## How it works

```mermaid
flowchart LR
    A["PDF / DOCX"] --> B["Ingest<br/>(Docling)"]
    B --> C["Chunk<br/>(heading-boundary)"]
    C --> D["Extract<br/>(LLM router)"]
    D -->|"rate limit / timeout / malformed"| E["Fallback provider"]
    D --> F["Validate<br/>(structural)"]
    E --> F
    F -->|fail| G["validation/gaps.yaml"]
    F -->|pass| H["Knowledge Object .md"]
    H --> I["manifest.yaml"]
```

Two guardrails run the whole time, not shown above for clarity: a **cost ceiling** that halts the
run cleanly (resumable) if it would be exceeded, and a **validation gate** that means nothing
malformed ever reaches disk.

Full module map (what's real vs. an empty placeholder) and the design rationale behind the
trickier decisions: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).

## CLI reference

| Command | What it does |
|---|---|
| `distl compile <file> --output <dir>` | Compile a source document into a new Knowledge Package. |
| `distl compile <dir> --resume [--source <path>]` | Resume a run halted by the cost ceiling; `--source` re-points at a moved source document. |
| `distl update <dir> <file>` | Re-compile only the sections of `<dir>` that changed in `<file>`. |
| `distl audit <dir> --sample <fraction>` | Interactively sample objects and judge traceability to source; reports a hallucination rate. |
| `distl merge-conflicts <dir>` | List objects where a human edit and a new extraction disagreed. |
| `distl tags review <dir>` | Accept or reject LLM-proposed tags not yet in the controlled vocabulary. |
| `distl baseline run <file> [-c N ...]` | Run at one or more concurrency levels, recording cost/latency/token metrics. |
| `distl baseline report --runs-dir <dir>` | Aggregate baseline runs into a Markdown table. |

## Configuration

Environment variables, prefixed `DISTL_`, loaded via Pydantic Settings (`src/distl/config.py`):

| Variable | Default | Meaning |
|---|---|---|
| `DISTL_GEMINI_API_KEY` | *(none)* | Primary extraction provider. |
| `DISTL_OPENROUTER_API_KEY` | *(none)* | Fallback provider. Unset = fallback skipped, recorded as a gap, never an error. |
| `DISTL_GEMINI_MODEL` | `gemini-3.5-flash` | Primary model. |
| `DISTL_OPENROUTER_FALLBACK_MODEL` | `openai/gpt-5-mini` | Fallback model. |
| `DISTL_CHUNK_SIZE_TOKENS` | `2000` | Token budget per chunk before a heading-group splits further. |
| `DISTL_MAX_CONCURRENCY` | `5` | Concurrent chunk extractions per run. |
| `DISTL_FALLBACK_COST_CEILING_USD` | `5.00` | Hard per-run spend ceiling; halts cleanly (resumable) if it would be exceeded. |
| `DISTL_RATE_LIMIT_RETRY_ATTEMPTS` | `2` | Retries on a rate-limit error before falling back. |
| `DISTL_TIMEOUT_RETRY_ATTEMPTS` | `1` | Retries on a timeout before falling back. |
| `DISTL_MANIFEST_TOKEN_BUDGET` | `8000` | Manifest size threshold before it shards into `manifest.d/`. |
| `DISTL_ID_MATCH_SIMILARITY_THRESHOLD` | `0.80` | Fuzzy title-similarity cutoff for `distl update`'s object matcher. Not yet tuned against real collision data. |
| `DISTL_CONTROLLED_TAG_VOCABULARY_PATH` | `./tags.yaml` | Relative paths resolve against the package directory, not the current working directory. |
| `DISTL_VISION_ENABLED` | `false` | Opt-in: read picture content via a vision LLM instead of just preserving the image. |
| `DISTL_VISION_MODEL_TIER1` | `qwen/qwen3-vl-8b-instruct` | Cheap/fast vision model, tried first. |
| `DISTL_VISION_MODEL_TIER2` | `qwen/qwen3-vl-235b-a22b-instruct` | Escalation target when tier 1's self-reported confidence is below the threshold. |
| `DISTL_VISION_CONFIDENCE_THRESHOLD` | `0.70` | Below this, tier 1's result is discarded and tier 2 is tried instead. |
| `DISTL_VISION_ESTIMATED_COST_TIER1_USD` | `0.01` | Pre-call cost-ceiling check estimate for tier 1 (real cost is reconciled after the call). |
| `DISTL_VISION_ESTIMATED_COST_TIER2_USD` | `0.05` | Same, for tier 2. |
| `DISTL_VISION_RETRY_ATTEMPTS` | `2` | Mirrors `DISTL_RATE_LIMIT_RETRY_ATTEMPTS`: retries a vision call on transient rates/timeout/malformed-response failures (default 2 = one retry). |

## Output layout

```
knowledge-package/
├── manifest.yaml                 # index: objects, tags, categories, gaps (or a sharded pointer)
├── manifest.d/                   # per-category shards, only once manifest.yaml crosses its budget
├── requirements/REQ-1.md
├── business-rules/BR-1.md
├── ...                           # one file per object, grouped by id-prefix category
├── source/                       # preserved images from pictures not (yet) turned into an object
                                   # -- not extracted at all, or vision-extraction failed/skipped
└── validation/
    ├── gaps.yaml                 # every skipped/failed/rejected item — never silent
    ├── pending-tags.yaml         # tags awaiting `distl tags review`
    ├── possible-id-matches.yaml  # below-threshold fuzzy matches for human review
    ├── merge-conflicts/*.md      # objects where a human edit and a new extraction disagree
    └── cost-ceiling-alert.md     # present only if the last run halted on the cost ceiling
```

Every object's frontmatter: `id`, `title`, `status`, `priority`, `description`, `actors`,
`dependencies`, `business_rules`, `constraints`, `related_apis`, `confidence`,
`source_location` (document/page/section), `extraction_method`, `version`, `superseded_by`,
`last_updated_by`, `last_updated_at`, `extraction_provider`, `extraction_model`, `tags`.

## Design decisions, briefly

**Why a manifest instead of a vector index?** Real documents compiled so far produce tens of
objects (highest observed: 62, on an 18-page document); sharding has been tested against a
synthetic 450-object package, not a real one that large yet. At these counts, a flat/sharded YAML
index that an agent reads directly is simpler, cheaper, and fully inspectable — no embedding
pipeline, no vector store to run. A retrieval-based index is a deliberate future option, not
built speculatively ahead of real usage reaching the counts that would justify it.

**Why two LLM providers?** Any single provider fails sometimes (rate limits, timeouts, content
filters). The router retries, then falls back to a second provider, rather than failing the whole
chunk — with a hard cost ceiling so a failure storm can't produce an unbounded bill.

**Why does a chunk never merge across headings?** It used to. A chunk spanning two sections could
only cite one (page, section) anchor for every object extracted from it — measured, this produced
citations off by as much as 85 pages on a real document. Forcing a chunk boundary at every heading
means every citation is exactly the section it came from, at the cost of more (smaller) LLM calls.
Traded deliberately: this project treats a wrong citation as a correctness bug, not a cost
optimization opportunity.

**Why only structural validation so far?** Semantic/cross-reference/duplicate/conflict/LLM-review
validation each need real failure-rate evidence to justify the added latency and cost before
being turned on — turning all of them on speculatively would mean tuning against guesses instead
of data. See [Not built yet](#not-built-yet).

## Measured, not claimed

Numbers below come from real API calls against real documents (18–143 pages), instrumented by
`src/distl/telemetry/metrics.py` — not projections.

| Metric | Result |
|---|---|
| Cost per document | $0.0016 – $0.1165 |
| Source-location accuracy (post chunking-boundary fix) | 27/28 objects (96%) independently verified correct, on a real compile |
| Source-location accuracy (pre-fix baseline) | 4/15 objects (27%) fully correct, across 3 documents |
| Table-heavy document coverage | 1 usable object → 41 objects, after table content stopped being dropped during ingestion |

**Not measured**: whether a compiled package actually makes a coding agent faster or more
accurate than handing it the raw document. No consumption-side trial has been run yet — this is
the single biggest open question before recommending `distl` for real use, not just a nice-to-have
follow-up metric.

## Not built yet

Real gaps, not modesty:

- **Validation stages 3–7** (semantic, cross-reference, duplicate, conflict, LLM review) — only
  stages 1–2 (chunk well-formedness, structural schema) run today.
- **Annotation semantics** (e.g. strikethrough → deprecated, highlight → important) — not read.
- **Diff-aware picture re-extraction on `distl update`** — vision extraction (opt-in, see [What
  distl does](#what-distl-does)) re-attempts every picture on every update call, not just ones in
  changed sections; text chunks already get this filtering, pictures don't yet.
- **Source types beyond PDF/DOCX** — Confluence, Jira, Notion, OpenAPI, DB-schema, and Figma
  adapters exist only as empty placeholder files.
- **Multi-document packages** — a package is scoped to exactly one source document.
- **Consumption-side validation** — see the callout above.

None of the above is scheduled on a fixed timeline — each gets built once real usage produces
evidence it's worth the added complexity, not built speculatively ahead of that. Track progress
or propose one of these via [issues](https://github.com/ruriazz/distl/issues).

## Development

```bash
pip install -e ".[dev]"
pytest
```

214 tests pass at the time of writing (94% statement coverage), run against real fixture documents
with fully mocked/stub provider calls — no network access or API key needed. The "Measured, not
claimed" numbers above come from separate, manual runs against real providers, not from the test
suite.

## Contributing

Issues and PRs are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for dev setup, test/lint
commands, and commit/PR conventions — read it before starting anything non-trivial, since this is
a single-maintainer project and scope is easiest to agree on before code is written, not after.
Participation is governed by [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). Found a security issue
rather than a bug? Report it per [SECURITY.md](SECURITY.md), not as a public issue.

## License

[MIT](LICENSE) — see [CHANGELOG.md](CHANGELOG.md) for what's actually shipped so far.
