Metadata-Version: 2.5
Name: kompend
Version: 2026.10.4
Summary: Compile a document corpus into an interlinked, OKF-conformant markdown wiki
Author-email: Jan Reimes <jan@reimes.de>
License: MIT
License-File: LICENSE
Keywords: acp,document-processing,knowledge-base,knowledge-graph,knowledge-management,llm,mcp,okf,open-knowledge-format,pdf-parsing,rag,semantic-search,vector-search,wiki
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database :: Front-Ends
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.13
Requires-Dist: aenum>=3.1.17
Requires-Dist: binaryornot>=0.6.0
Requires-Dist: charset-normalizer>=3.5.1
Requires-Dist: chonkie>=1.0.0
Requires-Dist: crawl4ai[all]>=0.5.0
Requires-Dist: datasketch>=1.10.0
Requires-Dist: fastapi>=0.110.0
Requires-Dist: fastmcp-slim[server]>=4.0.0
Requires-Dist: ison-py>=1.0.2
Requires-Dist: jinja2>=3.0.0
Requires-Dist: json-repair[schema]>=0.61.2
Requires-Dist: lazy-loader
Requires-Dist: logfire[system-metrics]>=4.37.0
Requires-Dist: mail-parser>=4.4.0
Requires-Dist: models-dev>=1.0.398
Requires-Dist: ollama>=0.6.2
Requires-Dist: py7zr>=1.1.3
Requires-Dist: pydantic-ai-harness[acp]>=0.7.0
Requires-Dist: pydantic-ai-rlm>=0.1.2
Requires-Dist: pydantic-ai[sentence-transformers]>=2.8.0
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pydantic>=2.13.4
Requires-Dist: python-slugify[unidecode]>=8.0.4
Requires-Dist: pyyaml>=6.0
Requires-Dist: rapidfuzz>=3.14.6
Requires-Dist: rich>=13.0.0
Requires-Dist: rumdl>=0.2.9
Requires-Dist: summarization-pydantic-ai>=0.1.10
Requires-Dist: tenacity>=9.0.0
Requires-Dist: tomli-w>=1.2.0
Requires-Dist: toon-format>=0.9.0b1
Requires-Dist: typer>=0.12.0
Requires-Dist: unrar2-cffi>=0.5.0
Requires-Dist: uvicorn>=0.29.0
Requires-Dist: wenmode>=0.14.1
Requires-Dist: zvec>=0.4.0
Provides-Extra: mineru
Requires-Dist: parsecraft[mineru]>=2026.10.2; extra == 'mineru'
Provides-Extra: otel
Requires-Dist: opentelemetry-exporter-otlp>=1.28.0; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.28.0; extra == 'otel'
Provides-Extra: parsecraft
Requires-Dist: parsecraft[docling,liteparse,pandoc,web]>=2026.10.1; extra == 'parsecraft'
Provides-Extra: systemone
Requires-Dist: parsecraft[auto]>=2026.10.1; extra == 'systemone'
Description-Content-Type: text/markdown

# kompend

<p align="center">
  <img src="./assets/readme/hero.svg" width="100%"
       alt="kompend — Knowledge Organizer for eXtended documents: compile scattered documents into an interlinked, queryable wiki with any LLM">
</p>

kompend refuses to pick a side in the LLM-wiki space. Existing tools either
demand a frontier-model subscription **and** lock you into their stack, or
insist everything run on an underpowered local model while lecturing you
about _privacy_. kompend is provider-agnostic, adapts to whatever LLM you have
— from a local 8B model to a frontier API — and uses structure-preserving
parsing so technical content (tables, equations, figures) survives
extraction intact. [Read the full motivation →](docs/about/motivation.md)

> **OKF v0.1 compliant** — produces [Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundles by default.
> **ACP agent** — implements the [Agent Client Protocol](https://agentclientprotocol.com) for credential-free LLM injection via `kompend agent`.
> **PydanticAI-native** — every LLM call routes through typed agents with failover chains; [models.dev](https://models.dev) catalog resolves any provider.

## How it works

<p align="center">
  <img src="./assets/readme/pipeline.svg" width="100%"
       alt="kompend 5-tier pipeline: Preprocessing → Chunking → Resolution → Compilation → Output">
</p>

kompend runs a 5-tier pipeline — each tier independently replaceable, with a
clear `Base` interface:

| Tier                 | What it does                                                                                                                                                                 | Needs LLM? |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------: |
| **T1 Preprocessing** | Parse PDF, DOCX, PPTX, EPUB, XLSX, HTML, URLs into a uniform `ParsedDocument` preserving structure (headings, tables, figures, equations, code).                             |     ❌     |
| **T2 Chunking**      | Split into a `ChunkTree` sized to the model's context window (auto-detected). Atomic elements never split; text runs via recursive paragraph→line→sentence→token boundaries. |     ✅     |
| **T3 Resolution**    | Extract entities, resolve cross-references, acronym/definition consistency. Optional VLM enricher for figure captioning.                                                     |     ✅     |
| **T4 Compilation**   | Build an interlinked `CompiledWiki` graph. Two backends: `TwoPhaseCompiler` (default, extract→generate) and `WiCERCompiler` (iterative compile→evaluate→refine).             |     ✅     |
| **T5 Output**        | Write OKF bundles (default), Markdown, JSON, or Typst. Optionally index for semantic search and synthesize an overview.                                                      |     ❌     |

`kompend ingest` stops at T1 — **no LLM, no cost**. `kompend compile` runs T2–T5.

## Quick start

```bash
uv tool install kompend

# 1. Initialize a project
kompend init ./my-wiki && cd ./my-wiki

# 2. Ingest sources (T1 only — no LLM needed)
kompend ingest                              # everything in sources/
kompend ingest paper.pdf notes.docx *.xlsx  # explicit files
kompend ingest --url https://arxiv.org/pdf/2605.07068v1

# 3. Compile the wiki (T2–T5 — needs an LLM)
kompend compile --set compile.llm.default_model=ollama/qwen3

# 4. Query it
kompend query "How does attention scale with sequence length?"
```

> **Debugging tip:** `kompend parse paper.pdf --elements` inspects one file's
> extracted structure before ingesting the full corpus. If parsing looks
> wrong, fix it first — the pipeline won't recover from bad extraction.

## Commands

| Command                                    | Purpose                                                   |
| ------------------------------------------ | --------------------------------------------------------- |
| `kompend init [DIR]`                          | Scaffold a wiki project (`--bare`, `--force`, `--git`)    |
| `kompend ingest FILES...`                     | Parse sources into `.kompend/cache/` (T0–1, no LLM)          |
| `kompend compile FILES...`                    | Full pipeline → interlinked wiki (T2–5, needs LLM)        |
| `kompend query "..."`                         | Ask a natural-language question against the wiki          |
| `kompend organize`                            | LLM-categorize `sources/` into a hierarchy and move files |
| `kompend parse FILES...`                      | Debug: parse documents and print structure (`--elements`) |
| `kompend doctor`                              | Validate dirs, parsers, LLM/VLM connectivity              |
| `kompend status`                              | Show source/article/word counts and last compile time     |
| `kompend config get/set/list/path`            | Inspect and edit configuration (5-layer precedence)       |
| `kompend lint run`                            | Lint wiki: 8 structural + 4 LLM rules (`--fix`, `--llm`)  |
| `kompend review list/approve/reject`          | Manage the review queue for compiled pages                |
| `kompend capture "..."` / `learn` / `distill` | Capture notes → synthesize into wiki pages                |
| `kompend write "TITLE"`                       | One-shot wiki page from a title + optional sources        |
| `kompend provenance [SLUG]`               | Trace a page to its source files                          |
| `kompend memory add/search/synthesize`        | Manage query memory and context synthesis                 |
| `kompend serve`                               | Start the web UI (`--host`, `--port`, `--open`)           |
| `kompend mcp`                                 | Start the MCP server (stdio, for AI clients)              |
| `kompend agent`                               | Start the ACP agent (editor integration)                  |

Full option reference: `kompend <command> --help`

## Key subsystems

| Subsystem            | Description                                                                                                                                               |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **RAG / Search**     | `HybridRetriever` fuses FTS + vector + token-overlap + raw sources via reciprocal rank fusion. Embeddings via `litellm:` prefix or sentence-transformers. |
| **Memory lifecycle** | Confidence decay, tier promotion (working → episodic → semantic → procedural), supersession detection.                                                    |
| **Review queue**     | Route compiled pages through approve/reject/lock before publishing.                                                                                       |
| **MCP server**       | Stdio server exposing `parse_document`, `compile_wiki`, `wiki_capture`, `wiki_learn` to Claude Code, Cursor, etc.                                         |
| **ACP agent**        | Agent Client Protocol server: editors inject LLM credentials via `providers/set` — no API keys in your config.                                            |
| **Configuration**    | 5-layer precedence: CLI > env (`KOMP_*`) > project (`.kompend/config.toml`, walks up) > global (`~/.config/kompend/`) > defaults.                               |

## LLM configuration

kompend uses `provider/model` strings resolved by `build_model()`:

1. **models.dev catalog** — custom providers with known API URLs (`claudinio/`, `opencode/`, `deepseek/`) construct OpenAI-compatible models.
2. **pydantic-ai native** — standard providers (`ollama/`, `openai/`, `anthropic/`, `google/`) via `infer_model()`.

API keys come from the env var each provider advertises. No `api_key` in config:

```toml
[compile.llm]
default_model = "ollama/qwen3"
fallback_chain = ["openai/gpt-4o"]   # tried when the primary fails
reasoning_effort = "none"             # tame over-thinking local models
```

kompend auto-detects each model's context window and sizes every call to fit.
**Recommended: a model with ≥ 32K context** (16K is the functional floor;
the generation stage reserves 16K output tokens). See the
[LLM configuration guide](./docs/guides/llm-config.md#context-window-requirements)
for the full breakdown.

See the [LLM configuration guide](./docs/guides/llm-config.md) for the full reference.

## Documentation

> See [`docs/index.md`](./docs/index.md) for the complete documentation index — architecture deep-dives, tier references, WiCER loop, CLI guides, integrations.

## Development

```bash
git clone <repo-url> && cd kompend
uv sync                    # install deps
uv run kompend --help         # verify CLI
uv run pytest tests/       # run tests
mise test                  # run all quality checks
```

## Installing kompend as a dependency

The name `kompend` is taken on PyPI by an unrelated certificate-management tool,
so kompend is not published there. Add it as a git dependency with a
`[tool.uv.sources]` override:

```toml
[tool.uv.sources]
kompend = { git = "https://github.com/jr2804/kompend.git", branch = "main" }
```

Then declare it normally (e.g. `"kompend[all]"` for every parser backend), and
`uv sync` / `uv add kompend[all]` will pull the real kompend from Codeberg.

## License

MIT. See [LICENSE](./LICENSE).

## Acknowledgments

- [Karpathy's compiled LLM-wiki Gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) — the core concept
- [Rohit G00's LLM Wiki v2 Gist](https://gist.github.com/rohitg00/2067ab416f7bbe447c1977edaaa681e2) — memory lifecycle, knowledge graph, hybrid search
- [WiCER paper](https://arxiv.org/pdf/2605.07068v1) (Huerta, Zinnia Tech) — iterative compilation loop
- [Docling](https://github.com/docling-project/docling) (IBM) — structure-preserving parsing
- [Chonkie](https://github.com/feyninc/chonkie) — multi-strategy chunking
- [pydantic-ai](https://ai.pydantic.dev) — typed agent framework
- [models.dev](https://models.dev) — provider/model catalog
