Metadata-Version: 2.4
Name: okf-kit
Version: 0.1.8
Summary: Turn any website into a portable, agent-ready Open Knowledge Format (OKF) bundle — build, sync, and chat, no LLM required to start.
Author: okf-kit contributors
License: Apache-2.0
Project-URL: Homepage, https://github.com/vinodborole/okf-kit
Project-URL: Repository, https://github.com/vinodborole/okf-kit
Project-URL: Issues, https://github.com/vinodborole/okf-kit/issues
Project-URL: Specification, https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md
Keywords: okf,open-knowledge-format,rag,llm,agents,crawler,knowledge-graph
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Requires-Dist: trafilatura>=1.8
Requires-Dist: lxml-html-clean>=0.1
Requires-Dist: selectolax>=0.3.21
Requires-Dist: pyyaml>=6.0
Provides-Extra: js
Requires-Dist: crawl4ai>=0.6.0; extra == "js"
Requires-Dist: trafilatura<2.1,>=1.8; extra == "js"
Provides-Extra: chat
Requires-Dist: openai>=1.60; extra == "chat"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: enrich
Requires-Dist: openai>=1.60; extra == "enrich"
Provides-Extra: mcp
Requires-Dist: mcp>=1.2.0; extra == "mcp"
Provides-Extra: all
Requires-Dist: okf-kit[anthropic,chat,enrich,js,mcp]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Dynamic: license-file

# okf-kit

[![CI](https://github.com/vinodborole/okf-kit/actions/workflows/ci.yml/badge.svg)](https://github.com/vinodborole/okf-kit/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/okf-kit.svg)](https://pypi.org/project/okf-kit/)
[![Python versions](https://img.shields.io/pypi/pyversions/okf-kit.svg)](https://pypi.org/project/okf-kit/)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![OKF spec](https://img.shields.io/badge/format-OKF%20v0.1-9A6B1F.svg)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)

**Turn any website into a portable, agent-ready knowledge bundle — no LLM required to start.**

`okf-kit` crawls a site into a
[Google Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
bundle: a directory of markdown concept files with YAML frontmatter and
per-directory `index.md` listings that any agent can navigate with plain file
reads. Build it, keep it in sync as the site changes, publish it, and chat with
it — locally, with your own key, or fully offline via Ollama.

```bash
pip install okf-kit
okf build https://docs.example.com -o docs-okf   # crawl → OKF bundle (no key, no browser)
okf chat docs-okf --provider ollama              # chat offline, no key
```

Or zero-install with [`uv`](https://docs.astral.sh/uv/):

```bash
uvx --from okf-kit okf build https://docs.example.com -o docs-okf
```

Part of the **[calknowledge](https://github.com/vinodborole/calknowledge)**
ecosystem — okf-kit is the lightweight, open library; calknowledge is the full
platform (LLM enrichment, RAG export, retrieval evals, GUI) built on top of it.

---

## Why

Everyone re-crawls and re-indexes the same docs privately and badly. okf-kit
makes a website's knowledge a **portable artifact**:

- **Agents can read an OKF bundle; they can't read your website.** The bundle is
  navigable markdown — no scraping, no SDK, no runtime.
- **Faithful markdown, not text soup.** Real extraction (headings, code, tables),
  boilerplate filtered, JS-rendered when needed.
- **Self-maintaining.** `okf sync` updates only what changed, so a published
  bundle in git produces small delta commits and never goes stale.
- **Works with any LLM, or none.** Chat via OpenAI, Ollama, vLLM, OpenRouter, or
  Claude — or get a zero-key retrieval answer with citations.

## Install

```bash
pip install okf-kit                 # core: build / sync / validate / zip / list / get / visualize
pip install "okf-kit[chat]"         # okf chat via OpenAI-compatible providers (OpenAI, Ollama, …)
pip install "okf-kit[anthropic]"    # Claude as a chat provider
pip install "okf-kit[js]"           # crawl JavaScript-rendered sites (pulls a Playwright Chromium)
pip install "okf-kit[mcp]"          # serve bundles to Claude Code / Cursor over MCP
pip install "okf-kit[enrich]"       # okf build --enrich (LLM descriptions + tags)
```

The default install has **no browser and no LLM SDK** — it installs in seconds.

> **Tip:** install into a dedicated virtualenv so okf-kit's dependencies don't
> mix with your other projects:
> ```bash
> python3 -m venv ~/okf && ~/okf/bin/pip install okf-kit
> ```
> This also avoids clashes if an existing environment already pins packages
> like `lxml` (e.g. a prior `crawl4ai` install) — a plain install would
> otherwise bump them.

## Commands

### Build

```bash
okf build https://docs.example.com -o docs-okf --max-depth 3 --max-pages 200
```

Domain-restricted BFS crawl → an OKF bundle: `pages/` mirror with frontmatter
concepts, a `.okf-kit/state.json` for sync, and an `index.md` in every directory
for agent navigation. Validated on exit. No API key needed.

By default the crawl is **scoped to the seed's path section** — `okf build
https://doc.rust-lang.org/book/` stays under `/book/` and won't wander into the
rest of the host. Override with `--path-prefix PATH` (a narrower/different
scope) or `--all-paths` (the whole host). Other flags: `--js` (JS-rendered
sites — build hints when a site needs it), `--no-robots`, `--enrich` (add LLM
descriptions/tags — needs `[enrich]` + `OPENAI_API_KEY`).

### Sync

```bash
okf sync docs-okf
```

Re-crawls the same site and updates **only the delta** — added pages written,
changed pages rewritten, removed pages deleted, unchanged pages left
byte-for-byte (stable git diffs). A safety valve aborts on a suspiciously empty
re-crawl (`--force` overrides).

### Chat

```bash
okf chat docs-okf --provider ollama                 # offline, no key
okf chat docs-okf --provider openai --trace         # any provider, with citations + a navigation trace
okf chat docs-okf                                   # no provider → zero-key retrieval answer
okf chat docs-okf --resume                          # continue the last session (history is local)
```

The agent navigates the bundle (`list_directory` / `read_concept`) to the most
specific concept and answers **only from what it read**, citing the paths.

| `--provider` | Endpoint | Key |
|---|---|---|
| `openai` | OpenAI | `OPENAI_API_KEY` |
| `ollama` | `localhost:11434` (local) | none |
| `openrouter` | OpenRouter | `OPENROUTER_API_KEY` |
| `anthropic` | Claude | `ANTHROPIC_API_KEY` |
| `custom` | `--base-url` | as configured |

Chat history is stored locally at `~/.okf/chats/<bundle>/`.

### Visualize

```bash
okf visualize docs-okf          # -> docs-okf/graph.html
```

A self-contained interactive graph (nodes = concepts, edges = internal links);
no backend, no CDN — open the HTML from `file://`.

### Serve over MCP

```bash
okf serve-mcp docs-okf          # or --all for every downloaded bundle
```

Exposes `list_bundles` / `list_directory` / `read_concept` / `search_bundle` over
stdio MCP for Claude Code/Desktop, Cursor, and any MCP client.

### Registry

```bash
okf list --remote               # browse published bundles
okf get backstage-docs          # download, validate, install to ~/.okf/bundles/
okf list                        # your local bundles
```

### Package for hand-off

```bash
okf zip docs-okf                # -> docs-okf.zip, ready to publish or share
```

## Publishing

See [docs/PUBLISHING.md](docs/PUBLISHING.md) — build a bundle, ship it as a
release zip with a weekly self-sync Action, and add it to the
[awesome-okf-kit](https://github.com/vinodborole/awesome-okf-kit) registry.
Publish only content you may redistribute.

## Bundle layout

```
docs-okf/
    index.md                 root directory listing (reserved, no frontmatter)
    log.md                   build/sync history
    pages/                   one concept per page (frontmatter + body + citations)
        index.md             directory listing (every directory has one)
        home.md
        docs/…
    .okf-kit/state.json      crawl config, per-page content hashes, link edges
```

## Development

`pip install -e ".[dev]"`, then `pytest -q` (37 tests, fully offline) and
`ruff check okf_kit tests`. See [CONTRIBUTING.md](CONTRIBUTING.md) and the
[CHANGELOG](CHANGELOG.md).

## License

Apache-2.0.
