Metadata-Version: 2.4
Name: llm-wiki-tools
Version: 0.4.1
Summary: CLI tools to read and manage LLM Wiki projects on GitHub
License: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: markitdown>=0.1.0
Requires-Dist: anthropic>=0.25.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Dynamic: license-file

# llm-wiki-tools

CLI tools for the **Agentic LLM Wiki** pattern. Manage project knowledge as a curated, traversal-organized Markdown wiki that AI agents can read, write, and patch surgically — entirely on your local disk.

**No MCP server. No remote backend. Just Python, Markdown, and Git.**

---

## The concept: knowledge as code

Inspired by Andrej Karpathy's LLM Wiki pattern, this package treats project documentation like software:

1. **Raw sources (`raw/`)** — immutable "source code": PDFs, docs, transcripts, emails. The wiki is built from these.
2. **Updates (`updates/`)** — staging area for new sources awaiting ingestion. After processing, files are archived into `raw/` automatically.
3. **The wiki (`wiki/`)** — "compiled binary": traversal concept pages in plain Markdown, optimized for LLM consumption. Versioned with Git.
4. **The agents (Claude Code)** — the "compiler". An `analyst` plans the structure, an `ingest-agent` materializes it; both invoke the CLIs below.

Pages in the wiki are **concept-oriented, not source-oriented**: a single source contributes to N pages, and a single page aggregates contributions from N sources. There is no one-to-one mapping between raw files and wiki pages. External traceability is preserved with inline `[fonte: <slug>]` markers, not with source-pages or metadata blocks.

---

## How it works

```text
       local disk (strict local — no remote backend)
       ────────────────────────────────────────────────

       wiki-my-project/
         updates/      ←  you drop new sources here
            │
            │  wiki-manage update  (reads, prints, then archives)
            ▼
         raw/          ←  immutable history; CLI-managed only
            │
            │  wiki-manage generate / update  (emits content)
            ▼
         analyst → plan → ingest-agent → wiki-manage write/patch/delete
            │
            ▼
         wiki/         ←  plain Markdown, concept pages
            │
            │  git add / commit / push  (you, manually)
            ▼
         GitHub        ←  storage and history, optional
```

Two CLIs:

- **`wiki-read`** — read pages and search the wiki. Used by every agent and by you.
- **`wiki-manage`** — ingest sources, write/patch/delete pages, run health checks. Used by the agents in the wiki repo.

Both operate **strictly on the local filesystem**. Pushing to GitHub is your responsibility, done manually with `git`. GitHub is just storage — the package never talks to it.

---

## Install

```bash
pip install llm-wiki-tools
```

For OCR and image understanding (PDFs with scanned text, image files), enable the vision extra:

```bash
pip install "llm-wiki-tools[vision]"
```

The vision extra uses Claude as the OCR backend via the official `anthropic` SDK.

---

## Requirements

- **Python 3.12+**
- **Anthropic API key** *(optional, vision only)* — set as `ANTHROPIC_API_KEY` or in `.project` to enable OCR of PDFs and images.

No GitHub token is needed by the CLI itself: Git operations are performed by you outside the package.

---

## Configuration (`.project`)

Place a `.project` file at the root of your wiki repo. The CLIs walk up from the current working directory to find it.

```json
{
  "project_id": "my-project",
  "wiki_repo": "username/wiki-my-project",
  "wiki_local_path": "/absolute/path/to/wiki-my-project",
  "anthropic_key": "sk-ant-...",
  "anthropic_model": "claude-3-5-sonnet-latest"
}
```

Required:
- `wiki_local_path` — absolute path to the wiki repo on disk. Must exist.

Optional:
- `project_id`, `wiki_repo` — identifiers used in informational output; not used for any network call.
- `anthropic_key` — enables vision-based OCR during raw/updates conversion.
- `anthropic_model` — defaults to `claude-3-5-sonnet-latest`.

---

## Directory layout (per wiki repo)
```
wiki-my-project/
.project
raw/         ←  immutable archive of all ingested sources
updates/     ←  drop zone for new sources awaiting ingest
wiki/        ←  the compiled Markdown wiki (versioned with Git)
```

- `raw/` is read-only for agents and humans. It is written exclusively by `wiki-manage update` as part of the archival step (see below).
- `updates/` is a transit zone. You drop new files there; the CLI moves them into `raw/` once successfully read.
- `wiki/` is the only directory you push to GitHub.

---

## Commands

### `wiki-read` — consult the wiki

Read-only. Operates on `wiki/` on the local disk.

```bash
wiki-read <page>                  # print page content
wiki-read --list                  # list all pages
wiki-read --search "<query>"      # full-text search
wiki-read --info                  # show project info from .project
```

Page names can omit `.md`. `wiki-read index` and `wiki-read index.md` are equivalent.

### `wiki-manage` — maintain the wiki

Mixed read/write. Operates on the local disk.

**Status and health**
```bash
wiki-manage status                # file counts for raw/, updates/, wiki/
wiki-manage lint                  # broken links, orphan pages, empty sections, missing required
```

**Ingestion**
```bash
wiki-manage generate              # read all raw/ files, emit content for the analyst (read-only)
wiki-manage update                # read all updates/ files, emit content, then archive them into raw/
```

`generate` is purely read-only. `update` archives the successfully-read files into `raw/` after emission. Files that fail to read remain in `updates/` for a retry.

**Surgical writing (via STDIN)**
```bash
cat << 'EOF' | wiki-manage write <page>
# Page Title
Content...
EOF

cat << 'EOF' | wiki-manage patch <page> "## Section Header" <action>
New content...
EOF
```

Patch actions:
- `append` — insert at the end of the section
- `prepend` — insert immediately after the header
- `replace` — replace the section body
- `delete` — remove the section (header included)

**Page removal**
```bash
wiki-manage delete <page>
```

Refuses to delete required pages (`index.md`, `changelog.md`).

---

## File lifecycle
```
[ you ]  →  updates/  →  [ wiki-manage update ]  →  raw/  (archive)
│
└─►  emitted content  →  analyst  →  plan
│
▼
ingest-agent
│
▼
wiki-manage write/patch
│
▼
wiki/
│
└─►  git push  (you)
```
Important properties:
- Archival of `updates/ → raw/` is automatic and irreversible (no backup; relies on Git history for audit).
- Files that fail conversion stay in `updates/`. The CLI never silently drops content.
- When a file in `updates/` has the same name as one in `raw/`, the emitted output tags it `[REPLACEMENT]` so the analyst knows to treat it as a revision (and to remove content that was present in the old version but absent in the new).

---

## Supported source formats

The package uses [`markitdown`](https://pypi.org/project/markitdown/) to convert binary formats into Markdown.

- **Plain text** (read as-is): `.txt`, `.md`, `.json`, `.jsonl`, `.csv`, `.py`, `.js`, `.ts`, `.go`, `.rs`, `.sql`, `.sh`, and other common code/text extensions.
- **Documents** (converted via markitdown): `.docx`, `.xlsx`, `.xls`, `.pptx`, `.pdf`, `.msg`, `.ipynb`, `.epub`, `.html`, `.xml`, `.zip`.
- **Images** (described via Claude if `anthropic_key` is set): `.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`. Without a key, images yield a placeholder.

Unsupported binary formats raise a clear error; the file stays in `updates/`.

---

## The agentic system around these CLIs

This package is the execution layer. The intended usage is inside a wiki repo configured with Claude Code, where two sub-agents and a small set of skills orchestrate the CLIs:

- **`analyst` (Opus)** — reads `.project`, `raw/`, `updates/`, and the existing wiki; designs the page structure following anti-drift and granularity rules; produces an execution-ready plan with explicit `CREATE` / `PATCH` / `DELETE` primitives. Read-only.
- **`ingest-agent` (Sonnet)** — receives the plan and executes it via `wiki-manage write/patch/delete`, runs `wiki-manage lint`, writes the changelog entry. No architectural decisions.
- **Skills** — `wiki-generate`, `wiki-update`, `wiki-lint`, `wiki-query`. Pre-flight, delegate to sub-agents, validate the plan, present the result. Never execute writes directly.

A reference scaffolding repo (`scaffolding-wiki`) provides the matching `CLAUDE.md`, agent prompts, and skill files. This package supplies only the CLIs they depend on.

---

## Wiki conventions

These are not enforced by the CLI — they are enforced by the agentic system. Two are worth stating here because they shape how `wiki-manage` is used:

- **Pure semantic division** — pages are traversal concepts (e.g. `architecture.md`, `requirements.md`, `auth.md`), not one-page-per-source. Never create a `sources/` directory.
- **Plain Markdown + inline citations** — no YAML frontmatter, no JSON metadata. External provenance is tracked with inline `[fonte: <slug>]` markers, where `<slug>` is the basename of the source file (no extension, lowercased, spaces/underscores/punctuation collapsed into single dashes).

Two pages are always required:
- `index.md` — map of the wiki, with one description line per page.
- `changelog.md` — reverse-chronological log of every operation.

---

## Testing locally

```bash
# install in editable mode
pip install -e ".[dev]"

# set up a fake wiki
mkdir -p /tmp/test-wiki/{raw,updates,wiki}
echo "some content" > /tmp/test-wiki/raw/doc.txt

cd /tmp
cat > .project << 'EOF'
{
  "project_id": "test",
  "wiki_repo": "username/wiki-test",
  "wiki_local_path": "/tmp/test-wiki"
}
EOF

wiki-read --info
wiki-manage status
wiki-manage generate

# run the test suite
cd /path/to/llm-wiki-tools
pytest -q

# cleanup
rm -rf /tmp/test-wiki /tmp/.project
```

---

## Publishing to PyPI

1. Bump version in `pyproject.toml` (patch for fixes/subcommands, minor for new binaries, major for breaking changes to `.project` or CLI).
2. Update this README and `CLAUDE.md` if the user-facing interface or architecture changed.
3. Build and upload:

```bash
pip install build twine
python -m build
twine upload dist/*
```
