Metadata-Version: 2.4
Name: llmwiki-harness
Version: 0.3.0
Summary: Context-Memory Harness for AI Agents. Context Window = RAM, Local Wiki = Disk.
Author: chancelu
License: MIT
Project-URL: Homepage, https://github.com/chancelu/llmwiki-harness
Project-URL: Issues, https://github.com/chancelu/llmwiki-harness/issues
Keywords: agent,memory,wiki,markdown,obsidian,llm,context,retrieval,rag,karpathy
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Provides-Extra: semantic
Requires-Dist: sentence-transformers; extra == "semantic"
Requires-Dist: numpy; extra == "semantic"
Provides-Extra: mcp
Requires-Dist: mcp>=1.2.0; extra == "mcp"
Dynamic: license-file

# LLMWiki

[![CI](https://github.com/chancelu/llmwiki-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/chancelu/llmwiki-harness/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/llmwiki-harness.svg)](https://pypi.org/project/llmwiki-harness/)
[![Python](https://img.shields.io/pypi/pyversions/llmwiki-harness.svg)](https://pypi.org/project/llmwiki-harness/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

> **Context Window = RAM, Local Wiki = Disk**
>
> A zero-dependency framework that turns your local Markdown vault (Obsidian, selfwiki, etc.) into long-term memory for any AI agent.

> PyPI note: the distribution is published as **`llmwiki-harness`** (`pip install llmwiki-harness`). The bare `llmwiki` name on PyPI belongs to an unrelated third-party project — do not `pip install llmwiki`. The Python import package and CLI are still called `llmwiki`.

## What This Is

Every serious agent user hits the same wall: the agent forgets everything between sessions. LLMWiki solves this by treating:

- **Your context window** as volatile RAM (fast, limited, per-session)
- **Your local Markdown wiki** as persistent Disk (slow, unlimited, cross-session)

It provides a universal **harness** that any agent framework can plug into — no Docker, no cloud, no vector DB required.

```
┌─────────────────────────────────────────────┐
│  Agent (OpenClaw / LangChain / AutoGen ...) │
│  ┌───────────────────────────────────────┐  │
│  │  L1: Context Window (RAM)             │  │
│  │  ├── Current conversation             │  │
│  │  └── ← Injected wiki knowledge        │  │
│  └───────────────────────────────────────┘  │
│              ↑ ↓ LLMWiki Harness            │
│  ┌───────────────────────────────────────┐  │
│  │  L3: Local Markdown Wiki (Disk)       │  │
│  │  ├── entities/  concepts/  projects/  │  │
│  │  ├── chronicle/daily/  (conversation) │  │
│  │  └── raw/  (session dumps)            │  │
│  └───────────────────────────────────────┘  │
└─────────────────────────────────────────────┘
```

## Features

| Feature | Status |
|---------|--------|
| **Multi-engine search** — ripgrep, SQLite FTS, pure Python fallback | ✅ |
| **CJK-aware full-text search** — SQLite FTS5 trigram tokenizer + bigram query splitting; Chinese/Japanese/Korean vaults just work | ✅ |
| **Multi-strategy retrieval** — keyword, graph (wikilink traversal), temporal | ✅ |
| **Knowledge graph edge table** — index-time wikilink graph with backlinks, 2-hop weighted traversal, dead-link/orphan detection | ✅ |
| **RRF fusion** — combine multiple retrieval strategies | ✅ |
| **Token budget management** — never overflow the context window | ✅ |
| **In-memory cache** — avoid repeated disk reads | ✅ |
| **OpenClaw adapter** — drop-in memory hook | ✅ |
| **MCP server** — works with Claude Desktop / Claude Code / Cursor / any MCP host | ✅ |
| **3-layer vault architecture** (Karpathy-native) | ✅ |
| **Zero dependencies** for core (optional enhancements via extras) | ✅ |

## Install

```bash
# Core (zero dependencies)
pip install llmwiki-harness

# With semantic search support
pip install llmwiki-harness[semantic]

# With MCP server support (Claude Desktop / Cursor / any MCP host)
pip install llmwiki-harness[mcp]

# Dev
pip install llmwiki-harness[dev]
```

## Quick Start

### 1. Initialize a vault

```bash
llmwiki init ~/Documents/selfwiki
```

This creates the directory structure:
```
~/Documents/selfwiki/
├── raw/              # Layer 1: session dumps
├── chronicle/daily/  # Layer 2: daily conversation logs
├── entities/         # Layer 3: atomic knowledge
├── concepts/
├── comparisons/
├── projects/
├── queries/
└── SCHEMA.md
```

### 2. Use in your agent

```python
from llmwiki import ContextMemoryHarness

harness = ContextMemoryHarness("~/Documents/selfwiki")
harness.build_index()

# Before each turn — retrieve relevant knowledge
context = harness.retrieve_and_assemble(
    query=user_message,
    token_budget=2000,
)

# Inject into your prompt
messages = [
    {"role": "system", "content": f"{system_prompt}\n\n{context}"},
    {"role": "user", "content": user_message},
]

# After each turn — capture to chronicle
harness.capture_turn(user_message, assistant_response)

# Periodically — curate chronicle into compiled notes
harness.curate()
```

### 3. OpenClaw adapter

```python
from llmwiki.adapters import OpenClawMemoryHook

hook = OpenClawMemoryHook("~/Documents/selfwiki")

# On each turn:
wiki_context = hook.on_turn_start(user_message)
# → inject into system prompt

hook.on_turn_end(user_message, assistant_response)
# → auto-captures to chronicle
```

## MCP Server

The fastest way to use LLMWiki: run it as an [MCP](https://modelcontextprotocol.io) server and plug it into Claude Desktop, Claude Code, Cursor, Codex, or any MCP-compatible host. The agent gets five memory tools:

| Tool | Purpose |
|------|---------|
| `memory_search(query, top_k)` | Raw ranked search over the wiki |
| `memory_recall(query, token_budget)` | Assembled context block, ready to inject into a prompt |
| `memory_capture(user_message, assistant_message)` | Store a conversation turn in the chronicle |
| `memory_curate()` | Distill the chronicle into compiled atomic notes |
| `memory_stats()` | Vault / index / cache statistics |

### Claude Desktop / Cursor (`claude_desktop_config.json`)

```json
{
  "mcpServers": {
    "llmwiki": {
      "command": "uvx",
      "args": ["--from", "llmwiki-harness[mcp]", "llmwiki-mcp"],
      "env": {
        "LLMWIKI_VAULT_PATH": "~/Documents/selfwiki"
      }
    }
  }
}
```

### Claude Code

```bash
claude mcp add llmwiki -- uvx --from "llmwiki-harness[mcp]" llmwiki-mcp
# then set the vault:  export LLMWIKI_VAULT_PATH=~/Documents/selfwiki
```

### Already installed via pip?

```bash
pip install llmwiki-harness[mcp]
llmwiki mcp                          # stdio server, vault from config/env
llmwiki -v ~/Documents/selfwiki mcp  # explicit vault path
```

The server speaks stdio (the MCP default for local servers). Compatible with both `mcp` 1.x and 2.x Python SDKs.

## CLI

```bash
llmwiki init <path>              # Initialize vault
llmwiki index [--force]          # Build search index
llmwiki search "prompt injection" # Search wiki
llmwiki curate [--llm]           # Run curation pipeline
llmwiki stats                    # Vault statistics
llmwiki health                   # Check for dead links, orphans
llmwiki graph "Zettelkasten"     # Show a note's links, backlinks, 2-hop neighbors
llmwiki config                   # Show configuration
llmwiki mcp                      # Start MCP server (stdio) for any MCP host
```

## Configuration

Create `llmwiki.yaml` in your vault root or `~/.config/llmwiki/config.yaml`:

```yaml
vault:
  path: ~/Documents/selfwiki

index:
  engine: ripgrep  # ripgrep | sqlite | hybrid
  incremental: true

retrieve:
  default_top_k: 5
  strategies: [keyword, graph, temporal]
  fusion_method: rrf

context:
  token_budget: 4000
  format: markdown
  priority: relevance  # relevance | recency | diversity | structured

cache:
  enabled: true
  maxsize: 100
  ttl: 300

curate:
  enabled: true
  archive_after_days: 30
```

## Architecture

### Core Components

| Module | Purpose |
|--------|---------|
| `Indexer` | Manages search indices (ripgrep, SQLite, etc.) |
| `LinkGraph` | Persistent wikilink edge table (SQLite): neighbors, backlinks, dead links, orphans |
| `Retriever` | Multi-strategy recall (keyword, graph, temporal) |
| `Assembler` | Token-budget-aware context assembly |
| `Cache` | In-memory LRU cache |
| `MCP Server` | Exposes memory tools to any MCP host over stdio |

### Data Flow

```
User Message → Retriever → [Keyword | Graph | Temporal] → RRF Fusion
                                                        ↓
                              Assembler ←── Token Budget Check
                                                        ↓
                                              System Prompt Injection

Turn End → Capture → chronicle/daily/YYYY-MM-DD.md
                              ↓ (scheduled curation)
                     compiled/entities/ | concepts/ | projects/
```

## Vault Schema (Karpathy 3-Layer)

```
┌─────────────────────────────────────────┐
│ Layer 3: Compiled Wiki (Query)          │
│ entities/ concepts/ comparisons/        │
│ projects/ queries/                      │
│ ↑ LLM curation (nightly)                │
├─────────────────────────────────────────┤
│ Layer 2: Chronicle (Daily Notes)        │
│ chronicle/daily/YYYY-MM-DD.md           │
│ ↑ auto-capture from agent turns         │
├─────────────────────────────────────────┤
│ Layer 1: Raw (Session Exports)          │
│ raw/session-{id}.md                     │
│ ↑ on_session_end / on_pre_compress      │
└─────────────────────────────────────────┘
```

## Ecosystem

- **TypeScript port for DeepSeek Harness**: [`dsh-llmwiki`](https://github.com/chancelu/dsh-llmwiki) — same vault format, native dsh plugin, on npm.

## What's New in 0.3.0

- **MCP server** — `llmwiki mcp` / `llmwiki-mcp` exposes five memory tools (`memory_search`, `memory_recall`, `memory_capture`, `memory_curate`, `memory_stats`) to Claude Desktop, Claude Code, Cursor, Codex, and any MCP host. Compatible with both mcp 1.x (`FastMCP`) and 2.x (`MCPServer`).
- **Persistent knowledge graph** — wikilinks are extracted at index time into a SQLite edge table (`.llmwiki/graph.db`). The graph retrieval strategy now does weighted 2-hop traversal (forward links 1.0, backlinks 0.8, hop-2 0.5) instead of re-parsing files on every query.
- **`llmwiki graph` / improved `llmwiki health`** — inspect any note's links, backlinks, and 2-hop neighborhood; health checks report dead links and orphan notes from the edge table.
- **CJK search fixed** — SQLite engine now prefers the FTS5 `trigram` tokenizer (with graceful fallback), and temporal/keyword matching splits CJK queries into bigrams. Chinese vaults are first-class.
- **FTS5 query sanitization** — natural-language queries no longer crash `MATCH` on quotes, hyphens, or AND/OR/NOT.
- **Tooling** — repo-wide black + ruff clean, CI now actually runs on `main` (it was misconfigured to `master`).

## From hermes-llmwiki

This project evolved from [`hermes-llmwiki`](https://github.com/chancelu/hermes-llmwiki). Key changes in 0.2.0:

- **Framework-agnostic**: No longer Hermes-only — works with any agent
- **Multi-engine search**: ripgrep + SQLite FTS + Python fallback
- **Multi-strategy retrieval**: keyword + graph + temporal + RRF fusion
- **Token budget management**: Dynamic context assembly
- **In-memory cache**: L1 RAM layer for frequent queries
- **OpenClaw adapter**: First-class adapter for OpenClaw agents

## Development

```bash
git clone https://github.com/chancelu/llmwiki-harness
cd llmwiki-harness
pip install -e ".[dev,mcp]"

pytest tests/          # 65 tests
black llmwiki/ tests/  # formatting (line-length 100)
ruff check llmwiki/ tests/
```

CI runs the test matrix (Linux / Windows / macOS × Python 3.10–3.13) plus black and ruff on every push to `main`.

Releases are published to PyPI via trusted publishing: pushing a `v*` tag triggers the `publish` workflow.

## License

MIT
