Metadata-Version: 2.4
Name: llmwikify
Version: 0.40.2
Summary: Knowledge + Chat + Research Assistant - LLM-maintained knowledge bases with chat & general research capabilities
Author-email: sn0wfree <linlu1234567@sina.com>
Maintainer-email: sn0wfree <linlu1234567@sina.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/sn0wfree/llmwikify
Project-URL: Documentation, https://llmwikify.readthedocs.io
Project-URL: Repository, https://github.com/sn0wfree/llmwikify
Project-URL: Issues, https://github.com/sn0wfree/llmwikify/issues
Project-URL: Changelog, https://github.com/sn0wfree/llmwikify/blob/main/CHANGELOG.md
Keywords: wiki,chat,research-assistant,llm,knowledge-base,mcp,agent
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Education
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Education
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jinja2>=3.1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.28.0
Requires-Dist: duckdb>=1.0.0
Requires-Dist: pyjwt<3,>=2.0
Requires-Dist: keyring<24,>=21.2.0
Requires-Dist: jieba>=0.42
Provides-Extra: mcp
Requires-Dist: fastmcp>=3.0.0; extra == "mcp"
Provides-Extra: config
Provides-Extra: watch
Requires-Dist: watchdog>=3.0.0; extra == "watch"
Provides-Extra: graph
Requires-Dist: networkx>=3.0; extra == "graph"
Requires-Dist: pyvis>=0.3.0; extra == "graph"
Requires-Dist: python-louvain>=0.16; extra == "graph"
Provides-Extra: web
Requires-Dist: fastapi>=0.104.0; extra == "web"
Requires-Dist: starlette>=0.27.0; extra == "web"
Requires-Dist: uvicorn>=0.23.0; extra == "web"
Requires-Dist: httpx>=0.24.0; extra == "web"
Provides-Extra: agent
Requires-Dist: croniter>=2.0.0; extra == "agent"
Requires-Dist: filelock>=3.13.0; extra == "agent"
Requires-Dist: duckduckgo-search>=4.0; extra == "agent"
Requires-Dist: tavily-python>=0.3.0; extra == "agent"
Requires-Dist: apscheduler<4,>=3.10.0; extra == "agent"
Provides-Extra: llm
Requires-Dist: tiktoken>=0.7.0; extra == "llm"
Provides-Extra: extractors
Requires-Dist: markitdown[all]>=0.1.0; extra == "extractors"
Requires-Dist: pymupdf>=1.23.0; extra == "extractors"
Requires-Dist: trafilatura>=1.7.0; extra == "extractors"
Requires-Dist: youtube-transcript-api>=0.6.0; extra == "extractors"
Provides-Extra: all
Requires-Dist: llmwikify[agent,config,extractors,graph,mcp,watch,web]; extra == "all"
Provides-Extra: dev
Requires-Dist: llmwikify[all]; extra == "dev"
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0; extra == "dev"
Requires-Dist: types-requests>=2.30; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine<5,>=4.0.0; extra == "dev"
Requires-Dist: keyring<24,>=21.2.0; extra == "dev"
Requires-Dist: jaraco.context<5.4; extra == "dev"
Requires-Dist: jaraco.functools<5,>=3; extra == "dev"
Requires-Dist: jaraco.classes<4; extra == "dev"
Requires-Dist: more-itertools<11; extra == "dev"
Dynamic: license-file

# llmwikify

> **Knowledge + Chat + Research Assistant — LLM-maintained knowledge bases with chat & general research capabilities.**

[![PyPI version](https://badge.fury.io/py/llmwikify.svg)](https://pypi.org/project/llmwikify/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Tests](https://img.shields.io/badge/tests-4300%2B%20passed-brightgreen.svg)](https://github.com/sn0wfree/llmwikify)
[![Version](https://img.shields.io/badge/version-0.40.2-blue.svg)](pyproject.toml)
[![CI Tests](https://github.com/sn0wfree/llmwikify/actions/workflows/tests.yml/badge.svg)](https://github.com/sn0wfree/llmwikify/actions/workflows/tests.yml)
[![Lint](https://github.com/sn0wfree/llmwikify/actions/workflows/lint.yml/badge.svg)](https://github.com/sn0wfree/llmwikify/actions/workflows/lint.yml)
[![codecov](https://codecov.io/gh/sn0wfree/llmwikify/branch/main/graph/badge.svg)](https://codecov.io/gh/sn0wfree/llmwikify/branch/main/graph/badge.svg)

**llmwikify** is a Python CLI + library + unified server for building **persistent,
LLM-maintained knowledge bases** with **chat** and **general research assistant** capabilities.

> **v0.40 Refocus** — quant research pipeline (paper → factor → backtest) has moved to
> [**quantnodes.research**](https://github.com/sn0wfree/quantnodes) (>=4.0). llmwikify
> now focuses exclusively on wiki + chat + research-assistant use cases.
>
> ⚠️ **v0.40.2 BREAKING for HTTP clients**: `POST /page` to an existing page
> now returns **409 Conflict** with a `confirmation_id` (one-shot, TTL 300s)
> instead of silently overwriting. Re-POST with the token (query string
> `?confirm_token=<id>` is the primary path; body `{"confirm_token": "..."}`
> is the back-compat shim) to confirm. See
> [docs/releases/v0.40.2.md](docs/releases/v0.40.2.md) for details.

![Dashboard](docs/screenshots/dashboard.png)

> ⚠️ **Beta Release** — APIs may shift between minor versions. Report issues on
> [GitHub](https://github.com/sn0wfree/llmwikify/issues).

---

## Why llmwikify?

| | |
|---|---|
| 📚 **Knowledge Base** | LLM-maintained wiki with auto-linting, multi-wiki registry, section anchors |
| 💬 **Chat Agent** | Streaming chat with tool calling, 50+ skills, subagents, memory |
| 🔬 **Research Assistant** | Adaptive ReAct loop, multi-source synthesis, quality gates |
| 🔍 **Smart Search** | SQLite FTS5 + optional QMD hybrid (BM25 + vector + LLM reranking) |
| 🔗 **Bidirectional Links** | Automatic `[[wikilink]]` detection with section-level granularity |
| 🧠 **Knowledge Graph** | 8 relation types, PageRank, community detection, interactive D3.js visualization |
| 🌐 **Unified Server** | MCP + REST + WebSocket + Web UI in one process |

---

## Features

### 💬 Chat + Skills Agent
Streaming chat with tool calling, confirmations, and 50+ built-in skills (CRUD,
research, subagent, scheduler). The agent can search your wiki, manage pages,
and orchestrate multi-step research — all with human-in-the-loop confirmations.

![Chat](docs/screenshots/chat.png)

### 🔬 Research Assistant
Adaptive ReAct research engine that breaks down questions into sub-queries,
gathers sources, applies quality gates, and synthesizes structured reports.
Configuration-driven via `~/.llmwikify/llmwikify.json`.

### 📚 Knowledge Base (Wiki)
LLM-maintained wiki with bidirectional wikilinks, multi-wiki registry,
section-level anchors, YAML frontmatter + DuckDB storage, FTS5 full-text search.

### Markdown Editor
Split-pane live markdown editor with page tree, front-matter panel, and wikilink autocomplete. Edit, preview, and manage your wiki pages in one view.

![Editor](docs/screenshots/editor.png)

### Knowledge Graph
Interactive D3.js force-directed graph with PageRank node sizing, community coloring, and bridge highlighting. Explore relationships between your wiki pages visually.

![Graph](docs/screenshots/graph.png)

### Dashboard
Track your knowledge growth with metrics cards, Wiki Dream activity timeline, and health indicators. See how your wiki evolves over time.

![Dashboard](docs/screenshots/dashboard.png)

---

## Quick Start

```bash
# Install (with PDF/document extractors + web server)
pip install 'llmwikify[extractors,web]'

# Create and enter wiki directory
mkdir my-wiki
cd my-wiki

# Initialize a wiki
llmwikify init

# Add a page manually
llmwikify write_page "hello" --content "# Hello World\n\nThis is my first page."

# Ingest a markdown source (PDF requires [extractors])
llmwikify ingest README.md

# Build the link index (finds [[wikilinks]])
llmwikify build-index

# Search the wiki
llmwikify search "hello"

# Read a page
llmwikify read_page hello

# Show wiki status
llmwikify status

# Start the server (MCP + REST + Web UI)
llmwikify serve --web --port 8765
# Open http://localhost:8765
```

### LLM Setup (for AI features)

```bash
# Option 1: One-shot — set up LLM with your wiki
export OPENAI_API_KEY=sk-...
llmwikify init --llm

# Option 2: Standalone — set up LLM separately
llmwikify init-llm
llmwikify init-llm --provider openai --api-key sk-...
llmwikify init-llm --provider anthropic  # uses ANTHROPIC_API_KEY
llmwikify init-llm --provider minimax    # uses MINIMAX_API_KEY

# Option 3: Interactive — init will prompt you
llmwikify init
# 💡 LLM features (analyze-source, synthesize, chat) need ~/.llmwikify/llmwikify.json
#    No LLM config detected. Set one up now? [y/N]: y
#    Detected OPENAI_API_KEY in env vars. Provider: openai
# ✅ LLM config written to ~/.llmwikify/llmwikify.json
```

#### Custom endpoint / OpenAI-compatible

```bash
llmwikify init-llm \
    --provider openai \
    --api-key sk-... \
    --base-url https://api.deepseek.com/v1
```

#### Non-interactive (CI / scripts)

```bash
llmwikify init --llm --no-llm-prompt --llm-overwrite
```

> **Without LLM**: `init`, `search`, `write_page`, `read_page`, `build-index`,
> `references`, `lint`, `graph-analyze`, `export-graph` all work offline.

---

## Doctor — Health Check

```bash
llmwikify doctor                     # Check everything (5s LLM test)
llmwikify doctor --skip-llm          # Skip LLM API call (faster)
llmwikify doctor --wiki-root /path   # Check a specific wiki
llmwikify doctor --json              # JSON output for CI/scripts
```

### What it checks

| # | Check | What |
|---|-------|------|
| 1 | **Config** | `~/.llmwikify/llmwikify.json` exists, parseable, has valid `api_key` |
| 2 | **Python** | Version >= 3.10 |
| 3 | **Core deps** | llmwikify, yaml, duckdb, jinja2 |
| 4 | **Optional extras** | fastapi, fastmcp, watchdog, networkx, markitdown, tiktoken, httpx |
| 5 | **LLM connectivity** | Actually calls provider API (5s timeout) with `"Say hi"` — verifies key works |
| 6 | **Wiki directory** | 4 functional paths (`raw/`, `wiki/`, `.llmwikify.db`, `wiki.md`) + all page-type subdirs declared in `wiki.md` (parsed dynamically from `Directory Structure` + `Page Types` sections). See [`docs/DOCTOR.md`](docs/DOCTOR.md#3-wiki-check-the-interesting-one) |
| 7 | **Permissions** | `~/.llmwikify/` and wiki root are writable |
| 8 | **WebUI bundle** | `ui/webui/dist/index.html` exists |
| 9 | **Server** | `GET /api/health` returns 200 |

### Severity and fix

Every fail or actionable warn carries a **fix dict** (commands + docs +
cost + risk + auto). In text mode these are aggregated at the bottom
under **📋 Recommended actions**:

```
📋 Recommended actions:

  [FAIL 1/1] wiki (4 functional paths missing)
      Fix:   llmwikify init
      Docs:  docs/ONBOARDING.md#init
      Cost:  ~5s    Risk: low    Auto: True
```

In JSON mode (`--json`), each check has a `fix` field on actionable
items. Severity: **FAIL** blocks exit 0; **WARN** and **INFO** do not.

### Exit codes

| Code | Meaning |
|------|---------|
| 0 | All checks passed (warnings/info allowed) |
| 1 | One or more checks failed |
| 2 | Config missing — run `llmwikify init-llm` |

### Examples

```bash
# CI integration (JSON + skip LLM)
llmwikify doctor --json --skip-llm | jq -e '.summary.failed == 0'

# Quick check before running expensive operations
llmwikify doctor --skip-llm  # ~1 second

# Check a specific wiki
llmwikify doctor --wiki-root /home/ll/Public/comovement
```

For the full reference (all 9 checks, fix dict schema, troubleshooting,
CI recipes), see [`docs/DOCTOR.md`](docs/DOCTOR.md).

---

## Tutorial

New to llmwikify? Start with our **5 end-to-end scenarios**:

| # | Scenario | Description |
|---|----------|-------------|
| 1 | Personal Reading Notes | PDF → searchable wiki with cross-references |
| 2 | Company Due-Diligence KB | Multi-source analysis → knowledge graph |
| 3 | Multi-Wiki Collaboration | Manage multiple wikis through one server |
| 4 | Chat + ReAct Agent | LLM-powered Q&A with tool calling |
| 5 | Quant Reproduction | Paper → Factor → Backtest → L5 reflection |

📖 **Full tutorial**: [`docs/TUTORIAL.md`](docs/TUTORIAL.md) (40-60 min read)
🎯 **Runnable examples**: [`examples/`](examples/README.md) (8 playbooks, no LLM required)

---

## Features at a Glance

| Feature | Description |
|---------|-------------|
| **Wiki Core** | FTS5 search, bidirectional references, query compounding, multi-wiki registry |
| **Smart Lint** | Broken links, orphans, contradictions, outdated pages, knowledge gaps |
| **Knowledge Graph** | 8 relation types, PageRank, community detection, HTML/SVG/GraphML export |
| **Chat + Agent** | ReAct streaming, 26 MCP tools, skills system, research engine |
| **Research Assistant** | Adaptive ReAct loop, multi-source synthesis, quality gates (general purpose) |
| **Web UI** | React SPA: editor, graph, dashboard, chat pages |
| **Extraction** | PDF, Word, Excel, PowerPoint, images, audio, web, YouTube |
| **MCP Server** | 26 tools over stdio + HTTP, multi-wiki support |

---

## Architecture

```mermaid
graph TB
    subgraph "Interfaces"
        CLI[CLI]
        MCP[MCP Server]
        REST[REST API]
        UI[Web UI]
    end
    
    subgraph "Apps"
        Wiki[Wiki Service]
        Chat[Chat + ReAct]
        Research[Research Engine]
        Agent[Agent Runtime]
    end
    
    subgraph "Kernel"
        Engine[Wiki Engine]
        Graph[Knowledge Graph]
        Search[Search Engine]
        Storage[SQLite Storage]
    end
    
    subgraph "Foundation"
        LLM[LLM Client]
        Extract[Extractors]
        Config[Configuration]
    end
    
    subgraph "Reproduction"
        Paper[Paper Pipeline]
        Factor[Factor Library]
        Backtest[Backtest Engine]
    end
    
    CLI --> Wiki
    MCP --> Wiki
    REST --> Wiki
    UI --> Chat
    Chat --> Engine
    Research --> LLM
```

> **v0.40 architecture**: strict 4-layer `foundation → kernel → apps →
> interfaces` with single-direction dependencies. The quant module
> (paper / factor / backtest / strategy) was stripped and migrated to
> the sibling project [`quantnodes`](docs/quantnodes.md).
> See [`docs/REFACTORING.md`](docs/REFACTORING.md) for the full
> refactor history, dependency matrix, and cross-project migration.

---

## Installation

```bash
pip install llmwikify              # Core (zero hard deps)
pip install llmwikify[all]         # Full features
pip install llmwikify[web]         # Web UI + REST
pip install llmwikify[mcp]         # MCP server
pip install llmwikify[extractors]  # PDF/Office/media
```

### Optional Extras

| Extra | Purpose |
|-------|---------|
| `extractors` | PDF / Office / images / audio / YouTube via MarkItDown |
| `mcp` | MCP server (`fastmcp`) |
| `watch` | Filesystem watching (`watchdog`) |
| `graph` | Graph visualization + community detection |
| `web` | FastAPI / Starlette / Uvicorn for the unified server |
| `agent` | Scheduler + filelock + DuckDuckGo / Tavily search |
| `llm` | `tiktoken` for token counting |
| `all` | Everything above |

---

## CLI Reference

### Core Wiki Operations

| Command | Description |
|---------|-------------|
| `init` | Initialize a wiki (creates dirs + `wiki.md` schema + `.llmwikify.db`) |
| `init --llm` | Initialize wiki + set up LLM config (one-shot) |
| `init-llm` | Set up LLM config only (auto-detects from `OPENAI_API_KEY` etc.) |
| `write_page` | Write/update a wiki page (`llmwikify write_page "Name" --content "..."`) |
| `read_page` | Read a wiki page (name only, no `.md` suffix) |
| `ingest` | Ingest a source file or URL (PDF requires `[extractors]`) |
| `batch` | Batch ingest a directory of sources |
| `status` | Show wiki stats (page count, index, links) |
| `log` | Record an operation in `log.md` |

### Search & Analysis

| Command | Description |
|---------|-------------|
| `search` | Full-text search (FTS5 backend) |
| `analyze-source` | LLM-powered source analysis and caching |
| `knowledge-gaps` | Detect missing pages, outdated content, redundancy |
| `suggest-synthesis` | LLM: generate cross-source synthesis suggestions |
| `synthesize` | Save query answer as a wiki page |
| `report` | Generate unexpected connections report |

### Link Index & References

| Command | Description |
|---------|-------------|
| `build-index` | Build bidirectional link index from `[[wikilinks]]` |
| `references` | Show inbound/outbound links, broken links, stats |
| `fix-wikilinks` | Fix broken wikilinks by adding directory prefix |

### Knowledge Graph

| Command | Description |
|---------|-------------|
| `graph-analyze` | PageRank, community detection, suggestions |
| `graph-query` | Query graph (neighbors / path / stats / context) |
| `community-detect` | Detect knowledge communities (Leiden/Louvain) |
| `export-graph` | Export visualization (HTML/SVG/GraphML) |

### Multi-Wiki

| Command | Description |
|---------|-------------|
| `wikis` | Multi-wiki management (list / add / remove / scan) |

### Server & MCP

| Command | Description |
|---------|-------------|
| `serve` | Start unified server — MCP + REST + Web UI (alias: `mcp`) |
| `watch` | Watch `raw/` directory for new files, auto-ingest |

### Quant Research

> **v0.40 BREAKING**: `quant-init` and `reproduce` commands removed.
> Quant research pipeline moved to [`quantnodes.research`](https://github.com/sn0wfree/quantnodes) (>=4.0).
> Use `quantnodes reproduce` instead.

### Database & Health

| Command | Description |
|---------|-------------|
| `db` | Database management (stats / list / clean / export) |
| `lint` | Health check (broken links, orphans, contradictions) |
| `doctor` | **System health check** (config, deps, LLM, wiki, permissions, server) |
| `sink-status` | Show query sink buffer status |

### Auth

| Command | Description |
|---------|-------------|
| `auth` | Auth bootstrap (init / create-token / list-tokens / revoke / whoami) |

### QMD Hybrid Search

| Command | Description |
|---------|-------------|
| `qmd` | QMD search engine (status / search / install / embed / mcp) |

> **MCP tools**: 26 `wiki_*` tools available when server runs with `--transport stdio` or HTTP. See [docs/MCP_SETUP.md](docs/MCP_SETUP.md).

---

## Python API

```python
from llmwikify import create_wiki

# Create or open a wiki
wiki = create_wiki("./my-wiki")

# Write a page
wiki.write_page("Python/Singleton", "# Singleton Pattern\nEnsures one instance...")

# Read a page
content = wiki.read_page("Python/Singleton")

# Search
results = wiki.search("singleton", limit=10)

# Inbound/outbound links
inbound = wiki.get_inbound_links("Python/Singleton")
outbound = wiki.get_outbound_links("Python/Singleton")

# Status / lint
status = wiki.status()
lint_result = wiki.lint()

wiki.close()
```

### Run the unified server programmatically

```python
from llmwikify import Wiki
from llmwikify.interfaces.server import WikiServer

wiki = Wiki("./my-wiki")
server = WikiServer(
    wiki,
    api_key="optional-secret",
    enable_mcp=True,
    enable_rest=True,
    enable_webui=True,
)
server.run(host="0.0.0.0", port=8765)
```

---

## MCP Server (26 Tools)

Wiki maintenance and query:

| Tool | Description |
|------|-------------|
| `wiki_init` | Initialize wiki structure |
| `wiki_ingest` | Ingest a source file |
| `wiki_write_page` | Write/update a wiki page |
| `wiki_read_page` | Read a wiki page |
| `wiki_search` | Full-text search (FTS5) |
| `wiki_lint` | Health check |
| `wiki_status` | Status overview |
| `wiki_references` | Page references |
| `wiki_synthesize` | Save query answer as wiki page |
| `wiki_graph` | Graph query / modify |
| `wiki_graph_analyze` | Graph export / detect / report |

Multi-wiki management:

| Tool | Description |
|------|-------------|
| `wiki_list` | List all registered wikis |
| `wiki_switch` | Switch to a different wiki |
| `wiki_register` | Register a new wiki |
| `wiki_search_cross` | Search across multiple wikis |
| `wiki_scan` | Scan directories for wikis |

---

## Documentation

- [`docs/TUTORIAL.md`](docs/TUTORIAL.md) — **5 end-to-end scenarios (must read)**
- [`docs/ONBOARDING.md`](docs/ONBOARDING.md) — New-user guide + onboarding improvements (v0.38+)
- [`examples/`](examples/README.md) — 8 runnable playbooks
- [`ARCHITECTURE.md`](ARCHITECTURE.md) — Layered architecture, modules, data flow
- [`docs/CONFIGURATION_GUIDE.md`](docs/CONFIGURATION_GUIDE.md) — All config options + Doctor
- [`docs/DOCTOR.md`](docs/DOCTOR.md) — `llmwikify doctor` reference: 9 checks, fix dicts, CI integration (v0.40+)
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — Development setup and workflow

---

## Contributing

Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for development
setup, coding standards, and the contribution workflow.

---

## Acknowledgments

- **[llm-wiki-kit](https://github.com/iamsashank09/llm-wiki-kit)** — Original inspiration
- **Andrej Karpathy** — [LLM Wiki Principles](docs/LLM_WIKI_PRINCIPLES.md)
- **Obsidian** — Markdown wiki platform
- **MCP** — Model Context Protocol

---

## License

MIT License — see [LICENSE](LICENSE).

## Contact

- **GitHub**: [@sn0wfree](https://github.com/sn0wfree)
- **Email**: linlu1234567@sina.com
- **Discussions**: [GitHub Discussions](https://github.com/sn0wfree/llmwikify/discussions)
