Metadata-Version: 2.4
Name: docharvest
Version: 11.5.1
Summary: DocHarvest — Turn any documentation site into LLM-ready Markdown, vector RAG JSONL, llms.txt, and styled offline PDFs. Zero-config CLI, React desktop GUI & FastMCP server.
Author-email: Rohan Shetty <shettyrohan2@gmail.com>
License: MIT
Project-URL: Homepage, https://rohannshetty.github.io/DocHarvest/
Project-URL: Documentation, https://github.com/RohannShetty/DocHarvest#-quick-start
Project-URL: Repository, https://github.com/RohannShetty/DocHarvest
Project-URL: Source, https://github.com/RohannShetty/DocHarvest
Project-URL: Issues, https://github.com/RohannShetty/DocHarvest/issues
Project-URL: Changelog, https://github.com/RohannShetty/DocHarvest/blob/main/CHANGELOG.md
Keywords: docharvest,gitbook,documentation,markdown,downloader,ai,rag,llms-txt,mcp-server,vector-db,docusaurus,readthedocs,mintlify,offline-docs,pdf-generator,nextra,vitepress,mkdocs,readme-io
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Documentation
Classifier: Topic :: Text Processing :: Markup
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: beautifulsoup4>=4.11.0
Requires-Dist: markdownify>=0.11.0
Requires-Dist: lxml>=4.9.0
Requires-Dist: mcp>=1.2.0
Requires-Dist: textual>=0.60
Requires-Dist: pyperclip>=1.8.0
Requires-Dist: pywebview>=6.2.1
Requires-Dist: fpdf2>=2.8.8
Provides-Extra: mcp
Requires-Dist: mcp>=1.2.0; extra == "mcp"
Provides-Extra: render
Requires-Dist: playwright>=1.40.0; extra == "render"
Provides-Extra: js
Requires-Dist: playwright>=1.40.0; extra == "js"
Provides-Extra: dev
Requires-Dist: pytest>=9.1.1; extra == "dev"
Requires-Dist: pytest-asyncio>=1.4.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

<div align="center">

<img src="assets/logo-icon.svg" alt="DocHarvest Logo" width="96" />

# DocHarvest

### Compile any docs portal into context your agent can verify

**CLI + Universal Skill first.** MCP is optional when you need a live tool surface.

| `673 pages` | `18.2 s` | `~83% less boilerplate` |
|---:|---:|---:|
| reference capture | reference capture | reference measurement |

[![Version: 11.5.1](https://img.shields.io/badge/version-11.5.1-3f3f46?style=flat-square&labelColor=18181b)](CHANGELOG.md)
[![License: MIT](https://img.shields.io/badge/license-MIT-71717a?style=flat-square&labelColor=18181b)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-a1a1aa?style=flat-square&labelColor=18181b)](pyproject.toml)
[![Tests: 1001 passing](https://img.shields.io/badge/tests-1001%20passing%20(2026--11--13)-f59e0b?style=flat-square&labelColor=18181b)](docs/lib/stats.ts)

> Reference measurement, not a guarantee: 673 pages captured in 18.2 seconds with approximately 83% boilerplate-token reduction. Reproduce the local retrieval benchmark with `python benchmarks/run.py`; replace this capture reference only with a fresh end-to-end measurement.
```bash
pip install docharvest && docharvest capture https://docs.example.com/
```
<br />

<img src="assets/capture_studio.png" alt="DocHarvest Desktop GUI — Capture Studio with shadcn/ui, 60fps motion progress, radial gauge, and live terminal logs" width="920" />

</div>

---

## ⚡ Overview

**Your coding agent doesn't read documentation — it reads web pages.** Navbars, cookie banners, search modals, and footer scripts make up **80–85% of a raw page's bytes** before a single API fact arrives. Chunks captured without source URLs make hallucinations unfalsifiable, and per-page cloud scraping API bills spike the moment you index a real documentation portal.

**DocHarvest** (package: `docharvest`) compiles any technical documentation site into a pristine, structured local knowledge base with verifiable SHA-256 provenance:

- ⚡ **Ultra-Fast Throughput**: Reference capture of **673 pages in 18.2 seconds** (~37 pages/sec) directly into SQLite FTS5 BM25 search and semantic graphs.
- 🎯 **~83% Token Reduction**: Eliminates boilerplate DOM trees, sidebars, anchor spam, and tracking wrappers, delivering high-density context directly to your LLMs.
- 📐 **OpenSpec SDD Native**: Integrated spec-driven development harnesses and prompt commands (`opsx-*`) across 14+ agent environments.
- 🖥️ **Universal Desktop & Browser GUI**: Seamless native window with zero-dependency standalone HTTP/SSE browser fallback across any default system browser.
- 🧠 **Universal Agent Skill (`SKILL.md`)**: Bundled, harness-neutral intelligence that equips 14+ coding agents (Cursor, Claude Code, Windsurf, Gemini CLI, Oh My Pi, Codex) with self-orchestrated harvesting and retrieval workflows via a single install command (`docharvest skill install docharvest -o <dir>`).
- 🔌 **Native FastMCP v2 Server**: 12 zero-friction tools in an opt-in full profile, with a smaller default profile for lower schema cost; MCP Resources (`docs://{domain}/book`), and MCP Prompts over stdio.
- 🔒 **Deterministic SHA-256 Provenance**: Every page is stored with an exact SHA-256 content hash in YAML frontmatter for caching, version diffing, and citation verification.

Whether you are feeding 500-page API manuals to **Cursor / Claude Code**, building vector RAG pipelines with **LangChain & LlamaIndex**, reading offline on an airplane, or archiving technical libraries — every capture ends in the same verifiable shape: clean markdown ready for your agent or your bookshelf.

---

## ⏱️ 30-Second Start

```bash
# 1. Install via pip (FastMCP v2 server & bundled skill included)
pip install docharvest

# 2. Capture a complete docs site with RAG JSONL and offline PDF
docharvest capture https://docs.openalgo.in/ --rag --pdf

# 3. Equip your AI agent with the Universal DocHarvest Skill
docharvest skill install docharvest -o .cursor/skills     # or .claude/skills, .agents/skills...
```

No API key. No account. No telemetry. When the command finishes you own a `book.md`, an `llms.txt`, an `llms-full.txt`, a RAG JSONL dataset, and a printable PDF — all local, all MIT. Full install paths (standalone `.exe`, uvx, optional extras) are in [Quick Start](#-quick-start).

---

## 🧩 Supported Documentation Platforms (8 Real Providers)

DocHarvest features dedicated, priority-ordered parsers that extract clean article content and strip headers, footers, sidebars, anchor hashes, and cookie banners:

| Provider | Priority | Discovery Method | Clean Content Target |
| :--- | :---: | :--- | :--- |
| **GitBook** | `100` | `.md` endpoint probing, sitemap, space discovery | Native markdown or `.page-inner` / `article` |
| **Mintlify** | `90` | `mintlify.json`, OpenAPI specs, CDN asset anchors | `#content`, `#main-content`, `article` |
| **Docusaurus** | `80` | `sitemap.xml`, `docusaurus.config.js`, sidebars | `article`, `.markdown`, `main .theme-doc-markdown` |
| **Nextra** | `75` | `sitemap.xml`, Next.js app routes, nextra scripts | `main.nextra-content`, `article` |
| **VitePress** | `72` | Sitemap, VitePress theme anchors, route index | `div.vp-doc`, `div.VPContent`, `main.VPDoc` |
| **MkDocs** | `70` | `search/search_index.json`, sitemap | `article.md-content__inner`, `div.md-typeset` |
| **ReadMe.io** | `65` | `sitemap.xml`, `/llms.txt`, developer hub routes | `div.rm-Article`, `div.rm-Markdown`, `#content` |
| **ReadTheDocs** | `60` | Sphinx `sitemap.xml`, `div.sphinxsidebar` | `div.document[role="main"]`, `div.body` |
| **Generic HTML / SPA** | `0` | BFS link crawl, `llms.txt`, `sitemap.xml` | `main`, `article`, `[role="main"]`, `#content` |

> [!TIP]
> **Dynamic JavaScript SPAs**: If a site is rendered entirely client-side via JavaScript (such as `omp.sh/docs`), install the optional Playwright extra (`pip install "docharvest[render]" && playwright install chromium`) and run with `--render` to execute JavaScript before extracting markdown.

---

## 🌟 Key Capabilities

- 🤖 **Zero-Noise LLM Context**: Auto-detects 8 documentation frameworks, probes native `.md` endpoints, and cleans DOM trees — measured at ~83% token reduction vs raw pages.
- 📦 **Five-Part Output Contract**: Every capture yields a modular `pages/` tree with SHA-256 YAML frontmatter, a consolidated `book.md` with TOC, `llms.txt`, a full-content `llms-full.txt`, and search index records.
- 🚀 **Export Studio & Local Search**: RAG JSONL for vector databases, pure-Python PDF handbooks (`fpdf2`, zero C-dependencies), and AST markdown chunks — all indexed into embedded SQLite FTS5 BM25 search.
- 🔌 **Native FastMCP v2 Server**: A minimal schema is the default; `docharvest mcp --profile full` exposes all 12 tools plus resources and prompts over stdio. Crash-safe atomic storage and semver snapshot diffing included.

### What DocHarvest Is *Not* For

A tool that claims to do everything has earned none of your trust. Honest scope:

- **Not for login-walled, paywalled, or CAPTCHA-protected content.** DocHarvest is built for public technical documentation and will not bypass access controls.
- **Not for internet-scale crawling.** Millions of arbitrary URLs is Common Crawl / Scrapy territory; this is a documentation compiler, not a search-engine crawler.
- **Not for e-commerce or social feeds.** Product catalogs and social streams are out of scope by design.
- **Not a cloud service.** No dashboard, no subscription, no telemetry — because nothing of yours ever leaves your machine.

---

## 🖥️ Desktop GUI: Document Library & Management

<div align="center">
  <img src="assets/document_library.png" alt="DocHarvest Desktop GUI — Document Library with search, rename, open folder, and multi-format exports" width="920" />
</div>

The desktop application includes a dedicated **Document Library** for managing all harvested technical docs:
- **Instant Search**: Real-time filtering across downloaded portals.
- **In-App Renaming**: Organize project labels without breaking file paths.
- **1-Click Export Studio**: Export selected docsets to Markdown, Vector JSONL, or PDF handbooks directly from the GUI.
- **Direct Finder/Explorer Integration**: Open raw markdown source folders with a single click.

### Reliable capture and local preferences

Queue documentation URLs in **Capture Studio → Batch → Run Batch**. Captures run sequentially with your selected options; failed starts are reported rather than counted as successes. Cancellation waits for the worker to finish, and a stalled batch keeps the worker busy until it actually stops. **Force Reset does not terminate a Python thread** or release a live worker's locks.

Open **Preferences** (`Ctrl+8`, outside form fields) to inspect merged configuration, review presets, and save capture settings to the global or project TOML file. Project settings override global settings. Capture Studio retains its own per-run controls; saved preferences do not automatically populate them.

```bash
docharvest gui                   # native desktop window
docharvest gui --browser         # local browser companion
```

The browser companion validates local origins and Host headers. It is **not an authenticated network service**: keep it on loopback and do not expose its port publicly. See the [GUI guide](docs/GUI_GUIDE.md) for recovery, configuration, development, and regression-test commands.

---

## 📋 The Five-Part Output Contract

**What a generic crawler hands your LLM** (every page, every time):

```html
<nav class="sidebar">…47 links…</nav>
<div class="cookie-banner">We value your privacy…</div>
<main>
  <h1>OAuth 2.0<a class="anchor" href="#oauth2">¶</a></h1>
  <pre><code><span class="token-keyword">import</span> <span class="token-variable">requests</span>…</code></pre>
</main>
```

**What DocHarvest delivers** — the same page, with cryptographic provenance:

````markdown
---
source_url: https://docs.openalgo.in/v/v2.0/api-reference/oauth
title: "OAuth 2.0 Authentication"
content_hash: "sha256-2fa9ca2a57c4e974f1725657f88f757e25b90adee3e18ef809f65932d283746c"
---

# OAuth 2.0 Authentication

## Request Signature

```python
import requests

response = requests.post(
    "https://api.openalgo.in/oauth/token",
    json={"client_id": "pk_live_..."},
)
```
````

The `content_hash` is the real SHA-256 of the markdown body shown above — paste it into any SHA-256 tool and it verifies.

Every crawl produces the same standardized, deterministic directory structure:

```
~/.docharvest/docs/
└── docs.openalgo.in/
    ├── pages/                     # Modular individual markdown files
    │   ├── 001_quickstart.md
    │   └── 002_api_reference.md
    ├── book.md                    # Consolidated single handbook with hierarchical TOC
    ├── llms.txt                   # Standardized AI discovery manifest
    ├── llms-full.txt              # Full-content manifest with per-page hashes
    ├── exports/
    │   ├── openalgo_rag.jsonl     # Tokenized vector chunks + metadata
    │   └── openalgo_handbook.pdf  # Publication-grade printable PDF (pure Python)
    └── .manifest.json             # Crawl metadata, engine version & cryptographic hashes
```

---

## 🚀 Quick Start

### Option 1: Standalone Executable (Zero Setup)
Download **[`docharvest-windows-latest.exe`](https://github.com/RohannShetty/DocHarvest/releases/latest)** from the latest release:
- **Double-click** to launch the **Desktop GUI Application**.
- Or execute directly in your terminal:
  ```powershell
  .\docharvest-windows-latest.exe crawl https://docs.openalgo.in/ --rag --pdf
  ```

### Option 2: Install via pip / PyPI
```bash
# Standard installation (100% local, zero C-dependencies)
pip install docharvest

# Optional headless browser rendering for dynamic JavaScript SPAs
pip install "docharvest[render]"
playwright install chromium

# MCP server: the FastMCP SDK ships in the base install — nothing extra needed.
# (The [mcp] extra is a backward-compatibility no-op; this line still resolves.)
pip install "docharvest[mcp]"

# Launch desktop GUI:
docharvest --gui

# Or run a CLI crawl:
docharvest capture https://docs.openalgo.in/ --rag --pdf
```

### Option 3: Ultra-Fast One-Liner via uv / uvx
```bash
# Launch GUI instantly without permanent installation:
uvx docharvest --gui

# Or install as a global CLI tool:
uv tool install docharvest
```

---

## 💻 CLI Command Reference

```bash
# Basic Documentation Crawl (aliases: capture, dl, crawl)
# `docharvest capture` is the canonical verb; `crawl` remains as a documented alias.
docharvest capture https://docs.openalgo.in/

# Full Compilation (Markdown + RAG JSONL + llms.txt + PDF Handbook)
docharvest capture https://docs.openalgo.in/ --rag --pdf

# Crawl Dynamic Client-Rendered SPAs (Playwright Headless Browser)
docharvest capture https://omp.sh/docs --render

# Restrict Crawl to Specific Path Prefix & Limit Depth
docharvest capture https://docs.example.com/ --scope /api/ --max-pages 50

# Capture a Local Dev Docs Server (host:port domains are fully supported)
docharvest capture http://localhost:3000/docs --output library

# Full-Text BM25 Search across Harvested Docs
docharvest search "OAuth 2.0 authentication token"

# List Harvested Document Domains in Local Library
docharvest ls

# Show Snapshot History & Diff Versions
docharvest history docs.example.com
docharvest diff docs.example.com v1.0.0 v1.0.1

# Start the default minimal FastMCP profile over stdio
docharvest mcp

# Expose all 12 named tools explicitly
docharvest mcp --profile full

# List / Install the Bundled Agent Skill
docharvest skill list
docharvest skill install docharvest -o .claude/skills

# Launch Desktop GUI Application (native window or browser)
docharvest --gui
docharvest gui --browser zen
```

---

## 🧠 Agent Skills: Universal MCP Skill for Every Agent Harness

The FastMCP server gives an agent **tools**; the bundled **docharvest** skill gives it **reasoning & workflow intelligence**. It teaches LLM agents when and how to call tools without redundant network crawls, how to bound context windows to avoid token bloat, and how to verify cryptographic SHA-256 provenance.

One skill, one `SKILL.md`: the matrix below describes **skill discovery directories**, not MCP server configuration. MCP registration paths are client-specific and are covered by `docharvest init --client <name>` where supported.

### 🌐 Universal Skill Matrix (17 documented environments)

Install the bundled skill into any agent's discovery directory in a single command with `docharvest skill install docharvest -o <target_dir>`:

| Agent / Harness | Skill Target Directory | 1-Command Skill Installation | Trigger / Invocation | Status |
| :--- | :--- | :--- | :--- | :--- |
| **Cursor** | `.cursor/skills` | `docharvest skill install docharvest -o .cursor/skills` | Auto-invoked or `@docharvest` | Current |
| **Claude Code** | `.claude/skills` | `docharvest skill install docharvest -o .claude/skills` | `claude` prompt / `/skill:docharvest` | Current |
| **Claude Desktop** | `~/.claude/skills` | `docharvest skill install docharvest -o ~/.claude/skills` | Natural language queries | Current |
| **Devin Desktop / legacy Windsurf** | `devin/` or `.codeium/skills` | `docharvest skill install docharvest -o <target_dir>` | Client-specific | Verify per client |
| **Oh My Pi (OMP)** | `.omp/skills` | `docharvest skill install docharvest -o .omp/skills` | `/skill:docharvest <query>` | Current |
| **VS Code (Copilot/MCP)** | `.agents/skills` | `docharvest skill install docharvest -o .agents/skills` | Chat agent prompt | Current |
| **Antigravity / Gemini CLI** | `.gemini/skills` | `docharvest skill install docharvest -o .gemini/skills` | CLI skill invocation | Current |
| **GitHub Copilot Workspace** | `.github/skills` | `docharvest skill install docharvest -o .github/skills` | Workspace agent prompt | Unverified |
| **JetBrains AI Assistant** | `.agents/skills` | `docharvest skill install docharvest -o .agents/skills` | AI Assistant prompt | Current |
| **Junie** | `.agents/skills` | `docharvest skill install docharvest -o .agents/skills` | Junie CLI prompt | Current |
| **Zed** | `.agents/skills` | `docharvest skill install docharvest -o .agents/skills` | Context server prompt | Current |
| **Cline** | `.agents/skills` | `docharvest skill install docharvest -o .agents/skills` | Task auto-execution | Current |
| **Continue.dev** | `.agents/skills` | `docharvest skill install docharvest -o .agents/skills` | `@docs` context prompt | Legacy / winding down |
| **Kiro** | `.kiro/skills` | `docharvest skill install docharvest -o .kiro/skills` | Kiro agent runner | Current |
| **OpenCode** | `.opencode/skills` | `docharvest skill install docharvest -o .opencode/skills` | Agent command bar | Current |
| **OpenAI Codex CLI** | `.agents/skills` | `docharvest skill install docharvest -o .agents/skills` | CLI assistant prompt | Current |
| **Universal / Standard Agents** | `.agents/skills` | `docharvest skill install docharvest -o .agents/skills` | Standard discovery loader | Current |

### What the Skill Encodes:

- **Autonomous Two-Stage Lookup**: Instructs the agent to check `list_domains` and `find_docs` first, reusing local cached libraries before triggering network crawls.
- **AST-Aware Token Slicing**: Employs `read_doc(domain, topic=..., max_tokens=4000)` to pull precise sub-headings without splitting code blocks or tables.
- **Topological Traversal**: Uses `query_doc_graph` and `get_related_concepts` to navigate prerequisite concepts and API references across complex documentation hierarchies.
- **Automated Version Audits**: Executes `list_versions`, `diff_versions`, and `get_changelog` to analyze documentation drifts and API updates over time.
- **Graceful CLI Seam**: Transparently falls back to `docharvest capture` and `docharvest search` when running in environments without active MCP server connections.

```bash
# Inspect all bundled skills:
docharvest skill list

# Install into your current workspace root:
docharvest skill install docharvest -o .agents/skills
```

---

## 🔌 AI Agent Integration: Native FastMCP v2 Server

DocHarvest includes a native **FastMCP (Model Context Protocol v2)** server over standard input/output (`stdio`). The default `minimal` profile exposes the capture and retrieval path with a smaller schema; `docharvest mcp --profile full` exposes all 12 high-level tools, **MCP Resources**, and **MCP Prompts**. It is compatible with `mcp<2` and `mcp>=2.1`.

> **Zero-Dependency SDK**: The `mcp` SDK ships directly inside the base package. `pip install docharvest` or `uvx docharvest mcp` works out of the box with no extra steps.

Tool profiles are documented in [`docs/TOOL_PROFILES.md`](docs/TOOL_PROFILES.md). Use `full` when an existing workflow needs versioning, exports, changelogs, or graph tools.

**MCP Registry id:** `io.github.RohannShetty/docharvest`

<!-- mcp-name: io.github.RohannShetty/docharvest — ownership token the official MCP registry checks in this README to
     verify ownership of the PyPI package `docharvest`. Do not remove or
     reformat: publishing via mcp-publisher fails without it. -->

### All 12 Native MCP Tools

1. `download_docs(url, max_pages=None, workers=8, path_scope=[], exclude_paths=[], site_versions=None, output_mode="both")`
   Captures any documentation URL into Markdown, `book.md`, `llms.txt`, and `llms-full.txt`.
2. `search_docs(query, domain=None, limit=10, max_tokens=2000)`
   Full-text search across downloaded documentation via SQLite FTS5 BM25; `max_tokens` bounds total returned snippets.
3. `find_docs(query, limit=10)`
   Resolves library/framework names ("react", "nextjs") to indexed domains in the local library.
4. `read_doc(domain, path=None, topic=None, max_tokens=4000, version=None)`
   Reads a specific page or topic section with AST-safe token bounding — code blocks and tables are never split.
5. `get_doc(domain, version=None)`
   Retrieves the compiled documentation content or preview for a domain.
6. `list_domains()`
   Returns metadata for all harvested documentation portals in local storage.
7. `query_doc_graph(domain, query, limit=10)`
   Queries the semantic entity & concept graph to discover connected API endpoints and sections without reading full files.
8. `get_related_concepts(domain, concept)`
   Returns 1-hop and 2-hop connected concepts and prerequisite sections.
9. `diff_versions(domain, v1, v2)`
   Computes unified diffs and line change statistics between two snapshots.
10. `list_versions(domain)`
    Lists available captured snapshots and timestamps for a domain.
11. `export_docs(domain, format="markdown")`
    Exports documentation into `"markdown"`, `"jsonl"`, or `"rag"` metadata formats.
12. `get_changelog(domain)`
    Auto-generates version changelogs across captured snapshot iterations.

### MCP v2 Resources & Prompts
- **Resources**: `docs://{domain}/book` (full handbook) and `docs://{domain}/manifest` (`llms.txt` index). The same library root also contains `llms-full.txt` for full-content ingestion.
- **Prompts**: `prompt://search-docset` (guided docset synthesis), `prompt://summarize-library` (library overview).

---

### Zero-Setup MCP Configuration Snippets (14 Clients)

#### 1. Claude Code
```bash
claude mcp add docharvest docharvest mcp
```
Or in `~/.claude.json`:
```json
{
  "mcpServers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}
```

#### 2. Claude Desktop (`claude_desktop_config.json`)
```json
{
  "mcpServers": {
    "docharvest": {
      "command": "uvx",
      "args": ["docharvest", "mcp"]
    }
  }
}
```

#### 3. Cursor (`.cursor/mcp.json`)
```json
{
  "mcpServers": {
    "docharvest": {
      "command": "python",
      "args": ["-m", "docharvest.mcp"]
    }
  }
}
```

<details>
<summary><strong>11 more client configs — Windsurf · VS Code · JetBrains · Zed · Cline · Continue · Kiro · OpenCode · Oh My Pi · Gemini CLI · Codex CLI</strong></summary>

#### 4. Windsurf (`~/.codeium/windsurf/mcp_config.json`)
```json
{
  "mcpServers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}
```

#### 5. VS Code (`.vscode/mcp.json`)
```json
{
  "servers": {
    "docharvest": {
      "type": "stdio",
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}
```

#### 6. JetBrains AI Assistant / PyCharm / IntelliJ
Configure via **Settings → Tools → Model Context Protocol (MCP)**:
- **Server Name**: `docharvest`
- **Command**: `docharvest`
- **Arguments**: `mcp`

#### 7. Zed (`settings.json`)
```json
{
  "context_servers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}
```

#### 8. Cline (`cline_mcp_settings.json`)
```json
{
  "mcpServers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"],
      "disabled": false,
      "autoApprove": ["search_docs", "get_doc", "list_domains", "query_doc_graph"]
    }
  }
}
```

#### 9. Continue.dev (`config.json`)
```json
{
  "experimental": {
    "modelContextProtocolServers": [
      {
        "transport": {
          "type": "stdio",
          "command": "docharvest",
          "args": ["mcp"]
        }
      }
    ]
  }
}
```

#### 10. Kiro (`.kiro/settings/mcp.json`)
```json
{
  "mcp": {
    "servers": {
      "docharvest": {
        "command": "docharvest",
        "args": ["mcp"]
      }
    }
  }
}
```

#### 11. OpenCode (`opencode.json`)
```json
{
  "mcp": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}
```

#### 12. Oh My Pi (`omp.sh`) (`.omp/mcp.json` project · `~/.omp/agent/mcp.json` user)
```json
{
  "$schema": "https://raw.githubusercontent.com/can1357/oh-my-pi/main/packages/coding-agent/src/config/mcp-schema.json",
  "mcpServers": {
    "docharvest": {
      "type": "stdio",
      "command": "uvx",
      "args": ["docharvest", "mcp"],
      "timeout": 0
    }
  }
}
```

`timeout: 0` disables the client-side request timeout. DocHarvest crawls a
documented site at roughly 0.075 s/page, so a full-site capture can exceed a
client's default timeout (OMP defaults to 30 s) and be aborted mid-crawl; set a
positive millisecond value instead if you prefer a hard ceiling, and use
`max_pages` to bound very large sites.

> **Not for Pi (`pi.dev`)**: Pi keeps its MCP configuration elsewhere, which is
> outside OMP's discovery list — use Pi's own configuration file.

#### 13. Antigravity / Gemini CLI (`mcp/docharvest.json`)
```json
{
  "name": "docharvest",
  "command": "docharvest",
  "args": ["mcp"]
}
```

#### 14. OpenAI Codex CLI (`codex_config.json`)
```json
{
  "mcp_servers": {
    "docharvest": {
      "command": "docharvest",
      "args": ["mcp"]
    }
  }
}
```

</details>

---

## 🐍 Python SDK Example

```python
from docharvest.api import capture, CaptureOptions

# Configure capture options
options = CaptureOptions(
    workers=8,
    output_mode="both",   # "both", "library", or "local"
    render=False,         # set True for dynamic JavaScript SPAs
)

# Execute deterministic capture
result = capture("https://docs.openalgo.in/", options=options)

print(f"Captured {result.pages_captured} pages using provider: {result.provider}")
print(f"Book file: {result.book_file}")
print(f"Manifest:  {result.manifest_file}")
```

---

## 🧪 Test Suite & Quality Benchmarks

DocHarvest is continuously tested across Windows, Linux, and macOS:

- **The full suite is green** (badge above, dated): engine discovery, BFS crawling, provider extraction, storage safety, DocGraph semantic search, MCP v2 tools, the desktop GUI bridge, and the showcase's own content contracts. Reproduce with `uv run pytest`.
- **Numbers nobody regenerates go stale, so these are generated.** `docs/lib/stats.ts` feeds the showcase, and `node docs/scripts/sync-stats.mjs --run` is the only supported way to change them — the site, this file and the real suite are kept in step, and a run with failures is published as failures rather than rounded away.
- **Coverage**: `uv run pytest --cov=docharvest` (see the CI job for the current figure).
- **Windows CRLF Safe**: All link and boilerplate stripping routines are cross-platform normalized against Windows CRLF and Unix LF linebreaks.

To run the test suite locally:
```bash
uv run pytest --cov=docharvest
```

---

## 👨‍💻 Author & Connect

Created with ❤️ by **Rohan Shetty**.

- 🌐 **Website**: [rohannshetty.github.io/DocHarvest](https://rohannshetty.github.io/DocHarvest/)
- 🐙 **GitHub**: [@RohannShetty](https://github.com/RohannShetty)
- 💼 **LinkedIn**: [linkedin.com/in/rohan-shettyy](https://www.linkedin.com/in/rohan-shettyy/)
- 🐦 **X (Twitter)**: [@rohan__shetty](https://x.com/rohan__shetty)
- 📧 **Email**: [shettyrohan2@gmail.com](mailto:shettyrohan2@gmail.com)

---

## 📄 License

This project is licensed under the [MIT License](LICENSE).
