Metadata-Version: 2.4
Name: mantisfetch
Version: 1.9.7
Summary: Open-source data collection and document parsing platform
Author-email: ReadyForAI <opensource@readyfor.ai>
License: MIT
Project-URL: Homepage, https://github.com/ReadyForAI/MantisFetch
Project-URL: Repository, https://github.com/ReadyForAI/MantisFetch
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.115.0
Requires-Dist: uvicorn[standard]>=0.30.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: playwright>=1.45.0
Requires-Dist: markitdown[docx,pdf,pptx,xls,xlsx]>=0.1.0
Requires-Dist: pymupdf>=1.24.0
Requires-Dist: python-docx>=1.1.0
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: lxml>=5.0.0
Requires-Dist: google-genai>=0.8.0
Requires-Dist: openai>=1.0.0
Requires-Dist: Pillow>=10.4.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: mcp<3,>=2
Dynamic: license-file

# MantisFetch

<p align="center">
  <img src="assets/banner.png" alt="MantisFetch Banner" width="100%">
</p>

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
[![CI](https://github.com/ReadyForAI/MantisFetch/actions/workflows/ci.yml/badge.svg)](https://github.com/ReadyForAI/MantisFetch/actions)

[English](#english) | [中文](#中文)

---

## English

**MantisFetch** is an Agent-native data collection and document parsing toolkit by [ReadyForAI](https://github.com/ReadyForAI).

### Why MantisFetch?

Most scrapers hand your Agent a wall of raw HTML — expensive to process and mostly noise. MantisFetch semantically distills web pages into structured sections and actions, cutting token usage by 90%+ compared to raw HTML. A three-tier loading model (digest → brief → section) lets Agents pull only what they need instead of loading entire documents into context. Diff-first incremental reads mean revisiting a page costs near-zero tokens, and table-aware extraction precomputes statistics so Agents can answer data questions without reading a single row.

### Features

- **Semantic distillation** — web pages → structured sections + interactive actions; 4,000–8,000 chars per distill vs. full HTML
- **Three-tier loading** — digest (~200 tokens) → brief (~1,500 tokens) → section (on-demand); Agents load only what they need
- **Diff-first** — `distill` returns only changed sections (`changed_sids`); repeat visits cost near-zero tokens
- **Table-aware** — auto-extracts tables with precomputed stats (min/max/avg); answer data questions without reading rows
- **Document parsing** — PDF, DOCX, PPTX, XLSX, CSV, HTML via [MarkItDown](https://github.com/microsoft/markitdown), with Gemini OCR fallback
- **Multi-LLM support** — Gemini (default), OpenAI, Ollama, Groq, or any OpenAI-compatible API
- **MCP server** — exposes `/web` and `/doc` as native MCP tools (streamable-HTTP at `/mcp`) for agent runtimes; loopback-only by default, with bearer-token + optional TLS for remote clients
- **WebMCP client** — discovers and invokes structured tools that Chrome 146+ pages expose via `navigator.modelContext`
- **i18n** — English and Chinese (set `LANG=zh`)

### Quick Start

#### Docker (recommended)

```bash
git clone https://github.com/ReadyForAI/MantisFetch.git
cd MantisFetch
GEMINI_API_KEY=your_key_here docker compose up -d

# Check health
curl http://localhost:9898/health
```

> `/web` and `/doc` are **loopback-only** without a token. Host requests reach the
> container across the Docker bridge (a non-loopback peer), so set
> `MANTISFETCH_MCP_TOKEN=…` and send `Authorization: Bearer …` to use them; only
> `/health` is exempt.

#### Python (local)

```bash
git clone https://github.com/ReadyForAI/MantisFetch.git
cd MantisFetch
pip install -r requirements.txt
playwright install chromium

export GEMINI_API_KEY=your_key_here
python mantisfetch_server.py     # listens on port 9898
```

### Docker

The `docker-compose.yml` provides a single-service setup. By default, the document library is bind-mounted to the current user's `~/.mantisfetch/docs` directory on the host. See [`DEPLOYMENT.md`](DEPLOYMENT.md) for container hardening, shared-volume ownership (SMB/NFS), and the single-process boundary.

**Image variants (`WITH_LOCAL_OCR` build arg):** the offline PaddleOCR stack is ~1.5 GB of the image (v1.9.1 measured: full 4.66 GB, slim 3.20 GB uncompressed; 1.21 GB vs 0.88 GB as a gzipped `docker save`). Build with or without it:

```bash
# Full — bundled offline OCR (default)
docker build -t readyforai/mantisfetch:latest .

# Slim — ~1.5 GB smaller, no local OCR; OCR runs via the configured LLM provider
docker build --build-arg WITH_LOCAL_OCR=false -t readyforai/mantisfetch:slim .
```

With `WITH_LOCAL_OCR=false` the image bakes `MANTISFETCH_LOCAL_OCR_ENABLED=false`, so all OCR routing skips the (absent) local worker and uses the LLM/vision provider — embedded images **and** scanned-PDF pages (which otherwise have no per-page local→LLM fallback). **An LLM provider must therefore be configured** for the slim image to OCR at all; startup prewarm is skipped to match. Inspect a running image with `docker inspect --format '{{index .Config.Labels "com.readyforai.mantisfetch.local-ocr"}}' <image>`.

```yaml
# docker-compose.yml (excerpt)
services:
  mantisfetch:
    build: .
    ports:
      - "9898:9898"
    volumes:
      - ${MANTISFETCH_HOST_DOCS_DIR:-${HOME}/.mantisfetch/docs}:/root/.mantisfetch/docs
```

**Environment variables (pass via `.env` or `docker compose` `environment` block):**

| Variable | Default | Description |
|---|---|---|
| `MANTISFETCH_LLM_PROVIDER` | `gemini` | LLM backend: `gemini` or `openai` |
| `MANTISFETCH_LLM_VENDOR` | `openai` | Vendor profile for OpenAI-compatible APIs: `openai`, `zhipu`, `kimi`, `aliyun`, `volcengine` |
| `GEMINI_API_KEY` | — | Google Gemini API key |
| `MANTISFETCH_LLM_API_KEY` | — | API key for OpenAI-compatible provider |
| `MANTISFETCH_LLM_BASE_URL` | vendor default | Base URL override for OpenAI-compat provider |
| `MANTISFETCH_LLM_MODEL` | **required** | Model name. No built-in default — one written into the code only stays correct until the vendor retires it |
| `MANTISFETCH_OCR_MODEL` | `MANTISFETCH_LLM_MODEL` | Optional OCR vision model override |
| `MANTISFETCH_OCR_IMAGE_INPUT_MODE` | `data_url` | OCR image serialization mode: `data_url`, `plain_base64`, `remote_url_only` |
| `MANTISFETCH_LLM_EXTRA_BODY_JSON` | — | Optional JSON object merged into text chat request body |
| `MANTISFETCH_OCR_EXTRA_BODY_JSON` | — | Optional JSON object merged into OCR request body |
| `MANTISFETCH_DOCS_DIR` | `~/.mantisfetch/docs` | Document library directory |
| `MANTISFETCH_STORE_SOURCE_FILES` | `true` | Persist uploaded source files under each document's `source/` directory |
| `MANTISFETCH_LIBRARY_RETENTION_DAYS` | `0` | Library-wide retention by `created_at`, in days; `0` = off. Deletes hourly through the same locked path as `DELETE /doc/library/{id}`. Set it ≥ every writer's own retention |
| `MANTISFETCH_RAW_MAX_MD_MB` | `2` | Raw-channel ceiling for markdown, in MiB (2 = 2,097,152 bytes) |
| `MANTISFETCH_RAW_MAX_IMAGE_MB` | `8` | Raw-channel ceiling for images, in MiB (8 = 8,388,608 bytes) |

#### Using a local Ollama model

```bash
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_API_KEY=ollama \
MANTISFETCH_LLM_BASE_URL=http://host.docker.internal:11434/v1 \
MANTISFETCH_LLM_MODEL=llama3 \
docker compose up
```

#### OpenAI-compatible vendor profiles

A vendor profile supplies the `base_url` and the protocol quirks (image encoding,
request-body extras) for an OpenAI-compatible vendor. It does **not** supply a model:
`MANTISFETCH_LLM_MODEL` is required, because a model name written into the code is only
correct until the vendor retires it — and then every call fails, far from the cause.
Startup logs a warning naming the key, and `GET /health` reports which model each role
resolved to. The global default OCR image input mode stays `data_url` for maximum
compatibility.

```bash
# Zhipu: one model for text, another for OCR
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=zhipu \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<their current text model> \
MANTISFETCH_OCR_MODEL=<their current vision model> \
docker compose up

# Kimi: one multimodal model covers both (omit MANTISFETCH_OCR_MODEL)
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=kimi \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<their current multimodal model> \
docker compose up

# Aliyun Bailian
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=aliyun \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<their current text model> \
MANTISFETCH_OCR_MODEL=<their current vision model> \
docker compose up

# Volcengine Ark: set your deployed endpoint/model explicitly
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=volcengine \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=your_endpoint_or_model \
MANTISFETCH_OCR_MODEL=your_vision_endpoint_or_model \
docker compose up
```

Vendor-specific request fields can be injected without code changes:

```bash
MANTISFETCH_OCR_EXTRA_BODY_JSON='{"image_url_detail":"high"}'
```

OCR image serialization can also be switched explicitly:

```bash
MANTISFETCH_OCR_IMAGE_INPUT_MODE=plain_base64
```

`remote_url_only` is recognized but currently fails fast in the built-in OCR pipeline,
because MantisFetch renders pages in-memory and does not yet upload them to a hosted URL.

#### Per-role providers + failover (dual-slot)

The variables above configure **one** provider shared by both roles (summarisation
and OCR). To run the two roles on **different vendors** — e.g. cheap text
summaries on one vendor, a strong vision model for OCR on another — and to fail
over to a backup when a call fails, set `MANTISFETCH_LLM_DEFAULT`. That switches
MantisFetch into the dual-slot scheme:

- Two credential **slots**, `DEFAULT` and `EXTRA`. Each slot's value is the vendor
  tag; its base URL and API key are read from the environment (nothing is baked
  into the image — point a slot at any OpenAI-compatible endpoint by setting its
  `_BASE_URL`).
- Each **role** picks a primary and an optional fallback model, written
  `<vendor>/<model>`. The vendor prefix routes the call to the slot with that tag.
- **Failover**: if the primary model's call fails, the same call is retried once
  against the role's `_FALLBACK` model. A `gemini/<model>` prefix uses the Gemini
  backend; any other prefix uses the OpenAI-compatible backend.

```bash
# Slot DEFAULT = Zhipu (strong OCR VLM); slot EXTRA = DeepSeek (cheap text)
MANTISFETCH_LLM_DEFAULT=zhipu
MANTISFETCH_LLM_DEFAULT_BASE_URL=https://open.bigmodel.cn/api/paas/v4
MANTISFETCH_LLM_DEFAULT_API_KEY=your_zhipu_key
MANTISFETCH_LLM_EXTRA=deepseek
MANTISFETCH_LLM_EXTRA_BASE_URL=https://api.deepseek.com/v1
MANTISFETCH_LLM_EXTRA_API_KEY=your_deepseek_key

# Summaries on DeepSeek, OCR on Zhipu; each with a cross-vendor fallback
MANTISFETCH_SUM_MODEL_DEFAULT=deepseek/deepseek-chat
MANTISFETCH_SUM_MODEL_FALLBACK=zhipu/glm-4.6
MANTISFETCH_OCR_MODEL_DEFAULT=zhipu/glm-4.6v
MANTISFETCH_OCR_MODEL_FALLBACK=deepseek/deepseek-chat
```

| Variable | Description |
|---|---|
| `MANTISFETCH_LLM_DEFAULT` | Vendor tag of the primary slot (enables the dual-slot scheme) |
| `MANTISFETCH_LLM_DEFAULT_BASE_URL` / `_API_KEY` | Base URL + key for the DEFAULT slot |
| `MANTISFETCH_LLM_EXTRA` | Vendor tag of the second slot |
| `MANTISFETCH_LLM_EXTRA_BASE_URL` / `_API_KEY` | Base URL + key for the EXTRA slot |
| `MANTISFETCH_SUM_MODEL_DEFAULT` / `_FALLBACK` | Summary primary / fallback model, `<vendor>/<model>` |
| `MANTISFETCH_OCR_MODEL_DEFAULT` / `_FALLBACK` | OCR primary / fallback model, `<vendor>/<model>` |

Failover triggers on the provider's failure signal (each provider already retries
transient errors internally, then reports a failure sentinel). Leave
`MANTISFETCH_LLM_DEFAULT` unset to keep the legacy single-provider behaviour.

### API

All endpoints are served on port **9898**.

#### Core

| Method | Path | Description |
|---|---|---|
| `GET` | `/health` | Aggregated health check for all services |
| `GET` | `/web/health` | Browser sub-app health |
| `GET` | `/doc/health` | DocReader sub-app health (includes `docs_dir`) |
| `POST` | `/web/capture` | Capture a URL and persist it to the document library |
| `POST` | `/doc/parse` | Upload and parse a document (PDF, DOCX, PPTX, XLSX, CSV, HTML), optionally storing metadata, source file references, and embedded Word image OCR results |

#### Browser Session API

Stateful browser sessions for multi-step web automation.

| Method | Path | Description |
|---|---|---|
| `POST` | `/web/session/new` | Open a new Playwright browser session |
| `POST` | `/web/session/goto` | Navigate the session to a URL |
| `POST` | `/web/session/distill` | Extract structured content from the current page |
| `POST` | `/web/session/read_sections` | Retrieve specific sections by ID from the last distill |
| `POST` | `/web/session/act` | Click, type, select, or scroll an interactive element |
| `POST` | `/web/session/scroll` | Scroll the page up or down by a pixel amount |
| `POST` | `/web/session/navigate` | Go back or forward in the browser history |
| `POST` | `/web/session/webmcp_discover` | Discover WebMCP tools exposed by the current page |
| `POST` | `/web/session/webmcp_invoke` | Invoke a WebMCP tool by name |
| `POST` | `/web/session/export_storage_state` | Export cookies and local storage for session reuse |
| `POST` | `/web/session/close` | Close the session and release browser resources |

#### Document Library API

Access documents stored by `/web/capture` and `/doc/parse`.

| Method | Path | Description |
|---|---|---|
| `GET` | `/doc/library/search` | Search by keyword, tag, and/or file type |
| `GET` | `/doc/library/search_text` | Search full text / sections with snippets and page hints |
| `POST` | `/doc/library/{doc_id}/search_sections` | Search within one document's sections (sid + page provenance) |
| `GET` | `/doc/library/{doc_id}/digest` | Short summary (~200 tokens) |
| `GET` | `/doc/library/{doc_id}/brief` | Extended summary (~1500 tokens) |
| `GET` | `/doc/library/{doc_id}/full` | Full document text |
| `GET` | `/doc/library/{doc_id}/sections` | List all sections with metadata |
| `GET` | `/doc/library/{doc_id}/section/{sid}` | Full text of a single section |
| `POST` | `/doc/library/{doc_id}/sections/batch` | Read multiple sections by sid in one request |
| `GET` | `/doc/library/{doc_id}/table/{table_id}` | Markdown table with column statistics |
| `GET` | `/doc/library/{doc_id}/table/{table_id}/json` | Structured table JSON (cells with row/column spans) |
| `POST` | `/doc/library/{doc_id}/chunks` | Section-boundary chunks for downstream retrieval/RAG |
| `GET` | `/doc/library/{doc_id}/images` | List embedded Word images with anchors and OCR results |
| `GET` | `/doc/library/{doc_id}/image/{image_id}` | Metadata and OCR text for one embedded Word image |
| `GET` | `/doc/library/{doc_id}/image/{image_id}/raw` | Raw image bytes (`variant=rendered\|original`) for visual reads |
| `GET` | `/doc/library/{doc_id}/summary` | Summary status (poll after a deferred parse) |
| `POST` | `/doc/library/{doc_id}/summary` | (Re)schedule or retry summary generation |
| `GET` | `/doc/library/{doc_id}/manifest` | Provenance metadata (source, timestamps, content hash) |
| `GET` | `/doc/library/{doc_id}/source` | The stored original file, served as the media type it was admitted as |
| `GET` | `/doc/library/{doc_id}/source/info` | Original-file metadata (no bytes); `offset`/`limit` read a line window of a text original |

#### MCP

MantisFetch also exposes `/web` and `/doc` as **Model Context Protocol** tools over streamable-HTTP at `/mcp`, for agent runtimes (e.g. NodalOS) that speak MCP. It is **loopback-only by default**; set `MANTISFETCH_MCP_TOKEN` (and optionally TLS) to allow remote clients.

Full API reference: see [`skills/mantisfetch-browser-SKILL.md`](skills/mantisfetch-browser-SKILL.md), [`skills/mantisfetch-docreader-SKILL.md`](skills/mantisfetch-docreader-SKILL.md), and [`skills/mantisfetch-mcp-SKILL.md`](skills/mantisfetch-mcp-SKILL.md).

DocReader notes:

- `POST /doc/parse` accepts an optional `metadata` form field containing a JSON object; shallow scalar values are indexed for later filtering.
- `POST /doc/parse` also accepts an optional `doc_id` form field. Valid values may contain letters, digits, and internal hyphens, for example `DOC-001`, `NBS250321`, or `doc001`. Reusing an existing `doc_id` returns `409` unless `replace=true`.
- `POST /doc/parse` accepts `summary_mode` (`sync` / `defer` / `off`); with `defer`, the summary is generated in the background — poll `GET /doc/library/{doc_id}/summary` for status, and `POST` the same path to retry a failed one.
- `POST /doc/parse` accepts `parse_mode` (`fast` / `accurate` / `full`, default `accurate`) to tune PDF parsing intensity vs. cost.
- `POST /doc/parse` accepts `store_only=true` for the **raw channel**: `md` and `png/jpg/jpeg/gif/webp` are stored as the original file with no parse products at all, and read back via `/doc/library/{doc_id}/source`. The response and manifest carry `kind: "raw"` (parsed documents read `"parsed"`), as do search hits, the sections listing and summary status; such documents 404 on digest/brief/full, 409 on a summary retry, and do not appear in `search_text`. `GET /doc/health` lists the channel's formats as `raw_formats`. A `409` on this channel returns an object under `detail` carrying the existing document's `kind`; the parse channel's `409` is unchanged. The raw and parsed format sets never overlap — a parseable extension asking for `store_only` is a `422`. Requires `MANTISFETCH_STORE_SOURCE_FILES=true`.
- `POST /doc/parse` reports duplicate content on the raw channel as `dedup: "hit"` plus `existing_doc_id`, and still creates the document under the `doc_id` the caller asked for — nothing is merged.
- `GET /doc/library/search` results carry `kind` as well, so a caller can tell a raw hit from a parsed one before reaching for a digest.
- `GET /doc/library/{doc_id}/table/{table_id}/json` returns structured cells; for tables reconstructed from scanned pages, merged cells now carry a recovered `colspan` (the Markdown form is unchanged).
- `POST /doc/parse` and `POST /web/capture` accept `content_type`, one of `General`, `Contract`, `Bid`, or `Knowledge` (case-insensitive on input; persisted in title case); the default is `General`.
- New ingested documents are stored under `${MANTISFETCH_DOCS_DIR}/<content_type>/<doc_id>`, while legacy flat `${MANTISFETCH_DOCS_DIR}/<doc_id>` documents remain readable.
- `GET /doc/library/search` also accepts query params prefixed with `metadata.` for equality-style filtering, for example `metadata.customer=ACME`.
- `GET /doc/library/search` and `GET /doc/library/search_text` accept `content_type` for category-filtered browsing.
- `GET /doc/library/search_text` returns `snippet`, `sid`, `page_range`, `page_start`, and `page_end` to support page-level follow-up actions.
- Parsed document manifests now include `metadata`, `source_file`, and enriched section page bounds.
- If `MANTISFETCH_DOC_ID_STRATEGY=source_filename`, uploaded files may use a sanitized source filename as the document directory name. Unsupported characters are stripped, separators such as spaces / `_` / `.` are normalized to `-`, and the service falls back to `DOC-xxx` when no usable characters remain.

### Web search (optional)

Off by default. Set `MANTISFETCH_SEARCH_PROVIDER` to enable two endpoints:

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/web/search` | Web search — returns ranked `{url, title, snippet, ...}`; captures nothing |
| `POST` | `/web/search_and_capture` | Search, then capture the top N hits (≤3, serial) into the library with search provenance — returns `doc_id`s ready for the three-tier read |

Providers: `searxng` (self-hosted, zero API cost — the recommended default), `tavily`, `bocha` (博查, China), `brave`. A comma-separated `MANTISFETCH_SEARCH_FALLBACK` switches provider only on connection error / 5xx / timeout (a 4xx config error or an empty result does **not** fall back). When unset, `/web/search*` return `404` and the MCP `web_search` / `web_search_capture` tools are not registered.

Zero-cost SearXNG in one command:

```bash
MANTISFETCH_SEARCH_PROVIDER=searxng docker compose --profile search up
```

For **China deployments**, SearXNG's default upstreams (Google, DuckDuckGo) are usually blocked — activate the China preset first (a Baidu / 360 / Sogou / Quark / Bing starting set; **validate it from your own network**): `cp configs/searxng/settings.cn.yml configs/searxng/settings.yml`.

`search_and_capture` reuses the `/web/capture` path with **two** layers of reuse:

1. **URL cache** — `MANTISFETCH_SEARCH_CAPTURE_TTL_HOURS` (default **24**) so repeated research queries do not re-fetch the same top hits. Independent of `MANTISFETCH_CAPTURE_TTL_HOURS` (still `0`/off for plain `/web/capture`).
2. **Content-hash reuse** — after distill, if the body `content_hash` already exists in the library (same article via AMP / tracking-param URLs), the existing `doc_id` is returned and tags/metadata are merged (first-touch for existing metadata keys).

A cache hit keeps the document's original top-level provenance (first-touch): a URL first captured directly stays `source=web_capture`. Filter search-sourced documents with `metadata.source=web_search`, **not** the top-level `source`.

| Variable | Default | Description |
|---|---|---|
| `MANTISFETCH_SEARCH_PROVIDER` | — | `searxng` / `tavily` / `bocha` / `brave`; unset disables search |
| `MANTISFETCH_SEARCH_FALLBACK` | — | Comma-separated fallback chain, e.g. `tavily,searxng` |
| `MANTISFETCH_SEARCH_PROVIDERS` | — | Addressable set for per-request `provider` selection (adds to primary + fallback); lets an agent target one backend per call, e.g. bocha for CN + tavily for EN |
| `MANTISFETCH_SEARXNG_URL` | — | SearXNG instance URL (the `search` compose profile defaults it to `http://searxng:8080`) |
| `MANTISFETCH_SEARCH_API_KEY` | — | Shared API key (fallback for Tavily / Bocha / Brave) |
| `MANTISFETCH_{TAVILY,BOCHA,BRAVE}_API_KEY` | — | Per-provider API key; overrides the shared one (set these to run several API providers at once) |
| `MANTISFETCH_SEARCH_MAX_RESULTS` | `10` | Default result cap (hard max 20) |
| `MANTISFETCH_SEARCH_MIN_INTERVAL_SEC` | `2` | Minimum seconds between searches. A search that arrives early waits its turn rather than failing |
| `MANTISFETCH_SEARCH_MAX_WAIT_SEC` | `30` | How long a search may wait for its turn before `429` instead. A burst of `max_wait / interval` searches is absorbed, plus the one that does not wait at all — 16 at the defaults; `0` = never wait |
| `MANTISFETCH_SEARCH_CAPTURE_TTL_HOURS` | `24` | URL-level reuse window for `/web/search_and_capture` only; `0` disables URL reuse on that path (content-hash reuse still applies) |

### Configuration

MantisFetch is configured entirely through environment variables. See the table in the **Docker** section above for LLM settings. Additional variables:

| Variable | Default | Description |
|---|---|---|
| `HOST` | `0.0.0.0` | Bind address |
| `PORT` | `9898` | HTTP listening port |
| `LANG` | `en` | UI language (`en` or `zh`) |
| `MANTISFETCH_TLS_CERTFILE` | — | TLS certificate path; set **with** `MANTISFETCH_TLS_KEYFILE` to serve HTTPS (setting only one is ignored) |
| `MANTISFETCH_TLS_KEYFILE` | — | TLS private key path (paired with `MANTISFETCH_TLS_CERTFILE`) |
| `MANTISFETCH_MCP_TOKEN` | — | Bearer token for the `/mcp`, `/web`, `/doc` and `/deliverables` surfaces; without it they are **loopback-only** (non-loopback callers get 403) |
| `MANTISFETCH_OWNER_SCOPED_DISCOVERY` | off | When `1`/`true`/`yes`/`on`, library search returns only the caller's own uploads, documents with `shared: true`, and web captures. The process refuses to start unless `MANTISFETCH_MCP_TOKEN` is set, and `/web` `/doc` `/deliverables` then require that bearer even from loopback (health checks stay open). Uploads without an explicit `doc_id` get an unguessable `R-<hex>` id |
| `MANTISFETCH_MCP_ALLOWED_HOSTS` | — | Extra hosts/origins (comma-separated) for the DNS-rebinding guard on `/mcp` and on token-less loopback `/web` `/doc` requests |
| `MANTISFETCH_MCP_PARSE_BUDGET_SEC` | `45` | Seconds an MCP `doc_parse` or `web_search_capture` call may take before it answers with what fits (a refusal, or a partial capture). Sized under the MCP client's per-request timeout — NodalOS caps upstream calls at 60 s — because a call that outlives it is dropped as a transport failure, which takes every tool of this server off the client until it reconnects. Raise it only with that cap |
| `MANTISFETCH_MCP_RESULT_BUDGET_BYTES` | `60000` | How large one MCP tool result may be on the wire (measured ASCII-escaped, as the 2026-07-28 face sends it). Larger results are paged or windowed (`doc_manifest`, `doc_sections`, `doc_tables`, `doc_full`, `doc_table`, `doc_chunks`, `doc_source`), and any other tool over it answers with a tool error saying how to ask instead — never a result the client cuts off mid-way. The default sits under NodalOS's 65,536-byte cut; a deployment whose client has no such limit can raise it. There is no off: 0 or an invalid value means the default |
| `MANTISFETCH_ALLOWED_DOC_ROOTS` | — | Allowlist roots for the MCP `doc_parse` `rel_path` source; unset disables local-path parsing over MCP |
| `MANTISFETCH_DELIVERABLES_ROOT` | — | Fence root for the read-only `GET /deliverables/{rel_path}` byte face; unset disables it (every request 404s). Must not overlap `MANTISFETCH_DOCS_DIR` or `MANTISFETCH_ALLOWED_DOC_ROOTS` |
| `MANTISFETCH_DELIVERABLES_MAX_MB` | `200` | Size cap for a single deliverable download; larger files get 413 |
| `MANTISFETCH_DOC_ID_STRATEGY` | `counter` | Document directory naming strategy: `counter` keeps `DOC-xxx`; `source_filename` derives a safe directory name from the uploaded filename stem |
| `MANTISFETCH_PARSE_QUEUE_MAX_WAIT_SEC` | `600` | How long a parse that declared no `budget_seconds` waits for a slot before `429`. One that did declare a budget waits inside it instead. Concurrency buys no parse throughput — text parsing is GIL-bound and OCR is already internally parallel — so the wait costs a caller nothing it would not have spent anyway |
| `MANTISFETCH_PARSE_QUEUE_MAX_BYTES` | ten times the upload cap (**2GB** at the default 200MB) | Staged upload bytes the parse queue may hold. A queued parse keeps its upload on disk, so the queue is bounded by disk rather than by count. The same budget also covers uploads still being received: a `/parse` request is admitted by its `Content-Length` (the largest request size if it declares none) against what is queued plus what is arriving, before any of its body is read, and gets `429` if it does not fit — so this one number bounds all upload bytes the service holds |
| `MANTISFETCH_MAX_PARSE_ROWS` | `100000` | Rows an XLSX may span across all its sheets — each sheet counts up to its highest row number, because conversion fills in the empty rows between filled ones. Counted by streaming the sheet XML before conversion; a workbook over it gets `422` and nothing is stored. It is a memory bound: conversion holds the whole workbook, and 100k rows × 10 columns measured about 2GB RSS. `.xls` is not counted |
| `MANTISFETCH_MAX_PARSE_CELLS` | `1000000` | The same bound across columns: each XLSX sheet counts as its rows × its widest column, summed over sheets, checked in the same pass with the same `422`. Without it two cells in A1 and XFD200 were a 4.8 KB file that took 80 s and 3.5 GB to convert. The default is the grid the row default was sized for; a long *and* wide sheet (e.g. 50k rows × 30 columns) needs it raised |
| `MANTISFETCH_MAX_CONCURRENT_CAPTURE` | `16` | Captures in flight at once. Measured saturation point: throughput holds at ~4.5 captures/second from 16 onward, so more concurrency buys latency, not work. Each in-flight capture costs ~6MB |
| `MANTISFETCH_MAX_CONCURRENT_SESSIONS` | `20` | Session creations in flight at once. Creation is cheap (0.03s idle, 0.29s with 200 live), so this bounds a burst rather than a resource |
| `MANTISFETCH_CONCURRENCY_MAX_WAIT_SEC` | `30` | How long a capture or session creation waits for a slot before `429`. At the saturated rate 30s absorbs ~135 queued captures — five agents fetching eight pages each finish in ~9s, ten agents in ~19s, none refused. `0` = refuse immediately |
| `MANTISFETCH_SESSION_MAXSIZE` | `200` | Live sessions kept before the oldest is evicted. **This is a memory budget**: a live session holds a browser context and costs ~15MB, so 200 is ~3GB resident. Size it for the box you deploy on |
| `MANTISFETCH_SESSION_TTL_SEC` | `1800` | Idle session lifetime before it is closed |
| `MANTISFETCH_CAPTURE_TTL_HOURS` | `0` | Reuse a prior `/web/capture` of the same URL + content_type made within this many hours instead of re-fetching; `0` disables (default). `force_refresh=true` always bypasses. Content-hash library reuse still applies after distill |
| `MANTISFETCH_SEARCH_CAPTURE_TTL_HOURS` | `24` | Same as above, but for `/web/search_and_capture` only (default on so research queries do not pile duplicate docs) |
| `MANTISFETCH_SUMMARY_BATCH_CONCURRENCY` | `1` | Maximum concurrent section-summary LLM batches per document |
| `MANTISFETCH_SUMMARY_REQUEST_MIN_INTERVAL_SEC` | `2.0` | Minimum spacing between summary LLM requests across the service |
| `MANTISFETCH_SUMMARY_SECTION_DETAIL_LIMIT` | `10` | Documents above this section count skip per-section LLM summaries and generate document-level summaries from section excerpts |

### For AI Agents

MantisFetch is designed to be deployed and operated autonomously by AI Agents. A self-contained deployment prompt (English + Chinese) is available at [`docs/agent-deployment-prompt.md`](docs/agent-deployment-prompt.md) — copy it directly into your Agent's system prompt, no modification needed.

### Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, code conventions, and the PR process.

### License

MIT — see [LICENSE](LICENSE).

---

## 中文

**MantisFetch** 是由 [ReadyForAI](https://github.com/ReadyForAI) 开源的 Agent 原生数据采集与文档解析工具包。

### 为什么选择 MantisFetch？

大多数爬虫工具返回原始 HTML——对 LLM 而言噪音极多、token 消耗过高。MantisFetch 专为 AI Agent 设计：网页经过语义蒸馏，转化为结构化段落（sections）和可交互元素（actions），相比原始 HTML 节省 90%+ 的 token。三层加载模型（digest → brief → section）让 Agent 按需取用，无需把整篇文档塞进上下文。增量读取（diff-first）机制确保重复访问同一页面时的额外开销接近于零；表格自动预计算统计值，Agent 无需逐行读取即可回答数据问题。

### 功能特性

- **语义蒸馏** — 网页 → 结构化段落 + 可交互元素；每次 distill 仅 4,000–8,000 字符，对比原始 HTML 节省 90%+ token
- **三层加载** — digest（约 200 token）→ brief（约 1,500 token）→ section（按需）；Agent 只取所需
- **增量读取** — `distill` 仅返回变化段落（`changed_sids`），重复访问近零 token 开销
- **表格感知** — 自动提取并预计算统计值（min/max/avg），无需读取原始行即可回答数据问题
- **文档解析** — 支持 PDF、DOCX、PPTX、XLSX、CSV、HTML，基于 [MarkItDown](https://github.com/microsoft/markitdown)（微软），可自动 Gemini OCR 兜底
- **多 LLM 支持** — Gemini（默认）、OpenAI、Ollama、Groq，以及任意 OpenAI 兼容接口
- **MCP 服务** — 把 `/web` 和 `/doc` 暴露为原生 MCP 工具（streamable-HTTP，挂载在 `/mcp`），供 Agent 运行时使用；默认仅 loopback，远程客户端需 bearer token + 可选 TLS
- **WebMCP 客户端** — 发现并调用 Chrome 146+ 页面通过 `navigator.modelContext` 暴露的结构化工具
- **多语言** — 支持中英文（设置 `LANG=zh` 切换为中文）

### 快速上手

#### Docker（推荐）

```bash
git clone https://github.com/ReadyForAI/MantisFetch.git
cd MantisFetch
GEMINI_API_KEY=your_key_here docker compose up -d

# 检查服务状态
curl http://localhost:9898/health
```

#### Python（本地运行）

```bash
git clone https://github.com/ReadyForAI/MantisFetch.git
cd MantisFetch
pip install -r requirements.txt
playwright install chromium

export GEMINI_API_KEY=your_key_here
python mantisfetch_server.py     # 监听 9898 端口
```

### Docker 配置

`docker-compose.yml` 提供单服务部署方案。默认会把文档库 bind mount 到宿主机当前用户的 `~/.mantisfetch/docs` 目录。

**镜像变体（`WITH_LOCAL_OCR` 构建参数）：** 离线 PaddleOCR 栈约占镜像 1.5 GB（v1.9.1 实测：未压缩时完整版 4.66 GB、精简版 3.20 GB；`docker save` 再 gzip 后分别为 1.21 GB 与 0.88 GB）。可选择带或不带：

```bash
# 完整版 —— 内置离线 OCR（默认）
docker build -t readyforai/mantisfetch:latest .

# 精简版 —— 体积小约 1.5 GB，不含本地 OCR；OCR 走配置的 LLM provider
docker build --build-arg WITH_LOCAL_OCR=false -t readyforai/mantisfetch:slim .
```

设 `WITH_LOCAL_OCR=false` 时镜像会烙入 `MANTISFETCH_LOCAL_OCR_ENABLED=false`，所有 OCR 路由都跳过（不存在的）本地 worker、改用 LLM/vision provider——包括内嵌图片**以及扫描版 PDF 页**（后者本身没有逐页 local→LLM 兜底）。因此精简版**必须配置 LLM provider** 才能做 OCR；启动期 prewarm 也随之跳过。用 `docker inspect --format '{{index .Config.Labels "com.readyforai.mantisfetch.local-ocr"}}' <image>` 查看某镜像是哪种变体。

```yaml
# docker-compose.yml（节选）
services:
  mantisfetch:
    build: .
    ports:
      - "9898:9898"
    volumes:
      - ${MANTISFETCH_HOST_DOCS_DIR:-${HOME}/.mantisfetch/docs}:/root/.mantisfetch/docs
```

**环境变量（通过 `.env` 文件或 `docker compose` 的 `environment` 块传入）：**

| 变量 | 默认值 | 说明 |
|---|---|---|
| `MANTISFETCH_LLM_PROVIDER` | `gemini` | LLM 后端：`gemini` 或 `openai` |
| `MANTISFETCH_LLM_VENDOR` | `openai` | OpenAI 兼容厂商配置：`openai`、`zhipu`、`kimi`、`aliyun`、`volcengine` |
| `GEMINI_API_KEY` | — | Google Gemini API Key |
| `MANTISFETCH_LLM_API_KEY` | — | OpenAI 兼容接口的 API Key |
| `MANTISFETCH_LLM_BASE_URL` | 厂商默认值 | 覆盖 OpenAI 兼容接口的 Base URL |
| `MANTISFETCH_LLM_MODEL` | **必填** | 模型名称。没有内置默认值 —— 写死在代码里的名字只在厂商下架它之前是对的 |
| `MANTISFETCH_OCR_MODEL` | `MANTISFETCH_LLM_MODEL` | 可选：单独指定 OCR 视觉模型 |
| `MANTISFETCH_OCR_IMAGE_INPUT_MODE` | `data_url` | OCR 图片序列化模式：`data_url`、`plain_base64`、`remote_url_only` |
| `MANTISFETCH_LLM_EXTRA_BODY_JSON` | — | 可选：合并到文本请求体中的 JSON 对象 |
| `MANTISFETCH_OCR_EXTRA_BODY_JSON` | — | 可选：合并到 OCR 请求体中的 JSON 对象 |
| `MANTISFETCH_DOCS_DIR` | `~/.mantisfetch/docs` | 文档库存储目录 |
| `MANTISFETCH_STORE_SOURCE_FILES` | `true` | 是否将上传原件保存在每个文档目录下的 `source/` 子目录 |
| `MANTISFETCH_LIBRARY_RETENTION_DAYS` | `0` | 库级保留天数（按 `created_at`），`0` = 关闭；每小时经与 `DELETE /doc/library/{id}` 相同的加锁路径删除。应 ≥ 各写入方自己的保留期 |
| `MANTISFETCH_RAW_MAX_MD_MB` | `2` | 原件通道 markdown 上限，单位 MiB（2 = 2,097,152 字节） |
| `MANTISFETCH_RAW_MAX_IMAGE_MB` | `8` | 原件通道图片上限，单位 MiB（8 = 8,388,608 字节） |

#### 使用本地 Ollama 模型

```bash
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_API_KEY=ollama \
MANTISFETCH_LLM_BASE_URL=http://host.docker.internal:11434/v1 \
MANTISFETCH_LLM_MODEL=llama3 \
docker compose up
```

#### OpenAI 兼容厂商配置

厂商 profile 提供 OpenAI 兼容厂商的 `base_url` 与协议细节（图片编码、请求体附加项），**但不提供模型名**：
`MANTISFETCH_LLM_MODEL` 是必填的 —— 写死在代码里的模型名只在厂商下架它之前是对的，之后每次调用都会失败，
而且失败点离原因很远。启动时会打一条点名该键的告警，`GET /health` 也会报出每个角色实际解析到的模型。
OCR 图片输入模式的全局默认值仍固定为 `data_url`，优先保证兼容面。

```bash
# 智谱：文本一个模型、OCR 另一个
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=zhipu \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<该厂商当前的文本模型> \
MANTISFETCH_OCR_MODEL=<该厂商当前的视觉模型> \
docker compose up

# Kimi：一个多模态模型两用（不设 MANTISFETCH_OCR_MODEL 即可）
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=kimi \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<该厂商当前的多模态模型> \
docker compose up

# 阿里百炼
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=aliyun \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<该厂商当前的文本模型> \
MANTISFETCH_OCR_MODEL=<该厂商当前的视觉模型> \
docker compose up

# 火山方舟：通常需要显式填写你的推理接入点 / 模型名
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=volcengine \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=your_endpoint_or_model \
MANTISFETCH_OCR_MODEL=your_vision_endpoint_or_model \
docker compose up
```

如果厂商需要额外请求参数，也可以直接透传 JSON：

```bash
MANTISFETCH_OCR_EXTRA_BODY_JSON='{"image_url_detail":"high"}'
```

也可以显式切换 OCR 图片输入模式：

```bash
MANTISFETCH_OCR_IMAGE_INPUT_MODE=plain_base64
```

其中 `remote_url_only` 目前只做了显式报错路径：MantisFetch 现在是把 PDF 页渲染到内存里直接上送，还没有内建“先上传图片再传远程 URL”的流程。

#### 分角色 provider + 故障切换（双槽位）

上面的变量配置的是**一个** provider，摘要和 OCR 共用。若想让两个角色跑在**不同厂商**上（比如摘要用便宜的文本模型、OCR 用能力强的视觉模型），并在调用失败时切到备用，设置 `MANTISFETCH_LLM_DEFAULT` 即可切换到双槽位方案：

- 两个凭据**槽位** `DEFAULT` 和 `EXTRA`。槽位的值是厂商标签，其 base_url 和 API key **从环境变量读取**（不写死进镜像——给槽位设 `_BASE_URL` 即可指向任意 OpenAI 兼容端点）。
- 每个**角色**各配一个主用和一个可选的备用模型，写成两段式 `<厂商>/<模型>`。前段厂商标签决定路由到哪个槽位。
- **故障切换**：主用模型调用失败时，同一次调用会用该角色的 `_FALLBACK` 模型再试一次。前段是 `gemini/<模型>` 走 Gemini 后端，其它走 OpenAI 兼容后端。

```bash
# 槽位 DEFAULT = 智谱（视觉 OCR 强）；槽位 EXTRA = DeepSeek（文本便宜）
MANTISFETCH_LLM_DEFAULT=zhipu
MANTISFETCH_LLM_DEFAULT_BASE_URL=https://open.bigmodel.cn/api/paas/v4
MANTISFETCH_LLM_DEFAULT_API_KEY=你的智谱key
MANTISFETCH_LLM_EXTRA=deepseek
MANTISFETCH_LLM_EXTRA_BASE_URL=https://api.deepseek.com/v1
MANTISFETCH_LLM_EXTRA_API_KEY=你的deepseek_key

# 摘要走 DeepSeek，OCR 走智谱；各自配一个跨厂商备用
MANTISFETCH_SUM_MODEL_DEFAULT=deepseek/deepseek-chat
MANTISFETCH_SUM_MODEL_FALLBACK=zhipu/glm-4.6
MANTISFETCH_OCR_MODEL_DEFAULT=zhipu/glm-4.6v
MANTISFETCH_OCR_MODEL_FALLBACK=deepseek/deepseek-chat
```

| 变量 | 说明 |
|---|---|
| `MANTISFETCH_LLM_DEFAULT` | 主槽位的厂商标签（设置后即启用双槽位方案） |
| `MANTISFETCH_LLM_DEFAULT_BASE_URL` / `_API_KEY` | DEFAULT 槽位的 base_url + key |
| `MANTISFETCH_LLM_EXTRA` | 第二个槽位的厂商标签 |
| `MANTISFETCH_LLM_EXTRA_BASE_URL` / `_API_KEY` | EXTRA 槽位的 base_url + key |
| `MANTISFETCH_SUM_MODEL_DEFAULT` / `_FALLBACK` | 摘要的主用 / 备用模型，`<厂商>/<模型>` |
| `MANTISFETCH_OCR_MODEL_DEFAULT` / `_FALLBACK` | OCR 的主用 / 备用模型，`<厂商>/<模型>` |

切换在 provider 报出失败信号时触发（每个 provider 自身已先重试瞬时错误，再返回失败哨兵）。不设 `MANTISFETCH_LLM_DEFAULT` 则保持原有的单 provider 行为。

### API 接口

所有接口均运行在 **9898** 端口。

#### 核心接口

| 方法 | 路径 | 说明 |
|---|---|---|
| `GET` | `/health` | 全服务聚合健康检查 |
| `GET` | `/web/health` | Browser 子服务健康检查 |
| `GET` | `/doc/health` | DocReader 子服务健康检查（含 `docs_dir`） |
| `POST` | `/web/capture` | 抓取 URL 并保存到文档库 |
| `POST` | `/doc/parse` | 上传并解析文档（PDF、DOCX、PPTX、XLSX、CSV、HTML），可附带 metadata、保留原始文件引用，并可抽取 Word 内嵌图片 OCR 结果 |

#### Browser Session API

有状态浏览器会话，支持多步骤网页自动化。

| 方法 | 路径 | 说明 |
|---|---|---|
| `POST` | `/web/session/new` | 打开新的 Playwright 浏览器会话 |
| `POST` | `/web/session/goto` | 在当前会话中导航到指定 URL |
| `POST` | `/web/session/distill` | 从当前页面提取结构化内容 |
| `POST` | `/web/session/read_sections` | 按 ID 获取上次 distill 的指定章节 |
| `POST` | `/web/session/act` | 对交互元素执行点击、输入、选择或滚动 |
| `POST` | `/web/session/scroll` | 按像素上下滚动页面 |
| `POST` | `/web/session/navigate` | 浏览器前进或后退 |
| `POST` | `/web/session/webmcp_discover` | 发现当前页面暴露的 WebMCP 工具 |
| `POST` | `/web/session/webmcp_invoke` | 按名称调用 WebMCP 工具 |
| `POST` | `/web/session/export_storage_state` | 导出 Cookie 和 LocalStorage 以复用会话 |
| `POST` | `/web/session/close` | 关闭会话并释放浏览器资源 |

#### Document Library API

访问由 `/web/capture` 和 `/doc/parse` 存入文档库的文档。

| 方法 | 路径 | 说明 |
|---|---|---|
| `GET` | `/doc/library/search` | 按关键词、标签和/或文件类型搜索 |
| `GET` | `/doc/library/search_text` | 按全文 / section 搜索，并返回片段与页码提示 |
| `POST` | `/doc/library/{doc_id}/search_sections` | 在单个文档的 section 内搜索（返回 sid + 页码 provenance） |
| `GET` | `/doc/library/{doc_id}/digest` | 简短摘要（约 200 token） |
| `GET` | `/doc/library/{doc_id}/brief` | 详细摘要（约 1500 token） |
| `GET` | `/doc/library/{doc_id}/full` | 完整文档正文 |
| `GET` | `/doc/library/{doc_id}/sections` | 列出所有章节及元数据 |
| `GET` | `/doc/library/{doc_id}/section/{sid}` | 单个章节的完整文本 |
| `POST` | `/doc/library/{doc_id}/sections/batch` | 一次请求按 sid 读取多个章节 |
| `GET` | `/doc/library/{doc_id}/table/{table_id}` | 带列统计的 Markdown 表格 |
| `GET` | `/doc/library/{doc_id}/table/{table_id}/json` | 结构化表格 JSON（单元格含行/列跨度） |
| `POST` | `/doc/library/{doc_id}/chunks` | 面向下游检索/RAG 的 section 边界分块 |
| `GET` | `/doc/library/{doc_id}/images` | 列出 Word 内嵌图片、锚点和 OCR 结果 |
| `GET` | `/doc/library/{doc_id}/image/{image_id}` | 单个 Word 内嵌图片的元数据和 OCR 文本 |
| `GET` | `/doc/library/{doc_id}/image/{image_id}/raw` | 图片原始字节（`variant=rendered\|original`），用于视觉读图 |
| `GET` | `/doc/library/{doc_id}/summary` | 摘要状态（延迟解析后轮询） |
| `POST` | `/doc/library/{doc_id}/summary` | （重新）调度或重试摘要生成 |
| `GET` | `/doc/library/{doc_id}/manifest` | 来源元数据（来源地址、时间戳、内容哈希） |
| `GET` | `/doc/library/{doc_id}/source` | 入库原件字节，按入库时记录的 media type 返回 |
| `GET` | `/doc/library/{doc_id}/source/info` | 原件元数据（不返字节）；带 `offset`/`limit` 时返回文本原件的行窗口 |

#### MCP

MantisFetch 还把 `/web` 和 `/doc` 暴露为 **Model Context Protocol** 工具（streamable-HTTP，挂载在 `/mcp`），供使用 MCP 的 Agent 运行时（如 NodalOS）使用。**默认仅 loopback**；设置 `MANTISFETCH_MCP_TOKEN`（并可选启用 TLS）以允许远程客户端。

完整 API 说明见 [`skills/mantisfetch-browser-SKILL.md`](skills/mantisfetch-browser-SKILL.md)、[`skills/mantisfetch-docreader-SKILL.md`](skills/mantisfetch-docreader-SKILL.md) 和 [`skills/mantisfetch-mcp-SKILL.md`](skills/mantisfetch-mcp-SKILL.md)。

DocReader 补充说明：

- `POST /doc/parse` 支持可选 `metadata` 表单字段，值为 JSON object；其中浅层标量字段会进入索引，供后续过滤。
- `POST /doc/parse` 也支持可选 `doc_id` 表单字段；合法值可包含字母、数字和中间连字符，例如 `DOC-001`、`NBS250321`、`doc001`。复用已有 `doc_id` 会返回 `409`，除非 `replace=true`。
- `POST /doc/parse` 支持 `summary_mode`（`sync` / `defer` / `off`）；用 `defer` 时摘要在后台生成 —— 轮询 `GET /doc/library/{doc_id}/summary` 查看状态，`POST` 同一路径可重试失败的摘要。
- `POST /doc/parse` 支持 `parse_mode`（`fast` / `accurate` / `full`，默认 `accurate`），用于在 PDF 解析强度与成本间权衡。
- `POST /doc/parse` 支持 `store_only=true` 走**原件通道**：`md` 与 `png/jpg/jpeg/gif/webp` 只存原件、完全不解析，通过 `/doc/library/{doc_id}/source` 读回。响应与 manifest 带 `kind: "raw"`（解析文档为 `"parsed"`），搜索命中、sections 列举、摘要状态同样带；这类文档在 digest/brief/full 上 404、摘要重试上 409，也不会出现在 `search_text` 结果里。`GET /doc/health` 以 `raw_formats` 列出本通道的格式。本通道的 `409` 在 `detail` 里返回对象、带上已有文档的 `kind`；解析通道的 `409` 不变。原件与解析两套扩展名集合互不相交 —— 可解析的扩展名带 `store_only` 会返回 `422`。要求 `MANTISFETCH_STORE_SOURCE_FILES=true`。
- 原件通道遇到内容重复时返回 `dedup: "hit"` 与 `existing_doc_id`，但仍按调用方给的 `doc_id` 建文档 —— 只告知，不合并。
- `GET /doc/library/search` 的结果也带 `kind`，调用方在去要 digest 之前就能分辨原件文档与解析文档。
- `GET /doc/library/{doc_id}/table/{table_id}/json` 返回结构化单元格；对从扫描页重建的表格，合并单元格现在带恢复的 `colspan`（Markdown 形式不变）。
- `POST /doc/parse` 和 `POST /web/capture` 支持 `content_type`，可选值为 `General`、`Contract`、`Bid`、`Knowledge`（输入大小写不敏感，存储统一为首字母大写）；默认 `General`。
- 新入库文档会保存到 `${MANTISFETCH_DOCS_DIR}/<content_type>/<doc_id>`，旧版平铺的 `${MANTISFETCH_DOCS_DIR}/<doc_id>` 文档仍可读取。
- `GET /doc/library/search` 支持 `metadata.*` 形式的查询参数做等值过滤，例如 `metadata.customer=ACME`。
- `GET /doc/library/search` 和 `GET /doc/library/search_text` 支持 `content_type`，用于按分类浏览。
- `GET /doc/library/search_text` 会返回 `snippet`、`sid`、`page_range`、`page_start`、`page_end`，便于后续定位页面。
- `manifest.json` 现在会包含 `metadata`、`source_file`，以及补强后的 section 页码边界。
- 如果设置 `MANTISFETCH_DOC_ID_STRATEGY=source_filename`，上传文件会优先使用“过滤后的源文件名”作为文档目录名；空格、下划线、点会归一成 `-`，不支持的字符会被剔除；若过滤后没有剩余可用字符，则自动回退为 `DOC-xxx`。

### 网络搜索（可选）

默认关闭。设置 `MANTISFETCH_SEARCH_PROVIDER` 后启用两个端点：

| 方法 | 端点 | 说明 |
|---|---|---|
| `POST` | `/web/search` | 纯搜索——返回排序后的 `{url, title, snippet, ...}`，不采集 |
| `POST` | `/web/search_and_capture` | 搜索后串行采集前 N 条（≤3）入库并写入搜索来源标记——返回可直接三级加载的 `doc_id` |

Provider：`searxng`（自托管、零 API 成本，推荐默认）、`tavily`、`bocha`（博查，国内）、`brave`。逗号分隔的 `MANTISFETCH_SEARCH_FALLBACK` **仅**在连接错误 / 5xx / 超时时切换 provider（4xx 配置错误或空结果**不**切换）。未设置时 `/web/search*` 返回 `404`，MCP 的 `web_search` / `web_search_capture` 工具不注册。

一条命令启动零成本 SearXNG：

```bash
MANTISFETCH_SEARCH_PROVIDER=searxng docker compose --profile search up
```

**国内部署**:SearXNG 默认上游(Google、DuckDuckGo)通常被墙——先启用国内预设(Baidu / 360 / Sogou / Quark / Bing 起步组合,**请在目标网络实测校准**):`cp configs/searxng/settings.cn.yml configs/searxng/settings.yml`。

`search_and_capture` 复用 `/web/capture` 路径，并带两层去重：

1. **URL 缓存** — `MANTISFETCH_SEARCH_CAPTURE_TTL_HOURS`（默认 **24**），避免重复研究查询反复重抓同一批结果；与普通 `/web/capture` 的 `MANTISFETCH_CAPTURE_TTL_HOURS`（默认仍为 `0`）独立。
2. **内容哈希复用** — distill 后若库中已有相同 body `content_hash`（同一文章经 AMP / 跟踪参数 URL），返回已有 `doc_id` 并合并 tags/metadata（已有 metadata 键 first-touch 保留）。

缓存命中保留文档原有顶层来源（first-touch）：先被直接采集的 URL 仍为 `source=web_capture`。过滤搜索来源文档请用 `metadata.source=web_search`，**不要**用顶层 `source`。

| 变量 | 默认 | 说明 |
|---|---|---|
| `MANTISFETCH_SEARCH_PROVIDER` | — | `searxng` / `tavily` / `bocha` / `brave`；不设则禁用搜索 |
| `MANTISFETCH_SEARCH_FALLBACK` | — | 逗号分隔的降级链，如 `tavily,searxng` |
| `MANTISFETCH_SEARCH_PROVIDERS` | — | 请求级 `provider` 选择的可寻址集合（在主 + 降级之外追加）；让 Agent 每次调用点名一家，如中文查 bocha、英文查 tavily |
| `MANTISFETCH_SEARXNG_URL` | — | SearXNG 实例地址（`search` compose profile 默认为 `http://searxng:8080`） |
| `MANTISFETCH_SEARCH_API_KEY` | — | 共享 API key（Tavily / 博查 / Brave 的回退） |
| `MANTISFETCH_{TAVILY,BOCHA,BRAVE}_API_KEY` | — | 各 provider 专属 key；优先于共享 key（同时配置多家 API provider 时用） |
| `MANTISFETCH_SEARCH_MAX_RESULTS` | `10` | 单次结果上限（硬顶 20） |
| `MANTISFETCH_SEARCH_MIN_INTERVAL_SEC` | `2` | 两次搜索的最小间隔秒数。来早了的搜索排队等，不再直接失败 |
| `MANTISFETCH_SEARCH_MAX_WAIT_SEC` | `30` | 一次搜索最多等多久轮到自己，超过才 `429`。能吸收 `max_wait / interval` 个排队的，再加上不用等的那一个——默认值下是 16；`0` = 从不等待 |

### 配置项

MantisFetch 所有配置均通过环境变量管理。LLM 相关配置见上方 **Docker 配置** 一节，其他变量如下：

| 变量 | 默认值 | 说明 |
|---|---|---|
| `HOST` | `0.0.0.0` | 绑定地址 |
| `PORT` | `9898` | HTTP 监听端口 |
| `LANG` | `en` | 界面语言（`en` 英文 / `zh` 中文） |
| `MANTISFETCH_TLS_CERTFILE` | — | TLS 证书路径；与 `MANTISFETCH_TLS_KEYFILE` **同时**设置才会启用 HTTPS（只设其一会被忽略） |
| `MANTISFETCH_TLS_KEYFILE` | — | TLS 私钥路径（与 `MANTISFETCH_TLS_CERTFILE` 配对） |
| `MANTISFETCH_MCP_TOKEN` | — | `/mcp` 接口的 bearer token；不设置则 `/mcp` 仅 loopback 可达 |
| `MANTISFETCH_OWNER_SCOPED_DISCOVERY` | 关 | 设为 `1`/`true`/`yes`/`on` 时，库检索只返回调用方自己的上传、`shared: true` 的文档和网页抓取。此时必须设置 `MANTISFETCH_MCP_TOKEN`，否则进程拒绝启动；`/web` `/doc` `/deliverables` 连本机回环也要带 bearer（健康检查除外）。未指定 `doc_id` 的上传改为不可枚举的 `R-<hex>` 编号 |
| `MANTISFETCH_MCP_ALLOWED_HOSTS` | — | DNS-rebinding 防护的额外 host/origin（逗号分隔），作用于 `/mcp` 以及不带 token 的 loopback `/web` `/doc` 请求 |
| `MANTISFETCH_MCP_PARSE_BUDGET_SEC` | `45` | MCP 的 `doc_parse` 与 `web_search_capture` 单次调用的时间上限，到点就用当时能给的结果回复（拒绝，或部分抓取结果）。它要小于 MCP 客户端的单请求超时（NodalOS 对上游调用封顶 60 秒）：超时的调用会被当作传输失败断开，客户端在重连前会丢掉本服务的全部工具。只能和那个上限一起调大 |
| `MANTISFETCH_MCP_RESULT_BUDGET_BYTES` | `60000` | 单个 MCP 工具结果在线上的体积上限（按 2026-07-28 协议代的 ASCII 转义形态计）。超过的结果会分页或分窗（`doc_manifest`、`doc_sections`、`doc_tables`、`doc_full`、`doc_table`、`doc_chunks`、`doc_source`），其他工具超限则返回说明改用什么方式的工具错误，绝不发出会被客户端半截切断的结果。默认值低于 NodalOS 的 65,536 字节截断线；客户端没有这类上限的部署可以调大。没有关闭选项：0 或无效值都按默认值处理 |
| `MANTISFETCH_ALLOWED_DOC_ROOTS` | — | MCP `doc_parse` `rel_path` source 的 allowlist 根目录；不设置则禁用 MCP 上的本地路径解析 |
| `MANTISFETCH_DELIVERABLES_ROOT` | — | 只读 `GET /deliverables/{rel_path}` 字节接口的围栏根目录；不设置则禁用（所有请求返回 404）。不得与 `MANTISFETCH_DOCS_DIR` 或 `MANTISFETCH_ALLOWED_DOC_ROOTS` 重叠 |
| `MANTISFETCH_DELIVERABLES_MAX_MB` | `200` | 单个交付物下载的大小上限；超出返回 413 |
| `MANTISFETCH_DOC_ID_STRATEGY` | `counter` | 文档目录命名策略：`counter` 保持 `DOC-xxx`；`source_filename` 基于上传文件名生成安全目录名 |
| `MANTISFETCH_PARSE_QUEUE_MAX_WAIT_SEC` | `600` | 没声明 `budget_seconds` 的解析最多等多久拿到槽，超过才 `429`；声明了预算的则在自己的预算内等。并发对解析吞吐毫无帮助——文本解析被 GIL 绑死，OCR 单文档已内部并行——所以排队花的时间本来也省不掉 |
| `MANTISFETCH_PARSE_QUEUE_MAX_BYTES` | 上传上限的十倍（默认 200MB 时为 **2GB**） | 解析队列允许占用的暂存上传字节数。排队中的解析会一直把上传留在盘上，所以队列按磁盘设限而不是按个数。同一额度也覆盖「正在接收中」的上传：`/parse` 请求在读取任何请求体之前，按 `Content-Length`（未声明长度的按单请求上限计）与「已排队 + 正在接收」的字节合计准入，放不下就 `429`——所以这一个数限定了服务持有的全部上传字节 |
| `MANTISFETCH_MAX_PARSE_ROWS` | `100000` | XLSX 所有 sheet 合计跨越的行数——每个 sheet 按最大行号计，因为转换会把有值行之间的空行全部补出来。转换前流式读 sheet XML 计数，超过即 `422`，什么都不入库。这是内存上限：转换会把整个工作簿读进内存，实测 10 万行 × 10 列约 2GB RSS。`.xls` 不计数 |
| `MANTISFETCH_MAX_PARSE_CELLS` | `1000000` | 列方向的同一上限：每个 XLSX sheet 按「行数 × 最宽列」计格数，跨 sheet 累加，与行数同一遍检查、同样 `422`。没有它时，只有 A1 和 XFD200 两格的 4.8KB 文件转换要 80 秒、3.5GB。默认值即行数默认值当初参照的规模；又长又宽的表（如 5 万行 × 30 列）需要调大 |
| `MANTISFETCH_MAX_CONCURRENT_CAPTURE` | `16` | 同时在飞的 capture 数。实测饱和点：16 起吞吐稳定在 ~4.5 次/秒，再加并发只涨延迟不涨产出。每个在飞的 capture 约 6MB |
| `MANTISFETCH_MAX_CONCURRENT_SESSIONS` | `20` | 同时建 session 的并发数。建一个很便宜（空载 0.03s，200 个活跃时 0.29s），所以这是突发门而不是资源门 |
| `MANTISFETCH_CONCURRENCY_MAX_WAIT_SEC` | `30` | capture / 建 session 最多等多久拿到槽，超过才 `429`。按饱和速率，30s 能吸收约 135 个排队的 capture——5 个 Agent 各抓 8 页约 9s 全成，10 个 Agent 约 19s，无一被拒。`0` = 立即拒绝 |
| `MANTISFETCH_SESSION_MAXSIZE` | `200` | 活跃 session 上限，超出淘汰最旧的。**这是一个内存预算**：一个活跃 session 持有一个浏览器上下文，约 15MB，所以 200 个约 3GB 常驻。请按部署机器的内存来定 |
| `MANTISFETCH_SESSION_TTL_SEC` | `1800` | session 空闲多久后关闭 |
| `MANTISFETCH_CAPTURE_TTL_HOURS` | `0` | 在这么多小时内对同一 URL + content_type 的 `/web/capture` 直接复用已有结果而不重抓；`0` 关闭（默认）。`force_refresh=true` 始终绕过。distill 后仍会按 content_hash 做库内内容级复用 |
| `MANTISFETCH_SEARCH_CAPTURE_TTL_HOURS` | `24` | 同上，但仅作用于 `/web/search_and_capture`（默认开启，避免研究查询堆积重复文档） |
| `MANTISFETCH_SUMMARY_BATCH_CONCURRENCY` | `1` | 单文档 section 摘要的最大 LLM batch 并发数 |
| `MANTISFETCH_SUMMARY_REQUEST_MIN_INTERVAL_SEC` | `2.0` | 全服务摘要 LLM 请求之间的最小间隔秒数 |
| `MANTISFETCH_SUMMARY_SECTION_DETAIL_LIMIT` | `10` | 超过该 section 数量后跳过逐 section LLM 摘要，改用 section 摘录生成文档级摘要 |

### 接入 AI Agent

MantisFetch 支持由 AI Agent 自主部署和操作。[`docs/agent-deployment-prompt.md`](docs/agent-deployment-prompt.md) 提供开箱即用的中英双语部署提示词，直接复制到 Agent 的系统提示词中即可使用，无需任何修改。

### 参与贡献

欢迎提交 PR！开发环境搭建、代码规范和 PR 流程请参阅 [CONTRIBUTING.md](CONTRIBUTING.md)。

### 许可证

MIT 协议，详见 [LICENSE](LICENSE)。
