Metadata-Version: 2.4
Name: free-search-mcp
Version: 0.9.1
Summary: Local-first, no-API-key search MCP server. Multi-engine web search, smart fetching, document reading.
Project-URL: Homepage, https://github.com/sweetcornna/free-search-mcp
Project-URL: Repository, https://github.com/sweetcornna/free-search-mcp
Project-URL: Issues, https://github.com/sweetcornna/free-search-mcp/issues
Project-URL: Changelog, https://github.com/sweetcornna/free-search-mcp/blob/main/CHANGELOG.md
License-Expression: MIT
License-File: LICENSE
Keywords: llm,mcp,model-context-protocol,playwright,scraping,search,web-search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Requires-Python: >=3.11
Requires-Dist: aiosqlite>=0.20
Requires-Dist: anyio>=4.4
Requires-Dist: curl-cffi>=0.7
Requires-Dist: ebooklib>=0.18
Requires-Dist: extruct>=0.18
Requires-Dist: httpx[socks]>=0.27
Requires-Dist: markdownify>=0.13
Requires-Dist: mcp[cli]>=2.0.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: pillow>=10.0
Requires-Dist: playwright>=1.47
Requires-Dist: pydantic-settings>=2.5
Requires-Dist: pydantic>=2.8
Requires-Dist: pypdf>=5.0
Requires-Dist: python-docx>=1.1
Requires-Dist: python-pptx>=1.0
Requires-Dist: rapidfuzz>=3.10
Requires-Dist: selectolax>=0.3.21
Requires-Dist: starlette>=0.40
Requires-Dist: trafilatura>=2.0
Requires-Dist: uvicorn>=0.30
Requires-Dist: w3lib>=2.0
Description-Content-Type: text/markdown

# free-search-mcp

<p align="center">
  <img src="https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/docs/search.gif" alt="free-search-mcp — one research() call returns a cited Markdown brief, no API key" width="820">
</p>

[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml)
[![MCP](https://img.shields.io/badge/MCP-2026--07--28-purple.svg)](https://modelcontextprotocol.io/specification/2026-07-28)

A **local-first, no-API-key** Model Context Protocol server that gives any
LLM (Claude, GPT, local Ollama, …) the ability to search the web, fetch and
clean up pages, and read documents — without you signing up for a single
search API.

It bundles together the best ideas from a handful of open-source MCPs into
one Python package, and adds the LLM-ergonomics and reliability work they
were each missing.

```text
research("how does reciprocal rank fusion work", depth=3)
   ↓
# Research brief: how does reciprocal rank fusion work
_engines: duckduckgo, mojeek, googlenews · sources: 3 · ~3,400 tokens_

## Sources
- [1] Reciprocal rank fusion | Elasticsearch Reference — <https://…>
- [2] Hybrid Search Scoring (RRF) | Microsoft Learn — <https://…>
- [3] RRF explained in 4 mins — Medium — <https://…>

## Documents
…full Markdown bodies of each page, ready for the LLM to read…
```

One tool call. Three sources. No API key. No `OPENAI_API_KEY`-but-for-search
shakedown.

---

## 🚀 One-click deploy

One command — the keyless engines work immediately, no signup, no key, no
checkout (needs [uv](https://docs.astral.sh/uv/)):

```bash
claude mcp add search -- uvx free-search-mcp      # Claude Code
codex mcp add search -- uvx free-search-mcp       # Codex
```

Any other MCP client: point it at the command `uvx free-search-mcp` (stdio). The
first run downloads the package from PyPI; every HTTP engine works with no
further setup.

Optional — browser-rendered engines (`startpage`, `zhihu`, …) and JS-heavy
page fetches need Chromium once:

```bash
uvx --from free-search-mcp playwright install chromium
```

Without it, HTTP search/fetch keep working, and any call that needs the
browser returns that exact install command instead of a cryptic failure.

Configuration (all optional) lives in `~/.config/search-mcp/.env` — see
[Configuration](#configuration).

### Full install (source checkout + client registration)

```bash
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client claude-code
```

It clones or updates the project under `~/.local/share/free-search-mcp`,
installs `uv`, syncs dependencies, installs Chromium for rendered engines
(with OS deps on Linux), smoke-tests the server, and registers the `search`
MCP server in Claude Code user scope.

Other client targets:

```bash
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client claude-desktop
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client codex
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client generic
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client add-mcp
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client all
```

Codex, Cursor, Cline, Continue, Zed, and generic agent guidance:
[docs/AGENT_USAGE.md](docs/AGENT_USAGE.md).

Optional extras, any time (the defaults already work without them):

```bash
uv run search-mcp-admin        # bilingual browser UI / 中英双语配置页: http://127.0.0.1:8765
uv run search-mcp-login zhihu  # one-time Zhihu login (persists cookies)
```

Prefer containers? `docker compose run --rm search-mcp`. Claude Desktop and
other clients: see [Install](#install) below.

---

## Why this exists

Existing search MCPs each do one thing well, but you usually want all of it:

| | Multi-engine | No API key | Smart fallback | PDF/DOCX | FTS5 cache | Filters | Trafilatura | LLM-tuned |
|---|---|---|---|---|---|---|---|---|
| `nickclyde/duckduckgo-mcp-server` | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ~ |
| `mrkrsl/web-search-mcp` | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ~ |
| `Aas-ee/open-webSearch` | ✓ | ✓ | ~ | ✗ | ✗ | ✗ | ✗ | ~ |
| `VincentKaufmann/noapi-google-search-mcp` | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ~ |
| **free-search-mcp** | **✓** | **✓** | **✓** | **✓** | **✓** | **✓** | **✓** | **✓** |

"LLM-tuned" here means: Markdown-first output, token estimates, smart
truncation at paragraph boundaries, "Best for / Not for / Returns / Common
mistakes" docstrings the model uses to pick the right tool, actionable
error hints, MCP prompts and resource templates, and a one-shot
`research()` that collapses search→fetch→fetch→fetch into a single turn.

"Trafilatura" means we extract main content using
[trafilatura](https://github.com/adbar/trafilatura) — winner of the
Bevendorff 2023 ROUGE benchmark (~0.85 vs ~0.55 for naive boilerplate
stripping). Each fetched page also returns `author`, `published_date`, and
`sitename` for free.

"Filters" means search/research accept `freshness`, `include_domains`,
`exclude_domains`, `category`
(`news`/`pdf`/`github`/`paper`/`forum`/`blog`/`image`/`dataset`),
`include_text`, `exclude_text`. `category` also **routes** the query to
sources that natively index it — see [Vertical sources](#vertical-sources-selected-automatically-by-category).

### Anti-detection &amp; resilience

- HTTP fast path uses [`curl_cffi`](https://github.com/lexiforest/curl_cffi)
  with a real Chrome 131 JA3/JA4 + HTTP/2 fingerprint, fixing the DDG
  "anomaly 202" rate-limit response that vanilla httpx triggered.
- Playwright fallback uses `launch_persistent_context` (cookies survive
  restarts on disk), prefers a real installed Chrome (`channel="chrome"`),
  drops the `--no-sandbox` fingerprint marker on macOS, and randomizes the
  viewport per session.
- Result dedup is **title-fuzzy + host-canonical** (rapidfuzz
  `token_set_ratio >= 92`, host normalized for `www./m./amp.` and
  country-TLD collapse), catching `bbc.co.uk` vs `bbc.com` duplicates that
  URL-only dedup misses.
- `search` includes an honest extractive `lead_snippet` — picks the top-3
  result whose snippet contains ≥2 query terms and is ≥80 chars; rendered
  as `> **Lead:** According to {host}: …`. No LLM call. Returns nothing
  if no snippet qualifies (no fake answer).

> ⚠️ We deliberately do **not** attempt to defeat proof-of-work captchas
> on Bing or Brave — that crosses the ToS line. When those engines
> challenge us, we fall back to other engines instead.

---

## Tools (10)

| Tool | Description |
|---|---|
| `search(query, ...filters)` | Parallel multi-engine search, RRF-merged, title-fuzzy + host-canonical deduped, with optional extractive `lead_snippet` |
| `research(question, depth?, ...filters)` | One-shot: search + fetch top N + return Markdown brief |
| `compare(question, urls=[2..5])` | Concurrent fetch of 2-5 URLs, side-by-side excerpts keyed by question |
| `fetch(url, render?, inline?, ...)` | Fetch any resource: reader-mode Markdown for pages, parsed text for documents, or a description (type/size/dimensions/sha256) for images and binaries. `inline=True` returns the image itself for vision models |
| `fetch_batch(urls, ...)` | Concurrent multi-URL fetch (max 20 per call) |
| `read_doc(source, start?, length?, ...)` | Parse PDF / DOCX / XLSX / PPTX / EPUB / CSV / code / zip-tar / HTML / TXT / MD with pagination |
| `extract_structured(url, ...)` | Pull JSON-LD / OpenGraph / Twitter cards / microdata via extruct |
| `cache_search(query, limit?, ...)` | FTS5 search across previously fetched pages |
| `engines()` | List engine names available to `search` |
| `download(url, ...)` | Save a file to `${SEARCH_MCP_CACHE_DIR}/downloads` by default; files auto-delete after 24h. Set `SEARCH_MCP_DOWNLOAD_ENABLED=false` to disable it. |

Plus **4 MCP prompts** (`Research thoroughly`, `Fact-check claim`,
`Compare sources`, `News brief`) and **2 resource templates**
(`cache://page/{url}`, `cache://search/{query_hash}`).

### Filters (search / research)

| Param | Values | Effect |
|---|---|---|
| `freshness` | `day` / `week` / `month` / `year` | Only results from the last N |
| `include_domains` | `["python.org", "djangoproject.com"]` | Restrict to these domains |
| `exclude_domains` | `["pinterest.com"]` | Remove these |
| `category` | `news` / `pdf` / `github` / `paper` / `forum` / `blog` | Content-type shortcut (paper = arxiv/acm/ieee/…, forum = reddit/HN/SE, etc.) |
| `include_text` | `"async"` | Substring required in title/snippet |
| `exclude_text` | `"beginner"` | Substring forbidden |
| `max_age_hours` | `24` | Override the 7-day default cache TTL on this call |

All tools default to `format="markdown"` — readable, ~40% fewer tokens than
JSON, with provenance and a token-budget header. Pass `format="json"` for
structured access.

### Tool annotations

Every tool ships correct `readOnlyHint`, `idempotentHint`, and
`openWorldHint` annotations so MCP clients can label them and gate
elevated actions.

### Engines

Default set (all-HTTP, **no browser**):
`duckduckgo`, `mojeek`, `googlenews`, `bing`.

When a search comes back empty (or nearly empty with gated/erroring
engines), the aggregator automatically runs one bounded **rescue pass**
via `searx` → `bing` and reports it as `rescued_via` — so a CAPTCHA wall
on the defaults degrades to slower results instead of no results.

Opt-in:
- `startpage` — browser-rendered (~5-10s/query); good for hard-to-reach
  results that the HTTP defaults miss.
- `brave`, `baidu` — intermittent challenges to headless clients.
- `searx` — meta-search proxy via public SearXNG instances; included for
  completeness but most public instances are slow/unreliable in 2026.
- `google` — keyless Google web SERP scrape (HTTP first, Playwright
  fallback when Google serves a JS/consent shell). `serpsearch` is a pure
  alias of `google` (all dedicated "SERP APIs" require a key, so the only
  keyless SERP is a direct scrape).
- `anysearch` — [AnySearch](https://github.com/anysearch-ai/anysearch-mcp-server)
  unified-search REST API, anonymous (keyless) tier; one HTTP call returns
  fused, re-ranked results. IP rate-limited; 429/5xx degrade to empty.
- `bilibili` — keyless Bilibili (哔哩哔哩) video search via the public
  `web-interface/search/all/v2` JSON API (synthetic `buvid3` cookie, no
  login). Returns video results only.
- `zhihu` — **best-effort** keyless Zhihu (知乎) search. Zhihu's
  `api/v4/search_v3` needs login cookies + `x-zse-96` signing, so the only
  no-key path is browser-rendering the public search page. Zhihu hard-gates
  headless clients, so a login wall / empty result is common and honest —
  treat it like `baidu`/`brave`.

- `sogou`, `so360` — Chinese web indexes, HTML scrapes, best-effort like
  `zhihu`. Note `sogou` returns **redirect URLs** (`sogou.com/link?url=…`)
  rather than target URLs; the blob is only resolvable by following it.
  `fetch` handles that, but host-based `category` filtering will discard
  them. `baidu` and `so360` return direct URLs.
- `wikipedia` — encyclopedia search; language follows `SEARCH_MCP_REGION`.
- `openlibrary` — book search over the Internet Archive catalogue.

> All keyless engines stay **opt-in** — they're not in the fast default pool,
> so the ~2x latency win of the all-HTTP defaults is preserved. Enable per
> call with `engines=["google","bilibili", ...]`, or globally via
> `SEARCH_MCP_DEFAULT_ENGINES`.

### Vertical sources (selected automatically by `category`)

These index something a general web engine can't. You normally **don't name
them** — passing `category=` to `search`/`research` routes to them:

| `category` | Engines | Why it matters |
|---|---|---|
| `paper` | `arxiv`, `openalex`, `crossref`, `pubmed` | Actually searches the literature instead of filtering web results by hostname |
| `github` | `github` (repos + issues/PRs), `github_code` (needs a token) | Real repository metadata, stars, last push |
| `forum` | `stackexchange`, `hackernews` | Accepted-answer and score signals |
| `news` | `googlenews`, `gdelt` | GDELT covers 100+ languages Google News never surfaces |
| `image` | `openverse` | Openly-licensed images; results are direct file URLs, so `fetch(inline=True)` works on them |
| `dataset` | `zenodo` | Datasets, software and their DOIs |

`image` and `dataset` **replace** the default pool rather than adding to it —
a web engine can't return an image file or a dataset record, so mixing it in
only crowds out the source that can. The others augment it, capped by
`SEARCH_MCP_CATEGORY_ENGINE_LIMIT` (default 3).

Naming engines explicitly (`engines=[...]`) turns the routing off.

Sources that publish a stricter rate limit than our default declare it
themselves, and are **skipped** rather than queued when their bucket is empty
— search is a parallel fan-out, so waiting on one slow source would add that
delay to every other engine's results.

### API-key engines & the admin backend

Keyless is the default, but you can also plug in keyed providers for higher
reliability/quality. These engines stay dormant (and return a clear "not
configured" hint) until you add a key:

| Engine | Provider | Free tier |
|---|---|---|
| `brave_api` | [Brave Search API](https://brave.com/search/api/) | 2,000 queries/mo |
| `serper` | [Serper](https://serper.dev) (Google) | 2,500 queries |
| `tavily` | [Tavily](https://app.tavily.com) (AI search) | 1,000 credits/mo |
| `google_cse` | [Google Custom Search](https://programmablesearchengine.google.com/) | 100 queries/day |
| `anysearch` | [AnySearch](https://anysearch.com) (key optional) | keyless works; key lifts limits |
| `github_code` | [GitHub](https://github.com/settings/tokens) | code search is auth-only; the keyless `github` engine covers repos + issues |
| `stackexchange` | [Stack Apps](https://stackapps.com/apps/oauth/register) (key optional) | 300 req/day keyless; a key lifts the quota |

**Simplest setup — the admin UI:**

```bash
uv run search-mcp-admin          # opens a local config page on 127.0.0.1:8765
```

It serves one bilingual page (English + 中文, bound to localhost only) with,
per provider: a **how-to-get-a-key / 如何获取密钥** guide + signup/docs links,
masked key fields, **Save / 保存** (applies live — no server restart),
**Test / 测试**, and **Clear / 清除**. The same page also includes
**Network / Proxy / 网络 / 代理** settings. Keys are written to
`~/.config/search-mcp/config.json` (`0600`); they're never echoed back to the
page. Prefer env vars? Set `SEARCH_MCP_<PROVIDER>_API_KEY` instead (these
override the saved file). Full walkthrough for each provider:
**[docs/API_KEYS.md](docs/API_KEYS.md)**.

```text
search("…", engines=["brave_api"])        # once a key is saved
search("…", engines=["tavily", "serper"]) # mix keyed + keyless freely
```

### When an engine is gated (proxy · fallback · login)

Some engines get blocked by the *provider* — Google/Bing serve a CAPTCHA to
datacenter IPs, Zhihu needs a login. We don't defeat CAPTCHAs (ToS); instead:

- **Proxy** (the real fix for IP gating): set a proxy in the admin UI
  "Network / Proxy" card or `SEARCH_MCP_PROXY` (`http`/`https`/`socks5`, optional
  `user:pass@`). It routes the HTTP engines, the browser, and `fetch` through a
  non-blocked IP. Scope it with `SEARCH_MCP_PROXY_ENGINES="google bing zhihu"`.
- **SearXNG auto-fallback**: when `google`/`serpsearch`/`bing` are CAPTCHA-gated,
  they transparently retry via the working `searx` meta-search — you still get
  results, honestly attributed to `searx`.
- **Gate diagnostics**: the response includes `gated_engines` + `gated_hint`
  telling you which engine was gated (`captcha`/`consent`/`login`) and how it was
  handled.
- **Zhihu login**: run `uv run search-mcp-login zhihu` (or the admin "Login"
  button) once — a browser opens, you log in, cookies persist, and `zhihu` search
  then works. Requires a desktop session.

Full guide: **[docs/PROXY_AND_GATES.md](docs/PROXY_AND_GATES.md)**.

> Brave/Bing/Baidu all gate headless browsers after a handful of calls (PoW
> CAPTCHAs, "something went wrong" pages, redirect wrappers). Pass
> `engines=["brave"]` etc. only when the defaults can't find what you need.

### Sparse-result diagnostics

When filters drop results so aggressively that ≤3 are returned, the
response includes `filter_diagnostics` so the LLM knows which knob to
relax. Example for `category="forum" + exclude_text="beginner"`:

```text
⚠️ **Filter diagnostics** (results were sparse)
Raw results: 20 across 3 engines → 0 after filters.
Top drops: category_forum (20).
Hint: Filters dropped 20 of 20 raw results. Most were excluded by
category=forum. Try widening or removing one filter.
```

---

## Install

### Zero-checkout (uvx, recommended)

```bash
claude mcp add search -- uvx free-search-mcp      # Claude Code
codex mcp add search -- uvx free-search-mcp       # Codex
uvx free-search-mcp                               # or run the stdio server directly
```

Optional extras, any time:

```bash
uvx --from free-search-mcp playwright install chromium   # browser-rendered engines
uvx --from free-search-mcp search-mcp-admin               # bilingual config UI (opens browser)
```

### One-click setup (source checkout)

```bash
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client claude-code
```

The remote installer clones or updates
`~/.local/share/free-search-mcp`, installs `uv` if needed, syncs dependencies,
installs Chromium, smoke-tests the MCP server, then registers it with the
requested client:

```bash
--client claude-code      # Claude Code user-scope config
--client claude-desktop   # claude_desktop_config.json
--client codex            # Codex config
--client generic          # print portable MCP JSON for other agents
--client add-mcp          # delegate to npx add-mcp
--client both             # Claude Code + Claude Desktop
--client all              # Claude Code + Claude Desktop + Codex
--client none             # install only, no client config changes
```

Prefer a local checkout?

```bash
git clone https://github.com/sweetcornna/free-search-mcp.git
cd free-search-mcp
./scripts/install.sh --client none
```

Prefer to do it by hand?

```bash
uv sync
uv run playwright install chromium
cp .env.example .env        # optional: customize engines/limits
```

Run as a stand-alone server (stdio transport):

```bash
uv run search-mcp
```

Or serve it over HTTP. MCP revision `2026-07-28` removed protocol-level
sessions, so the HTTP endpoint is stateless — no sticky routing, any replica
can answer any request:

```bash
uv run search-mcp --transport streamable-http --port 8000   # → http://127.0.0.1:8000/mcp
```

> The HTTP endpoint has **no authentication** and will fetch arbitrary URLs on
> behalf of whoever reaches it. It binds `127.0.0.1` by default; if you change
> `--host`, put an authenticating reverse proxy in front. A DNS-rebinding guard
> rejects unknown `Origin`/`Host` headers — add trusted browser origins with
> `SEARCH_MCP_HTTP_ALLOWED_ORIGINS`. This is a separate process and port from
> the admin UI (`search-mcp-admin`, port 8765), which stays loopback-only
> because it reads and writes your API keys.

### Docker (one-click, containerized)

```bash
docker compose build
docker compose run --rm search-mcp     # attaches stdio for MCP
```

### Config & env vars

All settings are env vars prefixed with `SEARCH_MCP_`. Copy `.env.example` →
`.env` and edit — it documents every knob, including how to enable the new
engines via `SEARCH_MCP_DEFAULT_ENGINES`. See the full table under
[Configuration](#configuration) and the [usage guide](docs/USAGE.md).

### Tests

```bash
uv run pytest -q                              # offline (default, no network)
SEARCH_MCP_TEST_NETWORK=1 uv run pytest -v    # live tests, hit the real web
```

---

## Wire into Claude Code

```bash
claude mcp add search -s user -- uvx free-search-mcp
```

From a source checkout instead: this repo ships a project-scoped `.mcp.json`,
so running `claude` inside the project auto-detects the `search` server; or
register the checkout globally:

```bash
claude mcp add search -s user -- uv --directory /absolute/path/to/free-search-mcp run search-mcp
```

## Wire into Codex

```bash
codex mcp add search -- uvx free-search-mcp
codex mcp list
```

(Source checkout: swap the command for
`uv --directory /absolute/path/to/free-search-mcp run search-mcp`.)

## Wire into Claude Desktop

Add this to `~/Library/Application Support/Claude/claude_desktop_config.json`
(macOS) or the equivalent on your platform:

```json
{
  "mcpServers": {
    "search": {
      "command": "uvx",
      "args": ["search-mcp"]
    }
  }
}
```

(Source checkout: `"command": "uv", "args": ["--directory",
"/absolute/path/to/free-search-mcp", "run", "search-mcp"]`.)

Restart Claude Desktop. The ten tools above will appear in the tool
drawer.

### Wire into other clients

The server speaks plain MCP over stdio (or streamable-http, see above). It
implements protocol revision `2026-07-28` and serves every earlier revision
from the same process, so both new and older clients work unchanged:

- Codex (`codex mcp add search -- uvx free-search-mcp`)
- Claude Code (`claude mcp add search -s user -- uvx free-search-mcp`)
- Cursor / Continue / Cline (use the JSON snippet above)
- Custom Python / TypeScript clients via the official MCP SDK

For agent-specific operating rules, tool-selection guidance, and a reusable
system-prompt snippet, see [docs/AGENT_USAGE.md](docs/AGENT_USAGE.md).

### Installer choice

`uvx free-search-mcp` (PyPI) is the fastest path and needs no checkout — HTTP
engines work immediately and Chromium is a single optional follow-up command.
`scripts/install.sh` remains the full bootstrap for people who want a source
checkout, Chromium with OS deps, a smoke test, and client registration in one
shot. Generic MCP installers still have their place: `add-mcp` can write
config for many clients at once, Smithery is strongest for registry/remote
MCP connections, and MCPB is the right future format for clickable desktop
bundles.

---

## Configuration

All settings can be overridden by environment variables prefixed with
`SEARCH_MCP_`. They can live in three places (highest precedence first):
real environment variables → `./.env` in the launch directory (source
checkouts) → `~/.config/search-mcp/.env` (the stable location for uvx/PyPI
installs; directory overridable via `SEARCH_MCP_CONFIG_DIR`).

API keys and the proxy are easiest to manage in the admin UI
(`search-mcp-admin`, opens your browser automatically; set
`SEARCH_MCP_ADMIN_NO_BROWSER=1` to suppress).

Available knobs:

| Var | Default | Meaning |
|---|---|---|
| `SEARCH_MCP_DEFAULT_ENGINES` | `["duckduckgo","mojeek","googlenews","bing"]` | JSON list |
| `SEARCH_MCP_SEARX_INSTANCES` | *(empty)* | pin known-good SearXNG instance URL(s), comma/space separated; overrides the built-in shortlist |
| `SEARCH_MCP_RESCUE_ENABLED` | `true` | auto-rescue empty searches via rescue engines |
| `SEARCH_MCP_RESCUE_ENGINES` | `["searx","bing"]` | rescue order (JSON list) |
| `SEARCH_MCP_RESCUE_TIMEOUT` | `10.0` | seconds; cap on the whole rescue pass |
| `SEARCH_MCP_MAX_RESULTS_PER_ENGINE` | `10` | |
| `SEARCH_MCP_RATE_LIMIT_PER_MINUTE` | `30` | per engine |
| `SEARCH_MCP_FETCH_RATE_LIMIT_PER_MINUTE` | `20` | shared `fetch` bucket |
| `SEARCH_MCP_CACHE_DIR` | `~/.cache/search-mcp` | |
| `SEARCH_MCP_CACHE_TTL_SECONDS` | `604800` | 7 days |
| `SEARCH_MCP_CACHE_MAX_MB` | `512` | size cap on the cache file; `0` disables |
| `SEARCH_MCP_FETCH_STRATEGY` | `auto` | `auto` / `http` / `browser` |
| `SEARCH_MCP_BROWSER_HEADLESS` | `true` | |
| `SEARCH_MCP_BROWSER_POOL_SIZE` | `2` | concurrent pages |
| `SEARCH_MCP_MAX_CONTENT_CHARS` | `50000` | per result truncation |
| `SEARCH_MCP_USER_AGENT` | desktop Chrome UA | used by the httpx (documents) and Playwright paths; the curl_cffi path derives its UA from browser impersonation |
| `SEARCH_MCP_DOWNLOAD_ENABLED` | `true` | set `false` to disable local file downloads |
| `SEARCH_MCP_DOWNLOAD_DIR` | `${SEARCH_MCP_CACHE_DIR}/downloads` | optional directory override; unset or blank uses the dynamic default (`/data/downloads` in Docker) |
| `SEARCH_MCP_DOWNLOAD_TTL_HOURS` | `24` | downloaded files are deleted after this; `0` keeps them forever |
| `SEARCH_MCP_DOWNLOAD_MAX_MB` | `100` | second-layer save cap; effective remote cap is the smaller of this and `SEARCH_MCP_MAX_RESPONSE_BYTES` (25,000,000 bytes by default) |
| `SEARCH_MCP_CATEGORY_ENGINE_LIMIT` | `3` | how many category-native engines `category=` may add |
| `SEARCH_MCP_CONTACT_EMAIL` | *(empty)* | optional; routes OpenAlex/Crossref/NCBI into their faster identified-caller pools |
| `SEARCH_MCP_TRANSPORT` | `stdio` | `stdio` / `streamable-http` |
| `SEARCH_MCP_HTTP_HOST` | `127.0.0.1` | streamable-http bind address |
| `SEARCH_MCP_HTTP_PORT` | `8000` | streamable-http port |
| `SEARCH_MCP_HTTP_PATH` | `/mcp` | streamable-http endpoint path |
| `SEARCH_MCP_HTTP_ALLOWED_ORIGINS` | *(empty)* | extra `Origin` values the DNS-rebinding guard accepts, comma/space separated (loopback is always allowed) |

---

## Architecture

```
   ┌─────────────────────────────────────────────────────┐
   │  MCP server (stdio | streamable-http)               │
   │  tools: search / research / fetch / fetch_batch /   │
   │         read_doc / cache_search / engines           │
   └────────────┬────────────────────────────────────────┘
                │
   ┌────────────▼────────────┐  ┌────────────────────────┐
   │  aggregator             │  │  fetcher               │
   │  - parallel engines     │  │  - httpx fast path     │
   │  - reciprocal rank      │  │  - playwright fallback │
   │    fusion               │  │  - markdownify         │
   │  - search cache (FTS5)  │  │  - page cache (FTS5)   │
   └────┬────────────────────┘  └────────────┬───────────┘
        │                                    │
   ┌────▼─────────────────┐  ┌──────────────▼─────────────┐
   │  engines/            │  │  browser pool              │
   │   duckduckgo.py      │  │   - persistent context     │
   │   mojeek.py          │  │   - stealth init script    │
   │   searx.py           │  │   - shared cookies         │
   │   startpage.py (opt) │  │   - semaphore-bounded pages│
   │   brave.py     (opt) │  └────────────────────────────┘
   │   bing.py      (opt) │
   │   baidu.py     (opt) │
   │   google.py    (opt) │
   │   serpsearch.py(opt) │
   │   anysearch.py (opt) │
   │   bilibili.py  (opt) │
   │   zhihu.py     (opt) │
   │   sogou.py     (opt) │
   │   so360.py     (opt) │
   │   arxiv/openalex/    │
   │   crossref/pubmed    │
   │   github/stackexch.  │
   │   hackernews/gdelt   │
   │   wikipedia/openlib. │
   │   openverse/zenodo   │
   └──────────────────────┘

   ┌────────────────────────────┐    ┌──────────────────┐
   │  documents/                │    │  ratelimit       │
   │   pypdf, python-docx,      │    │   token bucket   │
   │   markdownify              │    │   per engine     │
   └────────────────────────────┘    └──────────────────┘

   ┌────────────────────────────┐    ┌──────────────────┐
   │  formatting                │    │  research        │
   │   token estimate           │    │   composed       │
   │   smart truncation         │    │   workflow       │
   │   markdown renderers       │    │                  │
   └────────────────────────────┘    └──────────────────┘
```

### Engine adapter pattern

Each engine in `src/search_mcp/engines/` implements:

```python
class Engine:
    name: str
    needs_browser: bool          # Force Playwright?
    wait_selector: str | None    # CSS to wait for in browser mode

    def build_url(self, query: str, max_results: int) -> str: ...
    def parse(self, html: str) -> list[SearchResult]: ...
```

The base class handles transport (httpx → Playwright fallback), rate
limiting, and the case where HTTP returns a captcha shell instead of
results (auto-retries via the browser).

---

## Credits

This project stands on the shoulders of:

- [`mrkrsl/web-search-mcp`](https://github.com/mrkrsl/web-search-mcp) —
  smart httpx-then-Playwright fetch strategy, multi-engine fallback chain
- [`Aas-ee/open-webSearch`](https://github.com/Aas-ee/open-webSearch) —
  multi-engine breadth (Bing/DDG/Baidu/Brave/Startpage)
- [`VincentKaufmann/noapi-google-search-mcp`](https://github.com/VincentKaufmann/noapi-google-search-mcp) —
  anti-detection patterns (`navigator.webdriver`, UA, cookies), SQLite
  FTS5 cache idea, multi-format `read_document`
- [`nickclyde/duckduckgo-mcp-server`](https://github.com/nickclyde/duckduckgo-mcp-server) —
  per-engine rate limiting, LLM-friendly content cleanup
- [Mojeek](https://www.mojeek.com/) — independent search index that
  doesn't gate on User-Agent
- [Model Context Protocol](https://modelcontextprotocol.io/) and the
  [official Python SDK](https://github.com/modelcontextprotocol/python-sdk)

---

## License

MIT — see [LICENSE](LICENSE).
