Metadata-Version: 2.4
Name: skdev-websearch-mcp
Version: 1.0.0
Summary: Multi-engine web search + fetch MCP server for AI agents (10 tools)
Author: SK-DEV-AI
License-Expression: MIT
Project-URL: Homepage, https://github.com/SK-DEV-AI/websearch
Project-URL: Repository, https://github.com/SK-DEV-AI/websearch
Project-URL: Issues, https://github.com/SK-DEV-AI/websearch/issues
Keywords: mcp,mcp-server,web-search,search-engine,scraping,reranker,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx==0.28.1
Requires-Dist: mcp>=2.0.0
Requires-Dist: trafilatura>=2.3.1
Requires-Dist: markdownify>=1.0.0
Requires-Dist: protego>=0.7.0
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: pymupdf>=1.24.0
Requires-Dist: ebooklib>=0.18
Requires-Dist: python-docx>=1.0.0
Requires-Dist: lxml>=6.0.0
Requires-Dist: extruct>=0.18.0
Requires-Dist: html-to-markdown>=3.0.0
Requires-Dist: ddgs>=9.0.0
Requires-Dist: websockets>=14.0
Provides-Extra: scraping
Requires-Dist: scrapling>=0.4.15; extra == "scraping"
Provides-Extra: extract
Requires-Dist: crawl4ai>=0.9.4; extra == "extract"
Provides-Extra: ocr
Requires-Dist: docling==2.118.1; extra == "ocr"
Provides-Extra: all
Requires-Dist: scrapling>=0.4.15; extra == "all"
Requires-Dist: crawl4ai>=0.9.4; extra == "all"
Requires-Dist: docling==2.118.1; extra == "all"
Dynamic: license-file

# websearch MCP Server

[![skills.sh](https://skills.sh/b/SK-DEV-AI/skills)](https://skills.sh/SK-DEV-AI/skills) · `npx skills add SK-DEV-AI/skills`

Multi-engine web search + fetch MCP server. Parallel fan-out across DuckDuckGo, Bing Web/News RSS, Tavily, Wikipedia, arXiv, Google News RSS, Brave (last resort — rate-limited from some networks), Google AI Mode, and intent-gated vertical lanes (MDN, Stack Exchange, GitHub, HN, Semantic Scholar, OSV vuln, grep.app/Sourcegraph code, GDELT news) — with consensus-merge dedup, a local cross-encoder reranker, and Groq synthesis with grounding checks.

## Tools (10)

- **search** — Open research questions. depth=1 snippets, depth≥2 full-page fetch + rerank. `synthesize` returns a Groq answer with [N] citations.
- **fetch** — URL to markdown/text. Direct fetch with auto-fallback to the Helium CDP browser for blocked/JS pages. PDF/EPUB/DOCX supported. Paginated (`offset`). For structured JSON use `extract`.
- **screenshot** — ARIA accessibility snapshot (LLM-readable page text and interactive structure).
- **browser_action** — Drive a real browser page (navigate, click, type, press a key, evaluate JS, read HTML/text, ARIA tree, screenshot). Use when a page needs interaction; for plain reading use `fetch`.
- **wikipedia** — Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages.
- **arxiv** — arXiv paper search with boolean/phrase/wildcard query syntax (single-source; multi-source paper search lives in codesearch `papers`).
- **extract** — Structured JSON from a page via CSS (fast, free), LLM, or regex strategies. Repeat calls cached.
- **pdf_extract** — PDF to markdown/json/html. PyMuPDF fast path for text-layer PDFs, Docling+OCR fallback for scanned PDFs.
- **map_site** — Survey a site's pages via sitemap XML (primary) + HTML link fallback, before fetching individually.
- **ping** — Connectivity check for the internet and key search endpoints. Use before expensive calls when uncertain.

## Setup

Requires Python 3.14+.

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp run.example run   # fill in your API keys (run is gitignored — never commit it)
chmod +x run
```

Point your MCP client at `run` (stdio):

```json
{ "websearch": { "command": "/path/to/websearch/run" } }
```

### API keys (`run`)

| Key | Used for | Required? |
|---|---|---|
| `GROQ_API_KEYS` | query rewrite, synthesis, grounding checks | Recommended (search works without, no synthesis) |
| `TAVILY_KEYS` | Tavily engine lane | Optional |
| `NV_KEY` | NVIDIA embedding endpoint | Optional |

Everything else (DDG, Bing RSS, Wikipedia, arXiv, RSS) is keyless. Brave's HTML scrape is currently rate-limited from some networks, so Bing RSS is the primary general-web door and Brave is a last resort.

## Notes

- **Reranker (optional, local):** ranking uses a local `gte-reranker-modernbert-base` cross-encoder over a Unix socket (`RERANKER_MODEL` overrides the model). Without it, results return unranked with `relevance_score 0.0` — everything still works.
- **Helium CDP (optional):** bot-walled pages escalate to a Chrome DevTools browser at `127.0.0.1:9222` when available; without it those pages return a terminal verdict instead of hanging.
- **Caching:** fetch results cache for 1h (`cache_ttl=0` forces fresh). No database, no state beyond the cache dir.
- **Safety:** all user-URL fetches are SSRF-validated (private-IP blocking + redirect-hop revalidation + robots.txt gate).
- **Groq model split:** synthesis + grounding + extract-llm use `openai/gpt-oss-120b`; rewrite/classify/expand/decompose use `openai/gpt-oss-20b` (the previous `llama-3.1-8b-instant` was retired by Groq and now 404s). Constants `GROQ_SMART_MODEL`/`GROQ_CHEAP_MODEL` in config.py.
- **Keyless lanes (all advisory-only):** OSV.dev vuln vertical + NVD CVE enrichment; grep.app MCP code search (Sourcegraph backup); Bing Web RSS + Bing News RSS (Brave's scrape is rate-limited, GDELT 429s from some networks); Wikipedia vertical; Jina Reader pre-CDP tier; Wayback CDX-history recovery in enrich(); Common Crawl domain-survey assist in map_site.
- **Local reranker:** a `gte-reranker-modernbert-base` cross-encoder runs as one shared worker over a Unix socket; either server spawns it on demand and it unloads when idle. Optional — without it results are engine-ranked only.
- **Optional keys (graceful keyless fallback):** `SE_KEY`/`STACKEXCHANGE_KEY` (10k/day SE), `S2_KEY`/`S2_API_KEY` (dedicated S2 1 RPS).
- **Marginalia:** best-effort lane only (shared public key pool, 503s under load); independent-index recall, no dedicated key wired (kontakt@marginalia.nu for one).

