Metadata-Version: 2.4
Name: websearch-kit
Version: 0.23.0
Summary: Web search, fetch, and research pipeline for LLMs — usable as a Python SDK, a standalone MCP server, and an Open WebUI plugin.
Project-URL: Homepage, https://github.com/rmarnold/websearch-kit
Project-URL: Changelog, https://github.com/rmarnold/websearch-kit/blob/main/CHANGELOG.md
Author: Robert Arnold
License-Expression: MIT
License-File: LICENSE
Keywords: llm,mcp,model-context-protocol,open-webui,rag,scraping,search,web-search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx[brotli,http2,zstd]>=0.27
Requires-Dist: protego>=0.3
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: pydantic>=2.7
Requires-Dist: readability-lxml>=0.8.1
Requires-Dist: trafilatura<2.1,>=1.8.0
Provides-Extra: all
Requires-Dist: ddgs>=9.0; extra == 'all'
Requires-Dist: fastembed>=0.5; extra == 'all'
Requires-Dist: mcp<2,>=1.12; extra == 'all'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27; extra == 'all'
Requires-Dist: opentelemetry-sdk>=1.27; extra == 'all'
Requires-Dist: pypdf>=5.0; extra == 'all'
Requires-Dist: tiktoken>=0.7; extra == 'all'
Provides-Extra: ddgs
Requires-Dist: ddgs>=9.0; extra == 'ddgs'
Provides-Extra: impersonate
Requires-Dist: curl-cffi>=0.7; extra == 'impersonate'
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.12; extra == 'mcp'
Provides-Extra: otel
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.27; extra == 'otel'
Provides-Extra: owui
Provides-Extra: pdf
Requires-Dist: pypdf>=5.0; extra == 'pdf'
Provides-Extra: rerank
Requires-Dist: fastembed>=0.5; extra == 'rerank'
Provides-Extra: rerank-gpu
Requires-Dist: fastembed-gpu>=0.5; extra == 'rerank-gpu'
Provides-Extra: tokens
Requires-Dist: tiktoken>=0.7; extra == 'tokens'
Description-Content-Type: text/markdown

# websearch-kit

[![PyPI](https://img.shields.io/pypi/v/websearch-kit)](https://pypi.org/project/websearch-kit/)
[![Python](https://img.shields.io/pypi/pyversions/websearch-kit)](https://pypi.org/project/websearch-kit/)
[![CI](https://github.com/rmarnold/websearch-kit/actions/workflows/ci.yml/badge.svg)](https://github.com/rmarnold/websearch-kit/actions/workflows/ci.yml)
[![License audit](https://github.com/rmarnold/websearch-kit/actions/workflows/license-audit.yml/badge.svg)](https://github.com/rmarnold/websearch-kit/actions/workflows/license-audit.yml)
[![Live tests](https://github.com/rmarnold/websearch-kit/actions/workflows/live.yml/badge.svg)](https://github.com/rmarnold/websearch-kit/actions/workflows/live.yml)
[![License: MIT](https://img.shields.io/pypi/l/websearch-kit)](LICENSE)

> Web search, fetch, and research pipeline for LLMs — one engine, three surfaces:
> a Python **SDK**, a standalone **MCP server**, and an **Open WebUI** plugin.

Query expansion → multi-provider search → SSRF-guarded concurrent fetching
(40-UA browser profile, pinned-IP connect) → trafilatura extraction → BM25
rerank with adaptive context budgeting → numbered, citable context for your LLM.

**No fail-silent:** a call either raises a typed error or returns a result where
every dropped, blocked, truncated, or substituted item is enumerated as a
structured `Degradation`. On the live web that looks like:

```text
ok        : True   partial: True
sources   : 10                       # 5 fetched pages + 5 relevance-filtered snippets
warnings  :
  - [fetch] https://cloud.google.com/...: response exceeded byte cap (1054971 > 1048576 bytes)
stats     : 10 raw -> 10 unique, 5 fetched, context 23471 chars,
            timings {'search': 854, 'fetch': 1662, 'extract': 878, 'rank': 3}
```

**Status: 0.23.0.** See [SPEC.md](SPEC.md), [CHANGELOG.md](CHANGELOG.md).

## Features

- **One engine, three surfaces** — the SDK core is the only pipeline; the MCP
  server and Open WebUI adapters are thin translators over it, so behavior and
  config semantics never drift between surfaces.
- **Full research pipeline** — `research()` runs search → fetch → extract →
  rank → assemble in one call and returns a numbered `[N]` context block with
  1:1 source citations, ready to drop into a prompt.
- **Agentic deep research** — `deep_research()` runs a deterministic map-reduce
  over scoped sub-agents (PLAN → MAP → MERGE → REDUCE) and returns one
  synthesized, `[N]`-cited answer, so research quality stops depending on the
  calling model's own loop. Orchestration lives in code; an LLM (a `Reasoner` —
  an injected `generate` callback to the host's own model, or an
  OpenAI-compatible endpoint) is a stateless worker called once per scoped step.
  Inner `research` reuse keeps every guard; fully fail-soft. On all three
  surfaces: SDK, MCP (`deep_research` tool), and OWUI (a per-chat **Deep
  Research** toggle pill, or the `--deep` flag).
- **Preferred sources** — for questions whose best source is authoritative but
  not lexically matchy (forecasts, prices, scores, gov stats, API docs), a
  `preferred` flag seeds those sources into the pool and protects them from the
  zero-BM25 drop + budget starvation. Fed by a config `regex → domains` registry
  *and* the deep-research orchestrator's own per-question proposals; discovery via
  `site:` search / direct URL, always through the SSRF guard. Default-on,
  fail-soft (ADR 0019).
- **Multi-provider search** — zero-key `ddgs` out of the box; keyed
  Tavily / Brave / Serper / Exa and self-hosted SearXNG via config; ordered
  fallback chains with per-provider circuit breakers.
- **Hardened fetching** — SSRF guard (private / reserved / metadata IP ranges
  blocked at connect time with pinned-IP enforcement), per-response byte caps,
  rotating 40-UA browser profile, concurrent fetches with deadline budgeting.
- **TLS-impersonation profile (optional)** — `fetch_profile="impersonate"` (the
  `[impersonate]` extra) fetches via `curl_cffi` Chrome TLS impersonation to
  recover Cloudflare-fronted / Medium pages that **403 the default fetch on its
  TLS fingerprint** (a full browser header set flips none of them — the block is
  on the ClientHello). The SSRF guard is fully preserved: the validated IP is
  pinned via libcurl `CURLOPT_RESOLVE` with SNI + certificate verification kept
  on the real hostname, every redirect hop re-vetted, and `proxy_url` rejected
  (it would void the pin). Off by default; robots-off like `browser`.
- **Quality extraction & ranking** — trafilatura article extraction, BM25
  reranking (golden-tested math), adaptive context budgeting: the most relevant
  pages get more of the character budget, marginal ones shrink, noise is dropped.
- **Hybrid dense reranking (optional)** — `semantic_rerank` fuses BM25 with
  dense-embedding cosine via reciprocal-rank fusion, so synonym/paraphrase
  matches surface that pure lexical ranking misses (measured +0.021 nDCG@10
  on the eval corpus). Local ONNX (`fastembed`, the `[rerank]` extra) or any
  OpenAI-compatible `/embeddings` endpoint; off by default, a zero-BM25 source
  still drops as noise, and an embedding failure degrades to pure BM25. GPU is
  optional (`embedding_device="cuda"`, the `[rerank-gpu]` extra).
- **Content deduplication** — on by default: a word-shingle SimHash drops the
  same article reaching you under different URLs (syndicated wire stories,
  mirrors, scraped copies) that URL-identity dedup can't catch, while leaving
  distinct same-topic articles alone. `dedup_max_hamming=0` disables it.
- **Chunk-level assembly (default)** — each page's budget is spent on
  its most relevant paragraphs (in document order, with visible `[...]` gap
  markers) instead of keeping the page head, so an answer past the budget line
  still ships. `assembly_mode="pages"` restores the pre-0.5 head-truncation.
- **Char- or token-budgeting** — the per-source budget is measured in
  characters by default; set `budget_unit="tokens"` to spend it in model tokens
  instead, so CJK/code-heavy pages ship the context they were actually allotted.
  Exact counts via tiktoken (the `[tokens]` extra), else a dependency-free
  CJK-aware heuristic. `chars` mode is byte-identical to before.
- **Code & table fidelity** — structured content survives the pipeline coherent
  on doc pages: fenced code keeps its markers and structural lines (the prose
  cleaner used to drop them), original indentation is restored from the source
  `<pre>` with a language info-string (```` ```python ````), even when
  trafilatura glues text onto a fence delimiter; markdown tables are kept as
  atomic chunks with their header carried onto every piece of an oversized
  table, so rows never reach the model orphaned from their columns.
- **Prompt-injection flagging** — on by default (`detect_injection`): fetched
  content is scanned for injection-like text (instruction overrides, role/persona
  spoofing, prompt-extraction) and a match is surfaced as an info degradation. It
  flags, never censors — a best-effort signal on untrusted web data, not a
  safety boundary.
- **Recency-aware ranking** — pages carry their own declared publication dates
  (extracted from page metadata, explicit tags only — never guessed) merged
  with provider dates; a decay-weighted boost ranks fresher answers higher by
  default, works on any provider, and never penalizes undated pages.
  `recency_boost=0` restores pure BM25.
- **Date/time/location awareness** — every LLM-facing surface knows *now* and
  *here*: prompts carry the current date/time (configured `timezone`) and an
  opt-in `location` hint; every research context opens with a
  `Research performed: <ISO-8601>` header; the MCP server surfaces the locale
  in its instructions/tool descriptions so calling models localize queries
  with zero extra calls.
- **No-fail-silent contract** — every degradation (blocked URL, truncated page,
  provider fallback, budget cut) is a typed, enumerable warning; nothing
  disappears without a trace.
- **LLM query expansion (optional)** — expand a question into multiple search
  queries via any OpenAI-compatible endpoint or an injected callback.
- **Caching** — in-memory by default, sqlite persistence a config flag away.
- **OpenTelemetry metrics (optional)** — `metrics_enabled` + the `[otel]` extra
  pushes per-stage latency (by provider), fetch outcomes, cache hits,
  degradations, and budget utilization to an OTLP collector, derived from the
  run stats the pipeline already produces. Off by default, MCP-server-only,
  fail-soft (a metrics fault never fails a tool call).
- **Typed throughout** — pyright-strict clean, structured results on every
  surface (Pydantic models in the SDK, JSON structured output over MCP).
- **1000+ tests** including live-web smoke suites, hand-computed golden tests,
  and a recorded-corpus retrieval-quality eval gate (recall / nDCG / MRR /
  span-recall).

## How to use

### Python SDK

```bash
pip install "websearch-kit[ddgs]"   # ddgs = the zero-API-key search provider
```

```python
import asyncio
from websearch_kit import SearchKit

async def main():
    async with SearchKit() as kit:          # zero-config: ddgs, no keys, no LLM
        report = await kit.research("RISC-V vs ARM datacenter adoption")
        print(report.context)               # numbered [N] block for your LLM
        for s in report.sources:
            print(f"[{s.n}] {s.title} — {s.url}")
        print(report.warnings)              # everything the run degraded on

asyncio.run(main())
```

Beyond `research()`, the kit exposes the pipeline stages individually:

```python
results = await kit.search("python 3.14 free threading", count=5)     # snippets only
pages   = await kit.fetch(["https://docs.python.org/3.14/whatsnew/"])  # URLs, extracted
status  = await kit.health()                                           # provider probe
```

Prefer blocking code? `SyncSearchKit` mirrors the async API 1:1. Keyed
providers, fallback chains, sqlite caching, and LLM query expansion are all
config away — see [docs/deployment/sdk.md](docs/deployment/sdk.md) and
[examples/](examples/).

### MCP server

Add to your MCP client config (Claude Code, Claude Desktop, or any MCP client):

```json
{
  "mcpServers": {
    "websearch-kit": {
      "command": "uvx",
      "args": ["--from", "websearch-kit[mcp,ddgs]", "websearch-kit-mcp"]
    }
  }
}
```

Requires Python ≥ 3.10. If your client launches `uvx` against an older host
Python (e.g. LM Studio on macOS → system 3.9), prepend `"--python", "3.12"` to
`args` — uv downloads a modern interpreter if needed.

Four read-only tools with typed structured output, over stdio or streamable HTTP:

| Tool | What it does |
|------|--------------|
| `web_search` | Snippet-level results — context-economical |
| `fetch_page` | Read one URL as markdown, cursor pagination for long pages |
| `research` | Full pipeline → `[N]` context block + one resource link per citation |
| `deep_research` | Agentic multi-step research → one synthesized, `[N]`-cited answer (needs an orchestrator LLM) |
| `health` | Provider latency, circuit-breaker state, config checks |

For HTTP transport, scaling, and hardening flags see
[docs/deployment/mcp.md](docs/deployment/mcp.md) and
[examples/mcp_config_examples.md](examples/mcp_config_examples.md); a ready-to-use
system prompt for these tools is in
[examples/system_prompt.md](examples/system_prompt.md).

### Open WebUI

Import [`adapters/owui/websearch_kit_filter.json`](adapters/owui/websearch_kit_filter.json)
via **Admin Panel → Functions → Import** (OWUI's import expects its JSON
export format — or create a new Function and paste
[`websearch_kit_filter.py`](adapters/owui/websearch_kit_filter.py) instead).
It pip-installs this SDK automatically via its frontmatter `requirements:`
line and searches key-free out of the box (`ddgs`); valves switch it to your
instance's configured web search or a keyed provider.

Toggle the pill to research every message, or trigger one-off:

```
?? quantum routers --count 12 --lang en --reply de --fresh week
```

A Tool variant for model-invoked (agentic) use ships alongside it. See
[docs/deployment/owui.md](docs/deployment/owui.md).

## Documentation

- [SPEC.md](SPEC.md) — normative behavioral contract (pipeline semantics,
  ranking math, degradation codes, SSRF ruleset)
- [docs/architecture.md](docs/architecture.md) — how the engine is layered
- [docs/domains/](docs/domains/) — one standard document per domain
- [docs/adr/](docs/adr/) — the ten load-bearing architecture decisions
- [VERSIONING.md](VERSIONING.md) — SemVer policy and public-API definition
- [SECURITY.md](SECURITY.md) — SSRF posture and threat model

## License

MIT — with a CI-enforced permissive-only dependency policy (no GPL/AGPL;
`trafilatura>=1.8.0` pinned for its Apache-2.0 relicense).
