Metadata-Version: 2.4
Name: websearch-kit
Version: 0.4.0
Summary: Web search, fetch, and research pipeline for LLMs — usable as a Python SDK, a standalone MCP server, and an Open WebUI plugin.
Project-URL: Homepage, https://github.com/rmarnold/websearch-kit
Project-URL: Changelog, https://github.com/rmarnold/websearch-kit/blob/main/CHANGELOG.md
Author: Robert Arnold
License-Expression: MIT
License-File: LICENSE
Keywords: llm,mcp,model-context-protocol,open-webui,rag,scraping,search,web-search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx[brotli,http2,zstd]>=0.27
Requires-Dist: protego>=0.3
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: pydantic>=2.7
Requires-Dist: readability-lxml>=0.8.1
Requires-Dist: trafilatura>=1.8.0
Provides-Extra: all
Requires-Dist: ddgs>=9.0; extra == 'all'
Requires-Dist: fastembed>=0.5; extra == 'all'
Requires-Dist: mcp<2,>=1.12; extra == 'all'
Requires-Dist: pypdf>=5.0; extra == 'all'
Provides-Extra: ddgs
Requires-Dist: ddgs>=9.0; extra == 'ddgs'
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.12; extra == 'mcp'
Provides-Extra: owui
Provides-Extra: pdf
Requires-Dist: pypdf>=5.0; extra == 'pdf'
Provides-Extra: rerank
Requires-Dist: fastembed>=0.5; extra == 'rerank'
Description-Content-Type: text/markdown

# websearch-kit

> Web search, fetch, and research pipeline for LLMs — one engine, three surfaces:
> a Python **SDK**, a standalone **MCP server**, and an **Open WebUI** plugin.

Query expansion → multi-provider search → SSRF-guarded concurrent fetching
(40-UA browser profile, pinned-IP connect) → trafilatura extraction → BM25
rerank with adaptive context budgeting → numbered, citable context for your LLM.

**No fail-silent:** a call either raises a typed error or returns a result where
every dropped, blocked, truncated, or substituted item is enumerated as a
structured `Degradation`. On the live web that looks like:

```text
ok        : True   partial: True
sources   : 10                       # 5 fetched pages + 5 relevance-filtered snippets
warnings  :
  - [fetch] https://cloud.google.com/...: response exceeded byte cap (1054971 > 1048576 bytes)
stats     : 10 raw -> 10 unique, 5 fetched, context 23471 chars,
            timings {'search': 854, 'fetch': 1662, 'extract': 878, 'rank': 3}
```

**Status: 0.1.0.** See [SPEC.md](SPEC.md), [CHANGELOG.md](CHANGELOG.md).

## Features

- **One engine, three surfaces** — the SDK core is the only pipeline; the MCP
  server and Open WebUI adapters are thin translators over it, so behavior and
  config semantics never drift between surfaces.
- **Full research pipeline** — `research()` runs search → fetch → extract →
  rank → assemble in one call and returns a numbered `[N]` context block with
  1:1 source citations, ready to drop into a prompt.
- **Multi-provider search** — zero-key `ddgs` out of the box; keyed
  Tavily / Brave / Serper / Exa and self-hosted SearXNG via config; ordered
  fallback chains with per-provider circuit breakers.
- **Hardened fetching** — SSRF guard (private / reserved / metadata IP ranges
  blocked at connect time with pinned-IP enforcement), per-response byte caps,
  rotating 40-UA browser profile, concurrent fetches with deadline budgeting.
- **Quality extraction & ranking** — trafilatura article extraction, BM25
  reranking (golden-tested math), adaptive context budgeting: the most relevant
  pages get more of the character budget, marginal ones shrink, noise is dropped.
- **No-fail-silent contract** — every degradation (blocked URL, truncated page,
  provider fallback, budget cut) is a typed, enumerable warning; nothing
  disappears without a trace.
- **LLM query expansion (optional)** — expand a question into multiple search
  queries via any OpenAI-compatible endpoint or an injected callback.
- **Caching** — in-memory by default, sqlite persistence a config flag away.
- **Typed throughout** — pyright-strict clean, structured results on every
  surface (Pydantic models in the SDK, JSON structured output over MCP).
- **688+ tests** including live-web smoke suites and hand-computed golden tests.

## How to use

### Python SDK

```bash
pip install "websearch-kit[ddgs]"   # ddgs = the zero-API-key search provider
```

```python
import asyncio
from websearch_kit import SearchKit

async def main():
    async with SearchKit() as kit:          # zero-config: ddgs, no keys, no LLM
        report = await kit.research("RISC-V vs ARM datacenter adoption")
        print(report.context)               # numbered [N] block for your LLM
        for s in report.sources:
            print(f"[{s.n}] {s.title} — {s.url}")
        print(report.warnings)              # everything the run degraded on

asyncio.run(main())
```

Beyond `research()`, the kit exposes the pipeline stages individually:

```python
results = await kit.search("python 3.14 free threading", count=5)     # snippets only
pages   = await kit.fetch(["https://docs.python.org/3.14/whatsnew/"])  # URLs, extracted
status  = await kit.health()                                           # provider probe
```

Prefer blocking code? `SyncSearchKit` mirrors the async API 1:1. Keyed
providers, fallback chains, sqlite caching, and LLM query expansion are all
config away — see [docs/deployment/sdk.md](docs/deployment/sdk.md) and
[examples/](examples/).

### MCP server

Add to your MCP client config (Claude Code, Claude Desktop, or any MCP client):

```json
{
  "mcpServers": {
    "websearch-kit": {
      "command": "uvx",
      "args": ["--from", "websearch-kit[mcp,ddgs]", "websearch-kit-mcp"]
    }
  }
}
```

Four read-only tools with typed structured output, over stdio or streamable HTTP:

| Tool | What it does |
|------|--------------|
| `web_search` | Snippet-level results — context-economical |
| `fetch_page` | Read one URL as markdown, cursor pagination for long pages |
| `research` | Full pipeline → `[N]` context block + one resource link per citation |
| `health` | Provider latency, circuit-breaker state, config checks |

For HTTP transport, scaling, and hardening flags see
[docs/deployment/mcp.md](docs/deployment/mcp.md) and
[examples/mcp_config_examples.md](examples/mcp_config_examples.md).

### Open WebUI

Import [`adapters/owui/websearch_kit_filter.json`](adapters/owui/websearch_kit_filter.json)
via **Admin Panel → Functions → Import** (OWUI's import expects its JSON
export format — or create a new Function and paste
[`websearch_kit_filter.py`](adapters/owui/websearch_kit_filter.py) instead).
It pip-installs this SDK automatically via its frontmatter `requirements:`
line and searches key-free out of the box (`ddgs`); valves switch it to your
instance's configured web search or a keyed provider.

Toggle the pill to research every message, or trigger one-off:

```
?? quantum routers --count 12 --lang en --reply de --fresh week
```

A Tool variant for model-invoked (agentic) use ships alongside it. See
[docs/deployment/owui.md](docs/deployment/owui.md).

## Documentation

- [SPEC.md](SPEC.md) — normative behavioral contract (pipeline semantics,
  ranking math, degradation codes, SSRF ruleset)
- [docs/architecture.md](docs/architecture.md) — how the engine is layered
- [docs/domains/](docs/domains/) — one standard document per domain
- [docs/adr/](docs/adr/) — the eight load-bearing decisions
- [VERSIONING.md](VERSIONING.md) — SemVer policy and public-API definition
- [SECURITY.md](SECURITY.md) — SSRF posture and threat model

## License

MIT — with a CI-enforced permissive-only dependency policy (no GPL/AGPL;
`trafilatura>=1.8.0` pinned for its Apache-2.0 relicense).
