Metadata-Version: 2.4
Name: wildweb-mcp
Version: 0.1.1
Summary: An extensible MCP server that exposes web scraping as an MCP tool.
Author-email: Carel Wentzel <carel.wentzel@gmail.com>
License-Expression: MIT
Keywords: mcp,model-context-protocol,web-scraping,playwright,llm-tools
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp==1.13.1
Requires-Dist: playwright==1.65.0
Requires-Dist: trafilatura==2.3.1
Requires-Dist: tiktoken==0.14.0
Requires-Dist: httpx==0.28.1
Requires-Dist: lxml==6.1.3
Requires-Dist: pydantic==2.13.5
Provides-Extra: dev
Requires-Dist: pytest==7.4.4; extra == "dev"
Requires-Dist: hypothesis==6.154.1; extra == "dev"
Dynamic: license-file

# wildweb-mcp

An extensible [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server, written
in Python, that exposes web scraping as an MCP tool. It fetches a web page with a headless
Chromium browser (so client-rendered pages are captured), strips boilerplate (navigation, ads,
scripts, styles) with `trafilatura`, and returns token-efficient clean markdown to MCP clients.
No API key or third-party search/scrape API is required — it talks to the open web directly.

## Requirements

- Python **3.10+**
- Chromium browser binary — installed automatically on first run, see below

## Installation

Install into a virtualenv rather than system/global Python — it keeps `wildweb-mcp`'s pinned
dependencies isolated, and (as covered below) MCP clients need this venv's path anyway since they
launch the server directly, bypassing your shell:

```bash
python3 -m venv ~/.venvs/wildweb
~/.venvs/wildweb/bin/pip install wildweb-mcp
```

(`.venvs\wildweb\Scripts\pip.exe install wildweb-mcp` on Windows.)

### The Chromium browser

`pip install` installs the `playwright` Python package (it's a pinned dependency of
`wildweb-mcp`) but not the browser binary that Playwright drives.

**You don't need to install it yourself.** If Chromium isn't found when the server starts, it
runs `playwright install chromium` automatically — a one-time download (~300MB) into the shared
Playwright cache (`~/.cache/ms-playwright` on Linux/macOS,
`%USERPROFILE%\AppData\Local\ms-playwright` on Windows) — then retries. Every startup after that
is unaffected, since the binary is already there. This first-run install can take a few minutes
and needs network access; if it can't finish within 5 minutes (e.g. no network) or fails for
another reason, the server falls back to the existing clear startup error instead of hanging.
Progress and any install errors are logged — see [Where logs go](#where-logs-go) below.

If you'd rather install it ahead of time (e.g. to avoid the first-run delay, or in an offline/
sandboxed environment where the server shouldn't attempt a network call on its own), run it
yourself before starting the server — using the venv path from Installation above:

```bash
~/.venvs/wildweb/bin/playwright install chromium
```

(`~/.venvs/wildweb/Scripts/playwright.exe install chromium` on Windows.)

**A bare `playwright install chromium` will not work as a copy-paste pre-setup step** unless you
first activate that venv (`source ~/.venvs/wildweb/bin/activate`, or on Windows
`~/.venvs/wildweb/Scripts/activate`) — without either activation or the full path, your shell has
no `playwright` command on `PATH` at all (it only exists inside the venv you installed
`wildweb-mcp` into) and the command fails with "command not found." The full-path form above
works unconditionally, with nothing to activate first.

This has to be the `playwright` from the **same environment** `wildweb-mcp` was installed into —
if you installed globally/user-wide instead (`pip install --user` or system pip) rather than into
a venv, it lands in pip's script directory instead (e.g. `~/.local/bin` on Linux, your Python
install's `Scripts`/`bin` folder on macOS/Windows), and a bare `playwright install chromium` works
directly only if that directory is on `PATH`.

On Linux, Chromium may also need OS-level shared libraries. **This is never installed
automatically** — it requires root/sudo to run a system package manager, which the server will
not do on its own. If the server's startup error (or a manual `playwright install chromium`)
warns about missing dependencies, install them yourself:

```bash
playwright install-deps chromium
```

## Configuration

The server reads a single JSON configuration file. Copy the sample and edit it as needed:

```bash
cp config.example.json config.json
```

```json
{
  "request_timeout_ms": 30000,
  "token_budget": 4000,
  "log_level": "INFO",
  "continuation_ttl_seconds": 300,
  "browser": {
    "headless": true,
    "navigation_timeout_ms": 30000,
    "wait_until": "load",
    "max_retries": 2,
    "retry_backoff_ms": 500,
    "settle_ms": 0
  }
}
```

All fields are optional and fall back to documented defaults:

| Field                        | Default   | Description                                                        |
| ---------------------------- | --------- | ------------------------------------------------------------------ |
| `request_timeout_ms`         | `30000`   | Logical request timeout in milliseconds.                           |
| `token_budget`               | `null`    | Max tokens per tool output; `null` disables budgeting/truncation.  |
| `log_level`                  | `"INFO"`  | One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`.            |
| `continuation_ttl_seconds`   | `300`     | Time-to-live for continuation tokens, in seconds.                  |
| `browser.headless`           | `true`    | Launch Chromium headless.                                          |
| `browser.navigation_timeout_ms` | `30000` | Playwright navigation timeout; defaults to `request_timeout_ms`.  |
| `browser.wait_until`         | `"load"`  | One of `load`, `domcontentloaded`, `networkidle`, `commit`.        |
| `browser.max_retries`        | `2`       | Extra attempts after a transient navigation timeout or network error (connection reset, DNS hiccup); a non-2xx HTTP status is never retried. `0` disables retry. |
| `browser.retry_backoff_ms`   | `500`     | Base delay before each retry, doubled per attempt (500ms, then 1000ms, ...). |
| `browser.settle_ms`          | `0`       | Extra wait after navigation completes, before reading the page, so content a site renders asynchronously after load has a chance to appear. |
| `search_result_count`        | `10`      | Number of results the `search` tool returns per query.             |
| `search_timeout_ms`          | `10000`   | Timeout in milliseconds for a `search` request.                    |
| `user_agents`                | `null`    | Array of User-Agent strings to rotate through; `null` uses the built-in default pool. |
| `log_file`                   | `null`    | Path to also write logs to, capped at 10MB (see [Where logs go](#where-logs-go)); `null` logs to stderr only. |

### Simulating a modern browser

Every outgoing request (Playwright page navigation and the `search` tool's DuckDuckGo
request) identifies as an ordinary modern desktop browser rather than a generic
client/library default, since some sites are more likely to serve their normal response
to a request that looks like one. Requests draw from a rotating pool of real browser
User-Agent strings (Chrome, Firefox, Safari, Edge) round-robin, one per request, rather
than reusing a single fixed value.

Set `user_agents` to replace the default pool with your own:

```json
{
  "user_agents": [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15"
  ]
}
```

The array must be non-empty; omit the field or set it to `null` to use the default pool.

## Where logs go

All server logging — startup messages (including the automatic Chromium install above),
per-request success/failure entries, and errors — is written to **stderr**, never stdout.
This is deliberate: stdout is reserved exclusively for the MCP JSON-RPC protocol stream between
the server and its client, so anything else written there would corrupt that stream.

What you see depends on how your MCP client surfaces a subprocess's stderr. Most already capture
it to their own log file, so check there first:

- **Claude Desktop**: `~/Library/Logs/Claude/mcp-server-<name>.log` (macOS),
  `%APPDATA%\Claude\logs\mcp-server-<name>.log` (Windows)
- **Claude Code**: `claude mcp logs <name>`
- **Kiro CLI / opencode**: each client's own MCP connection log/output panel — check its docs for
  the exact location

Or run `wildweb` directly in a terminal to see log lines interleaved with your own shell.
`log_level` (see the [Configuration](#configuration) table above) controls verbosity: `DEBUG`
through `CRITICAL`, defaulting to `INFO`.

### Writing logs to a file too

If your client doesn't capture stderr usefully, or you just want one log location regardless of
which client launched the server, set `log_file` to a path:

```json
{
  "log_file": "/absolute/path/to/wildweb.log"
}
```

This writes the same entries to that file *in addition to* stderr — nothing stops going to
stderr. The file is capped at 10MB: once it hits that size, it's truncated (overwritten from
empty) rather than rotated to `.1`/`.2` backups, so it's always exactly one bounded file, never
growing unbounded across a long-running server process and never leaving old backup files to
clean up. If the path can't be opened (e.g. its parent directory doesn't exist), the server logs
a warning and continues with stderr-only logging rather than failing to start.

## Usage

A config file is optional. The config file path is resolved with the following
precedence:

1. `--config <path>` command-line argument (highest precedence)
2. `WILDWEB_CONFIG` environment variable
3. If neither is provided, the server runs with the built-in defaults listed
   under [Configuration](#configuration) — no config file is required.

```bash
# with no config at all — runs with built-in defaults
wildweb

# via command-line argument
wildweb --config config.json

# or via environment variable
WILDWEB_CONFIG=config.json wildweb
```

MCP clients connect over stdio and discover three tools:

- **`scrape`** — fetch a web page by URL and return its main textual content as clean markdown.
  When the content exceeds the configured `token_budget`, the result is truncated and a
  continuation token is returned. On a directory/listing page (a customer-stories grid, a blog
  index) where boilerplate removal would otherwise collapse the result down to just the one or two
  entries that look the most "article-like", a detected repeated group of same-class links is
  appended as a `## Links on this page` section instead of being silently dropped.
- **`scrape_continue`** — fetch the next portion of a previously scraped page using a
  continuation token returned by `scrape`.
- **`search`** — search the web for a query and return a list of results (title, URL, snippet).
  Input: a non-empty text query. Output: up to `search_result_count` ordered results, or an
  error. The `search` tool does **not** fetch or scrape any result's page content — pipe a
  result's `url` into `scrape` for that.

### Search provider caveat

The `search` tool is backed by a free, key-free `SearchProvider` implementation
(`DuckDuckGoSearchProvider`) that fetches and parses DuckDuckGo's public HTML results page
(`https://html.duckduckgo.com/html/`). This is an **unofficial surface**, not a documented or
supported API:

- DuckDuckGo can change its HTML markup, rate-limit, or block automated requests at any time
  without notice, which will surface as a `search_unavailable` error from the `search` tool.
- Using it at any significant volume may be subject to DuckDuckGo's Terms of Service; this
  provider is intended for light, personal-scale use.
- The `SearchProvider` interface (`src/wildweb/search/base.py`) exists specifically so a
  different, officially-supported provider (e.g. a paid search API) can be registered later
  without changing the `search` tool or server wiring — see `src/wildweb/search/registry.py`.

## Adding it to an MCP client

Once installed (see [Installation](#installation) above), the `wildweb` console script is what
MCP clients launch over stdio.

### Point your client at the venv's executable, not a bare command

An MCP client launches the server as a plain subprocess using exactly the `command` string in
its own config — that launch does not go through your shell, so it never activates the virtualenv
you installed into and often doesn't even inherit your normal shell `PATH`. A bare
`"command": "wildweb"` only works if `wildweb` happens to resolve on whatever `PATH` that
client's process has, which is unreliable. The robust fix, used throughout every example below:
give the client the venv's own executable path directly (the `~/.venvs/wildweb` venv from
Installation), instead of relying on `PATH` resolution:

```json
{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb"
}
```

(`~/.venvs/wildweb/Scripts/wildweb.exe` on Windows — adjust the path in every example below the
same way.) No `args` needed for this alone, no `PATH` dependency, nothing to activate. Replace
`/Users/you/.venvs/wildweb` with your own venv's actual path.

### Config: none, a file, or an environment variable

The simplest entry needs no config at all — the server runs with its built-in defaults:

```json
{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb"
}
```

To use a config file instead, add `--config` with an absolute path (the client usually
launches the server from its own working directory, not this project's):

```json
{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb",
  "args": ["--config", "/absolute/path/to/config.json"]
}
```

Alternatively, set `WILDWEB_CONFIG` as an environment variable on the entry instead of passing
`--config`.

### Claude Desktop

Add a `wildweb` entry under `mcpServers` in Claude Desktop's config file
(`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS,
`%APPDATA%\Claude\claude_desktop_config.json` on Windows). No config (built-in defaults):

```json
{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb"
    }
  }
}
```

With a config file:

```json
{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "args": ["--config", "/absolute/path/to/config.json"]
    }
  }
}
```

Restart Claude Desktop for the change to take effect.

### Claude Code

Add it as a user- or project-scoped server with the `claude mcp add` CLI. The first `wildweb` is
the server's name within Claude Code; the second is the command it runs. No config:

```bash
claude mcp add wildweb /Users/you/.venvs/wildweb/bin/wildweb
```

With a config file:

```bash
claude mcp add wildweb /Users/you/.venvs/wildweb/bin/wildweb -- --config /absolute/path/to/config.json
```

Or add the equivalent `mcpServers` block shown above directly to the project's `.mcp.json`.

### Kiro CLI

Add a `wildweb` entry under `mcpServers` in Kiro's MCP config file — globally at
`~/.kiro/settings/mcp.json`, or per-project at `.kiro/settings/mcp.json` (workspace settings
take precedence when both exist). No config:

```json
{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "disabled": false,
      "autoApprove": []
    }
  }
}
```

With a config file:

```json
{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "args": ["--config", "/absolute/path/to/config.json"],
      "disabled": false,
      "autoApprove": []
    }
  }
}
```

Kiro does not inherit your shell's `PATH` at all, which is exactly why the `command` above is the
venv's full path rather than a bare `wildweb`. Run `/mcp` inside Kiro CLI to confirm the server
loaded.

### opencode

Add a local server under `mcp` in `opencode.jsonc` (project root) or the global
`~/.config/opencode/opencode.jsonc`. No config:

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "wildweb": {
      "type": "local",
      "command": ["/Users/you/.venvs/wildweb/bin/wildweb"]
    }
  }
}
```

With a config file:

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "wildweb": {
      "type": "local",
      "command": [
        "/Users/you/.venvs/wildweb/bin/wildweb",
        "--config",
        "/absolute/path/to/config.json"
      ]
    }
  }
}
```

Or add it from the CLI. No config:

```bash
opencode mcp add wildweb -- /Users/you/.venvs/wildweb/bin/wildweb
```

With a config file:

```bash
opencode mcp add wildweb -- /Users/you/.venvs/wildweb/bin/wildweb --config /absolute/path/to/config.json
```

### Other clients

Any MCP client that launches servers over stdio (Cursor, Windsurf, etc.) accepts the same
`command`/`args` shape under whatever key its own config format uses for MCP servers — consult
that client's docs for the exact file location.
