Metadata-Version: 2.4
Name: wildweb-mcp
Version: 0.1.4
Summary: An extensible MCP server that exposes web scraping as an MCP tool.
Author-email: Carel Wentzel <carel.wentzel@gmail.com>
License-Expression: MIT
Keywords: mcp,model-context-protocol,web-scraping,playwright,llm-tools
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp==1.13.1
Requires-Dist: playwright==1.65.0
Requires-Dist: trafilatura==2.3.1
Requires-Dist: tiktoken==0.14.0
Requires-Dist: httpx==0.28.1
Requires-Dist: lxml==6.1.3
Requires-Dist: pydantic==2.13.5
Provides-Extra: dev
Requires-Dist: pytest==7.4.4; extra == "dev"
Requires-Dist: hypothesis==6.154.1; extra == "dev"
Dynamic: license-file

# wildweb-mcp

An extensible [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server, written
in Python, that exposes web scraping as an MCP tool. It fetches a web page with a headless
Chromium browser (so client-rendered pages are captured), strips boilerplate (navigation, ads,
scripts, styles) with `trafilatura`, and returns token-efficient clean markdown to MCP clients.
No API key or third-party search/scrape API is required — it talks to the open web directly.

## Requirements

- Python **3.10+**
- Chromium browser binary — installed automatically on first run, see below

## Installation

Install into a virtualenv rather than system/global Python — it keeps `wildweb-mcp`'s pinned
dependencies isolated, and (as covered below) MCP clients need this venv's path anyway since they
launch the server directly, bypassing your shell:

```bash
python3 -m venv ~/.venvs/wildweb
~/.venvs/wildweb/bin/pip install wildweb-mcp
```

(`.venvs\wildweb\Scripts\pip.exe install wildweb-mcp` on Windows.)

### Upgrading

Using the same venv, add `--upgrade`:

```bash
~/.venvs/wildweb/bin/pip install --upgrade wildweb-mcp
```

Confirm the new version:

```bash
~/.venvs/wildweb/bin/python -c "import wildweb; print(wildweb.__version__)"
```

No MCP client config changes needed — the `command` path stays the same, it's the same venv with
newer code inside it. Restart whichever MCP client has it running (Claude Desktop, Claude Code,
etc.) so it relaunches the server process with the update.

### The Chromium browser

`pip install` installs the `playwright` Python package (it's a pinned dependency of
`wildweb-mcp`) but not the browser binary that Playwright drives.

**You don't need to install it yourself.** If Chromium isn't found when the server starts, it
runs `playwright install chromium` automatically — a one-time download (~300MB) into the shared
Playwright cache (`~/.cache/ms-playwright` on Linux/macOS,
`%USERPROFILE%\AppData\Local\ms-playwright` on Windows) — then retries. Every startup after that
is unaffected, since the binary is already there. This first-run install can take a few minutes
and needs network access; if it can't finish within 5 minutes (e.g. no network) or fails for
another reason, the server falls back to the existing clear startup error instead of hanging.
Progress and any install errors are logged — see [Where logs go](#where-logs-go) below.

If you'd rather install it ahead of time (e.g. to avoid the first-run delay, or in an offline/
sandboxed environment where the server shouldn't attempt a network call on its own), run it
yourself before starting the server — using the venv path from Installation above:

```bash
~/.venvs/wildweb/bin/playwright install chromium
```

(`~/.venvs/wildweb/Scripts/playwright.exe install chromium` on Windows.)

**A bare `playwright install chromium` will not work as a copy-paste pre-setup step** unless you
first activate that venv (`source ~/.venvs/wildweb/bin/activate`, or on Windows
`~/.venvs/wildweb/Scripts/activate`) — without either activation or the full path, your shell has
no `playwright` command on `PATH` at all (it only exists inside the venv you installed
`wildweb-mcp` into) and the command fails with "command not found." The full-path form above
works unconditionally, with nothing to activate first.

This has to be the `playwright` from the **same environment** `wildweb-mcp` was installed into —
if you installed globally/user-wide instead (`pip install --user` or system pip) rather than into
a venv, it lands in pip's script directory instead (e.g. `~/.local/bin` on Linux, your Python
install's `Scripts`/`bin` folder on macOS/Windows), and a bare `playwright install chromium` works
directly only if that directory is on `PATH`.

On Linux, Chromium may also need OS-level shared libraries. **This is never installed
automatically** — it requires root/sudo to run a system package manager, which the server will
not do on its own. If the server's startup error (or a manual `playwright install chromium`)
warns about missing dependencies, install them yourself:

```bash
playwright install-deps chromium
```

## Configuration

The server reads a single JSON configuration file. Copy the sample and edit it as needed:

```bash
cp config.example.json config.json
```

```json
{
  "request_timeout_ms": 30000,
  "token_budget": 4000,
  "log_level": "INFO",
  "continuation_ttl_seconds": 300,
  "browser": {
    "headless": true,
    "navigation_timeout_ms": 30000,
    "wait_until": "domcontentloaded",
    "max_retries": 2,
    "retry_backoff_ms": 500,
    "settle_ms": 1000
  }
}
```

All fields are optional and fall back to documented defaults:

| Field                        | Default   | Description                                                        |
| ---------------------------- | --------- | ------------------------------------------------------------------ |
| `request_timeout_ms`         | `30000`   | Logical request timeout in milliseconds.                           |
| `token_budget`               | `null`    | Max tokens per tool output; `null` disables budgeting/truncation.  |
| `log_level`                  | `"INFO"`  | One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`.            |
| `continuation_ttl_seconds`   | `300`     | Time-to-live for continuation tokens, in seconds.                  |
| `browser.headless`           | `true`    | Launch Chromium headless.                                          |
| `browser.navigation_timeout_ms` | `30000` | Playwright navigation timeout; defaults to `request_timeout_ms`.  |
| `browser.wait_until`         | `"domcontentloaded"` | One of `load`, `domcontentloaded`, `networkidle`, `commit`. `load` waits for every subresource and can hang indefinitely on sites with a slow/stuck tracker or ad script. |
| `browser.max_retries`        | `2`       | Extra attempts after a transient navigation timeout or network error (connection reset, DNS hiccup); a non-2xx HTTP status is never retried. `0` disables retry. |
| `browser.retry_backoff_ms`   | `500`     | Base delay before each retry, doubled per attempt (500ms, then 1000ms, ...). |
| `browser.settle_ms`          | `1000`    | Extra wait after navigation completes, before reading the page, so content a site renders asynchronously after load has a chance to appear. Many modern (React/Next.js-style) sites return no content at all at `0` — set to `0` to disable if you don't need it. |
| `search_result_count`        | `10`      | Number of results the `search` tool returns per query.             |
| `search_timeout_ms`          | `10000`   | Timeout in milliseconds for a `search` request.                    |
| `user_agents`                | `null`    | Array of User-Agent strings to rotate through; `null` uses the built-in default pool. |
| `log_file`                   | `null`    | Path to also write logs to, capped at 10MB (see [Where logs go](#where-logs-go)); `null` logs to stderr only. |

### Simulating a modern browser

Every outgoing request (Playwright page navigation and the `search` tool's DuckDuckGo
request) identifies as an ordinary modern desktop browser rather than a generic
client/library default, since some sites are more likely to serve their normal response
to a request that looks like one. Requests draw from a rotating pool of real browser
User-Agent strings (Chrome, Firefox, Safari, Edge) round-robin, one per request, rather
than reusing a single fixed value.

Set `user_agents` to replace the default pool with your own:

```json
{
  "user_agents": [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15"
  ]
}
```

The array must be non-empty; omit the field or set it to `null` to use the default pool.

## Where logs go

All server logging — startup messages (including the automatic Chromium install above),
per-request success/failure entries, and errors — is written to **stderr**, never stdout.
This is deliberate: stdout is reserved exclusively for the MCP JSON-RPC protocol stream between
the server and its client, so anything else written there would corrupt that stream.

What you see depends on how your MCP client surfaces a subprocess's stderr. Most already capture
it to their own log file, so check there first:

- **Claude Desktop**: `~/Library/Logs/Claude/mcp-server-<name>.log` (macOS),
  `%APPDATA%\Claude\logs\mcp-server-<name>.log` (Windows)
- **Claude Code**: `claude mcp logs <name>`
- **Kiro CLI / opencode**: each client's own MCP connection log/output panel — check its docs for
  the exact location

Or run `wildweb` directly in a terminal to see log lines interleaved with your own shell.
`log_level` (see the [Configuration](#configuration) table above) controls verbosity: `DEBUG`
through `CRITICAL`, defaulting to `INFO`.

### Writing logs to a file too

If your client doesn't capture stderr usefully, or you just want one log location regardless of
which client launched the server, set `log_file` to a path:

```json
{
  "log_file": "/absolute/path/to/wildweb.log"
}
```

This writes the same entries to that file *in addition to* stderr — nothing stops going to
stderr. The file is capped at 10MB: once it hits that size, it's truncated (overwritten from
empty) rather than rotated to `.1`/`.2` backups, so it's always exactly one bounded file, never
growing unbounded across a long-running server process and never leaving old backup files to
clean up. If the path can't be opened (e.g. its parent directory doesn't exist), the server logs
a warning and continues with stderr-only logging rather than failing to start.

## Usage

A config file is optional. The config file path is resolved with the following
precedence:

1. `--config <path>` command-line argument (highest precedence)
2. `WILDWEB_CONFIG` environment variable
3. If neither is provided, the server runs with the built-in defaults listed
   under [Configuration](#configuration) — no config file is required.

```bash
# with no config at all — runs with built-in defaults
wildweb

# via command-line argument
wildweb --config config.json

# or via environment variable
WILDWEB_CONFIG=config.json wildweb
```

MCP clients connect over stdio and discover three tools:

- **`scrape`** — fetch a web page by URL and return its main textual content as clean markdown.
  When the content exceeds the configured `token_budget`, the result is truncated and a
  continuation token is returned. On a directory/listing page (a customer-stories grid, a blog
  index) where boilerplate removal would otherwise collapse the result down to just the one or two
  entries that look the most "article-like", a detected repeated group of same-class links is
  appended as a `## Links on this page` section instead of being silently dropped.
- **`scrape_continue`** — fetch the next portion of a previously scraped page using a
  continuation token returned by `scrape`.
- **`search`** — search the web for a query and return a list of results (title, URL, snippet).
  Input: a non-empty text query. Output: up to `search_result_count` ordered results, or an
  error. The `search` tool does **not** fetch or scrape any result's page content — pipe a
  result's `url` into `scrape` for that.

### Search provider caveat

The `search` tool is backed by a free, key-free `SearchProvider` implementation
(`DuckDuckGoSearchProvider`) that fetches and parses DuckDuckGo's public HTML results page
(`https://html.duckduckgo.com/html/`). This is an **unofficial surface**, not a documented or
supported API:

- DuckDuckGo can change its HTML markup, rate-limit, or block automated requests at any time
  without notice, which will surface as a `search_unavailable` error from the `search` tool.
- Using it at any significant volume may be subject to DuckDuckGo's Terms of Service; this
  provider is intended for light, personal-scale use.
- The `SearchProvider` interface (`src/wildweb/search/base.py`) exists specifically so a
  different, officially-supported provider (e.g. a paid search API) can be registered later
  without changing the `search` tool or server wiring — see `src/wildweb/search/registry.py`.

## Adding it to an MCP client

Once installed (see [Installation](#installation) above), the `wildweb` console script is what
MCP clients launch over stdio.

### Point your client at the venv's executable, not a bare command

An MCP client launches the server as a plain subprocess using exactly the `command` string in
its own config — that launch does not go through your shell, so it never activates the virtualenv
you installed into and often doesn't even inherit your normal shell `PATH`. A bare
`"command": "wildweb"` only works if `wildweb` happens to resolve on whatever `PATH` that
client's process has, which is unreliable. The robust fix, used throughout every example below:
give the client the venv's own executable path directly (the `~/.venvs/wildweb` venv from
Installation), instead of relying on `PATH` resolution:

```json
{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb"
}
```

(`~/.venvs/wildweb/Scripts/wildweb.exe` on Windows — adjust the path in every example below the
same way.) No `args` needed for this alone, no `PATH` dependency, nothing to activate. Replace
`/Users/you/.venvs/wildweb` with your own venv's actual path.

### Config: none, a file, or an environment variable

The simplest entry needs no config at all — the server runs with its built-in defaults:

```json
{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb"
}
```

To use a config file instead, add `--config` with an absolute path (the client usually
launches the server from its own working directory, not this project's):

```json
{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb",
  "args": ["--config", "/absolute/path/to/config.json"]
}
```

Alternatively, set `WILDWEB_CONFIG` as an environment variable on the entry instead of passing
`--config`.

### Claude Desktop

Add a `wildweb` entry under `mcpServers` in Claude Desktop's config file
(`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS,
`%APPDATA%\Claude\claude_desktop_config.json` on Windows). No config (built-in defaults):

```json
{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb"
    }
  }
}
```

With a config file:

```json
{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "args": ["--config", "/absolute/path/to/config.json"]
    }
  }
}
```

Restart Claude Desktop for the change to take effect.

### Claude Code

Add it as a user- or project-scoped server with the `claude mcp add` CLI. The first `wildweb` is
the server's name within Claude Code; the second is the command it runs. No config:

```bash
claude mcp add wildweb /Users/you/.venvs/wildweb/bin/wildweb
```

With a config file:

```bash
claude mcp add wildweb /Users/you/.venvs/wildweb/bin/wildweb -- --config /absolute/path/to/config.json
```

Or add the equivalent `mcpServers` block shown above directly to the project's `.mcp.json`.

### Kiro CLI

Add a `wildweb` entry under `mcpServers` in Kiro's MCP config file — globally at
`~/.kiro/settings/mcp.json`, or per-project at `.kiro/settings/mcp.json` (workspace settings
take precedence when both exist). No config:

```json
{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "disabled": false,
      "autoApprove": []
    }
  }
}
```

With a config file:

```json
{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "args": ["--config", "/absolute/path/to/config.json"],
      "disabled": false,
      "autoApprove": []
    }
  }
}
```

Kiro does not inherit your shell's `PATH` at all, which is exactly why the `command` above is the
venv's full path rather than a bare `wildweb`. Run `/mcp` inside Kiro CLI to confirm the server
loaded.

### opencode

Add a local server under `mcp` in `opencode.jsonc` (project root) or the global
`~/.config/opencode/opencode.jsonc`. No config:

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "wildweb": {
      "type": "local",
      "command": ["/Users/you/.venvs/wildweb/bin/wildweb"]
    }
  }
}
```

With a config file:

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "wildweb": {
      "type": "local",
      "command": [
        "/Users/you/.venvs/wildweb/bin/wildweb",
        "--config",
        "/absolute/path/to/config.json"
      ]
    }
  }
}
```

Or add it from the CLI. No config:

```bash
opencode mcp add wildweb -- /Users/you/.venvs/wildweb/bin/wildweb
```

With a config file:

```bash
opencode mcp add wildweb -- /Users/you/.venvs/wildweb/bin/wildweb --config /absolute/path/to/config.json
```

### Other clients

Any MCP client that launches servers over stdio (Cursor, Windsurf, etc.) accepts the same
`command`/`args` shape under whatever key its own config format uses for MCP servers — consult
that client's docs for the exact file location.
