Metadata-Version: 2.4
Name: mcp-server-webcrawl
Version: 0.9.0
Summary: MCP server for search and retrieval of web crawler content
Author: Ben Caulfield
Project-URL: Homepage, https://pragmar.com/mcp-server-webcrawl/
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp>=1.3.0
Requires-Dist: Pillow>=9.0.0
Requires-Dist: aiohttp>=3.8.0
Requires-Dist: warcio>=1.7.0
Requires-Dist: ply==3.11
Requires-Dist: html2text>=2020.1.16
Dynamic: license-file

# mcp-server-webcrawl

Bridge the gap between your web crawl and AI language models using Model Context Protocol (MCP).
With **mcp-server-webcrawl**, your AI client filters and analyzes web content under your direction or autonomously. The server includes a full-text search interface with boolean support, resource filtering by type, HTTP status,
and more.

**mcp-server-webcrawl** provides the LLM a complete menu with which to search your web content, and works with
a variety of web crawlers:

* [WARC](https://en.wikipedia.org/wiki/WARC_(file_format))
* [wget](https://en.wikipedia.org/wiki/Wget)
* [InterroBot](https://interro.bot)
* [Katana](https://github.com/projectdiscovery/katana)
* [SiteOne](https://crawler.siteone.io)

**mcp-server-webcrawl** is free and open source, and requires Claude Desktop, Python (>=3.10). It is installed on the command line, via pip install:

```bash
pip install mcp-server-webcrawl
```

## Features

* Claude Desktop ready
* Fulltext search support
* Filter by type, status, and more
* Multi-crawler compatible
* Supports advanced/boolean and field searching

## MCP Configuration

From the Claude Desktop menu, navigate to File > Settings > Developer. Click Edit Config to locate the configuration file, open in the editor of your choice and modify the example to reflect your datasrc path.

You can set up more mcp-server-webcrawl connections under mcpServers as needed.

```json
{
  "mcpServers": {
    "webcrawl": {
      "command": [varies by OS/env, see below],
       "args": [varies by crawler, see below]
    }
  }
}
```

For step-by-step setup, refer to the [Setup Guides](https://pragmar.github.io/mcp-server-webcrawl/guides.html).

### Windows vs. macOS

On Windows with Python installed on path, the command should simply be `mcp-server-webcrawl`.

On macOS, you must use the absolute path to the `mcp-server-webcrawl` executable in the `command` field, rather than just the command name.

For example:

```json
"command": "/Users/yourusername/.local/bin/mcp-server-webcrawl",
```

To find the absolute path of the `mcp-server-webcrawl` executable on your system:

1. Open Terminal
2. Run `which mcp-server-webcrawl`
3. Copy the full path returned and use it in your config file

### wget (using --mirror)

The datasrc argument should be set to the parent directory of the mirrors.

```
"args": ["--crawler", "wget", "--datasrc", "/path/to/wget/archives/"]
```

### WARC

The datasrc argument should be set to the parent directory of the WARC files.

```
"args": ["--crawler", "warc", "--datasrc", "/path/to/warc/archives/"]
```

### InterroBot

The datasrc argument should be set to the direct path to the database.

```
"args": ["--crawler", "interrobot", "--datasrc", "/path/to/Documents/InterroBot/interrobot.v2.db"]
```

### Katana

The datasrc argument should be set to the parent directory of the text cache files.

```
"args": ["--crawler", "katana", "--datasrc", "/path/to/katana/archives/"]
```

### SiteOne (using archiving)

The datasrc argument should be set to the parent directory of the archives, archiving
must be enabled.

```
"args": ["--crawler", "siteone", "--datasrc", "/path/to/SiteOne/archives/"]
```

## Boolean Search

The query engine supports field-specific (`field: value`) searches and complex boolean
expressions. Fulltext is supported as a combination of the url, content, and headers fields.

While the API interface is designed to be consumed by the LLM directly, it can be helpful
to familiarize yourself with the search syntax. Searches generated by the LLM are
inspectable, but generally collapsed in the UI. If you need to see the query, expand
the MCP collapsable.

**Example Queries**

| Query | Description |
|-------|-------------|
| privacy | single fulltext search |
| "privacy policy" | fulltext match exact phrase |
| privacy* | matches wildcard fulltext results starting with "privacy" |
| id: 12345 | matches a specific resource by ID |
| url: example.com/* | matches results with URL containing example.com |
| type: html | HTML pages only |
| status: 200 | matches specific HTTP status code (equal) |
| status: >=400 | matches specific HTTP status code (greater than or equal to) |
| content: javascript | find javascript in HTTP body (often, but not always HTML) |
| headers: application/json | match HTTP response headers |
| privacy AND policy | match both as fulltext search |
| privacy OR policy | match either as fulltext search |
| policy NOT privacy | match fullext policies not containing privacy |
| (login OR signin) AND form | match fullext login or signin with form |
| type: html AND status: 200 | match only HTML pages with HTTP success |

**Field Search Definitions**

| Field | Description |
|-------|-------------|
| id | database ID |
| url | resource URL |
| type | enumerated list of types (see types table) |
| status | HTTP response code of result |
| headers | HTTP response headers |
| content | HTTP body—HTML, CSS, JS, and more |

**Content Types**

| Type | Description |
|------|-------------|
| html | web page |
| iframe | embedded iframe |
| img | web image formats |
| audio | audio files |
| video | video files |
| font | web font files |
| style | CSS stylesheets |
| script | JavaScript files |
| rss | RSS syndication feed |
| text | plain text content |
| pdf | PDF files |
| doc | MS Word documents |
| other | uncategorized |
