Metadata-Version: 2.5
Name: cluefinch
Version: 0.1.3
Summary: Cluefinch MCP server with SearXNG web search, SSRF-safe page fetching, and BM25 research retrieval.
Project-URL: Homepage, https://github.com/cluefinch/mcp-server
Project-URL: Repository, https://github.com/cluefinch/mcp-server
Project-URL: Issues, https://github.com/cluefinch/mcp-server/issues
Project-URL: Security, https://github.com/cluefinch/mcp-server/security/policy
Author: MicroNet
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
License-File: THIRD_PARTY_NOTICES.md
Requires-Python: >=3.12.4
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: httpcore[asyncio]<2,>=1.0
Requires-Dist: httpx>=0.27
Requires-Dist: mcp<2,>=1.30
Requires-Dist: pydantic<3,>=2.11
Requires-Dist: rank-bm25>=0.2.2
Requires-Dist: trafilatura>=2.0
Description-Content-Type: text/markdown

# Cluefinch MCP

Cluefinch is a local-first, version-aware research navigation layer for agents.
It provides SearXNG search, HTML reading, and source collection without a
separate LLM or paid search API. The agent plans, evaluates evidence, and
writes conclusions; the server supplies research capabilities.

[Repository](https://github.com/cluefinch/mcp-server) |
[Issues](https://github.com/cluefinch/mcp-server/issues) |
[Agent workflow](https://github.com/cluefinch/mcp-server/blob/main/AGENTS.md)

## Install

Requires Python 3.12.4+. After the first PyPI release, use an activated virtual
environment:

```sh
pip install cluefinch
cluefinch
```

This starts an MCP **stdio server**, not an interactive shell or web interface.
Connect it to an MCP client. Search requires a separate SearXNG instance with
JSON output enabled. Its default address is `http://127.0.0.1:8081`; override
it with `MCP_SEARCH_SEARXNG_URL`. The package does not install SearXNG.
These commands describe installation after publication, not a claim that the
package is already published. Neither uv nor Docker is required to install it.

### Migrating from mcp-search

Do not install `mcp-search` and `cluefinch` in the same environment: both can own
the `mcp_search` files. Prefer a new virtual environment. To reuse an existing
one, run `pip uninstall mcp-search` before `pip install cluefinch`.
The Python package remains `mcp_search`; `python -m mcp_search` and all
`MCP_SEARCH_` settings remain supported.

## Install from source

From a checkout of this repository, with uv installed:

```sh
uv sync --locked
python scripts/init_env.py
docker compose up -d
uv run cluefinch
```

Docker is only needed for the supplied SearXNG deployment. Reuse an existing
SearXNG instance if available. The sample binds to loopback port 8081.
For a different port, set `SEARXNG_PORT` in `.env` and update the MCP client's
`MCP_SEARCH_SEARXNG_URL` accordingly.

`init_env.py` creates a unique secret without overwriting an existing `.env`.
Compose passes it as `SEARXNG_SECRET`. Never commit `.env` or your local
`searxng/settings.yml`. `SEARXNG_IMAGE` accepts a tested tag or digest;
the default `searxng/searxng:latest` is a development convenience, not a pinned
release configuration. Upstream changes can affect search results.

## Connect an MCP client

For clients using `mcpServers`, point to the interpreter in your environment:

```json
{
  "mcpServers": {
    "cluefinch": {
      "command": "/absolute/path/.venv/bin/python",
      "args": ["-m", "mcp_search"],
      "env": {"MCP_SEARCH_SEARXNG_URL": "http://127.0.0.1:8081"}
    }
  }
}
```

On Windows, use a path such as `C:\\path\\.venv\\Scripts\\python.exe`.
Client configuration formats vary. Alternatively configure the installed
`cluefinch` executable with no arguments.

## Tools

Tools publish typed input and output schemas, including nested sources and
excerpts. Responses are validated before sending. Text JSON and MCP
`structuredContent` have the same flat structure. Expected failures return
`error` and `hint` instead of success fields.

| Task                                 | Tool or action                                                                         |
|--------------------------------------|----------------------------------------------------------------------------------------|
| Find candidate sources               | `web_search(query="...", max_results=5)`                                               |
| Restrict a domain and exclude others | `web_search(query="...", domain="python.org", exclude_domains=["discuss.python.org"])` |
| Preview a page                       | `web_fetch(url="...", max_chars=500)`                                                  |
| Continue reading when needed         | Call `response.continuation.tool` with `response.continuation.arguments` if non-null   |
| Expand a selected excerpt            | Call `excerpt.expand.tool` with `excerpt.expand.arguments`                             |
| Collect selected pages               | `research_collect(urls=[...])`                                                         |
| Collect query variants               | `research_collect(queries=[...], topic="...", max_sources=5)`                          |

`topic` alone does not start a search. Explicit URLs take priority; remaining
candidate slots are shared round-robin between query variants. `max_sources`
limits attempted candidates, not guaranteed successful sources. The server
does not invent replacement queries or sources.

### Reading and version checks

`max_chars` defaults to 10,000, capped at 20,000 in the default configuration.
A preview reduces returned context, not the initial page download.
Ready actions contain `{tool, arguments}` with the final URL, exact offset,
reading budget, and `expected_content_hash`. Actions are optional; generating
them makes no additional network request. Clients may qualify the local tool
name with a server prefix.

Continuation preserves the effective reading budget. `continuation: null`
means the retained text has ended. Expansion starts at the excerpt's start;
its default budget is 3,000 characters, capped by `MAX_FETCH_CHARS`. You can
adjust the budget while keeping the expected hash.

If the retained text differs, the server returns `content_changed` with no
slice or automatic retry. Reacquire the source without the old hash and select
coordinates again. A changed hash does not prove an old citation was false.
**A matching hash is not a freshness check against the origin.** Cached text can
still match after the website changes. Hashes do not identify cache hits or
changes beyond the retained extraction.

Manual reading remains available through `start_offset=response.next_start`
or `start_offset=excerpt.start_char`. Supply the corresponding source hash as
`expected_content_hash` for server-side version checking. Without it, reading
is unguarded. For preceding context, reduce the offset without going below zero.
Offsets count Python Unicode characters, not bytes or UTF-16 code units.

`truncated` means more **retained** text remains after the slice.
`text_truncated` means extraction exceeded the retained-text limit; pagination
cannot recover that discarded tail. `total_chars` is the retained length.

### Sources and limitations

`source_id` is `src_` plus the first 16 hex characters of the normalized final
URL's SHA-256. It is stable across calls and restarts, but does not identify a
document version. `content_hash` hashes the entire retained extracted text.

Excerpts are literal slices: `excerpt.text == text[start_char:end_char]`.
BM25 uses positive IDF and bounded passages from long paragraphs. When no
positive matches exist, leading eligible passages are returned. There is no
lemmatization or semantic relevance assessment. `heading` is local context,
not a full outline. Author, publication date, and HTML canonical URL are not
part of this contract.

`source_type` is a domain-based category, not a credibility rating. Unrecognized
domains receive `unknown`. Domain search operators depend on external engine
support. Search pagination is not implemented; `max_results` controls result count.

Inspect `unresponsive_engines` and collection `gaps` for provider failures,
blocked/unreadable sources, duplicates, and limits. Empty gaps do not establish
topic completeness. The agent decides whether evidence is sufficient.

After upgrading, restart the MCP connection so the client reloads schemas.
Version 0.1.3 adds response fields; clients validating against an older strict
schema must update it. Source installations should run `uv sync --locked`.

## Configuration

Pass settings in the MCP process environment. Only Docker Compose automatically
reads `.env`. Restart the server after changing settings. Numeric configuration
values must be positive and finite.

| Variable, prefixed with `MCP_SEARCH_`         | Default                                                 |
|-----------------------------------------------|---------------------------------------------------------|
| `SEARXNG_URL`                                 | `http://127.0.0.1:8081`                                 |
| `ENGINES`                                     | `google,bing,duckduckgo,wikipedia`                      |
| `UA`                                          | `Cluefinch/<version> (+https://github.com/cluefinch/mcp-server)`                    |
| `MAX_RESULTS` / `MAX_SOURCES` / `MAX_QUERIES` | 20 / 10 / 10                                            |
| `MAX_FETCH_CHARS` / `MAX_TEXT_CHARS`          | 20,000 / 100,000 characters                             |
| `EXPAND_CHARS`                                | 3,000 characters, capped at `MAX_FETCH_CHARS`           |
| `MIN_EXTRACTED_CHARS`                         | 100 characters                                          |
| `MAX_RESPONSE_BYTES`                          | 5,242,880 bytes, checked before and after decompression |
| `MAX_REDIRECTS`                               | 5                                                       |
| `DOWNLOAD_TIMEOUT`                            | 60 seconds, including redirects                         |
| `FETCH_CONCURRENCY` / `HOST_INTERVAL`         | 3 / 5 seconds between request starts per hostname                                            |
| `SEARCH_INTERVAL`                             | 5 seconds after each SearXNG request completes                       |
| `FETCH_TTL` / `SEARCH_TTL`                    | 600 / 300 seconds                                       |
| `CACHE_ENTRIES`                               | 500 per cache                                           |

`MCP_SEARCH_UA` sets the outgoing User-Agent for page downloads, including pages
collected by `research_collect`. The default identifies Cluefinch and uses the
installed package version. Override it for a deployment-specific identity and
restart the MCP server. It does not change SearXNG's own requests to search engines.

Search requests are serialized with a five-second pause after completion, including
failures. Page downloads use a separate five-second start-to-start interval per
hostname and never overlap for that hostname. Up to three downloads to different
hosts can run concurrently; redirects also pass through the limiter. Cache hits
do not download the page or repeat the search. Limits are per MCP process, not
shared across processes or subdomains. Waiting for a page-download slot counts
toward `DOWNLOAD_TIMEOUT`.

These intervals reduce request frequency; they do not guarantee acceptance by a
website. The server does not currently enforce `robots.txt` or `Retry-After`, and
does not automatically retry failed downloads. Respect site and provider policies.

To add engines, update both SearXNG and `MCP_SEARCH_ENGINES`. When a call omits
`engines`, search uses SearXNG's enabled engines. Caches are process-local with
TTL and LRU limits. Character limits are not memory-size limits.

## Support and security boundaries

Supports HTML/XHTML over HTTP(S). JavaScript rendering, PDF, authentication,
CAPTCHA solving, and bypassing access restrictions are not implemented.
Insufficient extracted text produces an error; this can indicate a short page
as well as a JavaScript-rendered page. The transport requests gzip/identity;
unsupported encodings and malformed or concatenated gzip fail explicitly.
Decoding considers HTTP charset, BOM, and HTML meta-charset.

Outbound fetching rejects URL credentials, private/local/reserved and multicast
IP addresses, scoped hosts, and mixed public/private DNS answers. Redirects are
validated. Connections use validated numeric IPs while retaining the original
hostname for Host, TLS SNI, and certificate verification. Environment proxies
are disabled. Custom NAT64 prefixes or unusual routing require an infrastructure
egress policy. Request limits apply to actual HTTP transitions, including redirects.

These controls do not filter prompt injection. Retrieved content is untrusted
data, not instructions. The sample deployment is local, not a public hosted-service
security profile. See the [security policy](https://github.com/cluefinch/mcp-server/blob/main/SECURITY.md).

## Development

```sh
uv sync --locked
uv run pytest -q
uv run ruff check mcp_search tests scripts
uv run ruff format --check mcp_search tests scripts
uv build
```

Tests include real MCP stdio exchanges without external network access.
`uv run python scripts/smoke.py` additionally checks live search and pages;
it needs internet and SearXNG and accepts `--url` and `--query` overrides.
CI checks Windows/Linux with Python 3.12/3.13.

See [development checks](https://github.com/cluefinch/mcp-server/blob/main/docs/DEVELOPMENT.md)
and [publishing](https://github.com/cluefinch/mcp-server/blob/main/docs/PYPI_FIRST_RELEASE.md).

## License

Original code is licensed under [Apache-2.0](https://github.com/cluefinch/mcp-server/blob/main/LICENSE).
[NOTICE](https://github.com/cluefinch/mcp-server/blob/main/NOTICE) and
[third-party notices](https://github.com/cluefinch/mcp-server/blob/main/THIRD_PARTY_NOTICES.md)
describe component boundaries. SearXNG is a separate AGPL-licensed service;
the core license does not cover its code, dependencies, or retrieved content.

See [contributing](https://github.com/cluefinch/mcp-server/blob/main/CONTRIBUTING.md)
for Apache-2.0 and DCO requirements, and
[SearXNG integration requirements](https://github.com/cluefinch/mcp-server/blob/main/docs/SEARXNG_COMPLIANCE.md)
before bundling SearXNG or deploying a hosted service.
