Metadata-Version: 2.5
Name: mcp-toolproof
Version: 0.3.1
Summary: Automated tests for MCP servers: schema checks, YAML tests, fuzzing, benchmarks, tool-selection evals and a pytest plugin.
Project-URL: Homepage, https://github.com/NilkanthSuthar/toolproof
Project-URL: Documentation, https://nilkanthsuthar.github.io/toolproof/
Project-URL: Source, https://github.com/NilkanthSuthar/toolproof
Project-URL: Changelog, https://github.com/NilkanthSuthar/toolproof/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/NilkanthSuthar/toolproof/issues
Author: Nilkanth Suthar
Maintainer: Nilkanth Suthar
License-Expression: MIT
License-File: LICENSE
Keywords: benchmark,ci,evals,fuzzing,hypothesis,junit,mcp,model-context-protocol,pytest,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Framework :: Hypothesis
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: anyio>=4
Requires-Dist: httpx2>=2.10
Requires-Dist: hypothesis-jsonschema>=0.23
Requires-Dist: hypothesis>=6.100
Requires-Dist: jsonpath-ng>=1.6
Requires-Dist: jsonschema>=4.21
Requires-Dist: mcp<3,>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6
Requires-Dist: rich>=13
Requires-Dist: typer>=0.12
Provides-Extra: anthropic
Requires-Dist: anthropic>=1.0; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: anthropic>=1.0; extra == 'dev'
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: coverage[toml]>=7.5; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: openai>=1.40; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff<0.17,>=0.16; extra == 'dev'
Requires-Dist: twine>=5; extra == 'dev'
Requires-Dist: types-jsonschema; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocs<2,>=1.6; extra == 'docs'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Description-Content-Type: text/markdown

# toolproof

[![PyPI](https://img.shields.io/pypi/v/mcp-toolproof)](https://pypi.org/project/mcp-toolproof/)
[![Python](https://img.shields.io/pypi/pyversions/mcp-toolproof)](https://pypi.org/project/mcp-toolproof/)
[![CI](https://github.com/NilkanthSuthar/toolproof/actions/workflows/ci.yml/badge.svg)](https://github.com/NilkanthSuthar/toolproof/actions/workflows/ci.yml)
[![Docs](https://img.shields.io/badge/docs-online-blue)](https://nilkanthsuthar.github.io/toolproof/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/NilkanthSuthar/toolproof/blob/main/LICENSE)

**Automated tests for [Model Context Protocol](https://modelcontextprotocol.io) servers.** Check tool definitions, write YAML or pytest tests, fuzz every tool from its schema, benchmark latency, check that models pick the right tools, and fail the build when something breaks.

```bash
pip install mcp-toolproof
toolproof fuzz -- python my_server.py
```

**[Documentation](https://nilkanthsuthar.github.io/toolproof/)** · [Getting started](https://nilkanthsuthar.github.io/toolproof/getting-started/) · [Configuration](https://nilkanthsuthar.github.io/toolproof/guide/configuration/) · [CLI](https://nilkanthsuthar.github.io/toolproof/reference/cli/) · [Changelog](https://github.com/NilkanthSuthar/toolproof/blob/main/CHANGELOG.md)

## Features

- **Static checks**: every tool needs a name, a description, a valid JSON Schema and required fields that exist
- **YAML tests**: assert on errors, text, regex, JSONPath values, latency and the tool's own `outputSchema`
- **Fuzzing**: valid and invalid inputs generated from each tool's schema with Hypothesis; crashes, hangs, internal errors and leaked tracebacks, each shrunk to the smallest failing input
- **Benchmarks**: p50/p95/p99 latency, throughput and error rate under concurrency, with thresholds
- **Tool-selection evals**: give a model your tool list and check it picks the right tool and arguments for real requests; Anthropic, OpenAI-compatible or your own program, with cached answers
- **pytest plugin**: an `mcp_server` fixture for tests written in Python
- **Built for CI**: a [GitHub Action](https://nilkanthsuthar.github.io/toolproof/guide/github-action/), exit codes, JUnit, JSON and single-file HTML reports, a job summary, repeatable fuzz seeds, `${VAR}` secrets
- **Any server**: stdio or streamable HTTP, written in Python, TypeScript, Go or anything else; crashed servers are restarted and their stderr is shown

## Quick start

See what a server offers and whether its tool definitions are sound:

```bash
toolproof inspect -- python my_server.py
toolproof inspect --url http://localhost:8000/mcp
```

Write tests in `toolproof.yaml`:

```yaml
server:
  command: ["python", "my_server.py"]

tests:
  - name: toronto weather
    tool: get_weather
    args: { city: Toronto }
    expect:
      is_error: false
      jsonpath: { "$.temp_c": { type: number, min: -60, max: 60 } }
      max_latency_ms: 2000

  - name: rejects empty city
    tool: get_weather
    args: { city: "" }
    expect: { is_error: true }

bench:
  thresholds: { p95_ms: 200 }

fuzz:
  seed: 1
```

```bash
toolproof run toolproof.yaml --junit report.xml
```

Check that a model picks the right tool (`pip install "mcp-toolproof[anthropic]"` or `[openai]`):

```yaml
evals:
  provider: anthropic
  model: your-model-name
  cases:
    - prompt: Is it raining in Montreal right now?
      expect_tool: get_weather
      expect_args: { city: Montreal }
```

```bash
toolproof eval -c toolproof.yaml
```

In GitHub Actions:

```yaml
- uses: actions/setup-python@v7
  with:
    python-version: "3.12"
- run: pip install -e .              # your server and its dependencies
- uses: NilkanthSuthar/toolproof@v0.3.1
  with:
    config: toolproof.yaml
    python-version: ""               # run in the environment set up above
```

Or in Python, with the pytest plugin:

```python
def test_weather(mcp_server):
    r = mcp_server.call("get_weather", city="Toronto")
    assert not r.is_error
    assert r.data["temp_c"] > -60
```

## Fuzzing finds bugs without writing tests

[`examples/buggy_server.py`](https://github.com/NilkanthSuthar/toolproof/blob/main/examples/buggy_server.py) has five planted bugs. `toolproof fuzz` finds all of them on its own:

```
$ toolproof fuzz --seed 3 -- python examples/buggy_server.py

Fuzz (seed 3)
┌──────┬─────────────────┬───────┬────────────────────────────┬─────────────────┬─────────────────────────────────────────────┐
│      │ Tool            │ Calls │ Finding                    │ Smallest input  │ Details                                     │
├──────┼─────────────────┼───────┼────────────────────────────┼─────────────────┼─────────────────────────────────────────────┤
│ FAIL │ get_weather     │    21 │ crash (valid)              │ {"city": ""}    │ connection lost, server probably crashed    │
│      │                 │       │                            │                 │ (Connection closed)                         │
│      │                 │       │ accepted-invalid (invalid) │ {"city": []}    │ invalid input returned a success result     │
│      │                 │       │ crash (invalid)            │ {}              │ connection lost, server probably crashed    │
│      │                 │       │                            │                 │ (Connection closed)                         │
│ FAIL │ search_cities   │     0 │ bad-schema (valid)         │ {}              │ can't generate inputs: 'strng' is not valid │
│      │                 │       │                            │                 │ under any of the given schemas              │
│ FAIL │ slow_report     │     1 │ hang (valid)               │ {"city": ""}    │ timed out after 1.0s                        │
│ FAIL │ get_temperature │    24 │ output-schema (valid)      │ {"city": ""}    │ output schema: $.temp_c: '12.5' is not of   │
│      │                 │       │                            │                 │ type 'number'                               │
│      │                 │       │ accepted-invalid (invalid) │ {}              │ invalid input returned a success result     │
│ FAIL │ lookup          │    15 │ internal-error (valid)     │ {"city_id": {}} │ JSON-RPC error -32603: Internal server      │
│      │                 │       │                            │                 │ error                                       │
│      │                 │       │ internal-error (invalid)   │ {}              │ JSON-RPC error -32603: Internal server      │
│      │                 │       │                            │                 │ error                                       │
└──────┴─────────────────┴───────┴────────────────────────────┴─────────────────┴─────────────────────────────────────────────┘
Repeat this run with --seed 3

FAILED  |  fuzz: 9 findings in 5 tools  |  24.33s
```

> **Warning:** fuzzing calls your tools for real, many times, with odd arguments. Point it at a test instance, and skip tools that send, delete or spend. See the [fuzzing guide](https://nilkanthsuthar.github.io/toolproof/guide/fuzzing/).

## Tested on

toolproof is run against the official [MCP reference servers](https://github.com/modelcontextprotocol/servers): everything, filesystem, git, memory, sequential-thinking, time and fetch. In October 2026 that was about 3,000 fuzz calls, 15 [security boundary tests](https://nilkanthsuthar.github.io/toolproof/recipes/security-boundaries/) and benchmarks. All of them held up; the reference servers validate their input well.

## How it compares

[MCP Inspector](https://github.com/modelcontextprotocol/inspector) is for exploring a server by hand; toolproof is for tests that run unattended in CI. Other projects cover declarative YAML tests, snapshot regression gates, agent evaluations and security scanning. toolproof's focus is behaviour testing with schema-driven fuzzing and benchmarks, offline and without an LLM; only the optional evals use a model. See the [comparison](https://nilkanthsuthar.github.io/toolproof/about/comparison/) for details.

## Requirements

- Python 3.11 to 3.14 on Linux, macOS or Windows
- The server can be written in any language

## Contributing

Bug reports, fixes and new checks are welcome. See [CONTRIBUTING.md](https://github.com/NilkanthSuthar/toolproof/blob/main/CONTRIBUTING.md) for the development setup, and [SECURITY.md](https://github.com/NilkanthSuthar/toolproof/blob/main/SECURITY.md) to report a security problem privately.

## License

[MIT](https://github.com/NilkanthSuthar/toolproof/blob/main/LICENSE)
