Metadata-Version: 2.5
Name: mcp-crashtest
Version: 0.2.1
Summary: MCP Multi-step Deadlock & Vulnerability Crash Tester via State-Aware Random Walks with Rollback
Project-URL: Homepage, https://github.com/SnowyLumiere/mcp-crashtest
Project-URL: Repository, https://github.com/SnowyLumiere/mcp-crashtest
Project-URL: Issues, https://github.com/SnowyLumiere/mcp-crashtest/issues
Author: SnowyLumiere
License: MIT License
        
        Copyright (c) 2026 SnowyLumiere
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agents,deadlock,fuzzing,mcp,model-context-protocol,random-walk,testing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: mcp>=1.1.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# MCP-CrashTest

**Unit tests check single tool calls. Real agents fail on multi-step contracts.**

*State-aware random walks with POSIX inode-preserving rollback for Model Context Protocol (MCP) servers.*

[![Version](https://img.shields.io/badge/version-0.2.1-blue.svg)](pyproject.toml)
[![Python](https://img.shields.io/badge/python-%3E%3D3.10-brightgreen.svg)](pyproject.toml)
[![CI](https://github.com/SnowyLumiere/mcp-crashtest/actions/workflows/ci.yml/badge.svg)](https://github.com/SnowyLumiere/mcp-crashtest/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

[中文文档 (README_CN.md)](README_CN.md)

---

## Why MCP-CrashTest?

Tools like MCP Inspector and standard unit tests validate **single-step static calls**. In production, AI agents execute **5–20 step tool chains** where outputs from discovery/list tools feed downstream mutations, entities collide, and `stdio` pipes hang.

Inspired by reversing the state-aware random walk paradigm from **ToolSpace** ([arXiv:2611.05036](https://arxiv.org/abs/2611.05036))—instead of keeping successful trajectories for model training and discarding rollbacks—`MCP-CrashTest` mines a dependency graph from `tools/list`, runs guided multi-step random walks inside ephemeral OS sandboxes, rolls back on failure to explore neighboring branches, prunes noisy intermediate steps, and outputs deduplicated **minimal JSON-RPC reproducers**.

### What It Catches in Real MCP Servers

1. **Masked Errors (`isError: false` on SQL/Runtime Failures) — Protocol Violation (`CRITICAL`)**  
   In `mcp-server-sqlite` (verified on **2026-11-20** with `MCP-CrashTest v0.2.1` in **both** `--adapter none` pure zero-shot mode and `--adapter auto`), SQL errors return `isError: false`, causing agents to treat error strings as valid query results (context poisoning):
   ```json
   {
     "method": "tools/call",
     "params": {
       "name": "write_query",
       "arguments": {
         "query": "INSERT INTO users (id, name, email) VALUES (1, 'CollisionBob', 'bob_new@example.com');"
       }
     },
     "response": {
       "text": "Database error: UNIQUE constraint failed: users.id",
       "isError": false
     }
   }
   ```

2. **CLI Output Leaks Across Tools — Cross-Tool Contract Break (`HIGH`)**  
   In `mcp-server-git`, `git_branch` outputs human CLI formatting (`"* main"`). Passing that branch name directly to `git_checkout`, `git_diff`, `git_show`, or `git_merge` fails:
   ```json
   [
     { "tool": "git_branch", "response": { "text": "* main", "isError": false } },
     { "tool": "git_merge", "arguments": { "source_branch": "* main" }, "response": { "text": "Error executing tool git_merge: Ref '* main' did not resolve to an object", "isError": true } }
   ]
   ```

3. **Silent State Corruption & Stdio Deadlocks — Protocol Violations (`CRITICAL`)**  
   Read-after-write verification catches tools that report `isError: false` without persisting state, as well as hung `stdio` buffers or severed child-process pipes (with automatic subprocess recycling).

4. **Agent Contract Smells — UX & Consistency Pitfalls (`MEDIUM` / `LOW` / `INFO`)**  
   Explicitly separated from protocol bugs: undocumented glob-only `search_files` matching, relative vs. parent-joined path contracts, no-upsert skip behavior (`create_entities`), and intra-server `isError` inconsistencies.

---

## Quick Start

Requires **Python >= 3.10**.

### 1. Run Immediately via `uvx` or `pip`

```bash
# Run directly without cloning (once published on PyPI)
uvx mcp-crashtest --target sqlite --walks 15 --depth 6

# Or install from source
git clone https://github.com/SnowyLumiere/mcp-crashtest.git
cd mcp-crashtest
pip install -e ".[dev]"
```

### 2. Test Any Custom MCP Server (Zero-Shot Generic Mode)

Use `--` to pass the server command cleanly without option collisions, and `--adapter none` to run **pure zero-shot generic stateful fuzzing** with zero domain-specific assumptions:

```bash
mcp-crashtest \
  --target my-server \
  --adapter none \
  --sandbox directory \
  --seed 42 \
  --walks 20 \
  --depth 6 \
  --fail-on-crash \
  --output reports/my_server_report.md \
  -- python3 -m my_mcp_server --root {sandbox}
```

For stateless/API-only servers, use `--sandbox none` (context-memory rollback and branch exploration still run):

```bash
mcp-crashtest \
  --target my-api-server \
  --adapter none \
  --sandbox none \
  --walks 20 \
  -- node dist/index.js --env test
```

### 3. Built-in Official Server Presets

Requires `npx` (`filesystem` / `memory`) or `uvx` + `git` (`sqlite` / `git`):

```bash
mcp-crashtest --target sqlite --walks 15 --depth 6 --output reports/sqlite_crash_report.md
mcp-crashtest --target git --walks 15 --depth 6 --output reports/git_crash_report.md
mcp-crashtest --target filesystem --walks 15 --depth 6 --output reports/filesystem_crash_report.md
mcp-crashtest --target memory --walks 15 --depth 6 --output reports/memory_crash_report.md
```

---

## Architecture: Generic Engine vs. Domain Adapters

`MCP-CrashTest` separates the **generic zero-shot stateful engine** from optional **domain contract adapters** (`mcp_crashtest/adapters/`):

| Capability | Generic Mode (`--adapter none`) | Built-in Domain Adapters (`--adapter auto`) |
| :--- | :--- | :--- |
| **Schema Graph Mining** | Infers `CREATE`/`READ`/`MUTATE`/`DELETE` + `DATA`/`STATE`/`RETRY` edges from `tools/list`; handles Pydantic v2 `anyOf`/`oneOf`/`enum` and `properties: null` | Adds domain-specific edge bindings for `filesystem`, `sqlite`, `memory`, `git` |
| **Context Flow & Fuzzing** | Extracts IDs/keys/URIs/paths from args & JSON responses; injects boundary strings, traversal paths, extreme ints | Adds domain-specific probes (e.g. PK collisions, glob-vs-substring queries, raw `* main` branch passing) |
| **Post-Condition Probes** | Probe A (`RETRY_TRAP`), Generic Probe B (Read-After-Write on `id`/`key`/`uri`), Probe C (`SCHEMA_BREAK`), Probe D (`STDIO_DEADLOCK`) | Adds calibrated `CONTRACT_SMELL` checks (e.g. `search_files` glob pitfall, `describe_table` `[]`, `create_entities` no-upsert skip) |
| **Report Provenance** | Every finding is tagged with `source: generic` | Every adapter finding is explicitly tagged `source: adapter:<name>` |

### Sandbox Placeholders & Environment Injection

| `--sandbox` | CLI Placeholders | Injected Env Var | Seeded Initial Data |
| :--- | :--- | :--- | :--- |
| `directory` *(default)* | `{sandbox}`, `{work_dir}` | `MCP_SANDBOX_DIR` | `sample.txt`, `data_dir/config.json`, `data_dir/nested_note.txt` |
| `git` | `{sandbox}`, `{work_dir}` | `MCP_SANDBOX_DIR` | Initialized `main` branch + initial commit (`commit.gpgsign=false`) |
| `sqlite` | `{sandbox}`, `{db_path}` | `MCP_SANDBOX_DB_PATH` | `test_target.db` with seeded `users` table (2 rows) |
| `memory` | `{sandbox}`, `{memory_path}` | `MEMORY_FILE_PATH` | Isolated temp path `memory_graph.json` |
| `none` | — | — | No local filesystem sandbox (context rollback still active) |

---

## Live Audit Summary (v0.2.1 · 4 Official MCP Servers)

Verified on **2026-11-20** (`--walks 15 --depth 6 --rollbacks 3 --seed 42`, Python 3.12, `mcp` SDK v2.3.0 client with full v1.x/v2.x compatibility; pruned reports in [reports/](reports/)):

| Server | Mode | Protocol Violations | Contract Smells | Headline Minimal Reproducers | Report |
| :--- | :--- | :---: | :---: | :--- | :---: |
| **`mcp-server-sqlite`** | `--adapter none` *(pure zero-shot)* | **`3`** (`CRITICAL`) | `0` | `write_query`, `create_table`, `read_query` return `"Error: ..."` with `isError: false` (1-step each, `source: generic`) | — |
| **`mcp-server-sqlite`** | `--adapter auto` | **`3`** (`CRITICAL`) | `1` (`MEDIUM`) | SQL errors return `isError: false` (×3, 1-step); `describe_table` returns `[]` on missing table (1-step) | [sqlite](reports/sqlite_crash_report.md) |
| **`mcp-server-git`** | `--adapter auto` | `0` | **`4`** (`HIGH` `SCHEMA_BREAK`) | `git_branch` leaks `'* main'`, breaking `git_checkout`, `git_diff`, `git_show`, `git_merge` (2-step each) | [git](reports/git_crash_report.md) |
| **`@modelcontextprotocol/server-filesystem`** | `--adapter auto` | `0` | `4` (`MEDIUM`) | `list_directory` bare filenames vs `read_file`/`read_text_file`/`get_file_info` (2-step); `search_files` glob-only miss (1-step) | [filesystem](reports/filesystem_crash_report.md) |
| **`@modelcontextprotocol/server-memory`** | `--adapter auto` | `0` | `3` (`LOW` / `INFO`) | `create_entities` skips existing entities with `isError: false` (2-step); `delete_*` idempotent `0` differs from `add_*` (1-step) | [memory](reports/memory_crash_report.md) |

> **Notes**:
> - The `sqlite` preset launches `uvx --with "mcp<2" mcp-server-sqlite` because upstream `mcp-server-sqlite` currently uses the `mcp 1.x` `@server.list_resources()` API and fails to boot under `mcp 2.x`. `MCP-CrashTest` itself supports both `mcp` SDK v1.x (`inputSchema`/`isError`) and v2.x (`input_schema`/`is_error`).
> - [reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md](reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md) is preserved as the historical v0.2.0 archive.

---

## 4 Contract Rules for Agent-Safe MCP Servers

1. **Strict `isError` Semantics**: Never return `isError: false` on SQL errors, constraint violations, or failed mutations.
2. **Cross-Tool I/O Symmetry**: Discovery/list tool outputs must be directly consumable by downstream tools without CLI decorations (`* main`) or stripped parent paths.
3. **Explicit Matching & Upsert Contracts**: Document glob vs. substring matching and provide explicit upsert/conflict errors instead of ambiguous `isError: false` skips.
4. **Actionable Errors & Pipe Backpressure**: Return offending parameter names and suggested corrective tools; paginate large responses to prevent `stdio` deadlocks.

---

## CLI Reference

| Flag | Default | Description |
| :--- | :---: | :--- |
| `--target` | `"custom"` | Target label or built-in preset (`filesystem`, `sqlite`, `memory`, `git`) |
| `--adapter` | `"auto"` | Domain adapter mode: `auto`, `none` (pure generic), or explicit adapter name(s) |
| `--cmd` / `-- <cmd...>` | `None` | Server executable (prefer POSIX `--` passthrough to avoid flag clashes) |
| `--sandbox` | `"directory"` | `directory` \| `sqlite` \| `git` \| `memory` \| `none` |
| `--walks` | `15` | Number of random walk chains |
| `--depth` | `6` | Max step depth per walk |
| `--rollbacks` | `3` | Max rollback retries per walk |
| `--fuzz-ratio` | `0.25` | Boundary fuzzing probability (`0.0`–`1.0`) |
| `--timeout` | `8.0` | Per-step & `tools/list` timeout in seconds |
| `--init-timeout` | `20.0` | Session `initialize()` timeout in seconds |
| `--seed` | `None` | Local deterministic RNG seed (does not mutate global `random` state) |
| `--lang` | `"en"` | Report language: `en` \| `zh` |
| `--fail-on-crash` | `False` | Exit with code `1` if findings matching `--fail-on` are detected (for CI gates) |
| `--fail-on` | `"any"` | CI failure scope: `any` (all findings) \| `protocol` (`PROTOCOL_VIOLATION` only) |
| `--output` | `"reports/mcp_crash_report.md"` | Output Markdown report path |
| `--version` | — | Print version and exit |

---

## Known Limitations & Roadmap

- **Transport**: Currently supports **`stdio`** transport only. **Streamable HTTP / SSE** support is on the roadmap.
- **SQLite Sandbox**: In-place binary file rollback assumes per-call SQLite connections (such as `mcp-server-sqlite`). Long-lived connection pools with in-memory page caches require session restarts between walks.
- **Process Memory**: For servers keeping unpersisted in-process state, sandbox rollback restores disk state only.

---

## Python API & Running Tests

```python
import asyncio
from mcp_crashtest.cli import run_crash_test

async def main():
    crashes = await run_crash_test(
        cmd="python3",
        args=["my_server.py", "--root", "{sandbox}"],
        target_name="my-server",
        adapter_mode="none",
        sandbox_type="directory",
        seed=42,
        lang="en",
        output_report_path="reports/my_server.md",
    )
    print(f"Hits: {len(crashes)} | Unique: {len({c.fingerprint for c in crashes})}")

asyncio.run(main())
```

Run the test suite (zero third-party dependencies required for `unittest`):

```bash
PYTHONPATH=. python3 -m unittest discover -s tests -v
# or with pytest
PYTHONPATH=. pytest tests/ -v
```

---

## License

Released under the [MIT License](LICENSE).
