Metadata-Version: 2.5
Name: mcpgantlet
Version: 0.5.0
Summary: MCP servers ship to production with no way to know whether they survive concurrent load or are safe.
Author: Raghuram P
License: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: fastapi<0.142,>=0.121
Requires-Dist: httpx>=0.27
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Requires-Dist: uvicorn>=0.32; extra == 'dev'
Description-Content-Type: text/markdown

# mcpgantlet

**Conformance and load checks for MCP servers on spec `2026-07-28`.**

<!-- SCREENCAST: Act 1 — curl the audit against a real public server; the Origin probe fails.
     Act 2 — the same probe against Cloudflare, which returns 403; the control that proves the
     tool works. Act 3 — the revision check, showing none of the five implements the spec being
     audited, and the version-gap/real-defect split that keeps the report honest. -->

## Install

```bash
pipx install mcpgantlet
mcpgantlet audit http://localhost:8000/mcp
mcpgantlet rules            # every rule with the spec clause it cites
```

Auditing a host you do not operate requires `--i-have-permission`. Every probe is read-only, but
they are still unsolicited requests to someone else's server.

Exit codes: `0` conformant · `1` real defect found · `2` refused to probe.

A **version gap** — a rule that only exists in the audited revision, against a server targeting an
older one — is reported but never exits non-zero. No public server implements 2026-07-28 yet, so
failing on that would make this useless in CI against the servers that exist.

## Four of five public MCP servers accept requests from any website

Identical request, sent twice — once with no `Origin`, once with `Origin: https://attacker.example`:

| Server | Operator | No Origin | Hostile Origin | |
|---|---|---|---|---|
| learn.microsoft.com/api/mcp | Microsoft | 200 | **200** | accepts |
| knowledge-mcp.global.api.aws | AWS | 200 | **200** | accepts |
| mcp.deepwiki.com/mcp | Cognition | 200 | **200** | accepts |
| gitmcp.io/docs | idosal | 200 | **200** | accepts |
| **docs.mcp.cloudflare.com/mcp** | **Cloudflare** | 200 | **403** | **rejects** |

Cloudflare is the control: same probe, same request shape, correct rejection — so the four
acceptances are a property of those servers, not of the tool.

`Origin` validation is the spec's defence against DNS rebinding — an attack that bites hardest on
servers reachable on localhost or a private network, where the server trusts its network position.
It has been a MUST since the Streamable HTTP transport was introduced, so this is **not** a version
gap.

**What this is and is not.** Four servers do not enforce a MUST the spec places on them, and a
client cannot distinguish a conformant server from a non-conformant one without checking. That is
the result. It is *not* a claim that each deployment is exploitable — three are public and
unauthenticated, where a browser adds little an attacker could not do from their own server.

**And the finding that came from asking before measuring: none of the five implements the current
revision.** All target 2025-03-26 or 2025-06-18. A naive audit would have reported ~40 MUST
violations — 26 of them version gaps — which is the "97% flagged at under 50% precision" noise
this tool exists not to add to. Findings are classified so the two never get conflated.

Full method, disclosure position and the probe bug that got the right answer for the wrong reason:
[`bench/conformance/results/2026-08-21-third-party.md`](bench/conformance/results/2026-08-21-third-party.md)

**Live:** https://mcpgantlet.vercel.app — a strictly conformant MCP 2026-07-28 server to
test your client against, and the zero-false-positive claim as one request:

```bash
curl https://mcpgantlet.vercel.app/audit/self
# 10 probes, 0 violations, 0 inconclusive — CONFORMANT
```

There is deliberately no hosted auditor for arbitrary URLs: `docs/NON-GOALS.md` ruled that out
before any code existed, and an auditor pointed at whatever a stranger types is a
request-forgery gadget whether or not its probes are read-only.

The first server it audited was my own. It failed **6 of 8 MUST requirements**.

## Why this exists

Revision `2026-07-28` changed the shape of MCP's HTTP transport. It removed protocol-level
sessions, the GET stream endpoint, server-initiated requests and `Last-Event-ID`
resumability — and added mandatory request-metadata headers with server-side header/body
validation.

A server built to the previous shape does not merely look dated. It violates several MUSTs
of the current revision, and a client that mis-detects the era silently negotiates down.

I had read the research on this exact revision, written a server, reviewed it, and deployed
it to production. Then I pointed this tool at it:

| Rule | Severity | Observed | Verdict |
|---|---|---|---|
| `origin-403` | MUST | 403 | pass |
| `get-405` | MUST | 200 | **fail** |
| `delete-405` | MUST | 405 | pass |
| `protocol-version-header` | MUST | 200 | **fail** |
| `header-body-match` | MUST | 200 | **fail** |
| `unknown-method-404` | MUST | 200 | **fail** |
| `no-initialize` | MUST | 200, claims 2026-07-28 | **fail** |
| `notification-202` | MUST | 200 | **fail** |

The instructive one is `no-initialize`. The server answered the `initialize` handshake and
returned `protocolVersion: "2026-07-28"` — a revision that **removed** that handshake. A
client doing era detection would have seen a successful `initialize`, concluded the server
spoke a pre-2026 revision, and negotiated down. **Nothing would have errored.**

Now 10 pass, 0 fail, verified against the production deployment.
Full writeup: [`bench/conformance/results/first-finding.md`](bench/conformance/results/first-finding.md)

## Quickstart

```bash
uv venv && source .venv/bin/activate
uv pip install -e ".[dev]" fastapi uvicorn

python bench/conformance/audit.py https://your-server.example/mcp
```

No credentials. Ten probes, each citing the spec clause it enforces — so you can check the
rule rather than trust my reading of it.

## What it checks

**Conformance** — 11 rules, 8 MUST and 3 SHOULD, every one traceable to a clause:
`Origin` → 403 · GET/DELETE → 405 · `MCP-Protocol-Version` required · header/body agreement
→ 400 with `-32020` · unknown method → 404 with `-32601` · `initialize` not answered ·
notification → 202 · content type · session-id ignored.

**Load** — a concurrency ramp reporting p50/p95/p99, throughput and error onset, looking for
where a server stops degrading gracefully.

## Load results are attributed, not just measured

This is the part I got wrong first, and the fix is the interesting bit.

The initial ramp against a reference server reported **p99 degrading 769×** and throughput
collapsing from 2,024 to 284 rps with zero errors. Textbook cliff.

Then I ran a control with **4× the server workers**:

| Concurrency | 1 worker | 4 workers | Change |
|---|---|---|---|
| 1 | 1,096 rps | 1,424 rps | +30% |
| 10 | 945 rps | 956 rps | **+1%** |
| 50 | 346 rps | 351 rps | **+1%** |

If the server were the constraint, quadrupling its workers would move the cliff. It moved it
by one percent. **The bottleneck was my own client**, with both processes on one machine.

So the profiler now calibrates its own ceiling against a trivial in-process endpoint before
profiling anything, and reports a step near that ceiling as `harness_limited` rather than as
a server finding:

```
concurrency 50  harness_limited
  throughput 349 rps is at this harness's own ceiling of 367 rps. Degradation at or
  above this concurrency cannot be attributed to the server — move the client to a
  separate host to measure it.
```

A false "your server has a cliff" is worse than a missed one: it sends someone optimising a
server that was fine, and the first false positive destroys trust in every later report.

Full writeup: [`bench/load/results/harness-limit-2026-08-21.md`](bench/load/results/harness-limit-2026-08-21.md)

## Status against pre-registered criteria

| # | Criterion | Status |
|---|---|---|
| 1 | Every rule cites a spec clause | **met** — asserted in tests |
| 2 | Real defects in ≥3 real servers | **partial** — 1 server (mine), 6 MUST violations |
| 3 | Zero false positives on a conformant server | **met** — 0 failures against the reference |
| 4 | Load profiling finds a non-obvious cliff | **NOT met** — the cliff found was the harness's |
| 5 | Inconclusive reported, never guessed | **met** |
| 6 | Probes cannot damage a target | **met** — read-only or rejected requests only |

Criterion 4 is open and stated as open. A p99 degradation was measured but cannot be
attributed to the server, so claiming it would mean shipping a load tool whose headline
finding is an artefact of the tool.

## It will not load-test a host you do not control

```
$ python bench/load/profile.py https://someone-elses-server.example/mcp
refusing to load-test someone-elses-server.example: pass --i-have-permission if you control it.
Load testing an endpoint you do not control is abuse regardless of intent.
```

Conformance probes are read-only or deliberately-rejected requests and are safe against any
endpoint. Load generation is not, and the tool should not make it frictionless.

## Limitations

- **Load results from a single machine are harness-limited above ~10 concurrency.** Real
  profiling needs the client on a separate host. The tool now says so instead of pretending
  otherwise.
- **Only one server has been audited** — mine. Criterion 2 needs three, and third-party
  audits are conformance-only (read-only probes, no load).
- **`error_onset` is currently suppressed at a harness-limited concurrency.** Errors are the
  server's regardless of who the throughput bottleneck is, so this should be split. Asserted
  in a test so the gap is visible rather than silently wrong.
- **Conformance covers the transport, not tool behaviour.** Whether a tool returns correct
  results is the server author's domain.
- **Only revision `2026-07-28`.** A checker that accepts every revision cannot tell you
  which one you implement, which is the question worth answering.
- **The 11 rules are not the whole spec.** They are the ones mechanically checkable from
  outside with a handful of requests.

## Licence

MIT — see [LICENSE](LICENSE).
