Metadata-Version: 2.4
Name: mcp-gauntlet
Version: 0.3.0
Summary: An agentic evaluation harness for MCP servers — can an AI agent actually accomplish real tasks with this server's tools?
Project-URL: Homepage, https://github.com/GhalebDweikat/mcp-gauntlet
Project-URL: Repository, https://github.com/GhalebDweikat/mcp-gauntlet
Project-URL: Issues, https://github.com/GhalebDweikat/mcp-gauntlet/issues
Project-URL: Leaderboard, https://ghalebdweikat.github.io/mcp-gauntlet/
Author: Ghaleb Dweikat
License: MIT
License-File: LICENSE
Keywords: agents,evals,evaluation,gemini,groq,llm,mcp,mcp-server,model-context-protocol,prompt-injection,tool-use
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: anyio>=4.4
Requires-Dist: jsonschema>=4.20
Requires-Dist: mcp>=1.9
Requires-Dist: openai>=1.40
Requires-Dist: pydantic>=2.7
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# mcp-gauntlet

[![CI](https://github.com/GhalebDweikat/mcp-gauntlet/actions/workflows/ci.yml/badge.svg)](https://github.com/GhalebDweikat/mcp-gauntlet/actions/workflows/ci.yml)

**An agentic evaluation harness for MCP servers.** Point it at any
[Model Context Protocol](https://modelcontextprotocol.io) server and it answers
the question the static analyzers don't: **can an AI agent actually accomplish
real tasks using this server's tools?**

## Why

The existing MCP quality tools (`mcp-lighthouse`, `mcp-scorecard`, `mcp-checkup`)
are all **static** — they inspect schemas, count tokens, and lint descriptions.
None of them run an LLM agent against the server to see whether it can actually
complete tasks. That dynamic, agent-in-the-loop evaluation — with a real
**task-success rate**, not just a conformance check — is what mcp-gauntlet does.
The same live run also scans the tools' actual **outputs** for prompt-injection,
catching tool-poisoning that a static description scan can't (a server that looks
clean at list-time but poisons at call-time).

> Google Lighthouse tells you your web page is *well-formed*.
> mcp-gauntlet tells you your MCP server is *usable by an agent* — with a
> task-success rate to prove it.

## What it scores

Each run produces a graded report card (JSON + Markdown) across:

- **Schema Health** — valid JSON schemas, typed and described parameters.
- **Description Quality** — can an agent tell when and how to use each tool?
- **Security Signals** — a *static* scan for tool-poisoning / prompt-injection markers
  and hidden characters, covering the server's init instructions, its tool descriptions,
  and **every string in each tool's input schema** — titles, enums, defaults, examples,
  `$defs` entries and unknown extension keywords included. The whole schema is serialized
  into the model's prompt, so any string in it can carry a payload; scanning only
  `description` fields at the top level is trivially evaded by nesting one behind a
  `$ref`. A critical finding caps the overall grade.
- **Agent Task Success** — a live LLM agent attempts generated tasks using only
  the server's tools; LLM-judged and repeated for a success rate.
- **Tool-Selection Accuracy** — did the agent call the tools it was expected to?
- **Tool Reliability** — did the server's tools execute without error?
- **Response Safety** — a *dynamic* scan of the tools' live **outputs** for the
  same injection / poisoning markers, catching a server that looks clean at
  list-time but poisons at call-time. Reported (and it lowers the score) but
  doesn't cap on its own, since a fetch/filesystem server may faithfully pass
  through untrusted content.
- **Robustness** — does the server reject malformed input gracefully? A tool that
  publishes no argument schema at all scores zero here rather than being skipped:
  a server that declares no contract can't reject anything, and skipping it would
  make omitting schemas a way to score higher.

## Leaderboard

A live leaderboard ranks popular public MCP servers by their gauntlet score:
**[ghalebdweikat.github.io/mcp-gauntlet](https://ghalebdweikat.github.io/mcp-gauntlet/)**

Generate one yourself across any set of servers listed in a JSON file:

```bash
uv run mcp-gauntlet leaderboard --servers leaderboard.servers.json --out docs
```

Each server's raw result is saved to `servers/<name>.json` alongside its page, so the
site can be rebuilt for free after a presentation change — no re-running (or re-paying
for) the evaluation:

```bash
uv run mcp-gauntlet leaderboard --render-only --out docs
```

Only servers the agent actually scored share the ranked table. The overall is a weighted
mean over the dimensions *present*, so a server the agent never ran against — no LLM
configured, the backend rate-limited it, or every tool was excluded as possibly-mutating —
skips Agent Task Success (the heaviest dimension) and would score systematically higher on
a smaller denominator. Those are listed separately under **Partially evaluated**, each with
the reason it wasn't ranked, rather than mixed in where an untested server could outrank a
tested one.

## Quickstart

```bash
uv sync --extra dev

# Static + robustness checks only — no API key required
uv run mcp-gauntlet run "python -m mcp_gauntlet.fixtures.good_server" --no-agentic

# Full gauntlet, including the live agent (Groq's free tier works)
echo "GROQ_API_KEY=gsk_..." > .env
uv run mcp-gauntlet run "npx -y @modelcontextprotocol/server-everything"
```

The LLM backend is provider-agnostic — any OpenAI-compatible endpoint (Groq by
default; also OpenRouter, Together, or a local Ollama / vLLM). Runs are read-only by
default: tools that *look* mutating (by name/description or a self-declared MCP
`destructiveHint`) are excluded unless you pass `--allow-writes`. That exclusion is a
best-effort heuristic, not a guarantee — pair it with read-only credentials or a
throwaway environment for untrusted servers. Generated task sets are cached so scores
are reproducible across runs.

Bundled `good` / `bad` fixture servers make it easy to see the difference:

```bash
uv run mcp-gauntlet run "python -m mcp_gauntlet.fixtures.bad_server"   # capped C — tool poisoning
uv run mcp-gauntlet run "python -m mcp_gauntlet.fixtures.good_server"  # A
```

With an LLM key, the bad fixture also trips **Response Safety**: its `status_report`
tool has a clean description but poisons its *output*, so only the runtime scan
catches it — the static description scan can't.

A server that hangs can't stall the run: every tool call is bounded by
`--tool-timeout` (default 60s) and recorded as a failed call against the server's Tool
Reliability, and `--timeout` (default 900s, `0` disables) caps the evaluation as a whole
so a server that hangs during connect or `tools/list` still can't wedge the CLI. Raise
`--tool-timeout` if a server is legitimately slow rather than stuck — the report says so
when the limit is what stopped it.

## Configuration

Configure via a `.env` file (copy [`.env.example`](.env.example) and fill it in)
or real environment variables:

| Variable | Purpose |
|----------|---------|
| `GROQ_API_KEY` / `GEMINI_API_KEY` / `OPENAI_API_KEY` / `OPENROUTER_API_KEY` | API key for the provider the agent should use (only one needed). A free Groq key: [console.groq.com/keys](https://console.groq.com/keys). |
| `MCP_GAUNTLET_PROVIDER` | Which provider: `groq` (default), `gemini`, `openai`, `openrouter`, or `ollama` (local). |
| `MCP_GAUNTLET_MODEL` | Model override for that provider (e.g. `gemini-flash-latest`). Defaults to a sensible per-provider model. |

The `--provider` / `--model` CLI flags override these, and `--base-url` points at
any OpenAI-compatible endpoint — a local Ollama / vLLM / LM Studio or a gateway:

```bash
uv run mcp-gauntlet run "npx -y @scope/pkg" --base-url http://localhost:11434/v1 --model llama3.1
```

### No API key? Static mode

Everything except the live agent runs without an LLM. `mcp-gauntlet run <server>`
with no key configured reports a **static grade** from the LLM-free checks —
schema health, description quality, security signals, and robustness probes:

```bash
uv run mcp-gauntlet run "npx -y @modelcontextprotocol/server-everything" --no-agentic
```

Add `--no-probe` for a pure inspection that never executes any of the server's
tools. The leaderboard behaves the same way — with no key it ranks servers on the
static + robustness checks alone.

## Use it in CI

Gate your MCP server's pull requests on its gauntlet score. Copy
[`examples/gauntlet-ci.yml`](examples/gauntlet-ci.yml) into your server's repo as
`.github/workflows/gauntlet.yml`, point it at your server, and the build fails
when the score drops below your threshold:

```yaml
- name: Run the gauntlet
  run: uvx mcp-gauntlet@0.3.0 run "python -m your_server" --no-agentic --fail-under 60
```

Pin the version, as above: an unpinned `uvx mcp-gauntlet` would let a new release move
your gate without a commit.

The static + robustness checks need no API key. To include the live agent
evaluation, add an LLM key (e.g. `GROQ_API_KEY`) as a repository secret and drop
`--no-agentic`. The report is uploaded as a build artifact.

## License

MIT © Ghaleb Dweikat
