Metadata-Version: 2.4
Name: mcpevalkit
Version: 0.1.0
Summary: Evaluation tooling for Model Context Protocol (MCP) servers
Keywords: mcp,model-context-protocol,evals,evaluation,llm
Author: Bruce P
Author-email: Bruce P <treble37@users.noreply.github.com>
License-Expression: Apache-2.0
Classifier: Development Status :: 1 - Planning
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Dist: mcp>=1.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: jsonschema>=4.20
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/treble37/mcp-evals
Project-URL: Repository, https://github.com/treble37/mcp-evals
Project-URL: Issues, https://github.com/treble37/mcp-evals/issues
Description-Content-Type: text/markdown

# mcpevalkit

> The eval kit for MCP-based agents. Record real tool-call trajectories, replay them deterministically against mocked servers, and score step-level — not just final answers.

[![PyPI version](https://img.shields.io/pypi/v/mcpevalkit.svg)](https://pypi.org/project/mcpevalkit/)
[![Python versions](https://img.shields.io/pypi/pyversions/mcpevalkit.svg)](https://pypi.org/project/mcpevalkit/)
[![License: Apache 2.0](https://img.shields.io/pypi/l/mcpevalkit.svg)](./LICENSE)
[![CI](https://github.com/treble37/mcp-evals/actions/workflows/ci.yml/badge.svg)](https://github.com/treble37/mcp-evals/actions/workflows/ci.yml)

```text
$ make demo
mcpevalkit demo — coding agent demo
====================================================
  PASS  write-greeting
  PASS  write-notes-with-content
  PASS  read-sample
  PASS  list-workspace
  PASS  confirm-write-response
  FAIL  expects-search-but-lists
        tool_called: 'search_files' was never called (calls: ['list_directory'])
  FAIL  expects-different-content
        tool_called_with: no call to 'write_file' matched args {'content': 'beta'}
----------------------------------------------------
  5/7 passed (71%)
  (2 deliberate failure demo(s) shown above)
```

> 🚧 **Status: v0.1 in active development.** The public API is unstable and may
> change without notice until the first tagged release. Not yet published to PyPI.

---

## Why mcpevalkit exists

If your agent speaks MCP, the tool trajectory is the behavior you need to test.

MCP-based agents fail differently. They fail by picking the wrong tool, by calling it with bad parameters, by retrying in a loop, by giving up too early, by recovering well from an error in step 3 but missing the implication in step 7. A final-answer score can't see any of that.

Most eval tooling can judge the final answer. `mcpevalkit` is narrower: it records and replays MCP tool-call trajectories so you can catch agent regressions in CI before they reach production.

**It is opinionated about three things:**

1. **MCP is a first-class object.** Not an adapter, not a plugin — the recorder speaks MCP natively over stdio and SSE.
2. **Evals should be deterministic and cheap.** Record real trajectories once. Replay them against mocked MCP servers forever. CI runs in seconds, not dollars.
3. **Trajectory-level scoring is the point.** Judges for tool selection, parameter validity, and retry loops — not just a single rubric over the final response.

---

## Who it's for

`mcpevalkit` is for teams building MCP-based agents who need to answer practical release questions:

- Did the new prompt, model, or tool schema change the agent's tool-use behavior?
- Did the agent call the right tool with the right arguments?
- Did it recover from tool errors, or did it spin in a retry loop?
- Can this behavior be tested in pytest without hitting live services on every CI run?

It is also a lightweight consulting artifact: clone it into a client MCP-agent repo, define the expected tool behavior in YAML or pytest, record a known-good trajectory, and turn agent reliability into a repeatable release check.

`mcpevalkit` is not a dashboard, hosted observability product, generic RAG benchmark, or framework adapter layer. The project intentionally stays small: MCP trajectory evals, deterministic replay, and test-friendly reporting.

---

## Installation

```bash
pip install mcpevalkit
```

Requires Python 3.11+ and an MCP-compatible agent. Works with any LLM provider — bring your own.

> **Status:** v0.1 is in active development and the API may shift. Pin to a specific version in production.

---

## Quickstart

Your agent is any `async` callable that takes a recorder and a prompt and drives
MCP tools through `recorder.session(...)`:

```python
# my_agent.py
from mcpevalkit.recorder import McpServerConfig, Recorder

# Servers for the `mcpevalkit record` CLI (the eval suite declares its own).
mcp_servers = {
    "filesystem": McpServerConfig(
        transport="stdio",
        command="npx",
        args=("-y", "@modelcontextprotocol/server-filesystem", "/tmp/sandbox"),
    )
}

async def agent(recorder: Recorder, prompt: str) -> str:
    await recorder.session("filesystem").call_tool(
        "write_file", {"path": "/tmp/sandbox/hello.txt", "content": "hello world"}
    )
    return "done"
```

Record a real run of your agent against an MCP server — note the `:agent`
attribute on the target:

```bash
mcpevalkit record \
  --agent ./my_agent.py:agent \
  --prompt "Write 'hello world' to /tmp/sandbox/hello.txt" \
  --output traces/baseline.jsonl
```

Define an eval suite:

```yaml
# .mcpeval.yaml
suite: coding-agent-baseline

mcp_servers:
  filesystem:
    transport: stdio
    command: npx
    args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp/sandbox"]

evals:
  - id: write-file-correctly
    prompt: "Write 'hello world' to /tmp/sandbox/hello.txt"
    expects:
      - tool_called: write_file
      - tool_called_with:
          tool: write_file
          args:
            content: "hello world"
      - final_state_matches:
          kind: file_equals
          path: /tmp/sandbox/hello.txt
          contains: "hello world"
      - no_retry_loop: true
```

Run it (the agent under test is supplied with `--agent`):

```bash
mcpevalkit run --suite .mcpeval.yaml --agent ./my_agent.py:agent
```

### In pytest

**Zero-config:** drop a `.mcpeval.yaml` next to an `agent.py` that exposes an
`agent` callable, then run pytest with the plugin flag. Each eval becomes its
own test, and failures print the trajectory step by step:

```bash
pytest --mcpevalkit
```

**Fixtures:** for hand-written tests, `mcp_agent_run` runs an agent and returns
the recorded [`Trace`](src/mcpevalkit/trace.py) to assert on:

```python
# test_agent.py
from my_agent import agent, mcp_servers
from mcpevalkit.trace import ToolCall

async def test_writes_file(mcp_agent_run):
    trace = await mcp_agent_run(agent, "write hello", servers=mcp_servers)
    calls = [e.tool for e in trace if isinstance(e, ToolCall)]
    assert "write_file" in calls
```

---

## Core concepts

### Recorder

Connects to one or more MCP servers as a real client, observes the agent's tool-call sequence during a live run, and persists the full trajectory as a structured JSONL trace.

A trace captures: the prompt, every tool call (name, arguments, latency, result), every error, every retry, the final response, and timing metadata. Traces are designed to be diffed, replayed, and inspected by hand.

### Replay

Takes a recorded trace and replays it against a **mocked MCP server** — same tool surface, deterministic responses sourced from the trace. This makes evaluation:

- **Fast** — no network, no LLM-tool latency
- **Cheap** — no real API costs for downstream services
- **Deterministic** — same inputs every time, even if your tools touch the real world

Replay is the foundation of CI-friendly eval. Record once when something interesting happens; replay forever.

### Judges

Three layers of scoring, all composable. Deterministic judges need no LLM; the
LLM-judged ones take a bring-your-own callable and report confidence.

- **Step judges** — `ToolSelectedCorrectly`, `ArgsValid` (validates call arguments against a tool's JSON Schema). Deterministic.
- **Trajectory judges** — `ToolCalled`, `ToolCalledWith`, `NoRetryLoop` (deterministic); `TrajectoryEfficient` (LLM-judged).
- **Outcome judges** — `FinalStateMatches` (deterministic state assertions on disk or the final tool result); `FinalResponseSatisfies` (LLM-judged rubric over the response).

All layers report individually so you can see *where* a run failed, not just *that* it did.

In a `.mcpeval.yaml` suite, the assertion vocabulary maps onto these judges:
`tool_called`, `tool_called_with`, `no_retry_loop`, `final_state_matches`, and
`final_response_satisfies`. Print the full machine-readable schema any time with
`mcpevalkit schema`.

### Pytest plugin

The `pytest --mcpevalkit` plugin discovers `.mcpeval.yaml` files in your repo, materializes each eval as a pytest test, and reports pass/fail with per-step diagnostics in the test output. CI integration is one line.

---

## Example agent

The repository ships with a small example coding agent in
[`examples/coding_agent/`](examples/coding_agent/) — a deterministic, LLM-free
agent that drives the filesystem MCP server over stdio, scored against a fixed
task set. See its [`agent.py`](examples/coding_agent/agent.py),
[`.mcpeval.yaml`](examples/coding_agent/.mcpeval.yaml), and
[README](examples/coding_agent/README.md).

```bash
git clone https://github.com/treble37/mcp-evals
cd mcp-evals
uv sync
make demo
```

You'll get the scored trajectory report shown at the top of this page in a
second or two (it needs Node's `npx` for the filesystem server, but no API key —
the judge defaults to a deterministic fake). The agent is intentionally small
(~170 lines): it's a worked consumer of the library, not a separate product.
The same suite also runs under `pytest examples/coding_agent/ --mcpevalkit`.

---

## Design notes

### Why MCP-first?

A first-class MCP recorder is more accurate than a generic tool-call wrapper because it captures the full protocol surface — server capabilities, resource references, prompts, error semantics. Wrapping MCP behind a framework abstraction loses signal.

### Why trajectory-level?

Production agent failures are rarely "the final answer was wrong." They're "the agent picked the wrong tool at step 3 and recovered fine, but then misread its own state at step 6." Final-answer evaluation can't surface either of those. Step-level judges can.

### Why deterministic replay?

Eval suites that hit real LLM APIs and real tool servers don't run on CI — they're too slow, too expensive, and too flaky. Record once when something interesting happens; replay forever. This is the load-bearing design choice that makes `mcpevalkit` usable in real CI.

### Why a YAML eval format *and* a pytest plugin?

YAML is for non-engineering stakeholders defining expected behavior; pytest is for engineers integrating evals into existing test suites. Both compile to the same internal eval representation — you can use either or both.

---

## Roadmap

### v0.2 — Paired comparison

> Answer not just "is this agent good enough?" but **"is the new config better than the one in production?"**

- Run two configurations against the same eval suite
- Paired statistical comparison with reported confidence
- A `ship` / `don't-ship` decision rule with configurable thresholds
- CLI: `mcpevalkit compare --baseline traces/v1.jsonl --candidate traces/v2.jsonl`

### v0.3 and beyond

- Multi-turn conversation eval (currently single-turn only)
- Token / cost reporting per trajectory
- HTML report output for sharing with non-engineering reviewers

---

## Status

**v0.1 — Active development. API may shift.** Pin to a specific version in production.

Issue reports and PRs welcome. The fastest way to influence the shape of the library is to file an issue describing your MCP agent and how you wish you could test it.

---

## License

[Apache 2.0](./LICENSE) © Bruce P

---

## Acknowledgements

Built on top of the [Anthropic MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk).
