Metadata-Version: 2.4
Name: ninjaeval-mcp
Version: 0.1.0
Summary: NinjaEval as an MCP server: a coding agent tests a prompt, model or tool change and reads the verdict and the diagnosis before calling it done.
License: MIT
Project-URL: Homepage, https://ninjaeval.com
Project-URL: Documentation, https://ninjaeval.com/docs/developer-guide.html
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp<3,>=1.2
Dynamic: license-file

# NinjaEval MCP server

Lets your coding agent (Claude Code, Cursor, Copilot, any MCP client) test an AI change with
[NinjaEval](https://ninjaeval.com) before it calls the change done.

The agent edits your bot's system prompt, model or tools, then calls `test_change`. NinjaEval replays your test cases
on production and on the change and grades every answer. It returns **ready** or **blocked**, with the cause of each
regression and a fix that was re-run to check it. The agent applies the fix and tests again. Shipping stays with you:
the server can't promote a version to production.

## Set up

1. In NinjaEval, open **⚙ Workspace & API keys** and create a key for the agent (e.g. `coding-agent`).
2. Add the server to your MCP client. For Claude Code:

   ```bash
   claude mcp add ninjaeval --env NINJAEVAL_API_KEY=efk_... -- uvx ninjaeval-mcp
   ```

   Any other client (`mcpServers` in its JSON config):

   ```json
   { "mcpServers": { "ninjaeval": { "command": "uvx", "args": ["ninjaeval-mcp"],
                                    "env": { "NINJAEVAL_API_KEY": "efk_..." } } } }
   ```

   Or `pip install ninjaeval-mcp` and run `ninjaeval-mcp`.

It runs over stdio on your machine and opens no network port. It talks only to `api.ninjaeval.com`.

## Tools

| Tool | What it does |
|---|---|
| `whoami` | The workspace, the key's role, today's limits |
| `list_projects` | Your projects (AI apps under test) |
| `get_project` | Production's provider, model, system prompt and tools; the gate; suites; recent change tests |
| `test_change` | Tests a change against production: only the fields you pass change (`system_prompt`, `model`, `provider`, `params`, `tools`, `code_sha`, `note`, `stage`, `trials`). Waits for the verdict and the diagnoses |
| `get_change_test` | A test's verdict, rules, regressions with cause / explanation / fix (`prompt_edits`) / whether the fix was verified, cases still failing, and the next step |
| `diagnose_cases` | Diagnoses failing cases that weren't diagnosed automatically |
| `get_report` | The full report as Markdown, e.g. for a PR description |

Each response has a `next` field telling the agent what to do. When the cause is `TEST`, the test itself is wrong, so
the agent should tell you rather than bend the product to fit it.

## Settings

| Variable | Default | |
|---|---|---|
| `NINJAEVAL_API_KEY` | | Workspace key (`efk_…`) |
| `NINJAEVAL_API_KEY_FILE` | | Read the key from a file instead |
| `NINJAEVAL_API_URL` | `https://api.ninjaeval.com/api/v1/eng` | |
| `NINJAEVAL_APP_URL` | `https://ninjaeval.com` | Used for the links in responses |
| `NINJAEVAL_MCP_MAX_TESTS` | `10` | Change tests one session may start, so an agent can't loop on fixes indefinitely |

Each change test runs your real app and an AI judge, which takes minutes and counts toward the workspace's daily
limits. Endpoint projects call your service with the system prompt NinjaEval sends. If your service ignores it, test
code changes by deploying them where the endpoint runs and passing `code_sha`.
