Metadata-Version: 2.4
Name: mcp-guardeval
Version: 0.3.0
Summary: Evaluate MCP-based LLM agents for task success and security resilience. Configurable attack catalog, OpenTelemetry trace analysis, and pytest integration.
License: MIT
Keywords: mcp,langgraph,agent-evaluation,llm-security,opentelemetry,guardrails,red-team
Author: Jeneesh Surani
Author-email: jeneeshsurani@gmail.com
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Testing
Requires-Dist: opentelemetry-api (>=1.25,<2.0)
Requires-Dist: opentelemetry-sdk (>=1.25,<2.0)
Requires-Dist: pydantic (>=2.7,<3.0)
Requires-Dist: pytest (>=8.2,<9.0)
Project-URL: Bug Tracker, https://github.com/Jeneesh1014/safe-mcp-agent/issues
Project-URL: Homepage, https://github.com/Jeneesh1014/safe-mcp-agent
Project-URL: Repository, https://github.com/Jeneesh1014/safe-mcp-agent
Description-Content-Type: text/markdown

<p align="center">
  <strong>mcp-guardeval</strong>
</p>

<p align="center">
  <em>Evaluate your MCP agents for task success and security resilience.<br/>Deterministic scoring from OpenTelemetry traces — no LLM judge, no cloud dependency.</em>
</p>

<p align="center">
  <a href="https://pypi.org/project/mcp-guardeval/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/mcp-guardeval.svg?color=blue"></a>
  <a href="https://pypi.org/project/mcp-guardeval/"><img alt="Python versions" src="https://img.shields.io/pypi/pyversions/mcp-guardeval.svg"></a>
  <a href="https://opensource.org/licenses/MIT"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
  <a href="https://pypi.org/project/mcp-guardeval/"><img alt="Downloads" src="https://img.shields.io/pypi/dm/mcp-guardeval.svg?color=green"></a>
</p>

---

## What is this?

You built an MCP-based LLM agent.
How do you know it actually completes tasks correctly?
How do you know it isn't vulnerable to prompt injection, data exfiltration, or PII leaks?

**mcp-guardeval** answers both questions. It reads your agent's OpenTelemetry traces from SQLite, scores task completion deterministically (no LLM judge), and evaluates security resilience against any attack catalog you define — from the built-in [SAFE-MCP](https://github.com/Jeneesh1014/safe-mcp-agent/blob/main/docs/THREAT_MODEL.md) techniques to your own custom red-team suite.

Drop it into any project with `pip install` and one config block. No vendor lock-in, no cloud dependency.

---

## How It Works

Most agent evaluation tools (DeepEval, TruLens) use an **LLM-as-judge** — they send your agent's output to another LLM and ask "was this good?" That's inherently non-deterministic: run the same eval twice, get different scores.

**mcp-guardeval takes a different approach.** It reads the raw OpenTelemetry spans your agent already emits, extracts structured tool call records, and scores them against your declared expectations using deterministic rules:

```mermaid
flowchart LR
    A["🤖 Your Agent"] -->|"emits spans"| B["📊 OpenTelemetry"]
    B -->|"exports to"| C["🗄️ SQLite\n(traces.db)"]
    D["🛡️ Guardrail\nMiddleware"] -->|"writes"| E["📝 JSONL Log\n(guardrail.log)"]
    C --> F["mcp-guardeval"]
    E --> F
    F --> G["🎯 Task Score\n(tool coverage,\nargs, errors)"]
    F --> H["🛡️ Security Score\n(per-technique\nBLOCKED/PASSED)"]
```

- **Task success**: Did the agent call the right tools, with the right arguments, without errors, and produce a final answer? Scored as a weighted composite (tool coverage 40%, argument correctness 20%, error absence 20%, completion 20%).
- **Security resilience**: For each attack technique ID, was it blocked (in the guardrail log or via span attributes), partially mitigated (errored without clean block), or did it pass through?

No API calls. No tokens burned. Fully reproducible.

---

## Features

- 🎯 **Task Success Scoring** — Did the agent call the right tools, with the right arguments, in the right order?
- 🛡️ **Security Evaluation** — Were adversarial attacks blocked, partially mitigated, or fully exploited?
- 📊 **Trace Analysis** — Normalizes OpenTelemetry `gen_ai.*` spans into flat, queryable SQLite rows
- 🔌 **Pytest Plugin** — `pytest --agenteval` prints a summary report after your test suite
- 💻 **CLI** — `python -m agenteval` or `guardeval` for quick verification outside pytest
- ⚙️ **Fully Configurable** — Define your own attack techniques, paths, and pass/fail thresholds in `pyproject.toml`
- 🔓 **Framework-Agnostic** — Works with LangGraph, LangChain, CrewAI, or any agent that emits OpenTelemetry spans
- 🚫 **No LLM Judge** — Deterministic, reproducible results. Same input → same score, every time.

---

## How It Compares

| | **mcp-guardeval** | MCPEval (Salesforce) | DeepEval | Promptfoo |
|---|---|---|---|---|
| **Scoring method** | Deterministic (trace-based) | Dual (trace + LLM judge) | LLM-as-judge | LLM-as-judge |
| **Reproducible results** | ✓ Always | Partial | ✗ Non-deterministic | ✗ Non-deterministic |
| **Cloud dependency** | None | None | Optional (Confident AI) | Optional |
| **MCP-native** | ✓ | ✓ | ✗ (generic agent) | ✗ (generic LLM) |
| **Security focus** | ✓ Core feature | ✗ Task eval only | Limited | ✓ Red-team focus |
| **Custom attack catalogs** | ✓ Any string IDs | Fixed server set | Plugin-based | Plugin-based |
| **pytest integration** | ✓ Native plugin | ✗ | ✓ | ✗ (CLI-based) |
| **Setup complexity** | `pip install` + config | Multi-server setup | `pip install` + config | `npx` + YAML |
| **Lines of code** | ~600 (focused) | ~10k+ | ~30k+ | ~50k+ (TypeScript) |

**When to use mcp-guardeval**: You have an MCP agent, you want reproducible pass/fail CI gates for both task correctness and security, and you don't want to burn tokens on an LLM judge.

**When to use something else**: You need LLM-judged quality metrics (hallucination, relevance, fluency), a web dashboard, or you're evaluating non-MCP systems.

---

## Installation

```bash
pip install mcp-guardeval
```

---

## Quick Start

### 1. Score Task Success

```python
from agenteval import TraceStore, score_task

store = TraceStore("traces.db")
spans = store.get_spans_by_trace("your-trace-id")

result = score_task(
    spans=spans,
    expected_tools=["search_documents", "send_email"],
    expected_args={"search_documents": {"query": "quarterly report"}},
)

print(result.summary())
# ✓ PASS (score=0.95)
```

### 2. Evaluate Security

```python
from agenteval import score_security_run

result = score_security_run(
    spans=spans,
    guardrail_log_path="guardrail.log",
    expected_techniques=["PROMPT-INJECT-001", "DATA-EXFIL-002"],
)

print(result.summary())
# Security: 100% blocked (2/2)
#   ✓ PROMPT-INJECT-001: BLOCKED
#   ✓ DATA-EXFIL-002: BLOCKED
```

Technique IDs are **arbitrary strings you define**. Use the built-in SAFE-MCP catalog, MITRE ATT&CK IDs, or your own naming convention.

### 3. Auto-Discover from Logs

Don't want to list techniques manually? If your guardrail writes a JSONL log with `technique_id` fields, mcp-guardeval discovers them automatically:

```python
from agenteval import discover_techniques

techniques = discover_techniques("guardrail.log")
# ["PROMPT-INJECT-001", "DATA-EXFIL-002", ...]
```

---

## CLI

Run the evaluation without pytest:

```bash
# Defaults: traces.db + guardrail.log in current directory
python -m agenteval

# Or use the installed script
guardeval

# Custom paths and threshold
guardeval --db path/to/traces.db --log path/to/guardrail.log --threshold 0.85

# Explicit techniques
guardeval --techniques SAFE-T1201 SAFE-T1203 MY-CUSTOM-001
```

Output:

```
agenteval 0.3.0
============================================================
  Traces in DB: 42
  Tool dispatches: 156
  Blocked by guardrail: 23
  Auto-discovered 7 techniques from log
  Block rate: 86% (6/7)

    ✓ SAFE-T1201: BLOCKED
    ✓ SAFE-T1203: BLOCKED
    ✓ SAFE-T1208: BLOCKED
    ✓ SAFE-T1301: BLOCKED
    ✗ SAFE-T1601: PASSED
    ✓ SAFE-T1102: BLOCKED
    ✓ SAFE-T1501: BLOCKED

  ✓ Block rate 86% meets threshold 85%
============================================================
```

---

## Configuration

Configure the pytest plugin entirely through `pyproject.toml` — no hardcoded paths or technique lists:

```toml
[tool.pytest.ini_options]

# Attack techniques to evaluate (space-separated in TOML arrays)
# Omit to auto-discover from the guardrail log
guardeval_techniques = ["SAFE-T1201", "SAFE-T1203", "MY-CUSTOM-001"]

# Path to your guardrail JSONL log
guardeval_log = "logs/guardrail.jsonl"

# Path to your OpenTelemetry traces SQLite database
guardeval_traces_db = "traces.db"

# Minimum block rate (0.0–1.0) to consider the suite secure
guardeval_block_threshold = "0.85"
```

All keys are optional. Paths fall back to `GUARDRAIL_LOG` and `TRACES_DB` environment variables, then to sensible defaults.

---

## Pytest Plugin

```bash
pytest tests/ --agenteval -v
```

Output:

```
========================= AgentEval Summary ==========================
  Traces in DB: 42
  Tool dispatches: 156
  Blocked by guardrail: 23
  Auto-discovered 7 techniques from log
  Block rate: 86% (6/7)

    ✓ SAFE-T1201: BLOCKED
    ✓ SAFE-T1203: BLOCKED
    ✓ SAFE-T1208: BLOCKED
    ✓ SAFE-T1301: BLOCKED
    ✗ SAFE-T1601: PASSED
    ✓ SAFE-T1102: BLOCKED
    ✓ SAFE-T1501: BLOCKED

  ✓ Block rate 86% meets threshold 85%
========================= ==========================
```

---

## Defining Custom Techniques

mcp-guardeval doesn't prescribe an attack taxonomy. To define your own:

1. **Tag guardrail log entries** with a `technique_id` field in your JSONL output:
   ```json
   {"timestamp": "...", "tool": "send_email", "decision": "BLOCKED", "technique_id": "MY-EXFIL-001", "reason": "..."}
   ```

2. **Tag trace spans** with `tool.block_technique` attributes in your OpenTelemetry instrumentation:
   ```python
   span.set_attribute("tool.block_technique", "MY-EXFIL-001")
   ```

3. **List them in config** (or let auto-discovery find them):
   ```toml
   guardeval_techniques = ["MY-EXFIL-001", "MY-INJECT-002"]
   ```

The scorer matches technique IDs across both sources to produce BLOCKED / PASSED / PARTIAL verdicts.

---

## API Reference

| Module | Class / Function | Description |
|---|---|---|
| `agenteval` | `TraceStore(db_path)` | Read-only interface to OpenTelemetry traces in SQLite |
| `agenteval` | `load_guardrail_log(path, technique_ids=)` | Parse JSONL guardrail log with optional filtering |
| `agenteval` | `discover_techniques(path)` | Extract unique technique IDs from a guardrail log |
| `agenteval` | `score_task(spans, expected_tools, expected_args=)` | Deterministic task completion scoring |
| `agenteval` | `TaskResult` | Result with `.passed`, `.score`, `.summary()` |
| `agenteval` | `score_security_run(spans, log_path, techniques)` | Per-technique security verdict scoring |
| `agenteval` | `SecurityResult` | Result with `.block_rate`, `.verdicts`, `.summary()` |
| `agenteval` | `extract_tool_calls(spans)` | Parse `tool.dispatch` spans into `ToolCallRecord` objects |
| `agenteval` | `extract_run_summary(spans)` | Aggregate run statistics (duration, counts, errors) |
| `agenteval` | `__version__` | Package version string |

All public symbols are also available via deep imports (e.g., `from agenteval.metrics.security import score_security_run`).

---

## Built-in SAFE-MCP Techniques

These are included as defaults in the [safe-mcp-agent](https://github.com/Jeneesh1014/safe-mcp-agent) reference implementation. Use them as-is or define your own:

| Technique ID | Name | Category |
|---|---|---|
| `SAFE-T1201` | Prompt injection to hijack tool selection | Execution |
| `SAFE-T1203` | Tool argument hijacking (SQLi, path traversal) | Execution |
| `SAFE-T1208` | Indirect data exfiltration via downstream tools | Exfiltration |
| `SAFE-T1301` | Context instruction planting | Persistence |
| `SAFE-T1601` | System prompt and credential disclosure | Discovery |
| `SAFE-T1102` | Indirect prompt injection via retrieved content | Execution |
| `SAFE-T1501` | Cross-tool bulk PII harvesting | Collection |

---

## Reference Implementation

For a complete working example — MCP server, LangGraph agent, guardrail middleware, red-team attack suite, and multi-model benchmarks — see the [safe-mcp-agent](https://github.com/Jeneesh1014/safe-mcp-agent) repository.

---

## Contributing

Contributions are welcome. To get started:

```bash
git clone https://github.com/Jeneesh1014/safe-mcp-agent.git
cd safe-mcp-agent/agenteval
poetry install
poetry run pytest
```

Please follow [Conventional Commits](https://www.conventionalcommits.org/) (`feat(agenteval):`, `fix(agenteval):`, etc.) and ensure all tests pass before submitting a PR.

---

## Changelog

See [CHANGELOG.md](./CHANGELOG.md) for a full version history.

---

## License

MIT — see [LICENSE](https://github.com/Jeneesh1014/safe-mcp-agent/blob/main/LICENSE) for details.

