MCP Test Harnessv2.0.2

MCP Test Report

One test run. Multiple formats. Complete visibility.

RUN FAILED Harness 2.0.2
Protocol 2025-03-26 2026-07-06T20:10:00.000Z → 2026-07-06T20:10:02.340Z

Test report outputs

One test run. Multiple formats. Complete visibility. Click a card to expand full Console, JUnit, or JSON output.

ConsoleHuman-readable terminal outputClick for full output
  PASS  test_echo_tool_returns_message  (12.4ms)
  PASS  test_tool_schema_matches_openapi  (48.2ms)
  PASS  test_list_tools_includes_echo  (6.8ms)
  FAIL  test_echo_rejects_empty_payload  (31.5ms)
      Tool 'echo' response mismatch
      --- expected
…
  PASS  test_echo_tool_returns_message  (12.4ms)
  PASS  test_tool_schema_matches_openapi  (48.2ms)
  PASS  test_list_tools_includes_echo  (6.8ms)
  FAIL  test_echo_rejects_empty_payload  (31.5ms)
      Tool 'echo' response mismatch
      --- expected
      +++ actual
      @@ -1,3 +1,3 @@
       [{"text": "validation error", "isError": true}]
      +[{"text": "ok", "isError": false}]
  FAIL  test_report_snapshot_stable  (89.0ms)
      Snapshot mismatch for 'generate_report'
      --- expected
      +++ actual
      - {'version': '2.0.0'}
      + {'version': '2.0.1'}
  PASS  test_tool_idempotent_calculate  (156.3ms)
  PASS  test_injection_payloads_blocked  (420.7ms)
  FAIL  test_no_secret_leak_in_logs  (55.1ms)
      Tool 'get_config' output matched secret pattern(s): aws_key
  PASS  test_path_traversal_blocked  (210.4ms)
  PASS  test_echo_latency_p95_under_slo  (1240.0ms)
  FAIL  test_throughput_min_rps  (980.0ms)
      Throughput 42.1 rps below minimum 50.0 rps (p99=312ms)
  PASS  test_degrades_gracefully_on_bad_args  (72.0ms)
  PASS  test_chaos_delay_tolerance  (145.0ms)
  FAIL  test_chaos_503_unavailable  (88.0ms)
      ChaosFaultError: simulated 503 Service Unavailable
  PASS  test_chaos_schema_drift_probe  (62.0ms)
  ERR  test_survives_server_restart  (18.0ms)
      ConnectionResetError: MCP server process exited with code 1
      Traceback (most recent call last):
        File tests/test_resiliency.py, line 44
          await assert_tool_call(...)
  PASS  test_flaky_resource_read  (340.0ms)
      [!] flaky (passed on retry)
  FAIL  test_schema_probe_strict  (44.0ms)
      Schema validation failed for tool 'search'
  SKIP  test_prompts_when_available  (0.0ms)
  SKIP  test_optional_bedrock_pack  (0.0ms)
  TIME  test_long_running_export  (30000.0ms)

11 passed, 6 failed, 1 errored, 2 skipped
Total time: 2340.0ms

========================================
FAILURES
--------------------
tests/test_server.py
  test_echo_rejects_empty_payload — Tool 'echo' response mismatch
--- expected
+++ actual
@@ -1,3 +1,3 @@
 [{"text": "validation error", "isError": true}]
+[{"text": "ok", "isError": false}]
--------------------
tests/test_regression.py
  test_report_snapshot_stable — Snapshot mismatch for 'generate_report'
--- expected
+++ actual
- {'version': '2.0.0'}
+ {'version': '2.0.1'}
--------------------
tests/test_security.py
  test_no_secret_leak_in_logs — Tool 'get_config' output matched secret pattern(s): aws_key
--------------------
tests/test_performance.py
  test_throughput_min_rps — Throughput 42.1 rps below minimum 50.0 rps (p99=312ms)
--------------------
tests/test_resiliency.py
  test_chaos_503_unavailable — ChaosFaultError: simulated 503 Service Unavailable
--------------------
tests/test_resiliency.py
  test_survives_server_restart — ConnectionResetError: MCP server process exited with code 1
Traceback (most recent call last):
  File tests/test_resiliency.py, line 44
    await assert_tool_call(...)
--------------------
tests/test_schema.py
  test_schema_probe_strict — Schema validation failed for tool 'search'
--------------------
tests/test_performance.py
  test_long_running_export — Test exceeded timeout of 30000ms
JUnit XMLCI/CD integrationClick for full output
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="mcp-test" tests="21" failures="6" errors="2" skipped="2" time="2.340" timestamp="2026-07-06T20:10:02.340Z" hostname="LAPTOP-UBC5B3VU">
    <properties>
      <property name="harness_version" value="2.0.2" />
…
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="mcp-test" tests="21" failures="6" errors="2" skipped="2" time="2.340" timestamp="2026-07-06T20:10:02.340Z" hostname="LAPTOP-UBC5B3VU">
    <properties>
      <property name="harness_version" value="2.0.2" />
      <property name="mcp_protocol_version" value="2025-03-26" />
      <property name="transport" value="stdio" />
      <property name="server_command" value="python -m my_mcp_server --transport stdio" />
      <property name="run_started_at" value="2026-07-06T20:10:00.000Z" />
      <property name="run_finished_at" value="2026-07-06T20:10:02.340Z" />
    </properties>
    <testcase name="test_echo_tool_returns_message" classname="tests/test_server.py" time="0.012">
      <properties>
        <property name="tag" value="smoke" />
        <property name="tag" value="functional" />
      </properties>
    </testcase>
    <testcase name="test_tool_schema_matches_openapi" classname="tests/test_server.py" time="0.048">
      <properties>
        <property name="tag" value="functional" />
      </properties>
    </testcase>
    <testcase name="test_list_tools_includes_echo" classname="tests/test_server.py" time="0.007">
      <properties>
        <property name="tag" value="smoke" />
      </properties>
    </testcase>
    <testcase name="test_echo_rejects_empty_payload" classname="tests/test_server.py" time="0.032">
      <properties>
        <property name="tag" value="functional" />
      </properties>
      <failure message="Tool &apos;echo&apos; response mismatch">--- expected
+++ actual
@@ -1,3 +1,3 @@
 [{&quot;text&quot;: &quot;validation error&quot;, &quot;isError&quot;: true}]
+[{&quot;text&quot;: &quot;ok&quot;, &quot;isError&quot;: false}]</failure>
    </testcase>
    <testcase name="test_report_snapshot_stable" classname="tests/test_regression.py" time="0.089">
      <properties>
        <property name="tag" value="regression" />
        <property name="tag" value="snapshot" />
      </properties>
      <failure message="Snapshot mismatch for &apos;generate_report&apos;">--- expected
+++ actual
- {&apos;version&apos;: &apos;2.0.0&apos;}
+ {&apos;version&apos;: &apos;2.0.1&apos;}</failure>
    </testcase>
    <testcase name="test_tool_idempotent_calculate" classname="tests/test_regression.py" time="0.156">
      <properties>
        <property name="tag" value="regression" />
      </properties>
    </testcase>
    <testcase name="test_injection_payloads_blocked" classname="tests/test_security.py" time="0.421">
      <properties>
        <property name="tag" value="security" />
      </properties>
    </testcase>
    <testcase name="test_no_secret_leak_in_logs" classname="tests/test_security.py" time="0.055">
      <properties>
        <property name="tag" value="security" />
      </properties>
      <failure message="Tool &apos;get_config&apos; output matched secret pattern(s): aws_key"></failure>
    </testcase>
    <testcase name="test_path_traversal_blocked" classname="tests/test_security.py" time="0.210">
      <properties>
        <property name="tag" value="security" />
      </properties>
    </testcase>
    <testcase name="test_echo_latency_p95_under_slo" classname="tests/test_performance.py" time="1.240">
      <properties>
        <property name="tag" value="perf" />
        <property name="tag" value="latency" />
      </properties>
    </testcase>
    <testcase name="test_throughput_min_rps" classname="tests/test_performance.py" time="0.980">
      <properties>
        <property name="tag" value="perf" />
        <property name="tag" value="throughput" />
      </properties>
      <failure message="Throughput 42.1 rps below minimum 50.0 rps (p99=312ms)"></failure>
    </testcase>
    <testcase name="test_degrades_gracefully_on_bad_args" classname="tests/test_resiliency.py" time="0.072">
      <properties>
        <property name="tag" value="resiliency" />
        <property name="tag" value="chaos" />
      </properties>
    </testcase>
    <testcase name="test_chaos_delay_tolerance" classname="tests/test_resiliency.py" time="0.145">
      <properties>
        <property name="tag" value="chaos" />
        <property name="tag" value="chaos:delay" />
        <property name="tag" value="resiliency" />
      </properties>
    </testcase>
    <testcase name="test_chaos_503_unavailable" classname="tests/test_resiliency.py" time="0.088">
      <properties>
        <property name="tag" value="chaos" />
        <property name="tag" value="chaos:503" />
        <property name="tag" value="resiliency" />
      </properties>
      <failure message="ChaosFaultError: simulated 503 Service Unavailable"></failure>
    </testcase>
    <testcase name="test_chaos_schema_drift_probe" classname="tests/test_resiliency.py" time="0.062">
      <properties>
        <property name="tag" value="chaos" />
        <property name="tag" value="chaos:schema_drift" />
      </properties>
    </testcase>
    <testcase name="test_survives_server_restart" classname="tests/test_resiliency.py" time="0.018">
      <properties>
        <property name="tag" value="resiliency" />
      </properties>
      <error message="ConnectionResetError: MCP server process exited with code 1">Traceback (most recent call last):
  File tests/test_resiliency.py, line 44
    await assert_tool_call(...)</error>
    </testcase>
    <testcase name="test_flaky_resource_read" classname="tests/test_resiliency.py" time="0.340">
      <properties>
        <property name="tag" value="resiliency" />
      </properties>
    </testcase>
    <testcase name="test_schema_probe_strict" classname="tests/test_schema.py" time="0.044">
      <properties>
        <property name="tag" value="functional" />
      </properties>
      <failure message="Schema validation failed for tool &apos;search&apos;"></failure>
    </testcase>
    <testcase name="test_prompts_when_available" classname="tests/test_prompts.py" time="0.000">
      <properties>
        <property name="tag" value="functional" />
      </properties>
      <skipped message="server has no prompts capability" />
    </testcase>
    <testcase name="test_optional_bedrock_pack" classname="tests/test_prompts.py" time="0.000">
      <properties>
        <property name="tag" value="integration" />
      </properties>
      <skipped message="optional extra mcp-test-harness-bedrock not installed" />
    </testcase>
    <testcase name="test_long_running_export" classname="tests/test_performance.py" time="30.000">
      <properties>
        <property name="tag" value="perf" />
      </properties>
      <error message="Test exceeded timeout of 30000ms">Test timed out</error>
    </testcase>
  </testsuite>
</testsuites>
JSONMachine-readable dataClick for full output
{
  "metadata": {
    "harness_version": "2.0.2",
    "protocol_version": "2025-03-26",
    "server_capabilities": {
…
{
  "metadata": {
    "harness_version": "2.0.2",
    "protocol_version": "2025-03-26",
    "server_capabilities": {
      "tools": {
        "listChanged": true
      },
      "resources": {
        "subscribe": false
      },
      "logging": {},
      "prompts": {}
    },
    "started_at": "2026-07-06T20:10:00.000Z",
    "finished_at": "2026-07-06T20:10:02.340Z"
  },
  "environment": {
    "python_version": "3.12.4",
    "platform": "Windows-11-amd64",
    "cwd": "C:/demo/mcp-server",
    "server_command": "python -m my_mcp_server --transport stdio",
    "transport": "stdio",
    "bastion_paired": "CI verifies \u00b7 Bastion protects production",
    "report_generated_at": "2026-07-07T05:55:05.469847+00:00"
  },
  "summary": {
    "total": 21,
    "passed": 11,
    "failed": 6,
    "errored": 1,
    "skipped": 2,
    "timed_out": 1,
    "total_duration_ms": 2340.0,
    "duration_percentiles_ms": {
      "p50_ms": 88.0,
      "p95_ms": 30000.0,
      "p99_ms": 30000.0
    }
  },
  "tests": [
    {
      "name": "test_echo_tool_returns_message",
      "module": "tests/test_server.py",
      "file": "tests/test_server.py",
      "status": "passed",
      "duration_ms": 12.4,
      "tags": [
        "smoke",
        "functional"
      ],
      "started_at": "2026-07-06T20:10:01.120Z"
    },
    {
      "name": "test_tool_schema_matches_openapi",
      "module": "tests/test_server.py",
      "file": "tests/test_server.py",
      "status": "passed",
      "duration_ms": 48.2,
      "tags": [
        "functional"
      ],
      "started_at": "2026-07-06T20:10:00.117000Z"
    },
    {
      "name": "test_list_tools_includes_echo",
      "module": "tests/test_server.py",
      "file": "tests/test_server.py",
      "status": "passed",
      "duration_ms": 6.8,
      "tags": [
        "smoke"
      ],
      "started_at": "2026-07-06T20:10:00.234000Z"
    },
    {
      "name": "test_echo_rejects_empty_payload",
      "module": "tests/test_server.py",
      "file": "tests/test_server.py",
      "status": "failed",
      "duration_ms": 31.5,
      "tags": [
        "functional"
      ],
      "started_at": "2026-07-06T20:10:00.351000Z",
      "error": "Tool 'echo' response mismatch",
      "assertion_diff": "--- expected\n+++ actual\n@@ -1,3 +1,3 @@\n [{\"text\": \"validation error\", \"isError\": true}]\n+[{\"text\": \"ok\", \"isError\": false}]",
      "mcp_trace": {
        "event_count": 4,
        "stdio_pollution": [
          "WARN: deprecated config key in stderr"
        ],
        "events": [
          {
            "t_ms": 0.8,
            "direction": "request",
            "method": "tools/call",
            "bytes": 84,
            "payload": {
              "name": "echo",
              "arguments": {
                "message": "ping"
              }
            }
          },
          {
            "t_ms": 3.2,
            "direction": "response",
            "method": "tools/call",
            "bytes": 120,
            "payload": {
              "content": [
                {
                  "type": "text",
                  "text": "ping"
                }
              ]
            }
          },
          {
            "t_ms": 4.1,
            "direction": "request",
            "method": "tools/call",
            "bytes": 92,
            "payload": {
              "name": "echo",
              "arguments": {
                "message": ""
              }
            }
          },
          {
            "t_ms": 5.0,
            "direction": "error",
            "method": "tools/call",
            "bytes": 64,
            "payload": {
              "isError": true,
              "message": "message must be non-empty"
            },
            "stdio_pollution": true
          }
        ]
      }
    },
    {
      "name": "test_report_snapshot_stable",
      "module": "tests/test_regression.py",
      "file": "tests/test_regression.py",
      "status": "failed",
      "duration_ms": 89.0,
      "tags": [
        "regression",
        "snapshot"
      ],
      "started_at": "2026-07-06T20:10:00.468000Z",
      "error": "Snapshot mismatch for 'generate_report'",
      "assertion_diff": "--- expected\n+++ actual\n- {'version': '2.0.0'}\n+ {'version': '2.0.1'}"
    },
    {
      "name": "test_tool_idempotent_calculate",
      "module": "tests/test_regression.py",
      "file": "tests/test_regression.py",
      "status": "passed",
      "duration_ms": 156.3,
      "tags": [
        "regression"
      ],
      "started_at": "2026-07-06T20:10:00.585000Z"
    },
    {
      "name": "test_injection_payloads_blocked",
      "module": "tests/test_security.py",
      "file": "tests/test_security.py",
      "status": "passed",
      "duration_ms": 420.7,
      "tags": [
        "security"
      ],
      "started_at": "2026-07-06T20:10:00.702000Z"
    },
    {
      "name": "test_no_secret_leak_in_logs",
      "module": "tests/test_security.py",
      "file": "tests/test_security.py",
      "status": "failed",
      "duration_ms": 55.1,
      "tags": [
        "security"
      ],
      "started_at": "2026-07-06T20:10:00.819000Z",
      "error": "Tool 'get_config' output matched secret pattern(s): aws_key"
    },
    {
      "name": "test_path_traversal_blocked",
      "module": "tests/test_security.py",
      "file": "tests/test_security.py",
      "status": "passed",
      "duration_ms": 210.4,
      "tags": [
        "security"
      ],
      "started_at": "2026-07-06T20:10:00.936000Z"
    },
    {
      "name": "test_echo_latency_p95_under_slo",
      "module": "tests/test_performance.py",
      "file": "tests/test_performance.py",
      "status": "passed",
      "duration_ms": 1240.0,
      "tags": [
        "perf",
        "latency"
      ],
      "started_at": "2026-07-06T20:10:01.053000Z"
    },
    {
      "name": "test_throughput_min_rps",
      "module": "tests/test_performance.py",
      "file": "tests/test_performance.py",
      "status": "failed",
      "duration_ms": 980.0,
      "tags": [
        "perf",
        "throughput"
      ],
      "started_at": "2026-07-06T20:10:01.170000Z",
      "error": "Throughput 42.1 rps below minimum 50.0 rps (p99=312ms)"
    },
    {
      "name": "test_degrades_gracefully_on_bad_args",
      "module": "tests/test_resiliency.py",
      "file": "tests/test_resiliency.py",
      "status": "passed",
      "duration_ms": 72.0,
      "tags": [
        "resiliency",
        "chaos"
      ],
      "started_at": "2026-07-06T20:10:01.287000Z"
    },
    {
      "name": "test_chaos_delay_tolerance",
      "module": "tests/test_resiliency.py",
      "file": "tests/test_resiliency.py",
      "status": "passed",
      "duration_ms": 145.0,
      "tags": [
        "chaos",
        "chaos:delay",
        "resiliency"
      ],
      "started_at": "2026-07-06T20:10:01.404000Z"
    },
    {
      "name": "test_chaos_503_unavailable",
      "module": "tests/test_resiliency.py",
      "file": "tests/test_resiliency.py",
      "status": "failed",
      "duration_ms": 88.0,
      "tags": [
        "chaos",
        "chaos:503",
        "resiliency"
      ],
      "started_at": "2026-07-06T20:10:01.521000Z",
      "error": "ChaosFaultError: simulated 503 Service Unavailable"
    },
    {
      "name": "test_chaos_schema_drift_probe",
      "module": "tests/test_resiliency.py",
      "file": "tests/test_resiliency.py",
      "status": "passed",
      "duration_ms": 62.0,
      "tags": [
        "chaos",
        "chaos:schema_drift"
      ],
      "started_at": "2026-07-06T20:10:01.638000Z"
    },
    {
      "name": "test_survives_server_restart",
      "module": "tests/test_resiliency.py",
      "file": "tests/test_resiliency.py",
      "status": "error",
      "duration_ms": 18.0,
      "tags": [
        "resiliency"
      ],
      "started_at": "2026-07-06T20:10:01.755000Z",
      "error": "ConnectionResetError: MCP server process exited with code 1",
      "traceback": "Traceback (most recent call last):\n  File tests/test_resiliency.py, line 44\n    await assert_tool_call(...)"
    },
    {
      "name": "test_flaky_resource_read",
      "module": "tests/test_resiliency.py",
      "file": "tests/test_resiliency.py",
      "status": "passed",
      "duration_ms": 340.0,
      "tags": [
        "resiliency"
      ],
      "started_at": "2026-07-06T20:10:01.872000Z",
      "retry_count": 2,
      "attempt_results": [
        {
          "attempt": 1,
          "status": "timeout",
          "duration_ms": 5000.0,
          "error": "read_resource timed out"
        },
        {
          "attempt": 2,
          "status": "error",
          "duration_ms": 120.0,
          "error": "503 from upstream"
        },
        {
          "attempt": 3,
          "status": "passed",
          "duration_ms": 88.0,
          "error": null
        }
      ],
      "flaky": true
    },
    {
      "name": "test_schema_probe_strict",
      "module": "tests/test_schema.py",
      "file": "tests/test_schema.py",
      "status": "failed",
      "duration_ms": 44.0,
      "tags": [
        "functional"
      ],
      "started_at": "2026-07-06T20:10:01.989000Z",
      "error": "Schema validation failed for tool 'search'",
      "schema_violations": [
        {
          "json_path": "$.arguments.limit",
          "expected_type": "integer",
          "actual_value": "ten",
          "message": "limit must be an integer"
        }
      ]
    },
    {
      "name": "test_prompts_when_available",
      "module": "tests/test_prompts.py",
      "file": "tests/test_prompts.py",
      "status": "skipped",
      "duration_ms": 0.0,
      "tags": [
        "functional"
      ],
      "started_at": "2026-07-06T20:10:02.106000Z",
      "error": "server has no prompts capability"
    },
    {
      "name": "test_optional_bedrock_pack",
      "module": "tests/test_prompts.py",
      "file": "tests/test_prompts.py",
      "status": "skipped",
      "duration_ms": 0.0,
      "tags": [
        "integration"
      ],
      "started_at": "2026-07-06T20:10:02.223000Z",
      "error": "optional extra mcp-test-harness-bedrock not installed"
    },
    {
      "name": "test_long_running_export",
      "module": "tests/test_performance.py",
      "file": "tests/test_performance.py",
      "status": "timeout",
      "duration_ms": 30000.0,
      "tags": [
        "perf"
      ],
      "started_at": "2026-07-06T20:10:02.340000Z",
      "error": "Test exceeded timeout of 30000ms"
    }
  ],
  "unified_summary": {
    "gate": "fail",
    "overall": {
      "total": 21,
      "passed": 11,
      "failed": 8,
      "skipped": 2,
      "duration_ms": 2340.0
    },
    "categories": {
      "functional": {
        "total": 9,
        "passed": 4,
        "failed": 3,
        "score_pct": 44.4,
        "status": "fail"
      },
      "performance": {
        "total": 3,
        "passed": 1,
        "failed": 2,
        "score_pct": 33.3,
        "status": "fail"
      },
      "security": {
        "total": 3,
        "passed": 2,
        "failed": 1,
        "score_pct": 66.7,
        "status": "fail"
      },
      "resiliency": {
        "total": 6,
        "passed": 4,
        "failed": 2,
        "score_pct": 66.7,
        "status": "fail"
      }
    },
    "coverage": {
      "advertised": {
        "tools": [
          "calculate",
          "echo",
          "export_report",
          "get_config",
          "search"
        ],
        "resources": [
          "file:///config.json",
          "file:///health"
        ],
        "prompts": [
          "summarize"
        ]
      },
      "tested": {
        "tools": [
          "calculate",
          "echo",
          "get_config",
          "search"
        ],
        "resources": [
          "file:///config.json"
        ],
        "prompts": [],
        "auth_tools": [
          "echo",
          "search"
        ]
      },
      "gaps": {
        "untested_tools": [
          "export_report"
        ],
        "untested_resources": [
          "file:///health"
        ],
        "untested_prompts": [
          "summarize"
        ],
        "tools_missing_auth_tests": [
          "calculate",
          "get_config"
        ]
      },
      "summary": {
        "tools_advertised": 5,
        "tools_tested": 4,
        "tools_untested": 1,
        "tools_missing_auth": 2
      }
    },
    "coverage_headline": "4 tools tested, 1 untested, 2 missing auth tests"
  },
  "coverage": {
    "advertised": {
      "tools": [
        "calculate",
        "echo",
        "export_report",
        "get_config",
        "search"
      ],
      "resources": [
        "file:///config.json",
        "file:///health"
      ],
      "prompts": [
        "summarize"
      ]
    },
    "tested": {
      "tools": [
        "calculate",
        "echo",
        "get_config",
        "search"
      ],
      "resources": [
        "file:///config.json"
      ],
      "prompts": [],
      "auth_tools": [
        "echo",
        "search"
      ]
    },
    "gaps": {
      "untested_tools": [
        "export_report"
      ],
      "untested_resources": [
        "file:///health"
      ],
      "untested_prompts": [
        "summarize"
      ],
      "tools_missing_auth_tests": [
        "calculate",
        "get_config"
      ]
    },
    "summary": {
      "tools_advertised": 5,
      "tools_tested": 4,
      "tools_untested": 1,
      "tools_missing_auth": 2
    }
  },
  "security_findings": [
    {
      "test_name": "test_no_secret_leak_in_logs",
      "file": "tests/test_security.py",
      "status": "failed",
      "message": "Tool 'get_config' output matched secret pattern(s): aws_key",
      "rule": {
        "id": "mcp01-token-mismanagement",
        "name": "Token Mismanagement & Secret Exposure",
        "framework": "OWASP-MCP",
        "owasp_id": "MCP01",
        "severity": "high"
      }
    }
  ]
}
HTML Dashboard ↓ view below
Rich visual reporting (this page)
PASS 11 · FAIL 8 · SKIP 2
Pass rate 52% · 2.34s
Click to jump to charts & filters below

Export this run: mcp-test --report-format html --report-output report.html · --report-format junit · --report-format json · mcp-test export-pdf report.html

QUALITY GATE: FAILED
Passed
11
Failed
8
Skipped
2
Duration
2.34s

Pass rate

52%pass rate
Passed52%11
Failed29%6
Errors5%1
Skipped10%2
Timeouts5%1

Run timeline

Cumulative pass rate across tests in execution order

Top failing tests

  • test_echo_rejects_empty_payloadFAIL
    Tool 'echo' response mismatch
  • test_report_snapshot_stableFAIL
    Snapshot mismatch for 'generate_report'
  • test_no_secret_leak_in_logsFAIL
    Tool 'get_config' output matched secret pattern(s): aws_key
  • test_throughput_min_rpsFAIL
    Throughput 42.1 rps below minimum 50.0 rps (p99=312ms)
  • test_chaos_503_unavailableFAIL
    ChaosFaultError: simulated 503 Service Unavailable
  • test_survives_server_restartERROR
    ConnectionResetError: MCP server process exited with code 1
  • +2 more

Slowest tests

test_long_running_export
30000ms
test_echo_latency_p95_under_slo
1240ms
test_throughput_min_rps
980ms
test_injection_payloads_blocked
421ms
test_flaky_resource_read
340ms
test_path_traversal_blocked
210ms
test_tool_idempotent_calculate
156ms
test_chaos_delay_tolerance
145ms

Harness features in this run

Security payloads, performance SLOs, chaos, traces, schema validation, and more.

3Security tests
3Performance
6Resiliency / chaos
1MCP traces
1Flaky (retried)
1Schema checks
14Unique tags

Platform QA toolkit

Features exercised in this run — chaos, load, security, resiliency, regression, trace.

4
Chaos monkey
delay · 503 · truncate · drift
3
Load / throughput
assert_latency · assert_throughput
3
Security payloads
injection · traversal · secrets
5
Resiliency
degrades · reconnect · crash
2
Regression
assert_snapshot · idempotent
1
MCP trace
per-test JSON-RPC timeline

Chaos monkey

Protocol-aware faults on call_tool (@marker(tags=["chaos"], chaos_faults=[...])) — 3/4 survived.

delay503truncateschema_drift
TestFaults injectedOutcomeDuration
test_degrades_gracefully_on_bad_argsdelayPASS72ms
test_chaos_delay_tolerancedelayPASS145ms
test_chaos_503_unavailable503FAIL88ms
test_chaos_schema_drift_probeschema_driftPASS62ms

Load & performance SLOs

Suite duration percentiles plus assert_latency / assert_throughput / baseline checks.

P50
88.0ms
P95
30000.0ms
P99
30000.0ms

assert_throughput exercises concurrent call_tool with min_rps, max_p99_ms, and max_error_rate gates.

TestTagsOutcomeDurationSLO note
test_echo_latency_p95_under_sloperf, latencyPASS1240ms
test_throughput_min_rpsperf, throughputFAIL980ms
Throughput 42.1 rps below minimum 50.0 rps (p99=312ms)
test_long_running_exportperfTIMEOUT30000ms

Security findings & MCP-Bastion pairing

Test defenses in CI with the harness; enforce at runtime with MCP-Bastion.

CI gate (this report)

Installed mcp-bastion-python 2.0.0 · CI verifies · Bastion protects production

Export SARIF: mcp-test --sarif-output findings.sarif

Production (MCP-Bastion)

ThreatHarness verifiesBastion enforces
Prompt injectionassert_injection_blockedPromptGuard middleware
PII / secretsassert_no_secret_leakPresidio redaction
Auth boundariesassert_authorization_boundaryRBAC + tool policies
Rate limitsassert_throughput SLOsToken budgets & iteration caps
Path traversalassert_path_traversal_blockedPolicy-as-code rules

pip install mcp-bastion-python · PyPI

Unified test portal

Functional
44.4%
9 test(s) · fail
Performance
33.3%
3 test(s) · fail
Security
66.7%
3 test(s) · fail
Resiliency
66.7%
6 test(s) · fail

4 tools tested, 1 untested, 2 missing auth tests

MCP inventory coverage

Advertised vs exercised tools, resources, and prompts (coverage map).

Auth boundary tests: 2 of 4 exercised tools · 2 missing

Tools
4/5
✓ calculate✓ echo! export_report✓ get_config✓ search

1 untested

Resources
1/2
✓ file:///config.json! file:///health

1 untested

Prompts
0/1
! summarize

1 untested

Results by tag

security, perf, chaos, smoke — pass rate per marker category.

TagPassFailSkipPass rate
chaos310
75%
chaos:503010
0%
chaos:delay100
100%
chaos:schema_drift100
100%
functional221
40%
integration001
0%
latency100
100%
perf120
33%
regression110
50%
resiliency320
60%
security210
67%
smoke200
100%
snapshot010
0%
throughput010
0%

Diagnostics

Jump to flaky retries, schema drift, and per-test MCP trace timelines.

Flaky tests

Schema violations

MCP JSON-RPC traces

All Passed Failed Error Skipped Timeout
/ search · Esc clear · ? shortcuts
Showing all tests
Run window: 2026-07-06T20:10:00.000Z → 2026-07-06T20:10:02.340Z

Server & protocol

Protocol 2025-03-26

Capabilities tools, resources, logging, prompts

Environment

  • python_version 3.12.4
  • platform Windows-11-amd64
  • cwd C:/demo/mcp-server
  • server_command python -m my_mcp_server --transport stdio
  • transport stdio

Results by file

tests/test_performance.py (3)
TestTagsStartedStatusDurationDetails
test_echo_latency_p95_under_sloperf latency2026-07-06 20:10:01 UTCPASS1240.0 ms—
test_throughput_min_rpsperf throughput2026-07-06 20:10:01 UTCFAIL980.0 ms
Details
Throughput 42.1 rps below minimum 50.0 rps (p99=312ms)
test_long_running_exportperf2026-07-06 20:10:02 UTCTIMEOUT30000.0 ms
Details
Test exceeded timeout of 30000ms
tests/test_prompts.py (2)
TestTagsStartedStatusDurationDetails
test_prompts_when_availablefunctional2026-07-06 20:10:02 UTCSKIP0.0 ms
Details
server has no prompts capability
test_optional_bedrock_packintegration2026-07-06 20:10:02 UTCSKIP0.0 ms
Details
optional extra mcp-test-harness-bedrock not installed
tests/test_regression.py (2)
TestTagsStartedStatusDurationDetails
test_report_snapshot_stableregression snapshot2026-07-06 20:10:00 UTCFAIL89.0 ms
Details
Snapshot mismatch for 'generate_report'
--- expected
+++ actual
- {'version': '2.0.0'}
+ {'version': '2.0.1'}
test_tool_idempotent_calculateregression2026-07-06 20:10:00 UTCPASS156.3 ms—
tests/test_resiliency.py (6)
TestTagsStartedStatusDurationDetails
test_degrades_gracefully_on_bad_argsresiliency chaos2026-07-06 20:10:01 UTCPASS72.0 ms—
test_chaos_delay_tolerancechaos chaos:delay resiliency2026-07-06 20:10:01 UTCPASS145.0 ms—
test_chaos_503_unavailablechaos chaos:503 resiliency2026-07-06 20:10:01 UTCFAIL88.0 ms
Details
ChaosFaultError: simulated 503 Service Unavailable
test_chaos_schema_drift_probechaos chaos:schema_drift2026-07-06 20:10:01 UTCPASS62.0 ms—
test_survives_server_restartresiliency2026-07-06 20:10:01 UTCERROR18.0 ms
Details
ConnectionResetError: MCP server process exited with code 1
Traceback (most recent call last):
  File tests/test_resiliency.py, line 44
    await assert_tool_call(...)
test_flaky_resource_read flakyresiliency2026-07-06 20:10:01 UTCPASS340.0 ms
Details
Attempts:
  #1 timeout 5000.0ms read_resource timed out
  #2 error 120.0ms 503 from upstream
  #3 passed 88.0ms 
Flaky: passed after 2 retry(ies).
tests/test_schema.py (1)
TestTagsStartedStatusDurationDetails
test_schema_probe_strictfunctional2026-07-06 20:10:01 UTCFAIL44.0 ms
Details
Schema validation failed for tool 'search'
schema: $.arguments.limit — limit must be an integer (expected 'integer', got 'ten')
tests/test_security.py (3)
TestTagsStartedStatusDurationDetails
test_injection_payloads_blockedsecurity2026-07-06 20:10:00 UTCPASS420.7 ms—
test_no_secret_leak_in_logssecurity2026-07-06 20:10:00 UTCFAIL55.1 ms
Details
Tool 'get_config' output matched secret pattern(s): aws_key
test_path_traversal_blockedsecurity2026-07-06 20:10:00 UTCPASS210.4 ms—
tests/test_server.py (4)
TestTagsStartedStatusDurationDetails
test_echo_tool_returns_messagesmoke functional2026-07-06 20:10:01 UTCPASS12.4 ms—
test_tool_schema_matches_openapifunctional2026-07-06 20:10:00 UTCPASS48.2 ms—
test_list_tools_includes_echosmoke2026-07-06 20:10:00 UTCPASS6.8 ms—
test_echo_rejects_empty_payloadfunctional2026-07-06 20:10:00 UTCFAIL31.5 ms
Details
Tool 'echo' response mismatch
--- expected
+++ actual
@@ -1,3 +1,3 @@
 [{"text": "validation error", "isError": true}]
+[{"text": "ok", "isError": false}]

MCP trace (4 events)

Possible stdio pollution:

  • WARN: deprecated config key in stderr
0.8msrequesttools/call84 B
{'name': 'echo', 'arguments': {'message': 'ping'}}
3.2msresponsetools/call120 B
{'content': [{'type': 'text', 'text': 'ping'}]}
4.1msrequesttools/call92 B
{'name': 'echo', 'arguments': {'message': ''}}
5.0mserrortools/call64 B stdio pollution?
{'isError': True, 'message': 'message must be non-empty'}
Multiple formats. Maximum value.
Developer friendlyCI/CD ready Data drivenInsightful reports