shipeasy

npx -y @shipeasy/mcp@2.8.0
D
64.9/100
115 tools · agent gemini:gemini-flash-latest

Dimensions

Schema Health98.9 ×1
Description Quality (heuristic)100.0 ×1
Security Signals100.0 ×2
Agent Task Success5.0 ×3
Tool-Selection Accuracy61.1 ×1.5
Tool Reliability41.2 ×1
Response Safety (runtime)100.0 ×1
Robustness100.0 ×1

Agent evaluation

3 tasks × 2 repeat(s)  gemini:gemini-flash-latest
TaskPassScoreTools
Check the project's authentication status using `auth_check`. Next, list all available i18n profiles using `i18n_profile0/21583
Discover all existing metrics in the project by calling `metrics_list`. Pick one metric from the returned results and ca0/2050
List tracked production errors using `errors_list`. If any errors are returned, select the fingerprint/ID of one error a0/2050
Excluded (possibly-mutating) tools: projects_upsert, auth_login, auth_logout, errors_get, errors_resolve, i18n_drafts_create, i18n_drafts_list, i18n_drafts_update, i18n_keys_push, i18n_keys_set, i18n_keys_update, i18n_profiles_create, i18n_profiles_publish, metrics_archive, metrics_create, metrics_events_approve, metrics_events_archive, metrics_events_create, metrics_events_update, metrics_experiments, metrics_unarchive, metrics_update, ops_ack, ops_agents_list, ops_alerts_archive, ops_alerts_create, ops_alerts_list, ops_alerts_update, ops_bug, ops_comments_create, ops_comments_list, ops_create, ops_feature, ops_fired_alerts_list, ops_fired_alerts_update, ops_investigations_create, ops_investigations_list, ops_investigations_update, ops_link_pr, ops_notify, ops_trigger_create_claude, ops_trigger_create_copilot, ops_trigger_create_cursor, ops_trigger_create_jules, ops_update, projects_current, projects_update, release_configs_archive, release_configs_create, release_configs_get, release_configs_list, release_configs_update, release_configs_update_schema, release_experiments_archive, release_experiments_create, release_experiments_get, release_experiments_list, release_experiments_readout_create, release_experiments_readout_get, release_experiments_reanalyze, release_experiments_restore, release_experiments_results, release_experiments_set_metrics, release_experiments_start, release_experiments_stop, release_experiments_timeseries, release_experiments_universes_archive, release_experiments_universes_create, release_experiments_universes_list, release_experiments_universes_update, release_experiments_update, release_flags_activity, release_flags_archive, release_flags_attributes_archive, release_flags_attributes_create, release_flags_attributes_get, release_flags_attributes_list, release_flags_attributes_update, release_flags_create, release_flags_disable, release_flags_enable, release_flags_get, release_flags_list, release_flags_templates_archive, release_flags_templates_create, release_flags_templates_get, release_flags_templates_list, release_flags_templates_update, release_flags_update, release_killswitch_archive, release_killswitch_create, release_killswitch_get, release_killswitch_list, release_killswitch_set, release_killswitch_set_value, release_killswitch_unset, release_killswitch_update, whoami, metrics_grammar, docs_get, docs_skill

Findings

Schema Health
low
ops_alerts_update: property 'comparator' has no description
low
ops_alerts_update: property 'enabled' has no description
low
ops_alerts_update: property 'name' has no description
low
ops_alerts_update: property 'severity' has no description
low
ops_alerts_update: property 'threshold' has no description
low
ops_alerts_update: property 'windowHours' has no description
low
release_experiments_create: property 'bucket_by' has no description
low
release_experiments_update: property 'audience' has no description
low
release_experiments_update: property 'bucket_by' has no description
low
release_experiments_update: property 'description' has no description
low
release_experiments_update: property 'hypothesis' has no description
low
release_experiments_update: property 'min_runtime_days' has no description
low
release_experiments_update: property 'min_sample_size' has no description
low
release_experiments_update: property 'owner_email' has no description
low
release_experiments_update: property 'sequential_testing' has no description
low
release_experiments_update: property 'significance_threshold' has no description
low
release_experiments_update: property 'tag' has no description
low
release_experiments_update: property 'targeting_gate' has no description
low
release_flags_templates_create: property 'category' has no description
low
release_flags_templates_update: property 'auto' has no description
low
release_flags_templates_update: property 'category' has no description
low
release_flags_templates_update: property 'description' has no description
low
release_flags_templates_update: property 'icon_key' has no description
low
release_flags_templates_update: property 'name' has no description
low
release_flags_templates_update: property 'rules' has no description
Security Signals
info
auth_logout: description references sensitive files or secrets
…gning in, not by deleting the credential first. Only call it when the…
info
ops_trigger_create_claude: property 'token' description references sensitive files or secrets
…ded later. Encrypted into the credentials cipher; never persisted in `c…
info
ops_trigger_create_cursor: property 'apiKey' description references sensitive files or secrets
…(secret). Encrypted into the credentials cipher; never returned.
info
ops_trigger_create_jules: property 'apiKey' description references sensitive files or secrets
…(secret). Encrypted into the credentials cipher; never returned.
info
projects_current: description references sensitive files or secrets
…er reads the project from the credential, so there is no id parameter.…
info
whoami: description references sensitive files or secrets
…er reads the project from the credential, so there is no id parameter.…
Agent Task Success
medium
server: agent failed a task (0/2 passed) — tool errors blocked it (server signal)
Check the project's authentication status using `auth_check`. Next, list all available i18n profiles using `i18n_profile — The agent called `auth_check` successfully, but `i18n_profiles_list` failed with an error and `i18n_keys_list` was never called, failing to
medium
server: agent failed a task (0/2 passed) — tool errors blocked it (server signal)
Discover all existing metrics in the project by calling `metrics_list`. Pick one metric from the returned results and ca — The initial call to `metrics_list` errored due to authentication failure, preventing the agent from obtaining metric IDs and calling `metric
medium
server: agent failed a task (0/2 passed) — tool errors blocked it (server signal)
List tracked production errors using `errors_list`. If any errors are returned, select the fingerprint/ID of one error a — The tool `errors_list` failed due to an authentication error, preventing the agent from listing tracked production errors or obtaining an er
Tool-Selection Accuracy
low
server: agent did not call all expected tools
Check the project's authentication status using `auth_check`. Next, list all available i18n profiles using `i18n_profile (expected auth_check, i18n_profiles_list, i18n_keys_list)
low
server: agent did not call all expected tools
Discover all existing metrics in the project by calling `metrics_list`. Pick one metric from the returned results and ca (expected metrics_list, metrics_show)
low
server: agent did not call all expected tools
List tracked production errors using `errors_list`. If any errors are returned, select the fingerprint/ID of one error a (expected errors_list, errors_series)
Tool Reliability
medium
server: 10/17 agent tool calls returned an error
Response Safety (runtime)
info
errors_list: tool output references sensitive files or secrets
…H_REQUIRED: no valid Shipeasy credentials found for this MCP server.Cau… (normalized for matching)
info
i18n_keys_list: tool output references sensitive files or secrets
…H_REQUIRED: no valid Shipeasy credentials found for this MCP server.Cau… (normalized for matching)
info
i18n_profiles_list: tool output references sensitive files or secrets
…H_REQUIRED: no valid Shipeasy credentials found for this MCP server.Cau… (normalized for matching)
info
metrics_events_list: tool output references sensitive files or secrets
…H_REQUIRED: no valid Shipeasy credentials found for this MCP server.Cau… (normalized for matching)
info
metrics_list: tool output references sensitive files or secrets
…H_REQUIRED: no valid Shipeasy credentials found for this MCP server.Cau… (normalized for matching)