The leaderboards are withheld

Deliberately, until the scoring model stops moving.

mcp-gauntlet publishes scores about other people's software. Two boards used to live here — a set of reference servers, and a survey of 50 servers drawn from the official MCP registry. Both are down.

Why

Inside three days the harness was found to have graded servers on its own defects three separate times, and those servers were named publicly while it did so:

• Task generation invented filesystem paths that did not exist, so every call failed and the server wore it. Two servers published as D and C were in fact A.
• Twenty-five security findings about "references to sensitive files or secrets" were all false positives — among them a network-forensics server marked down for the phrase data exfiltration, which is its entire subject.
• The survey pinned each package to the version the registry recorded. That version lags badly — the registry said 1.7.1 where npm shipped 1.30.0 — so two servers were published as failures for code their maintainers had long since fixed.

Each was found, fixed and written up. But the boards were published from runs predating some of those fixes, and METHODOLOGY.md promises that any server appearing with a low score gets advance notice and a link to dispute it. That notice was never sent. Taking the boards down is the only way to honour both commitments at once.

The part that can still be checked

None of this bears on whether the harness is useful — it bears on how young its scoring model is. It ships a deliberately malicious demo server whose every tool description is innocuous, and catches five attacks in it that a description scanner cannot see. That runs locally, in one command, with no API key and no third party involved:

uvx mcp-gauntlet run "python -m mcp_gauntlet.fixtures.malicious_server" --no-agentic

The write-up

All eight of those failures, with the reasoning and the fixes: Eight times my evaluator measured something other than the server. The pattern turned out to be more useful than any individual fix.

What brings them back

A v1.0 whose scoring has held steady across releases, provenance recorded on every row, and the disclosure process exercised before anyone is named. The data is kept in the repository under boards-withheld/ rather than deleted, so the corrections stay auditable.