Benchmarks

Nexus-MCP is a local code-intelligence MCP server. These are live runs of Claude Code, Claude Sonnet, headless (claude -p) with Nexus-MCP against an agent with only Grep, Glob and Read. Updated 2026-10-28, version 2.1.0.

Live agent runs

Project	Files	Cost: Grep and Read	Cost: Nexus-MCP	Change	Turns	Correct (Grep / Nexus)	Commit, date
Private Python project	460	$0.571	$0.446	−22%	55 → 35	36 / 35 of 36	`a67e5b4`, 2026-10-28
Django	3,000	$0.894	$0.863	−3%	93 → 71	34 / 36 of 36	`a67e5b4`, 2026-10-28

12 questions per project, 3 runs each. The same code can vary by about 20% in cost with 3 runs per question (measured 2026-10-28). The saving is large on code the model does not know and small on code it knows from training.

Search accuracy (no LLM)

Retrieval suite	hit@1
`jobscout`	0.81
`nexus_mcp`	0.80
`shop_repo`	0.50
`flask`	0.64

Quick tier: `nexus-mcp-ci eval` (no LLM)

The free offline check: about 30 to 40 questions made from each project, with the right file from the parser. Tokens are a proxy (characters divided by 4). Chromium base/ took 25 minutes to index on a laptop; the Nexus-MCP source takes about 30 seconds (--example).

Project	Files	Questions	Verdict	Tokens: Grep and Read	Tokens: Nexus-MCP	Right file first	Top 3	Date
Chromium base/	3,638	40	Strong fit	17,998	950	40% → 70%	58% → 90%	2026-11-03
pallets/click	91	30	Strong fit	24,774	1,301	27% → 97%	63% → 100%	2026-11-03
Nexus-MCP source	59	30	Likely fit	5,385	1,234	70% → 93%	97% → 97%	2026-11-03

Head to head: Nexus-MCP and GitNexus

Chromium base/ (C++, 4,243 files, commit d8ebdd2): 8 tasks, 3 runs each, Claude Code 2.1.280, Claude Sonnet, headless (claude -p), 2026-11-04. Each total is the median over tasks of each task's median of 3 runs.

Condition	Version	Tokens (median)	Cost per task (median)	Cost of all runs	Turns	Correct
Grep and Read	built-in tools	62,450	$0.057	$1.58	5	24 of 24
Nexus-MCP	2.2.0 with the C++ parser fixes of 2.2.1	82,900	$0.062	$1.64	6	24 of 24
GitNexus	1.6.12	164,192	$0.085	$2.13	6	24 of 24

Task	Category	Grep and Read	Nexus-MCP	GitNexus
`cr-concept-weakptr-invalidate`	conceptual	50,057 · $0.077	82,568 · $0.069	110,006 · $0.088
`cr-impact-atexit-registercallback`	impact	58,666 · $0.040	72,676 · $0.046	212,910 · $0.090
`cr-impact-histogram-bucket-ranges`	impact	256,305 · $0.164	267,985 · $0.175	374,064 · $0.173
`cr-concept-json-reader-backend`	conceptual	56,211 · $0.037	109,361 · $0.055	180,053 · $0.076
`cr-needle-json-max-depth`	needle	66,234 · $0.029	49,847 · $0.022	97,562 · $0.028
`cr-needle-utf8-to-utf16`	needle	52,976 · $0.028	32,710 · $0.018	103,939 · $0.045
`cr-arch-task-scheduling`	architecture	132,197 · $0.083	111,439 · $0.090	191,516 · $0.099
`cr-concept-timer-hierarchy`	conceptual	71,500 · $0.074	83,231 · $0.082	148,332 · $0.082

All three conditions answered all 24 runs correctly.
Nexus-MCP used fewer tokens than GitNexus on 8 of 8 tasks and cost less on 6; GitNexus cost $0.002 less on one task, and one task was a tie.
Grep and Read cost less than both on this C++ code. Nexus-MCP cost less than Grep and Read on 3 of 8 tasks (the two needle tasks and WeakPtr).
GitNexus sends about 17.9k tokens of tool schema on each turn (17 tools); Nexus-MCP sends about 2.3k (8 tools).
The GitNexus condition gets its MCP server and the CLAUDE.md block that gitnexus analyze writes; its on-demand skills are not given.

How the comparison was kept fair

Same agent, model, prompt, budget and timeout in each condition.
Read, Grep and Glob in each condition; a tool server is added to them, not in place of them.
Each tool gets what its real user always has loaded: its MCP server and its always-loaded instructions.
Indexes built before the runs; a warm-up call so that each run starts with a warm prompt cache.
Isolated config folder and --strict-mcp-config for each run.
Pinned repo commit and tool versions; ground truth checked with grep against the commit.
The same mechanical scoring (file recall and required facts) for each condition; every task is published.

The tasks are in benchmarks/tasks/chromium-base.yaml. The method, and how to run the benchmark on any repo, are in benchmarks/README.md.

Community results

Runs shared with nexus-mcp-ci eval share and accepted by a maintainer. Numbers only: no code and no private names.

None yet. Yours can be the first.

Run it on your own code

uvx nexus-mcp-ci eval runs the free offline check on the current folder. uvx nexus-mcp-ci eval --agent claude runs the paid A/B with your own Claude Code login, after it shows the cost.