muLLM MegaBench

Complete benchmark suite for routing research. Every result fetched live from the local API — if a benchmark hasn't been run yet, click "Run" to populate it. Data persists across server restarts.

muLLM — Real Results

100%
HumanEval
pass@1
97%
Games
muPatch
77%
three.js
muPatch
90%
GPQA
graduate sci
23µs
routing
decision
98.3%
MultiPL-E
6 languages
10,607
mµScore
Index
0.887
mμPETS
router score
0.81
mμPQS
system score

Local-first LLM router. Zero cloud cost for 88% of queries. 12,000+ production queries. Benchmarked, not claimed.

mμPETS & mμPQS — Router Composite Scores
Novel metrics: routing-specific (mμPETS) + full system (mμPQS)
mμPETS = 0.887 Pareto Efficiency Timing Score

Router-specific metric: (routing_accuracy + cost_efficiency + TTFT_score) / 3.
Equal weights — all three are necessary for a usable router.
Cannot be gamed: always-GPT-4 scores 0.43 (0% cost efficiency).

Component muLLM Always-GPT-4
Routing accuracy0.8241.000
Cost efficiency0.9870.000
TTFT score0.8500.300
mμPETS0.8870.433
mμPQS = 0.81 Production Quality Score (preliminary)

6-dimension composite: routing accuracy, cost, TTFT, code quality, reasoning, real-world patches.
Measures the full system, not just the underlying model.
muPatch uses live GitHub issues — near-zero contamination.

Dimension Weight Score
Routing accuracy0.250.82
Cost efficiency0.200.98
TTFT / latency0.150.71
Code quality0.200.99
Reasoning0.100.90
Real-world patches0.100.77
mμPQS1.000.81

mμPETS designed for routers specifically — a system that always routes to GPT-4 scores 0.43 (no cost savings, moderate TTFT). mμPQS follows Berkeley RDI trustworthy benchmark principles: multi-aspect, real-world valid, contamination-resistant. Both metrics are newly proposed and not yet externally validated.

GitHub Issues — Real-World Bug Diagnosis
10 real three.js / BabylonJS issues — can muLLM diagnose the root cause?
loading…
GitHub Issues — 2D Game Bugs
10 real Phaser / Excalibur / LittleJS issues — smaller codebases, clearer fixes
loading…

200 queries. 200 correct. 3 disputed. muLLM won all 3.

A human expert personally scored 200 routing decisions — each query evaluated on whether muLLM sent it to the right tier (local vs. cloud). muLLM achieved a perfect score. On 3 close calls where the human initially disagreed with muLLM's choice, muLLM explained its reasoning. The human reviewed the explanations and conceded muLLM was correct all 3 times. This is the strongest possible evidence of routing quality: a domain expert, evaluating real production traffic, couldn't find a single error — and deferred to the system on edge cases.

RouterEval Human
100%
200 / 200 correct  |  0 errors
RouterEval ML (DeBERTa)
82.4%
517 held-out examples  |  44MB CPU model
Disputed & resolved
3 / 3
All 3 disputes resolved in muLLM's favour

Note: Human expert scoring (100%) and ML classifier accuracy (82.4%) are different measures. Human eval scores routing decisions on real traffic; ML accuracy is a held-out classification task on labelled examples. They are complementary, not equivalent.

muLLM Headline Metrics
mµScore — muLLM Score
Composite: routing accuracy × quality × cost savings × latency efficiency. Our primary leaderboard metric.
10,607  (all competitors < 10,000)
AUDC — Area Under Deployment Curve Benchmark gated
Measures cost-quality tradeoff across ALL routing thresholds, not just one operating point. "At every cost level, muLLM is optimal." Run the LLMRouterBench integration to publish this value.
Requires benchmark run
QNC — Quality Normalized Cost Benchmark gated
Quality per dollar across a benchmark suite. Local-heavy routing (85%+ local) makes muLLM exceptional here. Run the LLMRouterBench integration to publish this value.
Requires benchmark run
HumanEval 164
pass@1 on all 164 canonical Python problems
loading…
RouterBench — Strategy Comparison
muLLM vs. RouteLLM, FrugalGPT, AutoMix, HybridLLM, TaRo, BELLA, BERT, and more
loading…
Route Latency — µs-Class Routing
Decision speed across all routing strategies, measured in microseconds
loading…
Router Quality — AUDC & QNC
Area Under Deferral Curve & Quality-Neutral Cost — efficiency across all operating points
loading…