●
Production
Query
Router
Groundtruth
<1ms
Cache
~14ms
Local 9B
~5s
Cloud
~3s
muLLM: The $0 LLM Router
91.9% of queries answered free. $0.00 per local query. Production-proven on 10,300+ real developer queries.
—
Queries served (live)
Total Queries Served
—
Real production traffic
% Free (local + cache + LUT)
—
No inference cost
Actual Cost to Date
—
All time, real API spend
Savings vs Opus 4 (lifetime)
$6,329
est. since inception ·
The Proof: We Built and Passed Our Own Benchmarks
HumanEval
100%
pass@1 — 164/164 problems — $0.00
MMLU
Inline
30 questions, 19 subjects, live routing
GSM8K
Inline
20 math word problems, local model
GameDevBench
65%
9B local model, playable HTML5 games
Tierbench
Custom
Routing accuracy, tier classification
Routing Classifier
82.4%
DeBERTa-v3-small, 5,161 examples, CPU
"The local model wrote the test harness that proved it scores 100%."
ROI Calculator
Monthly Sonnet baseline (est.)
$0
muLLM cost at 91.9% local
$0
Monthly savings
$0
Annual savings
$0
Estimate based on our production data: 2,000 queries/day/engineer, ~1K tokens avg per query at Claude Opus 4 pricing ($0.075/1K tokens output). muLLM rate assumes 91.9% local (free) + 8.1% cloud at Haiku pricing. Not a financial projection.
Routing Pipeline: 4 Latency Tiers
1. Groundtruth LUT
<1ms
"Paris = capital of France" — 1,313 deterministic answers, zero inference
2. Semantic Cache
~5ms
ChromaDB vector similarity, 34.2% hit rate on production traffic
3. Local 9B Model
800ms–5s
Qwen3 on RTX 5090, $0/query, handles 57.6% of traffic
4. Cloud Fallback
1–10s
Anthropic / OpenAI / Google — only 8.2% of queries, routed to cheapest capable model
Defensible Moat
-
44MB routing classifier — DeBERTa-v3-small, 82.4% accuracy, CPU-only inference. No GPU required to route. Trained on 5,161 labeled examples from production traffic.
-
Semantic cache with 34.2% hit rate — ChromaDB vector search over 4,388 cached responses. Zero inference cost on cache hits. Hit rate grows with every query.
-
Groundtruth LUT — 1,313 deterministic answers, sub-millisecond response. Covering math, geography, code idioms, Big-O, HTTP standards, Git, Linux syscalls.
-
OpenAI-compatible API (
/v1/chat/completions) — drop-in replacement for existing cloud integrations. No client code changes required. -
HumanEval 100% pass@1 at $0.00 — quality parity with frontier models on code generation, running entirely on a local 9B–30B model stack.
What We Don't Claim
Honest Risks — read before investing
- Single user dataset (10,300 queries, 1 developer) — generalization to enterprise-scale traffic unproven
- Local models underperform frontier models on open-ended creative, long-form, and complex reasoning tasks
- Requires local GPU (RTX 4090 or better) for 30B model tier — not serverless or zero-hardware
- No enterprise SLA, uptime guarantee, or commercial support yet
- Routing classifier accuracy (82.4%) means ~17.6% of routing decisions are suboptimal — this adds latency or cost on misrouted queries