muLLM classifies every query in microseconds and sends it to local inference, semantic cache, or the cheapest cloud model that can handle it. 88% of queries cost nothing.
Every query cascades through cost tiers, stopping the moment a tier can answer confidently.
Groundtruth pattern matching answers deterministic questions in under 1ms — zero AI, zero cost. A local 30B model handles everything else that doesn't need the cloud. No round-trip. No API key consumed.
ChromaDB semantic cache finds near-duplicate queries with cosine similarity. 70% hit rate on organic usage. The cache learns your usage patterns — it gets smarter every day without any configuration.
When local and cache can't answer, the router picks the cheapest cloud model capable of the task — Gemini Flash for simple reasoning, GPT-4o only for genuinely hard work. No vendor lock-in.
Every query passes through a deterministic cascade — fastest tier first, cloud tier last.
12.7% of all queries. Pattern-matched against a curated fact database — HTTP codes, git commands, Big-O complexity, math, CUDA setup, POSIX signals.
<1msChromaDB vector search against all prior responses. 70% hit rate on organic usage. Threshold: 0.98 cosine similarity.
~8msQwen-3.5 9B on RTX 5090. 100% HumanEval pass@1. Handles code, analysis, writing, math — everything that doesn't require live data.
200–800msDeBERTa classifier (82.4% accuracy, 44MB CPU) selects the cheapest model: Gemini Flash 1.5 first, GPT-4o only for complex multi-step tasks.
800ms–4sEvery result is reproducible. Local model, local hardware, real benchmark datasets.
| Benchmark | Score | Cost | Tier | Notes |
|---|---|---|---|---|
| HumanEval pass@1 | 100% | $0.00 | local | 164/164 tasks, Qwen 9B, no cloud |
| MultiPL-E (6 languages) | 98.3% | $0.00 | local | JS, Python, Go, Java, Rust, C++ |
| Router Accuracy | 82.4% | $0.00 | classifier | DeBERTa-v3-small, 44MB, CPU-only |
| Routing Latency (groundtruth) | 23µs | $0.00 | tier 0 | vs 200–800ms cloud RTT |
| Semantic Cache Hit Rate | 70% | $0.00 | cache | Organic usage; 30% on novel queries |
| Production Cost Reduction | 97.6% | — | observed | $10.86 actual vs $449 estimated direct-to-cloud |
Full results → RouterBench · MegaBench · Accuracy Report
The local AI tier needs a GPU. But two of the four routing tiers are CPU-only, and smart cloud routing saves money on any hardware.
Groundtruth answers common questions in under 1ms without touching any model. The semantic cache returns prior answers for free. And when you do reach cloud, muLLM picks Gemini Flash ($0.075/M tokens) over GPT-4o ($5/M) whenever the task allows it.
12.7% of queries answered by regex pattern matching. HTTP status codes, git commands, Big-O notation — zero AI, <1ms, zero cost. Works on any hardware.
ChromaDB runs on CPU. Every cached query is served free forever. Warm up the cache once, save money on every repeat query permanently.
When cloud is needed, the DeBERTa classifier picks the cheapest sufficient model. Simple tasks go to Gemini Flash — 40–60% cheaper than GPT-4o on those workloads.
When 95%+ of queries cost $0.00 to serve locally or from cache, flat-rate pricing creates margins that per-token cloud billing cannot match.
The arbitrage: charge a flat $0.01/query while serving 88% of requests at $0.00. The remaining 12% cloud queries cost ~$0.002 each on average. A million daily queries at these economics is sustainable infrastructure, not a loss leader.
muLLMRouter introduces the CAA (Cost-Adjusted Accuracy) metric and a human-oracle routing dataset. Explore the full methodology behind the routing classifier and benchmark results.
View RouterBench →muLLM is running live. Send a query and watch the routing decision — which tier handled it, how long it took, and what it cost. Live routing archaeology available.
Open Chat →