Why muLLM?

Performance comparison

Key technical stats

Inference Backend Comparison — RTX 5090 · qwen3-coder

muLLM is a routing layer, not an inference engine. Tok/s is determined by the backend you choose. muLLM adds ≤1.8ms routing overhead per query — but routes 93% of queries to groundtruth or cache at $0.00, so average effective latency is lower than running any backend alone.

¹ Average TTFT when muLLM routing is considered: 93% of queries resolved in <2ms (groundtruth/cache), 7% hit the local backend. Weighted average: 0.93×1.76ms + 0.07×1,750ms = 124ms. Routing overhead from live bench data (p50=1.76ms, p95=48ms).
  ² Model A measured live: Ollama 229.8/230.9/230.1 tok/s · llama.cpp 295.0/303.4/303.3 tok/s · LiteLLM→llama.cpp 293.9/296.5/297.4 tok/s (3 runs each). Model: qwen3-coder-30b Q4_K_M (MoE, 3B active) on RTX 5090 32GB VRAM.
  ³ Model B measured live: llama.cpp 69.6/69.6/70.0 tok/s · LiteLLM→llama.cpp 69.6/69.7/69.7 tok/s · Ollama 60.5/61.3/61.2 tok/s (3 runs each). Model: Qwen3.6-27B-Q4_K_M on RTX 5090. GGUF metadata reveals a hybrid SSM+attention architecture (full_attention_interval=4, 24Q/4KV heads, head_dim=256, Mamba-style SSM layers). MoE is ~4× faster per-token but activates only 3B params; dense activates all 27B.
  ⁴ vLLM GGUF loader cannot map MoE expert weight keys (mlp.experts.gate_up_proj) for qwen3moe architecture. HF-format MoE model required. vLLM works for standard dense architectures (see Model C, ⁶).
  ⁵ Qwen3.6-27B (GGUF arch string: qwen35) is a hybrid SSM+attention model with Mamba-style state-space layers every 4th layer. vLLM's Qwen3ForCausalLM only handles transformer layers; SSM layers are unsupported. llama.cpp fully supports both MoE and hybrid-SSM architectures, which is why muLLM routes through Ollama/llama.cpp for these cutting-edge models.
  ⁶ Model C measured live on RTX 5090 32GB VRAM: Ollama (llama.cpp) 316.8/313.1/314.5 tok/s avg=315 tok/s · vLLM 0.10.1.1 163.2/163.7/162.0 tok/s avg=163 tok/s (3 runs each, 512 completion tokens, qwen3 GGUF Q4_K_M, vLLM with enforce-eager). Single-request Ollama/llama.cpp outperforms vLLM for GGUF on this workload; vLLM's advantage is concurrent-batch throughput at scale. muLLM routes to the fastest backend per request.

Connecting your backend