LiteInfer Benchmark Dashboard

6 run(s) in history  ·  6 engine(s) compared  ·  green = best per column red = worst per column  ·  x ratio is vs first (baseline) engine

throughput

All requests submitted at once — engine queues them, processes B=1 at a time. E2E includes queue wait.

📅 2026-05-09T22:46:53native-eager-kv-cache🤖 Llama-3.2-1B-Instruct32 prompts · all submitted at once
EngineBreq/stok/sE2E p50E2E p99
liteinfer11.75base72.66base11579.4 msbase18235.5 msbase
liteinfer-kvcache11.821.03x72.561.00x10167.2 ms1.14x17625.8 ms1.03x
liteinfer-native-kvcache11.821.04x72.661.00x10167.9 ms1.14x17602.9 ms1.04x
liteinfer-b444.402.51x180.532.48x4101.0 ms2.82x7278.4 ms2.51x
vllm12.891.64x180.002.48x5792.6 ms2.00x11045.3 ms1.65x
vllm-b4410.796.15x656.689.04x1623.6 ms7.13x2928.5 ms6.23x

latency

Sequential, no queue — each request sent only after previous finishes. Pure per-request engine latency.

📅 2026-05-09T22:49:32native-eager-kv-cache🤖 Llama-3.2-1B-Instruct20 prompts · sequential, no queue
EngineBTTFT p50TTFT p99E2E p50tok/s
liteinfer113.7 msbase15.7 msbase1732.8 msbase71.57base
liteinfer-kvcache115.3 ms0.89x16.5 ms0.95x1550.3 ms1.12x72.211.01x
liteinfer-native-kvcache114.2 ms0.97x15.2 ms1.03x1547.6 ms1.12x72.321.01x
vllm126.1 ms0.53x27.6 ms0.57x693.3 ms2.50x182.522.55x