LiteInfer Benchmark Dashboard

8 run(s) in history  ·  7 engine(s) compared  ·  green = best per column red = worst per column  ·  x ratio is vs first (baseline) engine

throughput

All requests submitted at once — engine queues them, processes B=1 at a time. E2E includes queue wait.

📅 2026-05-10T17:08:24paged-kv-cache🤖 Llama-3.2-1B-Instruct32 prompts · all submitted at once
EngineBreq/stok/sE2E p50E2E p99
liteinfer11.75base72.61base11607.0 msbase18248.8 msbase
liteinfer-kvcache11.811.03x72.471.00x10190.2 ms1.14x17647.9 ms1.03x
liteinfer-native-kvcache11.811.03x72.160.99x10232.8 ms1.13x17724.8 ms1.03x
liteinfer-paged-kvcache11.570.89x62.700.86x11771.8 ms0.99x20397.1 ms0.89x
liteinfer-b444.312.46x177.062.44x4220.3 ms2.75x7420.9 ms2.46x
vllm12.901.65x180.712.49x5795.9 ms2.00x10998.8 ms1.66x
vllm-b4410.776.14x655.899.03x1625.3 ms7.14x2931.8 ms6.22x

latency

Sequential, no queue — each request sent only after previous finishes. Pure per-request engine latency.

📅 2026-05-10T17:12:40paged-kv-cache🤖 Llama-3.2-1B-Instruct20 prompts · sequential, no queue
EngineBTTFT p50TTFT p99E2E p50tok/s
liteinfer113.7 msbase15.4 msbase1710.4 msbase72.37base
liteinfer-kvcache114.6 ms0.94x16.8 ms0.92x1540.8 ms1.11x72.641.00x
liteinfer-native-kvcache113.9 ms0.99x15.6 ms0.99x1539.5 ms1.11x72.651.00x
liteinfer-paged-kvcache114.7 ms0.94x16.7 ms0.92x1829.0 ms0.94x61.240.85x
vllm125.9 ms0.53x28.8 ms0.54x693.1 ms2.47x182.432.52x