inferlint

inferlint measures what your inference server actually did—and flags what it didn't tell you.

For vLLM, SGLang and TensorRT-LLM. Around a benchmark it reads the server's boot log and counters, measures achieved concurrency, preemptions, KV-cache use, GPU state and timing, and checks the run for 15 known problems. It tells the engine from what it reads, and works with the load tool you already use.

Requests over a 93-second run: the load tool kept 32 open, the server never ran more than 10 at once, the rest waited, and 22 preemptions happened with no log line. — vLLM 0.28 on an RTX 4090, a 27B model, 32 requests at once, 1,000 output tokens each. The same run on vLLM 0.29 ran at most 10 at once with 21 preemptions; on 0.30, 11 at once with 22.

10 of 32requests ran at once; the benchmark said 32 (T8)
22preemptions, and no line about them in the server log (T1)
10 → 7the most that fit, falling as answers grew (T9)

One command around your benchmark

# install
pipx install inferlint

# vLLM: start the server, record it while the benchmark runs, check it, stop it
inferlint xray -o run/ --serve "vllm serve MODEL" -- vllm bench serve --model MODEL --max-concurrency 32 ...

# SGLang: its metrics need --enable-metrics
inferlint xray -o run/ --serve "sglang serve --model-path MODEL --enable-metrics"     -- python -m sglang.benchmark.serving --model MODEL --max-concurrency 32 ...

# TensorRT-LLM: its metrics need return_perf_metrics: true and enable_iter_perf_stats: true in the --config file
inferlint xray -o run/ --serve "trtllm-serve serve MODEL --config llm_api.yaml"     -- python -m tensorrt_llm.serve.scripts.benchmark_serving --model MODEL --max-concurrency 32 ...

# a team on one engine can name it once
export INFERLINT_ENGINE=sglang

== is the GPU free?
[ T2] PASS  2 consecutive clear readings
== start the server (output in run/boot.log)
== boot log
[T12] PASS  no failure in log
[ T5] WARN  attention block size forced to 784 tokens (requested 16); each request's KV is allocated in 784-token units
[ T6] PASS  nothing to check: the server was started without --attention-backend, so vLLM chose its own (FLASH_ATTN) and nothing could be ignored; there was no draft model either
== probe: one request, then 8 at once
[ T7] PASS  1 + 8 concurrent requests served, server alive
== load: vllm bench serve --model MODEL ... --max-concurrency 32 --save-result --result-dir run --result-filename bench.json
… the benchmark's own output
== checks
[ T1] TRIPWIRE-FAILED  21 preemptions during the run
[T10] PASS  314.99 tokens/s over 101.590s; an integer-second timer would say 313.73 (-0.40%)
[T14] PASS  gauge denominator is 42 = num_gpu_blocks (43) - 1 null block
[ T9] WARN  ceiling moved between 5 and 10 as requests grew (one request held 9.5% to 19.0% of the cache); concurrency was not a constant of this run
[ T8] TRIPWIRE-FAILED  requested 32 concurrent, server never ran more than 10
[T15] PASS  client and server agree: 32 requests, 32,000 output tokens
== stop the server
[ T2] PASS  2 consecutive clear readings
12 checks (8 pass, 2 warn, 2 tripwire-failed)
report: run/report.html
files:  run

The output of the vLLM command on vLLM 0.28, from a second run of the test in the chart, with paths and the load command shortened. Tested live on vLLM 0.28, 0.29 and 0.30, SGLang 0.5.20 and TensorRT-LLM 1.3.0rc29, on an RTX 4090. No GPU at hand? The tutorial runs every check on the recorded files in two minutes.