vLLM 0.28 on an RTX 4090

The 15 tripwires

Each one is a way an inference benchmark can report a wrong number without raising an error.

1. T1Silent preemptionsilent-preemption

   When the KV cache has no free block for a running request, vLLM preempts a request: it frees that request's KV blocks and later recomputes them from the prompt and the output so far. vLLM increments vllm:num_preemptions_total and writes no log line at the default log level (verified on vLLM 0.28, 0.29 and 0.30).

   A preempted request waits in the queue again and repeats work, so latency and throughput include recomputation the client never sees. Searching the server log finds nothing.

   inferlint preemption before.json after.json
2. T2Orphaned engine processorphaned-engine

   pkill -f "vllm serve" matches only the API-server process. The engine process, which holds the weights and the KV cache in GPU memory, renames itself VLLM::EngineCore and can outlive its parent when the parent is killed.

   The next server starts in whatever GPU memory is left and sizes a smaller KV pool, with no error. One low memory reading is not proof the GPU is free, so inferlint waits for two in a row.

   inferlint teardown | inferlint gpu-inspect
3. T3Cleanup kills its own shellcleanup-self-kill

   pkill -f "vllm serve" inside a bash -c one-liner also matches that shell, because the shell's own command line contains the pattern.

   The shell is killed before the rest of the cleanup runs, and the script ends without an error message.

   inferlint teardown
4. T4KV pool size jumps between startspool-size-jump

   vLLM sizes the KV pool at each start from the GPU memory left after loading the model and capturing CUDA graphs, and the pool comes in discrete steps. Starts with identical flags can land on different steps; T11 is the cause.

   Two runs with the same flags can differ by a whole step (9.4% in the recorded example), which reads as a performance difference and is not one.

   inferlint check-log boot1.log boot2.log
5. T5Large KV block sizelarge-kv-blocks

   For hybrid attention/Mamba models, vLLM raises the attention block size (the KV cache allocation unit) from 16 tokens to hundreds, so that attention pages and Mamba state pages line up.

   Every request holds whole blocks, so a short request occupies a full block and far fewer requests fit than the pool's token count suggests.

   inferlint check-log boot.log
6. T6Requested attention backend ignoredbackend-ignored

   --attention-backend sets the target model's attention backend. With speculative decoding, the draft model selects its own backend from a candidate list and does not apply the flag.

   In the recorded case Triton was requested and logged for the target model, but the draft model selected FlashInfer, which failed on the first request.

   inferlint check-log boot.log
7. T7Crash after readycrash-after-ready

   A server can log 'Application startup complete' and then fail on its first request, when a code path start-up never ran is reached (for example a kernel compiled on first use).

   A gate that only waits for readiness passes a server that cannot serve. inferlint probe sends real requests; inferlint check-log reports T7 when the log shows a failure after ready.

   inferlint probe http://127.0.0.1:8000
8. T8Concurrency not reachedconcurrency-not-reached

   A load tool's concurrency is the number of requests it keeps open. vllm:num_requests_running is the number the scheduler actually runs; the rest wait in its queue. Admission-control limits (--max-num-queued-reqs, --max-num-queued-tokens, vLLM 0.29 and later) can also reject requests.

   A result labelled 'concurrency 32' can describe a server that never ran more than 10 requests at once.

   inferlint series run.series.jsonl --requested 32
9. T9Concurrency ceiling falls as sequences growfalling-ceiling

   With the KV cache full, the number of requests that fit is floor(1 / share), where share is the fraction of the cache one running request holds. The share grows with sequence length, so the ceiling falls during a run, and each fall preempts requests (T1).

   A run has no single maximum concurrency, so one figure for it describes none of the run.

   inferlint series run.series.jsonl
10. T10Integer-second timinginteger-second-timer

    Timing a run with whole seconds, such as $(date +%s), is off by up to one second at each end.

    On a 30-second run that is about 3%, enough to reverse a close comparison. inferlint uses the sub-second clocks recorded with each snapshot.

    inferlint rate before.json after.json
11. T11KV memory budget drifts between startskv-memory-drift

    The 'Available KV cache memory' vLLM reports at start-up can differ between starts with identical flags, mostly because CUDA-graph capture memory varies.

    A small drift can move the KV pool to a different step (T4). Record the per-start figure with every result.

    inferlint check-log boot1.log boot2.log
12. T12Start-up failure without a causeunexplained-failure

    When a server fails to start, a search of the log for 'Error' usually finds a wrapper line rather than the root cause, and the failure is recorded with no reason.

    A failure without its cause cannot be fixed, reproduced or compared.

    inferlint check-log boot.log
13. T13Stale waiter adopts the next serverstale-waiter

    A test script stopped while it waits for its server to become ready can leave its wait loop running. That loop sees the next script's server come up, reports it as its own, and later shuts it down.

    The next test dies partway through with no visible cause.

    see docs/orchestration.md
14. T14Reserved null blocknull-block

    vLLM exports num_gpu_blocks = N but reserves one block (the null block), and vllm:kv_cache_usage_perc counts out of N - 1 (verified on vLLM 0.28, 0.29 and 0.30).

    Capacity computed from the exported block count is one block too high.

    inferlint series run.series.jsonl --snapshot after.json
15. T15Client and server counts disagreeclient-server-mismatch

    The load tool's own counts for the run (requests completed, output tokens) are compared with the server's counters between the before and after snapshots.

    If the server did more work than the load tool sent, other traffic shared the server and the result is contaminated. If it did less, the load tool counted work the server did not do.

    inferlint xray -o results/ -- vllm bench serve ...