inferlint

vLLM 0.28 on an RTX 4090

vLLM 0.28.0 · Qwen3.8-27B-W4A16-AutoRound · 26,093-token KV pool · concurrency asked for: 32

What happened

The load tool kept 32 requests open for 1 min 34 s. The server finished 32 requests with no errors and generated 32,000 tokens. It never worked on more than 10 at once. The rest waited in a queue, 29 s on average before a request started. It paused a running request to make room 22 times (preemption); each one waited, then rebuilt its notes before carrying on. Its log does not mention this.

Ran at onceFail

10 of 32

The load tool kept 32 requests open.

PreemptionsFail

Requests evicted and recomputed. The server log mentions none.

Output tokens per second

341.4

over 93.7 s; a whole-second timer reads +0.77%

Usable KV blocks

43 reported, one held back; 784 tokens per block

The test

Measured time: 1 min 34 s (93.7 s), from the before to the after reading
Clock time: 2026-09-26 17:50:52 to 17:52:25 (UTC-7)
Server start-up: 2 min 4 s, before the test (from the boot log)
Requests finished: 32: 32 reached their token limit, no errors
Output tokens generated: 32,000 (1,000 per request)
Prompt tokens read: 2,819 (88 per request)
Average time per request: 54.4 s
Average wait for the first token: 29.6 s, 29.3 s of it waiting in the queue

The setup

Concurrency asked for: 32 requests at once
Requests sent by: the load tool. inferlint sends none: it only reads the server's counters.
Recording: the server read every 0.25 s, 365 readings
Server address: http://127.0.0.1:8000
Made from: boot.log, before.snapshot.json, after.snapshot.json, run.series.jsonl
Tripwires checked: 8 of 14. Not here: T2 and T3 (run at clean-up), T4 and T11 (need two or more boot logs), T7 (needs a live server), T13 (a rule for test scripts)

Words used in this report

Token: A piece of text the model reads or writes, about three-quarters of a word on average in English. Speed, memory and cost are all counted in tokens.
Prompt and output tokens: Prompt tokens are the text sent to the model. Output tokens are the text it writes back.
Request: One question sent to the server, and its answer.
Concurrency: How many requests are in progress at the same moment. 'Asked for' is how many the load tool kept open. 'Running' is how many the server was actually working on. The rest wait in a queue.
Waiting (queue): Requests the server has received but not started, because there is no room for them in the KV cache yet.
KV cache: GPU memory where the model keeps its working notes for every request in progress, so it does not have to reread the whole text for each new token. A request's notes grow as its answer gets longer.
KV pool: The size of the KV cache, in tokens, which vLLM sets aside when it starts. It is the GPU memory left after the model and its working buffers are loaded, so it can differ from one start to the next (T4, T11).
KV block: The pool is handed out in fixed-size blocks, like parking spaces. A request always takes whole blocks, so even a short one takes a full block. Most models use 16-token blocks. Models that mix attention and Mamba layers use blocks of hundreds of tokens (T5).
Reserved block: vLLM always keeps one block empty, so a pool of N blocks holds requests in N - 1 of them (T14).
Fit in cache (ceiling): How many requests of their current size fit when the cache is full. As answers grow, each needs more blocks and fewer fit (T9).
Preemption: When the cache is full and a running request needs another block, vLLM pauses a request and throws its notes away. To continue, the request must re-read everything so far to rebuild them. It costs time but writes no error and no log line (T1).
Output tokens per second: Tokens generated per second by all requests together, from the server's own counter.
Time to first token: Time from sending a request until the first token of the answer comes back. It includes the time spent waiting in the queue.
Boot log: The messages vLLM prints while it starts. They record the settings it actually chose, which can differ from the ones asked for.
Before and after readings: Copies of the server's counters taken just before and just after the test. The difference between them is what happened during the test.
Recording: The server's gauges, read several times a second during the test. The charts are drawn from it.
Attention backend: The code that does the model's main calculation on the GPU. vLLM has several and picks one unless it is told which (T6).
CUDA graphs: A speed-up in which vLLM records the GPU's steps once at start-up and replays them. It costs a little memory.

Results

2Fail
2Warning
0Can't tell
4Pass

A Fail means a tripwire caught its problem in this run, so some numbers from it cannot be taken at face value. It is about the measurement, not a crash. The server finished 32 requests with no errors.

What Pass, Warning, Fail and Can't tell mean

Result	What it means	What to do
Fail	The tripwire found its problem in this run. A number the run reports, such as users served, speed or delay, is not what it seems. Only T7 and T12 are about the server crashing; every other Fail is about the measurement.	Read the finding. Fix the setup and run again, or report the number together with the finding.
Warning	Nothing is wrong on its own, but a setting or an effect changes how the numbers should be read.	Keep the warning with the results, so whoever compares them knows.
Can't tell	The check could not decide, because a counter or log line it needs was missing, for example from a vLLM version it does not know. It never counts as a pass.	Supply the missing file, or check the vLLM version.
Pass	The tripwire looked for its problem in this run and did not find it.	Nothing. A pass covers this run only.

Fail
T1Silent preemption
22 preemptions during the run
Each paused request waits and then repeats work, so it finishes later than it should. Timing results include this hidden delay, and long requests suffer most. Searching the log for it finds nothing, even when it happened.
Fail
T8Fewer users than requested
requested 32 concurrent, server never ran more than 10
A result labelled '32 users' can describe a 10-user server.
Warning
T5Large memory blocks
attention block size forced to 784 tokens (requested 16); each request's KV is allocated in 784-token units
A short request still occupies a whole block, so far fewer requests fit than the pool's token count suggests.
Warning
T9Capacity shrinks as answers grow
ceiling moved between 7 and 10 as requests grew (one request held 9.5% to 14.3% of the cache); concurrency was not a constant of this run
A run has no single 'maximum concurrency'. It changes while the run is going, and every drop causes preemptions (T1).
Pass
T6Requested backend ignored
nothing to check: the server was started without --attention-backend, so vLLM chose its own (FLASH_ATTN) and nothing could be ignored; there was no draft model either
Pass
T10Whole-second timers
341.44 tokens/s over 93.720s
Pass
T12Failures with no reason
no failure in log
Pass
T14One block is reserved
gauge denominator is 42 = num_gpu_blocks (43) - 1 null block

What the server did

Requests over time

Requests the server was working on, requests queued behind them, and how many requests of their current size fit in the cache when it is full. Triangles mark preemptions.

running
waiting
how many fit in the cache
concurrency asked for
preemption

Data table

seconds	running	waiting	fit in cache
0.00	0	0	–
0.26	10	22	10
0.51	10	22	10
0.77	10	22	10
1.03	10	22	10
1.28	10	22	10
1.54	10	22	10
1.80	10	22	10
2.05	10	22	10
2.31	10	22	10
2.57	10	22	10
2.83	10	22	10
3.08	10	22	10
3.34	10	22	10
3.59	10	22	10
3.85	10	22	10
4.11	10	22	10
4.36	10	22	10
4.62	10	22	10
4.88	10	22	10
5.13	10	22	10
5.39	10	22	10
5.65	10	22	10
5.90	10	22	10
6.16	10	22	10
6.42	10	22	10
6.67	10	22	10
6.93	10	22	10
7.19	10	22	10
7.45	10	22	10
7.71	10	22	10
7.97	10	22	10
8.23	10	22	10
8.48	10	22	10
8.74	10	22	10
9.00	10	22	10
9.25	10	22	10
9.51	10	22	10
9.77	10	22	10
10.02	10	22	10
10.28	10	22	10
10.54	10	22	10
10.79	10	22	10
11.05	10	22	10
11.31	10	22	10
11.56	10	22	10
11.82	10	22	10
12.08	10	22	10
12.34	10	22	10
12.62	10	22	10
12.87	10	22	10
13.13	10	22	10
13.39	10	22	10
13.64	10	22	10
13.90	10	22	10
14.15	10	22	10
14.41	10	22	10
14.67	10	22	10
14.92	10	22	10
15.18	10	22	10
15.44	10	22	10
15.70	10	22	10
15.95	10	22	10
16.21	6	26	–
16.47	8	24	8
16.72	6	26	–
16.98	7	25	7
17.24	8	24	8
17.49	8	24	8
17.75	7	25	–
18.01	7	25	–
18.26	7	25	–
18.52	7	25	–
18.77	7	25	–
19.03	7	25	–
19.29	7	25	–
19.55	7	25	–
19.80	7	25	–
20.06	7	25	–
20.32	7	25	–
20.57	7	25	–
20.83	7	25	–
21.09	7	25	–
21.34	7	25	–
21.60	7	25	–
21.86	7	25	–
22.11	7	25	–
22.37	7	25	–
22.63	7	25	–
22.88	7	25	–
23.14	7	25	–
23.40	7	25	–
23.65	7	25	–
23.91	7	25	–
24.17	2	24	–
24.42	9	17	9
24.68	7	18	–
24.94	9	16	9
25.19	9	16	9
25.45	9	16	9
25.71	9	16	9
25.96	9	16	9
26.22	9	16	9
26.48	9	16	9
26.74	9	16	9
26.99	9	16	9
27.25	9	16	9
27.50	9	16	9
27.76	9	16	9
28.02	9	16	9
28.27	9	16	9
28.53	9	16	9
28.79	9	16	9
29.04	9	16	9
29.30	9	16	9
29.56	9	16	9
29.81	9	16	9
30.07	9	16	9
30.34	9	16	9
30.59	9	16	9
30.85	9	16	9
31.11	9	16	9
31.37	9	16	9
31.63	9	14	–
31.89	10	12	10
32.14	10	12	10
32.40	10	12	10
32.65	10	12	10
32.91	10	12	10
33.17	10	12	10
33.42	10	12	10
33.68	10	12	10
33.94	10	12	10
34.19	10	12	10
34.45	10	12	10
34.71	10	12	10
34.96	10	12	10
35.22	10	12	10
35.47	10	12	10
35.73	10	12	10
35.99	10	12	10
36.24	10	12	10
36.50	10	12	10
36.76	10	12	10
37.01	10	12	10
37.27	10	12	10
37.52	10	12	10
37.78	10	12	10
38.04	10	12	10
38.29	10	12	10
38.55	10	12	10
38.81	10	12	10
39.06	10	12	10
39.32	10	12	10
39.57	10	12	10
39.83	10	12	10
40.09	10	12	10
40.34	10	12	10
40.60	9	13	9
40.86	7	15	7
41.11	9	13	9
41.37	9	13	9
41.62	9	13	9
41.88	9	13	9
42.14	9	13	9
42.40	9	13	9
42.65	9	13	9
42.91	9	13	9
43.16	9	13	9
43.42	9	13	9
43.68	9	13	9
43.93	9	13	9
44.19	9	13	9
44.44	9	13	9
44.70	9	13	9
44.96	9	13	9
45.21	9	13	9
45.47	9	13	9
45.74	9	13	9
45.99	9	13	9
46.25	9	13	9
46.51	9	13	9
46.76	9	13	9
47.02	9	13	9
47.27	9	13	9
47.53	9	13	9
47.79	9	13	9
48.04	9	13	9
48.30	3	15	–
48.56	5	12	–
48.81	5	12	–
49.07	10	6	10
49.33	9	7	9
49.58	9	7	9
49.84	9	7	9
50.09	9	7	9
50.35	9	7	9
50.61	9	7	9
50.86	9	7	9
51.12	9	7	9
51.38	9	7	9
51.63	9	7	9
51.89	9	7	9
52.15	9	7	9
52.40	9	7	9
52.66	9	7	9
52.92	9	7	9
53.17	9	7	9
53.43	9	7	9
53.68	9	7	9
53.94	9	7	9
54.20	9	7	9
54.45	9	7	9
54.71	9	7	9
54.97	9	7	9
55.22	9	7	9
55.48	9	7	9
55.74	8	7	–
55.99	9	5	9
56.25	10	3	10
56.51	10	3	10
56.77	10	3	10
57.03	10	3	10
57.28	10	3	10
57.54	10	3	10
57.80	10	3	10
58.05	10	3	10
58.31	10	3	10
58.56	10	3	10
58.82	10	3	10
59.08	10	3	10
59.33	10	3	10
59.59	10	3	10
59.84	10	3	10
60.10	10	3	10
60.36	10	3	10
60.61	10	3	10
60.87	10	3	10
61.13	10	3	10
61.38	10	3	10
61.64	10	3	10
61.89	10	3	10
62.15	10	3	10
62.41	10	3	10
62.66	10	3	10
62.92	10	3	10
63.18	10	2	10
63.43	10	2	10
63.69	10	2	10
63.95	10	2	10
64.20	10	2	10
64.46	10	2	10
64.71	10	2	10
64.97	7	5	7
65.23	9	3	9
65.48	9	3	9
65.74	9	3	9
66.00	9	3	9
66.25	9	3	9
66.51	9	3	9
66.76	9	3	9
67.02	9	3	9
67.28	9	3	9
67.53	9	3	9
67.79	9	3	9
68.05	9	3	9
68.30	9	3	9
68.56	9	3	9
68.81	9	3	9
69.07	9	3	9
69.33	9	3	9
69.58	9	3	9
69.84	9	3	9
70.09	9	3	9
70.35	9	3	9
70.61	9	3	9
70.86	9	3	9
71.12	9	3	9
71.37	9	3	9
71.63	9	3	9
71.89	9	3	9
72.14	4	4	–
72.40	8	0	–
72.66	7	0	–
72.91	7	0	–
73.17	7	0	–
73.43	7	0	–
73.68	7	0	–
73.94	7	0	–
74.20	7	0	–
74.45	7	0	–
74.71	7	0	–
74.97	7	0	–
75.22	7	0	–
75.48	7	0	–
75.73	7	0	–
75.99	7	0	–
76.25	7	0	–
76.50	7	0	–
76.76	7	0	–
77.02	7	0	–
77.27	7	0	–
77.53	7	0	–
77.78	7	0	–
78.04	7	0	–
78.30	7	0	–
78.55	7	0	–
78.81	7	0	–
79.06	5	0	–
79.32	3	0	–
79.58	3	0	–
79.83	3	0	–
80.09	3	0	–
80.34	3	0	–
80.60	3	0	–
80.86	3	0	–
81.11	3	0	–
81.37	3	0	–
81.62	3	0	–
81.88	3	0	–
82.14	3	0	–
82.39	3	0	–
82.65	3	0	–
82.91	3	0	–
83.16	3	0	–
83.42	3	0	–
83.67	3	0	–
83.93	3	0	–
84.19	3	0	–
84.44	3	0	–
84.70	3	0	–
84.96	3	0	–
85.21	3	0	–
85.47	3	0	–
85.73	3	0	–
85.98	3	0	–
86.24	3	0	–
86.49	3	0	–
86.75	3	0	–
87.01	3	0	–
87.26	3	0	–
87.52	3	0	–
87.78	3	0	–
88.03	3	0	–
88.29	3	0	–
88.55	3	0	–
88.80	3	0	–
89.06	3	0	–
89.31	3	0	–
89.57	3	0	–
89.83	3	0	–
90.08	3	0	–
90.34	3	0	–
90.60	3	0	–
90.85	3	0	–
91.11	3	0	–
91.37	3	0	–
91.62	3	0	–
91.88	2	0	–
92.14	2	0	–
92.40	2	0	–
92.65	2	0	–
92.91	2	0	–
93.17	2	0	–
93.42	2	0	–

KV cache in use

Share of the cache holding requests. The gauge moves in steps of 1/42: one block is held back (T14).

preemption

Data table

seconds	% in use
0.00	0.0
0.26	95.2
0.51	95.2
0.77	95.2
1.03	95.2
1.28	95.2
1.54	95.2
1.80	95.2
2.05	95.2
2.31	95.2
2.57	95.2
2.83	95.2
3.08	95.2
3.34	95.2
3.59	95.2
3.85	95.2
4.11	95.2
4.36	95.2
4.62	95.2
4.88	95.2
5.13	95.2
5.39	95.2
5.65	95.2
5.90	95.2
6.16	95.2
6.42	95.2
6.67	95.2
6.93	95.2
7.19	95.2
7.45	95.2
7.71	95.2
7.97	95.2
8.23	95.2
8.48	95.2
8.74	95.2
9.00	95.2
9.25	95.2
9.51	95.2
9.77	95.2
10.02	95.2
10.28	95.2
10.54	95.2
10.79	95.2
11.05	95.2
11.31	95.2
11.56	95.2
11.82	95.2
12.08	95.2
12.34	95.2
12.62	95.2
12.87	95.2
13.13	95.2
13.39	95.2
13.64	95.2
13.90	95.2
14.15	95.2
14.41	95.2
14.67	95.2
14.92	95.2
15.18	95.2
15.44	95.2
15.70	95.2
15.95	95.2
16.21	85.7
16.47	100.0
16.72	83.3
16.98	97.6
17.24	100.0
17.49	92.9
17.75	83.3
18.01	83.3
18.26	83.3
18.52	83.3
18.77	83.3
19.03	83.3
19.29	83.3
19.55	83.3
19.80	83.3
20.06	83.3
20.32	83.3
20.57	83.3
20.83	83.3
21.09	83.3
21.34	83.3
21.60	83.3
21.86	83.3
22.11	83.3
22.37	83.3
22.63	83.3
22.88	83.3
23.14	83.3
23.40	83.3
23.65	83.3
23.91	83.3
24.17	31.0
24.42	97.6
24.68	81.0
24.94	92.9
25.19	92.9
25.45	92.9
25.71	92.9
25.96	92.9
26.22	92.9
26.48	92.9
26.74	92.9
26.99	92.9
27.25	92.9
27.50	92.9
27.76	92.9
28.02	92.9
28.27	92.9
28.53	92.9
28.79	92.9
29.04	92.9
29.30	92.9
29.56	92.9
29.81	92.9
30.07	92.9
30.34	92.9
30.59	92.9
30.85	92.9
31.11	92.9
31.37	92.9
31.63	88.1
31.89	95.2
32.14	95.2
32.40	95.2
32.65	95.2
32.91	95.2
33.17	95.2
33.42	95.2
33.68	95.2
33.94	95.2
34.19	95.2
34.45	95.2
34.71	95.2
34.96	95.2
35.22	95.2
35.47	95.2
35.73	95.2
35.99	95.2
36.24	95.2
36.50	95.2
36.76	95.2
37.01	95.2
37.27	95.2
37.52	95.2
37.78	95.2
38.04	95.2
38.29	95.2
38.55	95.2
38.81	95.2
39.06	95.2
39.32	95.2
39.57	95.2
39.83	95.2
40.09	95.2
40.34	95.2
40.60	95.2
40.86	92.9
41.11	97.6
41.37	100.0
41.62	100.0
41.88	100.0
42.14	100.0
42.40	100.0
42.65	100.0
42.91	100.0
43.16	100.0
43.42	100.0
43.68	100.0
43.93	100.0
44.19	100.0
44.44	100.0
44.70	100.0
44.96	100.0
45.21	100.0
45.47	100.0
45.74	100.0
45.99	100.0
46.25	100.0
46.51	100.0
46.76	100.0
47.02	100.0
47.27	100.0
47.53	100.0
47.79	100.0
48.04	100.0
48.30	42.9
48.56	52.4
48.81	52.4
49.07	100.0
49.33	92.9
49.58	92.9
49.84	92.9
50.09	92.9
50.35	92.9
50.61	92.9
50.86	92.9
51.12	92.9
51.38	92.9
51.63	92.9
51.89	92.9
52.15	92.9
52.40	92.9
52.66	92.9
52.92	92.9
53.17	92.9
53.43	92.9
53.68	92.9
53.94	92.9
54.20	92.9
54.45	92.9
54.71	92.9
54.97	92.9
55.22	92.9
55.48	92.9
55.74	81.0
55.99	97.6
56.25	97.6
56.51	97.6
56.77	97.6
57.03	97.6
57.28	97.6
57.54	97.6
57.80	97.6
58.05	97.6
58.31	97.6
58.56	97.6
58.82	97.6
59.08	97.6
59.33	97.6
59.59	97.6
59.84	97.6
60.10	97.6
60.36	97.6
60.61	97.6
60.87	97.6
61.13	97.6
61.38	97.6
61.64	97.6
61.89	97.6
62.15	97.6
62.41	97.6
62.66	97.6
62.92	97.6
63.18	95.2
63.43	95.2
63.69	95.2
63.95	95.2
64.20	95.2
64.46	95.2
64.71	95.2
64.97	100.0
65.23	97.6
65.48	97.6
65.74	97.6
66.00	97.6
66.25	97.6
66.51	97.6
66.76	97.6
67.02	97.6
67.28	97.6
67.53	97.6
67.79	97.6
68.05	97.6
68.30	97.6
68.56	97.6
68.81	97.6
69.07	97.6
69.33	97.6
69.58	97.6
69.84	97.6
70.09	97.6
70.35	97.6
70.61	97.6
70.86	97.6
71.12	97.6
71.37	97.6
71.63	97.6
71.89	97.6
72.14	52.4
72.40	83.3
72.66	73.8
72.91	76.2
73.17	76.2
73.43	76.2
73.68	76.2
73.94	76.2
74.20	76.2
74.45	76.2
74.71	76.2
74.97	76.2
75.22	76.2
75.48	76.2
75.73	76.2
75.99	76.2
76.25	76.2
76.50	76.2
76.76	76.2
77.02	76.2
77.27	76.2
77.53	76.2
77.78	76.2
78.04	76.2
78.30	76.2
78.55	76.2
78.81	76.2
79.06	52.4
79.32	28.6
79.58	28.6
79.83	28.6
80.09	28.6
80.34	28.6
80.60	28.6
80.86	28.6
81.11	28.6
81.37	28.6
81.62	28.6
81.88	28.6
82.14	28.6
82.39	28.6
82.65	28.6
82.91	28.6
83.16	28.6
83.42	28.6
83.67	28.6
83.93	28.6
84.19	28.6
84.44	28.6
84.70	28.6
84.96	28.6
85.21	28.6
85.47	38.1
85.73	31.0
85.98	31.0
86.24	31.0
86.49	31.0
86.75	31.0
87.01	31.0
87.26	35.7
87.52	35.7
87.78	35.7
88.03	35.7
88.29	35.7
88.55	35.7
88.80	35.7
89.06	35.7
89.31	35.7
89.57	35.7
89.83	35.7
90.08	35.7
90.34	35.7
90.60	35.7
90.85	35.7
91.11	35.7
91.37	35.7
91.62	35.7
91.88	23.8
92.14	23.8
92.40	23.8
92.65	23.8
92.91	23.8
93.17	23.8
93.42	23.8

Output tokens per second

Generated tokens over a sliding one-second window, from the server's own counter.

Data table

seconds	tokens/s
0.00	–
0.26	3.9
0.51	215.9
0.77	286.7
1.03	441.6
1.28	442.0
1.54	442.3
1.80	442.6
2.05	442.6
2.31	442.3
2.57	440.9
2.83	439.4
3.08	439.5
3.34	440.9
3.59	442.0
3.85	442.1
4.11	442.4
4.36	442.7
4.62	442.2
4.88	441.2
5.13	441.0
5.39	441.2
5.65	428.9
5.90	442.0
6.16	442.1
6.42	442.2
6.67	442.0
6.93	439.9
7.19	437.1
7.45	437.3
7.71	438.5
7.97	436.9
8.23	436.8
8.48	437.6
8.74	442.0
9.00	428.9
9.25	441.8
9.51	442.1
9.77	442.1
10.02	428.5
10.28	428.4
10.54	441.8
10.79	442.4
11.05	442.4
11.31	428.7
11.56	440.9
11.82	440.9
12.08	441.5
12.34	427.6
12.62	426.7
12.87	439.2
13.13	441.0
13.39	442.3
13.64	429.7
13.90	429.7
14.15	429.6
14.41	429.4
14.67	429.2
14.92	441.1
15.18	441.1
15.44	440.3
15.70	427.2
15.95	427.3
16.21	399.4
16.47	265.0
16.72	131.2
16.98	32.5
17.24	33.8
17.49	54.6
17.75	120.9
18.01	221.1
18.26	300.6
18.52	327.8
18.77	327.5
19.03	326.5
19.29	325.9
19.55	326.3
19.80	327.1
20.06	318.3
20.32	318.2
20.57	318.5
20.83	327.7
21.09	327.5
21.34	318.4
21.60	318.4
21.86	318.2
22.11	327.2
22.37	327.4
22.63	318.6
22.88	317.6
23.14	316.9
23.40	326.2
23.65	327.3
23.91	328.0
24.17	235.4
24.42	128.6
24.68	53.3
24.94	139.0
25.19	265.1
25.45	360.0
25.71	397.8
25.96	398.0
26.22	408.4
26.48	395.7
26.74	407.3
26.99	397.1
27.25	410.0
27.50	398.3
27.76	398.2
28.02	398.0
28.27	398.1
28.53	410.0
28.79	397.6
29.04	408.7
29.30	396.9
29.56	397.7
29.81	397.9
30.07	397.8
30.34	403.1
30.59	391.2
30.85	402.3
31.11	394.5
31.37	404.0
31.63	350.4
31.89	325.5
32.14	343.5
32.40	401.0
32.65	442.6
32.91	442.4
33.17	442.5
33.42	442.9
33.68	442.0
33.94	441.9
34.19	441.8
34.45	429.8
34.71	442.1
34.96	441.9
35.22	442.1
35.47	442.8
35.73	442.6
35.99	442.4
36.24	429.0
36.50	441.7
36.76	441.7
37.01	442.1
37.27	429.6
37.52	429.3
37.78	442.0
38.04	442.1
38.29	442.3
38.55	429.4
38.81	442.3
39.06	442.4
39.32	442.7
39.57	429.5
39.83	428.9
40.09	441.5
40.34	441.9
40.60	441.2
40.86	315.1
41.11	180.9
41.37	83.2
41.62	181.9
41.88	301.5
42.14	397.6
42.40	396.6
42.65	396.7
42.91	396.9
43.16	398.3
43.42	398.1
43.68	398.2
43.93	398.4
44.19	398.6
44.44	387.0
44.70	398.6
44.96	398.5
45.21	398.3
45.47	395.6
45.74	393.1
45.99	403.8
46.25	394.8
46.51	397.3
46.76	386.6
47.02	397.8
47.27	397.7
47.53	397.6
47.79	386.3
48.04	397.8
48.30	291.1
48.56	166.4
48.81	26.0
49.07	68.9
49.33	206.9
49.58	335.7
49.84	410.9
50.09	397.8
50.35	397.7
50.61	397.6
50.86	397.6
51.12	409.6
51.38	398.1
51.63	409.6
51.89	397.5
52.15	397.2
52.40	397.2
52.66	397.6
52.92	410.0
53.17	398.4
53.43	398.3
53.68	398.1
53.94	397.7
54.20	409.4
54.45	397.6
54.71	397.7
54.97	397.4
55.22	397.7
55.48	397.4
55.74	373.8
55.99	343.3
56.25	341.1
56.51	366.7
56.77	410.2
57.03	440.3
57.28	441.3
57.54	442.2
57.80	429.2
58.05	442.0
58.31	442.3
58.56	442.6
58.82	429.7
59.08	442.2
59.33	442.0
59.59	442.4
59.84	429.5
60.10	442.5
60.36	442.6
60.61	442.9
60.87	429.6
61.13	429.3
61.38	441.9
61.64	442.1
61.89	442.1
62.15	429.2
62.41	429.1
62.66	441.8
62.92	441.4
63.18	427.4
63.43	415.1
63.69	415.6
63.95	429.5
64.20	441.9
64.46	442.0
64.71	442.0
64.97	422.5
65.23	297.6
65.48	248.1
65.74	252.0
66.00	362.7
66.25	398.4
66.51	398.5
66.76	398.2
67.02	398.2
67.28	398.3
67.53	398.2
67.79	398.1
68.05	386.3
68.30	397.9
68.56	397.9
68.81	398.1
69.07	398.4
69.33	398.6
69.58	398.6
69.84	386.8
70.09	398.4
70.35	398.5
70.61	398.6
70.86	386.9
71.12	398.2
71.37	398.1
71.63	397.9
71.89	386.1
72.14	370.3
72.40	246.9
72.66	193.4
72.91	189.6
73.17	293.3
73.43	327.5
73.68	327.3
73.94	327.3
74.20	327.3
74.45	327.7
74.71	328.2
74.97	328.1
75.22	328.2
75.48	327.9
75.73	327.5
75.99	327.5
76.25	327.7
76.50	328.1
76.76	328.1
77.02	318.7
77.27	318.6
77.53	318.4
77.78	327.7
78.04	327.8
78.30	328.0
78.55	328.0
78.81	328.2
79.06	306.1
79.32	260.6
79.58	197.9
79.83	161.4
80.09	144.5
80.34	144.5
80.60	144.5
80.86	144.5
81.11	148.4
81.37	144.5
81.62	144.6
81.88	144.6
82.14	144.6
82.39	144.6
82.65	144.4
82.91	144.2
83.16	144.0
83.42	144.1
83.67	144.4
83.93	144.5
84.19	144.4
84.44	144.4
84.70	144.3
84.96	144.3
85.21	144.4
85.47	144.3
85.73	144.3
85.98	144.3
86.24	144.4
86.49	144.3
86.75	144.3
87.01	144.4
87.26	144.4
87.52	144.4
87.78	144.4
88.03	140.4
88.29	144.2
88.55	144.3
88.80	144.3
89.06	144.5
89.31	144.5
89.57	144.4
89.83	144.3
90.08	144.2
90.34	144.4
90.60	144.5
90.85	144.5
91.11	143.8
91.37	143.5
91.62	143.6
91.88	135.1
92.14	118.2
92.40	104.7
92.65	98.3
92.91	98.4
93.17	98.9
93.42	98.8

How many requests ran at once, and for how long

Each bar is the total time the server spent working on exactly that many requests at the same time. The load tool asked for 32 the whole time, but the server never ran more than 10.

Data table

running at once	seconds
10	33.64
9	28.24
8	1.28
7	13.85
6	0.51
5	0.77
4	0.26
3	12.82
2	1.80

The server that ran

From boot.log. Every result should carry these, because they change from one start to the next.

vLLM: 0.28.0
Model: Qwen3.8-27B-W4A16-AutoRound
KV pool: 26,093 tokens
Attention block: 784 tokens
Attention backend: FLASH_ATTN
Draft model backend: none
Memory for KV cache: 2.09 GiB
CUDA graphs: on, 0.18 GiB
Boot line 'maximum concurrency': 1.59x at 16,384 tokens
Model load: 17.71 GiB in 79 s
Server settings: gpu_memory_utilization=0.93, host=127.0.0.1, max_model_len=16384, max_num_seqs=32, served_model_name=['qwen38']

The fourteen tripwires

Each one is a way an inference benchmark can report a wrong number without raising an error.

T1Silent preemption
When the GPU memory that holds requests in progress (the KV cache) fills up, vLLM pauses a request and throws away its working notes. Before the request can continue, it has to re-read everything so far to rebuild them. vLLM writes nothing in its log when this happens. Only a counter moves.
Each paused request waits and then repeats work, so it finishes later than it should. Timing results include this hidden delay, and long requests suffer most. Searching the log for it finds nothing, even when it happened.
inferlint preemption before.json after.json
T2Engine left running
The usual way to stop vLLM, pkill -f "vllm serve", only reaches the front-end process. The engine process, which holds the model in GPU memory, renames itself VLLM::EngineCore and can keep running after the front end is gone.
The next test starts on a GPU that is already mostly full and produces wrong numbers with no error. One low memory reading is not proof the GPU is free.
inferlint teardown | inferlint gpu-inspect
T3Cleanup kills itself
Running pkill -f "vllm serve" inside a one-line shell command also kills that shell, because the shell's own command line contains the same words.
The rest of the cleanup never runs and the script ends with no message.
inferlint teardown
T4Memory pool changes between starts
Each time vLLM starts, it measures free memory and sets aside a pool for requests in progress. The same settings can get a different pool on the next start, and pool sizes come in fixed steps.
Two runs with identical settings can differ by a whole step (9.4% in the recorded example), which looks like a real speed difference and is not one.
inferlint check-log boot1.log boot2.log
T5Large memory blocks
Models that mix attention layers with Mamba layers make vLLM hand out cache memory in large blocks, hundreds of tokens each, instead of 16-token blocks.
A short request still occupies a whole block, so far fewer requests fit than the pool's token count suggests.
inferlint check-log boot.log
T6Requested backend ignored
The attention backend is the code that does the model's main calculation on the GPU. vLLM has several, such as FlashAttention, FlashInfer and Triton, and you can choose one when you start the server with --attention-backend. This checks that vLLM really used the one you asked for. With speculative decoding on, the small draft model that helps the main model picks its own backend and quietly ignores your choice.
In the recorded case Triton was requested, but the draft model picked FlashInfer, which crashed on the first request, while the log showed Triton in use.
inferlint check-log boot.log
T7Ready, then crashes
A server can start, announce that it is ready, and crash as soon as the first real request arrives.
A check that only waits for 'ready' passes a server that cannot serve anything.
inferlint probe http://127.0.0.1:8000
T8Fewer users than requested
A load tool can keep 32 requests open, but the server may only work on a few at a time and queue the rest.
A result labelled '32 users' can describe a 10-user server.
inferlint series run.series.jsonl --requested 32
T9Capacity shrinks as answers grow
The number of requests that fit at once is 1 divided by the share of cache one request holds. Each answer takes more cache as it gets longer, so fewer fit as a run goes on, and the server evicts some to make room.
A run has no single 'maximum concurrency'. It changes while the run is going, and every drop causes preemptions (T1).
inferlint series run.series.jsonl
T10Whole-second timers
Timing a run with whole seconds, such as $(date +%s), can be off by almost a second either way.
On a 30-second test that is about 3%, enough to flip a close comparison.
inferlint rate before.json after.json
T11KV memory varies between starts
The memory left for the pool after loading the model and capturing CUDA graphs can differ slightly from one start to the next.
A small difference can drop the pool by a whole step (T4).
inferlint check-log boot1.log boot2.log
T12Failures with no reason
When a server fails to start, simple log searches often miss the real error and record a blank reason.
A failure without a reason cannot be fixed, repeated or trusted.
inferlint check-log boot.log
T13Old script adopts the new server
A test script stopped while it waits for its server can mistake the next script's server for its own, report it ready, then shut it down.
The next test dies partway through for no visible reason.
see docs/orchestration.md
T14One block is reserved
vLLM reports N cache blocks but always keeps one empty. Its usage gauge counts out of N - 1.
Capacity worked out from the reported block count is slightly too high.
inferlint series run.series.jsonl --snapshot after.json