Tutorial

inferlint, step by step

inferlint measures what your inference server actually did—and flags what it didn't tell you. From a vLLM server's start-up log and counters it measures achieved concurrency, preemptions, KV-cache use, GPU state and timing, and checks the run for 14 known problems. Each check answers Pass, Warning, Fail or Can't tell. It works alongside whatever load tool you already use. The examples on this page all come from one recorded run: vLLM 0.28 on an RTX 4090, a 27B model, 32 requests at once, 1,000 output tokens each.

Asked for 32 at once. The server ran 10.

Requests in flight during the recorded run, read every 0.25 s

Hover over the chart, or focus it and use the arrow keys, to read any moment.

Show the numbers, every 5 seconds

10 of 32requests ran at once. The benchmark said 32. T8
22preemptions. The server log mentions none. T1
10 → 7the most that fit, falling as answers grew. T9

Quick start · 2 minutes · no GPU

Try it on the recorded run

The repository includes the files from the recorded run, so you can run every check on any computer, Linux, macOS or Windows, without a GPU or a server.

1Install

pipx install inferlint
inferlint --help

pipx gives inferlint its own environment and puts the inferlint command on your PATH. There is nothing to activate, and it works from any folder. Plain pip install inferlint works too. Needs Python 3.10 or newer; inferlint has no other dependencies.

2Get the recorded run

git clone https://github.com/radianvector/inferlint
cd inferlint

3Run the checks on it

First, a short name for the folder with the recorded files:

L=tests/fixtures/vllm-0.28/live

$L = "tests\fixtures\vllm-0.28\live"

On Windows, everything here works in PowerShell. teardown needs Linux, because it finds the server's processes the way Linux lists them. gpu-inspect shows the GPU on Windows too, but its free-card check (T2) needs Linux, so there it answers Can't tell. Run those two in WSL, where vLLM runs anyway.

The idea behind most of the checks

Why only 10? The GPU memory, in 3D

While a request is being answered, the server keeps its working notes in GPU memory, called the KV cache. vLLM hands that memory out in blocks. In this run there were 43 blocks of 784 tokens each, and one is held back, so 42 were usable (T14).

Each request needed 4 blocks to start, so 10 fit and the other 22 waited. As answers grew past 784 tokens they needed more blocks, and fewer fit. Press Play to replay the recorded run, or drag the slider through it.

Seconds into the run0.0

Running0

Waiting0

Blocks each, average–

Finished0

Preemptions so far0

Time in the recorded run

0 s 16 s: first preemption 93 s

The 42 usable blocks0 in use, 42 free

Each tower is a running request and each tile a waiting one; a tile with an outline has just been pushed back. Outlined spots are places with no room while requests wait. The recording has the total number of blocks in use, not each request's count, so the towers share that total evenly.

Check your own run

A GPU run with one command

You need Linux or WSL, an NVIDIA GPU, vLLM, and inferlint installed on the same machine (step 1 above). inferlint xray starts the server, checks it, records it while your benchmark runs, writes the report and stops the server. Your load tool goes after --; inferlint sends no load of its own.

inferlint xray -o run/ --serve "vllm serve $MODEL --max-model-len 16384" \
    -- vllm bench serve --model "$MODEL" --dataset-name random \
       --random-input-len 88 --random-output-len 1000 --ignore-eos \
       --num-prompts 32 --max-concurrency 32

You'll see (vLLM 0.28 on the RTX 4090 used for this page, shortened)

== is the GPU free?
[ T2] PASS  2 consecutive clear readings
== start the server (output in run/boot.log)
== boot log
[T12] PASS  no failure in log
[ T5] WARN  attention block size forced to 784 tokens (requested 16); each request's KV is allocated in 784-token units
[ T6] PASS  nothing to check: the server was started without --attention-backend, so vLLM chose its own (FLASH_ATTN) and nothing could be ignored; there was no draft model either
== probe: one request, then 8 at once
[ T7] PASS  1 + 8 concurrent requests served, server alive
== load: vllm bench serve --model MODEL ... --max-concurrency 32 --save-result --result-dir run --result-filename bench.json
…
== checks
[ T1] TRIPWIRE-FAILED  21 preemptions during the run
[T10] PASS  314.99 tokens/s over 101.590s; an integer-second timer would say 313.73 (-0.40%)
[T14] PASS  gauge denominator is 42 = num_gpu_blocks (43) - 1 null block
[ T9] WARN  ceiling moved between 5 and 10 as requests grew (one request held 9.5% to 19.0% of the cache); concurrency was not a constant of this run
[ T8] TRIPWIRE-FAILED  requested 32 concurrent, server never ran more than 10
[T15] PASS  client and server agree: 32 requests, 32,000 output tokens
== stop the server
[ T2] PASS  2 consecutive clear readings
12 checks (8 pass, 2 warn, 2 tripwire-failed)
report: run/report.html
files:  run

Above, paths are shortened, the load command is cut, and the benchmark's own output is left out (the … line). This was a second run of the same test as the recorded one, so its counts differ a little: 21 preemptions instead of 22, and a lowest ceiling of 5 instead of 7.

Every file lands in run/: the boot log, the before and after readings, the recording, the benchmark's own result file and report.html. The server is stopped at the end even if a step fails or you press Ctrl-C. To check a server that is already running, leave out --serve and give --url and --boot-log.

For vllm bench serve, xray adds --save-result so it can compare the benchmark's own counts with the server's (T15), and reads --max-concurrency as the concurrency asked for (T8).

Or step by step

The same routine as separate commands, for scripts that control each step. Use two terminals in the same folder: A runs the server, B runs everything else. The picture shows what runs when; click a number to open that step.

Before you start
1Check that the GPU is freeinferlint gpu-inspect
A server from an earlier test can still be holding the card. In the recorded run, a killed server's engine kept 21.3 GiB after its front end was gone. A new server started then would only get the memory that was left.
```
inferlint gpu-inspect || exit 1
```
You'll see (the RTX 4090 used for this page)
```
GPU 0   NVIDIA GeForce RTX 4090
        Ada Lovelace, compute capability 8.9, driver 616.56
        memory: 24,564 MiB total, 0 MiB used, 24,138 MiB free
[ T2] PASS  2 consecutive clear readings
```
It shows which GPU this is and its memory, then checks that no server process is holding it. It only looks; it never stops anything. It exits with 1 if the card is not clear, so a script stops there.

Checks T2
Start the server
2Start vLLM and keep its logvllm serve … | tee boot.log
In terminal A, start the server with your usual command and save everything it prints at start-up to a file. The log records how much memory the server set aside and how it divided it; inferlint reads it in the next step. Use a new log file for each start.
```
vllm serve "$MODEL" --max-model-len 16384 2>&1 | tee boot.log
```
Wait for Application startup complete; it took 2 min 4 s in the recorded run. Run the remaining steps in terminal B.
3Read the start-up loginferlint check-log boot.log
This checks how the server set itself up: whether start-up failed and why, how large its memory blocks are, and whether it used the attention method you asked for.
```
inferlint check-log boot.log
```
You'll see (recorded run)
```
[T12] PASS  no failure in log
[ T5] WARN  attention block size forced to 784 tokens (requested 16); each request's KV is allocated in 784-token units
[ T6] PASS  nothing to check: the server was started without --attention-backend, so vLLM chose its own (FLASH_ATTN) and nothing could be ignored; there was no draft model either
```
With two or more logs from the same settings, check-log also compares them (T4, T11). inferlint boot-facts boot.log lists everything it read from the log.

Checks T12 T5 T6 T4 T11
4Send it a few real requestsinferlint probe URL
Some settings start cleanly and then crash on the first request. The probe sends one request, then 8 at once, and checks that the server is still running afterwards.
```
inferlint probe http://127.0.0.1:8000
```
You'll see
```
[ T7] PASS  1 + 8 concurrent requests served, server alive
```
Checks T7
Measure
5Take a reading before the testinferlint snapshot URL -o before.json
This saves all of the server's counters, with the time, to a file: tokens generated so far, requests finished, preemptions and more. Step 8 takes a second reading, and the difference between the two is what happened during your test.
```
inferlint snapshot http://127.0.0.1:8000 -o before.json
```
6Start recordinginferlint watch URL … &
Two readings show totals. To see what happened in between (how many requests ran, how many waited, how full the memory was), record the server every quarter second while the test runs. The & runs it in the background.
```
inferlint watch http://127.0.0.1:8000 -o run.series.jsonl --interval 0.25 &
```
That recording is what the chart at the top of this page is drawn from.
7Run your load testyour usual client
Use whatever benchmark you normally use. inferlint doesn't need to know which: its numbers come from the server's own counters. Note how many requests your tool keeps open at once; you'll pass that number as --requested. For example, with vLLM's own client:
```
vllm bench serve --model "$MODEL" --dataset-name random \
    --random-input-len 88 --random-output-len 1000 --ignore-eos \
    --num-prompts 32 --max-concurrency 32
```
Option names change between vLLM versions; vllm bench serve --help lists yours.
8Take a reading after, and stop recordinginferlint snapshot … -o after.json
```
inferlint snapshot http://127.0.0.1:8000 -o after.json
kill %1        # stops the watch from step 6
```
You now have four files: boot.log, before.json, after.json and run.series.jsonl. Everything from here on works from these files, on any computer, with no GPU.
Read the results

9Check what happened during the runpreemption · rate · series

inferlint preemption before.json after.json
inferlint rate before.json after.json
inferlint series run.series.jsonl --snapshot after.json --requested 32

You'll see (recorded run)

[ T1] TRIPWIRE-FAILED  22 preemptions during the run
[T10] PASS  341.44 tokens/s over 93.720s; an integer-second timer would say 344.09 (+0.77%)
[T14] PASS  gauge denominator is 42 = num_gpu_blocks (43) - 1 null block
[ T9] WARN  ceiling moved between 7 and 10 as requests grew (one request held 9.5% to 14.3% of the cache); concurrency was not a constant of this run
[ T8] TRIPWIRE-FAILED  requested 32 concurrent, server never ran more than 10

The server finished all 32 requests without errors. The two failed tripwires are about the measurement: at most 10 requests ran at once, and 22 were paused and restarted along the way.

Checks T1 T10 T14 T9 T8

10Make the reportinferlint report -o run.html …
One HTML page with everything: a plain-English account of the run, every verdict, charts of what the server did, and the start-up facts. It needs no internet connection, so you can attach it to a ticket as it is.
```
inferlint report -o run.html --requested 32 --boot-log boot.log \
    --before before.json --after after.json --series run.series.jsonl
```
You'll see
```
8 checks (4 pass, 2 warn, 2 fail) -> run.html
```
Open run.html in any browser. The report for the recorded run shows what you get.
Clean up
11Stop the server properlyinferlint teardown
The usual pkill -f "vllm serve" misses vLLM's engine process, which renames itself VLLM::EngineCore and keeps the GPU memory. teardown finds every server process, stops them, and waits until two readings in a row show the card is free.
```
inferlint teardown --dry-run   # only lists what it would stop
inferlint teardown
```
You'll see (after the server was killed with pkill -9)
```
target  pid=1030     VLLM::EngineCore
(pkill -f 'vllm serve' would miss 1 and wrongly hit 0)
[ T2] PASS  2 consecutive clear readings
```
Checks T2 T3

Reading a result

[ T1]which check TRIPWIRE-FAILEDthe verdict 22 preemptions during the runwhat it found, in the run's own numbers

Verdict	What it means	What to do
Tripwire failed	The tripwire caught its problem in this run. For T7 and T12 the server itself failed. For every other tripwire the server worked, but a number from the run (concurrency, speed or delay) does not mean what it seems.	Fix the setup and run again, or report the number together with the finding.
Warning	Nothing is wrong on its own, but a setting or an effect changes how the numbers should be read.	Keep the warning with the results.
Can't tell	A counter or log line the check needs was missing. It never counts as a pass. The terminal shows it as `????`.	Supply the missing file, or check the vLLM version.
Pass	The check looked for its problem in this run and did not find it.	Nothing. A pass covers this run only.

In scripts

Every command ends with an exit code a script can test:

`0`	everything passed (warnings allowed)
`1`	a tripwire failed
`2`	a tripwire could not decide

--strict turns warnings into failures. --json prints the results as JSON, including the evidence behind each one.

The 15 tripwires

Grouped by when in a test they apply. Open one for what goes wrong, why it changes your numbers, and the command that checks it. inferlint explain T9 (or inferlint explain falling-ceiling) prints a technical version in the terminal.

Cheat sheet

When	Command	Checks
The whole run, one command	`inferlint xray -o run/ --serve "vllm serve …" -- vllm bench serve …`	all that apply
Before starting a server	`inferlint gpu-inspect`	T2
After start-up	`inferlint check-log boot.log`	T5 T6 T12 (T4 T11 with 2+ logs)
After start-up	`inferlint boot-facts boot.log`	lists what the log says
Before the test	`inferlint probe URL`	T7
Before and after the test	`inferlint snapshot URL -o before.json`	saves the counters
During the test	`inferlint watch URL -o run.series.jsonl &`	records the run
After the test	`inferlint preemption before.json after.json`	T1
After the test	`inferlint rate before.json after.json`	T10
After the test	`inferlint series run.series.jsonl --snapshot after.json --requested N`	T8 T9 T14
Any time after	`inferlint report -o run.html …`	all that apply
Stopping the server	`inferlint teardown`	T2 T3
Any time	`inferlint explain T1`	explains a check