Metadata-Version: 2.4
Name: gpubudget
Version: 0.3.2
Summary: VRAM budgets and cross-traffic tail latency for several models on one GPU
Author-email: Marouane Tijani <marouane@pexafy.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/matijani/three-models-one-gpu
Project-URL: Issues, https://github.com/matijani/three-models-one-gpu/issues
Keywords: triton,inference,gpu,latency,benchmark
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Dynamic: license-file

# three-models-one-gpu

Three encoders share one 20 GB card, and none of the three knobs that decide
whether that works does what its name suggests. `gpubudget` reads a Triton model
repository, works out what it will actually hold, and simulates what happens to
each model's tail latency once they are competing for the same device.

`instance_group { count: 2 }` reads like two servers. On one GPU it is two copies
of the weights and a second share of the same silicon. `max_batch_size: 32` reads
like a ceiling; under the TensorRT provider it is a reservation you pay for
whether or not the traffic ever produces a batch that big. `max_queue_delay_microseconds`
reads like a batching hint; below a certain arrival rate it is a fixed addition
to every single request. None of this fails loudly. The server starts, serves
your smoke test, passes review, and shows up six weeks later as a p95 nobody can
explain and an out-of-memory error that only happens on restart.

## install

```bash
pip install gpubudget
```

No dependencies. Python 3.10 or newer.

## the smallest thing that runs

```bash
gpubudget budget example
```

`example` is the three-model repository this README is about, shipped inside the
package, so there is nothing to clone before the first answer. From a checkout,
`gpubudget budget model_repository/` is the same thing. Either prints:

```
| model            | inst | batch | weights  | activation    | total    |
|------------------|------|-------|----------|---------------|----------|
| bge_m3           | 1    | 64    | 1.06 GiB | 64 MiB (est)  | 1.17 GiB |
| siglip2_text     | 1    | 64    | 703 MiB  | 96 MiB (est)  | 847 MiB  |
| siglip2_vision   | 2    | 32    | 4.51 GiB | 864 MiB (est) | 5.45 GiB |
| process overhead |      |       |          |               | 300 MiB  |
| allocator slack  |      |       |          |               | 554 MiB  |
| total            |      |       |          |               | 8.28 GiB |

ceiling 18.40 GiB of 20.00 GiB on gpu, headroom 10.12 GiB, load peak 19.93 GiB
verdict: fits at rest, will not survive a cold start
```

Ten gibibytes of headroom and it still will not start. That gap between 8.28 and
19.93 is the whole point of the tool, and it is the subject of the next section.

`gpubudget example --copy .` writes those configs somewhere you can edit them,
which is the point: change `count` or a batch ceiling and watch the numbers move.

The same thing from Python:

```python
from gpubudget import EXAMPLE_REPOSITORY, Device, budget_table, estimate_budget, load_repository

print(budget_table(estimate_budget(load_repository(EXAMPLE_REPOSITORY), Device.from_gib("card", 20))))
```

There is a longer example further down, under [from python](#from-python).

## what the arithmetic actually is

Four terms, and only the first is obvious.

**Weights, once per instance.** An `instance_group` with `count: 2` is two ONNX
Runtime sessions. Each one copies the initialisers to the device. There is no
sharing, and no warning anywhere that this is what you asked for. Here that is
4.51 GiB where the model on disk is 4.34 GiB in FP16.

**Activation working set, sized by a batch that may never happen.** TensorRT
sizes an execution context from the optimisation profile, so
`trt_profile_max_shapes` decides the reservation whatever the traffic does. The
shipped vision config pins that at 32 while `preferred_batch_size` tops out at 8,
which reserves 648 MiB for a batch the batcher cannot form. The ONNX Runtime CUDA
arena behaves differently again: it grows to the largest batch it has served and
never gives it back, so there the number that matters is the largest batch the
batcher *can* form, not the one it usually does.

**Process overhead.** The CUDA context, plus the cuBLAS and cuDNN kernel images
if `CUDA_MODULE_LOADING` is not `LAZY`. That last one is most of a gibibyte and it
is set by an environment variable most people never look at.

**Allocator slack.** The arena rounds up and does not compact. The sum of what
you asked for is not the sum of what is resident.

And then the load-time peak, which is larger than all of it and is where servers
actually die. With a cold TensorRT cache the FP32 ONNX initialisers stay resident
while the builder runs, the builder is allowed up to `max_workspace_size_bytes` on
top, and Triton loads several models at once. Three models, two of them
TensorRT, and the peak is 19.93 GiB against an 18.40 GiB ceiling — from a steady
state that fits four times over. The fix is `--model-load-thread-count=1`, or
building the engines somewhere with more memory and mounting the cache read-only.
`gpubudget budget` exits non-zero when this happens, so it belongs in CI.

## instance count is a priority setting, not a capacity setting

This is the part the documentation does not cover, and the reason the vision
config in this repository carries a comment claiming `count: 2` doubles
throughput. It does not.

Instances are not servers. An instance is a slot that lets a batch be *prepared*
concurrently; the kernels still land on one device, and two models that each fill
the SMs do not run in parallel, they interleave. So a second instance of the heavy
model does not add capacity. What it adds is a second share in how the device
splits its time, taken from the models you did not change.

At a load the device can carry, with the text and multilingual encoders at 30
requests a second each and the vision tower at six:

| siglip2_vision instances | siglip2_vision req/s | siglip2_text p50 ms | siglip2_text p95 ms | vram      |
|--------------------------|----------------------|---------------------|---------------------|-----------|
| 1                        | 6.1                  | 54                  | 89                  | 5.36 GiB  |
| 2                        | 6.1                  | 59                  | 109                 | 8.28 GiB  |
| 3                        | 6.1                  | 61                  | 126                 | 11.19 GiB |
| 4                        | 6.1                  | 62                  | 140                 | 14.10 GiB |

Throughput does not move, because throughput is set by arrivals and the device
was never the constraint. The interactive model's p95 rises by 57% and the bill
is 8.7 GiB. Push the vision rate to twelve a second, close to what this
configuration can serve, and the shape is the same: no extra throughput, text p50
from 63 ms to 120 ms.

The ceiling on what extra instances can ever buy is the fraction of a request that
is not device time — deserialising, the copy on, building the response. Divide it
out of your service curve before you spend gigabytes finding out.

## the batch you configure is not the batch you get

`preferred_batch_size` is a wish. The batcher can only group requests that are in
the queue at the same moment, and at interactive rates on a model that takes
hundreds of milliseconds, they are not.

| vision arrivals | mean batch | largest batch | req/s served | device busy | steady |
|-----------------|------------|---------------|--------------|-------------|--------|
| 2/s             | 1.0        | 2             | 2.1          | 63%         | yes    |
| 6/s             | 1.1        | 4             | 6.1          | 81%         | yes    |
| 12/s            | 1.9        | 8             | 12.2         | 97%         | yes    |
| 20/s            | 7.8        | 8             | 18.2         | 100%        | no     |
| 30/s            | 7.9        | 8             | 18.2         | 100%        | no     |

Batches of eight only appear once the device is already past what it can serve.
Every row where the configuration is viable forms batches of one or two. That is
the number the memory reservation should be sized from, and it appears nowhere in
the configuration file.

The queue delay follows the same logic from the other side:

| max_queue_delay | p50 at 2/s | at 20/s | at 100/s | at 300/s |
|-----------------|------------|---------|----------|----------|
| 0 ms            | 27         | 27      | 47       | 100      |
| 0.5 ms          | 28         | 28      | 47       | 99       |
| 5 ms            | 32         | 32      | 47       | 99       |

At two requests a second a 5 ms delay is 5 ms of pure waiting for company that
never arrives. At a hundred it is free, because by then the queue is never empty
and the request would have waited anyway. The crossover is a property of
utilisation, not of the batch size you asked for. The vision path in this
repository started at 10 ms and every uncontended image search paid all of it.

Both tables come from `bench/bench_frontier.py` and are the scheduler model's
output, not measurements of a GPU. Which brings us to whether the model is worth
anything.

## does the model agree with a real server

A queueing model that has only been compared against itself is a story.
`bench/bench_scheduler.py` runs the same traffic twice: once through `simulate()`,
and once over real HTTP against a server with real threads, a real socket, a real
dynamic batcher and a single device held for the modelled service time. It runs
on CPU in about four minutes, needs no GPU, and the results are in
`bench/results/scheduler-validation.json`.

Across four configurations and three models, twelve comparisons:

| quantity | median error | worst |
|----------|--------------|-------|
| p50, raw | 9.0%         | 10.3% |
| p95, raw | 6.1%         | 10.2% |
| p50, less measured overhead | 4.1% | 6.5% |
| p95, less measured overhead | 3.7% | 8.1% |

The model is consistently *optimistic*, by a median of 7.9%, and the gap is a
fixed 2.4 ms of HTTP and wakeup granularity that the model does not pretend to
include. It is a bias, not scatter — which matters, because the model is for
comparing configurations against each other, and a constant offset cancels.

Finding that number took two runs. The first reported 45 ms of overhead, which
was not overhead: the test server wrote its headers and its body as two separate
sends, and the second one sat waiting for a delayed ACK. Forty milliseconds,
flat, on every idle request over loopback, looking exactly like a slow model. If
you write your own harness, set `disable_nagle_algorithm` before you believe
anything it tells you.

## measuring your own hardware

Everything above uses `profiles/example-20gb.json`, which is illustrative and says
so in its own notes field. Two of its numbers are anchored to a real 20 GB
Ada-generation card; the rest is arithmetic from parameter counts. Do not size a
machine from it.

```bash
gpubudget bench example --url http://localhost:8000 --out profiles/mine.json
gpubudget simulate model_repository/ --profile profiles/mine.json \
    --traffic siglip2_text=30 --traffic bge_m3=30 --traffic siglip2_vision=6
gpubudget plan model_repository/ --profile profiles/mine.json --protect siglip2_text \
    --traffic siglip2_text=30 --traffic bge_m3=30 --traffic siglip2_vision=6
```

`bench` sweeps batch sizes sequentially on an idle device and fits each model to
an intercept and a slope. The intercept is what a batch costs before it has any
samples in it, the slope is the marginal sample, and the ratio between them tells
you whether batching is worth any queue delay at all. `plan` enumerates instance
counts and batch ceilings, drops the ones that will not fit or will not keep up,
and ranks what is left by the tail of the model you nominate.

`bench/bench_triton.py` does all three phases against a live server and writes a
measured-against-modelled table. Nothing in this repository ships results from
it, because somebody else's GPU is not useful to you.

There is no GPU here either way:

```bash
gpubudget serve model_repository/ --profile profiles/example-20gb.json --port 8000
gpubudget soak model_repository/ --url http://localhost:8000 \
    --traffic siglip2_text=30 --traffic siglip2_vision=6 --rates 1,2,3
```

`serve` runs a server that speaks the KServe v2 protocol with synthetic timings,
which is enough to exercise the harness end to end.

## from python

Everything the command line does is a library call. Four things are worth knowing
before you start: the example repository is importable, service curves come off
the profile by name, the percentiles live on `ModelResult.latency` rather than on
the result itself, and a config can be changed in memory instead of on disk.

```python
import gpubudget as gb

specs = gb.load_repository(gb.EXAMPLE_REPOSITORY)  # or "model_repository", or your own path
profile = gb.load_profile(gb.EXAMPLE_PROFILE)      # or the profiles/mine.json that 'bench' wrote
curves = {spec.name: profile.curve(spec.name) for spec in specs}

# 1. what it holds, and whether it survives a restart, which is a second question
budget = gb.estimate_budget(specs, profile.device, profile.memory)
print(f"at rest {gb.human_bytes(budget.steady_bytes)} of {gb.human_bytes(budget.ceiling_bytes)}")
print(f"loading {gb.human_bytes(budget.load_peak_bytes)}   fits={budget.fits} loads={budget.loads}")

# 2. what the tail looks like once they share the card
traffic = (
    gb.ModelTraffic("siglip2_text", 30.0),
    gb.ModelTraffic("bge_m3", 30.0),
    gb.ModelTraffic("siglip2_vision", 6.0),
)
workload = gb.Workload(traffic=traffic, duration_s=120.0, warmup_s=10.0, seed=11)
result = gb.simulate(specs, curves, workload)
text = result.by_name("siglip2_text")  # a ModelResult; the percentiles are on .latency
print(
    f"text {text.throughput_per_s:.1f}/s  "
    f"p50 {text.latency.p50_ms:.0f} ms  p95 {text.latency.p95_ms:.0f} ms  "
    f"device busy {result.gpu_busy_fraction:.0%}"
)

# 3. what dropping the heavy model to one instance would do, without touching a file
lean = [gb.apply_config(s, instances=1) if s.name == "siglip2_vision" else s for s in specs]
lean_text = gb.simulate(lean, curves, workload).by_name("siglip2_text")
lean_budget = gb.estimate_budget(lean, profile.device, profile.memory)
print(
    f"one vision instance: text p95 {lean_text.latency.p95_ms:.0f} ms  "
    f"at rest {gb.human_bytes(lean_budget.steady_bytes)}  loads={lean_budget.loads}"
)
```

```
at rest 8.28 GiB of 18.40 GiB
loading 19.93 GiB   fits=True loads=False
text 30.1/s  p50 61 ms  p95 112 ms  device busy 81%
one vision instance: text p95 90 ms  at rest 5.36 GiB  loads=True
```

`fits=True loads=False` is the whole first section in two booleans, and the last
line is the whole second one: the instance you removed was not buying throughput,
it was buying 2.9 GiB and 22 ms of tail. The file is `examples/from_python.py`
and it runs as it stands.

`gb.plan(...)` does the search behind `gpubudget plan`, `gb.run_load(...)` is the
open-loop harness behind `bench` and `soak`, and `gb.budget_table`, `gb.sim_table`
and `gb.plan_table` render exactly what the CLI prints.

## what did not work

**A closed-loop harness.** The first version ran N threads in a loop, each
sending the next request when the last reply came back. It cannot produce a
queue: when the server slows down the harness slows down with it, and the tail
flattens exactly where the real one takes off. The harness here fixes the send
schedule before the run starts and times every request from when it was
*scheduled*, not from when a worker got to it. It reports the difference, and
refuses to pretend the numbers are about the server when they are about itself.

**JSON tensors.** A batch of four 384×384 images is 1.7 million floats, and
`json.dumps` on that takes longer than the inference. A harness on the JSON path
measures its own encoder and concludes that batching does not help, which is the
opposite of the truth. The client here implements the binary tensor extension,
which is about forty lines and the difference between a benchmark and a fiction.

**TensorRT for the multilingual encoder.** The ONNX export fuses attention into
`MultiHeadAttention`, `SkipLayerNormalization` and `BiasGelu`, which are ONNX
Runtime contrib operators with no TensorRT plugin. The provider partitions around
them, builds an engine per fragment, and ends up slower than the provider it
replaced while using more memory. It stays on the CUDA provider, and the tool
warns if you try.

**CUDA graphs on the vision tower with `count: 2`.** Each capture pins its own
memory pool, and capturing concurrently on an engine that size takes the server
down during model load rather than under traffic. The saving was single digit
milliseconds on a call of over a hundred. It is off, and the budget warns about
the combination.

**Trusting a GPU-name lookup table for capacity.** The same part number ships
with different clocks and different memory, and the driver changed its allocator
between releases. Measure the card in front of you; that is what `bench` is for.

## failure modes this is looking for

Over the ceiling at steady state does not fail cleanly. The server starts, serves
single requests, and returns 500s only on the batches large enough to need the
memory that is not there — so the error rate correlates with your traffic and
nothing reproduces in staging.

Over the ceiling at load time fails cleanly but only on a cold cache, which means
it works in development, works on the first deploy after you warmed the cache by
hand, and fails on the node that comes up in the middle of the night.

TensorRT with dynamic input dimensions and no pinned optimisation profile builds
a fresh engine for every distinct sequence length it sees and keeps all of them.
Resident memory climbs for hours and then plateaus somewhere nobody predicted.
The multilingual encoder here has `dims: [-1]`, which is the worst case for this,
and it is a second reason it is not on TensorRT.

`dynamic_batching` with no `preferred_batch_size` falls back to `max_batch_size`.
One burst forms a batch that big, the arena grows to fit it, and it stays that way
until the server restarts.

And the one that is not a failure mode so much as a wasted week: a run where the
device is more than about 95% busy. Past that the queues never drain and every
percentile you measure is a property of how long you ran the benchmark. Both the
simulator and the load harness say so rather than printing a number.

## what this does not model

Multi-GPU, MIG and MPS. Sequence models with a KV cache, where the activation
estimate is wrong by a lot rather than by a third. Ensembles and the BLS
backend, where one request fans out into several. Pipeline parallelism. And the
device model is an approximation either way: kernels from different streams
neither serialise completely nor overlap freely, so `simulate()` offers `shared`
and `exclusive` and you should run both and see which tracks your hardware.

The activation estimator is shape arithmetic and is worth about ±30%. It is
marked `(est)` in every table until you replace it with a measurement, and the
tool says so in its warnings rather than letting you find out.

## layout

```
src/gpubudget/     pbtxt parser, budget, scheduler, load harness, cli
src/gpubudget/data/  the same configs and profile, shipped in the wheel as `example`
model_repository/  the three configs, with weights.json so CI can budget them
profiles/          calibration files; the shipped one is illustrative
bench/             the scripts that produced every number above
examples/          three short scripts, including the one in 'from python'
tests/             154 tests, no network
```

## licence

Apache-2.0.
