currently serving
TTFT p99
ms
TTFT p99
ms
/ SLA ms
TPOT p99
ms
/ SLA ms
E2E p99
ms
/ SLA ms
·
| *** |
·
ms
p50
p95
p99
ms
p50
p95
p99
tok/s
tok/s
KV cache
%
%
%
%
%
CUDA padding
%
ms
ms
ms
—
Compute roof
Memory roof
Compute roof
Memory roof
（ · ）·
Compute roof = ·
Memory roof = ·
= FLOP/byte
()
B=: TFLOPs/s
→
| Kernel | |||||
|---|---|---|---|---|---|
|
|
|
|||
|
·
-lineinfo
↗ （）
→
|
|||||
🟢 ·
%
/s
µs
%
%
%
str
0 ms
(
end−start) ·
|
||||||
▶
TTFT p99 ≤
ms
TPOT p99 ≤
ms
E2E p99 ≤
ms
·
/
| TTFT | · p50 / p95 / p99 |
| TPOT | · p50 / p95 / p99 |
| E2E | · p50 / p95 / p99 |
| Output | |
| Input | |
| Tokens | out · in |
| Endpoint | |
| Model | |
| API | |
| Prompt / Output | / tok |
| Concurrency | |
| Bounded by |