| model | size | params | backend | threads | test | t/s |
|---|---:|---:|---|---:|---|---:|
| Models | 216.80 MiB | 361.82 M | MemVanta CPU | 4 | pp32 | 58.12 ± 0.00 |
| Models | 216.80 MiB | 361.82 M | MemVanta CPU | 4 | tg16 | 32.78 ± 0.00 |
TTFT(126 input): 2415.94 ± 0.00 ms; output TPS(2): 32.54 ± 0.00
load=27.61 ms peak_rss=244.12 MiB kv_alloc=5.00 MiB batch=16 kv=f16
