| vLLM version | {% if snap.vllm_version %}{{ snap.vllm_version }}detected{% else %}not reported recommendations assume the newest tested vLLM{% endif %} |
|---|---|
| Served model | |
| Tokenizer |
{% if model.hf_id %}detected{% else %} Couldn't map the model's path to Hugging Face; enter its id. {% endif %} |
| Context length | {{ "{:,}".format(model.max_model_len) if model.max_model_len else "unknown" }}{% if model.max_model_len %}detected{% endif %} |
| KV cache | {{ "{:,}".format(snap.cache.kv_cache_size_tokens) }} tokens · {{ snap.cache.gpu_memory_utilization }} memory utilization · {{ snap.cache.cache_dtype }} dtypedetected |
| Prefix caching | {{ "on" if snap.cache.enable_prefix_caching else "off" }}detected |
| KV cache | not visible /metrics is unreachable, so memory findings will be estimates |
| Distance | {{ "%.0f"|format(snap.network.http_rtt_ms or 0) }} ms round trip
{% if (snap.network.http_rtt_ms or 0) > 20 %} Far from the server: time per token stays reliable, and first-token times are judged without the network. {% endif %} |