✓ 0ms Time-to-First-Token (TTFT):
Request B bypassed prompt prefill entirely for the first 32 tokens, achieving instant token streaming because key-value activations were already live in GPU VRAM!
Run continuous-batching inference and benchmark PagedAttention locally
Installation:
pip install mini-infer
Start High-Throughput OpenAI-Compatible Server:
# Spin up local engine with PagedAttention and Radix Prefix Caching
mini-infer serve --model Qwen/Qwen2.5-0.5B-Instruct --port 8000 --block-size 16 --enable-prefix-cache
Python API Usage:
from mini_infer.engine import LLMEngine
from mini_infer.cache.block_pool import BlockPool
from mini_infer.cache.prefix_cache import PrefixCache
engine = LLMEngine.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", enable_prefix_caching=True)
response = engine.generate("Explain how PagedAttention maps logical blocks to physical memory.")
print(response.text)