Metadata-Version: 2.4
Name: velocityllm
Version: 0.4.0
Summary: SLA-aware dynamic request scheduling for LLM serving on vLLM, with a built-in terminal benchmark
Author-email: Kartik Pagariya <kartik.profession@gmail.com>
License-Expression: MIT
Project-URL: Repository, https://github.com/kartikpagariya25/VelocityLLM
Keywords: llm,inference,scheduler,vllm,sla,gpu,benchmark
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi
Requires-Dist: uvicorn[standard]
Requires-Dist: pydantic>=2
Requires-Dist: aiohttp
Requires-Dist: pyyaml
Requires-Dist: Pillow
Provides-Extra: gpu
Requires-Dist: vllm; extra == "gpu"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: httpx; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# VelocityLLM

SLA-aware dynamic scheduling for LLM serving on vLLM. VelocityLLM sits in front of your model, decides how many requests the GPU should run at once, and keeps latency inside the promise you set. It ships with a terminal benchmark so you can see the difference on your own hardware.

## Install

```bash
pip install velocityllm            # scheduler, benchmark, simulated engine
pip install "velocityllm[gpu]"     # adds vLLM for real models on an NVIDIA GPU
```

Python 3.11 or newer.

## Compare static and dynamic scheduling on your models

Put each model in its own folder under `./models`, then:

```bash
velocityllm bench --models-dir ./models --requests 100 --repeats 3 --sla-ms 8000
```

Every model is served twice, once with a fixed batch size (static) and once with VelocityLLM (dynamic), under the same traffic. The table shows p50, p95 and p99 latency, tokens per second and the share of requests answered inside the SLA, with the change against static on a line of its own. Results are saved as CSV and JSON in `./velocity_bench/latest`.

Pick models and policies explicitly:

```bash
velocityllm bench --models-dir ./models --only qwen2.5-1.5b,llama-3.2-1b,gemma-2-2b --policies static,dynamic,smart
velocityllm bench --model qwen=/data/qwen2.5-1.5b --model llama=/data/llama-3.2-1b
velocityllm bench --mock --models-dir ./models      # no GPU, simulated engine
```

## Use it from Python

```python
import velocityllm

rows = velocityllm.benchmark(
    {"qwen2.5-1.5b": "/data/qwen2.5-1.5b", "gemma-2-2b": "/data/gemma-2-2b"},
    requests=100, repeats=3, sla_ms=8000,
)
for r in rows:
    print(r["model"], r["policy"], r["p50"], r["p95"], r["p99"], r["tokens_per_sec"])
```

## Serve a model

```bash
velocityllm serve --policy dynamic --model-path /data/qwen2.5-1.5b --sla-ms 8000
velocityllm serve --mock                                  # try it without a GPU
velocityllm loadgen --url http://localhost:8000/generate --num-requests 100 --concurrency 100 --pattern flood
```

Policies: `static` (fixed concurrency baseline), `dynamic` (SLA-aware admission with an adaptive concurrency limit) and `smart` (token-aware, KV-cache-aware, explains every decision).

## Arena dashboard

```bash
velocityllm arena --models-dir ./models
```

Opens the web dashboard where you pick a model, a user count and an SLA and watch the policies run side by side.
