Metadata-Version: 2.4
Name: velocityllm
Version: 0.4.2
Summary: SLA-aware request scheduling for LLM serving, with a mock engine, an API server and benchmark tools
Project-URL: Repository, https://github.com/kartikpagariya25/VelocityLLM
Keywords: llm,inference,scheduler,vllm,sla,gpu
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: fastapi
Requires-Dist: uvicorn[standard]
Requires-Dist: pydantic>=2
Requires-Dist: aiohttp
Requires-Dist: pyyaml
Provides-Extra: gpu
Requires-Dist: vllm; extra == "gpu"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: httpx; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"

<div align="center">

<img src="https://raw.githubusercontent.com/kartikpagariya25/VelocityLLM/main/assets/logo.png?token=GHSAT0AAAAAAEJZV3Q4BCKH3PZQZMI3T3X42WECZKA" alt="VelocityLLM" width="420"/>

**SLA-aware dynamic request scheduling for LLM serving on vLLM**

[![PyPI](https://img.shields.io/pypi/v/velocityllm?color=FF6A1A)](https://pypi.org/project/velocityllm/)
[![Python](https://img.shields.io/badge/python-3.11%2B-FF6A1A)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-MIT-FF6A1A)](https://github.com/kartikpagariya25/VelocityLLM/blob/main/LICENSE)

</div>

---

## What is VelocityLLM

LLM servers fail in a predictable way under load. Requests pile up in a queue, latency grows, and eventually clients time out after the work has already been spent. A static batch size makes this worse: it wastes GPU capacity when traffic is light and overloads the system when traffic spikes.

VelocityLLM is a scheduler that sits in front of an LLM engine and controls three things: which requests are admitted, the order they run in, and how much concurrent work the system accepts. The goal is to keep latency within a service-level agreement (SLA) during bursts, and to decline work early when it cannot be finished in time, rather than accepting it and failing later.

The scheduler only looks at request metadata such as arrival time, token counts, and priority. It never inspects model internals, so it works with any model the underlying engine supports.

## Features

**SLA-aware admission control.** Each incoming request is checked against a predicted completion time before it enters the queue. If the request cannot meet its deadline, it is rejected immediately with HTTP 429 and a retry hint. The client learns right away instead of waiting for a timeout.

**Priority scheduling with anti-starvation.** Requests are ordered as high, normal, or low priority. A request's effective priority rises the longer it waits, so low-priority traffic is delayed under load but is never blocked indefinitely.

**Adaptive concurrency.** An AIMD (additive increase, multiplicative decrease) controller raises the concurrency limit while latency and memory headroom are healthy, and cuts it sharply when they are not. This tracks the workload without manual tuning.

**Failure handling.** GPU out-of-memory errors trigger an emergency throttle and recovery rather than a crash. Client disconnects, malformed input, and sudden bursts are handled explicitly, and the server keeps serving other requests.

**Automatic hardware and model detection.** The NVIDIA GPU is detected automatically. When no GPU is present, the simulated engine runs so that the full pipeline can be exercised on any machine. The model is located from a command-line flag, an environment variable, a local models folder, or an interactive prompt.

**Built-in load testing and benchmarks.** A load generator supports four arrival patterns, and comparison tools measure static and dynamic policies under identical workloads.

## Installation

VelocityLLM requires Python 3.11 or newer.

```bash
pip install velocityllm
```

Installing with the GPU extra adds vLLM, the inference engine used for real models on NVIDIA hardware:

```bash
pip install "velocityllm[gpu]"
```

Confirm the installation:

```bash
velocityllm version
velocityllm detect
```

The `detect` command reports which GPU was found and which model the server will use, so you can check the setup before starting anything.

## Quick start

Start the server with the simulated engine. This needs no GPU and no model download, so it works on any machine:

```bash
velocityllm chargeVelocity --mock --port 8000
```

In another terminal, check that it is running:

```bash
curl http://localhost:8000/health
```

Send a request:

```bash
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain request scheduling in one sentence", "max_tokens": 64}'
```

The response includes the generated text, the time spent queued and executing, the tokens per second, and whether the SLA was met.

To serve a real model on a GPU machine, point VelocityLLM at the model folder:

```bash
velocityllm chargeVelocity --model-path /path/to/model
```

Or set it once in the environment:

```bash
export VELOCITY_MODEL_PATH=/path/to/model
velocityllm chargeVelocity
```

## Model and GPU detection

When you run `chargeVelocity` or `serve`, VelocityLLM resolves the model in this order:

1. The `--model-path` flag, if given.
2. The `VELOCITY_MODEL_PATH` environment variable, if set.
3. The first folder containing a `config.json` file, searched in `./models` and then `~/velocity-models`.
4. If none of the above applies and you are in an interactive terminal, a prompt asks you to type the path.

GPU detection uses `nvidia-smi`. If it finds an NVIDIA GPU, the server reports the GPU name and memory at startup and uses the real engine. If it finds no GPU, the server prints a notice and switches to the simulated engine automatically, so nothing needs to be configured.

## Commands

| Command | Purpose |
|---|---|
| `velocityllm chargeVelocity` | Start the server with automatic GPU and model detection |
| `velocityllm serve` | Run the scheduler server with full control over options |
| `velocityllm detect` | Show the detected GPU and model without starting a server |
| `velocityllm loadgen` | Send test traffic to a running server |
| `velocityllm version` | Show the installed version |

Run `velocityllm <command> --help` for the full option list of any command.

## Scheduling policies

The `--policy` option selects how the server schedules work:

| Policy | Behavior |
|---|---|
| `dynamic` (default) | Adaptive concurrency with SLA-aware admission and priority scheduling |
| `static` | Fixed concurrency. Used as the baseline for comparing scheduler performance |
| `smart` | Dynamic scheduling with the extended smart-scheduler settings |

## Server options

| Option | Description |
|---|---|
| `--mock` | Use the simulated engine. No GPU required |
| `--model-path PATH` | Folder containing the model weights |
| `--host HOST` | Address to bind to. Default `0.0.0.0` |
| `--port PORT` | Port to listen on. Default `8000` |
| `--sla-ms MS` | Target latency deadline in milliseconds. Default `3000` |
| `--max-concurrency N` | Upper limit for the adaptive controller. Default `32` |
| `--api-key KEY` | Require the `X-API-Key` header on every request except `/health` |
| `--config FILE` | Load settings from a YAML file. Command-line flags take precedence |
| `--structured-logs` | Write JSON log lines for production log collection |

## Configuration file

Long-running deployments are easier to manage with a configuration file:

```yaml
policy: dynamic
model_path: ""          # empty means auto-detect
port: 8000
sla_ms: 8000
max_concurrency: 32
max_model_len: 4096
api_key: change-me
```

Start the server with it:

```bash
velocityllm serve --config config.yaml
```

## API

| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/health` | Server status, active policy, and backend in use |
| `POST` | `/generate` | Submit a prompt and receive the response with timing data |
| `GET` | `/stats` | Scheduler statistics, including admissions, rejections, and queue state |

A successful `/generate` response has this shape:

```json
{
  "request_id": "req-c7a3b7997d85",
  "response": "...",
  "latency_seconds": 0.196,
  "queue_time_seconds": 0.002,
  "execution_time_seconds": 0.194,
  "tokens_generated": 14,
  "tokens_per_second": 72.1,
  "sla_met": true,
  "priority": "normal"
}
```

When the system cannot meet a request's SLA, it returns HTTP 429 with a retry hint instead of accepting work it cannot finish.

## Load testing

The load generator sends traffic to a running server using one of four arrival patterns:

```bash
velocityllm loadgen --url http://localhost:8000 --num-requests 200 --concurrency 16 --pattern poisson
```

| Pattern | Traffic shape |
|---|---|
| `flood` | All requests sent at once |
| `constant` | Steady request rate |
| `poisson` | Random arrivals with a constant average rate |
| `burst` | Periodic spikes separated by quiet periods |

Use the same pattern against `--policy static` and `--policy dynamic` to see how each scheduler handles the same workload.

## Requirements

- Python 3.11 or newer
- An NVIDIA GPU with CUDA drivers, for real model inference (optional)
- The `[gpu]` extra, which installs vLLM (optional)

Without a GPU, every feature except real model inference works, using the simulated engine.

## Links

- Source code and full documentation: https://github.com/kartikpagariya25/VelocityLLM
- Issues and feature requests: https://github.com/kartikpagariya25/VelocityLLM/issues
- License: MIT
