Metadata-Version: 2.4
Name: velocityllm
Version: 0.1.0
Summary: SLA-aware request scheduling for LLM serving, with a mock engine, an API server and benchmark tools
Project-URL: Repository, https://github.com/kartikpagariya25/VelocityLLM
Keywords: llm,inference,scheduler,vllm,sla,gpu
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: fastapi
Requires-Dist: uvicorn[standard]
Requires-Dist: pydantic>=2
Requires-Dist: aiohttp
Requires-Dist: pyyaml
Provides-Extra: gpu
Requires-Dist: vllm; extra == "gpu"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: httpx; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"

# VelocityLLM

An API server that sits in front of an LLM engine and decides **which requests to accept, in what order, and how many to run at once**, so that answers arrive inside a time promise (an SLA) instead of everything slowing down together.

It ships with:

- **`static`**: a fixed number of requests at a time, first come first served. A simple baseline.
- **`dynamic`**: watches the waiting line, the time promise and GPU memory, changes how many requests run at once, and says "no" early (with a reason) when an answer cannot arrive in time.
- A **mock engine**, so you can try everything on a normal computer with no GPU.
- Test-traffic tools and a benchmark service for comparing the policies.

## Install

```bash
pip install velocityllm
```

This installs the server, the mock engine and the tools. It does **not** install vLLM, because vLLM needs an NVIDIA GPU. For real models:

```bash
pip install "velocityllm[gpu]"
```

Requires Python 3.11 or newer.

## Quick start (no GPU)

```bash
velocityllm serve --policy dynamic --mock --port 8000
```

In a second terminal:

```bash
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Hello", "max_tokens": 20}'

curl http://localhost:8000/stats
```

The mock engine returns made-up text at a realistic speed. It is for trying the scheduler, not for real answers.

## Real model on a GPU

```bash
pip install "velocityllm[gpu]"
velocityllm serve --policy dynamic --model-path /path/to/model --sla-ms 8000
```

You need an NVIDIA GPU and the model files on disk (a folder with `config.json` and the weights).

## Commands

| Command | What it does |
|---|---|
| `velocityllm serve` | Runs the scheduler server (`--policy static` or `dynamic`, `--mock`, `--sla-ms`, `--port` ...) |
| `velocityllm loadgen` | Sends test traffic to a running server (`--num-requests`, `--pattern`, `--mixed-prompts` ...) |
| `velocityllm arena` | Runs the benchmark service behind the Arena dashboard. Only the API is included in this package; the web page is in the repository |

Add `--help` to any command to see all options.

## API

| Endpoint | Purpose |
|---|---|
| `POST /generate` | Send a prompt, get an answer, with queue time and whether the SLA was met |
| `POST /generate/stream` | Same, as a stream of tokens |
| `POST /v1/completions` | OpenAI-style completions |
| `GET /stats` | Live numbers: active, queued, accepted, rejected, latency percentiles |
| `GET /metrics` | The same numbers in Prometheus format |
| `GET /health` | Is the server alive |

A request the scheduler cannot serve in time is refused with HTTP 429 and a reason.

## Good to know

- Results depend on your GPU, model and traffic. Test on your own setup before relying on any policy.
- Only a normal computer is needed for the mock engine; the real-GPU path needs NVIDIA hardware and model weights.
- This is an early release (version 0.1).

## Source and issues

https://github.com/kartikpagariya25/VelocityLLM
