Metadata-Version: 2.5
Name: vkserve
Version: 0.1.19
Summary: An OpenAI-compatible LLM server that runs several models on any Vulkan GPU, built on compute-ml
Project-URL: Homepage, https://gitlab.com/cardona-oss/vkserve
Project-URL: Repository, https://gitlab.com/cardona-oss/vkserve
Project-URL: Documentation, https://cardona-oss.gitlab.io/vkserve/
Project-URL: Changelog, https://gitlab.com/cardona-oss/vkserve/-/blob/main/CHANGELOG.md
Project-URL: Issues, https://gitlab.com/cardona-oss/vkserve/-/issues
Author-email: Jorge Cardona <jorgeecardona@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: huggingface-hub>=0.26
Requires-Dist: jinja2>=3.1
Requires-Dist: litestar>=2.12
Requires-Dist: tokenizers>=0.20
Requires-Dist: uvicorn>=0.30
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocs>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.27; extra == 'docs'
Description-Content-Type: text/markdown

# vkserve

An OpenAI-compatible LLM server that runs several models on any Vulkan GPU, built on
[compute-ml](https://gitlab.com/cardona-oss/compute-ml). Status: prototype. The wheels on PyPI (Linux x86_64 and macOS arm64) include the C engine; other systems build it from source (see Install).

## Install

On Linux x86_64, `pip install vkserve` is enough. The wheel includes the C engine. It needs the
system Vulkan loader (`libvulkan.so.1`) and a GPU driver.

On macOS arm64, `pip install vkserve` is enough too. The wheel bundles MoltenVK (Vulkan on Metal)
and the Vulkan loader, so it needs no Vulkan SDK. Set `VK_DRIVER_FILES` to use another driver.

Other systems build the engine from a source checkout:

```bash
git clone https://gitlab.com/cardona-oss/vkserve.git
git clone https://gitlab.com/cardona-oss/compute-ml.git
cd vkserve
make engine
```

`make engine` needs compute-ml `071d587` (0.4.73) or newer. It stops with a message that names
the missing function when the checkout is older. Update it with `git -C ../compute-ml pull`.

Or point `VKSERVE_ENGINE_LIB` at an engine library you built elsewhere.

## Run it

1. Build the C engine. It compiles compute-ml from `../compute-ml`.

   ```bash
   make engine
   ```

2. Start the server on a model folder or a Hugging Face id. The id is downloaded on first use.

   ```bash
   uv run vkserve Qwen/Qwen3-0.6B --dtype q4 --slots 1 --max-len 3000
   ```

   The model id in the API is the last part of the name (`Qwen3-0.6B`); `--name` changes it.

   Pick the GPU with `--device` (an index or part of the name). `vkserve --list-devices` shows
   them. Repeat `--device` to run the model on several GPUs: requests go to the least busy one,
   and a GPU that cannot load the model is skipped.

   ```bash
   uv run vkserve --list-devices
   uv run vkserve Qwen/Qwen3-0.6B --device intel --device t2000
   ```

   Models: Qwen3, Llama 3.2, SmolLM2, Gemma 3 and Gemma 4, as safetensors with a
   `tokenizer.json`. A bare `.gguf` file is not supported yet. Several models in one server:
   repeat `--model name=ID,path=FOLDER_OR_HF_ID[,dtype=q8][,slots=4][,max_len=4096]`.

3. Send a request.

   ```bash
   curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' \
       -d '{"messages":[{"role":"user","content":"Name two planets."}],"max_tokens":30}'
   ```

## Use it from opencode

See [examples/opencode](https://gitlab.com/cardona-oss/vkserve/-/blob/main/examples/opencode/README.md): a config, a small agent, and the commands.

## Configuration

One TOML file, and every key is also a flag. Flags win over the file. See
[examples/vkserve.toml](https://gitlab.com/cardona-oss/vkserve/-/blob/main/examples/vkserve.toml).

```bash
uv run vkserve -t -c vkserve.toml                 # check the file, load nothing
uv run vkserve -c vkserve.toml                    # run it
uv run vkserve -c vkserve.toml --port 9000        # a flag overrides the file
uv run vkserve -c vkserve.toml --model qwen --slots 2   # change one model of the file
uv run vkserve --print-config -c vkserve.toml     # show every effective setting
```

- `[server]` keys are plain flags: `--host`, `--port`, `--log-level`, `--keep-going`,
  `--max-body-bytes`, `--pid-file`, `--log-file`, `--drain-timeout`.
- `--model NAME` starts a model block. The flags after it apply to that block: `--path`,
  `--dtype`, `--slots`, `--max-len`, `--prefill-chunk`, `--max-queue`, `--device` (repeat for
  several devices), `--temperature`, `--top-p`, `--top-k`.
- `--set server.port=9000` and `--set model.qwen.slots=2` reach any key.
- An unknown key is an error with a hint: `unknown key 'slot' ... (did you mean 'slots'?)`.

## Run it as a service

The master process serves HTTP. Each model on each device runs in its own worker process. A
worker that crashes fails its requests with 500 and starts again after 1, 2, 4 ... 30 seconds.
The master keeps running.

Send signals with `kill`, or with `-s` (it reads `pid_file`):

| Signal | `-s` | What it does |
|---|---|---|
| `SIGHUP` | `reload` | Read the configuration again. A new or changed model loads first, then the old one finishes its requests and stops. A bad file changes nothing. |
| `SIGTERM`, `SIGQUIT`, `SIGINT` | `stop`, `quit` | Stop accepting, wait up to `drain_timeout` for requests in progress, then exit. A second signal stops at once. |
| `SIGUSR1` | `reopen` | Reopen `log_file` (after logrotate moved it). |

`host`, `port`, `max_body_bytes`, `pid_file`, `drain_timeout` and `keep_going` need a restart.
A reload that changes them logs a warning and ignores them.

```bash
uv run vkserve -s reload -c vkserve.toml
```

A systemd unit is in [examples/vkserve.service](https://gitlab.com/cardona-oss/vkserve/-/blob/main/examples/vkserve.service).

## Features

- `POST /v1/chat/completions` and `/v1/completions`, streaming as server-sent events.
- Tool calls: `tools` in, OpenAI `tool_calls` out, `tool` messages back in.
- Several requests in one batched forward pass per step.
- Several GPUs: one worker per device, least-busy routing, failed workers skipped and restarted.
- Operations: `429` when the queue is full, `X-Request-ID`, Prometheus `/metrics` per device.
- Prefix reuse: a request that starts with a cached conversation prefills only the new tokens.
- Attention over long contexts (tested to 20,000 keys).

More: [docs/serving.md](https://gitlab.com/cardona-oss/vkserve/-/blob/main/docs/serving.md) (design and measurements).
