GGUF Server

Stopped

API Endpoint

http://127.0.0.1:8888/v1

Active Configuration

OpenAI-Compatible Endpoints

Point any OpenAI client at the base URL above. With an API key set, clients must send Authorization: Bearer <key>.

Command

Edit manually

The leading binary name is ignored — the bundled gguf-server engine always runs. Arguments are passed verbatim, including --host/--port.

Network

Host --host Port --port

Use 0.0.0.0 to accept connections from other devices. Default 8888 — vLLM uses 8000, llama.cpp 8080, LM Studio 1234, Ollama 11434.

API Key --api-key Model Alias --alias

The alias is the name reported by /v1/models. Blank uses the file name.

Performance

Detecting hardware…

Context Length --ctx-size

Maximum context window. Higher values need more VRAM/RAM.

GPU Layers --n-gpu-layers

Layers offloaded to the GPU. 0 = CPU only, 999 = all possible.

Main GPU --main-gpu

GPU index used for scratch buffers and the final layers.

Tensor Split `--tensor-split`

Auto

Layer allocation ratio across GPUs.

CPU Threads --threads

Threads used for generation. 0 lets the engine choose.

Parallel Slots --parallel

Simultaneous request slots for server batching.

Batch Size --batch-size Micro Batch Size --ubatch-size

Advanced

KV Cache — Keys --cache-type-k KV Cache — Values --cache-type-v

q8_0 saves VRAM with minimal quality loss.

Flash attention --flash-attn faster attention on compatible GPUs Continuous batching --cont-batching serve multiple requests in one pass Memory lock --mlock pin weights in RAM, no swapping Disable mmap --no-mmap load fully into RAM instead of mapping Verbose logs --verbose detailed request and JSON logging

Settings are saved automatically

No API key |(no model) |http://127.0.0.1:8888/v1 OpenAI-compatible

Stopped

Active Configuration

OpenAI-Compatible Endpoints

Command

GGUF Model File

Chat Template

Network

Performance

Tensor Split `--tensor-split`

Advanced

Save current setup

Saved presets

Hardware Monitor

CPU

Total Memory Capacity

GPU Devices

Server log

Stopped

Active Configuration

OpenAI-Compatible Endpoints

Command

GGUF Model File

Chat Template

Network

Performance

Tensor Split --tensor-split

Advanced

Save current setup

Saved presets

Hardware Monitor

CPU

Total Memory Capacity

GPU Devices

Server log

Tensor Split `--tensor-split`