Metadata-Version: 2.1
Name: gguf-server
Version: 0.0.2
Summary: Local OpenAI-compatible LLM server GUI for GGUF models with a bundled llama-server engine
Author-Email: gguf <info@gguf.us>
License: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Project-URL: Repository, https://github.com/gguf-org/gguf-desktop
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# gguf-server

Local LLM server GUI for GGUF models, packaged for Python. The GUI runs in
your browser against a local backend; inference is served by llama.cpp's
`llama-server`, compiled during `pip install` and bundled with the package as
a single binary. Model and template files are referenced by filesystem path
through a built-in file browser — nothing is uploaded or copied to temp
storage.

## Install

```bash
pip install gguf-server
```

Building the bundled engine requires a C/C++ toolchain and CMake ≥ 3.15 (on
Windows: MSVC Build Tools). The engine source is resolved from the vendored
`vendor/llama.cpp` copy (see `scripts/vendor_engine.py`), a sibling
`../llama.cpp` checkout, or `GGUF_SERVER_ENGINE_DIR`. Only `llama-server` and
the libraries it links against are compiled — the rest of llama.cpp's tools,
tests and examples are skipped.

### GPU and accelerator backends

The default build is CPU-only. Backends are opt-in and can be requested with
an environment variable or a CMake define — the env var is usually easier to
pass through `pip`:

```bash
GGUF_SERVER_CUDA=1   pip install gguf-server    # NVIDIA (needs the CUDA toolkit)
GGUF_SERVER_HIP=1    pip install gguf-server    # AMD (needs ROCm/HIP)
GGUF_SERVER_VULKAN=1 pip install gguf-server    # cross-vendor (needs the Vulkan SDK)
GGUF_SERVER_SYCL=1   pip install gguf-server    # Intel oneAPI

CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON" pip install gguf-server   # equivalent
```

Available: `CUDA`, `HIP`, `METAL`, `VULKAN`, `SYCL`, `OPENCL`, `MUSA`,
`WEBGPU`, `ZDNN`, `OPENVINO`, `HEXAGON`, `BLAS`, `RPC`. Each maps to the ggml
option of the same name; anything not listed can still be passed straight
through as `-DGGML_<NAME>=ON`. **On macOS, Metal is on by default** (with the
shader library embedded in the binary) — no flag needed.

CUDA builds compile kernels for many GPU architectures and can take a long
time; restrict them to your own card to speed this up, e.g.
`CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89"`. The GPU
runtime libraries (cuBLAS, ROCm, …) are linked dynamically and come from the
toolkit you built against, so keep it installed.

### HTTPS / OpenSSL

Off by default. `llama-server` only needs OpenSSL to *download* models over
HTTPS (`-hf` / URL arguments); this package always hands it local file paths.
Leaving it off also avoids a build break on Windows: llama.cpp's
`find_package(OpenSSL)` can pick up a foreign-ABI install — an MSYS2/MinGW
`libcrypto` alongside an MSVC toolchain puts MinGW headers on `cl.exe`'s
include path and fails `httplib.cpp`, `download.cpp` and `hf-cache.cpp` with
`winnt.h: fatal error C1189: No supported target architecture`. Re-enable
with `GGUF_SERVER_OPENSSL=1` if you want in-server HTTPS downloads.

### Embedded web UI

`llama-server`'s own web UI (served at the model endpoint) is fetched prebuilt
when the network allows. Pass `-DGGUF_SERVER_BUILD_UI=ON` to build it from
source with npm instead, or ignore it entirely — this package ships its own
GUI and the LLM server works either way.

### Windows notes

Both toolchains work; **which one you get depends on your shell**, so check the
`CMAKE_CXX_COMPILER` line in the configure output if a build misbehaves.

- **MSVC** — build from a Developer Command Prompt (`vcvars64.bat`) so `cl.exe`
  is on PATH. This is llama.cpp's primary Windows toolchain.
- **MinGW / MSYS2** — used automatically when `C:\msys64\mingw64\bin` is on
  PATH and MSVC is not. Supported here: the build defines
  `_WIN32_WINNT=0x0A00` (MinGW otherwise pins it to a Windows 7-era value,
  which makes the vendored cpp-httplib hard-error with *"doesn't support
  Windows 8 or lower"* and leaves `::CreateFile2` undeclared), and links the
  MinGW runtime statically so the binary doesn't need `libstdc++-6.dll`,
  `libgcc_s_seh-1.dll` or `libwinpthread-1.dll` beside it. Turn the latter off
  with `GGUF_SERVER_MINGW_STATIC=0`.

Two environment gotchas worth knowing:

- `'ninja' ... inappropriate file type or format` — a pyenv-win shim is
  shadowing the real ninja. Point CMake at a real one, e.g.
  `-DCMAKE_MAKE_PROGRAM="C:/Program Files (x86)/Microsoft Visual Studio/<ver>/BuildTools/Common7/IDE/CommonExtensions/Microsoft/CMake/Ninja/ninja.exe"`.
- A failed configure leaves a cache behind in `build/`, and later builds reuse
  it. Delete the directory (`rmdir /s /q build`) before retrying after you
  change toolchain or options.

## Usage

```bash
python -m gguf_server              # launch the GUI in the browser
gguf-server                        # same
gguf-server --port 8642 --no-browser
```

The GUI listens on port 8642; the LLM server it launches defaults to 8888.

GUI features (as in the desktop app's LLM panel):

- **Server** — start/stop, live status, copyable API base URL, active
  configuration summary, OpenAI-compatible endpoint reference, and a
  copyable/editable `llama-server` command line
- **Model** — `.gguf` model file, optional `--mmproj` vision projector, and
  chat template selection (auto-detect from metadata, file, or raw Jinja2)
- **Settings** — host/port, API key, model alias, context length, GPU layers,
  main GPU, per-GPU tensor split sliders, CPU threads, parallel slots, batch
  and micro-batch sizes, KV cache types, flash attention, continuous
  batching, mlock, mmap and verbose logging, plus hardware-derived
  "Recommended" defaults
- **Presets** — save the whole configuration, with JSON export/import
- **Hardware** — CPU, RAM/VRAM meters and per-GPU details, with live polling
- **Logs** — tail of the running server's log with auto-scroll

Engine CLI passthrough (runs the bundled `llama-server` binary):

```bash
python -m gguf_server engine -- --model model.gguf --port 8888
gguf-server engine -- --help
```

## How it works

- `pip install` compiles `llama-server` (static ggml + llama + common linked
  into one executable) via scikit-build-core and installs it into the
  package's `bin/` directory.
- `gguf-server` starts a stdlib HTTP server (default port 8642) serving the
  static GUI and a small JSON API, and opens the browser.
- Starting the server spawns one `llama-server` child process with its output
  redirected to a log file; a watcher thread polls the TCP port until it
  accepts connections, then reports the endpoint to the GUI. Stopping the GUI
  stops the child.
- File selection uses a server-side directory listing (`/api/browse`) so the
  GUI gets real filesystem paths — no drag & drop uploads of multi-GB models.
