Metadata-Version: 2.1
Name: gguf-server
Version: 0.0.3
Summary: Local OpenAI-compatible LLM server GUI for GGUF models with a bundled C/C++ inference engine
Author-Email: gguf <info@gguf.us>
License: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Project-URL: Repository, https://github.com/gguf-org/gguf-desktop
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# gguf-server

Local LLM server GUI for GGUF models, packaged for Python. The GUI runs in
your browser against a local backend; inference is served by the standalone
`gguf-server` C/C++ engine, compiled during `pip install` and bundled with the
package as a single binary. Model and template files are referenced by
filesystem path through a built-in file browser — nothing is uploaded or
copied to temp storage.

## Install

```bash
pip install gguf-server
```

Building the bundled engine requires a C/C++ toolchain and CMake ≥ 3.15 (on
Windows: MSVC Build Tools). The engine source is resolved from the vendored
`vendor/server` copy (see `scripts/vendor_engine.py`), a sibling `../server`
checkout, or `GGUF_SERVER_ENGINE_DIR`. That tree is self-contained — it carries
its own ggml kernels, GGUF runtime, common layer and HTTP server, with no
llama.cpp checkout and no external ggml — so the build compiles the server
binary and nothing else.

### GPU and accelerator backends

The default build is CPU-only. Backends are opt-in and can be requested with
an environment variable or a CMake define — the env var is usually easier to
pass through `pip`:

```bash
GGUF_SERVER_CUDA=1   pip install gguf-server    # NVIDIA (needs the CUDA toolkit)
GGUF_SERVER_HIP=1    pip install gguf-server    # AMD (needs ROCm/HIP)
GGUF_SERVER_VULKAN=1 pip install gguf-server    # cross-vendor (needs the Vulkan SDK)

CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON" pip install gguf-server   # equivalent
```

Available: `CUDA`, `HIP`, `VULKAN`, `METAL`. These are the kernels the engine
tree actually ships (`vendor/server/kernels/src`); the backends llama.cpp has
and this engine does not — SYCL, CANN, OpenCL, WebGPU, RPC, zDNN, ZenDNN,
OpenVINO, Hexagon, MUSA, BLAS — are simply absent. Each option maps to the
`ggml` option of the same name, and the finer-grained `GGML_*` knobs
(`GGML_NATIVE`, `GGML_AVX512`, `GGML_CUDA_FA_ALL_QUANTS`, …) can still be
passed straight through as `-DGGML_<NAME>=…`. **On macOS, Metal is on by
default** (with the shader library embedded in the binary) — no flag needed.

CUDA builds compile kernels for many GPU architectures and can take a long
time; restrict them to your own card to speed this up, e.g.
`CMAKE_ARGS="-DGGUF_SERVER_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89"`. The GPU
runtime libraries (cuBLAS, ROCm, …) are linked dynamically and come from the
toolkit you built against, so keep it installed.

### HTTPS / OpenSSL

Off by default here (the engine's own default is on). OpenSSL is only needed to
*download* models over HTTPS (`-hf` / URL arguments); this package always hands
the engine local file paths. Leaving it off also avoids a build break on
Windows: `find_package(OpenSSL)` can pick up a foreign-ABI install — an
MSYS2/MinGW `libcrypto` alongside an MSVC toolchain puts MinGW headers on
`cl.exe`'s include path and fails `httplib.cpp`, `download.cpp` and
`hf-cache.cpp` with `winnt.h: fatal error C1189: No supported target
architecture`. Re-enable with `GGUF_SERVER_OPENSSL=1` if you want in-server
HTTPS downloads.

### Embedded web UI

The engine embeds whatever static assets sit in `GGUF_SERVER_UI_DIR` (default
`vendor/server/ui/dist`, which ships empty), so out of the box it is API-only
and its own `/` returns 404. That has no effect here — this package serves its
own GUI. Point `GGUF_SERVER_UI_DIR` at a bundle if you want the engine's UI as
well.

### Windows notes

Both toolchains work; **which one you get depends on your shell**, so check the
`CMAKE_CXX_COMPILER` line in the configure output if a build misbehaves.

- **MSVC** — build from a Developer Command Prompt (`vcvars64.bat`) so `cl.exe`
  is on PATH. This is the primary Windows toolchain.
- **MinGW / MSYS2** — used automatically when `C:\msys64\mingw64\bin` is on
  PATH and MSVC is not. Supported here: the build defines
  `_WIN32_WINNT=0x0A00` (MinGW otherwise pins it to a Windows 7-era value,
  which makes the vendored cpp-httplib hard-error with *"doesn't support
  Windows 8 or lower"* and leaves `::CreateFile2` undeclared), and links the
  MinGW runtime statically so the binary doesn't need `libstdc++-6.dll`,
  `libgcc_s_seh-1.dll` or `libwinpthread-1.dll` beside it. Turn the latter off
  with `GGUF_SERVER_MINGW_STATIC=0`.

Two environment gotchas worth knowing:

- `'ninja' ... inappropriate file type or format` — a pyenv-win shim is
  shadowing the real ninja. Point CMake at a real one, e.g.
  `-DCMAKE_MAKE_PROGRAM="C:/Program Files (x86)/Microsoft Visual Studio/<ver>/BuildTools/Common7/IDE/CommonExtensions/Microsoft/CMake/Ninja/ninja.exe"`.
- A failed configure leaves a cache behind in `build/`, and later builds reuse
  it. Delete the directory (`rmdir /s /q build`) before retrying after you
  change toolchain or options.

## Usage

```bash
python -m gguf_server              # launch the GUI in the browser
gguf-server                        # same
gguf-server --port 8642 --no-browser
```

The GUI listens on port 8642; the LLM server it launches defaults to 8888.

GUI features (as in the desktop app's LLM panel):

- **Server** — start/stop, live status, copyable API base URL, active
  configuration summary, OpenAI-compatible endpoint reference, and a
  copyable/editable engine command line
- **Model** — `.gguf` model file, optional `--mmproj` vision projector, and
  chat template selection (auto-detect from metadata, file, or raw Jinja2)
- **Settings** — host/port, API key, model alias, context length, GPU layers,
  main GPU, per-GPU tensor split sliders, CPU threads, parallel slots, batch
  and micro-batch sizes, KV cache types, flash attention, continuous
  batching, mlock, mmap and verbose logging, plus hardware-derived
  "Recommended" defaults
- **Presets** — save the whole configuration, with JSON export/import
- **Hardware** — CPU, RAM/VRAM meters and per-GPU details, with live polling
- **Logs** — tail of the running server's log with auto-scroll

Engine CLI passthrough (runs the bundled `gguf-server` engine binary):

```bash
python -m gguf_server engine -- --model model.gguf --port 8888
gguf-server engine -- --help
```

## How it works

- `pip install` compiles the engine (static ggml + llama + llama-common + mtmd
  linked into one executable) via scikit-build-core and installs it into the
  package's `bin/` directory.
- `gguf-server` starts a stdlib HTTP server (default port 8642) serving the
  static GUI and a small JSON API, and opens the browser.
- Starting the server spawns one engine child process with its output
  redirected to a log file; a watcher thread polls the TCP port until it
  accepts connections, then reports the endpoint to the GUI. Stopping the GUI
  stops the child.
- File selection uses a server-side directory listing (`/api/browse`) so the
  GUI gets real filesystem paths — no drag & drop uploads of multi-GB models.
