Metadata-Version: 2.4
Name: edgeserve
Version: 0.1.1
Summary: OpenAI-compatible LLM server for TensorRT Edge-LLM on NVIDIA Jetson and NVIDIA GPUs
Author-email: suryachand2k1@gmail.com
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Surya-Rayala/edgeserve
Project-URL: Documentation, https://github.com/Surya-Rayala/edgeserve#readme
Project-URL: Source, https://github.com/Surya-Rayala/edgeserve
Project-URL: Issues, https://github.com/Surya-Rayala/edgeserve/issues
Keywords: llm,inference,server,tensorrt,tensorrt-edge-llm,jetson,openai,edge-ai
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: fastapi<1,>=0.115
Requires-Dist: uvicorn<0.55,>=0.30
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: prometheus-client>=0.20
Requires-Dist: httpx<1,>=0.27
Requires-Dist: onnx>=1.16
Requires-Dist: nvidia-ml-py>=12; sys_platform == "linux" and platform_machine == "x86_64"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: pytest-asyncio>=0.23; extra == "test"
Requires-Dist: openai>=1.40; extra == "test"
Requires-Dist: tomli>=2; python_version < "3.11" and extra == "test"
Dynamic: license-file

# edgeserve

Run LLMs on NVIDIA Jetson and NVIDIA GPUs behind an OpenAI-compatible API, using
[NVIDIA TensorRT Edge-LLM](https://github.com/NVIDIA/TensorRT-Edge-LLM).

- OpenAI API (chat, completions, streaming): works with the OpenAI SDKs and curl.
- Serves Hugging Face models and your own fine-tuned, pruned or quantized checkpoints, in FP16/BF16,
  INT4 (AWQ, GPTQ), INT8, FP8 or NVFP4, whichever your GPU supports.
- Converts the model to a TensorRT engine for your GPU on first start, and caches it.
- Keeps your request IDs and saves every request, with metrics per request and per token (latency,
  tokens/s, energy).
- Queues and batches simultaneous requests, with limits per client, and guards small devices against
  running out of memory.
- `edgeserve setup --network` adds HTTPS and an API key for other machines; `edgeserve keys add` gives each
  client its own key.

**Steps:** [1. Install](#1-install) · [2. Build the GPU runtime](#2-build-the-gpu-runtime) ·
[3. Export a model](#3-export-a-model) · [4. Serve](#4-serve) ·
[5. Send requests](#5-send-requests) ·
[6. Access from other machines](#6-access-from-other-machines). Also: [request format](#request-format),
[many requests at once](#many-requests-at-once), [saved requests](#saved-requests),
[your own models](#your-own-models), [batch jobs](#batch-jobs), [service](#run-as-a-service),
[uninstall](#uninstall), [commands](#command-reference), [options and settings](#options-and-settings),
[troubleshooting](#troubleshooting), [documentation](#documentation).

In the commands below, replace each `<placeholder>`, angle brackets included, with your own value.

## Requirements

| Device | Software | Status |
|---|---|---|
| Jetson Orin Nano / Orin NX / AGX Orin | JetPack 6.2.x or 7.2.x | tested (Orin Nano 8 GB, JetPack 6.2.1) |
| Jetson Thor | JetPack 7.x | expected to work |
| x86-64 PC with an NVIDIA RTX 30/40/50, A100, A10, L4, L40S or B200 | Ubuntu 22.04/24.04, CUDA 12 or 13, TensorRT 10 | expected to work |

Python 3.10 or newer and build tools are needed too. On a Jetson:

```bash
sudo apt install nvidia-jetpack git cmake build-essential python3-venv
```

On a PC, install the NVIDIA driver, CUDA and TensorRT 10 with its development packages, then
`sudo apt install git cmake build-essential python3-venv`. `edgeserve setup` checks all of this
and prints what is missing. edgeserve 0.1 is built on TensorRT Edge-LLM 0.11 (commit
`0f52705723ab`). Details per platform:
[docs/PLATFORMS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/PLATFORMS.md).

**Disk space.** edgeserve's tools take about 7 GB, the same whichever models you use: 1.7 GB for
the GPU runtime (step 2, which needs 3 GB free while it builds) and 5.5 GB for the exporter
(step 3). Each model needs its own space on top, which grows with its size: its download, its
exported folder and its engines. A 0.5B model takes up to 2 GB in INT4 and 3.5 GB in FP16
([details](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#disk-space)).

## 1. Install

With pip:

```bash
python3 -m venv <venv-dir>
source <venv-dir>/bin/activate
pip install --upgrade pip
pip install edgeserve
```

Or from the repository:

```bash
git clone https://github.com/Surya-Rayala/edgeserve.git
cd edgeserve
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .
```

`<venv-dir>` is any new folder for the Python environment, for example `~/edgeserve-venv`. Both ways
give the same `edgeserve` command, and both keep their files in the same places: settings in
`~/.config/edgeserve/`, the GPU runtime and the model exporter in `~/.local/share/edgeserve/`, engines
and build files in `~/.cache/edgeserve/`. In a new terminal, run the `source …/bin/activate` line again.

## 2. Build the GPU runtime

```bash
edgeserve setup
```

Run it once per machine, as your normal user (not with sudo). It downloads TensorRT Edge-LLM,
builds it for your GPU and tests it. This takes about 30 minutes on a Jetson Orin Nano (less on a
PC) and needs internet access and 3 GB of free disk. Run it again after upgrading edgeserve, JetPack,
CUDA or TensorRT: it rebuilds only what changed.

## 3. Export a model

edgeserve serves models in TensorRT Edge-LLM's format: a folder with the model converted to ONNX,
its weights and its tokenizer. `edgeserve export` makes this folder from a Hugging Face checkpoint:

```bash
edgeserve export <checkpoint> <model-dir>
```

| Argument | What it is |
|---|---|
| `<checkpoint>` | a Hugging Face model ID, such as `Qwen/Qwen2.5-0.5B-Instruct-AWQ`, or the folder of your own checkpoint ([your own models](#your-own-models)) |
| `<model-dir>` | a new folder for the exported model, such as `~/models/qwen2.5-0.5b-instruct-awq`; its name becomes the model's name |

The first export installs TensorRT Edge-LLM's exporter (a few minutes, about 5.5 GB of disk, once
for all models). An export takes a few minutes; it runs on the CPU, so it works on the Jetson itself
or on any Linux PC with edgeserve and git installed (no `edgeserve setup` needed there; copy
`<model-dir>` to the Jetson afterwards). It also copies the model's default sampling settings
(`generation_config.json`). For a gated model such as Llama, accept its license on huggingface.co
and pass your access token: `HF_TOKEN=<your token> edgeserve export <checkpoint> <model-dir>`. A
folder that TensorRT Edge-LLM 0.11 already exported needs no export: serve it as it is. GGUF files,
ONNX models from other tools and ready-made engines cannot be served ([what you can bring](#your-own-models)).

**Which models.** Text models of the Llama 3.x, Qwen2/Qwen2.5, Qwen3 and HunYuan (dense) families;
Qwen3 MoE, Qwen3.5 and Nemotron-H should work but are untested
([families and tested models](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#supported-families)).
Choose a precision your GPU supports:

| Weights | Hugging Face name looks like | Jetson Orin | RTX 30/40, A100, A10, L4, L40S | Thor, RTX 50, B200 |
|---|---|---|---|---|
| FP16 / BF16 (the original weights) | `Qwen/Qwen2.5-0.5B-Instruct` | ✅ | ✅ | ✅ |
| INT4 (AWQ or GPTQ) | `…-Instruct-AWQ`, `…-GPTQ-Int4` | ✅ | ✅ | ✅ |
| INT8 (SmoothQuant) | quantize it yourself ([how](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#quantizing-and-other-tools-by-hand)) | ✅ | ✅ | ✅ |
| FP8 | `nvidia/…-FP8` | ❌ | RTX 40, L4, L40S only | ✅ |
| NVFP4 | `nvidia/…-NVFP4` | ❌ | ❌ | ✅ |

Memory decides the model size. With `edgeserve export`, an INT4 model builds its engine in about the
memory it serves in; an FP16 model needs two to three times that. On an 8 GB Orin Nano, FP16 models
up to about 0.6B parameters and INT4 models of 0.5B to 4B were tested, and a pruned 6B INT4 model
with a reduced output vocabulary fits too. Boards and GPUs with more memory run correspondingly larger
models, in any precision they support. More: every argument, what the export does, exporter options
and errors in [docs/EXPORT.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md);
supported models and memory in [docs/MODELS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md).

## 4. Serve

On a Jetson, the model shares the board's memory with everything else running: close what you do not
need first ([how](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/PLATFORMS.md#free-memory-on-a-jetson)).

```bash
edgeserve serve --model <model-dir>
```

The first start converts the model to a TensorRT engine for your GPU (a minute to a few minutes)
and caches it; later starts take seconds. The server listens on `http://127.0.0.1:8000`, which
only this machine can reach. Stop it with Ctrl-C. To test it, run `edgeserve check` in a second
terminal.

Its options:

| Option | Default | What it does |
|---|---|---|
| `--model DIR` | none | the model folder; after `edgeserve setup --model <model-dir>`, plain `edgeserve serve` is enough |
| `--output-dir DIR` | none | save every request, answered or not, as JSON ([saved requests](#saved-requests)) |
| `--host ADDRESS`, `--port N` | 127.0.0.1, 8000 | where the server listens; [step 6](#6-access-from-other-machines) opens it to the network |
| `--max-batch-size N` | 1 | how many requests are generated together ([many requests at once](#many-requests-at-once)) |
| `--max-input-len N` | 3072 | the longest prompt, in tokens; a longer one is refused |
| `--max-new-tokens N` | the model's default, else 512 | the answer length when a request does not set `max_tokens` |
| `--cache-dir DIR` | `~/.cache/edgeserve` | where the converted engines are kept |
| `--queue-size N` | 16 | how many requests may wait for their turn; more are refused (503). `-1`: no limit |
| `--queue-timeout SECONDS` | 600 | a request that has waited this long fails (503). `0`: no limit |
| `--request-timeout SECONDS` | no limit | a request not finished this long after it arrived is stopped (504) |
| `--max-client-requests N` | no limit | how many requests one client may have waiting or running; more are refused (429) |
| `--client-rate-limit N` | no limit | requests a minute per client: N at once, then one every 60/N seconds; more are refused (429) |

To set an option permanently, see [options and settings](#options-and-settings).

### Many requests at once

Requests wait in a queue, and the oldest runs first. With `--max-batch-size N`, up to N waiting
requests of the same kind (chat or completion), with the same `temperature`, `top_p`, `top_k` and
`max_tokens`, are generated together. When it starts, the server reserves memory for N requests of
up to `--max-input-len` + `--max-new-tokens` tokens each, so lower those two to fit a larger N.
Changing any of the three builds a new engine (a few minutes; it is cached too). For example, to
batch up to 3 requests and keep every request until its turn:

```bash
edgeserve serve --model <model-dir> --max-batch-size 3 --max-input-len 1024 --max-new-tokens 256 --queue-size -1
```

- **A client** is one API key ([a key for each client](#a-key-for-each-client)), or one address on a
  server without keys.
- **`--queue-size -1`** keeps every request: there is no wait limit unless you also give
  `--queue-timeout`, and no connection limit. Otherwise one address can open at most 32
  connections, and all clients together 128.
- **A refused request** (429 or 503) gets a `Retry-After` header with the seconds to wait. The OpenAI
  SDK and `edgeserve batch` wait and retry by themselves; with curl, retry yourself. A request
  stopped by `--request-timeout` while running is not retried, as it would run out of time again.
- **A waiting client** has its own timeout: after 600 s without an answer, the OpenAI SDK drops the
  request and sends it again at the back of the queue, twice at most; `OpenAI(..., timeout=3600)`
  waits longer.

## 5. Send requests

With curl:

```bash
curl -s http://127.0.0.1:8000/v1/models
curl -s http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" \
  -H "X-Request-Id: demo-001" \
  -d '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'
```

With Python (`pip install openai`):

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")  # no key needed until step 6
model = client.models.list().data[0].id                                  # the served model's name
reply = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "What is a GPU? Answer in one sentence."}],
    max_tokens=100,
    extra_headers={"X-Request-Id": "demo-002"},
)
print(reply.choices[0].message.content)
print(reply.model_extra["metrics"]["decode_tokens_per_s"], "tokens/s")
```

### Request format

A chat request (`POST /v1/chat/completions`) is a JSON object; only `messages` is required:

```json
{
  "messages": [
    {"role": "system", "content": "Summarize the text in two sentences."},
    {"role": "user", "content": "<the text>"}
  ],
  "max_tokens": 160,
  "temperature": 0,
  "request_id": "meeting-0001",
  "metadata": {"source": "room-12"}
}
```

| Field | Meaning | Default |
|---|---|---|
| `messages` | the conversation, oldest first. `role`: `system` (instructions), `user`, `assistant` (an earlier answer) or `developer` (same as `system`); `content`: the text. Up to 64 messages of up to 128 KB each | required |
| `max_tokens` | the longest answer, in tokens (`max_completion_tokens` means the same) | `--max-new-tokens` (the model's `generation_config.json`, else 512) |
| `temperature` | randomness, 0 to 2. 0 always picks the most likely next token: the same answer every time | the model's `generation_config.json`, else 0 |
| `top_p`, `top_k` | when sampling, choose only among the most likely tokens: those that make up `top_p` (above 0, up to 1) of the probability, or the `top_k` (up to 1024) most likely | the model's `generation_config.json`, else no limit |
| `seed` | a number that makes sampling repeatable | random, saved in the record |
| `stop` | a string, or up to 4, that ends the answer | none |
| `stream` | `true`: send the answer in pieces as it is generated; add `"stream_options": {"include_usage": true}` for a last piece with the token counts and metrics | `false` |
| `request_id` | your ID for this request (the same as the `X-Request-Id` header) | `req_<random hex>` |
| `metadata` | up to 16 string pairs of your own (keys up to 64 characters, values up to 512), saved in the record | none |
| `return_record` | `true`: the response also holds the full record, with per-token timings (not with `stream`) | `false` |
| `model` | the served model's name, as `GET /v1/models` lists it; optional | the served model |
| `n` (only 1), `frequency_penalty` and `presence_penalty` (only 0), `logprobs` (only `false`), `user`, `safety_identifier`, `prompt_cache_key`, a message's `name` | accepted so that every OpenAI client works; they change nothing | — |

A completion request (`POST /v1/completions`) takes `prompt`, one string with the model's chat
template already applied, instead of `messages`. Any other field is refused with HTTP 400, never
ignored.

**Request IDs.** Send your ID in the `X-Request-Id` header or the `request_id` field: 1 to 512
printable ASCII characters (with letters, digits, `.`, `_` and `-`, the record's file name is the ID
itself). Without one, the server makes one (`req_…`). The ID comes back in the response's
`request_id` and `id` (`chatcmpl-<id>`) and in its `X-Request-Id` header. With
`edgeserve serve --output-dir <output-dir>`, every request is saved as
`<output-dir>/records/<id>.json`, which `GET /v1/records/<id>` also returns. A request with the ID
of one that is still running gets HTTP 409.

The response is OpenAI's, plus `request_id` and `metrics`:

```json
{"id": "chatcmpl-meeting-0001", "object": "chat.completion", "model": "<model name>",
 "choices": [{"index": 0, "message": {"role": "assistant", "content": "…"}, "finish_reason": "stop"}],
 "usage": {"prompt_tokens": 162, "completion_tokens": 72, "total_tokens": 234},
 "request_id": "meeting-0001", "metrics": {"ttft_ms": 303, "decode_tokens_per_s": 25.9, …}}
```

For streaming, send `"stream": true` (`stream=True` in the SDK). Every field, error code and
metric: [docs/API.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/API.md).

## 6. Access from other machines

```bash
edgeserve setup --network
```

This creates three things in `~/.config/edgeserve/`:

- **An API key.** It is printed once. To see it again:
  `grep EDGESERVE_API_KEY ~/.config/edgeserve/edgeserve.env`.
- **An HTTPS certificate**, and `edgeserve-ca.crt`, the file clients need to trust it.
- **The setting to listen on the network** (all interfaces, port 8000).

Restart `edgeserve serve`. From then on the server speaks only HTTPS and every `/v1/` request needs
the key, on this machine too (`edgeserve check` finds both by itself), so the `http://` examples of
step 5 stop working.

**On each client machine** you need the server's `edgeserve-ca.crt` and the key. Below,
`<server-ip>` is the server's IP address (`edgeserve setup --network` prints it) and `<user>` is your
user name on the server. The first two lines copy both from the server over SSH; you can also paste
the key yourself, e.g. `export EDGESERVE_API_KEY=es-…`.

Linux and macOS (Terminal):

```bash
scp <user>@<server-ip>:.config/edgeserve/edgeserve-ca.crt .
export EDGESERVE_API_KEY=$(ssh <user>@<server-ip> "sed -n 's/^EDGESERVE_API_KEY=//p' .config/edgeserve/edgeserve.env")
curl --cacert edgeserve-ca.crt -H "Authorization: Bearer $EDGESERVE_API_KEY" https://<server-ip>:8000/v1/models
curl --cacert edgeserve-ca.crt https://<server-ip>:8000/v1/chat/completions \
  -H "Authorization: Bearer $EDGESERVE_API_KEY" -H "Content-Type: application/json" -H "X-Request-Id: client-001" \
  -d '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'
```

Windows (PowerShell; `scp` and `curl.exe` come with Windows 10 and 11):

```powershell
scp <user>@<server-ip>:.config/edgeserve/edgeserve-ca.crt .
$env:EDGESERVE_API_KEY = ssh <user>@<server-ip> "sed -n 's/^EDGESERVE_API_KEY=//p' .config/edgeserve/edgeserve.env"
curl.exe --ssl-no-revoke --cacert edgeserve-ca.crt -H "Authorization: Bearer $env:EDGESERVE_API_KEY" https://<server-ip>:8000/v1/models
Set-Content -Encoding ascii request.json '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'
curl.exe --ssl-no-revoke --cacert edgeserve-ca.crt https://<server-ip>:8000/v1/chat/completions `
  -H "Authorization: Bearer $env:EDGESERVE_API_KEY" -H "Content-Type: application/json" -H "X-Request-Id: client-002" `
  -d "@request.json"
```

Type `curl.exe`, not `curl` (in Windows PowerShell, `curl` is a different command).
`--ssl-no-revoke` is needed because Windows otherwise asks whether the certificate was revoked,
which a private CA cannot answer. The request body goes in a file because PowerShell changes the
quotes inside inline JSON.

Python, on any of them: make a virtual environment with the OpenAI SDK
(`python3 -m venv <client-venv>` and `<client-venv>/bin/pip install openai httpx`; on Windows
`py -m venv <client-venv>` and `<client-venv>\Scripts\pip install openai httpx`), then run this
with its Python:

```python
import os, ssl
import httpx
from openai import OpenAI

client = OpenAI(base_url="https://<server-ip>:8000/v1", api_key=os.environ["EDGESERVE_API_KEY"],
                http_client=httpx.Client(verify=ssl.create_default_context(cafile="edgeserve-ca.crt")))
reply = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[{"role": "user", "content": "What is a GPU? Answer in one sentence."}],
    max_tokens=100,
    extra_headers={"X-Request-Id": "client-003"},
)
print(reply.model_extra["request_id"], reply.choices[0].message.content)
```

Every other request works as in [step 5](#5-send-requests), with `https://<server-ip>:8000`, the
key and the CA file.

- **New key:** delete the `EDGESERVE_API_KEY=` line in `~/.config/edgeserve/edgeserve.env`, run
  `edgeserve setup --network` (it prints the new key) and restart the server. You can also write
  your own key on that line.
- **New IP address or host name:** run `edgeserve setup --network` again and restart the server.
  Clients keep their `edgeserve-ca.crt`.
- **Firewall:** allow only your own network to reach port 8000. `edgeserve setup --network` prints
  the `ufw` commands for your network.
- **This machine only again:** set `EDGESERVE_HOST=127.0.0.1` in the settings file.
- **`/metrics` and `/docs`** need the key too, except when opened at `127.0.0.1` on the server
  itself; `/health` stays open.

### A key for each client

Rather than sharing the key from `edgeserve setup --network` (client `default`), you can give each
person or machine its own. You can then remove one without changing the others, and each saved
request shows which client sent it. On the server:

```bash
edgeserve keys add <name>       # make a key for client <name>; it is shown only now
edgeserve keys list             # the clients that have a key
edgeserve keys remove <name>    # that key stops working at once
```

The client uses its own key in place of the `default` key, and still needs `edgeserve-ca.crt`. To
skip copying the key by hand, make it straight into a file (instead of the plain `keys add` above),
then fetch it on the client over SSH, which also deletes the file:

```bash
edgeserve keys add <name> | awk '$1 ~ /^es-/ {print $1}' > ~/<name>.key && chmod 600 ~/<name>.key   # on the server
export EDGESERVE_API_KEY=$(ssh <user>@<server-ip> "cat <name>.key && rm <name>.key")                # on the client
```

On Windows (PowerShell), the client line is
`$env:EDGESERVE_API_KEY = ssh <user>@<server-ip> "cat <name>.key && rm <name>.key"`.

- Adding or removing a key works at once, also on a running server.
- A client reads only its own saved requests; the `default` key reads all of them.
- `--max-client-requests` and `--client-rate-limit` count each key separately.
- edgeserve stores only a hash of each key, so a lost key cannot be shown again: remove it and add a
  new one.

## Saved requests

With `edgeserve serve --output-dir <output-dir>`, every request is saved:

| File | What it holds |
|---|---|
| `<output-dir>/records/<id>.json` | the latest attempt of each request ID: the request, the answer, its client, metrics per request and per token, and its status: `completed`, `failed`, `cancelled` or `rejected` (refused before it was queued; `http_status` and `error` say why) |
| `<output-dir>/results.jsonl` | one line per attempt, in OpenAI's batch output format |

A refused request gets a `results.jsonl` line, and a record when it named its ID (`X-Request-Id`, or
`request_id` in a valid body) and that ID has no record yet: a refusal never replaces an earlier
record. At most 60 refusals a minute per client are saved, and requests without a valid key never.
`GET /v1/records/<id>` returns a record, also from another machine (with client keys, only the
client's own). The folder is readable only by the user running the server, as records hold prompts
and answers. Every field:
[docs/METRICS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/METRICS.md).

## Your own models

Find what you have:

| You have | What to do |
|---|---|
| A Hugging Face checkpoint folder (`config.json` and `*.safetensors`): the original model, or your fine-tuned, pruned or quantized version of it | export it: `edgeserve export <checkpoint-dir> <model-dir>`, then serve `<model-dir>` |
| A LoRA adapter | load its base model in FP16/BF16 (not 4-bit), merge the adapter (PEFT's `merge_and_unload()`), save the model and the tokenizer into one folder with `save_pretrained()`, then export that folder |
| A folder already exported (converted to ONNX) by TensorRT Edge-LLM 0.11: its `config.json` has `"edgellm_version": "0.11.…"` and sits next to `model.onnx` | nothing: serve it as it is, `edgeserve serve --model <model-dir>` (if the export made an `llm` folder, pass that) |
| A folder exported by another TensorRT Edge-LLM version | export the checkpoint again with `edgeserve export`; edgeserve refuses the folder and says so |
| An ONNX model made by another tool (such as Optimum), a GGUF file (llama.cpp, Ollama), an engine built by TensorRT-LLM or another tool, or an INT4 model in another format (bitsandbytes, compressed-tensors, EXL2, HQQ) | cannot be served: export the Hugging Face checkpoint it was made from, or an AWQ or GPTQ version of it |
| A model of another family (such as Mistral, Gemma or Phi), or one that needs images or audio | not supported, except Qwen3.5, whose image part the export leaves out ([supported families](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#supported-families)) |

**A checkpoint folder** (`<checkpoint-dir>`) holds these files, as `save_pretrained()` writes them:

| File | Needed | What it holds |
|---|---|---|
| `config.json` | yes | the architecture, with `model_type` such as `llama`, `qwen2` or `qwen3` ([supported families](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#supported-families)) |
| `model.safetensors` (or shards with `model.safetensors.index.json`; `pytorch_model.bin` also works) | yes | the weights |
| `tokenizer.json`, `tokenizer_config.json` | yes | the tokenizer, and the chat template (`chat_template` in `tokenizer_config.json`, or a `chat_template.jinja` file) |
| `generation_config.json` | no | default sampling settings and end-of-sequence tokens; copied into `<model-dir>` |

- **Fine-tuned:** no changes needed; save the model and its tokenizer into one folder.
- **Pruned:** fewer layers, or fewer heads or a smaller hidden size in every layer, need only the
  matching numbers in `config.json`. A different MLP width in each layer needs
  `"intermediate_sizes": [<width of layer 0>, <width of layer 1>, …]` in the checkpoint's `config.json`,
  one entry per layer (multiples of 128 for INT4). An exported `<model-dir>` needs nothing more: the
  widths are then part of its `model.onnx`, and `edgeserve info --model <model-dir>` lists them (`mlp_widths`).
- **Quantized:** INT4 AWQ or GPTQ (group size 128), or FP8, NVFP4, INT8 SmoothQuant or INT4 AWQ made
  with NVIDIA ModelOpt or Edge-LLM's quantizer, for GPUs that run them
  ([INT4 quality](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#quality-of-int4-models)).
- **Smaller output vocabulary:** add `-- --reduced-vocab-dir <vocab-dir>` to the export command
  (`<vocab-dir>` is the folder with `vocab_map.safetensors`).

**An exported folder** is served as a whole: copy all of it
([every file](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#the-model-folder)). `edgeserve info --model <model-dir>` checks it
without starting a server. A folder made with Edge-LLM's own `tensorrt-edgellm-export` (without
edgeserve) differs in three ways: it has no `generation_config.json` (copy the checkpoint's in, or
requests decode greedily), its INT4 AWQ and asymmetric GPTQ weights are slightly clipped, and unless
it was exported with `--externalize-weights`, its engine build needs more memory. If you have the
checkpoint, export it again with `edgeserve export`
([INT4 quality](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#quality-of-int4-models)).

To host your model for your network, starting at every boot:

```bash
edgeserve export <checkpoint-dir> <model-dir>        # once (skip it for a folder that is already exported)
edgeserve setup --network --model <model-dir>        # make it the default model; API key and HTTPS
edgeserve service install                            # serve it now and at every boot
```

Details, quantizing a model yourself and checking a pruned export: [docs/EXPORT.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#your-own-checkpoints).

## Batch jobs

```bash
edgeserve batch <requests.jsonl> --output-dir <results-dir>
```

`<requests.jsonl>` has one request per line, for example:

```jsonl
{"id": "q1", "messages": [{"role": "user", "content": "What is CUDA?"}], "max_tokens": 200}
{"id": "q2", "prompt": "<a prompt with the chat template already applied>"}
```

The OpenAI Batch format (`custom_id`, `method`, `url`, `body`) works too. Results go into
`<results-dir>/records/<id>.json`, `<results-dir>/results.jsonl` and `<results-dir>/run.json`
(counts, throughput, latency, energy). Running the same command again skips the finished IDs, so
an interrupted batch resumes. For another machine's server, set `EDGESERVE_API_KEY` as in
[step 6](#6-access-from-other-machines) and add `--server https://<server-ip>:8000 --ca-cert edgeserve-ca.crt`.

## Run as a service

```bash
edgeserve setup --model <model-dir>    # the model the service serves (add --network for other machines)
edgeserve service install              # starts the server now and at every boot; asks for your sudo password
journalctl -u edgeserve -f             # its log
edgeserve service remove               # stop and remove it
```

Run these commands as your normal user, not with sudo. The service uses the settings file: after
changing it, restart the service with `sudo systemctl restart edgeserve`.
`edgeserve service install --print` only shows the unit file.

## Uninstall

First [remove the Hugging Face downloads](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#disk-space)
you no longer need (they are in `~/.cache/huggingface`), then:

```bash
edgeserve service remove     # if you installed the service
rm -rf ~/.config/edgeserve ~/.local/share/edgeserve ~/.cache/edgeserve
```

Finally, delete `<venv-dir>` (or the cloned `edgeserve` folder) and the model folders you no longer
need. After a new install, `edgeserve setup --network` makes a new key and `edgeserve-ca.crt`: give
both to the clients again.

## Command reference

| Command | What it does |
|---|---|
| `edgeserve setup [--network] [--model DIR]` | build the GPU runtime; `--model` sets the default model; `--network` adds an API key, HTTPS and listening on the network |
| `edgeserve export CHECKPOINT MODEL_DIR [-- exporter options]` | export a Hugging Face checkpoint (an ID or a folder) into a model folder |
| `edgeserve serve` | run the server |
| `edgeserve build` | only convert the model to an engine (or check the cached one), then exit |
| `edgeserve info` | settings and where each comes from, the GPU runtime, the hardware; with `--model`, the model, its memory estimate and its cached engine |
| `edgeserve check [--server URL] [--ca-cert FILE]` | test a running server end to end |
| `edgeserve keys add NAME`, `edgeserve keys list`, `edgeserve keys remove NAME` | API keys, one per client |
| `edgeserve batch INPUT --output-dir DIR [--server URL] [--ca-cert FILE] [--concurrency N]` | run a JSONL file of requests |
| `edgeserve service install [--print]`, `edgeserve service remove` | run the server at boot (systemd) |
| `edgeserve --version` | the edgeserve and TensorRT Edge-LLM versions |

## Options and settings

Each option can also be set as a line `NAME=value` in `~/.config/edgeserve/edgeserve.env` (the file
`edgeserve setup` writes; restart the server after editing it) or as an environment variable. An
option on the command line wins over the environment, which wins over the file. `edgeserve info`
shows where each value came from.

**The model and its engine** (`serve`, `build` and `info`):

| Option | Setting | Default | Meaning |
|---|---|---|---|
| `--model DIR` | `EDGESERVE_MODEL` | none | the model folder (an absolute path in the file) |
| `--max-input-len N` | `EDGESERVE_MAX_INPUT_LEN` | 3072 | longest prompt in tokens; longer prompts are refused, never cut |
| `--max-new-tokens N` | `EDGESERVE_MAX_NEW_TOKENS` | the model's `generation_config.json`, else 512 | output length when a request sets no `max_tokens` |
| `--max-batch-size N` | `EDGESERVE_MAX_BATCH_SIZE` | 1 | requests generated together; each one needs its own memory |
| `--cache-dir DIR` | `EDGESERVE_CACHE_DIR` | `~/.cache/edgeserve` | where engines are kept |

**The server** (`serve`):

| Option | Setting | Default | Meaning |
|---|---|---|---|
| `--output-dir DIR` | `EDGESERVE_OUTPUT_DIR` | none | save every request here ([saved requests](#saved-requests)) |
| `--host ADDRESS` | `EDGESERVE_HOST` | 127.0.0.1 | `127.0.0.1`: this machine only; `0.0.0.0`: the network (needs an API key) |
| `--port N` | `EDGESERVE_PORT` | 8000 | port |
| `--queue-size N` | `EDGESERVE_QUEUE_SIZE` | 16 | requests that may wait beyond the running batch; `-1`: no limit, nor a wait or connection limit |
| `--queue-timeout SECONDS` | `EDGESERVE_QUEUE_TIMEOUT` | 600 | how long a request may wait; `0`: no limit |
| `--request-timeout SECONDS` | `EDGESERVE_REQUEST_TIMEOUT` | 0 | how long a request may take from arrival to its last token; `0`: no limit |
| `--max-client-requests N` | `EDGESERVE_MAX_CLIENT_REQUESTS` | 0 | requests one client may have waiting or running; `0`: no limit |
| `--client-rate-limit N` | `EDGESERVE_CLIENT_RATE_LIMIT` | 0 | requests per minute per client; `0`: no limit |

**Keys, certificates and the runtime** (no command-line option, so a key never shows in process lists; set
them in the settings file or the environment):

| Setting | Meaning |
|---|---|
| `EDGESERVE_API_KEY` | the key of client `default`, sent as `Authorization: Bearer <key>`; more keys: [edgeserve keys](#a-key-for-each-client) |
| `EDGESERVE_TLS_CERT`, `EDGESERVE_TLS_KEY` | HTTPS certificate and private key (PEM files) |
| `EDGESERVE_NATIVE_DIR` | use another GPU runtime build (development) |

Changing `--max-input-len`, `--max-new-tokens` or `--max-batch-size` builds a new engine (cached
too). `check` and `batch` use this machine's server unless given `--server URL`, and trust a
server's certificate with `--ca-cert FILE`; `batch --concurrency N` sets how many requests are in
flight (default: the server's batch size + 1). Environment only: `EDGESERVE_CONFIG_DIR` (another
settings folder), `EDGESERVE_API_KEY` on a client (the key `check` and `batch` send), and for
`edgeserve setup`, `TRT_PACKAGE_DIR` (a TensorRT tarball folder) and `JOBS` (parallel compile jobs).

## Troubleshooting

| Problem | Fix |
|---|---|
| `edgeserve setup` lists missing packages | install them with the `apt` line it prints, then run it again |
| "the GPU runtime is not built yet" or "… from other sources" | run `edgeserve setup` |
| `/health/ready` says `building_engine` | normal on the first start of a model or new limits |
| "not enough memory", or the engine build "ran out of memory" | [free memory](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/PLATFORMS.md#free-memory-on-a-jetson) on a Jetson; use an INT4 or a smaller model, lower `--max-input-len`, keep `--max-batch-size 1` |
| "exported with TensorRT Edge-LLM x.y" | export the model again with `edgeserve export` |
| "is a Hugging Face checkpoint, not an exported model" | export it first: `edgeserve export <checkpoint-dir> <model-dir>`, then serve `<model-dir>` |
| "is a GGUF file", "holds GGUF files" or "holds an ONNX model made by another tool" | export the Hugging Face checkpoint the model was made from |
| "has no config.json" | `--model` must be the folder `edgeserve export` made; for `export`, the checkpoint folder must hold `config.json` |
| HTTP 401 | send the key: `Authorization: Bearer <key>`; a key removed with `edgeserve keys remove` no longer works |
| HTTP 429 `too_many_requests` or `rate_limit_exceeded` | the client is over `--max-client-requests` or `--client-rate-limit`: retry after the `Retry-After` seconds |
| HTTP 503 `server_is_overloaded` or `queue_timeout` | the queue is full or the wait too long: retry after `Retry-After`; raise `--queue-size` or `--queue-timeout`, or batch more with `--max-batch-size` |
| HTTP 504 `time_limit_exceeded` | the request took longer than `--request-timeout`: raise the limit, or ask for fewer `max_tokens` (a request stopped while running is not sent again automatically) |
| certificate errors | give the client `edgeserve-ca.crt`; use a host name or address the certificate lists, or run `edgeserve setup --network` again after an address change |
| `<host>.local` does not connect | use the IP address; mDNS can announce a Docker bridge address (fix: `deny-interfaces=docker0` in `/etc/avahi/avahi-daemon.conf`) |
| other machines cannot connect | run `edgeserve setup --network` and restart; check the firewall (`sudo ufw status`) |
| answers differ from Hugging Face | the server log lists `generation_config.json` options Edge-LLM does not support (e.g. `repetition_penalty`); for INT4 models see [INT4 quality](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#quality-of-int4-models) |
| `edgeserve export` fails | the error table in [docs/EXPORT.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#errors) |

## Documentation

| Topic | Document |
|---|---|
| Exporting: every argument, what the export does, your own fine-tuned, pruned and quantized checkpoints, INT4 quality, exporter options, errors | [docs/EXPORT.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md) |
| Models: supported families, tested models, precision per GPU, memory, disk space | [docs/MODELS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md) |
| The API: endpoints, request fields, sampling, streaming, errors, request IDs, records | [docs/API.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/API.md) |
| Metrics: every per-request and per-token metric and how it is measured | [docs/METRICS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/METRICS.md) |
| Platforms: requirements per device, Jetson settings, measured performance, Edge-LLM patches | [docs/PLATFORMS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/PLATFORMS.md) |
| How it works: processes, GPU runtime, engine cache, request lifecycle, worker protocol | [docs/ARCHITECTURE.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/ARCHITECTURE.md) |

## Development

```bash
pip install -e ".[test]"
python -m pytest            # no GPU needed; needs g++ and nlohmann-json3-dev
```

## License

Apache License 2.0 ([LICENSE](https://github.com/Surya-Rayala/edgeserve/blob/main/LICENSE)). edgeserve
downloads and builds NVIDIA TensorRT Edge-LLM (Apache 2.0), which is not part of this package. Models
have their own licenses.
