Metadata-Version: 2.4
Name: edgeserve
Version: 0.1.2
Summary: OpenAI-compatible LLM server for TensorRT Edge-LLM on NVIDIA Jetson and NVIDIA GPUs
Author-email: suryachand2k1@gmail.com
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Surya-Rayala/edgeserve
Project-URL: Documentation, https://github.com/Surya-Rayala/edgeserve#readme
Project-URL: Source, https://github.com/Surya-Rayala/edgeserve
Project-URL: Issues, https://github.com/Surya-Rayala/edgeserve/issues
Keywords: llm,inference,server,tensorrt,tensorrt-edge-llm,jetson,openai,edge-ai
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: fastapi<1,>=0.115
Requires-Dist: uvicorn<0.55,>=0.30
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: prometheus-client>=0.20
Requires-Dist: httpx<1,>=0.27
Requires-Dist: onnx>=1.16
Requires-Dist: nvidia-ml-py>=12; sys_platform == "linux" and platform_machine == "x86_64"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: pytest-asyncio>=0.23; extra == "test"
Requires-Dist: openai>=1.40; extra == "test"
Requires-Dist: tomli>=2; python_version < "3.11" and extra == "test"
Dynamic: license-file

# edgeserve

Run LLMs on NVIDIA Jetson and NVIDIA GPUs behind an OpenAI-compatible API, using
[NVIDIA TensorRT Edge-LLM](https://github.com/NVIDIA/TensorRT-Edge-LLM).

- OpenAI API (chat, completions, streaming): works with the OpenAI SDKs and curl.
- Serves Hugging Face models and your own fine-tuned, pruned or quantized checkpoints, in FP16/BF16,
  INT4 (AWQ, GPTQ), INT8, FP8 or NVFP4, whichever your GPU supports.
- Converts the model to a TensorRT engine for your GPU on first start, and caches it.
- Keeps your request IDs and saves every request, with metrics per request and per token (latency,
  tokens/s, energy).
- Queues and batches simultaneous requests, with limits per client, and guards small devices against
  running out of memory.
- `edgeserve setup --network` adds HTTPS and an API key for other machines; `edgeserve keys add` gives each
  client its own key.

**Steps:** [1. Install](#1-install) · [2. Build the GPU runtime](#2-build-the-gpu-runtime) ·
[3. Export a model](#3-export-a-model) · [4. Serve](#4-serve) ·
[5. Send requests](#5-send-requests) ·
[6. Access from other machines](#6-access-from-other-machines). Also: [request format](#request-format),
[many requests at once](#many-requests-at-once), [saved requests](#saved-requests),
[your own models](#your-own-models), [batch jobs](#batch-jobs), [service](#run-as-a-service),
[uninstall](#uninstall), [commands](#command-reference), [options and settings](#options-and-settings),
[troubleshooting](#troubleshooting), [documentation](#documentation).

In the commands below, replace each `<placeholder>`, angle brackets included, with your own value.

## Requirements

| Device | Software | Status |
|---|---|---|
| Jetson Orin Nano / Orin NX / AGX Orin | JetPack 6.2.x or 7.2.x | tested (Orin Nano 8 GB, JetPack 6.2.1) |
| Jetson Thor | JetPack 7.x | expected to work |
| x86-64 PC with an NVIDIA RTX 30/40/50, A100, A10, L4, L40S or B200 | Ubuntu 22.04/24.04, CUDA 12 or 13, TensorRT 10 | expected to work |

Python 3.10 or newer and build tools are needed too. On a Jetson:

```bash
sudo apt install nvidia-jetpack git cmake build-essential python3-venv
```

On a PC, install the NVIDIA driver, CUDA and TensorRT 10 with its development packages, then
`sudo apt install git cmake build-essential python3-venv`. `edgeserve setup` checks all of this
and prints what is missing. edgeserve 0.1 is built on TensorRT Edge-LLM 0.11 (commit
`0f52705723ab`). Details per platform:
[docs/PLATFORMS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/PLATFORMS.md).

**Disk space.** edgeserve's tools take about 7 GB, the same whichever models you use: 1.7 GB for
the GPU runtime (step 2, which needs 3 GB free while it builds) and 5.5 GB for the exporter
(step 3). Each model needs its own space on top, which grows with its size: its download, its
exported folder and its engines. A 0.5B model takes up to 2 GB in INT4 and 3.5 GB in FP16
([details](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#disk-space)).

## 1. Install

With pip:

```bash
python3 -m venv <venv-dir>
source <venv-dir>/bin/activate
pip install --upgrade pip
pip install edgeserve
```

Or from the repository:

```bash
git clone https://github.com/Surya-Rayala/edgeserve.git
cd edgeserve
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .
```

`<venv-dir>` is any new folder for the Python environment, for example `~/edgeserve-venv`. Both ways
give the same `edgeserve` command, and both keep their files in the same places: settings in
`~/.config/edgeserve/`, the GPU runtime and the model exporter in `~/.local/share/edgeserve/`, engines
and build files in `~/.cache/edgeserve/`. In a new terminal, run the `source …/bin/activate` line again.

## 2. Build the GPU runtime

```bash
edgeserve setup
```

Run it once per machine, as your normal user (not with sudo). It downloads TensorRT Edge-LLM,
builds it for your GPU and tests it. This takes about 30 minutes on a Jetson Orin Nano (less on a
PC) and needs internet access and 3 GB of free disk. Run it again after upgrading edgeserve, JetPack,
CUDA or TensorRT: it rebuilds only what changed.

## 3. Export a model

edgeserve serves models in TensorRT Edge-LLM's format: a folder with the model converted to ONNX,
its weights and its tokenizer. `edgeserve export` makes this folder from a Hugging Face checkpoint:

```bash
edgeserve export <checkpoint> <model-dir>
```

| Argument | What it is |
|---|---|
| `<checkpoint>` | a Hugging Face model ID, such as `Qwen/Qwen2.5-0.5B-Instruct-AWQ`, or the folder of your own checkpoint ([your own models](#your-own-models)) |
| `<model-dir>` | a new folder for the exported model, such as `~/models/qwen2.5-0.5b-instruct-awq`; its name becomes the model's name |

The first export installs TensorRT Edge-LLM's exporter (a few minutes, about 5.5 GB of disk, once
for all models). An export takes a few minutes; it runs on the CPU, so it works on the Jetson itself
or on any Linux PC with edgeserve and git installed (no `edgeserve setup` needed there; copy
`<model-dir>` to the Jetson afterwards). It also copies the model's default sampling settings
(`generation_config.json`). For a gated model such as Llama, accept its license on huggingface.co
and pass your access token: `HF_TOKEN=<your token> edgeserve export <checkpoint> <model-dir>`. A
folder that TensorRT Edge-LLM 0.11 already exported needs no export: serve it as it is. GGUF files,
ONNX models from other tools and ready-made engines cannot be served ([what you can bring](#your-own-models)).

**Which models.** Text models of the Llama 3.x, Qwen2/Qwen2.5, Qwen3 and HunYuan (dense) families;
Qwen3 MoE, Qwen3.5 and Nemotron-H should work but are untested
([families and tested models](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#supported-families)).
Choose a precision your GPU supports:

| Weights | Hugging Face name looks like | Jetson Orin | RTX 30/40, A100, A10, L4, L40S | Thor, RTX 50, B200 |
|---|---|---|---|---|
| FP16 / BF16 (the original weights) | `Qwen/Qwen2.5-0.5B-Instruct` | ✅ | ✅ | ✅ |
| INT4 (AWQ or GPTQ) | `…-Instruct-AWQ`, `…-GPTQ-Int4` | ✅ | ✅ | ✅ |
| INT8 (SmoothQuant) | quantize it yourself ([how](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#quantizing-and-other-tools-by-hand)) | ✅ | ✅ | ✅ |
| FP8 | `nvidia/…-FP8` | ❌ | RTX 40, L4, L40S only | ✅ |
| NVFP4 | `nvidia/…-NVFP4` | ❌ | ❌ | ✅ |

Memory decides the model size. With `edgeserve export`, an INT4 model builds its engine in about the
memory it serves in; an FP16 model needs two to three times that. On an 8 GB Orin Nano, FP16 models
up to about 0.6B parameters and INT4 models of 0.5B to 4B were tested, and a pruned 6B INT4 model
with a reduced output vocabulary fits too. Boards and GPUs with more memory run correspondingly larger
models, in any precision they support. More: every argument, what the export does, exporter options
and errors in [docs/EXPORT.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md);
supported models and memory in [docs/MODELS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md).

## 4. Serve

On a Jetson, the model shares the board's memory with everything else running: close what you do not
need first ([how](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/PLATFORMS.md#free-memory-on-a-jetson)).

```bash
edgeserve serve --model <model-dir>
```

The first start converts the model to a TensorRT engine for your GPU (a minute to a few minutes)
and caches it; later starts take seconds. The server listens on `http://127.0.0.1:8000`, which
only this machine can reach. Stop it with Ctrl-C. To test it, run `edgeserve check` in a second
terminal.

Its options:

| Option | Default | What it does |
|---|---|---|
| `--model DIR` | none | the model folder; after `edgeserve setup --model <model-dir>`, plain `edgeserve serve` is enough |
| `--output-dir DIR` | none | save every request, answered or not, as JSON ([saved requests](#saved-requests)) |
| `--host ADDRESS`, `--port N` | 127.0.0.1, 8000 | where the server listens; [step 6](#6-access-from-other-machines) opens it to the network |
| `--max-batch-size N` | 1 | how many requests are generated together ([many requests at once](#many-requests-at-once)) |
| `--max-input-len N` | 3072 | the longest prompt, in tokens; a longer one is refused |
| `--max-new-tokens N` | the model's default, else 512 | the answer length when a request does not set `max_tokens` |
| `--cache-dir DIR` | `~/.cache/edgeserve` | where the converted engines are kept |
| `--queue-size N` | 16 | how many requests may wait for their turn; more are refused (503). `-1`: no limit |
| `--queue-timeout SECONDS` | 600 | a request that has waited this long fails (503). `0`: no limit |
| `--request-timeout SECONDS` | no limit | a request not finished this long after it arrived is stopped (504) |
| `--max-client-requests N` | no limit | how many requests one client may have waiting or running; more are refused (429) |
| `--client-rate-limit N` | no limit | requests a minute per client: N at once, then one every 60/N seconds; more are refused (429) |

To set an option permanently, see [options and settings](#options-and-settings).

### Many requests at once

Requests wait in a queue, and the oldest runs first. With `--max-batch-size N`, up to N waiting
requests of the same kind (chat or completion), with the same `temperature`, `top_p`, `top_k` and
`max_tokens`, are generated together. When it starts, the server reserves memory for N requests of
up to `--max-input-len` + `--max-new-tokens` tokens each, so lower those two to fit a larger N.
Changing any of the three builds a new engine (a few minutes; it is cached too). For example, to
batch up to 3 requests and keep every request until its turn:

```bash
edgeserve serve --model <model-dir> --max-batch-size 3 --max-input-len 1024 --max-new-tokens 256 --queue-size -1
```

- **A client** is one API key ([a key for each client](#a-key-for-each-client)), or one address on a
  server without keys.
- **`--queue-size -1`** keeps every request: there is no wait limit unless you also give
  `--queue-timeout`, and no connection limit. Otherwise one address can open at most 32
  connections, and all clients together 128.
- **A refused request** (429 or 503) gets a `Retry-After` header with the seconds to wait. The OpenAI
  SDK and `edgeserve batch` wait and retry by themselves; with curl, retry yourself. A request
  stopped by `--request-timeout` while running is not retried, as it would run out of time again.
- **A waiting client** has its own timeout: after 600 s without an answer, the OpenAI SDK drops the
  request and sends it again at the back of the queue, twice at most; `OpenAI(..., timeout=3600)`
  waits longer.

### When the engine build runs out of memory

Building an engine needs more memory than serving the model: for a few minutes the builder holds the
model's weights and its own working copies. So a model that fits for serving can still fail to
build, and edgeserve reports that the build "ran out of memory". This is most likely with a large
vocabulary or other FP16 layers kept inside the engine. On a Jetson the CPU and GPU share the
memory, and its built-in swap (zram) is compressed memory, so it adds little room.

A swap file on the disk lets the build move memory it is not using to the disk and finish. Put it
on an SSD or NVMe (swap wears out SD cards):

```bash
sudo fallocate -l 8G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon --priority 1 /swapfile   # zram, if any, stays first
swapon --show
```

Then start `edgeserve serve` again. Serving is unaffected: GPU memory is never moved to disk, and
edgeserve's memory checks count only free memory, not swap. The engine is cached, so the swap is
needed only while it builds; remove it with `sudo swapoff /swapfile && sudo rm /swapfile`.

## 5. Send requests

With curl:

```bash
curl -s http://127.0.0.1:8000/v1/models
curl -s http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" \
  -H "X-Request-Id: demo-001" \
  -d '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'
```

With Python (`pip install openai`):

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")  # no key needed until step 6
model = client.models.list().data[0].id                                  # the served model's name
reply = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "What is a GPU? Answer in one sentence."}],
    max_tokens=100,
    extra_headers={"X-Request-Id": "demo-002"},
)
print(reply.choices[0].message.content)
print(reply.model_extra["metrics"]["decode_tokens_per_s"], "tokens/s")
```

### Request format

A chat request (`POST /v1/chat/completions`) is a JSON object; only `messages` is required:

```json
{
  "messages": [
    {"role": "system", "content": "Summarize the text in two sentences."},
    {"role": "user", "content": "<the text>"}
  ],
  "max_tokens": 160,
  "temperature": 0,
  "request_id": "meeting-0001",
  "metadata": {"source": "room-12"}
}
```

| Field | Meaning | Default |
|---|---|---|
| `messages` | the conversation, oldest first. `role`: `system` (instructions), `user`, `assistant` (an earlier answer) or `developer` (same as `system`); `content`: the text. Up to 64 messages of up to 128 KB each | required |
| `max_tokens` | the longest answer, in tokens (`max_completion_tokens` means the same) | `--max-new-tokens` (the model's `generation_config.json`, else 512) |
| `temperature` | randomness, 0 to 2. 0 always picks the most likely next token: the same answer every time | the model's `generation_config.json`, else 0 |
| `top_p`, `top_k` | when sampling, choose only among the most likely tokens: those that make up `top_p` (above 0, up to 1) of the probability, or the `top_k` (up to 1024) most likely | the model's `generation_config.json`, else no limit |
| `seed` | a number that makes sampling repeatable | random, saved in the record |
| `stop` | a string, or up to 4, that ends the answer | none |
| `stream` | `true`: send the answer in pieces as it is generated; add `"stream_options": {"include_usage": true}` for a last piece with the token counts and metrics | `false` |
| `request_id` | your ID for this request (the same as the `X-Request-Id` header) | `req_<random hex>` |
| `metadata` | up to 16 string pairs of your own (keys up to 64 characters, values up to 512), saved in the record | none |
| `return_record` | `true`: the response also holds the full record, with per-token timings (not with `stream`) | `false` |
| `model` | the served model's name, as `GET /v1/models` lists it; optional | the served model |
| `n` (only 1), `frequency_penalty` and `presence_penalty` (only 0), `logprobs` (only `false`), `user`, `safety_identifier`, `prompt_cache_key`, a message's `name` | accepted so that every OpenAI client works; they change nothing | — |

A completion request (`POST /v1/completions`) takes `prompt`, one string with the model's chat
template already applied, instead of `messages`. Any other field is refused with HTTP 400, never
ignored.

**Request IDs.** Send your ID in the `X-Request-Id` header or the `request_id` field: 1 to 512
printable ASCII characters (with letters, digits, `.`, `_` and `-`, the record's file name is the ID
itself). Without one, the server makes one (`req_…`). The ID comes back in the response's
`request_id` and `id` (`chatcmpl-<id>`) and in its `X-Request-Id` header. With
`edgeserve serve --output-dir <output-dir>`, every request is saved as
`<output-dir>/records/<id>.json`, which `GET /v1/records/<id>` also returns. A request with the ID
of one that is still running gets HTTP 409.

The response is OpenAI's, plus `request_id` and `metrics`:

```json
{"id": "chatcmpl-meeting-0001", "object": "chat.completion", "model": "<model name>",
 "choices": [{"index": 0, "message": {"role": "assistant", "content": "…"}, "finish_reason": "stop"}],
 "usage": {"prompt_tokens": 162, "completion_tokens": 72, "total_tokens": 234},
 "request_id": "meeting-0001", "metrics": {"ttft_ms": 303, "decode_tokens_per_s": 25.9, …}}
```

For streaming, send `"stream": true` (`stream=True` in the SDK). Every field, error code and
metric: [docs/API.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/API.md).

## 6. Access from other machines

```bash
edgeserve setup --network
```

This creates three things in `~/.config/edgeserve/`:

- **An API key.** It is printed once. To see it again:
  `grep EDGESERVE_API_KEY ~/.config/edgeserve/edgeserve.env`.
- **An HTTPS certificate**, and `edgeserve-ca.crt`, the file clients need to trust it.
- **The setting to listen on the network** (all interfaces, port 8000).

Restart `edgeserve serve`. From then on the server speaks only HTTPS and every `/v1/` request needs
the key, on this machine too (`edgeserve check` finds both by itself), so the `http://` examples of
step 5 stop working.

**On each client machine** you need the server's `edgeserve-ca.crt` and the key. Below,
`<server-ip>` is the server's IP address (`edgeserve setup --network` prints it) and `<user>` is your
user name on the server. The first two lines copy both from the server over SSH; you can also paste
the key yourself, e.g. `export EDGESERVE_API_KEY=es-…`.

Linux and macOS (Terminal):

```bash
scp <user>@<server-ip>:.config/edgeserve/edgeserve-ca.crt .
export EDGESERVE_API_KEY=$(ssh <user>@<server-ip> "sed -n 's/^EDGESERVE_API_KEY=//p' .config/edgeserve/edgeserve.env")
curl --cacert edgeserve-ca.crt -H "Authorization: Bearer $EDGESERVE_API_KEY" https://<server-ip>:8000/v1/models
curl --cacert edgeserve-ca.crt https://<server-ip>:8000/v1/chat/completions \
  -H "Authorization: Bearer $EDGESERVE_API_KEY" -H "Content-Type: application/json" -H "X-Request-Id: client-001" \
  -d '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'
```

Windows (PowerShell; `scp` and `curl.exe` come with Windows 10 and 11):

```powershell
scp <user>@<server-ip>:.config/edgeserve/edgeserve-ca.crt .
$env:EDGESERVE_API_KEY = ssh <user>@<server-ip> "sed -n 's/^EDGESERVE_API_KEY=//p' .config/edgeserve/edgeserve.env"
curl.exe --ssl-no-revoke --cacert edgeserve-ca.crt -H "Authorization: Bearer $env:EDGESERVE_API_KEY" https://<server-ip>:8000/v1/models
Set-Content -Encoding ascii request.json '{"messages": [{"role": "user", "content": "What is a GPU? Answer in one sentence."}], "max_tokens": 100}'
curl.exe --ssl-no-revoke --cacert edgeserve-ca.crt https://<server-ip>:8000/v1/chat/completions `
  -H "Authorization: Bearer $env:EDGESERVE_API_KEY" -H "Content-Type: application/json" -H "X-Request-Id: client-002" `
  -d "@request.json"
```

Type `curl.exe`, not `curl` (in Windows PowerShell, `curl` is a different command).
`--ssl-no-revoke` is needed because Windows otherwise asks whether the certificate was revoked,
which a private CA cannot answer. The request body goes in a file because PowerShell changes the
quotes inside inline JSON.

Python, on any of them: make a virtual environment with the OpenAI SDK
(`python3 -m venv <client-venv>` and `<client-venv>/bin/pip install openai httpx`; on Windows
`py -m venv <client-venv>` and `<client-venv>\Scripts\pip install openai httpx`), then run this
with its Python:

```python
import os, ssl
import httpx
from openai import OpenAI

client = OpenAI(base_url="https://<server-ip>:8000/v1", api_key=os.environ["EDGESERVE_API_KEY"],
                http_client=httpx.Client(verify=ssl.create_default_context(cafile="edgeserve-ca.crt")))
reply = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[{"role": "user", "content": "What is a GPU? Answer in one sentence."}],
    max_tokens=100,
    extra_headers={"X-Request-Id": "client-003"},
)
print(reply.model_extra["request_id"], reply.choices[0].message.content)
```

Every other request works as in [step 5](#5-send-requests), with `https://<server-ip>:8000`, the
key and the CA file.

- **New key:** delete the `EDGESERVE_API_KEY=` line in `~/.config/edgeserve/edgeserve.env`, run
  `edgeserve setup --network` (it prints the new key) and restart the server. You can also write
  your own key on that line.
- **New IP address or host name:** run `edgeserve setup --network` again and restart the server.
  Clients keep their `edgeserve-ca.crt`.
- **Firewall:** allow only your own network to reach port 8000. `edgeserve setup --network` prints
  the `ufw` commands for your network.
- **This machine only again:** set `EDGESERVE_HOST=127.0.0.1` in the settings file.
- **`/metrics` and `/docs`** need the key too, except when opened at `127.0.0.1` on the server
  itself; `/health` stays open.

### A key for each client

Rather than sharing the key from `edgeserve setup --network` (client `default`), you can give each
person or machine its own. You can then remove one without changing the others, and each saved
request shows which client sent it. On the server:

```bash
edgeserve keys add <name>       # make a key for client <name>; it is shown only now
edgeserve keys list             # the clients that have a key
edgeserve keys remove <name>    # that key stops working at once
```

The client uses its own key in place of the `default` key, and still needs `edgeserve-ca.crt`. To
skip copying the key by hand, make it straight into a file (instead of the plain `keys add` above),
then fetch it on the client over SSH, which also deletes the file:

```bash
edgeserve keys add <name> | awk '$1 ~ /^es-/ {print $1}' > ~/<name>.key && chmod 600 ~/<name>.key   # on the server
export EDGESERVE_API_KEY=$(ssh <user>@<server-ip> "cat <name>.key && rm <name>.key")                # on the client
```

On Windows (PowerShell), the client line is
`$env:EDGESERVE_API_KEY = ssh <user>@<server-ip> "cat <name>.key && rm <name>.key"`.

- Adding or removing a key works at once, also on a running server.
- A client reads only its own saved requests; the `default` key reads all of them.
- `--max-client-requests` and `--client-rate-limit` count each key separately.
- edgeserve stores only a hash of each key, so a lost key cannot be shown again: remove it and add a
  new one.

## Saved requests

With `edgeserve serve --output-dir <output-dir>`, every request is saved:

| File | What it holds |
|---|---|
| `<output-dir>/records/<id>.json` | the latest attempt of each request ID: the request, the answer, its client, metrics per request and per token, and its status: `completed`, `failed`, `cancelled` or `rejected` (refused before it was queued; `http_status` and `error` say why) |
| `<output-dir>/results.jsonl` | one line per attempt, in OpenAI's batch output format |

A refused request gets a `results.jsonl` line, and a record when it named its ID (`X-Request-Id`, or
`request_id` in a valid body) and that ID has no record yet: a refusal never replaces an earlier
record. At most 60 refusals a minute per client are saved, and requests without a valid key never.
`GET /v1/records/<id>` returns a record, also from another machine (with client keys, only the
client's own). The folder is readable only by the user running the server, as records hold prompts
and answers. Every field:
[docs/METRICS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/METRICS.md).

## Your own models

Find what you have:

| You have | What to do |
|---|---|
| A Hugging Face checkpoint folder (`config.json` and `*.safetensors`): the original model, or your fine-tuned, pruned or quantized version of it | export it: `edgeserve export <checkpoint-dir> <model-dir>`, then serve `<model-dir>` |
| A LoRA adapter | load its base model in FP16/BF16 (not 4-bit), merge the adapter (PEFT's `merge_and_unload()`), save the model and the tokenizer into one folder with `save_pretrained()`, then export that folder |
| A folder already exported (converted to ONNX) by TensorRT Edge-LLM 0.11: its `config.json` has `"edgellm_version": "0.11.…"` and sits next to `model.onnx` | nothing: serve it as it is, `edgeserve serve --model <model-dir>` (if the export made an `llm` folder, pass that) |
| A folder exported by another TensorRT Edge-LLM version | export the checkpoint again with `edgeserve export`; edgeserve refuses the folder and says so |
| An ONNX model made by another tool (such as Optimum), a GGUF file (llama.cpp, Ollama), an engine built by TensorRT-LLM or another tool, or an INT4 model in another format (bitsandbytes, compressed-tensors, EXL2, HQQ) | cannot be served: export the Hugging Face checkpoint it was made from, or an AWQ or GPTQ version of it |
| A model of another family (such as Mistral, Gemma or Phi), or one that needs images or audio | not supported, except Qwen3.5, whose image part the export leaves out ([supported families](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#supported-families)) |

**A checkpoint folder** (`<checkpoint-dir>`) holds these files, as `save_pretrained()` writes them:

| File | Needed | What it holds |
|---|---|---|
| `config.json` | yes | the architecture, with `model_type` such as `llama`, `qwen2` or `qwen3` ([supported families](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#supported-families)) |
| `model.safetensors` (or shards with `model.safetensors.index.json`; `pytorch_model.bin` also works) | yes | the weights |
| `tokenizer.json`, `tokenizer_config.json` | yes | the tokenizer, and the chat template (`chat_template` in `tokenizer_config.json`, or a `chat_template.jinja` file) |
| `generation_config.json` | no | default sampling settings and end-of-sequence tokens; copied into `<model-dir>` |

- **Fine-tuned:** no changes needed; save the model and its tokenizer into one folder.
- **Pruned:** fewer layers, or fewer heads or a smaller hidden size in every layer, need only the
  matching numbers in `config.json`. A different MLP width in each layer needs
  `"intermediate_sizes": [<width of layer 0>, <width of layer 1>, …]` in the checkpoint's `config.json`,
  one entry per layer (multiples of 128 for INT4). An exported `<model-dir>` needs nothing more: the
  widths are then part of its `model.onnx`, and `edgeserve info --model <model-dir>` lists them (`mlp_widths`).
- **Quantized:** INT4 AWQ or GPTQ (group size 128), or FP8, NVFP4, INT8 SmoothQuant or INT4 AWQ made
  with NVIDIA ModelOpt or Edge-LLM's quantizer, for GPUs that run them
  ([INT4 quality](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#quality-of-int4-models)).
- **Smaller output vocabulary:** add `-- --reduced-vocab-dir <vocab-dir>` to the export command
  (`<vocab-dir>` is the folder with `vocab_map.safetensors`).

**An exported folder** is served as a whole: copy all of it
([every file](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#the-model-folder)). `edgeserve info --model <model-dir>` checks it
without starting a server. A folder made with Edge-LLM's own `tensorrt-edgellm-export` (without
edgeserve) differs in three ways: it has no `generation_config.json` (copy the checkpoint's in, or
requests decode greedily), its INT4 AWQ and asymmetric GPTQ weights are slightly clipped, and unless
it was exported with `--externalize-weights`, its engine build needs more memory. If you have the
checkpoint, export it again with `edgeserve export`
([INT4 quality](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#quality-of-int4-models)).

To host your model for your network, starting at every boot:

```bash
edgeserve export <checkpoint-dir> <model-dir>        # once (skip it for a folder that is already exported)
edgeserve setup --network --model <model-dir>        # make it the default model; API key and HTTPS
edgeserve service install                            # serve it now and at every boot
```

Details, quantizing a model yourself and checking a pruned export: [docs/EXPORT.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#your-own-checkpoints).

## Batch jobs

```bash
edgeserve batch <requests.jsonl> --output-dir <results-dir>
```

`<requests.jsonl>` has one request per line, for example:

```jsonl
{"id": "q1", "messages": [{"role": "user", "content": "What is CUDA?"}], "max_tokens": 200}
{"id": "q2", "prompt": "<a prompt with the chat template already applied>"}
```

The OpenAI Batch format (`custom_id`, `method`, `url`, `body`) works too. Results go into
`<results-dir>/records/<id>.json`, `<results-dir>/results.jsonl` and `<results-dir>/run.json`
(counts, throughput, latency, energy). Running the same command again skips the finished IDs, so
an interrupted batch resumes. For another machine's server, set `EDGESERVE_API_KEY` as in
[step 6](#6-access-from-other-machines) and add `--server https://<server-ip>:8000 --ca-cert edgeserve-ca.crt`.

## Run as a service

```bash
edgeserve setup --model <model-dir>    # the model the service serves (add --network for other machines)
edgeserve service install              # starts the server now and at every boot; asks for your sudo password
journalctl -u edgeserve -f             # its log
edgeserve service remove               # stop and remove it
```

Run these commands as your normal user, not with sudo. The service uses the settings file: after
changing it, restart the service with `sudo systemctl restart edgeserve`.
`edgeserve service install --print` only shows the unit file.

## Uninstall

First [remove the Hugging Face downloads](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md#disk-space)
you no longer need (they are in `~/.cache/huggingface`), then:

```bash
edgeserve service remove     # if you installed the service
rm -rf ~/.config/edgeserve ~/.local/share/edgeserve ~/.cache/edgeserve
```

Finally, delete `<venv-dir>` (or the cloned `edgeserve` folder) and the model folders you no longer
need. After a new install, `edgeserve setup --network` makes a new key and `edgeserve-ca.crt`: give
both to the clients again.

## Command reference

| Command | What it does |
|---|---|
| `edgeserve setup [--network] [--model DIR]` | build the GPU runtime; `--model` sets the default model; `--network` adds an API key, HTTPS and listening on the network |
| `edgeserve export CHECKPOINT MODEL_DIR [-- exporter options]` | export a Hugging Face checkpoint (an ID or a folder) into a model folder |
| `edgeserve serve` | run the server |
| `edgeserve build` | only convert the model to an engine (or check the cached one), then exit |
| `edgeserve info` | settings and where each comes from, the GPU runtime, the hardware; with `--model`, the model, its memory estimate and its cached engine |
| `edgeserve check [--server URL] [--ca-cert FILE]` | test a running server end to end |
| `edgeserve keys add NAME`, `edgeserve keys list`, `edgeserve keys remove NAME` | API keys, one per client |
| `edgeserve batch INPUT --output-dir DIR [--server URL] [--ca-cert FILE] [--concurrency N]` | run a JSONL file of requests |
| `edgeserve service install [--print]`, `edgeserve service remove` | run the server at boot (systemd) |
| `edgeserve --version` | the edgeserve and TensorRT Edge-LLM versions |

## Options and settings

Each option can also be set as a line `NAME=value` in `~/.config/edgeserve/edgeserve.env` (the file
`edgeserve setup` writes; restart the server after editing it) or as an environment variable. An
option on the command line wins over the environment, which wins over the file. `edgeserve info`
shows where each value came from.

**The model and its engine** (`serve`, `build` and `info`):

| Option | Setting | Default | Meaning |
|---|---|---|---|
| `--model DIR` | `EDGESERVE_MODEL` | none | the model folder (an absolute path in the file) |
| `--max-input-len N` | `EDGESERVE_MAX_INPUT_LEN` | 3072 | longest prompt in tokens; longer prompts are refused, never cut |
| `--max-new-tokens N` | `EDGESERVE_MAX_NEW_TOKENS` | the model's `generation_config.json`, else 512 | output length when a request sets no `max_tokens` |
| `--max-batch-size N` | `EDGESERVE_MAX_BATCH_SIZE` | 1 | requests generated together; each one needs its own memory |
| `--cache-dir DIR` | `EDGESERVE_CACHE_DIR` | `~/.cache/edgeserve` | where engines are kept |

**The server** (`serve`):

| Option | Setting | Default | Meaning |
|---|---|---|---|
| `--output-dir DIR` | `EDGESERVE_OUTPUT_DIR` | none | save every request here ([saved requests](#saved-requests)) |
| `--host ADDRESS` | `EDGESERVE_HOST` | 127.0.0.1 | `127.0.0.1`: this machine only; `0.0.0.0`: the network (needs an API key) |
| `--port N` | `EDGESERVE_PORT` | 8000 | port |
| `--queue-size N` | `EDGESERVE_QUEUE_SIZE` | 16 | requests that may wait beyond the running batch; `-1`: no limit, nor a wait or connection limit |
| `--queue-timeout SECONDS` | `EDGESERVE_QUEUE_TIMEOUT` | 600 | how long a request may wait; `0`: no limit |
| `--request-timeout SECONDS` | `EDGESERVE_REQUEST_TIMEOUT` | 0 | how long a request may take from arrival to its last token; `0`: no limit |
| `--max-client-requests N` | `EDGESERVE_MAX_CLIENT_REQUESTS` | 0 | requests one client may have waiting or running; `0`: no limit |
| `--client-rate-limit N` | `EDGESERVE_CLIENT_RATE_LIMIT` | 0 | requests per minute per client; `0`: no limit |

**Keys, certificates and the runtime** (no command-line option, so a key never shows in process lists; set
them in the settings file or the environment):

| Setting | Meaning |
|---|---|
| `EDGESERVE_API_KEY` | the key of client `default`, sent as `Authorization: Bearer <key>`; more keys: [edgeserve keys](#a-key-for-each-client) |
| `EDGESERVE_TLS_CERT`, `EDGESERVE_TLS_KEY` | HTTPS certificate and private key (PEM files) |
| `EDGESERVE_NATIVE_DIR` | use another GPU runtime build (development) |

Changing `--max-input-len`, `--max-new-tokens` or `--max-batch-size` builds a new engine (cached
too). `check` and `batch` use this machine's server unless given `--server URL`, and trust a
server's certificate with `--ca-cert FILE`; `batch --concurrency N` sets how many requests are in
flight (default: the server's batch size + 1). Environment only: `EDGESERVE_CONFIG_DIR` (another
settings folder), `EDGESERVE_API_KEY` on a client (the key `check` and `batch` send), and for
`edgeserve setup`, `TRT_PACKAGE_DIR` (a TensorRT tarball folder) and `JOBS` (parallel compile jobs).

## Troubleshooting

| Problem | Fix |
|---|---|
| `edgeserve setup` lists missing packages | install them with the `apt` line it prints, then run it again |
| "the GPU runtime is not built yet" or "… from other sources" | run `edgeserve setup` |
| `/health/ready` says `building_engine` | normal on the first start of a model or new limits |
| "not enough memory" | [free memory](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/PLATFORMS.md#free-memory-on-a-jetson) on a Jetson; use an INT4 or a smaller model, lower `--max-input-len`, keep `--max-batch-size 1` |
| the engine build "ran out of memory", although the model fits for serving | add a swap file for the build: [when the engine build runs out of memory](#when-the-engine-build-runs-out-of-memory) |
| "exported with TensorRT Edge-LLM x.y" | export the model again with `edgeserve export` |
| "is a Hugging Face checkpoint, not an exported model" | export it first: `edgeserve export <checkpoint-dir> <model-dir>`, then serve `<model-dir>` |
| "is a GGUF file", "holds GGUF files" or "holds an ONNX model made by another tool" | export the Hugging Face checkpoint the model was made from |
| "has no config.json" | `--model` must be the folder `edgeserve export` made; for `export`, the checkpoint folder must hold `config.json` |
| HTTP 401 | send the key: `Authorization: Bearer <key>`; a key removed with `edgeserve keys remove` no longer works |
| HTTP 429 `too_many_requests` or `rate_limit_exceeded` | the client is over `--max-client-requests` or `--client-rate-limit`: retry after the `Retry-After` seconds |
| HTTP 503 `server_is_overloaded` or `queue_timeout` | the queue is full or the wait too long: retry after `Retry-After`; raise `--queue-size` or `--queue-timeout`, or batch more with `--max-batch-size` |
| HTTP 504 `time_limit_exceeded` | the request took longer than `--request-timeout`: raise the limit, or ask for fewer `max_tokens` (a request stopped while running is not sent again automatically) |
| certificate errors | give the client `edgeserve-ca.crt`; use a host name or address the certificate lists, or run `edgeserve setup --network` again after an address change |
| `<host>.local` does not connect | use the IP address; mDNS can announce a Docker bridge address (fix: `deny-interfaces=docker0` in `/etc/avahi/avahi-daemon.conf`) |
| other machines cannot connect | run `edgeserve setup --network` and restart; check the firewall (`sudo ufw status`) |
| answers differ from Hugging Face | the server log lists `generation_config.json` options Edge-LLM does not support (e.g. `repetition_penalty`); for INT4 models see [INT4 quality](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#quality-of-int4-models) |
| `edgeserve export` fails | the error table in [docs/EXPORT.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md#errors) |

## Documentation

| Topic | Document |
|---|---|
| Exporting: every argument, what the export does, your own fine-tuned, pruned and quantized checkpoints, INT4 quality, exporter options, errors | [docs/EXPORT.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/EXPORT.md) |
| Models: supported families, tested models, precision per GPU, memory, disk space | [docs/MODELS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/MODELS.md) |
| The API: endpoints, request fields, sampling, streaming, errors, request IDs, records | [docs/API.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/API.md) |
| Metrics: every per-request and per-token metric and how it is measured | [docs/METRICS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/METRICS.md) |
| Platforms: requirements per device, Jetson settings, measured performance, Edge-LLM patches | [docs/PLATFORMS.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/PLATFORMS.md) |
| How it works: processes, GPU runtime, engine cache, request lifecycle, worker protocol | [docs/ARCHITECTURE.md](https://github.com/Surya-Rayala/edgeserve/blob/main/docs/ARCHITECTURE.md) |

## Development

```bash
pip install -e ".[test]"
python -m pytest            # no GPU needed; needs g++ and nlohmann-json3-dev
```

## License

Apache License 2.0 ([LICENSE](https://github.com/Surya-Rayala/edgeserve/blob/main/LICENSE)). edgeserve
downloads and builds NVIDIA TensorRT Edge-LLM (Apache 2.0), which is not part of this package. Models
have their own licenses.
