Metadata-Version: 2.4
Name: gpuroutertest
Version: 0.3.0
Summary: Launch, load-test, and tear down a GPU-backed HTTP API on your own AWS account with one command.
Author: Susmit Kulkarni
License: MIT
Project-URL: Homepage, https://github.com/HolboxAI/gpuroutertest
Project-URL: Issues, https://github.com/HolboxAI/gpuroutertest/issues
Keywords: aws,ec2,gpu,api,deploy,cli,benchmark,load-testing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.26
Requires-Dist: typer>=0.9
Provides-Extra: benchmark
Requires-Dist: openai>=1.0; extra == "benchmark"
Requires-Dist: httpx>=0.23; extra == "benchmark"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: moto[ec2]>=5; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: build>=1; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Requires-Dist: openai>=1.0; extra == "dev"
Requires-Dist: httpx>=0.23; extra == "dev"
Dynamic: license-file

# gpuroutertest

Launch a ready-to-call **HTTP API on your own AWS GPU** with a single command. `gpuroutertest` picks a GPU that fits, boots the instance, loads the model you choose, and hands you a **URL + API key**. Load-test it, then tear it down just as fast.

Built for people who want a private GPU-backed endpoint on AWS **without** hand-rolling EC2, security groups, drivers, and server flags. No console clicking, no infra code — one command up, one command down.

## Install

```bash
pip install gpuroutertest
```

You need AWS credentials (`aws configure`, `AWS_PROFILE`, or SSO) with permission to manage EC2, plus GPU (`G`/`P`) instance quota in your region. The optional `--spot` failover ([below](#spot-instances-with-automatic-failover)) additionally needs permission for Elastic IPs, Lambda, EventBridge, and IAM (to create its recovery role).

## What it does

1. **`deploy`** — launch a GPU instance, load a model, and expose it at `http://<host>:8000/v1` behind an API key.
2. **call it** — a standard chat-style REST API; point any HTTP client at the URL.
3. **`--test`** — a built-in load-tester that reports latency and throughput, measured from both the client and the server.
4. **`delete`** — terminate the instance and stop billing.

## CLI quickstart

```bash
# See the available models and the GPU tiers they run on
gpuroutertest list-models

# Launch one. A model is just an id string — the same one you pass to `deploy`,
# that `list-models` prints, and that you send as "model" when calling the API.
gpuroutertest deploy zai-org/glm-4-9b-chat-hf --size small --region us-east-1 --profile myprofile

# List what's running (state lives in AWS tags, so keys are never lost)
gpuroutertest ps --region us-east-1

# Inspect / fetch the endpoint / read boot logs
gpuroutertest status   <api_key|instance-id> --region us-east-1
gpuroutertest endpoint <api_key|instance-id> --region us-east-1
gpuroutertest logs     <api_key|instance-id> --region us-east-1

# Tear everything down (stops billing)
gpuroutertest delete   <api_key|instance-id> --region us-east-1 -y
```

Every command takes `--profile/-p` and `--region/-r`. `deploy` also supports:

| Flag | Purpose |
|------|---------|
| `--size` | `small` / `medium` / `large` tier from the registry |
| `--hf-token` | access token for gated models (or set `HF_TOKEN`) |
| `--model-uri` | pull weights from a same-region public S3 prefix (`s3://bucket/prefix`) instead of HuggingFace — see [Loading weights from S3](#loading-weights-from-s3-large-models) |
| `--cidr` | restrict who can reach port 8000 (default `0.0.0.0/0`) |
| `--ttl` | auto-terminate after N minutes — a cost guard for dev instances |
| `--timeout` | minutes to wait for the health check (default 30) |
| `--no-wait` | return as soon as the instance is running |
| `--spot` | run on spot capacity behind a stable Elastic IP, and auto-relaunch onto a new spot instance if reclaimed — see [Spot instances with automatic failover](#spot-instances-with-automatic-failover) |

## Loading weights from S3 (large models)

By default `deploy` pulls the model straight from HuggingFace. For **small models that's perfectly fine** — leave it alone. But for **large models** (hundreds of GB), the HuggingFace download becomes the slow, flaky part of every boot. For those, host the weights once in a **public S3 bucket in the same region** as your GPUs and point `deploy` at it:

```bash
gpuroutertest deploy zai-org/GLM-5.2-FP8 \
  --size large \
  --model-uri s3://glm-5-2-fp8-weights/zai-org/GLM-5.2-FP8 \
  --region us-east-1 \
  --ttl 120
```

With `--model-uri`, the instance fetches the weights from that S3 prefix with [s5cmd](https://github.com/peak/s5cmd) (`--no-sign-request`) instead of HuggingFace. Because the bucket is in-region, the transfer is **free and multi-GB/s**, and vLLM serves the local copy under the original model id — callers address the endpoint exactly the same way. Without `--model-uri`, nothing changes; it's opt-in per deploy.

Requirements:

- The bucket must be in the **same region** as `--region` (that's what makes it free + fast; cross-region would be slow and incur egress).
- The prefix must point at the folder that directly contains `config.json`, the safetensors shards, `model.safetensors.index.json`, and the tokenizer files.
- The value must start with `s3://` (validated before any AWS call, so typos fail fast).

**Seeding the bucket** (one-time, per model): [`scripts/seed_model_to_s3.py`](scripts/seed_model_to_s3.py) streams a HuggingFace repo straight into a public S3 bucket without ever staging it on local disk (so it needs no free space for the model), and is resumable. For an overnight, unattended run use [`scripts/seed_overnight.sh`](scripts/seed_overnight.sh), which re-runs the seeder until every file is confirmed uploaded.

```bash
python scripts/seed_model_to_s3.py \
  --model  zai-org/GLM-5.2-FP8 \
  --bucket glm-5-2-fp8-weights \
  --region us-east-1
```

## Spot instances with automatic failover

Spot GPUs are far cheaper than on-demand, but AWS can reclaim them with only a
~2-minute warning. `--spot` runs your endpoint on spot capacity and arms an
automatic failover so it comes back on its own after a reclaim:

```bash
gpuroutertest deploy zai-org/glm-4-9b-chat-hf --size small --region us-east-1 --spot
```

**What you get:** a **stable endpoint URL**. `--spot` allocates an Elastic IP up
front and serves the endpoint from it, so the URL never changes — even as the
underlying instance is replaced.

**How the jump works.** At deploy time, in addition to the instance, `--spot`
provisions three small pieces of infrastructure, all tagged with the deployment's
API key so they're torn down together on `delete`:

1. an **Elastic IP** — the stable address in front of the endpoint;
2. an **EventBridge rule** that fires on the `EC2 Spot Instance Interruption
   Warning` event; and
3. a per-deployment **Lambda** that the rule invokes.

```text
Spot reclaim warning (~2 min notice)
        ↓
EventBridge rule  →  recovery Lambda
        ↓
Launch a replacement spot instance (same model, size, and boot config;
tries each AZ on capacity errors)
        ↓
Re-point the Elastic IP at the replacement  →  endpoint URL unchanged
```

The replacement is an exact clone of the original launch — same model, weights
source (HuggingFace or `--model-uri` S3), size, and vLLM flags. The rule isn't
pinned to an instance id, so it keeps protecting each replacement in turn across
any number of reclaims; the Lambda ignores warnings for instances that aren't
part of this deployment.

**The one caveat — there is a downtime gap.** Loading a model into GPU memory
takes several minutes to (for large models) 8–15 minutes, which is far longer
than the ~2-minute reclaim notice. So failover is *automatic recovery*, not
*zero-downtime*: after a reclaim the endpoint is unreachable until the
replacement finishes booting and reloading the model, then it comes back at the
same URL. If you need to ride through a reclaim with no gap, run two independent
deployments behind your own load balancer. (Because inference is stateless there
is nothing to checkpoint — the replacement simply reloads the same weights.)

**Status and teardown.** `gpuroutertest status <api_key>` shows `Capacity: spot
(failover armed)` for a spot deployment. `gpuroutertest delete <api_key>` removes
the instance, the Lambda, the EventBridge rule, and the Elastic IP in one shot
(the shared IAM recovery role is left in place for reuse and costs nothing).

## Calling the endpoint

`deploy` prints an endpoint URL and API key. It's a standard chat-style REST API, so any HTTP client works:

```bash
curl $ENDPOINT/chat/completions \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"zai-org/glm-4-9b-chat-hf","messages":[{"role":"user","content":"Hello"}]}'
```

## Load-testing (`--test`)

Once an endpoint is up, `--test` drives it under a fixed, repeatable workload and
reports how it performs. It measures the same run from **two vantage points**:

- **Client side** — what a real caller experiences, *including* the network round-trip to AWS.
- **Server side** — the machine's own numbers, read from its built-in `/metrics` page.

Comparing the two tells you how much of the latency is the server itself versus
the network in between.

```bash
gpuroutertest --test \
  --endpoint http://<host>:8000/v1 \
  --api-key <api_key> \
  --model zai-org/glm-4-9b-chat-hf \
  --concurrency 10 \
  --prompt-length 1k
```

Pick which side(s) to show:

- **`--test`** (bare) or **`--test=both`** — client + server side by side, plus the network gap.
- **`--test=client`** — client numbers only (skips the server scrape).
- **`--test=server`** — server numbers only.

### Flags

| Flag | Purpose |
|------|---------|
| `--test[=client\|server\|both]` | Run a load test and pick which side(s) to report (bare = both) |
| `--endpoint` | Base URL of the server (e.g. `http://host:8000/v1`) |
| `--api-key` | API key for the endpoint |
| `--model` | Model id the server expects |
| `--concurrency` / `-c` | Requests kept in flight at once (N), default `1` |
| `--prompt-length` / `-l` | Test-input size tier: `100`, `1k`, or `8k` (default `100`) |
| `--requests` | Total requests to send (default: one per bundled input; repeats to fill larger counts) |
| `--warmup` | Throwaway warmup requests fired before timing (default `3`, `0` to disable) |
| `--metrics-url` | Override the server `/metrics` URL (default: derived from `--endpoint`) |
| `--matrix` | Run the full sweep (concurrency 1/10/100 × input 100/1k/8k) instead of a single run |

### What the numbers mean

- **TTFT (Time To First Token)** — how long until the *first* piece of the response comes back. This is the responsiveness a caller feels (like time-to-first-byte).
- **TPM (Tokens per Minute)** — output throughput. *(A "token" is a chunk of the response, roughly ¾ of a word.)*
- **Avg / P50 / P95** — the average, the median (typical request), and the 95th percentile (your slow 5% — the tail). When P95 sits far above P50, some requests are queueing.

**Warmup matters.** The very first request to a freshly-booted server pays a
one-time cold-start cost (loading data into GPU memory, warming caches) that can
be several seconds. `--warmup` fires a few throwaway requests first — discarded,
*before* the clock starts — so that cost never skews your results.

### Sample output

```
Model: zai-org/glm-4-9b-chat-hf
Prompt Length: 1K
Concurrency: 10

                      Client        Server
Avg TTFT             1694 ms       1077 ms
P50 TTFT             1663 ms       1300 ms
P95 TTFT             2408 ms       2380 ms
TPM                     9757          9757

Network latency (client - server TTFT): 617 ms
(server observed 24 request(s) during the run)
```

Server-side numbers are best-effort: if `/metrics` can't be reached (e.g. blocked
by a security group), the test falls back to client-side numbers with a warning.

Test inputs come from bundled files in
[`sdk/gpuroutertest/prompts/`](sdk/gpuroutertest/prompts/) — one input per entry,
separated by a lone `---` line. Edit them to use your own. When you ask for more
requests than there are unique inputs, they repeat — so pass enough `--requests`
to keep high concurrency busy.

The load-tester needs two optional packages:

```bash
pip install gpuroutertest[benchmark]
```

### Full matrix (`--matrix`)

Add `--matrix` to sweep the whole grid in one shot — concurrency `1/10/100` ×
input size `100/1k/8k` (9 runs). It prints the table and also saves it to
**`inference_benchmark.txt`**. A failing cell shows up as an `ERROR` row instead
of aborting the sweep; `Srv TTFT` / `TPM` / `Network Latency` show `-` for any
cell with no server-side numbers. `TPM` and `Network Latency` are server-side
(from vLLM's `/metrics`).

```bash
gpuroutertest --test --matrix \
  --endpoint http://<host>:8000/v1 --api-key <api_key> \
  --model zai-org/glm-4-9b-chat-hf --requests 100
```

```
Benchmark Report
Model: zai-org/glm-4-9b-chat-hf
(TTFT/Network Latency in ms; TPM in tokens/min; both server-side, from vLLM. Network Latency = client - server TTFT)

Length    Conc    Avg TTFT    Srv TTFT           TPM   Network Latency  Fail
---------------------------------------------------------------------------
100          1         604         106          1415               499     0
100         10         690         184          8208               506     0
100        100         734         225         15857               509     0
1K          10        1715        1034          7413               681     0
8K          10        7406        6654          4924               751     0
```

## Library usage

```python
import gpuroutertest as gr

dep = gr.deploy("zai-org/glm-4-9b-chat-hf", size="small", region="us-east-1", profile="myprofile")
print(dep.endpoint_url, dep.api_key)

# For large models, pull weights from a same-region public S3 bucket instead of
# HuggingFace (see "Loading weights from S3" above):
# dep = gr.deploy("zai-org/GLM-5.2-FP8", size="large", region="us-east-1",
#                 model_uri="s3://glm-5-2-fp8-weights/zai-org/GLM-5.2-FP8")

# Run on spot capacity with automatic failover behind a stable Elastic IP
# (see "Spot instances with automatic failover" above):
# dep = gr.deploy("zai-org/glm-4-9b-chat-hf", size="small", region="us-east-1", spot=True)

# ... call the HTTP API at dep.endpoint_url with dep.api_key ...

gr.destroy(dep.api_key, region="us-east-1")
```

You can also run a load test from Python:

```python
from gpuroutertest import run_benchmark

result = run_benchmark(endpoint=dep.endpoint_url, api_key=dep.api_key,
                       model="zai-org/glm-4-9b-chat-hf", concurrency=10, prompt_length="1k")
print(result.avg_ttft_ms, result.tpm, result.network_latency_ms)
```

## Models

Models live in [`sdk/gpuroutertest/registry.json`](sdk/gpuroutertest/registry.json). Each entry is **keyed by its model id** and holds the per-size config (instance type, disk, server flags). That same id is what you pass to `deploy`, what `list-models` prints, and what you send as `"model"` when calling the endpoint. Add a model by adding an entry — no code changes required.

## Notes & limitations

- The security group opens **port 8000**; it defaults to the whole internet and is protected only by the API key. Use `--cidr` in real use.
- `--spot` recovers automatically but **not instantly**: there's a downtime gap while the replacement reloads the model (see [Spot instances with automatic failover](#spot-instances-with-automatic-failover)).
- Traffic is plain **HTTP** (no TLS). Put it behind a proxy or load balancer for anything beyond dev.
- Instance boot logs are available via `gpuroutertest logs` (EC2 console output). The server's own container logs require SSH/SSM, which are intentionally not provisioned.

## License

MIT — see [LICENSE](LICENSE).
