Metadata-Version: 2.4
Name: frugal-llm
Version: 0.1.0
Summary: Benchmark latency, throughput and cost of any OpenAI-compatible LLM endpoint.
Keywords: llm,benchmark,inference,latency,throughput,vllm,ollama,openai
Author: Leke-Oduoye Erioluwa
Author-email: Leke-Oduoye Erioluwa <lekeoduoyee@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Benchmark
Requires-Dist: httpx>=0.28.1
Requires-Dist: rich>=15.0.0
Requires-Dist: typer>=0.27.3
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/Lekeboi/Frugal_LLM
Project-URL: Repository, https://github.com/Lekeboi/Frugal_LLM
Project-URL: Issues, https://github.com/Lekeboi/Frugal_LLM/issues
Description-Content-Type: text/markdown

# frugal-llm

How fast, and how expensive, is it to serve *your* model on *your* hardware?

`frugal-llm` points at any OpenAI-compatible endpoint — vLLM, TGI, SGLang,
llama.cpp, Ollama or a commercial API — and reports latency percentiles,
throughput and cost per million output tokens at each concurrency level you ask for.
It speaks only `POST /v1/chat/completions` with streaming, so there is nothing
engine-specific to install or configure.

## Quickstart

Requires [uv](https://docs.astral.sh/uv/). Against a local Ollama:

```bash
ollama pull qwen2.5:0.5b
uvx frugal-llm bench --url http://localhost:11434/v1 --model qwen2.5:0.5b
```

With the defaults (50 measured requests × 3 repeats, up to 128 output tokens) that
takes about 12 minutes on a laptop CPU. For a first look in about 2 minutes:

```bash
uvx frugal-llm bench --url http://localhost:11434/v1 --model qwen2.5:0.5b -n 10 --max-tokens 64
```

A concurrency sweep with a cost model, against vLLM on a GPU you rent for £0.52/hour:

```bash
uvx frugal-llm bench --url http://localhost:8000/v1 --model mistral-7b-awq \
  -c 1,4,16,64 -n 200 --cost-per-hour 0.52 --label vllm-awq-L4
```

Results land in `./results/` as JSON (every individual sample) and Markdown (the
table below, ready to paste into an issue or a chat thread).

## Example output

The quickstart above, run unmodified on a laptop (Intel Core 7 150U, 16 GB RAM, no
GPU, on mains power) against Ollama 0.40.4. This is the Markdown file it wrote, as is:

```
## frugal-llm — qwen2.5:0.5b @ http://localhost:11434/v1

Config: max_tokens=128, prompt_tokens≈200, requests=50, warmup=5, repeats=3, label=ollama-quickstart

| Conc | TTFT p50 | TTFT p95 | ITL p50 |      Thrpt tok/s | Err |
|-----:|---------:|---------:|--------:|-----------------:|----:|
|    1 |   2222ms |   2408ms |    27ms | 18.1 (18.0-18.4) |  0% |

- Latency percentiles pool the 3 repeats at each level (never across levels). Throughput and cost show the median run (min-max).
- Median output length was below --max-tokens 128 (c=1: 78): the model often stopped early. Throughput and cost reflect these shorter generations.
- Prompt length as counted by the server: median 233 tokens (requested ≈200, including the chat template).
- HTTP/1.1, one connection per concurrent request. Measures speed and cost only; output quality is not measured.

Generated by frugal-llm 0.1.0 on 2026-10-14T17:35:23Z
```

Read it as: about 2.2 s before the first token appears (prefill of ~233 prompt tokens
on a laptop CPU), then a token every ~27 ms, which works out at ~18 tokens/s including
that wait. No cost column because no `--cost-per-hour` was given.

### Same model, two servers

The point of measuring rather than guessing. The same model file (Qwen2.5-0.5B-Instruct,
Q4_K_M) on the same laptop, swept across concurrency with
`-c 1,2,4 -n 20 --max-tokens 64`:

**Ollama 0.40.4** (default settings)

| Conc | TTFT p50 | TTFT p95 | ITL p50 |      Thrpt tok/s | Err |
|-----:|---------:|---------:|--------:|-----------------:|----:|
|    1 |   2196ms |   2280ms |    26ms | 15.5 (14.8-15.8) |  0% |
|    2 |   5604ms |   6100ms |    26ms | 15.6 (14.9-15.7) |  0% |
|    4 |  11784ms |  13001ms |    26ms | 14.6 (14.4-15.7) |  0% |

**llama.cpp `llama-server` b11604** (`-np 4`, four parallel slots)

| Conc | TTFT p50 | TTFT p95 | ITL p50 |      Thrpt tok/s | Err |
|-----:|---------:|---------:|--------:|-----------------:|----:|
|    1 |   1560ms |   1692ms |    30ms | 14.0 (13.3-15.2) |  0% |
|    2 |   1626ms |   3130ms |    64ms | 16.7 (15.5-17.7) |  0% |
|    4 |   1697ms |   5988ms |   135ms | 19.9 (16.6-20.3) |  0% |

Ollama, with its defaults on this machine, handles one request at a time: throughput
stays flat at ~15 tok/s and every extra concurrent user waits in a queue, so median
TTFT grows from 2.2 s to 11.8 s. llama.cpp with four slots processes requests
together: median TTFT stays ~1.6 s and throughput rises to ~20 tok/s, paid for with
slower token streaming (ITL 30 → 135 ms) as the CPU is shared. Which is better
depends on whether your users feel the wait for the first token or the reading speed
— which is exactly the trade-off this tool exists to show. These are laptop numbers
for a tiny model; they say nothing about your GPU.

## Options

| Flag | Default | Meaning |
|---|---|---|
| `--url` | required | Base URL including `/v1`, e.g. `http://localhost:11434/v1` |
| `--model` | required | Model identifier as the server knows it |
| `--api-key` | none | Bearer token; also read from `FRUGAL_LLM_API_KEY`. Never written to results |
| `-c, --concurrency` | `1` | Comma-separated levels to sweep, e.g. `1,4,16,64`. Run one after another |
| `-n, --requests` | `50` | Measured requests per run. Use several times the concurrency |
| `--max-tokens` | `128` | Hard cap on output length |
| `--prompt-tokens` | `200` | Approximate length of generated prompts |
| `--prompts` | none | JSONL of `{"prompt": "..."}`, used in rotation instead of generated prompts |
| `--warmup` | `5` | Minimum warm-up requests per level; at least one per concurrent worker |
| `--repeats` | `3` | Independent runs per level. Single runs report noise as signal |
| `--cost-per-hour` | none | Hourly cost of the hardware; enables the cost column |
| `--currency` | `GBP` | Label only |
| `--timeout` | `120` | Seconds, applied per streamed chunk and as a total per-request deadline |
| `--out` | `./results` | Output directory |
| `--format` | `both` | `md`, `json` or `both` |
| `--label` | none | Free-text tag stored with the results and used in file names |

Re-aggregate a saved run without re-running it (for example after an upgrade changes
how reports are rendered):

```bash
uvx frugal-llm report results/20261014T120000Z-vllm-awq-L4.json
```

## What the numbers mean

| Metric | Definition |
|---|---|
| **TTFT** | Time to first token: request sent → first chunk with non-empty content (prefill). p50/p95/p99 |
| **ITL** | Inter-token latency: mean gap between content chunks within a request, excluding the first (decode). p50/p95 across requests |
| **E2E** | Request sent → final chunk. p50/p95/p99 in the JSON |
| **Throughput** | Output tokens of completed requests ÷ wall-clock measurement window |
| **Cost / 1M tokens** | `(cost_per_hour / 3600) / tokens_per_sec × 1,000,000` |
| **Err** | Failed or timed-out requests ÷ attempted. The report names each failure kind |

Latency is only ever reported as percentiles, never a mean: the tail is the point.
Each concurrency level is its own distribution; samples are never pooled across
levels. Within a level, latency samples from all repeats are pooled, while
throughput and cost show the median run with the min–max range across repeats.

## What this measures and what it doesn't

**It measures** speed and cost of generation as seen by a client: how long until the
first token, how fast tokens stream, how many tokens per second the endpoint sustains
at a given concurrency, and what that costs at the hourly rate you supply.

**It does not measure:**

- **Output quality or accuracy.** Not at all, in v0.1. A fast, cheap model that
  answers badly will look good here. Evaluate quality separately before deciding.
- **Your real traffic.** Load is closed-loop (N requests always in flight), with fixed
  output length and synthetic or supplied prompts. Bursty traffic, long contexts or
  variable output lengths will behave differently.
- **Network you don't benchmark through.** Numbers include the path between the
  client and the server. Run the client next to the server to measure the server.
- **Exact token counts when the server hides them.** Output tokens come from the
  server's `usage` report; if it doesn't send one, frugal-llm counts content chunks,
  which undercounts servers that pack several tokens per chunk. The report says
  which method was used.
- **More than the machine's state at the time.** Power mode, temperature and other
  load all move the numbers. Repeats show the spread within one session, not between
  sessions; benchmark on mains power with nothing else running.
- **Cost beyond the hourly rate.** Idle time, egress, storage and engineering time
  are not included.

## Why the numbers can be trusted

- **Every rule is tested against a mock server** that emits chunks at precisely
  controlled times, so the suite proves the tool measures the server and not itself:
  TTFT against a known prefill delay, genuinely concurrent requests (past 100),
  warm-up excluded, levels never pooled, percentile and cost arithmetic by hand.
- **Warm-up does double duty.** Warm-up requests warm the server *and* open every
  worker's connection, so measured requests don't pay TCP/TLS handshakes and the
  TTFT tail isn't handshake noise.
- **Every concurrent worker owns exactly one connection.** A shared connection pool
  either caps you at httpx's default 100 connections (silently serialising the load)
  or, sized up, burns client CPU matching requests to connections — at 150 concurrent
  requests that turned a 1 s server delay into a 6 s round.
- **The client checks itself.** It samples its own event-loop lag during every run
  and warns in the report when the client, not the server, may have shaped the
  latencies (for example on an overloaded laptop at high concurrency).
- **Prompts are never reused**, within a run or across runs (each run gets a random
  seed, recorded in the results). Servers keep prompt caches between requests;
  re-sending last run's prompts measured a 28 ms TTFT where a new prompt took
  ~1000 ms. The server's own cache report is recorded too, and the report warns when
  cached prompts are flattering TTFT (for example from a repeating `--prompts` file).
- **A machine that sleeps mid-run is caught.** The tool keeps the OS awake while it
  runs, and any run in which the client froze for over a second (a laptop entering
  standby, say) is excluded from the table and named in the report.
- **Failures fail loudly.** A preflight request catches a wrong URL, model or key
  before anything is measured; an unreachable endpoint aborts without writing
  partial results. Timeouts are per phase and named (`read_timeout`,
  `total_timeout`, ...), so a stalled server is distinguishable from a slow one.
- **HTTP/1.1 only.** HTTP/2 multiplexing would change what "concurrency" means.
  Recorded in every result file.

## Development

```bash
uv sync
uv run pytest
uv run ruff check . && uv run ruff format --check .
```

The tests need no GPU and no model: they run against `tests/mock_server.py`, which
you can also start by hand (`uv run python tests/mock_server.py`) and point the CLI at.

## License

Apache-2.0. See [LICENSE](LICENSE).
