Metadata-Version: 2.4
Name: frugal-llm
Version: 0.2.0
Summary: Benchmark latency, throughput, cost and quantisation quality loss of any OpenAI-compatible LLM endpoint.
Keywords: llm,benchmark,inference,latency,throughput,quantization,vllm,ollama,llama.cpp,openai
Author: Leke-Oduoye Erioluwa
Author-email: Leke-Oduoye Erioluwa <lekeoduoyee@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Benchmark
Requires-Dist: httpx>=0.28.1
Requires-Dist: rich>=15.0.0
Requires-Dist: typer>=0.27.3
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/Lekeboi/Frugal_LLM
Project-URL: Repository, https://github.com/Lekeboi/Frugal_LLM
Project-URL: Issues, https://github.com/Lekeboi/Frugal_LLM/issues
Description-Content-Type: text/markdown

# frugal-llm

How fast, and how expensive, is it to serve *your* model on *your* hardware, and what
do you lose by serving a cheaper version of it?

`frugal-llm` points at any OpenAI-compatible endpoint — vLLM, TGI, SGLang,
llama.cpp, Ollama or a commercial API — and reports:

- **`bench`**: latency percentiles, throughput and cost per million output tokens at each
  concurrency level you ask for;
- **`quality`** and **`compare`**: how much a quantised model's answers change against
  full precision, on shipped task sets, with honest statistics;
- **`cost`**: the traffic above which self-hosting beats a commercial API's per-token price.

It speaks only the OpenAI-compatible `/v1` API, so there is nothing engine-specific to
install or configure.

## Quickstart

Requires [uv](https://docs.astral.sh/uv/). Against a local Ollama:

```bash
ollama pull qwen2.5:0.5b
uvx frugal-llm bench --url http://localhost:11434/v1 --model qwen2.5:0.5b
```

With the defaults (50 measured requests × 3 repeats, up to 128 output tokens) that
takes about 12 minutes on a laptop CPU. For a first look in about 2 minutes:

```bash
uvx frugal-llm bench --url http://localhost:11434/v1 --model qwen2.5:0.5b -n 10 --max-tokens 64
```

A concurrency sweep with a cost model, against vLLM on a GPU you rent for £0.52/hour:

```bash
uvx frugal-llm bench --url http://localhost:8000/v1 --model mistral-7b-awq \
  -c 1,4,16,64 -n 200 --cost-per-hour 0.52 --label vllm-awq-L4
```

Results land in `./results/` as JSON (every individual sample) and Markdown (the
table below, ready to paste into an issue or a chat thread).

## Example output

The quickstart above, run unmodified on a laptop (Intel Core 7 150U, 16 GB RAM, no
GPU, on mains power) against Ollama 0.40.4. This is the Markdown file it wrote, as is:

```
## frugal-llm — qwen2.5:0.5b @ http://localhost:11434/v1

Config: max_tokens=128, prompt_tokens≈200, requests=50, warmup=5, repeats=3, label=ollama-quickstart

| Conc | TTFT p50 | TTFT p95 | ITL p50 |      Thrpt tok/s | Err |
|-----:|---------:|---------:|--------:|-----------------:|----:|
|    1 |   2222ms |   2408ms |    27ms | 18.1 (18.0-18.4) |  0% |

- Latency percentiles pool the 3 repeats at each level (never across levels). Throughput and cost show the median run (min-max).
- Median output length was below --max-tokens 128 (c=1: 78): the model often stopped early. Throughput and cost reflect these shorter generations.
- Prompt length as counted by the server: median 233 tokens (requested ≈200, including the chat template).
- HTTP/1.1, one connection per concurrent request. Measures speed and cost only; output quality is not measured.

Generated by frugal-llm 0.1.0 on 2026-10-14T17:35:23Z
```

Read it as: about 2.2 s before the first token appears (prefill of ~233 prompt tokens
on a laptop CPU), then a token every ~27 ms, which works out at ~18 tokens/s including
that wait. No cost column because no `--cost-per-hour` was given.

### Same model, two servers

The point of measuring rather than guessing. The same model file (Qwen2.5-0.5B-Instruct,
Q4_K_M) on the same laptop, swept across concurrency with
`-c 1,2,4 -n 20 --max-tokens 64`:

**Ollama 0.40.4** (default settings)

| Conc | TTFT p50 | TTFT p95 | ITL p50 |      Thrpt tok/s | Err |
|-----:|---------:|---------:|--------:|-----------------:|----:|
|    1 |   2196ms |   2280ms |    26ms | 15.5 (14.8-15.8) |  0% |
|    2 |   5604ms |   6100ms |    26ms | 15.6 (14.9-15.7) |  0% |
|    4 |  11784ms |  13001ms |    26ms | 14.6 (14.4-15.7) |  0% |

**llama.cpp `llama-server` b11604** (`-np 4`, four parallel slots)

| Conc | TTFT p50 | TTFT p95 | ITL p50 |      Thrpt tok/s | Err |
|-----:|---------:|---------:|--------:|-----------------:|----:|
|    1 |   1560ms |   1692ms |    30ms | 14.0 (13.3-15.2) |  0% |
|    2 |   1626ms |   3130ms |    64ms | 16.7 (15.5-17.7) |  0% |
|    4 |   1697ms |   5988ms |   135ms | 19.9 (16.6-20.3) |  0% |

Ollama, with its defaults on this machine, handles one request at a time: throughput
stays flat at ~15 tok/s and every extra concurrent user waits in a queue, so median
TTFT grows from 2.2 s to 11.8 s. llama.cpp with four slots processes requests
together: median TTFT stays ~1.6 s and throughput rises to ~20 tok/s, paid for with
slower token streaming (ITL 30 → 135 ms) as the CPU is shared. Which is better
depends on whether your users feel the wait for the first token or the reading speed
— which is exactly the trade-off this tool exists to show. These are laptop numbers
for a tiny model; they say nothing about your GPU.

## Options

| Flag | Default | Meaning |
|---|---|---|
| `--url` | required | Base URL including `/v1`, e.g. `http://localhost:11434/v1` |
| `--model` | required | Model identifier as the server knows it |
| `--api-key` | none | Bearer token; also read from `FRUGAL_LLM_API_KEY`. Never written to results |
| `-c, --concurrency` | `1` | Comma-separated levels to sweep, e.g. `1,4,16,64`. Run one after another |
| `-n, --requests` | `50` | Measured requests per run. Use several times the concurrency |
| `--max-tokens` | `128` | Hard cap on output length |
| `--prompt-tokens` | `200` | Approximate length of generated prompts |
| `--prompts` | none | JSONL of `{"prompt": "..."}`, used in rotation instead of generated prompts |
| `--warmup` | `5` | Minimum warm-up requests per level; at least one per concurrent worker |
| `--repeats` | `3` | Independent runs per level. Single runs report noise as signal |
| `--cost-per-hour` | none | Hourly cost of the hardware; enables the cost column |
| `--currency` | `GBP` | Label only |
| `--timeout` | `120` | Seconds, applied per streamed chunk and as a total per-request deadline |
| `--out` | `./results` | Output directory |
| `--format` | `both` | `md`, `json` or `both` |
| `--label` | none | Free-text tag stored with the results and used in file names |

Re-aggregate a saved run without re-running it (for example after an upgrade changes
how reports are rendered):

```bash
uvx frugal-llm report results/20261014T120000Z-vllm-awq-L4.json
```

## What the numbers mean

| Metric | Definition |
|---|---|
| **TTFT** | Time to first token: request sent → first chunk with non-empty content (prefill). p50/p95/p99 |
| **ITL** | Inter-token latency: mean gap between content chunks within a request, excluding the first (decode). p50/p95 across requests |
| **E2E** | Request sent → final chunk. p50/p95/p99 in the JSON |
| **Throughput** | Output tokens of completed requests ÷ wall-clock measurement window |
| **Cost / 1M tokens** | `(cost_per_hour / 3600) / tokens_per_sec × 1,000,000` |
| **Err** | Failed or timed-out requests ÷ attempted. The report names each failure kind |

Latency is only ever reported as percentiles, never a mean: the tail is the point.
Each concurrency level is its own distribution; samples are never pooled across
levels. Within a level, latency samples from all repeats are pooled, while
throughput and cost show the median run with the min–max range across repeats.

## Measuring quality loss

> New in 0.2.0, and so far exercised only on a small (0.5B) model on development subsets.
> The result format is provisional; see [Results format](#results-format).

Speed is only half the question. Before serving a cheaper (quantised) model, you want to
know what it costs in output quality. frugal-llm answers that relative to the full-precision
(FP16) model, on the same hardware and engine:

```bash
# FP16 twice (the second run measures run-to-run noise), then the cheaper format
frugal-llm quality --url http://localhost:8080/v1 --model my-model --task mmlu --label fp16-a
frugal-llm quality --url http://localhost:8080/v1 --model my-model --task mmlu --label fp16-b
frugal-llm quality --url http://localhost:8080/v1 --model my-model --task mmlu --label q4_k_m
frugal-llm compare results/*-fp16-a.json results/*-q4_k_m.json --noise-floor results/*-fp16-b.json
```

- **Task sets ship with the tool**, so everyone runs identical items: all 1,319 GSM8K test
  questions (arithmetic reasoning) and a fixed 1,000-question MMLU subset (knowledge, all 57
  subjects). Both are MIT-licensed; see `src/frugal_llm/tasks/`.
- **Greedy decoding, fixed prompts, fixed answer-extraction rules**, written down before any
  code existed. Development runs on real models exposed problems (answers read from the wrong
  place, outputs cut off too early), so some rules were revised before any result was
  published; every change, and the evidence for it, is logged in
  [docs/quality-method.md](docs/quality-method.md).
- **`compare` reports both what changed and whether it matters:** accuracy with confidence
  intervals, the paired difference against FP16 with an exact significance test, how often
  the final answers agree, and the FP16-versus-FP16 noise floor. It also states the smallest
  difference the comparison could have detected, so "no difference detected" is never
  dressed up as "no quality loss".
- **Long runs survive interruption:** every answer is saved as it arrives, and `--resume`
  continues where a run stopped. Rate-limited requests (HTTP 429) are retried, honouring
  `Retry-After`, because a quality score needs every question answered.

## Is self-hosting cheaper than an API?

> New in 0.2.0. The result format is provisional; see [Results format](#results-format).

A rented GPU costs the same per hour busy or idle, so "N× cheaper" claims are really claims
about an assumed load. `frugal-llm cost` answers the real question instead: **above what
sustained traffic does self-hosting beat paying per token?** It works from saved `bench`
results, so prices can be changed without re-running anything:

```bash
frugal-llm cost results/<bench-run>.json \
  --hourly-rate 0.44 --rate-source "provider, GPU model, on-demand" --rate-date 2026-10-15 \
  --api "name=some-model,in=0.15,out=0.60,source=https://provider.example/pricing,date=2026-10-15"
```

For each concurrency level it reports the measured capacity, the cost per 1,000 requests at
full use, and against each API price the **break-even traffic** (requests per day) and
**break-even utilisation**: the share of capacity you must keep busy, around the clock, for
self-hosting to cost no more. If even 100% use costs more than the API, it says so.
Requests are compared whole, at the measured tokens in and out, because a GPU-hour has no
separate input and output price. Rates and prices must carry a source and a date, and are
never built into the tool. Method and caveats: [docs/cost-method.md](docs/cost-method.md).

## What this measures and what it doesn't

**`bench` measures** speed and cost of generation as seen by a client: how long until the
first token, how fast tokens stream, how many tokens per second the endpoint sustains
at a given concurrency, and what that costs at the hourly rate you supply.

**`quality` measures** how much a cheaper format changes a model's answers on two checkable
task sets, relative to FP16 (see above).

**Neither measures:**

- **General output quality.** `bench` measures no quality at all: a fast, cheap model that
  answers badly will look good there. `quality` covers arithmetic reasoning and recalled
  knowledge only, not instruction following, long context, safety or writing. Both task sets
  are public and may be in a model's training data, so absolute scores are not evidence of
  general ability; only the comparison against FP16 is meaningful.
- **Your real traffic.** Load is closed-loop (N requests always in flight), with fixed
  output length and synthetic or supplied prompts. Bursty traffic, long contexts or
  variable output lengths will behave differently.
- **Network you don't benchmark through.** Numbers include the path between the
  client and the server. Run the client next to the server to measure the server.
- **Exact token counts when the server hides them.** Output tokens come from the
  server's `usage` report; if it doesn't send one, frugal-llm counts content chunks,
  which undercounts servers that pack several tokens per chunk. The report says
  which method was used.
- **More than the machine's state at the time.** Power mode, temperature and other
  load all move the numbers. Repeats show the spread within one session, not between
  sessions; benchmark on mains power with nothing else running.
- **Cost beyond the hourly rate.** Idle time, egress, storage and engineering time
  are not included.

## Why the numbers can be trusted

- **Every rule is tested against a mock server** that emits chunks at precisely
  controlled times, so the suite proves the tool measures the server and not itself:
  TTFT against a known prefill delay, genuinely concurrent requests (past 100),
  warm-up excluded, levels never pooled, percentile and cost arithmetic by hand.
- **Warm-up does double duty.** Warm-up requests warm the server *and* open every
  worker's connection, so measured requests don't pay TCP/TLS handshakes and the
  TTFT tail isn't handshake noise.
- **Every concurrent worker owns exactly one connection.** A shared connection pool
  either caps you at httpx's default 100 connections (silently serialising the load)
  or, sized up, burns client CPU matching requests to connections — at 150 concurrent
  requests that turned a 1 s server delay into a 6 s round.
- **The client checks itself.** It samples its own event-loop lag during every run
  and warns in the report when the client, not the server, may have shaped the
  latencies (for example on an overloaded laptop at high concurrency).
- **Prompts are never reused**, within a run or across runs (each run gets a random
  seed, recorded in the results). Servers keep prompt caches between requests;
  re-sending last run's prompts measured a 28 ms TTFT where a new prompt took
  ~1000 ms. The server's own cache report is recorded too, and the report warns when
  cached prompts are flattering TTFT (for example from a repeating `--prompts` file).
- **A machine that sleeps mid-run is caught.** The tool keeps the OS awake while it
  runs, and any run in which the client froze for over a second (a laptop entering
  standby, say) is excluded from the table and named in the report.
- **Failures fail loudly.** A preflight request catches a wrong URL, model or key
  before anything is measured; an unreachable endpoint aborts without writing
  partial results. Timeouts are per phase and named (`read_timeout`,
  `total_timeout`, ...), so a stalled server is distinguishable from a slow one.
- **HTTP/1.1 only.** HTTP/2 multiplexing would change what "concurrency" means.
  Recorded in every result file.

## Results format

Every run writes JSON with every individual sample, so results can be re-analysed later
(`report`, `compare` and `cost` all work from saved files). Each file records its
`schema_version` (`0.1` for `bench`, `quality-0.1` for `quality`) and the full environment:
the client machine, what the server reported about itself, and your `--meta key=value` notes.

**The format is provisional** (`"schema_status": "provisional"` in every file) until the
first full study on real hardware is complete. Quality and cost measurement have so far run
only against a small model, and the real study may show that something must be recorded
differently. Until then, results from different versions may not be directly comparable,
and there is no process yet for contributing results; both will come once the format is
stable.

## Development

```bash
uv sync
uv run pytest
uv run ruff check . && uv run ruff format --check .
```

The tests need no GPU and no model: they run against `tests/mock_server.py`, which
you can also start by hand (`uv run python tests/mock_server.py`) and point the CLI at.

## License

Apache-2.0. See [LICENSE](LICENSE).
