Metadata-Version: 2.5
Name: bench-my-llm
Version: 0.5.0
Summary: Dead-simple LLM benchmarking CLI. Measure TTFT, TPS, latency, cost, and quality for any OpenAI-compatible API.
Project-URL: Homepage, https://github.com/manasvardhan/bench-my-llm
Project-URL: Repository, https://github.com/manasvardhan/bench-my-llm
Project-URL: Issues, https://github.com/manasvardhan/bench-my-llm/issues
Author-email: Manas Vardhan <manasvardhan@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: benchmark,cli,latency,llm,openai
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: numpy>=1.24
Requires-Dist: openai>=1.0
Requires-Dist: rich>=13.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Provides-Extra: yaml
Requires-Dist: pyyaml>=6.0; extra == 'yaml'
Description-Content-Type: text/markdown


# 🏎️ bench-my-llm

> **New here?** Start with the [Getting Started Guide](GETTING_STARTED.md).

[![PyPI version](https://img.shields.io/pypi/v/bench-my-llm?color=orange)](https://pypi.org/project/bench-my-llm/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![CI](https://github.com/manasvardhan/bench-my-llm/actions/workflows/ci.yml/badge.svg)](https://github.com/manasvardhan/bench-my-llm/actions)

**Stop guessing which model is faster. Measure it.**

Point `bench-my-llm` at any OpenAI-compatible API and get latency, throughput, cost, and quality metrics in seconds. Compare models side by side. Get a beautiful terminal report. Ship with confidence.

## ✨ Features

- 🔥 **TTFT Measurement** - Time to first token via streaming
- ⚡ **Tokens per Second** - Real throughput numbers
- 📊 **p50 / p95 / p99 Latencies** - Production-grade percentiles
- 💰 **Cost Estimation** - Know what you're spending
- 🎯 **Quality Scoring** - Compare responses against reference answers
- 🏁 **Model Comparison** - Side-by-side with winner highlights
- 📦 **Built-in Prompt Suites** - Reasoning, coding, creative, factual
- 🔌 **Any OpenAI-compatible API** - OpenAI, Anthropic, Ollama, vLLM, Together, and more
- 💾 **Export to JSON, CSV, Markdown, HTML** - Pipe into CI, dashboards, or share a report
- 📈 **Historical Trends** - Track quality, speed, and cost per model over time with sparklines
- 🦙 **Ollama Auto-Detection** - Find local models and benchmark them with a single flag
- 🧮 **Custom Prompt Suites** - Load your own domain-specific prompts from JSON or YAML files
- 📦 **Suite Packs** - Install, update, and publish shareable prompt suites by name from a simple JSON registry
- 🧾 **Run Metadata and Notes** - Every saved result records the endpoint, temperature, suite, tool version, and an optional note so old numbers stay interpretable
- 🔬 **Metadata Filtering and Diffing** - Filter saved runs to like-for-like configs and diff any two to see which metadata changed and how the numbers moved
- 🏷️ **Metadata Grouping** - Group saved runs by endpoint, temperature, model, or tool version and compare aggregate metrics per group

## 🚀 Quick Start

```bash
pip install bench-my-llm
```

### Single Model Benchmark

```bash
bench-my-llm run --model gpt-4o --suite reasoning
```

```
┌──────────────────────────────────────────────────────────┐
│  🏎️  Benchmark Report                                    │
│  bench-my-llm results for gpt-4o                         │
│  Suite: reasoning | Prompts: 5 | Cost: $0.0043           │
└──────────────────────────────────────────────────────────┘

          Latency Summary
┌────────┬────────────┬────────────────────┐
│ Metric │ TTFT (ms)  │ Total Latency (ms) │
├────────┼────────────┼────────────────────┤
│ p50    │ 234.1      │ 1,523.4            │
│ p95    │ 312.7      │ 2,187.9            │
│ p99    │ 348.2      │ 2,401.3            │
│ Mean   │ 251.3      │ 1,687.2            │
└────────┴────────────┴────────────────────┘

       Throughput & Quality
┌───────────────────┬─────────────┐
│ Metric            │ Value       │
├───────────────────┼─────────────┤
│ Mean TPS          │ 67.3 tok/s  │
│ Median TPS        │ 64.8 tok/s  │
│ Quality Score     │ 82%         │
│ Estimated Cost    │ $0.0043     │
└───────────────────┴─────────────┘
```

### Model Comparison

```bash
bench-my-llm compare gpt-4o gpt-4o-mini --suite reasoning
```

```
┌──────────────────────────────────────────────────────────┐
│  🏁 Model Comparison                                     │
│  gpt-4o vs gpt-4o-mini                                   │
└──────────────────────────────────────────────────────────┘

              Head-to-Head
┌────────────────────────┬─────────┬─────────────┐
│ Metric                 │ gpt-4o  │ gpt-4o-mini │
├────────────────────────┼─────────┼─────────────┤
│ TTFT p50 (ms)          │ 234.1   │ 142.3  🏆   │
│ TTFT p95 (ms)          │ 312.7   │ 198.4  🏆   │
│ Total Latency p50 (ms) │ 1523.4  │ 876.2  🏆   │
│ Mean TPS               │ 67.3 🏆 │ 54.1        │
│ Cost (USD)             │ $0.0043 │ $0.0008 🏆  │
│ Quality Score          │ 0.82 🏆 │ 0.71        │
└────────────────────────┴─────────┴─────────────┘

🏆 Winner: gpt-4o-mini (4/6 metrics)
```

## 📖 Usage

### Custom Prompt Suites

Benchmark with your own domain-specific prompts by passing a JSON or
YAML file path to `--suite`:

```bash
bench-my-llm run --model gpt-4o --suite my_suite.json
bench-my-llm compare gpt-4o gpt-4o-mini --suite ./suites/support.yaml
```

Suite file format (see `examples/custom_suite.json` for a full example):

```json
{
  "name": "support-bot",
  "description": "Customer support prompts",
  "prompts": [
    {
      "text": "Explain our 30-day return policy in two sentences.",
      "category": "support",
      "reference": "Items can be returned within 30 days for a full refund.",
      "max_tokens": 256
    }
  ]
}
```

Only `text` is required per prompt. `category` defaults to `custom`,
`reference` to empty (skips quality scoring), and `max_tokens` to 512.
Files are validated with friendly error messages, including typo
detection for unknown keys. YAML files need the optional extra:
`pip install 'bench-my-llm[yaml]'`.

### Suite Packs

Share suites with your team and install them by name. A registry is a
JSON document (local file or HTTP URL) mapping pack names to suite
files:

```json
{
  "packs": {
    "support-basics": {
      "version": "1.0.0",
      "description": "Customer support prompts",
      "url": "support_basics.json"
    }
  }
}
```

Pack URLs can be absolute (http, https, or a filesystem path) or
relative to the registry location, so a registry can live next to its
suite files in a repo or on any static file host.

```bash
# Install a pack (validated before it lands on disk)
bench-my-llm suite install support-basics --registry https://example.com/registry.json

# Or set a default registry once
export BENCH_MY_LLM_REGISTRY=https://example.com/registry.json
bench-my-llm suite install support-basics

# See what is installed, then benchmark with a pack by name
bench-my-llm suite list
bench-my-llm run --model gpt-4o --suite support-basics

# Remove a pack
bench-my-llm suite remove support-basics
```

Keep installed packs current with `suite update`. It re-fetches each
pack's registry from the source recorded at install time, reinstalls
packs whose registry version changed, and reports the rest as up to
date.

```bash
# Check and update every installed pack
bench-my-llm suite update

# Update specific packs only
bench-my-llm suite update support-basics other-pack

# Reinstall even when the version is unchanged
bench-my-llm suite update --force

# Point at a different registry, or get machine-readable results
bench-my-llm suite update support-basics -r new-registry.json
bench-my-llm suite update --json-output
```

Update failures (a registry that moved, a pack pulled from its
registry) are reported per pack without aborting the rest, and the
command exits nonzero if anything failed.

Installed packs live in `~/.bench-my-llm/suites` (override the base
directory with `BENCH_MY_LLM_HOME`). Try it locally with the bundled
example: `bench-my-llm suite install example-basics -r examples/registry.json`.

Publishing your own packs is the reverse direction: validate a suite
file and add it to a registry in one step.

```bash
# Validate a suite and print its registry entry as JSON
bench-my-llm suite publish my_suite.json

# Write the entry into a registry file (created if missing)
bench-my-llm suite publish my_suite.json -r registry.json -n my-pack -v 1.0.0

# Bump an existing pack
bench-my-llm suite publish my_suite.json -r registry.json -n my-pack -v 1.1.0 --force
```

The suite is validated before anything is written, existing registry
entries are preserved, and the URL is stored relative to the registry
when the suite file lives in the same directory tree, so the pair
stays portable. Host the resulting registry anywhere static files can
be served and teammates can `suite install` from it.

### Prompt Suites

| Suite | Description | Prompts |
|-------|-------------|---------|
| `reasoning` | Logic, math, step-by-step | 5 |
| `coding` | Code generation and explanation | 5 |
| `creative` | Writing, storytelling, metaphors | 5 |
| `factual` | Knowledge recall, definitions | 5 |
| `all` | Everything combined | 20 |

### Export Results

```bash
bench-my-llm run --model gpt-4o --suite all --output results.json
bench-my-llm report results.json

# Convert saved results to other formats
bench-my-llm export results.json --format markdown
bench-my-llm export results.json --format csv -o results.csv
```

### Run Metadata and Notes

Every benchmark run captures its environment automatically: endpoint,
temperature, suite (name and prompt count), tool version, and a UTC
timestamp. Add a free-form note with `--note` to remember why you ran it:

```bash
bench-my-llm run --model gpt-4o --suite reasoning \
  --note "baseline before prompt rewrite" --output baseline.json
```

The terminal report prints a Run Metadata panel, and the saved JSON gains
a `metadata` object:

```json
{
  "metadata": {
    "notes": "baseline before prompt rewrite",
    "endpoint": "https://api.openai.com/v1",
    "temperature": 0.0,
    "suite_version": "reasoning (5 prompts)",
    "tool_version": "0.3.0",
    "captured_at": "2026-08-29T05:00:00+00:00"
  }
}
```

Older result files without a metadata block still load fine. Python API:
`RunMetadata` and the `metadata` field on `BenchmarkRun`.

### HTML Reports

Turn any saved results file into a shareable, self-contained HTML report
with summary cards, latency tables, per-prompt quality bars, and a model
comparison section for multi-model files. No JavaScript, no external
assets, safe to attach to a PR or email:

```bash
bench-my-llm export results.json --format html -o report.html
bench-my-llm export comparison.json --format html --title "GPT vs Claude" -o report.html
open report.html
```

### Cost-Adjusted Leaderboard

Rank saved runs by value, not just raw speed. The composite score weighs
quality (50%), cost (30%, lower is better), and throughput (20%):

```bash
bench-my-llm leaderboard results/*.json
bench-my-llm leaderboard a.json b.json --sort quality-per-dollar
bench-my-llm leaderboard results/*.json --json-output
```

Sort options: `value` (default), `quality`, `cost`, `speed`, `quality-per-dollar`.

### Historical Trends

Save each benchmark run to a results directory, then track how every
model's numbers move over time. Runs are grouped per model and ordered
chronologically, with a terminal sparkline and the change from the first
run to the latest:

```bash
bench-my-llm run -m gpt-4o -s reasoning -o results/gpt4o-$(date +%F).json
bench-my-llm trends ./results/
bench-my-llm trends ./results/ --metric latency
bench-my-llm trends ./results/ -m gpt-4o --metric cost --json-output
```

```
Model     Runs  Trend      First   Latest  Change
gpt-4o       5  ▂▃▅▆█        61%      74%  ▲ +21.3%
llama3       4  ▅▄▃▂         55%      48%  ▼ -12.7%
```

Metrics: `quality` (default), `tps`, `ttft`, `latency`, `cost`. For
`ttft`, `latency`, and `cost`, lower is better and the arrows account
for that.

### Filter and Diff Runs by Metadata

Every saved run carries a metadata block (endpoint, temperature, tool
version, suite). Use it to narrow a results directory down to
like-for-like configurations, then diff any two runs to see which
metadata changed and how the numbers moved:

```bash
bench-my-llm runs filter ./results/ --endpoint api.openai.com -t 0
bench-my-llm runs filter ./results/ --tool-version 0.4 --json-output
bench-my-llm runs diff ./results/ before.json after.json
```

`runs filter` matches `--model`, `--endpoint`, and `--tool-version` on
substrings and `--temperature` exactly; with no criteria it lists every
run. `runs diff` selects each run by file name or a unique substring and
prints a metadata change table plus per-metric deltas (using the same
lower-is-better arrows as trends), so an environment change that shifted
the results is easy to spot:

```
        Metadata Changes
Field         Old    New
temperature   0      0.7
tool_version  0.3.0  0.4.0

              Metric Changes
Metric   Old      New      Change
latency  800ms    400ms    ▲ -50.0%
```

### Group Runs by Metadata

To compare whole configurations instead of run pairs, group a results
directory by a metadata field and read aggregate metrics per group:

```bash
bench-my-llm runs group ./results/ --by endpoint
bench-my-llm runs group ./results/ --by temperature --model gpt-4o
bench-my-llm runs group ./results/ --by tool_version --json-output
```

Grouping fields are `model` (the default), `suite`, `endpoint`,
`temperature`, `tool_version`, and `suite_version`. Each group reports
the mean quality, tokens per second, TTFT p50, latency p50, and cost
across its member runs; `--json-output` adds min, max, and the member
file names. Runs without a value for the field land in an `(unset)`
group. This answers questions like "how did average quality differ
between my two endpoints across all runs" in one command:

```
   Runs Grouped by endpoint (5 run(s), 2 group(s))
endpoint                   Runs  Quality  TTFT p50   Cost
http://localhost:1234/v1      3      72%     220ms  $0.0000
https://api.openai.com/v1     2      89%      95ms  $0.0041
```

### Local Models (Ollama)

bench-my-llm auto-detects a running Ollama server. List what is
installed locally:

```bash
bench-my-llm ollama
```

```
              Local Ollama Models (2)
Model         Params  Quant    Size    Family
llama3:8b     8B      Q4_0     4.7 GB  llama
phi3:mini     3.8B    Q4_K_M   2.3 GB  phi3
```

Then benchmark with the `--ollama` flag, no URL or key needed. Tagless
names resolve automatically (`llama3` finds `llama3:latest`):

```bash
bench-my-llm run --ollama -m llama3
bench-my-llm compare --ollama llama3 phi3 -s coding
bench-my-llm ollama --json-output          # machine-readable model list
bench-my-llm ollama --url http://gpu:11434 # remote Ollama host
```

If a model is not installed, the error lists everything that is. Manual
configuration still works for any OpenAI-compatible endpoint:

```bash
bench-my-llm run --model llama3 --base-url http://localhost:11434/v1 --api-key ollama
```

### CI Integration

Add to your GitHub Actions workflow:

```yaml
- name: Benchmark LLM
  run: |
    pip install bench-my-llm
    bench-my-llm run --model gpt-4o-mini --suite reasoning --output benchmark.json
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

- name: Upload results
  uses: actions/upload-artifact@v4
  with:
    name: benchmark-results
    path: benchmark.json
```

## 🛠️ Development

```bash
git clone https://github.com/manasvardhan/bench-my-llm.git
cd bench-my-llm
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
```

## 📄 License

MIT. See [LICENSE](LICENSE).
