Metadata-Version: 2.1
Name: local-inferbench
Version: 0.3.0
Summary: A benchmarking framework for Ollama models
Author: Devin Hill
License: MIT
Project-URL: Homepage, https://github.com/iamdevinhill/local-inferbench
Project-URL: Repository, https://github.com/iamdevinhill/local-inferbench
Project-URL: Bug Tracker, https://github.com/iamdevinhill/local-inferbench/issues
Project-URL: PyPI, https://pypi.org/project/local-inferbench/
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: httpx>=0.25
Requires-Dist: psutil>=5.9
Requires-Dist: pydantic>=2.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: rich>=13.0
Provides-Extra: gpu
Requires-Dist: nvidia-ml-py>=12.0; extra == "gpu"
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == "pandas"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Provides-Extra: all
Requires-Dist: local-inferbench[dev,gpu,pandas]; extra == "all"

# local-inferbench

A benchmarking framework for Ollama models. Think `pytest` for local LLM performance.

Run standardized benchmarks across Ollama models. Collect hardware-aware metrics, quality scores, and compare results over time.

## Installation

```bash
pip install local-inferbench
```

With GPU monitoring (NVIDIA):

```bash
pip install local-inferbench[gpu]
```

## Quick Start

### CLI

```bash
# Run a benchmark (includes quality scoring by default)
inferbench run --model qwen3:0.6b --profile quick

# Compare multiple models in one run (shows comparison table at the end)
inferbench run --model qwen3:0.6b --model qwen3:1.7b --profile quick

# Compare last 2 runs
inferbench compare --last 2

# Get model recommendations for your hardware
inferbench recommend --suggest-pull

# Re-score a past run with per-prompt breakdown
inferbench score 1

# View run history
inferbench history

# Export results
inferbench export 1 --format json --output results.json

# List available profiles
inferbench profiles

# Show hardware info
inferbench info
```

### Python API

```python
from local_inferbench import Benchmark, BenchmarkConfig

bench = Benchmark(models=["qwen3:0.6b"], config=BenchmarkConfig(profile="quick", warmup_runs=2))
results = bench.run()
results[0].summary()
```

### Compare Multiple Models

```python
from local_inferbench import Benchmark, BenchmarkConfig
from local_inferbench.results import ComparisonResult

bench = Benchmark(
    models=["qwen3:0.6b", "qwen3:1.7b"],
    config=BenchmarkConfig(profile="quick"),
)
results = bench.run()

comparison = ComparisonResult(results=results)
comparison.table()
print(f"Fastest: {comparison.fastest().model_id}")
```

### Model Recommendations

```python
from local_inferbench import recommend_models

# Get recommendations for installed models based on your hardware
recs = recommend_models()
for r in recs:
    print(f"{r.model_name}: {r.tier} ({r.estimated_vram_gb:.1f} GB)")
```

## Features

### Performance Metrics
- **Time to First Token (TTFT):** mean, median, p95, min, max
- **Throughput:** tokens/sec mean, median, p95
- **Prompt eval speed:** prompt tokens processed per second
- **Hardware:** GPU utilization, VRAM usage, CPU usage, RAM usage (sampled in background)

### Quality Scoring
- **Length:** Is the response appropriately sized for the category?
- **Coherence:** Repetition detection, entropy analysis, structural checks
- **Relevance:** Keyword overlap with prompt, echo detection
- **Completeness:** Did the model finish naturally or get truncated?
- **Category-specific:** Code patterns for code prompts, step-by-step for reasoning, structured output for extraction, etc.

### Hardware-Aware Model Recommendations
- Estimates VRAM requirements from model size and quantization
- Classifies models as optimal/feasible/too_large for your hardware
- Suggests well-known models to download that fit your system

## Benchmark Profiles

| Profile | Prompts | Purpose |
|---------|---------|---------|
| `quick` | 10 | Fast sanity check across core categories |
| `standard` | 50 | Thorough benchmark across diverse categories |
| `stress` | 20 | Sustained throughput with long-form generation |

Custom profiles are supported via YAML files — pass a file path to `--profile`.

## Results Storage

Results are stored in SQLite at `~/.local-inferbench/results.db`. Use `inferbench history` to browse, `inferbench compare` to diff runs, and `inferbench export` to get JSON/CSV.

## Requirements

- Python 3.10+
- A running [Ollama](https://ollama.com) server

## License

MIT
