Metadata-Version: 2.1
Name: tokmark
Version: 0.2.0
Summary: Benchmark LLMs on your local hardware — TTFT, tokens/sec, VRAM, cost estimate
Author: Moti Israeli
License: MIT
Project-URL: Homepage, https://github.com/motiisr/tokmark
Project-URL: Repository, https://github.com/motiisr/tokmark
Project-URL: Issues, https://github.com/motiisr/tokmark/issues
Keywords: llm,benchmark,ollama,local,gpu
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: rich>=13.0
Requires-Dist: httpx>=0.27
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-httpx>=0.30; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"

# ModelbBench

Benchmark LLMs on your local hardware. Get TTFT, tokens/sec, VRAM usage, and estimated self-hosting cost — in one command.

```
pip install modelbench
modelbench run llama3.2:3b
```

## Output

```
ModelbBench v0.1.0 — llama3.2:3b on ollama (rtx4080super)
──────────────────────────────────────────────────────────
  Model load time     8.2s
  Prompt suite        v1 (5 prompts, temp=0.0)
  Warm-up             5 requests (not counted)

┌─────────────────────┬──────────┬──────────┬──────────┐
│ Metric              │   p50    │   p90    │   p99    │
├─────────────────────┼──────────┼──────────┼──────────┤
│ Time to first token │  180ms   │  210ms   │  280ms   │
│ End-to-end latency  │  4200ms  │  5100ms  │  6800ms  │
│ Tokens / sec        │  45.2    │    —     │    —     │
│ VRAM (peak)         │  3840 MB │    —     │    —     │
└─────────────────────┴──────────┴──────────┴──────────┘
```

## Requirements

- Python 3.11+
- [Ollama](https://ollama.com) installed and running
- NVIDIA GPU (VRAM stats require `nvidia-smi`)

## Commands

```bash
# Run benchmark
modelbench run llama3.2:3b

# View stored results
modelbench show
modelbench show llama3.2:3b
```

Results are saved to `~/.modelbench/benchmarks.json`.

## Stack

- Backend: Ollama (V1)
- Platform: WSL2 / Linux + NVIDIA GPU

## License

MIT
