Metadata-Version: 2.4
Name: local-bench-ai
Version: 0.2.6
Summary: Benchmark runner for local-bench.ai - the community quality leaderboard for local AI setups.
Project-URL: Homepage, https://local-bench.ai
Project-URL: Methodology, https://local-bench.ai/methodology
Project-URL: Submit, https://local-bench.ai/submit
Project-URL: Repository, https://github.com/local-bench/local-bench
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: anyio>=4
Requires-Dist: httpx>=0.27
Requires-Dist: langdetect>=1.0.9
Requires-Dist: math-verify>=0.9.0
Provides-Extra: build
Requires-Dist: datasets>=2.20; extra == "build"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: nltk>=3.9; extra == "dev"
Requires-Dist: jinja2>=3.1; extra == "dev"
Provides-Extra: hf
Requires-Dist: transformers>=4.51; extra == "hf"
Requires-Dist: jinja2>=3.1; extra == "hf"

# local-bench-ai

CLI benchmark runner for [local-bench.ai](https://local-bench.ai) — a community quality
leaderboard for local AI setups. It benchmarks GGUF models served by llama.cpp (or any
OpenAI-compatible endpoint), scores them on the six-axis Local Intelligence Index
(Agentic, Knowledge, Instruction-Following, Tool calling, execution-verified Coding, Math),
and packs signed, reproducible result bundles you can submit for maintainer review.

## Quickstart

```bash
pip install "local-bench-ai[hf]"   # Python 3.11+

# 1. Fetch the ranked suite from the site (hash-verified)
localbench fetch-suite --site https://local-bench.ai \
  --suite suite-v1-full-exec-6axis-v1 --accept-suite-terms

# 2. Cache the tokenizer/chat template for offline identity
localbench cache-tokenizer <hf-model-id>

# 3. Strongest provenance: the CLI launches the pinned llama.cpp server itself
localbench bench \
  --runtime llama.cpp --server-bin <path-to-llama-server> \
  --model-file <model.gguf> --model-id <model-slug> \
  --hf-model-id <hf-model-id> \
  --suite full-exec-6axis-v1 --bench all \
  --lane bounded-final-v2 --profile auto --tier standard \
  --ctx 32768 --seed 1234 --out runs/my-bench

# 4. Submit for maintainer review (nothing auto-publishes)
localbench submit run --run runs/my-bench
```

The site's [submit page](https://local-bench.ai/submit) generates these commands for your
exact model and runtime, including the full identity flag set for bring-your-own-server
runs. Publishable bounded-final-v2 runs require a 32k server context.

## What makes rows trustworthy

- Suites are hash-pinned releases; sampler settings are pinned (greedy, seeded).
- Coding is BigCodeBench-Hard, re-executed by the maintainer's sandbox verifier before it
  can rank; community agentic results are labeled self-reported.
- Every number on the board links to a receipt with the full run manifest.
- Nothing ranks without maintainer review.

Methodology: <https://local-bench.ai/methodology>
