Metadata-Version: 2.4
Name: localarena
Version: 0.2.0
Summary: Live multi-provider model evaluation, scoring, reports, and Elo
Project-URL: Homepage, https://github.com/maziyarpanahi/localarena
Project-URL: Repository, https://github.com/maziyarpanahi/localarena
Project-URL: Issues, https://github.com/maziyarpanahi/localarena/issues
Author: Maziyar Panahi
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: arena,elo,evaluation,leaderboard,local-models
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# LocalArena

Live multi-provider model evaluation, deterministic scoring, standalone HTML
reports, and Elo standings.

```bash
pip install localarena
```

LocalArena calls the endpoints you explicitly configure. It does not bundle
recorded answers, copy public leaderboard results, download models, or fall
back to another provider.

## Command line

Create a JSON configuration with one or more model targets and prompt tasks,
then run the complete Cartesian product:

```bash
localarena run evaluation.json \
  --output results.json \
  --report report.html
```

Built-in profiles:

- `llamacpp` — `http://127.0.0.1:8080/v1`
- `ollama` — `http://127.0.0.1:11434/v1`
- `lmstudio` — `http://127.0.0.1:1234/v1`
- `openrouter` — `OPENROUTER_API_KEY`
- `openai` — `OPENAI_API_KEY`
- `custom` — an explicit compatible base URL

```bash
localarena providers
localarena models ollama
localarena report results.json --output report.html
```

Config files use `api_key_env` and `headers_env`; literal API keys and
sensitive literal headers are rejected. Prompt and answer content is excluded
from results by default. Use `--include-content` only when it is safe to retain
that content.

See the
[complete configuration example](https://github.com/maziyarpanahi/localarena/blob/main/examples/evaluation-config.json)
and
[provider/task guide](https://github.com/maziyarpanahi/localarena/blob/main/docs/provider-and-task-support.md).

## Python API

```python
from localarena import (
    EvaluationRunner,
    NumericMatch,
    ModelTarget,
    PromptTask,
    create_provider,
    write_html_report,
)

target = ModelTarget(
    name="local-model",
    provider=create_provider("ollama"),
    model="qwen3:0.6b",
    max_tokens=64,
    temperature=0,
)
task = PromptTask.from_text(
    "arithmetic",
    "Return only the result of 6 multiplied by 7.",
    evaluator=NumericMatch(42),
)

run = EvaluationRunner(
    [target],
    [task],
    max_concurrency=2,
).run(name="Live evaluation")

print(run.to_json())
write_html_report(run, "report.html")
```

Deterministic evaluators include exact, containment, regular-expression, JSON,
and numeric scoring. `ModelJudge` supports live rubric scoring with another
configured target. Generation failures and scoring failures remain individual
result rows instead of aborting the run.

## Elo arena

The lower-level portable arena remains available:

```python
from localarena import Arena, Result

arena = Arena(["model-a", "model-b"])
arena.record("model-a", "model-b", Result.LEFT)

for row in arena.standings():
    print(row.rank, row.name, row.rating)
```

Arena and evaluation snapshots are JSON-safe. The npm package implements the
same arena and evaluation run contracts.
