Metadata-Version: 2.4
Name: tourney
Version: 0.1.1
Summary: Local-first benchmarking runner for AI models: run your prompts against the field.
Project-URL: Homepage, https://github.com/k-rthik/tourney
Author: Karthik PV
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai,benchmark,eval,evaluation,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: jsonschema>=4.21
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: tenacity>=8.3
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: fastapi>=0.111; extra == 'all'
Requires-Dist: uvicorn>=0.30; extra == 'all'
Provides-Extra: dashboard
Requires-Dist: fastapi>=0.111; extra == 'dashboard'
Requires-Dist: uvicorn>=0.30; extra == 'dashboard'
Provides-Extra: dev
Requires-Dist: fastapi>=0.111; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-socket>=0.7; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: uvicorn>=0.30; extra == 'dev'
Description-Content-Type: text/markdown

# tourney

[![PyPI](https://img.shields.io/pypi/v/tourney)](https://pypi.org/project/tourney/)
[![Python](https://img.shields.io/pypi/pyversions/tourney)](https://pypi.org/project/tourney/)
[![CI](https://github.com/k-rthik/tourney/actions/workflows/ci.yml/badge.svg)](https://github.com/k-rthik/tourney/actions/workflows/ci.yml)
[![License](https://img.shields.io/pypi/l/tourney)](https://github.com/k-rthik/tourney/blob/main/LICENSE)

Local-first benchmarking runner for AI models. Run your prompts against the field.

`tourney` runs your own prompts and datasets against multiple model APIs (your keys, your machine) and measures **quality, latency, cost, and token usage** — with the statistical rigor researchers expect: bootstrap confidence intervals, pinned model IDs, seeds, config hashes, and full run metadata in a local SQLite database. No hosted service, no telemetry.

## Install

```bash
pip install "tourney[dashboard]"      # or plain `pip install tourney` for CLI/library only
```

## Five-minute quickstart

```bash
tourney init            # scaffolds benchmark.yaml + cases.jsonl
tourney run benchmark.yaml
tourney show latest --failures
tourney serve           # local dashboard at http://127.0.0.1:8355
```

`benchmark.yaml`:

```yaml
name: math-word-problems
models:
  - provider: openai
    model: gpt-4o-2024-08-06          # pin dated IDs for reproducibility
  - provider: anthropic
    model: claude-sonnet-5
  - provider: openai_compat           # anything OpenAI-compatible: Ollama, vLLM, OpenRouter...
    model: llama3.1:8b
    base_url: http://localhost:11434/v1
defaults: {temperature: 0, max_tokens: 256, seed: 42}
prompt:
  system: "Answer with only the final number."
  user: "{{ question }}"              # Jinja2 over each case's input
dataset: {path: cases.jsonl}
graders:
  - {type: numeric, tolerance: 0.001}
run: {concurrency: 8, retries: 3, timeout_s: 60}
```

API keys come from environment variables (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or a custom `api_key_env`) — never from config files.

```
math-word-problems · run 01J9X4… · $0.213
┌──────────────────────┬───────┬──────────────┬─────────┬─────────┬─────────┬────────┐
│ model                │ score │ 95% CI       │ p50 lat │ p99 lat │ cost    │ errors │
├──────────────────────┼───────┼──────────────┼─────────┼─────────┼─────────┼────────┤
│ claude-sonnet-5      │ 0.960 │ [0.89, 1.00] │   944ms │  2107ms │ $0.0920 │      0 │
│ gpt-4o-2024-08-06    │ 0.940 │ [0.86, 0.98] │   812ms │  2431ms │ $0.1210 │      0 │
└──────────────────────┴───────┴──────────────┴─────────┴─────────┴─────────┴────────┘
```

## Why another eval tool?

- **Local-first.** Results live in `./.tourney/results.db` (SQLite). Copy it, query it, commit exports.
- **Reproducible.** Config hash, tourney/Python versions, and git commit stored per run. Deterministic graders by default; stats are seeded.
- **Cheap to iterate.** Responses are content-address cached — rerunning an unchanged benchmark costs $0.
- **CI-native.** `tourney run --json --fail-under 0.85` exits non-zero when a model regresses.
- **Honest statistics.** Bootstrap CIs over cases; errors score 0 instead of being dropped.

## Graders

| type | what it checks |
|---|---|
| `exact_match` | normalized string equality (`normalize: [strip, lowercase, collapse_whitespace]`) |
| `contains` | substring presence |
| `regex` | pattern match (`pattern:`) |
| `numeric` | last number in the response vs `expected`, within `tolerance` |
| `json_schema` | output is valid JSON, optionally conforming to `schema:` |

Multiple graders per benchmark average into the case score. LLM-as-judge is planned as strictly opt-in.

## Library

```python
import tourney

result = tourney.run("benchmark.yaml")
result.summary()      # per-model dicts: score, CI, latency percentiles, cost
result.to_records()   # per-completion rows -> pd.DataFrame(result.to_records())
```

## CLI reference

```
tourney init                     scaffold a new benchmark
tourney run <config>             run it (--model, --limit, --no-cache, --json, --fail-under)
tourney list                     recent runs
tourney show <run|latest>        summary table (--failures for the failing cases)
tourney export <run> --format    jsonl | csv | md
tourney serve                    local dashboard (requires tourney[dashboard])
```

## Extending

Third-party providers and graders register via entry points (`tourney.providers`, `tourney.graders`) — publish a package with a `Provider` or `Grader` subclass and it becomes available by name in configs. A `mock` provider ships in the box for dry runs and tests.

## Development

```bash
pip install -e ".[dev]"
pytest        # fully offline — network is disabled in the test suite
```

Apache-2.0.
