Metadata-Version: 2.4
Name: tolokaforge
Version: 0.2.6
Summary: Tolokaforge - LLM Tool-Use Benchmarking Harness
Project-URL: Homepage, https://github.com/Toloka/tolokaforge
Project-URL: Repository, https://github.com/Toloka/tolokaforge
Project-URL: Documentation, https://github.com/Toloka/tolokaforge/tree/main/docs
Project-URL: Issues, https://github.com/Toloka/tolokaforge/issues
Author: Renaud de la Gueronniere, Tolokaforge Contributors
Maintainer: Renaud de la Gueronniere
License: Apache-2.0
License-File: LICENSE
Keywords: agents,benchmark,evaluation,llm,mcp,tool-use
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: addict>=2.4.0
Requires-Dist: click>=8.1.0
Requires-Dist: deepdiff>=6.0.0
Requires-Dist: docker>=7.0.0
Requires-Dist: docstring-parser>=0.16
Requires-Dist: grpcio-health-checking>=1.60.0
Requires-Dist: grpcio-tools>=1.60.0
Requires-Dist: grpcio>=1.60.0
Requires-Dist: httpx>=0.25.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: jsonpath-ng>=1.6.0
Requires-Dist: jsonschema>=4.20.0
Requires-Dist: litellm<2.0.0,>=1.83.14
Requires-Dist: loguru>=0.7.0
Requires-Dist: mcp>=0.1.0
Requires-Dist: odata-query>=0.10.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic[email]>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rich>=13.0.0
Requires-Dist: sqlalchemy>=2.0.48
Requires-Dist: starlette>=0.52.1
Requires-Dist: structlog>=24.0.0
Requires-Dist: tenacity>=8.2.0
Requires-Dist: toml>=0.10.0
Requires-Dist: typesense>=2.0.0
Provides-Extra: adapters
Requires-Dist: tolokaforge-adapter-terminal-bench; extra == 'adapters'
Provides-Extra: all
Requires-Dist: fastapi>=0.108.0; extra == 'all'
Requires-Dist: openpyxl>=3.1.0; extra == 'all'
Requires-Dist: pillow>=12.1.0; extra == 'all'
Requires-Dist: playwright>=1.40.0; extra == 'all'
Requires-Dist: python-docx>=1.0.0; extra == 'all'
Requires-Dist: python-pptx>=0.6.21; extra == 'all'
Requires-Dist: rank-bm25>=0.2.2; extra == 'all'
Requires-Dist: tolokaforge-adapter-terminal-bench; extra == 'all'
Requires-Dist: uvicorn>=0.25.0; extra == 'all'
Provides-Extra: browser
Requires-Dist: pillow>=12.1.0; extra == 'browser'
Requires-Dist: playwright>=1.40.0; extra == 'browser'
Provides-Extra: office
Requires-Dist: openpyxl>=3.1.0; extra == 'office'
Requires-Dist: python-docx>=1.0.0; extra == 'office'
Requires-Dist: python-pptx>=0.6.21; extra == 'office'
Provides-Extra: rag
Requires-Dist: rank-bm25>=0.2.2; extra == 'rag'
Provides-Extra: server
Requires-Dist: fastapi>=0.108.0; extra == 'server'
Requires-Dist: uvicorn>=0.25.0; extra == 'server'
Provides-Extra: terminal-bench
Requires-Dist: tolokaforge-adapter-terminal-bench; extra == 'terminal-bench'
Description-Content-Type: text/markdown

# Tolokaforge

A benchmarking harness for evaluating tool-using LLM agents. Multi-turn agent/user loops, sandboxed execution, deterministic grading, and rich telemetry — across any provider via LiteLLM.

## Highlights

- **Agent + User Loop** – Multi-turn conversations where both agent and user models call tools.
- **Sandboxed Execution** – Tool calls proxy into Dockerized services with no external network access.
- **MCP-Compatible Tooling** – Tasks declare tools via Model Context Protocol or built-ins.
- **Deterministic Grading** – JSONPath assertions, state hashes, transcript rules, optional LLM judges.
- **Rich Metrics** – pass@k, cost/token estimates, latency percentiles, failure attribution.
- **Distributed Runner** – SQLite for local runs, Postgres for multi-machine execution.
- **Bring-Your-Own Models** – Any provider supported by LiteLLM (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, OpenRouter, and more).

## Installation

```bash
pip install tolokaforge                # core
pip install "tolokaforge[browser]"     # + Playwright
pip install "tolokaforge[all]"         # everything
```

Dev install:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
```

See [Python Package Guide](docs/PYTHON_PACKAGE.md) for all extras and programmatic API usage.

## Quick Start

```bash
# 1. Configure provider keys
cp .env.example .env

# 2. Run one of the included examples
uv run tolokaforge run --config examples/native/coding/run_config.yaml

# 3. Check results
uv run tolokaforge status --run-dir results/coding_example
uv run tolokaforge analyze --trajectory results/coding_example/trials/<task_id>/0/trajectory.yaml
```

That's it. Docker services for browser / mobile / RAG tasks start automatically via
[`auto_start_services`](tolokaforge/core/models.py) (default: `true`).

### What is a run config?

A run config (e.g. `examples/native/coding/run_config.yaml`) is a single YAML file
that fully specifies an evaluation. The harness reads it and runs the benchmark:

```yaml
models:                       # which LLM(s) drive the agent + user simulator
  agent:    {provider: openrouter, name: anthropic/claude-sonnet-4-6, ...}
  user:     {provider: openrouter, name: anthropic/claude-sonnet-4-6, ...}

orchestrator:                 # how the run is executed
  workers: 4                  # parallel trials
  repeats: 1                  # trials per task
  max_turns: 20

evaluation:                   # what to evaluate
  tasks_glob: "**/task.yaml"  # which tasks to load (relative to task_packs root or repo)
  task_packs:                 # optional: directories that contain task.yaml files
    - "examples/native/coding/dataset"
  output_dir: "results/coding_example"
```

To write your own benchmark, copy a working example as a starting point:

```bash
cp examples/native/coding/run_config.yaml my_run.yaml
$EDITOR my_run.yaml         # change model, tasks_glob, output_dir
uv run tolokaforge run --config my_run.yaml
```

Every example under [`examples/`](examples/) ships a `run_config.yaml` next to its
task data. There is no global "default" config — the run config and the tasks it
points at always travel together.

For distributed execution and advanced workflows see the [Runner Guide](docs/RUNNER.md).

## Project Structure

```
tolokaforge/          # Installable Python package
├── cli/              # CLI commands (run, validate, status, analyze)
├── core/             # Orchestration, grading, metrics, queue
├── tools/            # Built-in + MCP tool system
└── env/              # Environment services (JSON DB, mock web, RAG)
examples/             # Reference task layouts with runnable run_config.yaml
├── native/           # default `native` adapter
│   ├── browser_task/
│   ├── coding/
│   ├── native_shared_domain/
│   └── tool_use/
└── terminal_bench/   # `terminal_bench` adapter (Docker compose)
```

## Documentation

| Topic | Link |
| --- | --- |
| Getting started | [docs/GETTING_STARTED.md](docs/GETTING_STARTED.md) |
| Task authoring | [docs/TASKS.md](docs/TASKS.md) |
| Grading system | [docs/GRADING.md](docs/GRADING.md) |
| Tool reference | [docs/TOOLS.md](docs/TOOLS.md) |
| Browser/mobile tools | [docs/BROWSER_TOOLS.md](docs/BROWSER_TOOLS.md) |
| Runner & distributed execution | [docs/RUNNER.md](docs/RUNNER.md) |
| Adapter architecture | [docs/ADAPTER_ARCHITECTURE.md](docs/ADAPTER_ARCHITECTURE.md) |
| Analytics & failure attribution | [docs/ANALYTICS.md](docs/ANALYTICS.md) |
| Python package API | [docs/PYTHON_PACKAGE.md](docs/PYTHON_PACKAGE.md) |
| Task packs | [docs/TASK_PACKS.md](docs/TASK_PACKS.md) |
| Configuration reference | [docs/REFERENCE.md](docs/REFERENCE.md) |
| Security model | [docs/SECURITY.md](docs/SECURITY.md) |
| Docker runtime | [docs/BENCHMARK_BACKEND_DESIGNS.md](docs/BENCHMARK_BACKEND_DESIGNS.md) |
| Benchmark types | [docs/BENCHMARK_TYPES.md](docs/BENCHMARK_TYPES.md) |
| Testing guide | [tests/README.md](tests/README.md) |

## Examples

| Example | Description |
| --- | --- |
| [`examples/native/coding/`](examples/native/coding/) | Simplest native pattern — file-write grading |
| [`examples/native/tool_use/`](examples/native/tool_use/) | Structured tool-call grading |
| [`examples/native/browser_task/`](examples/native/browser_task/) | Browser tool against mock-web fixtures |
| [`examples/native/native_shared_domain/`](examples/native/native_shared_domain/) | `_shared/domain.yaml` + FastMCP pattern |
| [`examples/terminal_bench/`](examples/terminal_bench/) | Docker-compose stacks with `terminal_bench` adapter |

## Testing

```bash
make test              # all tests
make test-unit         # fast, isolated
make test-functional   # mocked externals
```

See [tests/README.md](tests/README.md) for integration/E2E tests and contribution guidelines.

## License

Apache-2.0 — see [LICENSE](LICENSE).

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## Citation

Use [CITATION.cff](CITATION.cff) or [CITATION.bib](CITATION.bib) when referencing Tolokaforge in papers or reports.
