Metadata-Version: 2.5
Name: evalsi-harness
Version: 0.1.0
Summary: The light built-in agent harness for Evals.si: tool loop, MCP tools, sandboxes, user simulator, record and replay.
Project-URL: Homepage, https://abhishek-rnjn.github.io/Evals.si/
Project-URL: Documentation, https://abhishek-rnjn.github.io/Evals.si/
Project-URL: Repository, https://github.com/Abhishek-Rnjn/Evals.si
Project-URL: Issues, https://github.com/Abhishek-Rnjn/Evals.si/issues
Project-URL: Changelog, https://github.com/Abhishek-Rnjn/Evals.si/blob/main/CHANGELOG.md
License-Expression: Apache-2.0
Requires-Python: >=3.11
Requires-Dist: evalsi[server]==0.1.0
Requires-Dist: httpx>=0.27
Description-Content-Type: text/markdown

# evalsi-harness

The light built-in agent harness for [Evals.si](https://github.com/Abhishek-Rnjn/Evals.si).
It drives an agent through a task in an Evals.si sandbox and grades the end state.

It runs either of these:
- the built-in tool-calling agent on any OpenAI-compatible or Anthropic model;
- a bring-your-own agent over A2A, MCP, OpenAI Responses or HTTP, or as a CLI run inside the sandbox.

```bash
pip install evalsi-harness
evalsi run -f agent-run.yaml
```

What it provides:
- **Sandbox tools:** `bash`, `read_file` and `write_file`, plus an escalation request the harness policy decides on.
- **Other tools:** MCP tools over streamable HTTP or stdio; mocked tools; deterministic fault injection.
- **Budgets:** steps, tokens, spend and wall-clock time.
- **Simulated user:** an LLM user for multi-turn tasks.
- **Record and replay:** model calls and tool I/O are captured, replay can branch from step N, and sandbox commands are re-executed so the environment is rebuilt.
- **Environment checkers:** exit code, JSON, JUnit or a Python function.
- **OTel spans:** every step becomes a span.
- **The agent's diff:** recorded for the `code-quality` evaluator when the workdir is a git repository.
- **Benchmark importers:** `harbor://` (Terminal-Bench 2 and other Harbor datasets) and `terminal-bench://` (Terminal-Bench 1) task directories, with their Dockerfiles translated into environments.

See [docs/guides/agent-runs.md](../../docs/guides/agent-runs.md).
