Metadata-Version: 2.5
Name: beetle-evals
Version: 0.1.0
Summary: TinyEvals for your AI agents and apps
Project-URL: Homepage, https://github.com/Neilblaze/beetle
Project-URL: Source, https://github.com/Neilblaze/beetle
Project-URL: Issues, https://github.com/Neilblaze/beetle/issues
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/banner.png" alt="Beetle — TinyEvals for your AI agents and LLM/SLM apps" width="840">
</p>

<p align="center">
  <b>TinyEvals for your AI agents and LLM/SLM apps.</b><br>
  <sub>Zero runtime dependencies · ~2,100 lines of Python · numbers you can put in a merge decision</sub>
</p>

<p align="center">
  <a href="https://github.com/Neilblaze/beetle/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/Neilblaze/beetle/actions/workflows/ci.yml/badge.svg"></a>
  <a href="https://www.python.org/downloads/"><img alt="Python" src="https://img.shields.io/badge/python-%E2%89%A5%203.10-4c8cf5"></a>
  <a href="pyproject.toml"><img alt="Runtime deps" src="https://img.shields.io/badge/runtime%20deps-0-2eb872"></a>
  <a href="pyproject.toml"><img alt="Typed" src="https://img.shields.io/badge/mypy-strict-2a6db2"></a>
  <a href="CODE_OF_CONDUCT.md"><img alt="Contributor Covenant" src="https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa"></a>
  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache%202.0-777"></a>
</p>

<p align="center">
  <a href="#install">Install</a> ·
  <a href="#quick-start">Quick start</a> ·
  <a href="#how-it-works">How it works</a> ·
  <a href="#the-built-in-evaluators">Evaluators</a> ·
  <a href="#llm-judges">Judges</a> ·
  <a href="#command-line">CLI</a> ·
  <a href="#examples">Examples</a> ·
  <a href="CONTRIBUTING.md">Contributing</a>
</p>

![breaker](https://user-images.githubusercontent.com/48355572/209539106-8e1cbfc6-2f3d-4afd-b96a-890d967dd9ab.png)



Beetle runs your system over a suite of cases, scores each run and tells you whether today is
worse than yesterday. Zero runtime dependencies, roughly 2,100 lines of Python.

Most eval harnesses make it easy to produce a number and hard to trust it. Beetle leans the other
way. An evaluator that crashes is recorded as an error rather than a zero, no mean is reported
without a sample size and an interval next to it, and a run that mostly failed cannot present
itself as a run that mostly passed.

## Install

```bash
pip install git+https://github.com/Neilblaze/beetle
```

Python 3.10 or newer. Nothing else — the wheel carries no runtime dependency, which CI verifies
against the built package's metadata on every push. Linux and macOS are covered in CI.

For a checkout you intend to hack on, see [CONTRIBUTING.md](CONTRIBUTING.md).

## Quick start

```python
from beetle import Case, run
from beetle.evaluators import Contains

report = run([Case("2+2?", expected="4")], lambda case: "4", [Contains("4")])
print(report.pass_rate)
```

A **task** is any callable that takes a `Case`. An **evaluator** is any object with
`evaluate(case, result) -> Score`, or a plain function of the same shape, so there is nothing to
register and no plugin system to learn. Writing your own takes one function.

```python
from beetle import Score


def mentions_units(case, result):
    ok = "km" in str(result.output)
    return Score.of("mentions_units", 1.0 if ok else 0.0, ok, "found km" if ok else "no units")
```

`Score.of` asks you to state `passed` explicitly rather than deriving it from a threshold, and
`Score.error(name, reason)` is there for the case where you could not judge at all. Reaching for
the second one instead of scoring a `0.0` is the whole difference between a report that is
honest and one that is merely confident.

`render(report)` gives you the run as text, which is also what the command line prints.

```text
run 20260816T154204Z-a27bb538  (3 case(s) x 1 trial(s))
beetle 0.1.0 | python 3.12.12 | seed 0 | concurrency 1

cases: 2 passed, 1 failed, 0 errored/timed out  (pass rate  66.7% of judged)

evaluator                      pass     n  err  mean
Contains                      66.7%     3    0   0.667 [0.21, 0.94]

1 case(s) did not pass:
  FAIL     four
           Contains = 0.00  'the answer is' not present in output
```

## How it works

```mermaid
flowchart LR
    LIB("beetle.run"):::e
    CLI("beetle CLI"):::e
    SUITE("Suite of Case"):::d
    RUN("runner.py"):::c
    TASK("your task"):::u
    TRAJ("Trajectory"):::d
    EVAL("evaluators/"):::c
    SCORE("Score"):::d
    AGG("aggregate.py"):::c
    REP("Report"):::d
    OUT("console · runs/ · compare"):::o

    LIB --> RUN
    CLI --> RUN
    SUITE --> RUN
    RUN --> TASK --> TRAJ --> EVAL --> SCORE --> AGG --> REP --> OUT

    classDef e fill:#ede9fe,stroke:#8b5cf6,color:#3b0764,rx:8,ry:8
    classDef u fill:#fef3c7,stroke:#f59e0b,color:#451a03,rx:8,ry:8
    classDef c fill:#ccfbf1,stroke:#14b8a6,color:#042f2e,rx:8,ry:8
    classDef d fill:#e0f2fe,stroke:#38bdf8,color:#082f49,rx:8,ry:8
    classDef o fill:#ffe4e6,stroke:#fb7185,color:#4c0519,rx:8,ry:8
```

There are two ways in. From Python you call `run`, and on the command line you point Beetle at
your own code with `module:attr`, e.g. `--task mymod:agent`. A path works too, so
`--task ./evals/agent.py:run` is fine.

The task runs once per trial and every evaluator then judges that one recorded result, so adding
an evaluator never means running the task again. Results come back in suite order no matter how
they were scheduled, which is what makes two runs comparable line by line.

| Type | What it holds |
|---|---|
| `Case` | one unit of evaluation, i.e. the input, an optional `expected`, optional `expected_tools`, metadata and a stable id |
| `Trajectory` | what the task did, i.e. its final output, steps, tool calls and workspace |
| `Score` | one judgement, with a name, a value, a status and a reason |
| `Status` | `PASS`, `FAIL`, `ERROR`, `TIMEOUT`, where the last two carry no value at all |
| `Report` | per-trial records, per-evaluator summaries and a manifest |

A case with no explicit `id` derives one from a hash of its input, so the same case keeps the
same id across runs and across machines. Random ids would make two runs of one suite
incomparable. Duplicate ids inside a suite are rejected at construction rather than at the point
where they would quietly corrupt a comparison.

## The built-in evaluators

All of them live in `beetle.evaluators` and score in `[0.0, 1.0]` unless a rubric says otherwise.

| Evaluator | What it asserts |
|---|---|
| `Equals(expected)` | the output equals the expectation, taken from the evaluator or from `case.expected` |
| `Contains(value, case_sensitive=True)` | a substring is present in a string output |
| `Matches(pattern, flags=0)` | a regular expression hits; the pattern is compiled at construction, not per case |
| `Approximately(expected, absolute=, relative=)` | two numbers agree within an absolute and/or relative tolerance |
| `JSONSubset(expected)` | the output *contains* the expected shape; extra fields you do not care about are fine |
| `ToolsCalled(expected, ordered=False)` | the expected tools were called, honouring multiplicity; scores the fraction matched |
| `ToolCalledWith(tool, arguments)` | some call to that tool carried at least these arguments |
| `ToolNotCalled(forbidden)` | the agent stayed away from the tools you named |
| `StepBudget(max_steps)` | the agent finished inside a step budget, which is how loops and thrashing show up |
| `FileExists(path, present=True)` | a path does, or deliberately does not, exist in the workspace |
| `FileContains(path, text)` | a workspace file exists and contains the text |
| `FileUnchanged(path, sha256)` | a file still hashes to what it did, which catches an agent that "passed" by weakening its own tests |
| `CommandSucceeds(command, timeout, expect_code=0)` | a command run in the workspace exits as expected |
| `LLMJudge(model, rubric, pass_at)` | a model's named rubric verdict clears a threshold |
| `correctness(model)`, `instruction_following(model)` | two ready-made judges built on `LLMJudge` |

They are deliberately unforgiving about their own configuration. An unset expectation raises
rather than quietly passing, an empty needle is rejected rather than matching everything, and a
non-string output fails with its type named rather than being coerced through `str()`, where
`"42"` would happily be found inside the integer `4200`.

> [!TIP]
> Every evaluator in a run needs a distinct name, because the name is the key its summary is
> reported under. Two bare `Contains(...)` instances both report as `Contains` and are rejected
> before the run starts. Pass `name=` to tell them apart: `Contains("ok", name="has_ok")`.

## Scoring how the system worked

Return a `Trajectory` instead of a bare value when the answer alone is not the thing you care
about.

```python
from beetle import Case, Step, ToolCall, Trajectory, run
from beetle.evaluators import CommandSucceeds, ToolsCalled


def agent(case):
    calls = (ToolCall("search", {"query": case.input}),)
    return Trajectory(
        final_output="done", steps=(Step("assistant", "", calls),), workspace="/tmp/run-42"
    )


run(
    [Case("fix the failing test")],
    agent,
    [ToolsCalled(("search",)), CommandSucceeds(["pytest", "-q"], timeout=60)],
)
```

That last pair is how you score a coding agent. Beetle never creates or cleans up the workspace,
because your task owns that and simply reports where it worked. Workspace evaluators only read
inside the directory they were given: an absolute path, or one climbing out through `..`, is a
configuration error rather than a quietly satisfied assertion about a file the agent never
touched.

> [!WARNING]
> `CommandSucceeds` is a scoring primitive and not a sandbox. It runs exactly what you configured
> with the privileges of the process running Beetle, so keep untrusted input away from it. It
> takes an argument list rather than a shell string, so nothing in a case can be read as shell
> syntax, and on timeout it kills the whole process group. See [SECURITY.md](SECURITY.md).

## LLM judges

Beetle ships no provider client and makes no network calls of its own. A judge accepts any object
with `complete(prompt, system=...)`, so you write a short adapter for whatever you already use
(see `examples/model_adapter.py`) and keep control of credentials, retries and model choice.

Judges answer with a named rubric level rather than a free-form number, since asking a model for a
calibrated decimal invites it to drift onto whatever scale it feels like. Case data is fenced
inside markers derived from its own content, so an output containing instructions cannot break out
into the instruction channel. A refusal, an unparseable reply, or a model that raises an exception
all produce an errored score, which is counted separately and left out of every mean.

```python
from beetle.evaluators import Level, LLMJudge, Rubric

concise = Rubric(
    question="Is the answer as short as it can be while still answering the question?",
    levels=(
        Level("padded", 0.0, "Restates the question, hedges, or adds filler."),
        Level("acceptable", 0.5, "Mostly direct, with some unnecessary scaffolding."),
        Level("tight", 1.0, "Answers and stops."),
    ),
)

judge = LLMJudge(model=my_model, rubric=concise, pass_at=1.0, name="concise")
```

A rubric needs at least two levels, unique labels, and level names a verdict line can actually
say — all checked when you build it, not when the bill arrives. `abstain` is reserved, so a judge
that cannot grade the material says so instead of guessing.

> [!TIP]
> `FakeModel` returns scripted replies, so you can exercise judge wiring without a provider.
> Beetle's own suite uses it, which is why the tests need no network and no keys.

## Repeats, flake and intervals

Agents are not deterministic, and one trial per case cannot tell a fixed regression apart from a
coin flip. `repeat` runs every case several times and reports how often the trials disagreed with
each other.

```python
report = run(suite, agent, [Contains("the answer is")], repeat=5)
```

```text
run 20260818T120705Z-39efad03  (2 case(s) x 5 trial(s))
beetle 0.1.0 | python 3.12.12 | seed 0 | concurrency 1

cases: 8 passed, 2 failed, 0 errored/timed out  (pass rate  80.0% of judged)

evaluator                      pass     n  err  mean
Contains                      80.0%    10    0   0.800 [0.49, 0.94]  flake 20%

2 case(s) did not pass:
  FAIL     wobbly trial 2
           Contains = 0.00  'the answer is' not present in output
  FAIL     wobbly trial 4
           Contains = 0.00  'the answer is' not present in output
```

The bracket after the mean is a 95% interval: a Wilson score interval for binary outcomes, which
behaves sensibly at small `n` and at rates of 0 or 1, and a seeded percentile bootstrap for
continuous scores, so the same run always reports the same interval. `flake` is the mean
per-case disagreement across trials, and a trial that errored before an evaluator could run still
counts as a trial — ignoring it would report a case that fails half the time as perfectly stable.

This is also what keeps `compare` honest. A drop is only called significant when the two
confidence intervals do not overlap, because reporting 0.90 → 0.85 on twelve cases as a
regression trains people to ignore the tool.

## Command line

```bash
beetle run cases.jsonl --task mymod:agent --evaluators mymod:evaluators \
    --repeat 5 --concurrency 8 --out runs/today --fail-under 0.9

beetle compare runs/yesterday runs/today
beetle report runs/today
```

`--task` and `--evaluators` take `module:attr`, resolved the way an import statement would.
`--evaluators` accepts a single evaluator, a list of them, or a zero-argument factory returning
either, which is how you build a judge that needs a configured client.

| Flag | On | Effect |
|---|---|---|
| `--repeat N` | `run` | trials per case, default 1 |
| `--concurrency N` | `run` | cases in flight, default 1 |
| `--timeout S` | `run` | per-case deadline; unset means a case runs until it returns |
| `--retries N` | `run` | retries per case on exception, default 0 |
| `--seed N` | `run` | seed for bootstrap intervals, so a run reproduces its own numbers |
| `--out DIR` | `run` | write the run here, streaming each trial as it completes |
| `--fail-under R` | `run` | exit 1 when the pass rate falls below `R` |
| `--allow-errors` | `run` | do not exit 1 merely because some cases errored |
| `--fail-under-delta D` | `compare` | exit 1 on a drop past `D`, independent of significance |
| `--json` | all three | emit machine-readable JSON instead of the text report |

A `.jsonl` suite holds one case per line, and `.json` works too, either as a list or as an object
with a `cases` key. Unknown fields are rejected rather than dropped, so a typo in `expected` is
caught instead of silently unsetting the expectation.

```json
{"id": "en", "input": "en", "expected": "hello"}
{"id": "fr", "input": "fr", "expected": "bonjour"}
```

Exit codes are a contract, because CI needs to tell a regression apart from a broken command.
`0` means every gate was met, `1` means the run finished but a gate failed, `2` means bad usage or
unreadable input, and `3` means an unexpected internal error. `compare` names the cases that newly
fail and only calls a change significant when the two confidence intervals do not overlap.

```text
evaluator                     baseline  candidate    delta
Contains                        100.0%      75.0%   -25.0%

* = intervals do not overlap; other movement is within noise

newly failing (1): gamma

REGRESSION
```

## Runs on disk

`--out runs/today`, or `write_run(report, "runs/today")` from Python, leaves three files:

| File | What it is |
|---|---|
| `manifest.json` | how the run was configured: version, platform, seed, concurrency, timeout, a content hash of the suite, and every evaluator's declared range and direction |
| `cases.jsonl` | one record per trial, appended as it completes, so an interrupted run still leaves evidence |
| `report.json` | the whole run, including aggregates |

`report.json` is a public contract. Fields are added, never repurposed, `schema_version` is
bumped only on a breaking change, and a version Beetle does not recognise is refused rather than
guessed at. Reading a run never unpickles and never evaluates anything; it is JSON in, dataclasses
out.

Re-running only what did not pass is a suite operation, not a flag:

```python
first = run(suite, agent, evaluators)
retry = run(suite.select(first.unresolved_case_ids()), agent, evaluators)
```

`unresolved_case_ids()` is everything that did not pass — failures, errors and timeouts alike,
since all three are worth another look and none of them is a result you would ship.

## Execution model

The public API is synchronous. `run` awaits a coroutine task internally, so there is no second
async surface to learn. Concurrency is a single parameter, `max_concurrency`, which defaults to 1.
`timeout` and `retries` are arguments to the run, applied to every case in it.

> [!IMPORTANT]
> Neither is guessed for you, and with no `timeout` a case runs until it returns, so set one
> whenever a task can hang.

A case that overruns its timeout is recorded as `TIMEOUT` and the run carries on. Python cannot
kill a thread, so the case is abandoned rather than stopped. A replacement worker starts at once
so the abandoned thread cannot hold a concurrency slot, and workers are daemon threads so it
cannot keep the interpreter alive either. It does keep running until it returns on its own, which
is why genuinely runaway work should be bounded inside the task, the way `CommandSucceeds` bounds
it by killing its subprocess group.

Nothing in a task or an evaluator can end a run. Both are called inside a boundary that turns an
exception into a recorded outcome against the case it happened in — `KeyboardInterrupt` and
`SystemExit` excepted, since those are the operator asking to stop.

## Examples

Every file under `examples/` runs offline with no keys, and CI executes all of them on every push.

| Example | What it shows |
|---|---|
| [`quickstart.py`](examples/quickstart.py) | three cases, one task, one evaluator, one report |
| [`tool_trajectory.py`](examples/tool_trajectory.py) | asserting which tools an agent called, with which arguments, in which order |
| [`coding_agent.py`](examples/coding_agent.py) | scoring a coding agent, including one that "passes" by neutering the tests |
| [`llm_judge.py`](examples/llm_judge.py) | a rubric judge scoring free-form output, driven by `FakeModel` |
| [`model_adapter.py`](examples/model_adapter.py) | the entire provider integration story, in about fifteen lines |
| [`error_handling.py`](examples/error_handling.py) | a task that raises, and why the pass rate does not silently absorb it |
| [`persistence.py`](examples/persistence.py) | write a run, read it back unchanged, re-run only what did not pass |
| [`ci_gate.py`](examples/ci_gate.py) | two versions of one system, compared; exits 1 on a detected regression |
| [`suite.py`](examples/suite.py) | the task and evaluators the CLI examples point at |

```bash
python examples/quickstart.py

beetle run examples/cases.jsonl \
    --task examples/suite.py:task \
    --evaluators examples/suite.py:evaluators
```

## What Beetle leaves out

Chaos and fault injection, red teaming, simulated users, trace ingestion from observability
backends, dataset generation and any hosted dashboard. Each of those is a separate product that
happens to sit near evaluation, and folding them in would cost the small surface that makes this
one worth reading end to end.

The way to add one is a package that imports Beetle, which needs nothing from us: a task is any
callable taking a `Case`, and an evaluator is any object with `evaluate(case, result)`.

## Contributing

Issues and pull requests are welcome. [CONTRIBUTING.md](CONTRIBUTING.md) covers the setup, the
four gates CI enforces — zero runtime dependencies, no `.py` file over 200 lines, `mypy --strict`,
and a test suite that fails if any test opens a socket — and the checklist a new evaluator has to
pass. Participation is governed by the [Code of Conduct](CODE_OF_CONDUCT.md).

## Security

Please do not report vulnerabilities in the issue tracker. [SECURITY.md](SECURITY.md) has the
reporting process and the threat model, including what is deliberately out of scope:
`CommandSucceeds` is not a sandbox, `--task` imports what you point it at, and a judge verdict is
advisory rather than an authorisation decision.

## Citation

```bibtex
@software{beetle,
  title    = {Beetle: TinyEvals for your AI agents and LLM/SLM apps},
  author   = {Banerjee, Pratyay},
  year     = {2026},
  url      = {https://github.com/Neilblaze/beetle},
  license  = {Apache-2.0},
  version  = {0.1.0}
}
```

GitHub's "Cite this repository" button reads [CITATION.cff](CITATION.cff), which is the same
metadata in machine-readable form.

## License

Apache 2.0. See [LICENSE](LICENSE).
