Metadata-Version: 2.4
Name: metrik-bench
Version: 0.1.1
Summary: Metrik Benchmark Lab — sandboxed evaluation, or a refusal. Never an unsandboxed run.
Project-URL: Homepage, https://github.com/Asmodeus14/Metrik
Project-URL: Repository, https://github.com/Asmodeus14/Metrik
Project-URL: Issues, https://github.com/Asmodeus14/Metrik/issues
Author: The Metrik Authors
License-Expression: Apache-2.0
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: metrik-sdk==0.1.1
Description-Content-Type: text/markdown

# metrik-bench

Sandboxed evaluation — or a refusal. Never an unsandboxed run.

```console
$ metrik bench --check          # what isolation is available here, and what is not
$ metrik bench <model> --task humaneval
```

## Why this package is unusual

HumanEval and MBPP ask a model to write code and then **run it**. A benchmark runner that
`exec()`s model output on your machine is a remote-code-execution primitive with extra steps.

[ADR 019](../../docs/planning/adr/019-benchmark-sandbox.md) answers that by refusing rather
than degrading, and records the clause as **not revisitable**. There is no in-process fallback,
no `--i-know-what-im-doing` flag, and no configuration that produces one.

That is enforced in two places, deliberately:

- **The type.** `BenchResult` cannot be constructed for a code-executing task without a sandbox
  attestation, and `isolation: "none"` fails validation. A runner that finds no container
  runtime can emit nothing at all.
- **The runner.** It checks for a runtime *before* generating anything, so the refusal arrives
  in a second rather than after an hour of generation.

## What a refusal looks like

```
error: humaneval executes model-generated code and no container runtime is available
  -> Install Docker Desktop and start it, then re-run. `metrik bench --check` shows
     what was probed. Tasks that do not execute code (mmlu) run without a container.
```

Note the last sentence: MMLU is multiple choice and needs no container. Requiring one for
everything would push people to disable the check they actually need.
