# inspect-evals-lint

Static checks for [Inspect AI](https://inspect.aisi.org.uk/) evaluations: file structure, test coverage conventions, best practices and sandbox image pinning. Documentation, including a page per rule, is at [inspect-evals-lint.generality.org](https://inspect-evals-lint.generality.org/).

These checks began life as the `autolint` tool inside [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals). They are packaged here so any repository of Inspect evaluations can run the same checks, including standalone repos built from the [inspect-evals-template](https://github.com/Generality-Labs/inspect-evals-template) and submitted to the inspect_evals register.

Nothing is imported or executed from the evaluation being checked. Every check is static analysis over Python source (via `ast`), `eval.yaml`, compose files and `pyproject.toml`.

## Install

```bash
uv add --dev inspect-evals-lint
# or
pip install inspect-evals-lint
```

## Usage

```bash
inspect-evals-lint gpqa                     # one evaluation (or helper package, e.g. utils)
inspect-evals-lint gpqa utils               # several
inspect-evals-lint --all                    # every evaluation and helper package in the repo
inspect-evals-lint --all --select IEBP      # only the best-practice rules
inspect-evals-lint gpqa --ignore readme,IETS
inspect-evals-lint --all --output-format json > lint.json
inspect-evals-lint --all --output-format markdown   # a summary for a pull request comment
inspect-evals-lint --preset single-eval --task src/castle/castle.py   # the package holding a task file
inspect-evals-lint --list-rules             # every rule with its code, category, scope and summary
inspect-evals-lint --explain IEBP002        # one rule's documentation
```

Run it inside the project's environment (`uv run inspect-evals-lint`): `external_dependencies` maps import names to distributions through the packages installed there. `--select` replaces the configured selection for one run and `--ignore` adds to it; both take rule names, codes or code prefixes, comma-separated. With more than one package the text output ends with an overall summary, a per-rule compliance table and the failures grouped by rule.

`--output-format json` writes one document to stdout and sends progress to stderr, so the output can be piped straight into other tooling ([docs/output.md](output.md) has the schema). `--output-format github` writes one GitHub Actions annotation per finding instead, each message starting with the file, line and rule, so a lint step marks up the pull request and the job log still names every location; in an Actions job it also appends the Markdown summary to the job summary (`GITHUB_STEP_SUMMARY`). `--output-format markdown` writes, per package, a headline of rules met with a per-category split and a collapsible list of the rules not met, with warnings or suppressed, for a pull request comment or job summary.

`--task <path>` (repeatable) lints the package holding a task file instead of a named package: the evaluation is the directory holding the file, its parent is the source root, and enclosing packages form the import prefix, so a repository can be linted from a register entry's `task_path` without knowing its layout. A task file with no `__init__.py` beside it is a usage error naming the problem. Everything but the layout comes from the configuration or `--preset`. It carries `schema_version`, `passed`, run-wide `summary` counts and a `packages` list. Each package has its `kind` (`eval` or `helper`), `rules` (one entry per rule that ran, at its worst status, with indexes into that package's `diagnostics`), `outcomes` (one per rule that passed or did not apply) and `diagnostics` (one per finding, with `rule`, `code`, `category` (`file_structure`, `code_quality`, `tests`, `best_practices` or `security`), `severity`, `status`, `message`, `file` relative to the repository root, `line`, `column` and a `hint`). Every finding points at a file, and at a line where the finding is in a file's contents, so `# inspect-evals-lint: ignore[<rule>]` on that line suppresses it whatever the rule. JSON schema version 2 adds the `security` category and `IESC` rule prefix. Consumers of version 1 must update their category and code validation before upgrading. Existing rule codes and categories are unchanged.

The repository root is the nearest `pyproject.toml` carrying a `[tool.inspect-evals-lint]` table (falling back to the nearest `pyproject.toml`, then the current directory). Pass `--root` to override.

Exit codes: `0` all checks passed (warnings, skips and suppressions count as passing), `1` at least one check failed, `2` usage or configuration error.

## Configuration

Configuration lives in `pyproject.toml`. Pick a layout preset and override any field:

```toml
[tool.inspect-evals-lint]
preset = "multi-eval"   # or "single-eval" / "monorepo"
```

<!-- config-table:start -->

| Key                         | `multi-eval` preset                                          | `monorepo` preset                                            | `single-eval` preset                                         | Meaning                                                                                                                                                                                                                                                                                                               |
| --------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source-root`               | `src`                                                        | `src/inspect_evals`                                          | `src`                                                        | Directory (relative to the repo root) holding one sub-directory per evaluation.                                                                                                                                                                                                                                       |
| `tests-root`                | `tests`                                                      | `tests`                                                      | `tests`                                                      | Directory holding `<tests_root>/<name>/` test packages.                                                                                                                                                                                                                                                               |
| `tests-layout`              | `per-eval`                                                   | `per-eval`                                                   | `flat`                                                       | `per-eval` requires `<tests_root>/<name>/` and `__init__.py` files throughout it; `flat`, for a single-evaluation repository, also accepts test files directly under `tests_root` and does not require `__init__.py` files, since there is no second test tree to collide with.                                       |
| `readme-location`           | `eval-dir`                                                   | `eval-dir`                                                   | `repo-root`                                                  | `eval-dir` requires `README.md` inside the evaluation directory; `repo-root` also accepts the repository's top-level `README.md`.                                                                                                                                                                                     |
| `eval-yaml-required`        | `true`                                                       | `true`                                                       | `true`                                                       | Whether a missing `eval.yaml` fails. False skips instead, for repositories whose metadata lives in the inspect_evals register; a present file is still validated.                                                                                                                                                     |
| `import-prefix`             | `""`                                                         | `inspect_evals`                                              | `""`                                                         | Dotted import prefix for evaluations, e.g. `inspect_evals`; empty when an eval imports as `<name>`.                                                                                                                                                                                                                   |
| `registry`                  | `entry-points`                                               | `module`                                                     | `entry-points`                                               | How tasks are registered: a Python module that imports every eval, `[project.entry-points.inspect_ai]`, or not checked.                                                                                                                                                                                               |
| `registry-module`           | unset                                                        | `src/inspect_evals/_registry.py`                             | unset                                                        | Path of the registry module (relative to the repo root); required when `registry == "module"`.                                                                                                                                                                                                                        |
| `helper-dirs`               | `["utils"]`                                                  | `["utils"]`                                                  | `["utils"]`                                                  | Sub-directories of `source_root` holding shared code rather than an evaluation. They are linted with the helper scope: the rules that guard code behaviour (private imports, score values, model roles, dependencies, tests for custom components) but not the ones about an evaluation's structure and registration. |
| `ignore-dirs`               | `["examples"]`                                               | `[]`                                                         | `["examples"]`                                               | Sub-directories of `source_root` that are never linted.                                                                                                                                                                                                                                                               |
| `eval-yaml-required-fields` | `["title", "description", "group", "contributors", "tasks"]` | `["title", "description", "group", "contributors", "tasks"]` | `["title", "description", "group", "contributors", "tasks"]` | Top-level keys every `eval.yaml` must define.                                                                                                                                                                                                                                                                         |
| `isolated-packages-dir`     | unset                                                        | `packages`                                                   | unset                                                        | Directory of per-eval `<dir>/<name>/pyproject.toml` files that declare an eval's dependencies instead of a root extra.                                                                                                                                                                                                |
| `per-eval-dependency-group` | `false`                                                      | `true`                                                       | `false`                                                      | Whether every evaluation with third-party imports must own an `[project.optional-dependencies]` group named after itself, so one evaluation's dependencies install without the rest (the inspect_evals convention). False accepts `[project].dependencies` and any extra, as a standalone repository declares them.   |
| `select`                    | `["IE"]`                                                     | `["IE"]`                                                     | `["IE"]`                                                     | Rules to run: names, codes or code prefixes. The default prefix selects every rule.                                                                                                                                                                                                                                   |
| `ignore`                    | `[]`                                                         | `[]`                                                         | `[]`                                                         | Rules never to run, in the same forms as `select`. Wins over `select`.                                                                                                                                                                                                                                                |
| `exclude`                   | `[]`                                                         | `[]`                                                         | `[]`                                                         | Glob patterns, relative to the repository root, of files the AST-based rules never read. For code that is shipped into a sandbox rather than run on the host, such as challenge sources that are not even valid Python 3.                                                                                             |
| `per-file-ignores`          | `[]`                                                         | `[]`                                                         | `[]`                                                         | `(glob, selectors)` pairs: findings in files matching the glob are suppressed for the selected rules.                                                                                                                                                                                                                 |
| `allowlists.<rule>`         | `{}`                                                         | `{}`                                                         | `{}`                                                         | Per rule, the `(package, key)` pairs it reports as warnings instead of failures. Only rules declared with `allowlist=True` accept one.                                                                                                                                                                                |
| `<rule>`                    | `{}`                                                         | `{ dockerfile_locking = { host-lock-coupling = "warn" } }`   | `{}`                                                         | `[tool.inspect-evals-lint.<rule>]` tables, passed through to the rule that declares them. Keys may be kebab-case; a preset's default for a key applies when the table leaves it unset.                                                                                                                                |

<!-- config-table:end -->

The presets describe how a repository is laid out. `multi-eval` is several evaluation packages side by side under `src/`, each registered by an entry point, as the [inspect-evals-template](https://github.com/Generality-Labs/inspect-evals-template) produces. `single-eval` is one evaluation package that is the whole repository: tests directly under `tests/` or in one `tests/<name>/`, the README at the repository root. `monorepo` is inspect_evals itself. When nothing names a preset (no `[tool.inspect-evals-lint]` table, or a table without `preset`) the source root decides: exactly one evaluation package gives `single-eval`, anything else `multi-eval`, and the run says which it chose and why. `--preset` overrides the table for one run.

The former names `template` and `register` still work for one release and print a notice naming their replacement. `template` was `multi-eval`. `register` was `single-eval` with `eval-yaml-required = false`, which is not a property of the layout but of the destination: the inspect_evals register entry holds the metadata. A repository that relies on that sets `eval-yaml-required = false` in its own table, and the register lint service passes it explicitly (`REGISTER_CONFIG` in the Python API). A fuller table:

```toml
[tool.inspect-evals-lint]
preset = "monorepo"
ignore = ["IETS004"]                                   # by name, code or prefix
exclude = ["src/inspect_evals/*/challenges/**"]         # sandbox code: never parsed
per-file-ignores = { "src/inspect_evals/*/data/**" = ["IEBP003"] }

[tool.inspect-evals-lint.allowlists.model_role_resolution]
moru = ["grader"]                                       # package = [keys the rule names]

[tool.inspect-evals-lint.allowlists.sandbox_image_pinning]
cybench = ["example/untagged"]

[tool.inspect-evals-lint.dockerfile_locking]                # a rule's own options
host-lock-coupling = "allow"
```

An allowlisted finding is reported as a warning with its message prefixed `Allowlisted:`, and an entry that no finding matches is itself a warning at `pyproject.toml`, so the list can only shrink. A table named after a rule holds that rule's options, merged over the preset's defaults key by key; each rule's page lists what it accepts. Removed keys (`disabled-checks`, `non-eval-dirs`, `sandbox-image-allowlist`, `model-role-allowlist`) are rejected with a message naming their replacement.

Only Python packages are linted: a sub-directory of `source-root` without an `__init__.py` (a README left behind after a move, a data directory) is skipped by `--all` and reported as a skip when named directly, so it needs no `ignore-dirs` entry. `ignore-dirs` is for packages you really do not want checked.

### Helper packages

Shared code that evaluations import, such as inspect_evals' `utils` package, is not an evaluation but does most of the same things: it grades, it resolves models, it imports third-party packages. Directories listed in `helper-dirs` are linted with the checks that guard that behaviour (`private_api_imports`, `score_constants`, `unscored_reason`, `get_model_location`, `model_role_resolution`, `sample_ids`, `task_overridable_defaults`, `sandbox_image_pinning`, `external_dependencies`, `tests_init` and the `custom_*_tests` checks) and not with the ones about an evaluation's structure and registration (`main_file`, `init_exports`, `readme`, `registry`, `eval_yaml`, `tests_exist`, `e2e_test`, `record_to_sample_test`). Two checks adapt: `external_dependencies` requires a helper's module-level third-party imports to be in `[project].dependencies`, since every evaluation that imports the helper loads them, while imports inside a function, a `try` block or an `if TYPE_CHECKING:` block only need declaring in some group or isolated package; and the `custom_*_tests` checks look for a helper's tests anywhere under `tests-root`, not only in `tests/<name>/`. Allowlists are keyed by package name, so `utils = ["grader"]` under `allowlists.model_role_resolution` works for a helper too. A failing helper check fails the run like any other.

An upstream repository listed in the [inspect_evals register](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/register/README.md) is usually `single-eval`. Run it from outside the repo with `inspect-evals-lint --root <clone> --preset single-eval --all --output-format json`; the register lint service does the same through `lint_task_files`, which also leaves `eval.yaml` optional because the register entry holds the metadata.

## For agents and tools

The documentation site is built to be read by machines as well as people:

- [`llms.txt`](https://inspect-evals-lint.generality.org/llms.txt) is an index of the whole site in the [llms.txt](https://llmstxt.org) convention; [`llms-full.txt`](https://inspect-evals-lint.generality.org/llms-full.txt) is every page concatenated.
- Every page is also served as raw Markdown at the same path with a `.md` suffix, for example [`rules/IEBP002.md`](https://inspect-evals-lint.generality.org/rules/IEBP002.md).
- [`rules.json`](https://inspect-evals-lint.generality.org/rules.json) lists every rule with its code, name, category, scopes, summary and page URLs.
- Locally, `inspect-evals-lint --explain <code>` prints the same page text, `--list-rules --output-format json` lists the rules, and `--output-format json` gives findings with file, line, column and a hint ([schema](output.md)).

## Suppressing a finding

Every finding has a file and, where it is in a file's contents, a line, so one comment syntax covers every rule:

- Line: `# inspect-evals-lint: ignore[IEBP003]` on the offending line, or on any line of a multi-line statement (formatters move trailing comments inside parenthesised imports). Names, codes and code prefixes are accepted, comma-separated. In a Dockerfile, whose instructions take no trailing comment, put it on the line above the instruction.
- File: `# inspect-evals-lint: ignore-file[IEBP003]` within the first ten lines of the file.
- Paths: `per-file-ignores` in the configuration table, for whole directories.
- Not linted at all: `exclude`, for code that is shipped into a sandbox rather than run on the host.

The comment is namespaced with the tool's name rather than reusing ruff's `# noqa`: ruff reads every `# noqa` comment and warns about codes it does not know, so a shared spelling would turn each suppression into a ruff warning. Suppressed findings still appear in reports, marked `[suppressed]`, and count as passing. A marker the linter does not read suppresses nothing and is reported as a warning by `suppression_syntax` (IECQ005) with the replacement in its hint: the former `noautolint` comments and `.noautolint` files, a bare `ignore` or `ignore[]`, an `ignore-file` past the header, or a selector that names no rule.

## Checks

[docs/CHECKS.md](CHECKS.md) lists every rule by category and links to a page per rule under [docs/rules/](CHECKS.md), generated from each rule's docstring by `python -m inspect_evals_lint.docs`; `inspect-evals-lint --explain <code>` prints the same text. Each rule is a decorated function in `src/inspect_evals_lint/rules/`: adding one means adding the function, its docstring, its `references` (the [Inspect documentation](https://inspect.aisi.org.uk/) sections that explain the convention, shown under *See also* on the rule's page and listed in `rules.json`; a rule with no upstream home for its convention has none) and its tests, and regenerating the docs (pre-commit checks they are current; the docs workflow also checks every reference resolves).

## Python API

```python
from pathlib import Path
from inspect_evals_lint import lint_repository, load_config

root = Path(".")
run = lint_repository(root, load_config(root))      # RunReport: every evaluation and helper package
for package in run.packages:
    print(package.name, package.kind, package.passed(), package.summary())
    for d in package.diagnostics:
        print(f"  {d.rule.code} {d.location}: {d.message}")
```

`lint_package(root, name, config)` lints one package and returns a `PackageReport`. `lint_task_files(root, ["src/castle/castle.py"])` lints the packages holding task files (one per distinct layout, see `task_layout`) and raises `UnsupportedLayoutError` for a bare module. `report.score()` and `run.score()` give a `Score`: every rule counted once at the worst status it reported (`fail`, `warn`, `suppressed`, `pass`, `skip` in that order), with `passing` (pass + warn), `applicable` (everything but skip) and a per-category split, the numbers the register badges show. `render_markdown(run, source_link=...)` produces the Markdown summary, with locations linked wherever `source_link(path, line)` returns a URL. `rules()` lists every registered `Rule`; `get_rule("IEBP002")` or `get_rule("model_role_resolution")` looks one up. `run.to_dict()` is the JSON document described in [docs/output.md](output.md).

## Development

```bash
uv sync
uv run pre-commit install   # optional: run the lint stack on every commit
uv run pytest                 # per-rule tests live under tests/rules/
uv run basedpyright src
```

`docs/CHECKS.md`, `docs/rules/`, `docs/index.md` and the configuration table above are generated by `uv run python -m inspect_evals_lint.docs`; pre-commit fails if they are out of date. `uv run --group docs mkdocs serve` previews the site, which `docs.yml` publishes from main. Linting (ruff, [zizmor](https://docs.zizmor.sh/), mdformat) runs via [pre-commit](https://pre-commit.com); CI runs the same stack plus basedpyright and pytest via the shared [`python-ci`](https://github.com/Generality-Labs/python-project-template) reusable workflow.

## Releasing

See [RELEASING.md](https://github.com/Generality-Labs/inspect-evals-lint/blob/main/RELEASING.md).


---

# Rule index

Every rule has a code (`IEFS`, `IECQ`, `IETS`, `IEBP`, `IESC` prefixes for the categories below) and a name; either is accepted by `--select`, `--ignore` and in suppression comments, and a prefix selects a whole category. A rule reports one diagnostic per site it finds something wrong at, each with a file and, where the finding is in a file's contents, a line and column. A rule with nothing to point at reports `pass`, or `skip` with the reason. Diagnostics are `fail` or `warn`; only `fail` makes the run exit non-zero. `inspect-evals-lint --explain <code>` prints the same text as a rule's page.

## File structure

| Code                        | Rule               | Applies to   | Summary                                                                |
| --------------------------- | ------------------ | ------------ | ---------------------------------------------------------------------- |
| [IEFS001](rules/IEFS001.md) | `package_location` | eval, helper | The package exists at <source-root>/<name>/ with an __init__.py        |
| [IEFS002](rules/IEFS002.md) | `main_file`        | eval         | Some module defines a @task function, preferably <name>.py or tasks.py |
| [IEFS003](rules/IEFS003.md) | `init_exports`     | eval         | __init__.py exports every @task function in the package                |
| [IEFS004](rules/IEFS004.md) | `registry`         | eval         | The evaluation is registered so inspect eval can find its tasks        |
| [IEFS005](rules/IEFS005.md) | `eval_yaml`        | eval         | eval.yaml exists, is a mapping, and defines the required fields        |
| [IEFS006](rules/IEFS006.md) | `readme`           | eval         | README.md exists and has no TODO markers                               |

## Code quality

| Code                        | Rule                    | Applies to   | Summary                                                                               |
| --------------------------- | ----------------------- | ------------ | ------------------------------------------------------------------------------------- |
| [IECQ001](rules/IECQ001.md) | `private_api_imports`   | eval, helper | No imports from private inspect_ai modules                                            |
| [IECQ002](rules/IECQ002.md) | `score_constants`       | eval, helper | Score() values use the CORRECT/INCORRECT constants, not string literals               |
| [IECQ003](rules/IECQ003.md) | `unscored_reason`       | eval, helper | Score.unscored() passes a reason= and the legacy unscored_reason metadata key is gone |
| [IECQ004](rules/IECQ004.md) | `external_dependencies` | eval, helper | Third-party imports are declared in pyproject.toml                                    |
| [IECQ005](rules/IECQ005.md) | `suppression_syntax`    | eval, helper | Every suppression marker is one the linter reads                                      |

## Tests

| Code                        | Rule                    | Applies to   | Summary                                                              |
| --------------------------- | ----------------------- | ------------ | -------------------------------------------------------------------- |
| [IETS001](rules/IETS001.md) | `tests_exist`           | eval         | A test directory exists for the evaluation                           |
| [IETS002](rules/IETS002.md) | `tests_init`            | eval, helper | The test directory and its sub-directories contain __init__.py       |
| [IETS003](rules/IETS003.md) | `e2e_test`              | eval         | Some test runs eval() against a mockllm/ model                       |
| [IETS004](rules/IETS004.md) | `record_to_sample_test` | eval         | record_to_sample is exercised by a test when the evaluation uses it  |
| [IETS005](rules/IETS005.md) | `custom_solver_tests`   | eval, helper | Every @solver or @agent function name appears somewhere in the tests |
| [IETS006](rules/IETS006.md) | `custom_scorer_tests`   | eval, helper | Every @scorer function name appears somewhere in the tests           |
| [IETS007](rules/IETS007.md) | `custom_tool_tests`     | eval, helper | Every @tool function name appears somewhere in the tests             |

## Best practices

| Code                        | Rule                            | Applies to   | Summary                                                                                             |
| --------------------------- | ------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- |
| [IEBP001](rules/IEBP001.md) | `get_model_location`            | eval, helper | get_model() is only called inside @solver, @scorer or @agent functions                              |
| [IEBP002](rules/IEBP002.md) | `model_role_resolution`         | eval, helper | get_model(role=...) resolves deliberately: an explicit model, default= or required=True             |
| [IEBP003](rules/IEBP003.md) | `sample_ids`                    | eval, helper | Every Sample() passes id=                                                                           |
| [IEBP004](rules/IEBP004.md) | `task_overridable_defaults`     | eval, helper | @task parameters naming a solver, scorer, metric, grader or model have defaults                     |
| [IEBP005](rules/IEBP005.md) | `sandbox_image_pinning`         | eval, helper | Registry images in compose files use an immutable tag or digest                                     |
| [IEBP006](rules/IEBP006.md) | `gpu_sandbox_check`             | eval         | An evaluation requiring a GPU ships a maintenance sandbox check task                                |
| [IEBP007](rules/IEBP007.md) | `dockerfile_locking`            | eval         | Dockerfile builds consume locked inputs: committed locks, digest-pinned images, fixed sources       |
| [IEBP008](rules/IEBP008.md) | `duplicate_filter_acknowledged` | eval, helper | Every filter_duplicate_ids() call states how many duplicates it drops and links the upstream report |
| [IEBP009](rules/IEBP009.md) | `known_broken_reported`         | eval, helper | Every drop_known_broken() entry maps a sample id to the URL of its upstream report                  |

## Security

| Code                        | Rule                 | Applies to   | Summary                                                                    |
| --------------------------- | -------------------- | ------------ | -------------------------------------------------------------------------- |
| [IESC001](rules/IESC001.md) | `sandbox_privileges` | eval, helper | Compose services do not grant additional sandbox privileges or host access |

## Helper packages

Directories listed in `helper-dirs` (by default `utils`) hold code that evaluations import rather than an evaluation. They run the rules about behaviour and not the ones about an evaluation's structure and registration. Run: `package_location`, `private_api_imports`, `score_constants`, `unscored_reason`, `external_dependencies`, `suppression_syntax`, `tests_init`, `custom_solver_tests`, `custom_scorer_tests`, `custom_tool_tests`, `get_model_location`, `model_role_resolution`, `sample_ids`, `task_overridable_defaults`, `sandbox_image_pinning`, `duplicate_filter_acknowledged`, `known_broken_reported`, `sandbox_privileges`. Not run: `main_file`, `init_exports`, `registry`, `eval_yaml`, `readme`, `tests_exist`, `e2e_test`, `record_to_sample_test`, `gpu_sandbox_check`, `dockerfile_locking`. Every rule declares its scopes in its `@rule` decorator, so a new rule decides up front whether shared code is in scope.

## Categories

Each rule belongs to one category. The JSON output and `registry.CATEGORIES` expose this set; changing it requires a JSON schema version update and coordinated consumer changes. Schema version 2 adds `security`. Existing rule codes and categories are unchanged.

- File structure covers package layout, registration, and metadata.
- Code quality covers Python API conventions and dependency declarations.
- Tests covers the presence of tests for evaluation components and execution.
- Best practices covers evaluation configuration, dataset handling, and reproducibility. Image pinning and dependency locking remain here because their primary check is whether inputs are fixed.
- Security covers isolation and access to the host, other containers, or execution capabilities. Runtime-user and build-isolation checks would also belong here as separate rules.

## Suppression

- Line: `# inspect-evals-lint: ignore[<rule>]` on the offending line, or on any line of a multi-line statement; names, codes and code prefixes, comma-separated. In a Dockerfile, on the line above the instruction. In a Compose file, on the setting's line or the list item's line.
- File: `# inspect-evals-lint: ignore-file[<rule>]` within the first ten lines.
- Paths: `per-file-ignores = { "<glob>" = ["<rule>"] }` in `[tool.inspect-evals-lint]`.
- Never read: `exclude = ["<glob>"]` keeps files out of the AST-based rules entirely, for code shipped into a sandbox.

Suppressed findings still appear in reports, marked `[suppressed]`, and count as passing. A marker the linter does not read (the former `noautolint` comments and `.noautolint` files, `ignore` without a rule list, a selector naming no rule) suppresses nothing and is reported by `suppression_syntax` with the replacement in its hint.


---

# Output formats

`--output-format text` (the default) prints a report per package and, with more than one package, an overall summary, a per-rule compliance table and the failures grouped by rule. The other three formats are described here. Under each, progress messages go to stderr and stdout carries only the document, so the output can be redirected or piped.

## JSON (`--output-format json`)

One document per run. `RunReport.to_dict()` produces the same mapping from the Python API.

```json
{
  "schema_version": 2,
  "version": "0.3.0",
  "root": "/abs/path/to/repo",
  "passed": false,
  "summary": {"pass": 40, "fail": 2, "warn": 1, "skip": 6, "suppressed": 0},
  "score": {"pass": 38, "fail": 2, "warn": 1, "skip": 6, "suppressed": 0, "applicable": 41, "passing": 39, "score": 0.9512,
            "by_category": {"best_practices": {"pass": 8, "fail": 2, "warn": 1, "skip": 2, "suppressed": 0, "applicable": 11, "passing": 9, "score": 0.8182}}},
  "packages": [
    {
      "name": "gpqa",
      "kind": "eval",
      "passed": false,
      "skipped": null,
      "summary": {"pass": 20, "fail": 2, "warn": 0, "skip": 3, "suppressed": 0},
      "score": {"pass": 19, "fail": 1, "warn": 0, "skip": 3, "suppressed": 0, "applicable": 20, "passing": 19, "score": 0.95, "by_category": {"...": "..."}},
      "rules": [
        {"rule": "package_location", "code": "IEFS001", "category": "file_structure", "status": "pass", "diagnostics": []},
        {"rule": "sample_ids", "code": "IEBP003", "category": "best_practices", "status": "fail", "diagnostics": [0]}
      ],
      "outcomes": [
        {"rule": "package_location", "code": "IEFS001", "category": "file_structure", "status": "pass", "message": "Package located at src/inspect_evals/gpqa"}
      ],
      "diagnostics": [
        {
          "rule": "sample_ids",
          "code": "IEBP003",
          "category": "best_practices",
          "severity": "error",
          "status": "fail",
          "message": "Sample() call without id=",
          "file": "src/inspect_evals/gpqa/gpqa.py",
          "line": 42,
          "column": 12,
          "hint": "pass a stable id= so the sample survives shuffles and reruns"
        }
      ]
    }
  ]
}
```

| Field                      | Meaning                                                                                                                                                                                                                                                                                                                                                                               |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schema_version`           | Integer, bumped for incompatible output changes, including the category set. Check it first.                                                                                                                                                                                                                                                                                          |
| `version`                  | The inspect-evals-lint version that produced the document.                                                                                                                                                                                                                                                                                                                            |
| `root`                     | Absolute repository root the run used. Every `file` below is relative to it when it falls under it.                                                                                                                                                                                                                                                                                   |
| `passed`                   | `true` when no diagnostic in any package has status `fail`. Mirrors the exit code.                                                                                                                                                                                                                                                                                                    |
| `summary`                  | Counts of every outcome and diagnostic status across the run.                                                                                                                                                                                                                                                                                                                         |
| `score`                    | Rules met out of rules applicable. Every rule that ran counts once, at the worst status it reported (`fail`, `warn`, `suppressed`, `pass`, `skip` in that order). `passing` is `pass` + `warn`; `applicable` is everything but `skip`; `score` is their ratio, or `null` when nothing applied; `by_category` repeats the counts per category. The register badges show these numbers. |
| `packages[].kind`          | `eval` or `helper`. Filter on this rather than expecting separate lists.                                                                                                                                                                                                                                                                                                              |
| `packages[].skipped`       | Why the package was not linted at all (listed in `ignore-dirs`), else `null`.                                                                                                                                                                                                                                                                                                         |
| `packages[].score`         | The package's own score, in the same shape as the run's.                                                                                                                                                                                                                                                                                                                              |
| `packages[].rules[]`       | One entry per rule that ran, in execution order, at the worst status it reported (`fail`, `warn`, `suppressed`, `pass`, `skip`). `diagnostics` is a list of indexes into this package's `diagnostics` array; a pass or skip with no findings has `[]`. Counting these by `status` reproduces `packages[].score`. This field was added in schema version 1 and remains in version 2.   |
| `packages[].outcomes[]`    | One per rule that ran and had nothing to point at: `status` is `pass` or `skip`, `message` says why.                                                                                                                                                                                                                                                                                  |
| `packages[].diagnostics[]` | One per finding. `severity` is `error` or `warning`; `status` is `fail`, `warn` or `suppressed`. `line` and `column` are 1-based and `null` when the finding is about a file or directory as a whole. `hint` is what to do about it, or `null`.                                                                                                                                       |

`category` is one of `file_structure`, `code_quality`, `tests`, `best_practices`, and `security`. Schema version 2 adds `security` and the `IESC` rule prefix. Existing codes, categories, fields, and score calculations are unchanged. Consumers must accept both versions during migration, add the category to validation and displays, and accept `IESC` codes. Adding or changing categories requires a schema version update.

Update the register publisher and documentation dashboard before deploying a linter release that emits version 2. The register service wraps lint reports in its own result and summary formats; their schema versions are independent of the nested lint report version.

## GitHub Actions (`--output-format github`)

One [workflow command](https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions) per failing or warning diagnostic, so a lint step annotates the pull request at the right line, followed by a one-line summary:

```
::error file=src/inspect_evals/gpqa/gpqa.py,line=42,col=12,title=IEBP003 sample_ids::src/inspect_evals/gpqa/gpqa.py:42:12 IEBP003 sample_ids: Sample() call without id=; pass a stable id= so the sample survives shuffles and reruns
::warning file=pyproject.toml,title=IEBP005 sandbox_image_pinning::pyproject.toml IEBP005 sandbox_image_pinning: Allowlist entry 'x/y' for sandbox_image_pinning on 'cybench' is no longer needed; remove it from [tool.inspect-evals-lint.allowlists.sandbox_image_pinning]
inspect-evals-lint: 128/130 packages passed; 2 failed, 1 warnings, 0 suppressed
```

The message repeats the location and the rule because that is all the job log shows: GitHub renders a command as `##[error]<message>` and moves `file=` and `line=` to the annotations panel, which records at most ten errors and ten warnings per step. When a run exceeds that, a line after the summary says so. Skips, passes and suppressed findings produce no annotation. Property values and messages are percent-escaped the way the workflow-command syntax requires.

In a GitHub Actions job, where `GITHUB_STEP_SUMMARY` names the job summary file, the same run also appends the [Markdown](#markdown-output-format-markdown) summary to it under an `## inspect-evals-lint` heading, so every finding is readable with its location and hint without opening the log. Set the variable to a file path to get the same outside Actions; unset it to skip.

## Markdown (`--output-format markdown`)

A summary shaped for a pull request comment or a job summary. Per package: a heading, a headline of rules met with the per-category split, then a collapsible list of the rules not met, with warnings or suppressed, each finding on its own line with its location and the hint beneath. Rules that passed or did not apply are counted but not listed. One closing line says how the counts work.

```markdown
### `gpqa`

**19/20 checks met** · Structure 6/6 · Code quality 4/4 · Tests 6/6 · Best practices 3/4

<details>
<summary>1 rule(s) not met, with warnings or suppressed</summary>

- **Not met** [`sample_ids`](https://inspect-evals-lint.generality.org/rules/IEBP003/)
  - `src/inspect_evals/gpqa/gpqa.py:42` Sample() call without id=<br>  Hint: pass a stable id= so the sample survives shuffles and reruns

</details>
```

Locations are code spans from the CLI. From the Python API, `render_markdown(run, source_link=...)` links each location wherever the callback returns a URL, for example into the repository at the linted commit, and `docs_base` redirects the rule links. Messages, hints and paths from the linted repository are escaped so they cannot open fences or break tables.


---

# IEFS001: package_location

The package exists at <source-root>/<name>/ with an __init__.py.

**Category:** file_structure · **Applies to:** eval, helper

## What it does

Checks that the directory named on the command line, or found under `source-root`, is a Python package. Every other rule depends on this one and does not run when it does not hold.

## Why is this bad?

A directory that exists but has no `__init__.py` is documentation or data, not code: a README left where evaluations used to live, a fixtures folder. It is reported as a skip rather than a failure, and the same test keeps it out of `--all` discovery, so it needs no configuration.

## Example

```text
src/inspect_evals/gdm_capabilities/README.md     # skipped: not a package
src/inspect_evals/gpqa/__init__.py               # linted
```

## See also

- [Tasks: Packaging](https://inspect.aisi.org.uk/tasks.html#packaging)
- [Extensions: Components: Registration](https://inspect.aisi.org.uk/extensions-components.html#registration)

Suppress on a line with `# inspect-evals-lint: ignore[IEFS001]` or `ignore[package_location]`; select or ignore it in configuration by either, or by the prefix `IEFS`.


---

# IEFS002: main_file

Some module defines a @task function, preferably <name>.py or tasks.py.

**Category:** file_structure · **Applies to:** eval

## What it does

Reads every Python file in the package for `@task` functions. Passes when `<name>.py` or `tasks.py` defines one, preferring whichever does so a stray empty `<name>.py` does not hide the tasks in `tasks.py`. Warns, once per module, when the tasks live only in other modules. Fails when no module defines a task, or when the conventional file exists but does not parse.

## Why is this bad?

A package with no task is not an evaluation, whatever else it contains. Keeping the tasks in a predictably named module is a lesser matter, so it is a warning: it lets readers find them without opening every file, but a package that names its task module after the benchmark still runs.

## Example

```text
src/my_eval/my_eval.py     # defines @task my_eval()
src/my_eval/tasks.py       # accepted alternative
src/my_eval/v8.py          # defines the tasks: warned, not failed
```

## See also

- [Tasks: Task Basics](https://inspect.aisi.org.uk/tasks.html#task-basics)
- [Extensions: Components: Registration](https://inspect.aisi.org.uk/extensions-components.html#registration)

Suppress on a line with `# inspect-evals-lint: ignore[IEFS002]` or `ignore[main_file]`; select or ignore it in configuration by either, or by the prefix `IEFS`.


---

# IEFS003: init_exports

__init__.py exports every @task function in the package.

**Category:** file_structure · **Applies to:** eval

## What it does

Reads the task functions from every module that defines one and checks each name appears in `__init__.py`, either in `__all__` or imported with `from ... import`. One diagnostic per missing task, with the import to add.

## Why is this bad?

`inspect eval my_eval/task` resolves tasks through the package, so a task the package does not export is a task nobody can run by name.

## Example

```python
# __init__.py
from .my_eval import my_eval, my_eval_hard

__all__ = ["my_eval", "my_eval_hard"]
```

## See also

- [Extensions: Components: Registration](https://inspect.aisi.org.uk/extensions-components.html#registration)

Suppress on a line with `# inspect-evals-lint: ignore[IEFS003]` or `ignore[init_exports]`; select or ignore it in configuration by either, or by the prefix `IEFS`.


---

# IEFS004: registry

The evaluation is registered so inspect eval can find its tasks.

**Category:** file_structure · **Applies to:** eval

## What it does

With `registry = "entry-points"` (the template layout) the package or its module must appear under `[project.entry-points.inspect_ai]` in `pyproject.toml`. With `registry = "module"` (the inspect_evals monorepo) the registry module must import it. `registry = "none"` skips the rule.

## Why is this bad?

An unregistered evaluation runs from a file path in development and then cannot be found by name anywhere else.

## Example

```toml
[project.entry-points.inspect_ai]
my_eval = "my_eval"
```

## Options

- `registry`
- `registry-module`

## See also

- [Extensions: Components: Registration](https://inspect.aisi.org.uk/extensions-components.html#registration)
- [Extensions: Components: Tasks](https://inspect.aisi.org.uk/extensions-components.html#tasks)

Suppress on a line with `# inspect-evals-lint: ignore[IEFS004]` or `ignore[registry]`; select or ignore it in configuration by either, or by the prefix `IEFS`.


---

# IEFS005: eval_yaml

eval.yaml exists, is a mapping, and defines the required fields.

**Category:** file_structure · **Applies to:** eval

## What it does

Parses `eval.yaml` in the package and reports one diagnostic per required field that is missing, or one for a file that is not valid YAML or not a mapping. With `eval-yaml-required = false` a missing file is a skip, for repositories whose metadata lives in the inspect_evals register; a present file is still validated.

## Why is this bad?

`eval.yaml` is what listings, the register and the README generator read. A missing field there is a missing field everywhere downstream.

## Example

```yaml
title: GPQA
description: Graduate-level science questions.
group: Knowledge
contributors: [someone]
tasks:
  - name: gpqa_diamond
```

## Options

- `eval-yaml-required`
- `eval-yaml-required-fields`

Suppress on a line with `# inspect-evals-lint: ignore[IEFS005]` or `ignore[eval_yaml]`; select or ignore it in configuration by either, or by the prefix `IEFS`.


---

# IEFS006: readme

README.md exists and has no TODO markers.

**Category:** file_structure · **Applies to:** eval

## What it does

Checks the package has a `README.md`; with `readme-location = "repo-root"` the repository's top-level README is accepted when the package has none. Warns once per line that still contains `TODO:`.

## Why is this bad?

The README is the evaluation's front door. A missing one leaves users guessing at what the evaluation measures; a leftover `TODO:` is a section the author meant to write.

## Options

- `readme-location`

Suppress on a line with `# inspect-evals-lint: ignore[IEFS006]` or `ignore[readme]`; select or ignore it in configuration by either, or by the prefix `IEFS`.


---

# IECQ001: private_api_imports

No imports from private inspect_ai modules.

**Category:** code_quality · **Applies to:** eval, helper

## What it does

Flags `from inspect_ai.<...>._<name> import ...` wherever a dotted segment starts with an underscore. One diagnostic per import.

## Why is this bad?

Private modules change without notice. An evaluation importing one breaks on the next `inspect_ai` release with no deprecation period.

## Example

```python
from inspect_ai.scorer._metric import Score   # private
```

Use instead:

```python
from inspect_ai.scorer import Score
```

Suppress on a line with `# inspect-evals-lint: ignore[IECQ001]` or `ignore[private_api_imports]`; select or ignore it in configuration by either, or by the prefix `IECQ`.


---

# IECQ002: score_constants

Score() values use the CORRECT/INCORRECT constants, not string literals.

**Category:** code_quality · **Applies to:** eval, helper

## What it does

Flags `Score(value="C")` and the other literals `"I"`, `"CORRECT"` and `"INCORRECT"`. One diagnostic per call.

## Why is this bad?

`inspect_ai`'s metrics compare against the constants. A literal that drifts from them, or that `inspect_ai` later changes, scores silently wrong.

## Example

```python
return Score(value="C")
```

Use instead:

```python
from inspect_ai.scorer import CORRECT

return Score(value=CORRECT)
```

## See also

- [Custom Scorers: Score](https://inspect.aisi.org.uk/custom-scorers.html#score)
- [Custom Scorers: Score Value](https://inspect.aisi.org.uk/custom-scorers.html#score-value)

Suppress on a line with `# inspect-evals-lint: ignore[IECQ002]` or `ignore[score_constants]`; select or ignore it in configuration by either, or by the prefix `IECQ`.


---

# IECQ003: unscored_reason

Score.unscored() passes a reason= and the legacy unscored_reason metadata key is gone.

**Category:** code_quality · **Applies to:** eval, helper

## What it does

Flags `Score.unscored(...)` calls whose `reason=` is missing, `None` or empty, and any occurrence of the string `"unscored_reason"` outside a docstring. Only attribute calls count, so a locally defined metric named `unscored()` is not mistaken for the constructor. One diagnostic per site.

## Why is this bad?

`Score.reason` (inspect_ai 0.3.261) is the first-class record of why a sample was left unscored; metrics and log tooling read it there. The interim `metadata["unscored_reason"]` convention is superseded, and an unscored sample with no reason cannot be told apart from one the scorer forgot.

## Example

```python
return Score.unscored(explanation="grader returned nothing")
```

Use instead:

```python
return Score.unscored(reason="grader_failed", explanation="grader returned nothing")
```

## See also

- [Custom Scorers: Unscored Samples](https://inspect.aisi.org.uk/custom-scorers.html#unscored-samples)
- [Scoring Policy: Recording the reason](https://inspect.aisi.org.uk/scoring-policy.html#recording-the-reason)

Suppress on a line with `# inspect-evals-lint: ignore[IECQ003]` or `ignore[unscored_reason]`; select or ignore it in configuration by either, or by the prefix `IECQ`.


---

# IECQ004: external_dependencies

Third-party imports are declared in pyproject.toml.

**Category:** code_quality · **Applies to:** eval, helper

## What it does

Collects every import in the package and treats one as external when it is not in the standard library, not in `[project].dependencies` or something those dependencies require in turn, not local to the package and not one of the repository's own packages (the `import-prefix` package, and every package under `source-root` such as a `utils` helper). The transitive requirements are read from `uv.lock` when the repository commits one, else from the distributions installed in the current environment; with neither, only the declared names count and the hint on an undeclared import says so. For an evaluation, each external import must be declared in some `[project.optional-dependencies]` group or `[dependency-groups]` entry (other than `dev`), or in the isolated package's `pyproject.toml` when `isolated-packages-dir` is set. With `per-eval-dependency-group = true` (the `monorepo` preset) the evaluation must also own a group named after itself unless it is isolated, so one evaluation's dependencies can be installed without the rest; a standalone repository declares its dependencies in `[project].dependencies` and any extra it likes.

For a helper package the rule is different, because every evaluation that imports the helper loads whatever it imports at module level: those imports must be in `[project].dependencies`, while imports inside a function, a `try` block or an `if TYPE_CHECKING:` block are deferred and only need to be declared in some group or in any isolated package. Helpers need no group of their own. One diagnostic per import, at its first site.

Import-to-distribution mapping uses the packages installed in the current environment plus a few static aliases, so run the linter inside the project's environment. Names are compared in PEP 503 normalised form.

## Why is this bad?

An import nobody declared works on the author's machine and fails for the next person with `ModuleNotFoundError`, often only when a particular sample runs.

## Example

A standalone repository:

```toml
[project]
dependencies = ["inspect_ai", "datasets>=4.0"]

[project.optional-dependencies]
modal = ["inspect_sandboxes"]
```

The inspect_evals monorepo, with `per-eval-dependency-group = true`:

```toml
[project.optional-dependencies]
my_eval = ["datasets>=4.0", "scikit-learn"]
```

## Options

- `per-eval-dependency-group`
- `isolated-packages-dir`
- `import-prefix`

## See also

- [Tasks: Packaging](https://inspect.aisi.org.uk/tasks.html#packaging)

Suppress on a line with `# inspect-evals-lint: ignore[IECQ004]` or `ignore[external_dependencies]`; select or ignore it in configuration by either, or by the prefix `IECQ`.


---

# IECQ005: suppression_syntax

Every suppression marker is one the linter reads.

**Category:** code_quality · **Applies to:** eval, helper

## What it does

Reads the `# inspect-evals-lint: ignore[...]` comments in the package's Python files and Dockerfiles (`exclude`d files are skipped) and Compose files, and warns about each marker that suppresses nothing: a comment in the removed `# noautolint` syntax or a `.noautolint` file, an `ignore` or `ignore-file` without a bracketed rule list, an `ignore-file` past the first ten lines, and a selector that names no rule. One warning per marker, at its line. The other selectors in the same comment still apply.

## Why is this bad?

A marker the linter does not read does nothing, silently: the finding it was meant to cover is reported under its own rule while the reader of the code believes it is handled. Earlier releases stopped with a configuration error instead, which lost every other result for the package, so a repository linted by a third party (the register lint service) had no results at all until it migrated.

## Example

```python
from inspect_ai.model._model import thing  # noautolint: private_api_imports
```

Use instead:

```python
from inspect_ai.model._model import thing  # inspect-evals-lint: ignore[private_api_imports]
```

Suppress on a line with `# inspect-evals-lint: ignore[IECQ005]` or `ignore[suppression_syntax]`; select or ignore it in configuration by either, or by the prefix `IECQ`.


---

# IETS001: tests_exist

A test directory exists for the evaluation.

**Category:** tests · **Applies to:** eval

## What it does

Looks for `<tests-root>/<name>/`. With `tests-layout = "flat"` test files directly under `<tests-root>/` are accepted when that directory is absent, as single-evaluation repositories usually have.

## Why is this bad?

An evaluation with no tests at all has never been run against the mock model, so nothing guards its wiring.

## Options

- `tests-root`
- `tests-layout`

Suppress on a line with `# inspect-evals-lint: ignore[IETS001]` or `ignore[tests_exist]`; select or ignore it in configuration by either, or by the prefix `IETS`.


---

# IETS002: tests_init

The test directory and its sub-directories contain __init__.py.

**Category:** tests · **Applies to:** eval, helper

## What it does

Checks `<tests-root>/<name>/` and every directory beneath it for an `__init__.py`, ignoring cache directories. One diagnostic per directory. Skipped with `tests-layout = "flat"`, the single-evaluation layout, whether the tests sit directly under the tests root or in a `tests/<name>/` directory of their own: with one evaluation there is no second test tree to collide with. Also skipped for a helper package with no `tests/<name>/` directory.

## Why is this bad?

Per-evaluation test trees with duplicate module basenames (`test_scorer.py` in two evaluations) collide during pytest collection unless each tree is a package.

## Options

- `tests-layout`

Suppress on a line with `# inspect-evals-lint: ignore[IETS002]` or `ignore[tests_init]`; select or ignore it in configuration by either, or by the prefix `IETS`.


---

# IETS003: e2e_test

Some test runs eval() against a mockllm/ model.

**Category:** tests · **Applies to:** eval

## What it does

Looks through the test directory for a file that calls `eval()` or `eval_async()` (or an alias imported from `inspect_ai`) and mentions a model under the `mockllm/` provider. `mockllm/model` is the usual name, but any name after the prefix is the same mock, and a test that scripts `custom_outputs` per case commonly names each one (`mockllm/epochs`).

## Why is this bad?

An end-to-end run against the mock model catches wiring mistakes, a dataset that no longer loads, a solver that does not compose, without spending tokens.

## Example

```python
from inspect_ai import eval

def test_e2e():
    logs = eval(my_eval(), model="mockllm/model", limit=2)
    assert logs[0].status == "success"
```

## See also

- [Reference: eval()](https://inspect.aisi.org.uk/reference/inspect_ai.html#eval)

Suppress on a line with `# inspect-evals-lint: ignore[IETS003]` or `ignore[e2e_test]`; select or ignore it in configuration by either, or by the prefix `IETS`.


---

# IETS004: record_to_sample_test

record_to_sample is exercised by a test when the evaluation uses it.

**Category:** tests · **Applies to:** eval

## What it does

When any file in the package mentions `record_to_sample`, some test file must mention it too. Skipped when the evaluation has none.

## Why is this bad?

`record_to_sample` is where a dataset's fields become an evaluation's inputs and targets. A field renamed upstream fails silently there unless a test pins a real record.

## See also

- [Datasets: Field Mapping](https://inspect.aisi.org.uk/datasets.html#field-mapping)

Suppress on a line with `# inspect-evals-lint: ignore[IETS004]` or `ignore[record_to_sample_test]`; select or ignore it in configuration by either, or by the prefix `IETS`.


---

# IETS005: custom_solver_tests

Every @solver or @agent function name appears somewhere in the tests.

**Category:** tests · **Applies to:** eval, helper

## What it does

Finds functions decorated with `@solver` or `@agent` in the package and checks each name appears in a test file. An agent is the solver of a sandboxed evaluation, so it is held to the same standard. For an evaluation the search covers `tests/<name>/`; for a helper package, the whole tests root, because shared components are usually tested next to the evaluation that motivated them. One diagnostic per untested function. This is a presence check, not a quality check.

## Why is this bad?

A custom solver or agent is the evaluation's own logic, the part no upstream test covers. One test that at least constructs it catches import errors and signature changes.

## See also

- [Solvers: Custom Solvers](https://inspect.aisi.org.uk/solvers.html#custom-solvers)
- [Custom Agents](https://inspect.aisi.org.uk/agent-custom.html)

Suppress on a line with `# inspect-evals-lint: ignore[IETS005]` or `ignore[custom_solver_tests]`; select or ignore it in configuration by either, or by the prefix `IETS`.


---

# IETS006: custom_scorer_tests

Every @scorer function name appears somewhere in the tests.

**Category:** tests · **Applies to:** eval, helper

## What it does

Finds functions decorated with `@scorer` in the package and checks each name appears in a test file. For an evaluation the search covers `tests/<name>/`; for a helper package, the whole tests root, because shared components are usually tested next to the evaluation that motivated them. One diagnostic per untested function. This is a presence check, not a quality check.

## Why is this bad?

A custom scorer is the evaluation's own logic, the part no upstream test covers. One test that at least constructs it catches import errors and signature changes.

## See also

- [Custom Scorers](https://inspect.aisi.org.uk/custom-scorers.html)

Suppress on a line with `# inspect-evals-lint: ignore[IETS006]` or `ignore[custom_scorer_tests]`; select or ignore it in configuration by either, or by the prefix `IETS`.


---

# IETS007: custom_tool_tests

Every @tool function name appears somewhere in the tests.

**Category:** tests · **Applies to:** eval, helper

## What it does

Finds functions decorated with `@tool` in the package and checks each name appears in a test file. A tool registered under another name with `@tool(name="submit")` is also satisfied by a mention of that name, since that is what the tests and the transcript call it. For an evaluation the search covers `tests/<name>/`; for a helper package, the whole tests root, because shared components are usually tested next to the evaluation that motivated them. One diagnostic per untested function. This is a presence check, not a quality check.

## Why is this bad?

A custom tool is the evaluation's own logic, the part no upstream test covers. One test that at least constructs it catches import errors and signature changes.

## See also

- [Tools: Custom Tools](https://inspect.aisi.org.uk/tools.html#custom-tools)

Suppress on a line with `# inspect-evals-lint: ignore[IETS007]` or `ignore[custom_tool_tests]`; select or ignore it in configuration by either, or by the prefix `IETS`.


---

# IEBP001: get_model_location

get_model() is only called inside @solver, @scorer or @agent functions.

**Category:** best_practices · **Applies to:** eval, helper

## What it does

Warns on each `get_model()` call that is not inside a function decorated with `@solver`, `@scorer` or `@agent`. An agent is the solver of a sandboxed evaluation, and its `execute` runs per sample just as a solver's does.

## Why is this bad?

Resolving a concrete model at import time or inside `@task` fixes it before the caller can choose one. Resolving it inside the solver, scorer or agent keeps the task declarative and lets `--model-role` and task parameters override it.

## Example

```python
GRADER = get_model("openai/gpt-4o")   # resolved at import

@scorer(metrics=[accuracy()])
def graded():
    async def score(state, target):
        return await GRADER.generate(...)
```

Use instead:

```python
@scorer(metrics=[accuracy()])
def graded(model: str | Model | None = None):
    async def score(state, target):
        grader = get_model(model, role="grader", default="openai/gpt-4o")
        ...
```

## See also

- [Models: Role Resolution](https://inspect.aisi.org.uk/models.html#role-resolution)
- [Solvers: Models in Solvers](https://inspect.aisi.org.uk/solvers.html#models-in-solvers)
- [Custom Scorers: Models in Scorers](https://inspect.aisi.org.uk/custom-scorers.html#models-in-scorers)
- [Custom Agents: Parameters](https://inspect.aisi.org.uk/agent-custom.html#parameters)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP001]` or `ignore[get_model_location]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IEBP002: model_role_resolution

get_model(role=...) resolves deliberately: an explicit model, default= or required=True.

**Category:** best_practices · **Applies to:** eval, helper · **Allowlist:** `[tool.inspect-evals-lint.allowlists.model_role_resolution]`

## What it does

Flags each `get_model(role=...)` call that passes no explicit model, no `default=` and no `required=True`. A literal `default=None`, `model=None` or `required=False` changes nothing at runtime and does not count. Each diagnostic is keyed by the role name (`<dynamic>` for a non-literal), which is what an allowlist entry names.

## Why is this bad?

A role that is not bound at invocation falls back to the model under evaluation. A grader then grades the model's own output, and the scores still look plausible. Pinning a default or requiring the role makes the fallback a choice rather than an accident.

## Example

```python
grader = get_model(role="grader")
```

Use instead:

```python
grader = get_model(role="grader", default="openai/gpt-4o")
# or
grader = get_model(role="grader", required=True)
```

## Options

- `allowlists.model_role_resolution`: `{ package = ["role"] }` entries reported as warnings while an existing surface is burned down.

## See also

- [Models: Model Roles](https://inspect.aisi.org.uk/models.html#model-roles)
- [Models: Role Defaults](https://inspect.aisi.org.uk/models.html#role-defaults)
- [Tasks: Model Roles](https://inspect.aisi.org.uk/tasks.html#model-roles)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP002]` or `ignore[model_role_resolution]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IEBP003: sample_ids

Every Sample() passes id=.

**Category:** best_practices · **Applies to:** eval, helper

## What it does

Flags each `Sample(...)` call without an `id=` keyword.

## Why is this bad?

Without a stable id a sample is identified by its position. Shuffling, `--limit`, reruns and dataset updates all change positions, so results can no longer be compared sample by sample.

## Example

```python
Sample(input=record["question"], target=record["answer"])
```

Use instead:

```python
Sample(input=record["question"], target=record["answer"], id=record["id"])
```

## See also

- [Datasets: Dataset Samples](https://inspect.aisi.org.uk/datasets.html#dataset-samples)
- [Log Files: IDs and Shuffling](https://inspect.aisi.org.uk/eval-logs.html#ids-and-shuffling)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP003]` or `ignore[sample_ids]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IEBP004: task_overridable_defaults

@task parameters naming a solver, scorer, metric, grader or model have defaults.

**Category:** best_practices · **Applies to:** eval, helper

## What it does

Flags each parameter of a `@task` function whose name contains `solver`, `scorer`, `metric`, `metrics`, `grader` or `model` and has no default.

## Why is this bad?

These are the pieces callers most often want to swap. With defaults the task runs unconfigured, `inspect eval my_eval/task` just works, and each piece can still be overridden with `-T`.

## Example

```python
@task
def my_eval(solver, grader_model):
    ...
```

Use instead:

```python
@task
def my_eval(solver: Solver | None = None, grader_model: str | None = None):
    ...
```

## See also

- [Tasks: Parameters](https://inspect.aisi.org.uk/tasks.html#parameters)
- [Tasks: Solver Parameter](https://inspect.aisi.org.uk/tasks.html#solver-parameter)
- [Tasks: Scorer Override](https://inspect.aisi.org.uk/tasks.html#scorer-override)
- [Extensions: Components: Tasks](https://inspect.aisi.org.uk/extensions-components.html#tasks)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP004]` or `ignore[task_overridable_defaults]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IEBP005: sandbox_image_pinning

Registry images in compose files use an immutable tag or digest.

**Category:** best_practices · **Applies to:** eval, helper · **Allowlist:** `[tool.inspect-evals-lint.allowlists.sandbox_image_pinning]`

## What it does

Reads every `compose*.y*ml` and `docker-compose*.y*ml` under the package, `exclude`d directories included, and flags each service whose `image` is untagged or `:latest`. Services built locally (`build:`) and `${VAR}` interpolated references are skipped. Each diagnostic is keyed by the image reference, which is what an allowlist entry names.

## Why is this bad?

A floating reference resolves to whatever the registry holds today. A push upstream silently changes the evaluation environment, and results stop being comparable across runs without anything in the repository changing.

## Example

```yaml
services:
  default:
    image: aisiuk/inspect-tool-support
```

Use instead:

```yaml
services:
  default:
    image: aisiuk/inspect-tool-support:1.4.2
    # or: aisiuk/inspect-tool-support@sha256:...
```

## Options

- `allowlists.sandbox_image_pinning`: `{ package = ["image/ref"] }` entries reported as warnings while they are pinned.

## See also

- [Sandboxing: Task Configuration](https://inspect.aisi.org.uk/sandboxing.html#task-configuration)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP005]` or `ignore[sandbox_image_pinning]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IEBP006: gpu_sandbox_check

An evaluation requiring a GPU ships a maintenance sandbox check task.

**Category:** best_practices · **Applies to:** eval

## What it does

When `eval.yaml` declares `metadata.requires.gpu`, `tasks` must include a task whose name ends `_sandbox_check` and which is declared with `kind: maintenance`.

## Why is this bad?

GPU sandbox images cannot be exercised in ordinary CI, so a broken image (a missing package, the wrong Python, a CUDA toolchain that does not work) would only show up as errored samples in a real run. The check task certifies the image on GPU hardware through the evaluation's own scorer, and `kind: maintenance` keeps its accuracy out of listings that present model results.

## Example

```yaml
tasks:
  - name: kernelbench
  - name: kernelbench_sandbox_check
    kind: maintenance
metadata:
  requires:
    gpu: true
```

## See also

- [Sandboxing: Container Resources](https://inspect.aisi.org.uk/sandboxing.html#container-resources)
- [Modal sandbox: Docker Compose (GPU reservations)](https://meridianlabs-ai.github.io/inspect_sandboxes/modal.html#docker-compose)
- [Daytona sandbox: Docker Compose (GPU reservations)](https://meridianlabs-ai.github.io/inspect_sandboxes/daytona.html#docker-compose)
- [Kubernetes sandbox: Targeting kubeconfig contexts (GPU nodes)](https://k8s-sandbox.aisi.org.uk/tips/configuration/#targeting-specific-or-multiple-kubeconfig-contexts)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP006]` or `ignore[gpu_sandbox_check]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IEBP007: dockerfile_locking

Dockerfile builds consume locked inputs: committed locks, digest-pinned images, fixed sources.

**Category:** best_practices · **Applies to:** eval

## What it does

Reads every `Dockerfile*` under the evaluation statically (nothing is executed) and reports one warning per instruction with a build input that is not locked, pointing at the instruction's first line:

- Dependency installs must consume a committed lock or hashed snapshot: `uv sync --locked` with `uv.lock` copied or bind-mounted in beforehand, or `pip install --require-hashes -r <snapshot>`. `uv sync` alone (may update the lock), `uv sync --frozen` (skips freshness validation), `uv lock` during the build, `pip install <packages>`, requirements files without `--require-hashes`, and installing the project with dependency resolution all warn; `pip install . --no-deps` after locked dependencies is accepted. Manifests are not inspected: ranges there are fine when the consumed lock resolves them.
- Image and source inputs must be immutable: `FROM` and `COPY --from` references need an `@sha256` digest (`scratch` and earlier build stages are exempt), Git dependencies on a pip command line need a full commit, and an installer script piped from `curl` or `wget` into a shell warns because nothing verifies it. `COPY --from` a digest-pinned tool image is the accepted way to bring in `uv`.
- Dynamic references (`${VAR}` images, remote or interpolated `COPY` sources, templated package names) and unsupported package managers (`npm`, `cargo`, `conda`, ...) are reported as unverified, never as passing.
- `COPY` sources resolve against the build context: the `# BUILD_CONTEXT=<path relative to the repository root>` directive inspect_evals uses, else the `build.context` of a compose service that builds the Dockerfile (compose resolves `dockerfile` relative to it), else the Dockerfile's directory. Missing sources and sources outside the repository warn. BuildKit heredocs are skipped and `.dockerignore` is not consulted.

Limits, kept visible: OS package installs (`apt-get install` and the like) cannot be locked by this rule, so a Dockerfile whose other inputs are locked still passes, with each such step named in the pass message. Build-isolation dependencies of source builds and the interpreter version against the sandbox project's `requires-python` are not checked. Passing means the supported installs consume locked inputs, not that a rebuild is byte-identical. Every finding is a warning in this release. Dockerfiles matched by `exclude` are not read; an evaluation without a Dockerfile skips.

## Why is this bad?

A digest-pinned published image fixes what runs today, but not what a rebuild produces: a Dockerfile that installs whatever the index or registry serves on the day gives a different sandbox each time the image is rebuilt, so results stop being comparable without anything in the repository changing, and a broken upstream release can turn into errored samples. Consuming a committed lock makes the rebuild a function of the repository alone.

## Example

```dockerfile
FROM python:3.12-slim
COPY pyproject.toml ./
RUN curl -LsSf https://astral.sh/uv/install.sh | sh && uv sync
```

Use instead:

```dockerfile
# BUILD_CONTEXT=.
FROM python:3.12-slim@sha256:<digest>
COPY --from=ghcr.io/astral-sh/uv:0.12.5@sha256:<digest> /uv /usr/local/bin/uv
COPY src/inspect_evals/my_eval/sandbox/pyproject.toml src/inspect_evals/my_eval/sandbox/uv.lock ./
RUN uv sync --locked --no-dev --no-install-project
```

## Options

- `dockerfile_locking.host-lock-coupling`: `"warn"` reports a build that copies the repository root's `pyproject.toml` or `uv.lock`, because unrelated host dependency updates would then change the sandbox image; `"allow"` accepts it, for a standalone repository whose root project is the sandbox. `warn` in the `monorepo` preset, `allow` elsewhere.

## See also

- [Sandboxing: Task Configuration](https://inspect.aisi.org.uk/sandboxing.html#task-configuration)
- [Sandboxing: Prebuilt Images](https://inspect.aisi.org.uk/sandboxing.html#prebuilt-images)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP007]` or `ignore[dockerfile_locking]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IEBP008: duplicate_filter_acknowledged

Every filter_duplicate_ids() call states how many duplicates it drops and links the upstream report.

**Category:** best_practices · **Applies to:** eval, helper

## What it does

Flags each `filter_duplicate_ids(...)` call that lacks a `max_duplicates=` keyword, lacks a `reason=` keyword, or gives a literal `reason` with no `http://` or `https://` URL in it. A `reason` passed as a variable is taken at face value, as is `**kwargs`. One diagnostic per call.

## Why is this bad?

Dropping samples that share an id is only safe when they are the same record twice. A call with no count is a workaround nobody has measured, and a reason with no link is a defect nobody upstream knows about; both let a bad id key silently truncate the dataset, which is how WorldSense lost half its trials. The count bounds the damage a revision bump can do and the link is the evidence a reviewer can check.

## Example

```python
dataset = filter_duplicate_ids(dataset)
```

Use instead:

```python
dataset = filter_duplicate_ids(
    dataset,
    max_duplicates=11,
    reason="8 groups of identical rows, see https://github.com/org/repo/issues/268",
)
```

## See also

- [Datasets: Dataset Samples](https://inspect.aisi.org.uk/datasets.html#dataset-samples)
- [Log Files: IDs and Shuffling](https://inspect.aisi.org.uk/eval-logs.html#ids-and-shuffling)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP008]` or `ignore[duplicate_filter_acknowledged]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IEBP009: known_broken_reported

Every drop_known_broken() entry maps a sample id to the URL of its upstream report.

**Category:** best_practices · **Applies to:** eval, helper

## What it does

Flags each `drop_known_broken(...)` call without a `broken=` keyword, and each entry of its `broken` dict whose value is a literal string with no `http://` or `https://` URL. The dict may be written inline or bound to a module-level name in the same file; a name the file does not define, a non-literal value and `**kwargs` are taken at face value. Entry diagnostics point at the entry's line so a suppression can sit beside it.

## Why is this bad?

A hard-coded exclusion list is a workaround for a dataset defect. Without the report beside each id, nobody upstream knows about the defect, a reviewer cannot check the claim, and the entry outlives the fix. The URL is the evidence and the reminder.

## Example

```python
KNOWN_BROKEN = {"ruin_names_100": "options split on commas"}
```

Use instead:

```python
KNOWN_BROKEN = {"ruin_names_100": "https://github.com/org/repo/issues/19"}
```

## See also

- [Datasets: Dataset Samples](https://inspect.aisi.org.uk/datasets.html#dataset-samples)

Suppress on a line with `# inspect-evals-lint: ignore[IEBP009]` or `ignore[known_broken_reported]`; select or ignore it in configuration by either, or by the prefix `IEBP`.


---

# IESC001: sandbox_privileges

Compose services do not grant additional sandbox privileges or host access.

**Category:** security · **Applies to:** eval, helper · **Allowlist:** `[tool.inspect-evals-lint.allowlists.sandbox_privileges]`

## What it does

Reads parsed YAML from every `compose*.y*ml` and `docker-compose*.y*ml` under the package, including nested files and overrides. `exclude` does not apply: Compose files configure the sandbox from the host even when they sit beside challenge code that is excluded. Reports one error per service, field and value for the settings described below.

GPU reservations under `deploy.resources.reservations.devices`, ordinary named or anonymous volumes, and namespaces shared by `service:` reference are accepted. YAML anchors are resolved; commented-out settings are ignored. A value tagged `!override` or `!reset` is checked as written.

Each finding points at the line of the setting, or of the list item for list settings such as `volumes` and `cap_add`. A setting merged from a YAML anchor points at the anchor, so a suppression comment there covers every service that merges it.

## Why is this bad?

These settings can give model-controlled processes access to host data, devices, namespaces, or the container engine, or remove runtime restrictions. Some evaluations need them. Each exception needs review and an allowlist entry; a finding does not establish that a setting is exploitable.

The effects depend on the host platform, daemon configuration, and remaining permissions. In particular, sharing a namespace does not automatically authorize every operation on the resources it exposes.

### Privileges and device access

- `privileged: true` grants all Linux capabilities, exposes host devices, and relaxes security profiles. This can allow sandbox processes to alter host resources and take control of the host. [Docker runtime privileges](https://docs.docker.com/engine/containers/run/#runtime-privilege-and-linux-capabilities).
- Any nonempty `cap_add` requests additional Linux capabilities. Each capability authorizes particular operations: for example, `SYS_ADMIN` includes mounting filesystems, while `SYS_PTRACE` permits process inspection subject to other restrictions. The implications depend on the capabilities requested and the namespaces in which they apply. [Linux capabilities](https://man7.org/linux/man-pages/man7/capabilities.7.html).
- Any nonempty `devices` exposes selected devices and their drivers to the container. For example, access to a disk device can expose data beyond the container's mounted directories. The effect depends on the device and its allowed operations. [Docker device access](https://docs.docker.com/reference/cli/docker/container/run/#device).
- Any nonempty `device_cgroup_rules` adds device permissions by type and major/minor number. Rules can allow reading, writing, and creating device nodes; wildcards can cover whole classes of devices. These permissions can apply when a device node becomes available later. [Linux device access rules](https://docs.kernel.org/admin-guide/cgroup-v1/devices.html).

### Disabled security restrictions

The following values in `security_opt` are reported with either `:` or `=` as the separator. Each disables a different restriction.

- `seccomp=unconfined` removes the container's system-call filter. Code can attempt kernel operations that the default profile would reject, subject to remaining permission checks. [Docker seccomp profiles](https://docs.docker.com/engine/security/seccomp/).
- `apparmor=unconfined` removes AppArmor profile enforcement where AppArmor is enabled. File access, mounts, and other operations lose the additional restrictions imposed by that profile. [Docker AppArmor profiles](https://docs.docker.com/engine/security/apparmor/).
- `label=disable` disables SELinux labeling for the container. On an SELinux-enabled host, this removes label-based confinement that helps separate container processes and resources. `label=type:spc_t` (the super-privileged container type) and `label=type:unconfined_t` run the container in an SELinux domain without that confinement, with the same effect. [Docker security options](https://docs.docker.com/reference/cli/docker/container/run/#security-opt).
- `systempaths=unconfined` removes the runtime's masking and read-only protection of system paths. Kernel information and controls at those paths become accessible subject to remaining permissions. [Docker system-path security option](https://docs.docker.com/reference/cli/docker/container/run/#security-opt).

### Shared namespaces

| Setting                       | Implication                                                                                                                                                                                                                                                                                  |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `network_mode: host`          | Shares host networking. Processes can reach services on host loopback and listen on host ports without a port mapping. [Docker host networking](https://docs.docker.com/engine/network/drivers/host/).                                                                                       |
| `pid: host`                   | Makes host processes visible. Signaling or inspecting them then depends on credentials, capabilities, and other controls. [Linux PID namespaces](https://man7.org/linux/man-pages/man7/pid_namespaces.7.html).                                                                               |
| `ipc: host`                   | Shares host IPC resources, including System V shared memory, semaphores, and message queues. Processes may read data or interfere with applications when IPC permissions allow it. [Linux IPC namespaces](https://man7.org/linux/man-pages/man7/ipc_namespaces.7.html).                      |
| `userns_mode: host`           | Disables per-container user-namespace remapping where the Docker daemon enables it. Container user IDs lose that remapping's separation from host IDs. [Docker user-namespace remapping](https://docs.docker.com/engine/security/userns-remap/#disable-namespace-remapping-for-a-container). |
| `uts: host`                   | Shares the host's hostname and NIS domain name. A process with the required capability can change these identifiers for other processes in that namespace. [Linux UTS namespaces](https://man7.org/linux/man-pages/man7/uts_namespaces.7.html).                                              |
| `cgroup: host`                | Exposes the host view of cgroup paths and hierarchy. This reveals information outside the container's private view; it does not itself remove resource limits or grant write access. [Linux cgroup namespaces](https://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html).              |
| `network_mode: container:...` | Joins another container's networking, including its loopback services and listening ports. [Docker container networking mode](https://docs.docker.com/engine/network/#container-networks).                                                                                                   |
| `pid: container:...`          | Shares another container's process namespace, allowing process visibility and operations subject to permissions. [Docker PID settings](https://docs.docker.com/reference/cli/docker/container/run/#pid).                                                                                     |
| `ipc: container:...`          | Shares another container's IPC resources, allowing data access or interference subject to permissions. [Docker IPC settings](https://docs.docker.com/reference/cli/docker/container/run/#ipc).                                                                                               |

`container:` references depend on containers outside the declared Compose service relationships. The rule cannot establish their configuration. References using `service:` remain within those declared relationships and are outside this check.

### Host files and engine access

- Host bind mounts in either short or long `volumes` syntax expose a host path. Writable mounts can let sandbox code change host files; read-only mounts still expose their contents, potentially including credentials or evaluation answers. Relative paths and Windows host paths are checked too, as is a short-syntax source with a path separator outside its interpolations, such as `${HOME}/.ssh`: a volume name cannot contain one. [Docker bind mounts](https://docs.docker.com/engine/storage/bind-mounts/).
- Named volumes whose local `driver_opts.o` contains `bind` or `rbind` also expose the host path in `driver_opts.device`. Review them as host bind mounts even though the service refers to a volume name. [Compose local-driver bind example](https://docs.docker.com/reference/compose-file/volumes/#driver_opts). The `rbind` option also includes existing mounts below the source path, potentially exposing additional filesystems. [Linux recursive bind mounts](https://man7.org/linux/man-pages/man8/mount.8.html).
- Windows named pipe mounts (`type: npipe`) expose a host communication endpoint. The implications depend on the service listening on that pipe and the caller's permissions. [Compose mount types](https://docs.docker.com/reference/compose-file/services/#volumes).
- Container engine sockets receive an additional callout when the source or target names a Docker, Podman, containerd, CRI-O or BuildKit socket (`docker.sock`, `docker_engine`, `podman.sock`, `containerd.sock`, `crio.sock`, `cri-dockerd.sock`, `buildkitd.sock`), or the source is a directory holding `/var/run/docker.sock`, such as `/var/run`. Access to an engine API can allow creation of containers with host mounts or elevated privileges. [Docker daemon access](https://docs.docker.com/engine/security/#docker-daemon-attack-surface). A read-only socket mount does not restrict which API operations a connected client can request. Review must therefore account for [Docker's API authorization policy](https://docs.docker.com/engine/extend/plugins_authorization/), which allows all operations by default.
- `secrets` and `configs` whose top-level definition has a `file` or `environment` source copy a host file or environment variable into the container, at `/run/secrets/<name>` for a secret. Review them as you would a read-only bind mount of that file. `content` and `external` sources are accepted. Keys name the file, or `env:<variable>`. [Compose secrets](https://docs.docker.com/reference/compose-file/secrets/), [Compose configs](https://docs.docker.com/reference/compose-file/configs/).
- `use_api_socket: true` provides the engine socket and the user's credentials. In addition to engine access, this can permit registry operations using those credentials. [Compose API socket access](https://docs.docker.com/reference/compose-file/services/#use_api_socket).
- `volumes_from: [container:...]` imports another container's mounts, potentially exposing files or sockets whose sources are not declared in this Compose file. A read-only import still exposes readable data. [Compose imported volumes](https://docs.docker.com/reference/compose-file/services/#volumes_from).

### Privileged lifecycle commands

The rule also reports `privileged: true` within each lifecycle hook:

- `pre_start` runs an initialization step with extra privileges in a temporary container before the service starts. [Compose pre-start hooks](https://docs.docker.com/reference/compose-file/services/#pre_start).
- `post_start` runs a command with extra privileges in the running service container after startup. [Compose post-start hooks](https://docs.docker.com/reference/compose-file/services/#post_start).
- `pre_stop` runs a command with extra privileges before the service container stops. [Compose pre-stop hooks](https://docs.docker.com/reference/compose-file/services/#pre_stop).

These hooks need review even when the service itself has no `privileged` setting. Check what the command does and whether sandbox code can modify any scripts or inputs it will use with those privileges.

## Example

```yaml
services:
  default:
    image: example/sandbox:1.0
    privileged: true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
```

Remove the settings when they are unnecessary. If required, document the reason next to `default:privileged` and `default:volumes:/var/run/docker.sock` entries in `allowlists.sandbox_privileges`.

## Scope and related checks

This is a static check of declared Compose settings. It does not run Docker, read environment files, inspect images, combine overrides, follow `include` or `extends` files, or inspect Kubernetes settings. Interpolated privilege, namespace, capability, device, security-option, and mount values that cannot be checked produce warnings because [Compose interpolation](https://docs.docker.com/reference/compose-file/interpolation/) can change the effective settings at runtime. A pass means no listed settings were found in the files checked.

Service `environment` and `env_file` values are not checked, though they can also carry host values into the container. Published ports, external networks, and host-gateway mappings need a network-exposure policy. Missing hardening settings such as `read_only` or `no-new-privileges` are outside this rule's checks.

## Options

- `allowlists.sandbox_privileges`: `{ package = ["service:field:value"] }` entries report warnings while present. Each finding names its key in the hint. The value is the capability, device, rule, namespace mode, security option, host path, secret or config source, or container, so allowing `default:cap_add:SYS_PTRACE` does not allow `ALL`, and allowing one host path does not allow another. Settings without a value use `service:field`, such as `default:privileged` and `default:post_start.privileged`. A key applies to that service across the package's Compose files. Removed settings leave stale entries for the runner to report.

## See also

- [Sandboxing](https://inspect.aisi.org.uk/sandboxing.html)

Suppress on a line with `# inspect-evals-lint: ignore[IESC001]` or `ignore[sandbox_privileges]`; select or ignore it in configuration by either, or by the prefix `IESC`.
