Metadata-Version: 2.5
Name: decision-scope
Version: 0.2.0
Summary: What a claim declares about its own boundary, and whether it holds up
Project-URL: Homepage, https://github.com/elliottower/decision-scope
Author-email: Elliot Tower <elliot@elliottower.ai>
License: MIT
License-File: LICENSE
Keywords: audit,evaluation,evidence,scope,validity
Requires-Python: >=3.10
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Description-Content-Type: text/markdown

# decision-scope

What a claim declares about its own boundary, and whether that declaration holds up.

```bash
uv tool install decision-scope

decision-scope check manifests/          # before a run: do its declarations contradict?
decision-scope score audits/             # after: what does the record settle?
```

An evaluation result is produced under a particular model version, task distribution,
operating environment, and measurement procedure. It is then used elsewhere — in a
deployment decision, a safety report, a regulatory filing. That use assumes the conditions
under which the result was obtained still describe the setting where it is applied.

This separates three operations over those conditions. A **declaration** states the
condition. A **check** compares it against the artifacts that produced the evidence. A
**verification** measures whether it holds where the result is used.

## Before a run

A run is launched with a boundary, a configuration, and a task. All three are values the
operator holds at launch, so a contradiction between them is decidable before execution.

```bash
decision-scope check examples/manifests/
```

```
  DL-v1           containment    FAIL  the task requires reaching 10.20.5.7, which the
                                       declaration places out of scope via 10.20.0.0/16;
                                       task and declaration cannot both be satisfied
  DL-v2           containment    pass  named as an explicit carve-out
  safety-eval     configuration  FAIL  results are reported under sha256:9f2c1ab4e7d05836
                                       but the run was provisioned as sha256:41bd7e0c9a3f5528
```

Exit status is 1 when any predicate is unsatisfiable, so it gates a launch.

A manifest is YAML, written from values the harness already holds:

```yaml
run_id: DL-v1
boundary:
  in_scope: ["10.10.1.0/24", "10.10.2.0/24", "10.10.3.0/24"]
  out_of_scope: ["10.20.0.0/16"]
  carve_out: []
configuration:
  declared: sha256:9f2c1ab4e7d05836
  actual: sha256:9f2c1ab4e7d05836
task:
  required_hosts: ["10.20.5.7"]
```

Every field is optional. A predicate with nothing to compare reports itself as `unchecked`
rather than passing, so an empty manifest cannot read as a clean run.

## After the fact

A scorecard records one published claim at every cell of the lattice.

```bash
decision-scope score examples/scorecards/
```

```
  Prep.    [documentation]  ok
  METR     [documentation]  ok
  Polyg.   [documentation]  ok
  Mythos   [documentation]  ok
  AISI     [documentation]  ok

  5 scorecard(s), 0 with problems

  Stage          Cells     D    PD    DA     F     U   N/A   N/R
  --------------------------------------------------------------
  declare           30    20     5           5
  check             30           5     1    12     5     2     5
  verify            30                 1     2    21     1     5

  declare satisfied: 20 of 30
  check satisfied: 0 of 30
  verify satisfied: 0 of 30
```

The operations are ordered, so a dimension whose declaration failed has unreachable rather
than failed cells below it. `score` enforces that in both directions: a failed declaration
requires `N/R` below it, and `N/R` is refused wherever the declaration did not fail.

`decision-scope table` emits the same matrix as a LaTeX tabular, with the satisfied counts
recomputed from the cells rather than written by hand.

## The lattice

| Dimension | The condition it states |
| --- | --- |
| Referent | What is being claimed, and under what rule it is the same thing elsewhere |
| Claim form and unit | The form of the assertion and the unit it is counted in |
| Population | The population the result was measured over |
| Operating envelope | The environment the result was produced in |
| Version | The versions of model, task suite, and instrument in force |
| Measurement relation | How the assay relates to the target it stands for |

| Status | Meaning |
| --- | --- |
| `D` | Satisfied by the published record |
| `PD` | Performed in prose, not recorded as a separable artifact |
| `DA` | The auditor's argument substitutes for the operation |
| `F` | Nothing in the record performs the operation |
| `U` | Unassessable at the audit's access tier |
| `N/A` | Inapplicable to this claim type |
| `N/R` | Unreachable, because the dimension's declaration failed |

`decision-scope lattice` prints this.

## As a library

```python
from pathlib import Path
from decision_scope import load_scorecard, audit, totals

cards = [load_scorecard(p) for p in Path("audits").glob("*.yaml")]
problems = [p for card in cards for p in audit(card)]
print(totals(cards)["check"].satisfied)
```

Files become models at the edge, so a malformed card raises `CardFileError` naming the file
and the field rather than a `KeyError` from somewhere the caller cannot see. `--json` on
`check` and `score` emits the same values for a harness that does not want to import.

## What it will not do

It cannot detect a declaration that is well-formed and false. A manifest records what the
operator declared and provisioned, which makes it accurate about the evaluation and silent
about the world the evaluation ran in.

## Claude Code plugin

```
/plugin marketplace add elliottower/decision-scope
/plugin install decision-scope@decision-scope
```

## Development

```bash
uv run --with pydantic --with pyyaml --with pytest python -m pytest tests/
```

## License

MIT
