Metadata-Version: 2.4
Name: dankaro-agent-quality
Version: 0.1.0
Summary: Release gates for AI agent output: sources, baselines, statistics, stability, contradictions and adversarial review.
Author-email: Marius Suteu <marius@dankaro.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Dankaro-projects/dankaro-agent-toolkit
Project-URL: Documentation, https://github.com/Dankaro-projects/dankaro-agent-toolkit/tree/main/quality-harness#readme
Project-URL: Changelog, https://github.com/Dankaro-projects/dankaro-agent-toolkit/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/Dankaro-projects/dankaro-agent-toolkit/issues
Keywords: ai agents,llm evaluation,quality gates,evidence,baselines,release gating
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pre-commit>=3; extra == "dev"
Dynamic: license-file

# Dankaro Agent Quality Harness

The harness evaluates the result of an AI agent before that result is released. It treats the following as release failures that can be measured: claims without sources, comparisons without a declared baseline, evidence that cites itself, excessive complexity, results that change between runs, contradictions and missing deliverables.

The central design rule is that a filled field proves nothing on its own. A source field that contains text is not evidence that the text points to anything a reader can check.

## Gates

| Gate | The release is blocked when |
|---|---|
| `contract` | the result does not match the typed task and result contract |
| `outcome` | a required deliverable is missing, or a placeholder remains in the output |
| `evidence` | a factual claim has no source that a reader can follow |
| `provenance` | the evidence is circular, or a recommendation rests on facts that were not checked in this run |
| `baseline` | a comparison names no baseline, or does not include a naive one |
| `statistics` | an effect has no sample size, or its significance rests on a single test |
| `simplicity` | the number of controls, decisions or setup steps exceeds its limit |
| `explainability` | an interpreted claim gives no reasoning |
| `contradiction` | one key holds conflicting values, either in this run or in the claim ledger |
| `stability` | repeated runs give different values for the same key |
| `adversarial` | a deterministic critic or a registered reviewer raises an objection |

By default the gates are conjunctive: every gate must pass on its own, or the release is blocked. The harness also reports a weighted overall score so that you can follow a project over time, but that score cannot block a release by itself. A release should only fail for a reason that a specific gate names. To let the overall score block as well, set `conjunctive = false` in `thresholds.toml`.

## Install

```bash
pip install dankaro-agent-quality
```

The package has no dependencies beyond the Python standard library and needs Python 3.11 or later.

## Command line

```bash
dankaro-quality --version
dankaro-quality init                        # write thresholds.toml to edit
dankaro-quality evaluate result.json        # full report in Markdown
dankaro-quality evaluate result.json --format json --out reports
dankaro-quality validate a.json b.json      # one line per file
```

`evaluate` and `validate` exit with `0` when every result passes and `2` when any result is blocked. The thresholds come from the file named with `--settings`; without that option, from `./thresholds.toml` if it exists; otherwise from the defaults shipped with the package.

## Working on the harness

```bash
python -m pip install -e .
python -m unittest discover -s tests -v
dankaro-quality evaluate fixtures/passing.json
```

## What counts as a source

The evidence gate requires a locator, not just a string. A source such as `x` or `trust me` is blocked. The following forms are accepted:

```
https://example.com/report          measurement://prototype/workflow-v2
runs/2026-08-06/RUN_REPORT.md:24    docs/report.md
$ pytest -q tests/test_pipeline.py  Companies House 05662277
sheet 12                            a1b2c3d (commit)
```

A source that refers to the agent's own earlier output, such as "our earlier draft" or "as stated above", is rejected as circular by both the evidence gate and the provenance gate. A number in an earlier draft is not a source. It exists only because someone wrote it down before.

## The baseline rule

A comparison must state what it was compared against, and at least one comparison must be against a naive baseline. Examples of naive baselines are uniform rotation, the base rate, the hand set prior, an empty field, the most recent item and the most popular item.

```python
AgentResult(
    output="The scheduler outperforms the alternative.",
    comparisons=(
        Comparison(
            subject="thompson scheduler",
            baseline="uniform rotation",
            baseline_kind=BaselineKind.NAIVE,
            metric="fund multiple",
            effect="+3.94x",
            sample_size=40,
            tests=("paired t", "Wilcoxon signed-rank", "sign"),
        ),
    ),
)
```

This gate exists because of a measured case. In a simulation, a policy compared only against a plausible alternative won by 2.08 times (t = 2.08). When a third policy was added that read no input and rotated uniformly, the first policy never beat it at any coverage level tested: +0.66 (p = 0.23), −0.69 (p = 0.52) and +0.64 (p = 0.58). The original comparison was not wrong. It was incomplete, and the missing baseline reversed its meaning.

## The claims contract

Every substantive statement in a result is a `Claim`.

- A `fact` needs a source that resolves, a `confidence` value and a `verified` flag.
- An `inference`, a `judgment` or a `recommendation` needs a `rationale`.

The gates also read these fields:

| Field | Meaning |
|---|---|
| `key` and `value` | The unit that can be compared. The contradiction and stability gates need it |
| `verified` | `True` only if the fact was measured in this run. `False` means it was recalled or inherited |
| `id` and `supersedes` | The claim's identity, and the earlier claim it explicitly replaces |
| `sample_size` and `tests` | The data and the tests behind a statistical claim |

The `verified` flag separates "I checked this" from "this is how it usually works". Recalled facts are allowed, but they cannot make up most of the evidence, and they cannot be the only support for a recommendation.

## Stability measures content, not wording

The stability gate compares keyed claims across repeated runs. The same key with a different value counts as drift and blocks the release. The same key and value in different words does not count as drift. A result that contains no keyed claims is reported as not comparable, rather than as stable, because a run that cannot be measured has not passed.

## The claim ledger across runs

Checking for contradictions inside one result does not catch the failure that happens in practice: a claim disproved in March reappears in a June report, because the June run never saw the correction. To catch it, point the harness at a claim ledger that persists between runs:

```toml
[global]
claim_ledger = "memory/claims.jsonl"   # leave empty to search upwards for memory/claims.jsonl
```

The ledger holds one JSON object per line. Each object has a `status` of `live` or `demoted`, and may have a `watch` regular expression:

```json
{"id":"CLM-1","text":"Thompson beats greedy by 2.08x","status":"demoted",
 "reason":"Never beats round-robin; the gain belongs to breadth",
 "demoted_by":"review/verify_baseline.py","watch":"2\\.08x"}
```

A run whose output matches the `watch` pattern of a demoted claim is blocked. If the new result names that claim in `supersedes`, it is allowed through, because an explicit replacement is legitimate. A missing or malformed ledger is treated as empty and never causes an error.

## Simplicity limits

```python
AgentResult(..., exposed_controls=5, required_decisions=3, configuration_steps=1)
```

The limits are set in `thresholds.toml`: by default at most 7 exposed controls, 5 required decisions and 3 configuration steps. Keeping them in a settings file makes simplicity an engineering constraint that the team agrees once, rather than a design opinion that is argued again in every sprint.

## Hooks and adversaries

```python
hooks = HookRegistry()

@hooks.add_pre_run
def require_owner(task):
    return [] if task.metadata.get("owner") else [Finding("OWNER", "Task owner missing")]
```

An error from a hook that runs before the agent prevents the agent from running. Hooks that run afterwards, and adversaries, contribute to the release decision. Model based review should stay secondary to the deterministic checks: a critic model may point out a problem, but it should not be the only authority on evidence, required deliverables or numeric thresholds.

## Enforcing the gates with a decorator

```python
@quality_gated(harness, repeats=3)
def product_agent(task):
    return AgentResult(...)
```

A blocked run raises `RuntimeError`. The full structured report is available on `exception.quality_report`.

## Regression fixtures

Each file in `fixtures/regression/` records a case that an earlier version of the harness passed when it should have blocked it. If one of these fixtures starts to pass, a gate has been weakened.

| Fixture | The case it records |
|---|---|
| `junk_source.json` | The source `x` once scored 100 on the evidence gate |
| `circular_source.json` | An agent cited its own earlier draft as evidence |
| `no_baseline.json` | A claimed win with no declared comparison |
| `single_test.json` | Significance that rested on a single test |

## Repository map

```text
src/dankaro_quality/
├── contracts.py      task, result, claim and comparison types
├── sources.py        rules for what counts as a source
├── ledger.py         the claim ledger that persists between runs
├── harness.py        orchestration and the release decision
├── hooks.py          hooks before and after a run, and adversaries
├── adversaries.py    deterministic critics
├── reporting.py      JSON and Markdown reports
├── serialization.py
├── cli.py
└── evaluators/       one module for each gate
```
