Metadata-Version: 2.5
Name: agent-build-gates
Version: 0.3.0
Summary: Quality gates for agent-built code: an anti-simulation tripwire, an AWS-account tripwire, a local-path tripwire, a denylist tripwire, and an eval harness with confidence intervals and significance testing.
Project-URL: Homepage, https://github.com/snymanpaul/aws_agent_1/tree/main/packages/agent-build-gates
Project-URL: Source, https://github.com/snymanpaul/aws_agent_1
Project-URL: Issues, https://github.com/snymanpaul/aws_agent_1/issues
Author: Paul Snyman
License-Expression: MIT
License-File: LICENSE
Keywords: ai-agents,aws,code-quality,evals,static-analysis
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Provides-Extra: strands
Requires-Dist: strands-agents>=1.48; extra == 'strands'
Description-Content-Type: text/markdown

# agent-build-gates

Quality gates for agent-built code. Five of them, all pure standard library, plus an optional sixth.

When an agent writes code at volume, the failure that costs most is not a bug. It is work that
looks complete and never touched the service it claims to demonstrate: a class that stands in for
a queue, a validator that returns `ALLOW` because the API was unreachable, a fallback chain whose
models never got called. Asking for honesty does not catch that. A script that fails the build does.

```bash
pip install agent-build-gates
```

Zero dependencies. `ship-gate` needs the `[strands]` extra, because it drives real agent
runs and costs money; the other five are pure standard library.

## The gates

### `no-sim-check`

Flags substituted integrations: code that fabricates a success for a call it could have made.

```bash
no-sim-check $(git ls-files '*.py')
```

It looks for substitute-object vocabulary (`mock`, `stub`, `fake`, `dummy`, `hardcoded`), returns
that fabricate a result, "in production this would" deferrals, and a bare `return True` straight
out of an `except`. Two scoping rules keep the signal usable, both learned from classifying every
hit in a 275-file repo:

- **Identifier-aware boundaries.** A plain `\b` stops at underscores and CamelCase humps, so
  `class MockSQSQueue`, `mock_client` and `_simulate_human_response` are invisible to it while
  their docstrings trip it. This uses boundaries that break on both.
- **Prose is exempt.** The two vocabulary rules never fire on comments or docstrings, because a
  comment cannot fake an integration. Only code can. The deferral rule still reads comments, since
  a deferral comment is the marker for a call that was never made.

Escape a justified line with a trailing `# nosim:ok <reason>`.

The question the tool cannot answer for you, and the one that separates a real hit from noise:
**was a real call available and skipped?** A helper that genuinely raises to drive a recovery path
is fault injection and legitimate. Code that fabricates a success is not.

### `check-no-aws-ids`

Blocks AWS account identifiers from entering tracked `.md` and `.py` files: 12-digit account ids,
SSO admin profile strings, `sso_account` references, and ARNs carrying an account field. <!-- noaws:ok describing the patterns -->

```bash
check-no-aws-ids $(git ls-files '*.md' '*.py')   # explicit files
check-no-aws-ids                                  # every tracked .md/.py in the current repo
```

Paths resolve against your working directory, so it behaves the same installed from a wheel or
vendored into a repo. Extend the known-safe literals at the call site rather than editing the
module: `scan(files, allow=ALLOW | {"123456789012"})`. <!-- noaws:ok doc example -->

Escape a single line with `noaws:ok`, matched anywhere on it, so it works as `# noaws:ok reason`
in Python and `<!-- noaws:ok reason -->` in Markdown. Anything that must legitimately contain
account-shaped strings needs this: this gate's own tests, and documentation like the paragraph
above. The marker covers only its own line, never the rest of the file.

### `check-no-local-paths`

Blocks local machine paths from files bound for a public repo: absolute home directories
(`/Users/<name>/`, `/home/<name>/`), home-relative project folders (`~/Code`, `~/Documents`, <!-- localpath:ok describing the patterns -->
`~/Desktop`, `~/Downloads`, `~/Docs`), `/Volumes/`, and macOS per-user temp <!-- localpath:ok describing the patterns -->
(`/private/var/folders`, `/private/tmp`). Plain `/tmp` is not flagged. <!-- localpath:ok describing the patterns -->

```bash
check-no-local-paths $(git ls-files)
check-no-local-paths --frozen-regex '"id": "obs-(0717|0750)"' log.jsonl
```

Binary files are skipped. Each hit prints as `file:line:text`, the text cut to 160 characters. Escape
a single line with `localpath:ok`, as `# localpath:ok reason` in code or `<!-- localpath:ok reason -->`
in Markdown. For an append-only file whose old lines were public before the check existed and cannot
be edited, `--frozen-regex` skips the lines it matches, so the old lines are named instead of the whole
file being skipped and new lines are still checked.

### `check-denylist`

Blocks literal strings that must not reach a public repo, such as another team's project name. Those
cannot be built into a published gate without publishing the names being protected, so they live in a
file your repo does not track, one term per line (blank lines and `#` comments ignored).

```bash
check-denylist $(git ls-files)                       # reads ./.git-denylist
check-denylist --denylist ~/terms.txt --ignore-case $(git ls-files)
```

Matching is case-sensitive unless you pass `--ignore-case`. A hit prints `denylist: '<term>' found in
<file>` and never the surrounding line. A missing denylist file exits 2, not 0: a gate that silently
scans nothing is worse than one that fails. A caller that wants "no file, no check" tests for the file
first. There is deliberately no per-line escape marker, because one in a tracked file would let a
protected name through with a comment.

### `eval_harness`

Multi-run evaluation with confidence intervals and significance testing, so a single lucky run
cannot be reported as a result.

```python
from agent_build_gates import Case, run_suite, gate, label_match, wilson, case_means

cases = [Case("input text", expected="positive")]  # at least 6 cases to compare to a baseline

result = run_suite(cases, my_run_fn, {"correct": label_match}, n=5)
passed, reasons = gate(result, baseline=previous, min_quality=0.8, metric="correct")
print(wilson(case_means(result, "correct")))
```

`run_fn(input) -> (output, tokens)` is yours, so the harness never assumes a framework. `gate`
combines a quality floor, a cost ceiling and a paired sign-flip test against a frozen baseline,
and returns every reason it failed rather than the first.

The case is the unit of evidence, not the run. The n runs of each case are averaged into one
score before any interval or test, because repeats of one prompt are not independent samples.
A paired test on k cases cannot return p below `2 * 0.5**k`, so with fewer than 6 cases the
baseline comparison cannot detect a regression; `gate` then fails with "regression undetectable"
rather than passing. `label_match` scores an answer correct only when the whole normalised output
is the expected label, so "positive, not negative" does not count for both. `wilson` gives the
interval on the rate; pass it `case_means`, not the flat run scores.

Each result carries `cases_sha256`, a fingerprint of its case set, and whatever you pass as
`run_suite(..., meta={...})` (model, temperature, a prompt hash). Both are saved with a baseline,
and `gate` refuses to compare against a baseline measured on a different case set.

## `ship-gate`, behind an extra

`ship_gate` composes the harness into one auditable GO or NO-GO verdict over real agent runs, and
writes the verdict plus the underlying runs to JSON so the decision can be re-examined. It drives
a framework and it spends money, so it is optional:

```bash
pip install 'agent-build-gates[strands]'
```

Importing `agent_build_gates` never pulls that in. Import `agent_build_gates.ship_gate` directly
when you want it; without the extra it raises an ImportError naming the install command.

## Testing your gates

These gates are themselves tested, which is the point. A tripwire with no test proving it fires is
an unverified claim about the thing that verifies everything else. When this package's own
anti-simulation gate was finally given paired positive and negative controls, it turned out to be
wrong in both directions at once: it fired on comments describing deliberate fault injection, and
it missed the substituted classes it existed to catch. Fixing it took one repo from 133 reported
hits to 0 and surfaced nine real substituted integrations that had been invisible.

Measure the precision and recall of your own gates before you trust a green run.

## Changes

- **0.3.0 (2026-10-19)**: added `check-no-local-paths` and `check-denylist`.

## License

MIT.
