Metadata-Version: 2.4
Name: agent-security-harness
Version: 4.28.2
Summary: 649 security tests for AI agent systems - MCP, A2A, L402, x402, UCP/ACP merchant-journey, AP2 mandate + Fireblocks x402 hardening + Visa TAP / Mastercard Agentic Tokens + denial-of-settlement finality, decision governance, AIUC-1 compliance, NIST AI 800-2 aligned
Author-email: "Michael K. Saleme" <research@cognitivethoughtengine.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/msaleme/red-team-blue-team-agent-fabric
Project-URL: Documentation, https://github.com/msaleme/red-team-blue-team-agent-fabric#readme
Project-URL: Repository, https://github.com/msaleme/red-team-blue-team-agent-fabric
Project-URL: Issues, https://github.com/msaleme/red-team-blue-team-agent-fabric/issues
Project-URL: Changelog, https://github.com/msaleme/red-team-blue-team-agent-fabric/blob/main/CHANGELOG.md
Project-URL: Research, https://doi.org/10.5281/zenodo.19162104
Project-URL: PyPI, https://pypi.org/project/agent-security-harness/
Keywords: ai-security,mcp,a2a,l402,x402,agent-security,red-team,blue-team,owasp,nist,aiuc-1,agentic-ai,protocol-testing,decision-governance,security-testing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Provides-Extra: mcp-server
Requires-Dist: mcp<3,>=2.0.0; extra == "mcp-server"
Dynamic: license-file

# Agent Security Harness

[![PyPI version](https://badge.fury.io/py/agent-security-harness.svg)](https://pypi.org/project/agent-security-harness/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![Apache 2.0 License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/security%20tests-649-green.svg)](#three-layers-of-agent-decision-security)
[![OWASP Agentic T1-T17](https://img.shields.io/badge/OWASP%20Agentic%20v1.1-13%2F17%20direct-blue.svg)](docs/OWASP-AGENTIC-V1.1-COVERAGE.md)
[![ClawScan](https://img.shields.io/badge/ClawScan-Benign-brightgreen)](https://clawhub.ai/msaleme/agent-security-harness)
[![Static Analysis](https://img.shields.io/badge/Static%20Analysis-Benign-brightgreen)](https://clawhub.ai/msaleme/agent-security-harness)
[![VirusTotal](https://img.shields.io/badge/VirusTotal-0%2F92_Clean-brightgreen)](https://www.virustotal.com/gui/url/37318967b56cd3cc1678972ebf0c53dbd37868b67ba3f6891447d53d51767cd2)

**Even if an agent is properly authenticated and authorized, can it still be manipulated into unsafe or policy-violating behavior?**

**A test that did not reach the target is not a passing security test.**

This harness sends adversarial traffic at a live endpoint, records what the target actually
serviced, and reports PASS, FAIL or **INCONCLUSIVE**. It will not convert a request the target
never answered — or a capability the target never exposed, or an answer that establishes
nothing either way — into evidence that a control held.
That distinction is enforced by `testing/test_serviced_guard.py` and
`testing/test_x402_capability_controls.py` rather than asserted here, and the repairs that put
it there are in [CHANGELOG.md](CHANGELOG.md).

### How that is checked: five target shapes

A verdict is only worth what it does when the target changes. Four scripts point every
suite at a sentinel that differs only in how it answers, and four state files under
`testing/` pin what each one claims:

```bash
python3 scripts/dead_host_sweep.py        # a closed port: nothing answers
python3 scripts/permissive_host_sweep.py  # HTTP 200, grants everything
python3 scripts/refusing_host_sweep.py    # HTTP 403, refuses everything
python3 scripts/empty_answer_sweep.py     # HTTP 200 `{}`: answers, does nothing
```

The fourth transport shape is the one the other three cannot see. A target answering 200
with an empty body satisfies every serviced-request guard -- the answer is real -- while
supplying no refusal, no decision, no inventory and no prose. The fourth external review
found 47 rows passing against it, including two AIUC-1 controls; the remainder and the
reading of each row is in `testing/test_empty_answer_is_not_a_control.py`.

The fifth shape is a **live agent** rather than a transport, and it is asserted in
`testing/test_refusal_establishes_a_pass.py` rather than run as a script: one agent that
complies with every request without using an indicator word, and one that declines in plain
prose. A prose-graded module must pass **nothing** against the first and **something** against
the second. Those two assertions are what v4.17.0 repaired eight modules against.

The first two ask whether a verdict can be **wrong**: a PASS against either is a control
reported as holding when it was never exercised. The third asks whether a verdict can be
**right**, which the other two cannot see — a suite that cannot pass a target which refuses
every attack scores zero against all three, and that is what a healthy module looks like from
the first two poles. The fourth asks whether a verdict needs the target to have *done*
anything at all.

These are diagnostic sentinels, not conformant implementations. A non-zero row is a reading
list rather than a defect count: it may be correct by construction (`over_refusal_harness`
asks whether *legitimate* requests are wrongly blocked, so it should pass a target that blocks
nothing), it may need a marker the generic sentinel does not emit, or it may be a real
inversion. Only reading the test separates them.

Sentinels are the floor. The first run against a real MCP implementation found two defect
classes all three had structurally missed, because every sentinel spoke the response idiom the
harness already expected.

```
$ agent-security test mcp --url http://localhost:8080/mcp
Running MCP Protocol Security Tests...
 MCP-001: Tool List Integrity Check [PASS] (0.234s)
 MCP-002: Tool Registration via Call Injection [PASS] (0.412s)
 MCP-003: Capability Escalation via Initialize [FAIL] (0.156s)
...
Results: 8/10 passed (80% pass rate) - see report.json
```

> Illustrative output. A target the harness cannot reach, or that answers without
> servicing the request, reports **INCONCLUSIVE** — never PASS. See
> [v4.13.1](CHANGELOG.md) for why that distinction is enforced rather than assumed.

649 executable security tests across 50 test-bearing modules on `main` (verified 2026-11-02 via `scripts/count_tests.py`; the v4.28.2 release carries 649, the same as `main`). MCP + A2A + L402 + x402 wire-protocol testing, plus UCP/ACP merchant-journey, AP2 mandate-chain, Fireblocks x402 hardening, Visa TAP / Mastercard Agentic Token funding-instrument, and denial-of-settlement finality conformance across the full agentic-payments stack. Decision-layer attack scenarios. One `pip install` away.

If this evidence discipline is useful in your agent-security work, **star this
repository to follow releases**.

## Evidence before coverage

Every claim in this project is bounded by the
[E1-E5 Evidence Class Taxonomy](docs/EVIDENCE-CLASS-TAXONOMY.md): observation,
runtime characterization, enforcement, persistence/replay resistance, and
isolation. A second axis, **I0-I2**, states who produced the oracle:
self-authored, independently reimplemented, or an independent sensor the target
does not control. Strength and independence are different properties, so both
are cited. A result is not promoted beyond what its retained artifact and
execution record demonstrate. Author-performed mappings and test runs are not
independent certification.

The [AIUC-1 Evidence Field Guide](https://msaleme.github.io/aiuc1-readiness/) is a
plain-language companion that applies this same taxonomy. It adds one distinction
the ladder above does not encode: whether evidence is *mapped* (a documented
requirement relationship), *executed* (a recorded run against a stated target and
pinned revision), or *independently reviewed* (assessed by a qualified outside
party). Those describe the status of evidence and are orthogonal to E1-E5, which
describes its strength. A mapping alone is E1-level material regardless of how many
requirements it covers. The taxonomy in this repository is canonical; the field
guide is hosted outside it and is not version-pinned.

**[OWASP Agentic AI v1.1 Threat Coverage Report](docs/OWASP-AGENTIC-V1.1-COVERAGE.md)** — commit-pinned mapping from the full **T1–T17** taxonomy to executable tests: **13 direct, 4 partial, 0 not evidenced**, across 96 mapped tests and 66 named OWASP scenarios. Mitigation-control validation is tracked separately from threat coverage (11 validated, 10 partial, 1 guidance-only), and every gap, evidence class and reproduction command is in the report. ([T1–T15 submission view](docs/OWASP-AGENTIC-T1-T15-SUBMISSION-COVERAGE.md) · [canonical mapping](docs/coverage/owasp-agentic-v1.1.yaml))

> Adapted from OWASP *Agentic AI — Threats and Mitigations* v1.1 under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). A test-capability report — not a certification, conformance claim, or OWASP endorsement. The adjudication is author-performed and is not independent review.

## Quick Start

```bash
pipx install agent-security-harness
```

`pipx` is the recommended install because this is a command-line tool. It builds
an isolated environment and puts `agent-security` on your PATH, with no `sudo`
and no PATH edits. Get it with `brew install pipx`, `apt install pipx`, or
`python3 -m pip install --user pipx`.

<details>
<summary>Installing with pip instead</summary>

Use a virtual environment:

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install agent-security-harness
```

**`pip install agent-security-harness` outside a virtual environment fails on
most current systems**, including Homebrew Python on macOS and Linux, Debian,
Ubuntu, and Fedora:

```
error: externally-managed-environment
```

Those interpreters ship a [PEP 668](https://peps.python.org/pep-0668/) marker
that blocks installs into the system environment. `--user` does not bypass it.
`--break-system-packages` does, and is a bad idea on a Homebrew Python. Use
`pipx` or a venv.

</details>

<details>
<summary>Keeping it updated</summary>

```bash
pipx upgrade agent-security-harness
```

[topgrade](https://github.com/topgrade-rs/topgrade) picks this up automatically
through its `pipx` step, or its `pip3` step for a `--user` install. No
configuration needed.

There is no Homebrew formula for this package.

</details>

```bash
# Confirm which build you're on:
agent-security --version

# See it work immediately — no server needed:
agent-security test mcp --simulate

# Then test your real MCP server:
agent-security test mcp --url http://localhost:8080/mcp

# Test an x402 payment endpoint. Its probes predate the published x402 transport: against a
# spec v1/v2 server only 5 tests grade, the rest report INCONCLUSIVE and say why (see CHANGELOG).
agent-security test x402 --url https://your-x402-endpoint.com

# Human-oversight surface: reviewer exposure (T10) and agent-to-human
# manipulation (T15). An unreachable target reports INCONCLUSIVE, never PASS.
agent-security test hitl --url http://localhost:8080

# Which tests back a given OWASP threat, scenario or mitigation control
python scripts/owasp_agentic_select.py --threat T16
python scripts/owasp_agentic_select.py --control P5-REA-001
```

**Exit status.** Every `agent-security test <harness>` run, and every harness run directly
with `python -m protocol_tests.<module>`, exits with the same three codes:

| Code | Meaning |
|------|---------|
| `0` | every result PASSED |
| `1` | at least one genuine FAIL (whatever else the run contains) |
| `2` | no FAIL, but at least one result is INCONCLUSIVE or not executed, or no result was produced (this includes `--simulate`) |

Nonzero always means "not clean", so a pipeline that only checks for nonzero behaves as
before, except for `receipt-claim`, `cloud-agents`, `autogen`, `crewai-cve`,
`mcp-tool-poisoning` and `capability-residue`, which exited `0` whatever their results said
until 2026-09-24 and now follow this table (see CHANGELOG). Before 2026-09-24 most harnesses
exited `1` for an INCONCLUSIVE row as well as for a FAIL; branch on `1` specifically if you want to gate on a failed control. argparse also exits
`2` on a usage error, and that run writes no report. Exception: the `--trials N` statistical
paths of `l402` and `x402` still exit `0`/`1` (they do not track INCONCLUSIVE per trial).
Modules run directly with their own `--simulate` (`cloud-agents`,
`crewai-cve`, `mcp-tool-poisoning`) exit with what their reference rows say; through
`agent-security test ... --simulate` every row is INCONCLUSIVE and the run exits `2`.

See [docs/QUICKSTART.md](docs/QUICKSTART.md) for mock server setup, rate limiting, MCP server mode, and CI/CD integration.

---

## Three Layers of Agent Decision Security

| Layer | What it covers | Example focus |
|-------|----------------|---------------|
| **Protocol Integrity** | Prevent spoofing, replay, downgrade, diversion, and malformed protocol behavior | MCP, A2A, L402, x402 wire-level tests |
| **Operational Governance** | Validate session state, capability boundaries, platform actions, trust chains, and execution context | capability escalation, facilitator trust, provenance, session security |
| **Decision Governance** | Test whether an agent should act at all under its authority, confidence, scope, and policy constraints | autonomy scoring, scope creep, return-channel poisoning, normalization-of-deviance |
| **Human Oversight** | Test whether the human review layer can be saturated, starved, blinded, or turned against its own principal | approval flooding, risk starvation, stripped decision context, agent→human manipulation (OWASP T10/T15) |

---

## How This Differs From Other Projects

| Capability | [Snyk Agent Scan](https://github.com/snyk/agent-scan) (2.9K stars) | [Cisco MCP Scanner](https://github.com/cisco-ai-defense/mcp-scanner) (1.0K stars) | [NVIDIA Garak](https://github.com/NVIDIA/garak) (8.7K stars) | **This framework** |
|---|---|---|---|---|
| **What it does** | Scans agent/MCP configs for tool poisoning and skill security | YARA + LLM-as-judge for malicious tools | LLM model vulnerability testing | Active protocol exploitation + decision governance |
| **Approach** | Static analysis | Static + LLM classification | Model-layer probing | **Wire-protocol adversarial testing** |
| **MCP coverage** | Tool descriptions, config files | Tool descriptions, YARA rules | - | **46 tests: protocol (32) + supply-chain (4) + tool-poisoning repro (10), real JSON-RPC 2.0 attacks** |
| **A2A coverage** | - | - | - | **13 tests** |
| **L402/x402 coverage** | - | - | - | **85 tests** (L402 33 + x402 52); x402 probes use a pre-spec header format, so against a spec x402 v1/v2 server 5 tests grade and the rest are INCONCLUSIVE |
| **Merchant journey (UCP/ACP)** | - | - | - | **12 tests: agent-profile + cross-merchant cart + delegated checkout** |
| **Funding instrument (Visa TAP / MC Agentic Tokens)** | - | - | - | **12 tests: holder/merchant/amount/velocity scope + dynamic cryptogram + revocation** |
| **Settlement finality (denial-of-settlement)** | - | - | - | **8 tests: release-before-finality, reorg revocation, withheld-settlement liveness** |
| **Payment authz/hardening** | - | - | - | **AP2 mandate (17) + Fireblocks x402 (17)** |
| **Enterprise platforms** | - | - | - | **25 cloud + 58 enterprise** (core 31 + extended 27) |
| **Human oversight (T10/T15)** | - | - | - | **8 tests: reviewer exposure + agent→human manipulation** |
| **APT simulation** | - | - | - | **GTG-1002 (17 tests)** |
| **Jailbreak/over-refusal** | - | - | Yes | **50 tests (25 + 25 FPR)** |
| **AIUC-1 certification** | - | - | - | **E1 mapping: 23 of 50 active requirements listed, 19 with named tests** (release 2026-10-15; [crosswalk](docs/AIUC1-CROSSWALK.md)) |
| **OWASP Agentic v1.1** | - | - | - | **T1–T17 commit-pinned: 13 direct, 4 partial, 0 not evidenced** |
| **Research backing** | - | Cisco blog | Papers | **7 DOIs + 3 NIST submissions** (self-authored; see [Research](#research)) |
| **MCP server mode** | - | - | - | **Yes - invoke from any AI agent** |
| **Statistical testing** | - | - | - | **Wilson CIs, multi-trial** |
| **Total tests** | Config checks | YARA rules | Model probes | **649 active tests across 50 test-bearing modules** |

Star counts verified 2026-08-02 via the GitHub API. Invariant Labs' `mcp-scan` now redirects to
`snyk/agent-scan` and is listed once rather than as two separate projects. Competitor rows describe
what each tool does by design; a dash means the capability is outside its stated scope, not a defect.

**What the table does not measure.** Every row above is coverage: what gets tested, and how much.
Coverage says nothing about what a result licenses you to claim. That is what the
[E1-E5 Evidence Class Taxonomy](docs/EVIDENCE-CLASS-TAXONOMY.md) is for, and it is applied to this
project's own output first: the AIUC-1 crosswalk is E1 material however many of its
requirements it lists, and this harness holds **no I2 evidence** at all, because it reads protocol responses
the target itself emits. No claim here is deliberately made about how other projects handle this.

**Outside checks.** The `Research backing` row says self-authored because it is. The exceptions
are two reports from one outside party: [#304](https://github.com/msaleme/red-team-blue-team-agent-fabric/issues/304), which reimplemented
the receipt-claim oracle from the published contract and replayed a pinned corpus, and
[#622](https://github.com/msaleme/red-team-blue-team-agent-fabric/issues/622), which ran the released harness against targets the reporter
wrote and found four target shapes that still produced verdicts. Two reports from one party are not
independent review of the harness as a whole, and this project still does not have that.

**Use both.** Scan with [Snyk Agent Scan](https://github.com/snyk/agent-scan) or [Cisco MCP Scanner](https://github.com/cisco-ai-defense/mcp-scanner) for static analysis. Test with this framework for active exploitation. They're complementary layers.

---

## Research

Seven public preprints and notes deposited on Zenodo (not represented as peer-reviewed publications) and three NIST submissions underpin the methodology. Every DOI below was re-verified on 2026-08-02 by content negotiation against `doi.org` — title and authorship confirmed:

| Publication | DOI |
|---|---|
| **Constitutional Self-Governance for Autonomous AI Agents** — 12 governance mechanisms, 85 days production data, 48 agents | [10.5281/zenodo.19162103](https://doi.org/10.5281/zenodo.19162103) |
| **Detecting Normalization of Deviance in Multi-Agent Systems** — First empirical demonstration that automated harnesses detect behavioral drift | [10.5281/zenodo.19195515](https://doi.org/10.5281/zenodo.19195515) |
| **Decision Load Index (DLI): A Quantitative Framework for Agent Autonomy Risk** — Measuring cognitive burden of AI agent oversight | [10.5281/zenodo.18207847](https://doi.org/10.5281/zenodo.18207847) |
| **Beyond Identity Governance: A Protocol-Level Security Testing Framework for Multi-Agent Systems** | [10.5281/zenodo.19343033](https://doi.org/10.5281/zenodo.19343033) |
| **Community-Driven Security for AI Agents: Evolution of an Adversarial Test Corpus** | [10.5281/zenodo.19343107](https://doi.org/10.5281/zenodo.19343107) |
| **Claim-Level Negative Testing for Agent-Governance Evidence** — Receipt-claim decomposition; the RCL-001..011 receipt-verification module in this harness | [10.5281/zenodo.21418701](https://doi.org/10.5281/zenodo.21418701) |
| **Signing Is Not Authorization: Claim-Level Negative Vectors for Agent-Payment Receipts** — payment-authority application of the receipt-claim decomposition; RCL-001..011 under adversarial payment receipts | [10.5281/zenodo.21535452](https://doi.org/10.5281/zenodo.21535452) |

**On citation counts.** These records carry internal citation lineage across later work in this
portfolio. An OpenAlex `cited_by` audit on 2026-08-02 found **30 citation edges and 0 qualifying
independent citations** — every edge is a self-citation, collapsing to nine duplicate/version records
across three title families. Nothing here should be read as third-party validation or scholarly
adoption. The external checks this project has received are two reports from one party, listed
under [Outside checks](#outside-checks).

> **Correction (2026-08-02).** Two entries previously in this table cited DOIs belonging to other
> researchers — `10.5281/zenodo.15105866` (a MALDI mass-spectrometry dataset by Ranes et al.) and
> `10.5281/zenodo.15106553` (an e-learning article by Toshtemirov). They were attributed here to
> *"Normalization of Deviance in Autonomous Agent Systems"* and *"Cognitive Style Governance for
> Multi-Agent Deployments"*. No Zenodo record under those titles by this author was located, so both
> rows were removed rather than re-pointed. The surviving *Detecting Normalization of Deviance in
> Multi-Agent Systems* record (`19195516`) is unaffected and was already listed separately.

---

## Related Projects

### Constitutional Governance (WHY layer)

The [constitutional-agent](https://github.com/CognitiveThoughtEngine/constitutional-agent-governance) package provides the governance gates and hard constraints that complement this test harness. Six gates, 12 hard constraints, amendment protocol — enforced in code, not YAML policy files. `pip install constitutional-agent`.

### Architecture context (why this layer exists)

This harness tests whether a control *holds*. It does not say which controls an agent workforce
should have in the first place — that is an architecture question, and it is written up as
**Enterprise Agent Architecture**: the case for treating autonomous agents as a fifth architecture
domain alongside Business, Information, Application and Technology.

[cognitivethoughtengine.com/eaa](https://cognitivethoughtengine.com/eaa?utm_source=github&utm_medium=readme&utm_campaign=harness)
— the four-layer model (Agent/Workforce, Capability/Tool, Control Plane, Governance), a maturity
model, a standards landscape, and the live reference environment the telemetry above comes from.

---

## Documentation

| Resource | Link |
|---|---|
| Expanded Quick Start | [docs/QUICKSTART.md](docs/QUICKSTART.md) |
| Full Test Inventory (649 tests) | [docs/TEST-INVENTORY.md](docs/TEST-INVENTORY.md) |
| OWASP Agentic v1.1 Coverage (T1–T17) | [docs/OWASP-AGENTIC-V1.1-COVERAGE.md](docs/OWASP-AGENTIC-V1.1-COVERAGE.md) |
| Canonical coverage mapping (source of truth) | [docs/coverage/owasp-agentic-v1.1.yaml](docs/coverage/owasp-agentic-v1.1.yaml) |
| Testing for a published CVE: CVE to test index | [docs/cve/README.md](docs/cve/README.md) |
| Release history & known gaps | [ROADMAP.md](ROADMAP.md) · [CHANGELOG.md](CHANGELOG.md) |
| E1-E5 Evidence Class Taxonomy (canonical) | [docs/EVIDENCE-CLASS-TAXONOMY.md](docs/EVIDENCE-CLASS-TAXONOMY.md) |
| Reproducing this harness (and disagreeing with it) | [docs/REPRODUCING.md](docs/REPRODUCING.md) |
| The external reimplementation, written up (E2/I1 for the fixture corpus) | [docs/RCL-CROSS-IMPLEMENTATION-INTEROP.md](docs/RCL-CROSS-IMPLEMENTATION-INTEROP.md) |
| The acceptance criterion that does not require the mechanism (5 recorded instances) | [docs/VERIFICATION-DESIGN-DEFECT.md](docs/VERIFICATION-DESIGN-DEFECT.md) |
| Attestation registry (client, opt-in publishing) | [docs/attestation-registry.md](docs/attestation-registry.md) |
| Attestation registry server contract (self-hosted; no registry is operated by this project) | [docs/ATTESTATION-REGISTRY-SERVER-CONTRACT.md](docs/ATTESTATION-REGISTRY-SERVER-CONTRACT.md) |
| AIUC-1 Evidence Field Guide (external, not version-pinned) | [msaleme.github.io/aiuc1-readiness](https://msaleme.github.io/aiuc1-readiness/) |
| AIUC-1 Crosswalk | [docs/AIUC1-CROSSWALK.md](docs/AIUC1-CROSSWALK.md) |
| Advanced Capabilities | [docs/ADVANCED.md](docs/ADVANCED.md) |
| MCP Server | [docs/mcp-server.md](docs/mcp-server.md) |
| CI/CD GitHub Action | [docs/github-action.md](docs/github-action.md) |
| Payment Attack Taxonomy | [docs/PAYMENT-ATTACK-TAXONOMY.md](docs/PAYMENT-ATTACK-TAXONOMY.md) |
| Decision Governance Checklist | [docs/DECISION-GOVERNANCE-CHECKLIST.md](docs/DECISION-GOVERNANCE-CHECKLIST.md) |
| Decision Governance Benchmark Leaderboard | [benchmarks/LEADERBOARD.md](benchmarks/LEADERBOARD.md) |
| Related Work | [docs/RELATED-WORK.md](docs/RELATED-WORK.md) |
| Comparison (detailed) | [docs/COMPARISON.md](docs/COMPARISON.md) |
| Privacy & Telemetry | [docs/PRIVACY.md](docs/PRIVACY.md) |

---

## Roadmap

**Current: v4.28.2** (2026-11-02). Recent shipped work — v4.5 skill supply chain and governance
modification · v4.6–v4.9 payment-stack depth (AP2 mandate chain, UCP/ACP merchant journey, card-network
agentic tokens, settlement finality, Fireblocks x402) · v4.10 benchmark integrity · v4.11–v4.12
decision-governance corpus currency and provenance repair · **v4.13 OWASP Agentic v1.1 T1–T17 coverage
mapping and the human-in-the-loop harness** · v4.13.1 a correctness fix to that harness · v4.14.0
endpoint provenance · v4.16.0 three target shapes: a verdict must be able to be wrong AND to be right · v4.17.0 eight modules could not tell a refusal from a compliance · v4.18.0 INCONCLUSIVE became a field, and the read-list emptied · v4.19.0 a correctness disclosure: verdicts moved in both directions · v4.20.0 the last absence-graded verdicts got a positive control · v4.21.0 the instrument was the thing under test: 7 of 9 defects were in the measurement apparatus · v4.21.1 the published wheel reported 17/17 against a host that did not exist · v4.21.2 a fourth review asked what the repaired layer assumes: 47 rows called a control held on a reply that said nothing · v4.21.3 a fifth review asked whether the checks can fail: the workflow contract tests passed with every step disabled, and nine of twelve register denominators could be replaced by a constant · v4.22.0 `scripts` shipped as a namespace portion and the console printed INCONCLUSIVE as FAIL in twenty-nine harnesses; four new families took the release from 623 to 640 · v4.23.0 exit status separates FAIL (1) from INCONCLUSIVE (2), six harnesses that always exited 0 now follow that contract, an MCP bootstrap failure emits NOT_EXECUTED rows, and the Action's default `fail_on: critical`, which could never fire, now does · v4.24.0 a verdict needs a target that answered: a guard runs every URL-taking harness against a closed port, a 404-everywhere and a 403-everywhere target, where a PASS or FAIL is a defect; the A2A, cloud-agent, CrewAI CVE, CVE-005 and REST adapter families were repaired and left its register (578 cells to 340) · v4.25.0 the no-surface register is empty: the x402/L402, refusal-recognising, AIUC-1/extended-thinking/PTC and over-refusal/prompt-caching/GTG-1002 families were repaired (340 cells to 0), so every URL-taking harness returns INCONCLUSIVE against those three targets · v4.26.0 an external reproduction (#622, VrtxOmega) confirmed the v4.25.0 claim on its three targets and found four more shapes still producing verdicts: a redirect loop, an empty 500, an empty 200 and an empty 204; one shared rule, `http_helpers.contentless_answer`, makes them INCONCLUSIVE (346 cells to 0), and the guard now covers nine poles with both registers empty · v4.26.1 served answers reach the right verdict (#631, #628, VrtxOmega): capability-profile (CP-001..010), INT-001 and MAG-019 can PASS on a served answer, and autogen keeps the real HTTP status on JSON answers; two false PASSes in v4.26.0 are closed (a JSON `null`, `[]` or `""` at 200, and a plain-text complying answer, each PASSed all ten AG-* rows; both are now INCONCLUSIVE) · v4.27.0 two new test families for checks that exist but do not run: MCP-022 sends a validly issued token for another resource server (GHSA-w4fh-qvv9-3v23, whose fix ships off by default), and APB-001..APB-003 test what a remembered human approval authorizes at runtime (bound to its arguments, single-use where declared, a declared deny rule enforced at execution); the two families took the release from 640 to 644 · v4.28.0 ACSH-001..ACSH-005 test whether an agent honors an OWASP ACS Guardian's decision, observed at a sink tool rather than taken from the decision event; and a correctness disclosure (#653): six stateful live rows (CTK-004, CTK-005, AP2-011, AP2-013, UCP-006, ACP-004) sent only their final request, so every release through v4.27.0 could grade a correct target FAIL; they now replay their setup and grade the final step only after the setup was accepted; 644 to 649 tests · v4.28.1 a correctness fix (#657): APB-002 could PASS on an execution record that de-duplicates or keeps only the latest entry; every APB PASS now needs a record shown append-only between reads, and APB-002 first requires a fresh call approved and run twice to appear twice, else INCONCLUSIVE; 649 tests, unchanged · **v4.28.2 two corrections from an outside review: the AIUC-1 mapping carries the published 2026-10-15 titles and no coverage percentage (#671), and a spec x402 server gets INCONCLUSIVE on the 49 tests whose pre-spec probes it never reads, where it had received 43 PASS and 11 FAIL (#672); 649 tests, unchanged** · v4.15.0 unserviced requests are no longer recorded as passes; that did not hold in every harness until v4.25.0, and not on every shape until v4.26.0. Since v4.26.0 the guard in `testing/test_verdicts_need_a_surface.py` runs every URL-taking harness against nine poles: a closed port, a 404-everywhere and a bare 403-everywhere target (since v4.25.0), and a same-location 302 redirect loop, an empty HTTP 500, an empty 200, an empty 204, a bare 401 and a TLS failure (since v4.26.0; VrtxOmega reported the first four in #622, the last two were already clean and are kept as regression poles). The first three poles have a shrink-only register, `VERDICT_WITHOUT_SURFACE`, which is **empty**; the other six have their own, `VERDICT_ON_A_CONTENTLESS_ANSWER`, which is **empty**; and the guard fails any new PASS or FAIL returned on any of the nine. It names three exceptions: A2A-001 FAILs on 404 (an Agent Card that cannot be fetched, the #594 contract); X4-001 and L4-001 FAIL on an empty 200 or 204, because an unpaid request was served without a payment challenge ("Expected HTTP 402, got 2xx"; every other x402/L402 row on an empty 2xx is INCONCLUSIVE, owner decision on #622); and the self-tests CREW-002, CVE-007 and CVE-008 PASS everywhere because they judge this repository's scanner on fixtures, not the target (see
[CHANGELOG.md](CHANGELOG.md)).

The release carries **649** tests; `main` is at **649**. They agree as of v4.28.2, and the
two numbers stay separately stated because that is temporary: the next merge to `main`
reopens the gap. They did not agree at v4.21.3, which
carried 623 while `main` stood at 640: the difference was MEM-013..MEM-016, the memory
revocation-enforcement family, WT-001..WT-004, the workspace-trust ingestion family,
HI-001..HI-004, the hidden-instruction family, and CR-001..CR-005, the capability-residue
family, all of which landed after that tag. The same divergence held at v4.15.0, where the
release carried 603 while `main` was at 608. Both numbers were correct for their own ref,
and agreement, when it holds, is a property of a fresh release rather than a guarantee.
That is why
`scripts/check_public_metadata.py` compares public claims against the release rather than against `main`,
and why `docs/release-claims.json` pins the release count to a commit where re-running the command
reproduces it.

**Next — Standards & Evidence.** Reproducible settlement-time payment evidence, a methodology paper,
and the attestation/evidence schema submitted to a standards venue. Coverage breadth and test count are
explicitly *not* goals. Full detail and the anti-goals in [ROADMAP.md](ROADMAP.md).

---

## Used By

| Who | Use Case |
|-----|----------|
| [FransDevelopment / Open Agent Trust Registry](https://github.com/FransDevelopment/open-agent-trust-registry) | OATR SDK v1.2.0 test fixtures (X4-021 through X4-027) -- Ed25519 attestation verification |

### Outside checks

This project has received two external reports, both from one party,
[@VrtxOmega](https://github.com/VrtxOmega), named with their consent (granted 2026-08-31 on #304 and
2026-09-26 on #622). Neither is independent review of the harness as a whole, which this project
still does not have.

**A second implementation (I1 for the receipt-claim corpus).** In
[veritas-agent-trust-lab](https://github.com/VrtxOmega/veritas-agent-trust-lab), they wrote a **separate Node
verifier** for the portable receipt-claim oracle fixtures and ran it against a pinned commit and fixture
hash, matching all 11 RCL results including both acceptance controls — so it did not pass by rejecting
everything. Reported in [#304](https://github.com/msaleme/red-team-blue-team-agent-fabric/issues/304).
The result is written up in
[docs/RCL-CROSS-IMPLEMENTATION-INTEROP.md](docs/RCL-CROSS-IMPLEMENTATION-INTEROP.md),
with their own report and verifier runner linked as the primary artifacts.

The exchange also produced two corrections to this repository: `signature_algorithm` now names the
actual encoding and states plainly that it is **not** RFC 8785 JCS, and `coverage_gaps` now declares
that freshness is exercised only in the stale direction. Both are in `fixtures/rcl/`.

**A run against their own targets (I0 for the harness).** In
[#622](https://github.com/msaleme/red-team-blue-team-agent-fabric/issues/622), they installed the published v4.25.0 wheel, ran it against
targets they wrote, and classified the rows themselves. The release claim held on the three shapes
it named: against a closed port, a 404-everywhere and a 403-everywhere target, no URL-taking harness
returned PASS or FAIL beyond the declared exceptions. They then tried six shapes the claim did not
name, and four still produced verdicts: a redirect loop (45 rows), an empty 500 (35), an empty 200
(134) and an empty 204 (144). Those four are now poles in the CI guard, and v4.26.0 returns
INCONCLUSIVE on them, except X4-001 and L4-001, which FAIL on an empty 200 or 204 by design (an
unpaid request served without a payment challenge). Their [public, rerunnable retest](https://github.com/VrtxOmega/veritas-agent-trust-lab/actions/runs/36254603030)
of the published v4.26.0 wheel found no unexpected PASS or FAIL on any of the nine poles. Running
our code is I0 by the table in [`docs/REPRODUCING.md`](docs/REPRODUCING.md): it shows where a stated
claim holds and where it breaks, not that the verdict logic is right.

Both were submitted as reports and explicitly **not** as contributions, endorsements,
certifications, or adoption.

**"A third independent implementation is worth more to this corpus than any additional vector."**
That is [`fixtures/rcl/CONFORMANCE.md`](fixtures/rcl/CONFORMANCE.md) §7, verbatim, and it is not
modesty. Two implementations agreeing establishes that they share a reading; only a third can tell
you whether that reading is the contract or a coincidence.

The bar is lower than it sounds. The corpus is eleven vectors, one fixture file, and a pinned hash;
the run above was a single Node script written against the published format.
[`docs/REPRODUCING.md`](docs/REPRODUCING.md) has the pin, the canonical byte basis that is the most
common reason a correct implementation computes a different digest, and the table of what does and
does not count as independent. §7 of `CONFORMANCE.md` adds the one rule that decides whether your
result counts at all: **write the verifier from the published contract, not from
`protocol_tests/receipt_claim_harness.py`.** Reading our implementation first turns an independent
result into a port, and the value is entirely in the independence. File it with the
[reproduction report template](https://github.com/msaleme/red-team-blue-team-agent-fabric/issues/new?template=reproduction-report.yml).

**A result that contradicts ours gets published too.** #304 matched 11 of 11 and *still* surfaced
two defects in our own contract description, both corrected above. #622 confirmed the claim as
stated and then broke it on four shapes the claim had not named; the fix and its counts are in the
[v4.26.0 changelog](CHANGELOG.md#4260---2026-09-26). Disagreement is the more useful outcome, not the awkward one — a reproduction that only
agrees may just mean two implementations share the same wrong assumption.

### Merged outside contributions

| PR | From | What it added |
|----|------|---------------|
| [#101](https://github.com/msaleme/red-team-blue-team-agent-fabric/pull/101) | [FransDevelopment](https://github.com/FransDevelopment) | OATR SDK v1.2.0 test fixtures: suspended issuer and grace period |
| [#516](https://github.com/msaleme/red-team-blue-team-agent-fabric/pull/516) | [Salento91](https://github.com/Salento91) | Four community fixtures for constraint preservation across delegation: a constraint kept, safely narrowed, dropped or widened (`community_modules/contrib/`) |

A merged contribution is code this project reviewed and accepted. It is not validation of the harness.

*Using the harness? Open a PR to add yourself, or tag us in your project.*

---

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines, [SECURITY_POLICY.md](SECURITY_POLICY.md) for security policy, and [CONTRIBUTION_REVIEW_CHECKLIST.md](CONTRIBUTION_REVIEW_CHECKLIST.md) for the PR checklist.

## Citation

If you cite this work in research:

> Saleme, M. K. (2026). *Agent Security Harness* — multi-protocol agent security testing framework. ORCID: [0009-0003-6736-1900](https://orcid.org/0009-0003-6736-1900). https://github.com/msaleme/red-team-blue-team-agent-fabric

Related Zenodo preprints are listed in full under [Research](#research); that table is the single
source of truth for DOIs in this repository. Do not cite a DOI for this work that does not appear there.

---

## License

Apache License 2.0 -- see [LICENSE](LICENSE).
