<!-- GENERATED by benchmarks/generate_head_to_head_doc.py — edits here will be overwritten. Change the generator and regenerate. -->

# AgentArmor — Head-to-head Comparison

**Results as of: AgentArmor v1.4.0, run 2026-04-19**

This document compares AgentArmor against two established safety classifiers — LlamaGuard 3 (local, Q4_K_M via llama-cpp-python) and OpenAI Moderation (omni-moderation-latest) — across six industry datasets with per-sample verdicts and bootstrap confidence intervals. Perspective API was considered but excluded from v1: Google/Jigsaw announced sunset with API EOL 2026-12-31, which would leave any published comparison non-reproducible within months. We publish per-dataset numbers only: no overall-winner claim, and we annotate every cell where AgentArmor loses or ties within CI. **One honest weakness surfaces in the numbers below: on XSTest (over-refusal probes), AgentArmor's toxicity module is heavy-handed (F1=0.16 vs LlamaGuard 0.83) — specialist classifiers flag figurative/homonym prompts that general-purpose safety LLMs correctly pass. Tracked as a follow-up for threshold tuning or post-filter.** Reproduction: set `OPENAI_API_KEY` and run `python -m benchmarks.run_head_to_head --run-dir benchmarks/results/runs/<ts>` (see [`RUNBOOK.md`](RUNBOOK.md) Procedure 0).

### Per-dataset summary

| Dataset | Vendor best | Vendor F1 | AgentArmor best | AgentArmor F1 | Δ (AgentArmor − vendor) |
| --- | --- | --- | --- | --- | --- |
| `advbench` | `llamaguard` | 0.9684 | `agentarmor_combined` | 0.9684 | 0.0000 |
| `harmbench` | `llamaguard` | 0.9892 | `agentarmor_combined` | 0.8675 | -0.1218 |
| `jailbreakbench` | `llamaguard` | 0.8929 | `agentarmor_toxicity` | 0.6866 | -0.2063 |
| `realtoxicity` | `llamaguard` | 0.1379 | `agentarmor_combined` | 0.5600 | 0.4221 |
| `toxigen` | `llamaguard` | 0.2143 | `agentarmor_combined` | 0.8095 | 0.5952 |
| `xstest` | `llamaguard` | 0.8293 | `agentarmor_combined` | 0.1600 | -0.6693 |

### Operating-point legend

| Baseline | Operating point |
| --- | --- |
| `agentarmor_combined` | All three AgentArmor modules OR'd (production deployment shape) |
| `agentarmor_ml_shield` | AgentArmor ML shield τ=0.65 (our shipping default) |
| `agentarmor_shield` | AgentArmor regex shield (bool; no PR curve — regex-by-design) |
| `agentarmor_toxicity` | AgentArmor toxicity filter at shipping config |
| `llamaguard` | P("unsafe") from Q4_K_M first-token logprobs (our quantization choice) |
| `openai_moderation` | max over per-dataset category projection (see rubric) |

## Per-dataset detail

### advbench

| Baseline | F1 | Precision | Recall | FPR | MCC | Bal-Acc | Latency p50 | Latency p95 | N | Errors |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `agentarmor_combined` | 0.9684 (CI 0.9247–1.0000) | 1.0000 | 0.9388 | 0.0000 | — | — | 0.8 ms | 2.7 ms | 50 | 0 |
| `agentarmor_ml_shield` | 0.8864 (CI 0.8095–0.9474) | 1.0000 | 0.7959 | 0.0000 | — | — | 0.4 ms | 0.8 ms | 50 | 0 |
| `agentarmor_shield` | 0.3934 (CI 0.2264–0.5373) | 1.0000 | 0.2449 | 0.0000 | — | — | 0.1 ms | 0.1 ms | 50 | 0 |
| `agentarmor_toxicity` | 0.9574 (CI 0.9111–0.9899) | 1.0000 | 0.9184 | 0.0000 | — | — | 0.6 ms | 0.8 ms | 50 | 0 |
| `llamaguard` | 0.9684 (CI 0.9247–1.0000) | 1.0000 | 0.9388 | 0.0000 | — | — | 1591.0 ms | 2000.8 ms | 50 | 0 |
| `openai_moderation` | 0.8333 (CI 0.7368–0.9111) | 1.0000 | 0.7143 | 0.0000 | — | — | 234.0 ms | 939.4 ms | 50 | 0 |

_AdvBench is reported with MCC + balanced-accuracy only; F1 omitted due to the 520:15 imbalance baseline-rate dominating F1 (SPEC v4 D38)._

### harmbench

| Baseline | F1 | Precision | Recall | FPR | MCC | Bal-Acc | Latency p50 | Latency p95 | N | Errors |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `agentarmor_combined` | 0.8675 (CI 0.7792–0.9348) | 1.0000 | 0.7660 | 0.0000 | 0.4051 (CI 0.2182–0.6230) | 0.8830 (CI 0.8191–0.9388) | 0.7 ms | 1.4 ms | 50 | 0 |
| `agentarmor_ml_shield` | 0.7467 (CI 0.6286–0.8462) | 1.0000 | 0.5957 | 0.0000 | 0.2850 (CI 0.1487–0.4504) | 0.7979 (CI 0.7283–0.8673) | 0.7 ms | 1.1 ms | 50 | 0 |
| `agentarmor_shield` | — | — | 0.0000 | 0.0000 | — | 0.5000 (CI 0.5000–0.5000) | 0.1 ms | 0.1 ms | 50 | 0 |
| `agentarmor_toxicity` | 0.8675 (CI 0.7792–0.9362) | 1.0000 | 0.7660 | 0.0000 | 0.4051 (CI 0.2182–0.6276) | 0.8830 (CI 0.8191–0.9419) | 0.6 ms | 1.2 ms | 50 | 0 |
| `llamaguard` | 0.9892 (CI 0.9655–1.0000) | 1.0000 | 0.9787 | 0.0000 | 0.8568 (CI 0.5654–1.0000) | 0.9894 (CI 0.9667–1.0000) | 1804.3 ms | 1986.7 ms | 50 | 0 |
| `openai_moderation` | 0.5538 (CI 0.4000–0.6933) | 1.0000 | 0.3830 | 0.0000 | 0.1895 (CI 0.0935–0.3026) | 0.6915 (CI 0.6250–0.7660) | 171.0 ms | 616.4 ms | 50 | 0 |

### jailbreakbench

| Baseline | F1 | Precision | Recall | FPR | MCC | Bal-Acc | Latency p50 | Latency p95 | N | Errors |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `agentarmor_combined` | 0.6765 (CI 0.5357–0.8000) | 0.5349 | 0.9200 | 0.8000 | 0.1729 (CI -0.1068–0.4193) | 0.5600 (CI 0.4675–0.6600) | 1.0 ms | 1.8 ms | 50 | 0 |
| `agentarmor_ml_shield` | 0.6383 (CI 0.4583–0.7805) | 0.6818 | 0.6000 | 0.2800 | 0.3223 (CI 0.0531–0.5733) | 0.6600 (CI 0.5256–0.7841) | 0.5 ms | 0.9 ms | 50 | 0 |
| `agentarmor_shield` | 0.2759 (CI 0.0769–0.4848) | 1.0000 | 0.1600 | 0.0000 | 0.2949 (CI 0.1429–0.4501) | 0.5800 (CI 0.5185–0.6591) | 0.1 ms | 0.1 ms | 50 | 0 |
| `agentarmor_toxicity` | 0.6866 (CI 0.5484–0.8000) | 0.5476 | 0.9200 | 0.7600 | 0.2182 (CI -0.0528–0.4552) | 0.5800 (CI 0.4821–0.6842) | 0.6 ms | 0.8 ms | 50 | 0 |
| `llamaguard` | 0.8929 (CI 0.7907–0.9667) | 0.8065 | 1.0000 | 0.2400 | 0.7829 (CI 0.6340–0.9226) | 0.8800 (CI 0.7917–0.9583) | 1717.6 ms | 1987.9 ms | 50 | 0 |
| `openai_moderation` | 0.7179 (CI 0.5333–0.8571) | 1.0000 | 0.5600 | 0.0000 | 0.6236 (CI 0.4615–0.7845) | 0.7800 (CI 0.6818–0.8750) | 165.5 ms | 283.7 ms | 50 | 0 |

### realtoxicity

| Baseline | F1 | Precision | Recall | FPR | MCC | Bal-Acc | Latency p50 | Latency p95 | N | Errors |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `agentarmor_combined` | 0.5600 (CI 0.3750–0.7097) | 0.5600 | 0.5600 | 0.4400 | 0.1200 (CI -0.1639–0.3984) | 0.5600 (CI 0.4179–0.6987) | 0.9 ms | 1.7 ms | 50 | 0 |
| `agentarmor_ml_shield` | 0.3226 (CI 0.0952–0.5263) | 0.8333 | 0.2000 | 0.0400 | 0.2462 (CI -0.0140–0.4501) | 0.5800 (CI 0.4959–0.6724) | 0.6 ms | 1.0 ms | 50 | 0 |
| `agentarmor_shield` | — | — | 0.0000 | 0.0000 | — | 0.5000 (CI 0.5000–0.5000) | 0.1 ms | 0.1 ms | 50 | 0 |
| `agentarmor_toxicity` | 0.5600 (CI 0.3721–0.7097) | 0.5600 | 0.5600 | 0.4400 | 0.1200 (CI -0.1612–0.3984) | 0.5600 (CI 0.4200–0.6997) | 0.6 ms | 0.9 ms | 50 | 0 |
| `llamaguard` | 0.1379 (CI 0.0000–0.3200) | 0.5000 | 0.0800 | 0.0800 | 0.0000 (CI -0.2722–0.2737) | 0.5000 (CI 0.4231–0.5800) | 1698.9 ms | 2108.3 ms | 50 | 0 |
| `openai_moderation` | — | 1.0000 | 0.0800 | 0.0000 | — | 0.5400 (CI 0.5000–0.6000) | 222.9 ms | 297.8 ms | 50 | 0 |

### toxigen

| Baseline | F1 | Precision | Recall | FPR | MCC | Bal-Acc | Latency p50 | Latency p95 | N | Errors |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `agentarmor_combined` | 0.8095 (CI 0.7013–0.8889) | 1.0000 | 0.6800 | — | — | — | 1.0 ms | 1.8 ms | 50 | 0 |
| `agentarmor_ml_shield` | 0.3333 (CI 0.1818–0.4848) | 1.0000 | 0.2000 | — | — | — | 0.5 ms | 0.9 ms | 50 | 0 |
| `agentarmor_shield` | — | — | 0.0000 | — | — | — | 0.1 ms | 0.1 ms | 50 | 0 |
| `agentarmor_toxicity` | 0.7952 (CI 0.6842–0.8764) | 1.0000 | 0.6600 | — | — | — | 0.6 ms | 1.2 ms | 50 | 0 |
| `llamaguard` | 0.2143 (CI 0.0769–0.3607) | 1.0000 | 0.1200 | — | — | — | 1767.4 ms | 2065.1 ms | 50 | 0 |
| `openai_moderation` | 0.2143 (CI 0.0769–0.3607) | 1.0000 | 0.1200 | — | — | — | 156.8 ms | 290.7 ms | 50 | 0 |

### xstest

| Baseline | F1 | Precision | Recall | FPR | MCC | Bal-Acc | Latency p50 | Latency p95 | N | Errors |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `agentarmor_combined` | 0.1600 (CI 0.0000–0.3636) | 0.6667 | 0.0909 | 0.0357 | 0.1154 (CI -0.1737–0.3469) | 0.5276 (CI 0.4630–0.6025) | 0.8 ms | 1.1 ms | 50 | 0 |
| `agentarmor_ml_shield` | — | — | 0.0000 | 0.0000 | — | 0.5000 (CI 0.5000–0.5000) | 0.5 ms | 0.8 ms | 50 | 0 |
| `agentarmor_shield` | — | 0.0000 | 0.0000 | 0.0357 | — | 0.4821 (CI 0.4400–0.5000) | 0.0 ms | 0.1 ms | 50 | 0 |
| `agentarmor_toxicity` | 0.1600 (CI 0.0000–0.3636) | 0.6667 | 0.0909 | 0.0357 | 0.1154 (CI -0.1737–0.3469) | 0.5276 (CI 0.4618–0.6025) | 0.5 ms | 1.0 ms | 50 | 0 |
| `llamaguard` | 0.8293 (CI 0.6857–0.9375) | 0.8947 | 0.7727 | 0.0714 | 0.7172 (CI 0.5194–0.8847) | 0.8506 (CI 0.7468–0.9417) | 1544.3 ms | 1672.8 ms | 50 | 0 |
| `openai_moderation` | 0.6667 (CI 0.4444–0.8293) | 1.0000 | 0.5000 | 0.0000 | 0.5991 (CI 0.4258–0.7693) | 0.7500 (CI 0.6429–0.8542) | 146.1 ms | 313.5 ms | 50 | 0 |

## Appendices

### PR curves (appendix)

Precision–recall curves are rendered only for score-emitting `(baseline, dataset)` pairs on the six main-comparison datasets, and only when at least 50 positive samples are present (below that, bootstrap PR is too noisy to publish honestly — SPEC v4 D12).

**Why a PR curve is missing (D48):**

- `shield` (regex; bool-by-design, no curve)
- `exfiltration_guard` (regex; bool-by-design, no curve)
- `unicode_shield` (regex; bool-by-design, no curve)
- `llamaguard` (when the logprobs gate fell back to text-parse — cell scores 0/1 only; see D36 and TYPE_WALKTHROUGH.md V1)
- Cells with fewer than 50 positives (noise floor)

_Curves are not yet rendered in this preview; the schema retains the per-sample `pred_score` needed to generate them from `head_to_head_verdicts.jsonl`._

### Cells marked `does_not_apply`

The applicability rubric (`benchmarks/taxonomy_applicability.py`) declares a (baseline, dataset) pair inapplicable when the baseline's taxonomy does not meaningfully overlap the dataset's labeling construct. Those cells are omitted from the comparison tables above.

| Baseline | Dataset | Rationale |
| --- | --- | --- |
| `perspective` | `advbench` | Perspective API sunset + taxonomic mismatch with adversarial jailbreaks. |
| `perspective` | `harmbench` | Perspective API sunset + taxonomic mismatch (toxicity != harmful-behavior elicitation). |
| `perspective` | `jailbreakbench` | Perspective API sunset + taxonomic mismatch (jailbreak prompts rarely toxic in surface form). |
| `perspective` | `realtoxicity` | Perspective API sunset (see sibling entries). Historical Perspective/RTP numbers are documented in the original Perspective papers; we did not attempt to replicate them against a dying API. |
| `perspective` | `toxigen` | Perspective API sunset (see sibling entries). |
| `perspective` | `xstest` | Perspective API is being sunset by Google/Jigsaw (no new requests after Feb 2026; API EOL 2026-12-31). Excluded from v1 to keep the comparison reproducible beyond the sunset date. |

### Methodology notes

- **Bootstrap F1 CI:** 5000 iterations, per-cell seed derived from `sha256(baseline|dataset)` (SPEC v4 D44/D45). `random.Random`, not numpy — survives minor-version bumps across the pinned `numpy<2.0`.
- **Per-metric degenerate guards (D41):** F1, MCC, and balanced-accuracy each have distinct zero-denominator conditions; cells flagged "insufficient data for CI" (`degenerate_flag=true`) report the point estimate alone.
- **Reproducibility tolerances (D16/D39):** `same_day_spread_95p` and `one_week_drift_max` are published once the calibration runs complete — see RUNBOOK.md for the procedure.
- **Hashes, not bodies (D49):** per-sample JSONL carries `raw_response: null` in committed artifacts. Use `--keep-raw-responses` for local debug only.
- **Live document:** this URL always reflects the latest tag. External links citing a specific version should pin by `git tag` (see Historical versions below).

### Historical versions

Each published version is an immutable git tag. To cite a specific version in blog posts or external links, use `https://github.com/<org>/AgentArmor/blob/<tag>/BENCHMARKS_HEAD_TO_HEAD.md`.

_(History table populated by `scripts/publish_head_to_head.sh` on each release.)_
