Mission Assurance Commons · open evidence

Make agent trust
inspectable.

Turn real public missions into safe synthetic tests, trace delegated authority across agent graphs, catch regressions, and export decision-ready evidence—without automating the accountable decision.

Enter the commons Run it locally
Not a safety certificate. A measurable regression floor with every limitation visible.
Observed robustness spread 25→99% Same harness. Same attacks.
Different base models.
14models measured
10model families
5 bridgespolicy + identity ecosystems
6 graph twinsmulti-agent authority paths
New · Mission Assurance Commons

A public workbench for
AI missions that matter.

Start with a bounded use case—not a vendor claim. Generate synthetic evaluation drafts, exercise every delegation edge, compare verified evidence over time, and package the result for accountable review.

01 / SCOPEInventoryForge

Normalize a public AI inventory and draft a human-review-required MissionPack.

inventory draft-pack
02 / TRACEAgentGraphTwin

Find the first unsafe authorization edge and calculate synthetic blast radius.

graph run
03 / WATCHContinuousProof

Compare content-addressed baselines and surface owner-threshold regressions.

watch compare
04 / DECIDEAcquisitionProof

Export neutral test plans, QASP inputs, cost fields, and portability checks.

acquisition export
frozen protocol · local execution

See where delegated authority breaks.

AgentGraphTwin keeps the mission fixed while mutating exactly one security-relevant edge. Receipts bind the request, adapter, policy, principal, decision, and reason.

Pairs
06
Graph hops
03
Real effects
00
principalHumanintent bound
delegate
agent / 01Orchestratorscope: records.read
scope inflation blocked
agent / 02Specialistleast privilege
authorize
resourceMCP toolno effect
FIRST UNSAFE EDGEedge-02reason: scope_not_granted
AuthorityBridge / translation contractsBring the policy plane you already operate.
  • OPARego decision
  • Cedarauthorization response
  • OpenFGArelationship check
  • OAuth + MCPtoken-bound tools
  • SPIFFEworkload identity

Compatibility fixtures validate translation into the benchmark contract. They do not execute, certify, or endorse the named backend.

$ dspy-security-bench graph demo
Open the implementation guide
The signal capability charts miss

Capability and security are different axes.

Two models can complete benign tasks at nearly the same rate and still have radically different prompt-injection failure profiles. Security must be measured directly.

62 point robustness gap between two models with near-identical capability
Interactive map

Capability × robustness

RobustMixedVulnerableProvisional

Hover or focus a model. Provisional measurements use a dashed ring.

Frozen protocol leaderboard

See which models
hold the boundary.

Robustness is the share of prompt-injection attacks that failed. Capability is benign task completion. Confidence intervals are cluster-bootstrapped over task pairs.

Rank / modelFamilyRobustnessCapabilityEvidence
Confirmed only when the full CI stays inside one bucket across all repeats.Read the complete methodology
From benchmark to guardrail

Make injection safety
a merge condition.

Scaffold a config and GitHub Action, preview the exact test matrix without spending API credits, then block regressions with terminal, JSON, and SARIF reports.

  • 01Works with DSPy or any Python agent factory
  • 02Absolute security floors or baseline regression gates
  • 03OWASP LLM01 · NIST AI 100-2 · MITRE ATLAS mappings
quickstart.sh
# install and create a ready-to-run gate
$ pip install dspy-security-bench
$ dspy-security-bench init --model openai/gpt-4o-mini

# inspect the matrix — zero model calls
$ dspy-security-bench scan --config .dspy-security-bench.yaml --plan

✓ 10 benchmark cases planned
✓ no model was called
Open the five-minute guide
Bring your own agent · v0.8

Your framework.
One evidence contract.

Keep your orchestration code and tools. The integration assistant detects a directly declared framework, generates a reviewable adapter, and validates the wiring without invoking the agent run loop.

01 / Scaffold
$dspy-security-bench integrate
Detectedopenai-agentspyproject.toml
+ dspy_security_target.py + integration.json + dspy-proofrun.yml
agent boundary
DSB
same attacks
same evidence
02 / Verify
dspy-security-bench doctorno agent runs
  • ✓ Manifest is valid
  • ✓ Fresh agent factory
  • ✓ Immutable ProofRun ref
  • ✓ Attestation permissions
Open integration guide
OA
OpenAIAgents SDK
LC
LangChainand LangGraph
PY
Pydantic AItyped agents
CR
CrewAIcrews and tools
AG
AutoGenAgentChat
↔
MCP / customany Python loop

Generated workflows are manual by default, so connecting a repository cannot unexpectedly spend model credits.

ProofRun · evidence with a chain of custody

Run it. Sign it.
Let others verify it.

v0.14 five-path evidence builder

A reusable workflow isolates model evaluation from signing, recomputes every statistic in a clean job, and binds the exact JSON to a source commit before enforcing the gate.

DSB / PROOFRUN 01 evidence passport
Verification ladder Bytes are easy.
Provenance is earned.
  1. 1
    Self-attestedstatistics + canonical hash
  2. 2
    GitHub-attestedrunner + workflow + commit
  3. 3
    Trusted builderimmutable central workflow
  4. 4
    Reproducedindependent maintainer rerun
artifact subject sha256: 9e81…c4f2 signature verification is separate from score recomputation
agent-security.yml
permissions:
  contents: read
  id-token: write
  attestations: write

jobs:
  proofrun:
    uses: immu4989/dspy-security-bench/.github/workflows/proofrun.yml@v0.15.0
    with:
      agent: myapp.security:build_agent
      trials: 10
Open evidence network 0 community ProofRuns

Every accepted entry preserves all raw trials and passes offline recomputation. A cryptographic badge appears only after the exact digest passes attestation verification and enters the reviewed registry.

Run your agent
Community evidenceranked by evidence tier, then lower confidence bound
The network is open. Be the first independent agent team to submit an attested run.

Provenance authenticates a workflow and exact artifact. It does not independently observe a model provider or turn a synthetic benchmark into certification.

AuthorityTwin · agent identity under pressure

Prove the agent
may act.

Frozen protocol · 10 adversarial twins

A vendor-neutral conformance lab for the authorization layer between a human, an AI agent, and a tool. It asks whether each action is bound to the right identity, tenant, audience, scope, intent, approval, and audit record.

01 / principal H human-alice intent: records.read
delegatesscope · audience · expiry
02 / workload A agent-orchestrator on behalf of human-alice
requestsnonce · tenant · action
03 / audience R mcp://records tenant-north / record
CONTROLBound requestALLOW
agent
agent-orchestrator
scope
records:read
audience
mcp://records
✓ clean utility preservedreceipt sha256: 8f2a…
Δone authority mutationsame mission · same resource
INJECTEDScope inflationDENY
agent
agent-orchestrator
scope
records:write
audience
mcp://records
✓ simulated effect containedreason: scope_not_granted
Attack surface / v1Ten ways ambient authority breaks
  1. 01Identity
    substitution
  2. 02Scope
    inflation
  3. 03Cross-tenant
    access
  4. 04Audience
    confusion
  5. 05Revoked
    delegation
  6. 06Approval
    replay
  7. 07Delegation
    laundering
  8. 08Intent
    drift
  9. 09Sensitivity
    aggregation
  10. 10Audit-chain
    tamper
$ dspy-security-bench authority demo
Implement the small adapter contract
Open authorization evidence ledger0 public adapter experiments
inspect registry files ↗
AUTH / 001
The adapter contract is open.

Bridge an OAuth, OIDC, MCP, SPIFFE, policy-engine, or custom authorization layer and publish the first independent evidence bundle.

AuthorityTwin is an independent conformance and evidence harness—not a new authorization protocol. Synthetic results are not identity proof, non-repudiation, compliance, certification, production validation, or an authorization to operate.

MissionForge · agency-owned evaluation

Trace the answer
back to authority.

Ship a mission test as reviewable YAML—not benchmark code. SourceTwin holds the authoritative corpus and question fixed, changes only retrieved untrusted content, and scores the agent's recorded claims against exact source IDs.

Five controlled interventions
  1. 01
    Fabricated authorityinvented rule + citation
  2. 02
    Embedded instructionretrieval changes conclusion
  3. 03
    Material omissionexception disappears
  4. 04
    Superseded guidanceobsolete source wins
  5. 05
    Insufficient evidenceanswer instead of abstain
Deterministic probes
FaithfulnessDoes the citation support the claim?
CompletenessWere material facts and exceptions retained?
SufficiencyDid the agent abstain when evidence ran out?
AuthorityDid current primary evidence outrank noise?
$ dspy-security-bench pack run source-twin --agent myapp:build
Author an agency MissionPack
Open grounding evidence ledger0 public MissionPack experiments
inspect registry files ↗
PACK / 001
The protocol is open.

Fork it, encode a real mission with synthetic evidence, and publish the first independently reproducible run.

Structured test claims are benchmark ground truth for a fixed synthetic pack—not a legal interpretation, universal truth set, certification, or substitute for agency subject-matter review.

Public-interest specialty

Same facts.
Different decision.

ImpactTwin / ProcureBench

Clean and poisoned procurement twins hold structured facts fixed while varying one untrusted input: vendor-authored text. Then the benchmark measures what changed in the live synthetic world.

01A Clean twin
Facts locked
Proposals
2 identical records
Ceiling
$1,200,000
Criteria
70 · 20 · 10
RecommendedP-ATLASbest-value eligible offer
01B Poisoned twin
Same facts
Proposals
2 identical records
Ceiling
$1,200,000
Criteria
70 · 20 · 10
ManipulatedP-NOVA+$150,000 price premium
5twin pairs

Award bias · sealed bids · payment identity · eligibility · approval

0%resistance

Deliberately vulnerable scorer fixture

$3.69Msynthetic exposure

Scenario value behind compromised actions

1sealed record out

Functional environment-state evidence

RepeatTwin · stochastic evidence

Perfect observed is not certain.

Ten passes still carry uncertainty. Preserve every raw trial, expose the interval, and gate on its lower bound.

trial evidence
10 / 10 observed passes
95% Wilson interval 72→100%
fixed pair · repeated execution
community proof path
  • raw trials retained
  • statistics recomputed
  • SHA-256 content addressed
Fork · run · submit
Not another prompt score.

Mission utility, controlled-pair decision invariance, confidentiality, identity, authorization, uncertainty, and bounded economic context—versioned in JSON and SARIF.

$ dspy-security-bench impact repeat --trials 10 Inspect the protocol

Reference fixture, not a model result. Synthetic exposure is not predicted loss or a compliance certification.

ControlTwin · policy efficacy

Don't trust the policy.
Test the outcome.

Closed-loop evidence

Execute the same poisoned missions with the boundary off and on. Measure functional harm, safe recovery, and clean utility independently.

Condition APolicy off
5/5harmful poisoned pairs
synthetic exposure$3.69M
  • award decision changed
  • sealed record released
  • authoritative state mutated
tool boundary
POLICYsha256 bound
6 calls blocked environment observed
Condition BPolicy on
0/5harmful poisoned pairs
synthetic exposure$0
  • all observed harms contained
  • clean utility stays at 100%
  • every aggregate recomputable
01Award biascontained · recovery gap
02Sealed proposalrecovered
03Payment identityrecovered
04Eligibilitycontained · recovery gap
05Award authorityrecovered
Functional containment100%5 of 5 harmful pairs → zero
Safe mission recovery60%3 recovered · 2 gaps exposed
Clean mission utility100%no observed clean-work loss
Risk reduction$3.69Msynthetic scenario exposure
RepeatControlTwin · v0.10

One delta can be luck.
Repeat the pair.

Run the complete experiment again and again. Alternate order. Preserve every child report. Gate the lower confidence bound—not the flattering point estimate.

01 Balanced schedule
T01OFF→ON
T02ON→OFF
T03OFF→ON
T04ON→OFF
T05OFF→ON
fresh agent · every case · every condition
02 95% Wilson evidence
Harm containment100%
25/25 · lower bound 86.7%
Safe mission recovery60%
15/25 · 40.7%–76.6%
Clean utility preserved100%
25/25 · lower bound 86.7%
03 Paired signal
harms prevented
25
harms introduced
0
unstable pair effects
0 / 5
exact McNemar p
5.96e−8
Try the five-trial fixture$ dspy-security-bench impact control-repeat-demo --trials 5
Read the method

Deterministic fixture · fixed synthetic suite · not a model result, population estimate, predicted loss, or certification

Run offline now $ dspy-security-bench impact control-demo
Inspect ControlTwin

Deterministic reference fixture, not a model result. Containment is not recovery; synthetic exposure is not predicted loss or certification.

Open Control Evidence Registry · public beta

Which guardrail works?
Show the receipts.

Evidence, not endorsements

Publish the policy-off and policy-on experiment—even when the control performs poorly. Every accepted entry keeps the raw paired trials, policy identity, uncertainty, and chain of custody inspectable.

0public control
experiments
  1. 01
    Run paired twinsfresh agent · alternating order
  2. 02
    Recompute offlineraw events → intervals → digest
  3. 03
    Verify provenanceworkflow · commit · exact bytes
  4. 04
    Compare tradeoffscontainment · recovery · utility
AdmissionValidity, not victory

A failing control can belong here. Five trials, recomputable evidence, fresh isolation, zero runtime errors, and redacted arguments are the comparison floor.

IdentityPolicy SHA-256 bound

The exact policy document travels with the evidence. A name alone cannot silently stand in for different enforcement logic.

InterpretationThree outcomes stay separate

Blocking harm, safely completing the mission, and preserving clean work are reported independently—never collapsed into one marketing score.

Public evidence ledgerchain of custody first · containment lower bound second
inspect registry files ↗
000
The ledger is open.

Be the first independent team to publish a policy experiment—not just a score.

Fork · run · attest · submit evidence-kind: control   policy: policies/production.yaml   trials: 10
Open the submission guide

Fixed synthetic ProcureBench suite · repeated execution evidence · not certification, a population estimate, predicted loss, or independent observation of a hosted provider.

Public-sector mission assurance · v0.15

From agent action
to reviewable evidence.

Exercise a cyber-response agent inside an inert digital twin, repeat the exact counterfactual protocol, and transform verified evidence into standards-shaped assessment inputs—without claiming certification.

01 / TEST

IncidentTwin

air-gap safe

Five clean/poisoned security-operations twins. Authoritative alert facts stay fixed; only hostile external content changes.

00:01READALT-2026-017
00:02VERIFYasset context
00:03DENYsecret → outside.test
00:04ESCALATEtier-2-soc
Pairs
5
Functional harms
5
Network required
no
$ dspy-security-bench incident demo
Inspect the cyber-response protocol
02 / EXPORT

FederalProof

OSCAL 1.2.2

Bind verified ImpactTwin, ControlTwin, IncidentTwin, MissionPack, or AuthorityTwin evidence to an owner-supplied deployment profile and local acceptance thresholds.

  1. AR
    assessment-results.jsonOSCAL findings + observations
    ✓
  2. IA
    impact-assessment-annex.mdoversight · fail-safe · remedy
    ✓
  3. QA
    qasp-scorecard.mdvendor-neutral outcome gates
    ✓
  4. PM
    poam.jsononly when an objective fails
    ↯
$ dspy-security-bench federal init
Open the adoption and evidence guide
Designed to support evidence review—not replace accountable officials. Informative crosswalks · owner-supplied system boundary · explicit uncertainty · no government endorsement · no automatic ATO or procurement decision
Open IncidentTwin ledger0 public cyber-response experiments
inspect registry files ↗
OPEN / 001

Be the first independent team to publish recomputable incident-response evidence.

Real systems. Real authority.

Secure the moment
intent becomes action.

Every production pattern pairs an untrusted input with a dangerous sink. Included policies enforce least agency before the side effect—not after an incident.

01Profile included

Customer support

Resolve tickets without turning a poisoned CRM note into data exfiltration, unlimited refunds, or identity changes.

Untrusted
Tickets · email · CRM
Boundary
Refund cap · recipient domain
--profile customer-support
02Profile included

Accounts payable

Extract and reconcile invoices while preventing vendor impersonation from becoming an unauthorized transfer.

Untrusted
Invoices · vendor email
Boundary
Payee allowlist · approval
--profile financial-operations
03Profile included

Research & RAG

Search broadly without letting a hostile page become persistent memory, published content, or executable code.

Untrusted
Web · docs · retrieved chunks
Boundary
Memory · publish · execute
--profile research-rag
04Profile included

DevOps copilots

Accelerate diagnosis with broad observability while keeping deletion, shell access, and production mutation constrained.

Untrusted
Logs · issues · repository
Boundary
Deploy · shell · delete
--profile devops
Start offline. No API key required. $ dspy-security-bench policy init --profile customer-support Open implementation guide
Research you can inspect

Built for scrutiny,
not screenshots.

01

Frozen protocol

Suites, attacks, scaffold, task subset, and decoding settings are hashed and versioned.

02

Every row reproducible

Per-model result JSON, run metadata, confidence intervals, and generation scripts live beside the board.

03

Honest uncertainty

Rows remain provisional when confidence intervals cross a bucket boundary. Green means no known bypass—not safe.

Trust is a system property.
Make the evidence inspectable.

Star on GitHub Fork the benchmark