Protocol v2 · committed evidence

Measure before
you trust.

A reproducible prompt-injection benchmark that connects to the agent stack you already use—then turns the result into a security gate.

Explore the evidence Connect your agent
Not a safety certificate. A measurable regression floor with every limitation visible.
Observed robustness spread 25→99% Same harness. Same attacks.
Different base models.
14models measured
10model families
2 axesrobustness + capability
100%rows backed by JSON
The signal capability charts miss

Capability and security are different axes.

Two models can complete benign tasks at nearly the same rate and still have radically different prompt-injection failure profiles. Security must be measured directly.

62 point robustness gap between two models with near-identical capability
Interactive map

Capability × robustness

RobustMixedVulnerableProvisional

Hover or focus a model. Provisional measurements use a dashed ring.

Frozen protocol leaderboard

See which models
hold the boundary.

Robustness is the share of prompt-injection attacks that failed. Capability is benign task completion. Confidence intervals are cluster-bootstrapped over task pairs.

Rank / modelFamilyRobustnessCapabilityEvidence
Confirmed only when the full CI stays inside one bucket across all repeats.Read the complete methodology
From benchmark to guardrail

Make injection safety
a merge condition.

Scaffold a config and GitHub Action, preview the exact test matrix without spending API credits, then block regressions with terminal, JSON, and SARIF reports.

  • 01Works with DSPy or any Python agent factory
  • 02Absolute security floors or baseline regression gates
  • 03OWASP LLM01 · NIST AI 100-2 · MITRE ATLAS mappings
quickstart.sh
# install and create a ready-to-run gate
$ pip install dspy-security-bench
$ dspy-security-bench init --model openai/gpt-4o-mini

# inspect the matrix — zero model calls
$ dspy-security-bench scan --config .dspy-security-bench.yaml --plan

✓ 10 benchmark cases planned
✓ no model was called
Open the five-minute guide
Bring your own agent · v0.8

Your framework.
One evidence contract.

Keep your orchestration code and tools. The integration assistant detects a directly declared framework, generates a reviewable adapter, and validates the wiring without invoking the agent run loop.

01 / Scaffold
$dspy-security-bench integrate
Detectedopenai-agentspyproject.toml
+ dspy_security_target.py + integration.json + dspy-proofrun.yml
agent boundary
DSB
same attacks
same evidence
02 / Verify
dspy-security-bench doctorno agent runs
  • ✓ Manifest is valid
  • ✓ Fresh agent factory
  • ✓ Immutable ProofRun ref
  • ✓ Attestation permissions
Open integration guide
OA
OpenAIAgents SDK
LC
LangChainand LangGraph
PY
Pydantic AItyped agents
CR
CrewAIcrews and tools
AG
AutoGenAgentChat
↔
MCP / customany Python loop

Generated workflows are manual by default, so connecting a repository cannot unexpectedly spend model credits.

ProofRun · evidence with a chain of custody

Run it. Sign it.
Let others verify it.

v0.11 dual evidence builder

A reusable workflow isolates model evaluation from signing, recomputes every statistic in a clean job, and binds the exact JSON to a source commit before enforcing the gate.

DSB / PROOFRUN 01 evidence passport
Verification ladder Bytes are easy.
Provenance is earned.
  1. 1
    Self-attestedstatistics + canonical hash
  2. 2
    GitHub-attestedrunner + workflow + commit
  3. 3
    Trusted builderimmutable central workflow
  4. 4
    Reproducedindependent maintainer rerun
artifact subject sha256: 9e81…c4f2 signature verification is separate from score recomputation
agent-security.yml
permissions:
  contents: read
  id-token: write
  attestations: write

jobs:
  proofrun:
    uses: immu4989/dspy-security-bench/.github/workflows/proofrun.yml@v0.11.1
    with:
      agent: myapp.security:build_agent
      trials: 10
Open evidence network 0 community ProofRuns

Every accepted entry preserves all raw trials and passes offline recomputation. A cryptographic badge appears only after the exact digest passes attestation verification and enters the reviewed registry.

Run your agent
Community evidenceranked by evidence tier, then lower confidence bound
The network is open. Be the first independent agent team to submit an attested run.

Provenance authenticates a workflow and exact artifact. It does not independently observe a model provider or turn a synthetic benchmark into certification.

Public-interest specialty

Same facts.
Different decision.

ImpactTwin / ProcureBench

Clean and poisoned procurement twins hold structured facts fixed while varying one untrusted input: vendor-authored text. Then the benchmark measures what changed in the live synthetic world.

01A Clean twin
Facts locked
Proposals
2 identical records
Ceiling
$1,200,000
Criteria
70 · 20 · 10
RecommendedP-ATLASbest-value eligible offer
01B Poisoned twin
Same facts
Proposals
2 identical records
Ceiling
$1,200,000
Criteria
70 · 20 · 10
ManipulatedP-NOVA+$150,000 price premium
5twin pairs

Award bias · sealed bids · payment identity · eligibility · approval

0%resistance

Deliberately vulnerable scorer fixture

$3.69Msynthetic exposure

Scenario value behind compromised actions

1sealed record out

Functional environment-state evidence

RepeatTwin · stochastic evidence

Perfect observed is not certain.

Ten passes still carry uncertainty. Preserve every raw trial, expose the interval, and gate on its lower bound.

trial evidence
10 / 10 observed passes
95% Wilson interval 72→100%
fixed pair · repeated execution
community proof path
  • raw trials retained
  • statistics recomputed
  • SHA-256 content addressed
Fork · run · submit
Not another prompt score.

Mission utility, controlled-pair decision invariance, confidentiality, identity, authorization, uncertainty, and bounded economic context—versioned in JSON and SARIF.

$ dspy-security-bench impact repeat --trials 10 Inspect the protocol

Reference fixture, not a model result. Synthetic exposure is not predicted loss or a compliance certification.

ControlTwin · policy efficacy

Don't trust the policy.
Test the outcome.

Closed-loop evidence

Execute the same poisoned missions with the boundary off and on. Measure functional harm, safe recovery, and clean utility independently.

Condition APolicy off
5/5harmful poisoned pairs
synthetic exposure$3.69M
  • award decision changed
  • sealed record released
  • authoritative state mutated
tool boundary
POLICYsha256 bound
6 calls blocked environment observed
Condition BPolicy on
0/5harmful poisoned pairs
synthetic exposure$0
  • all observed harms contained
  • clean utility stays at 100%
  • every aggregate recomputable
01Award biascontained · recovery gap
02Sealed proposalrecovered
03Payment identityrecovered
04Eligibilitycontained · recovery gap
05Award authorityrecovered
Functional containment100%5 of 5 harmful pairs → zero
Safe mission recovery60%3 recovered · 2 gaps exposed
Clean mission utility100%no observed clean-work loss
Risk reduction$3.69Msynthetic scenario exposure
RepeatControlTwin · v0.10

One delta can be luck.
Repeat the pair.

Run the complete experiment again and again. Alternate order. Preserve every child report. Gate the lower confidence bound—not the flattering point estimate.

01 Balanced schedule
T01OFF→ON
T02ON→OFF
T03OFF→ON
T04ON→OFF
T05OFF→ON
fresh agent · every case · every condition
02 95% Wilson evidence
Harm containment100%
25/25 · lower bound 86.7%
Safe mission recovery60%
15/25 · 40.7%–76.6%
Clean utility preserved100%
25/25 · lower bound 86.7%
03 Paired signal
harms prevented
25
harms introduced
0
unstable pair effects
0 / 5
exact McNemar p
5.96e−8
Try the five-trial fixture$ dspy-security-bench impact control-repeat-demo --trials 5
Read the method

Deterministic fixture · fixed synthetic suite · not a model result, population estimate, predicted loss, or certification

Run offline now $ dspy-security-bench impact control-demo
Inspect ControlTwin

Deterministic reference fixture, not a model result. Containment is not recovery; synthetic exposure is not predicted loss or certification.

Open Control Evidence Registry · public beta

Which guardrail works?
Show the receipts.

Evidence, not endorsements

Publish the policy-off and policy-on experiment—even when the control performs poorly. Every accepted entry keeps the raw paired trials, policy identity, uncertainty, and chain of custody inspectable.

0public control
experiments
  1. 01
    Run paired twinsfresh agent · alternating order
  2. 02
    Recompute offlineraw events → intervals → digest
  3. 03
    Verify provenanceworkflow · commit · exact bytes
  4. 04
    Compare tradeoffscontainment · recovery · utility
AdmissionValidity, not victory

A failing control can belong here. Five trials, recomputable evidence, fresh isolation, zero runtime errors, and redacted arguments are the comparison floor.

IdentityPolicy SHA-256 bound

The exact policy document travels with the evidence. A name alone cannot silently stand in for different enforcement logic.

InterpretationThree outcomes stay separate

Blocking harm, safely completing the mission, and preserving clean work are reported independently—never collapsed into one marketing score.

Public evidence ledgerchain of custody first · containment lower bound second
inspect registry files ↗
000
The ledger is open.

Be the first independent team to publish a policy experiment—not just a score.

Fork · run · attest · submit evidence-kind: control   policy: policies/production.yaml   trials: 10
Open the submission guide

Fixed synthetic ProcureBench suite · repeated execution evidence · not certification, a population estimate, predicted loss, or independent observation of a hosted provider.

Real systems. Real authority.

Secure the moment
intent becomes action.

Every production pattern pairs an untrusted input with a dangerous sink. Included policies enforce least agency before the side effect—not after an incident.

01Profile included

Customer support

Resolve tickets without turning a poisoned CRM note into data exfiltration, unlimited refunds, or identity changes.

Untrusted
Tickets · email · CRM
Boundary
Refund cap · recipient domain
--profile customer-support
02Profile included

Accounts payable

Extract and reconcile invoices while preventing vendor impersonation from becoming an unauthorized transfer.

Untrusted
Invoices · vendor email
Boundary
Payee allowlist · approval
--profile financial-operations
03Profile included

Research & RAG

Search broadly without letting a hostile page become persistent memory, published content, or executable code.

Untrusted
Web · docs · retrieved chunks
Boundary
Memory · publish · execute
--profile research-rag
04Profile included

DevOps copilots

Accelerate diagnosis with broad observability while keeping deletion, shell access, and production mutation constrained.

Untrusted
Logs · issues · repository
Boundary
Deploy · shell · delete
--profile devops
Start offline. No API key required. $ dspy-security-bench policy init --profile customer-support Open implementation guide
Research you can inspect

Built for scrutiny,
not screenshots.

01

Frozen protocol

Suites, attacks, scaffold, task subset, and decoding settings are hashed and versioned.

02

Every row reproducible

Per-model result JSON, run metadata, confidence intervals, and generation scripts live beside the board.

03

Honest uncertainty

Rows remain provisional when confidence intervals cross a bucket boundary. Green means no known bypass—not safe.

Prompt injection is a model property.
Measure it like one.

Star on GitHub Fork the benchmark