--- --- {% include head-meta.html title="SuperRed: Open-source AI red-teaming and AI system security framework" description="SuperRed is a modular Python framework for red-teaming AI systems: run any attacker against any AI system under any threat model, and score it on standard safety benchmarks." %} {% include site-header.html %}

SuperRed

Run any attacker against any AI system with any threat model

or and Get started

Why SuperRed

Attacks, systems, and benchmarks, freely composable

Module · Attacker

Optimizer

Drives the attack.

Framework

Controller

Orchestrates the run and enforces the security scope.

Module · System

Target

The AI system under test.

Module · Benchmark

Security claim

Defines the goal, scores the run.

Attacks and benchmarks live in one-off codebases, each wired to the system it was first demonstrated on. SuperRed pulls the pieces apart: the optimizer, target, and security claim are interchangeable modules that meet only at the controller. Swap any one and the comparison becomes a change of a single argument, not a re-implementation project.

Showcase

Ready-made attackers, targets, and benchmarks, plus the research built on them

Attacks, targets, and benchmarks, ready to run

Over a dozen faithfully implemented attacks, including AutoDAN-Turbo, PAIR, TAP, Crescendo, and AgentVigil, alongside targets and safety benchmarks such as HarmBench, DecodingTrust, and AgentDojo.

Browse the modules →

Research built with SuperRed

From new attack strategies to full system evaluations, SuperRed is the substrate researchers and builders use to do serious red-teaming work.

See the research →

{% assign mx = site.data.matrix %}

{% for t in mx.targets %}

{% endfor %} {% for opt in mx.optimizers %} {% assign ri = forloop.index0 %}

{% for t in mx.targets %} {% assign k = ri | plus: forloop.index0 %} {% assign n = t.claims | size %}

{% if n == 1 %} {% assign c1 = t.claims[0] %} {{ c1 }} {% else %} {% assign i1 = k | modulo: n %} {% assign k2 = k | plus: 1 %} {% assign i2 = k2 | modulo: n %} {% assign c1 = t.claims[i1] %} {% assign c2 = t.claims[i2] %} {{ c1 }} {{ c2 }} {% endif %}

{% endfor %} {% endfor %}

A meeting point

Attackers, targets, and benchmarks are all first-class

Attack, system, and benchmark are the same kind of thing here: interchangeable modules. Hold any one fixed to evaluate it; every contribution strengthens a shared toolbox.

For red-teamers

An interactive environment for developing attacks, so you can build and score novel strategies on real environments as you go.

Powerful attackers of every kind: deterministic to agentic, fixed strategies to RL.
Iterate against many targets as you build, and evaluate on standard benchmarks in a few lines.

For system builders

Declare your system's trust boundaries once, then see exactly which attacks succeed at each level of access.

Test any kind of target: fixed-response fixtures, simulated environments, sandboxes, or live deployments.
Wrap an existing system as a target; porting is straightforward.

For evaluators

Package a benchmark once as a security claim, then score any attacker against any target, or reuse the ones already ported.

Benchmark under many threat models for fine-grained, comparable insight, not assumptions that vary from benchmark to benchmark.
Score deterministically or agentically, whichever the benchmark calls for.

High-level features

What the framework gives you

Flexible threat models

Switch on exactly the capabilities the attacker gets, from a single injected prompt up to full compromise, and change them between runs.

Fair and reproducible

A fixed model and a per-task cost budget bound every attacker equally.

One threat model (scope, llm, budget)

system
├─ system_prompt
├─ model_responses
└─ architecture
   └─ high_level_architecture

user
…

the trust boundaries the attacker may cross: here, the system boundary and everything under it (read & write)

grants the attacker access to controllables observables feedback

Metrics and reports

Every run records its model, cost, and success rate to a report.

Built to scale

Run many threat models in parallel, each against its own instance of the system.

SuperRed results report for a HarmBench threat model: 42% attack success rate over 5 of 12 goals, mean score 0.57, max 0.94, $11.23 optimizer spend, run-outcome breakdown, and a table of per-task goals with outcomes, best scores, runs, and cost.

Easy to get started

Install, write a short script, run an evaluation

Point an attacker at a target, choose what counts as broken, and say what the attacker may control:

{% highlight python %} chatbot = lambda: ChatbotTarget(model="gpt-4o-mini", api_key=key) controller = Controller( optimizer_factory=lambda: PAIROptimizer(), # the attacker target_factory=TargetFactory(create=chatbot), # the system under test security_claim=harmbench_claim(), # what counts as broken scope=frozenset({USER_TAG}), # the attacker's capabilities llm_config=LLMConfig(model="gpt-4o", api_key=key), # the attacker's model task_cost_cap_usd=5.0, # per-task attacker budget (USD) ) await controller.run() {% endhighlight %}

The full walk-through, in the guide →

Documentation

Two ways in

Guide

Hands-on and example-driven: getting started, the core concepts of models and events, and worked examples. Start here.

Open the guide →

Reference

The framework interfaces in technical detail: the controller, optimizer, target, task, security claim, and core types.

Open the reference →

Used by researchers from

{%- comment -%} Brand-styled text logos (not official logo files). The set is repeated four times so the track always stays wider than the viewport and the -25% loop is seamless. Captured in a snippet so every copy stays identical. {%- endcomment -%} {% capture ub_set %}

{% endcapture %} {{ ub_set }}{{ ub_set }}{{ ub_set }}{{ ub_set }}

{% include site-footer.html %}