--- --- SuperRed: AI red-teaming and AI system security framework {% include site-header.html %}

SuperRed

Open source AI red-teaming and AI system security framework

Run any attacker against any AI system with any threat model

SuperRed is a modular framework for red-teaming AI systems. Attackers, targets, and the benchmarks that score them are independent modules you combine freely, and you define the threat model down to exactly how much access and budget the attacker gets. The same setup evaluates a new attack against various systems and benchmarks, or a new system against various attacks.

pip install superred and Get started

Why SuperRed

Attacks, systems, and benchmarks, freely composable

Module · Attacker

Optimizer

Drives the attack.

Framework

Controller

Orchestrates the run and enforces the security scope.

Module · System

Target

The AI system under test.

Module · Benchmark

Security claim

Defines the goal, scores the run.

Published attacks on AI systems and the benchmarks that measure them are scattered across one-off codebases. Each attack is wired to the system it was first demonstrated on, and each benchmark ships its own runner. Answering a simple question (does this attack beat that system, and how much access does it take?) usually means re-implementing one against the other by hand.

SuperRed pulls those pieces apart. The controller is the framework, and the optimizer, target, and security claim are interchangeable modules you plug into it. They never touch each other: they meet only at the controller, which runs them and enforces how much access the attacker actually gets, so swapping any one of them leaves the other two untouched.

So a new attack can be measured against many systems and benchmarks on the day it is published, and a new system can be put in front of every attack already in the library. The comparison stops being a re-implementation project and becomes a change of one argument.

Showcase

Ready-made attackers, targets, and benchmarks, plus the research built on them

Attacks, targets, and benchmarks, ready to run

Over a dozen faithfully implemented attacks, including AutoDAN-Turbo, PAIR, TAP, Crescendo, and AgentVigil, alongside targets and safety benchmarks such as HarmBench, DecodingTrust, and AgentDojo.

Browse the modules →

Research built with SuperRed

From new attack strategies to full system evaluations, SuperRed is the substrate researchers and builders use to do serious red-teaming work.

See the research →
{% assign mx = site.data.matrix %}

A meeting point

Attackers, targets, and benchmarks are all first-class

An attack, a system under test, and the benchmark that scores it are the same kind of thing here: interchangeable modules. Hold any of them fixed to evaluate it, so everyone shares a single toolbox where every contribution strengthens the rest.

For red-teamers

An interactive environment for developing attacks, so you can build and score novel strategies on real environments as you go.

  • Powerful attackers of every kind: deterministic to agentic, fixed strategies to RL.
  • Iterate against many targets as you build, and evaluate on standard benchmarks in a few lines.

For system builders

Declare your system's trust boundaries once, then see exactly which attacks succeed at each level of access.

  • Test any kind of target: fixed-response fixtures, simulated environments, sandboxes, or live deployments.
  • Wrap an existing system as a target; porting is straightforward.

For evaluators

Package a benchmark once as a security claim, then score any attacker against any target, or reuse the ones already ported.

  • Benchmark under many threat models for fine-grained, comparable insight, not assumptions that vary from benchmark to benchmark.
  • Score deterministically or agentically, whichever the benchmark calls for.

High-level features

What the framework gives you

Flexible threat models

Switch on exactly the capabilities the attacker gets, from a single injected prompt up to full compromise, and change them between runs.

Fair and reproducible

A fixed model and a per-task cost budget bound every attacker equally.

One threat model (scope, llm, budget)
Scope the system's trust boundaries 
system
├─ system_prompt
├─ model_responses
└─ architecture
   └─ high_level_architecture
user
…

the trust boundaries the attacker may cross: here, the system boundary and everything under it (read & write)

grants the attacker access to controllables observables feedback
Attacker LLM gpt-5.5 mythos-5 claude-opus-4.5 gemini-3-pro llama-4-405b

the fixed model the attacker may call

Budget $5.00 / task $1.00 $20.00 $100.00 unlimited

a per-task cost cap on LLM spend

Metrics and reports

Every run records its model, cost, and success rate to a report.

Built to scale

Run many threat models in parallel, each against its own instance of the system.

SuperRed results report for a HarmBench threat model: 42% attack success rate over 5 of 12 goals, mean score 0.57, max 0.94, $11.23 optimizer spend, run-outcome breakdown, and a table of per-task goals with outcomes, best scores, runs, and cost.

Easy to get started

Install, write a short script, run an evaluation

Install the framework from PyPI: pip install superred

Then point an attacker at a target, choose what counts as broken, and say what the attacker is allowed to control:

{% highlight python %} chatbot = lambda: ChatbotTarget(model="gpt-4o-mini", api_key=key) controller = Controller( optimizer_factory=lambda: PAIROptimizer(), # the attacker target_factory=TargetFactory(create=chatbot), # the system under test security_claim=harmbench_claim(), # what counts as broken scope=frozenset({USER_TAG}), # the attacker's capabilities llm_config=LLMConfig(model="gpt-4o", api_key=key), # the attacker's model task_cost_cap_usd=5.0, # per-task attacker budget (USD) ) await controller.run() {% endhighlight %}

The full walk-through, in the guide →

Documentation

Two ways in

Guide

Hands-on and example-driven: getting started, the core concepts of models and events, and worked examples. Start here.

Open the guide →

Reference

The framework interfaces in technical detail: the controller, optimizer, target, task, security claim, and core types.

Open the reference →
{% include site-footer.html %}