AI red-teaming and AI system security framework
Point an automated attacker at any AI system under a precisely defined level of access.
Showcase
Over a dozen faithfully implemented attacks, including AutoDAN-Turbo, PAIR, TAP, Crescendo, and AgentVigil, alongside targets and safety benchmarks such as HarmBench, DecodingTrust, and AgentDojo.
Browse the modules →From new attack strategies to full system evaluations, Superred is the substrate researchers and builders use to do serious red-teaming work.
See the portfolio →Why superred
Published attacks on AI systems and the benchmarks that measure them are scattered across one-off codebases. Each attack is wired to the system it was first demonstrated on; each benchmark ships its own runner. Answering a simple question (does this attack beat that system, and how much access does it take?) usually means re-implementing one against the other by hand.
Superred makes the three moving parts independent and composable. The attacker (an optimizer), the system under test (a target), and the definition of success (a security claim) are separate, interchangeable modules. Any attacker can be pointed at any target and scored by any claim, under a flexible threat model you state explicitly.
A meeting point
An attack and a system under test are the same kind of thing here: interchangeable modules. Fix one to evaluate the other, so red-teamers and system builders share a single toolbox where every contribution strengthens both sides.
A library of published attacks, ready to run against any target and scored by real benchmarks. A fixed model and budget keep comparisons fair.
Declare your system's trust boundaries once, then see exactly which attacks succeed at each level of access.
High-level features
Decide exactly what an attacker may do, and change it freely between runs. A threat model is just the set of attacker capabilities you switch on, one per surface the attacker could control, from a single injected prompt up to full compromise.
Any optimizer, any target, any security claim. Components are independent packages that combine freely.
A fixed model and a cost budget per task bound every attacker equally.
Superred tracks the model, cost, and success rates of every run and writes them to a report you can analyze.
Run many threat models in parallel, each against its own independent instance of the system under test.
In one sentence
Superred runs an optimizer against a target to pursue an adversarial goal from a security claim, under a security scope you define, recording every event and producing a report with success metrics.
Easy to get started
Install the framework and whichever modules you want to combine:
pip install superred superred-pair-optimizer superred-chatbot-target superred-secclaim-harmbenchThen point an attacker at a target, choose what counts as broken, and say what the attacker is allowed to control:
chatbot = lambda: ChatbotTarget(model="gpt-4o-mini", api_key=key)
controller = Controller(
optimizer_factory=lambda: PAIROptimizer(), # the attacker
target_factory=TargetFactory(create=chatbot), # the system under test
security_claim=harmbench_claim(), # what counts as broken
scope=frozenset({USER_TAG}), # the attacker's capabilities
llm_config=LLMConfig(model="gpt-4o", api_key=key, max_cost=5.0), # model + budget
)
await controller.run()Documentation