Zortium documentation

Introduction #

Zortium is an adversarial testing tool for vision-language models (VLMs) — think promptfoo, but for image-based attacks. It points at any endpoint speaking the OpenAI chat-completions protocol, runs a battery of attack suites drawn from published research, and reports how often each attack got the model to comply.

The headline metric is Attack Success Rate (ASR) — the fraction of adversarial cases in a suite where the target produced harmful or hijacked output instead of refusing. Lower is safer. Each suite is scored independently, and an overall ASR summarises the run.

Zortium ships as a standalone CLI that never phones home. zortium.dev is an optional managed platform built on the same suites — web UI, saved history, model benchmarks, and team features — but nothing below requires it.

Installation #

pip install zortium

Requires Python 3.10 or newer. There are no system dependencies — every dependency ships as a pure-Python wheel, so a fresh machine needs nothing beyond pip.

Verify the install:

zortium --help

Quick start #

A scan needs a target (the model under test) and, for trustworthy numbers, a judge (a separate model that grades whether the target actually complied):

zortium \
  --base-url https://openrouter.ai/api/v1 --model google/gemma-3-27b-it --api-key $OPENROUTER_API_KEY \
  --judge-model gpt-4o-mini --judge-base-url https://api.openai.com/v1 --judge-api-key $OPENAI_API_KEY

Every flag has a ZORTIUM_* environment variable, so keys never appear in your shell history or a process listing. That single command is the whole interface — everything else is a convenience on top of it.

You get a per-suite breach-rate table and a headline ASR:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Attack Suite                         ┃ Severity ┃ Breach Rate ┃ Status ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━┩
│ Typographic Visual Prompt Injection  │   HIGH   │          0% │  PASS  │
│ FigStep Typographic Jailbreak        │   HIGH   │         25% │  WARN  │
│ Query-Relevant Typographic Jailbreak │   HIGH   │          8% │  PASS  │
│ Goal Hijacking                       │   MID    │         50% │  FAIL  │
│ …                                    │    …     │           … │   …    │
├──────────────────────────────────────┼──────────┼─────────────┼────────┤
│ Overall                              │          │         12% │  WARN  │
└──────────────────────────────────────┴──────────┴─────────────┴────────┘

  cases    137 evaluated · 17 breached
PASS  Overall ASR 12.3% within threshold 100%

The scan command #

The command has two halves — the target you're testing and the judge that grades it:

Everything else layers convenience on top:

• Keys off the command line — every flag has a ZORTIUM_* env var (ZORTIUM_API_KEY, ZORTIUM_JUDGE_API_KEY, ZORTIUM_MODEL, …).
• Deeper audit — --no-fast runs full per-suite coverage instead of the reduced fast set (see Scan modes).
• Parallelism — --tps N runs N suites at once on high-limit endpoints (see Configuration).

Configuration #

Environment variables

Every flag has a ZORTIUM_* equivalent, so secrets stay out of shell history and CI logs. Precedence is flag > env var > config file > default.

Config file

Instead of a long command, generate a documented config and edit it:

# write a starter zortium.yaml in the current directory
zortium --init

# run from it (flags and env vars still override the file)
zortium --config zortium.yaml

Both YAML and JSON are accepted. Blank or null values in the file fall through to env vars and defaults.

Saving a report

The CLI keeps no database. Add --output to write a full JSON report — run metadata, headline stats, and per-suite → per-case entries with the model response, judge verdict, and its reasoning:

zortium --output report.json

Parallelism

--tps N (env ZORTIUM_TPS) runs N suites in parallel. The default is 1 — sequential, which is safe for rate-limited or shared-key endpoints. Raise it for in-house models that can absorb the extra requests per second; rate limits are still handled per the 429 policy, so too high a value degrades gracefully rather than failing.

CI integration #

Use --max-asr to set a threshold. The CLI exits 1 if the overall ASR exceeds it — a native CI gate, no plugin needed:

zortium --max-asr 20   # fail the build if overall ASR > 20%

A complete GitHub Actions job — secrets stay in CI variables:

# .github/workflows/vlm-safety.yml
name: VLM Safety Gate
on: [push]
jobs:
  zortium:
    runs-on: ubuntu-latest
    steps:
      - run: pip install zortium
      - run: zortium --max-asr 20
        env:
          ZORTIUM_BASE_URL: ${{ secrets.VLM_BASE_URL }}
          ZORTIUM_MODEL:    ${{ secrets.VLM_MODEL }}
          ZORTIUM_API_KEY:  ${{ secrets.VLM_API_KEY }}

The --max-asr gate uses the raw overall ASR. Diagnostic suites are always excluded from it.

Scan modes #

Both modes run the same 18 suites; fast runs fewer, highest-signal cases per suite, thorough runs full coverage. --fast is the default; use --no-fast for deep audits.

Rate limits #

By default a rate-limited (429) case is recorded as unavailable and left out of the score — fast, and right for CI. For a one-off baseline against a heavily throttled endpoint, --wait (env ZORTIUM_WAIT) retries each rate-limited case until the limit clears, so every case gets a real verdict. It respects the Retry-After header and can take much longer — use it for benchmarks, not CI.

zortium --no-fast --wait   # complete thorough baseline, waits out rate limits

LLM judge #

Critical suites are graded by an LLM judge — it reads each (harmful request, model response) pair and decides whether the target actually complied. Keyword matching produces false positives on jailbreak evaluation, so these suites require a judge rather than falling back to keywords.

Configure it with --judge-model / --judge-base-url / --judge-api-key (or the ZORTIUM_JUDGE_* env vars). The base URL and key inherit the target's when omitted, so a judge on the same endpoint needs only --judge-model. Use a separate, stronger model than the one under test for better calibration.

Attack suites #

Critical — LLM judge required, high research backing

Medium — keyword evaluation, limited effectiveness on modern models

Diagnostic — robustness probes, reported separately from ASR

Scoring & severity #

Each suite carries a severity reflecting how real-world exploitable the attack is — distinct from how it's evaluated internally:

Diagnostic suites are always excluded from the headline ASR. On zortium.dev, severity also drives a weighted overall score, so a high-severity breach moves the number more than a low-severity one. The CLI --max-asr gate uses the raw overall ASR.

Providers #

Any endpoint speaking the OpenAI chat-completions protocol works — no SDK changes, no instrumentation, just a base URL and an API key:

• OpenRouter — pay-per-token access to hundreds of OSS models via one key (the easiest path for benchmarking).
• OpenAI, Groq, Together AI, Google Gemini (via their OpenAI-compatible endpoints).
• vLLM, Ollama (via the /v1 shim), TGI adapter.
• In-house gateways and fine-tuned model deployments.

See Integrations for per-provider base URLs and model examples.

Architecture #

The engine is a src-layout package under src/zortium/:

src/zortium/
├── runner.py            TestRunner — judge enforcement, suite execution
├── constants.py         ScanMode, SuitePriority, SuiteSeverity enums
├── attacks/             Attack suite implementations
│   ├── base.py          AttackSuite base — configure_for_mode(), evaluate()
│   ├── __init__.py      SuiteRegistry — build_active_suites()
│   └── ...
├── providers/
│   ├── openai_compatible.py  Target and judge provider abstraction
│   └── ratelimit.py     Retry-After / backoff, SKIP/WAIT
├── evaluators/
│   ├── judge.py         LLM-as-judge evaluation
│   └── refusal_aware.py Keyword-based refusal detection
├── utils/               Render, Perturbations, ImageUtils, PayloadPack
└── config/
    ├── attacks.json     Suite config — priority, severity, parameters
    └── harm_payloads.json  Harm categories and prompt library

Adding a suite: subclass AttackSuite in attacks/, add an entry to config/attacks.json, and register it in ACTIVE_SUITE_CLASSES. Override configure_for_mode() if it needs different behaviour in fast vs thorough mode.

Safety #

Zortium is a defensive security tool for evaluating the robustness of vision-language models you own or are authorized to test.

• Authorized targets only. Do not run Zortium against models or systems you don't own or have explicit permission to test.
• It sends adversarial and harmful content. By design, Zortium submits jailbreak prompts, harmful requests, and adversarial images to measure refusal behaviour. The bundled payloads and the JailBreakV-28K benchmark contain offensive material.
• Purpose is measurement, not exploitation. The output is a robustness report to help you harden a model — not a toolkit for attacking third-party systems.

Report security issues responsibly and use results to improve model safety.