Metadata-Version: 2.4
Name: agentsec-bench
Version: 0.1.2
Summary: AgentSec-Bench: a security benchmark for LLM agents and harnesses, built on Inspect AI
Author: Santhosraj
License: MIT
Project-URL: Homepage, https://github.com/Santhosraj/AgentSec-Bench
Project-URL: Repository, https://github.com/Santhosraj/AgentSec-Bench
Project-URL: Issues, https://github.com/Santhosraj/AgentSec-Bench/issues
Keywords: llm,benchmark,evaluation,ai-safety,security,agents,inspect-ai
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Security
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: inspect-ai>=0.3.50
Requires-Dist: anthropic>=0.40.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Provides-Extra: scan
Requires-Dist: bandit>=1.7; extra == "scan"
Requires-Dist: semgrep>=1.80; sys_platform != "win32" and extra == "scan"
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: google
Requires-Dist: google-genai>=1.0; extra == "google"
Provides-Extra: mistral
Requires-Dist: mistralai>=1.0; extra == "mistral"
Provides-Extra: groq
Requires-Dist: groq>=0.11; extra == "groq"
Provides-Extra: all
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: google-genai>=1.0; extra == "all"
Requires-Dist: mistralai>=1.0; extra == "all"
Requires-Dist: groq>=0.11; extra == "all"
Requires-Dist: bandit>=1.7; extra == "all"
Requires-Dist: semgrep>=1.80; sys_platform != "win32" and extra == "all"
Requires-Dist: pytest>=8.0; extra == "all"
Requires-Dist: ruff>=0.6; extra == "all"
Dynamic: license-file

# AgentSec-Bench

> A security benchmark for LLM agents & harnesses. *(Python package: `agentsec`.)*

[![CI](https://github.com/Santhosraj/AgentSec-Bench/actions/workflows/ci.yml/badge.svg)](https://github.com/Santhosraj/AgentSec-Bench/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**AgentSec-Bench** measures how well **LLM agents and their harnesses** perform on **security**
tasks. It is built on [Inspect AI](https://inspect.aisi.org.uk/) (UK AISI) and evaluates
agents against **live, sandboxed targets** across four capability axes:

| Axis | What it measures | Grading |
|------|------------------|---------|
| **offensive** | Find & exploit vulns (CTF-style) against a live container; submit a flag. | Automated (flag match) |
| **defensive** | Detect insecure code and patch it without breaking functionality. | Hybrid (hidden security tests + judge) |
| **secure_coding** | Build a feature to spec without introducing vuln classes. | Hybrid (semgrep/bandit + judge) |
| **harness_safety** | Resist prompt injection, refuse misuse, respect sandbox boundaries. | Hybrid (judge + sandbox signals) |

## Difficulty tiers & partial credit

Challenges are tagged `easy | medium | hard | stretch`. Run a single tier (or several) and
the hardest challenges award **gradated partial credit** — e.g. 0.33 for reaching stage 1 of
a 3-stage exploit chain — so the benchmark discriminates across the capability range instead
of collapsing to 0/1.

```bash
inspect eval src/agentsec/tasks/offensive.py -T difficulty=stretch --model google/gemini-2.5-flash
inspect eval src/agentsec/tasks/offensive.py -T difficulty=hard,stretch --epochs 3   # repeat stochastic hard tasks
```

A challenge declares `subtasks` (weighted `exec`/`flag` checks) for partial credit; one
without subtasks scores all-or-nothing via the axis default scorer. See
[CHALLENGES.md](CHALLENGES.md) for the schema. The hard/stretch set includes a multi-network
SSRF→internal-metadata chain, weak-secret JWT forgery, a pickle-deserialization RCE fix, a
functionally-graded secure file-upload, and an obfuscated multi-step prompt injection.

## The differentiator: benchmarking *harnesses*, not just models

Most security benchmarks pin a harness and vary the model. `agentsec` does the opposite as
well: via Inspect's **agent bridge**, external harnesses (Claude Code, Codex CLI, Gemini CLI)
have their model calls proxied through Inspect's provider. Every harness therefore runs on the
**same underlying model**, isolating *harness quality* from *model quality*. Swap `--model` to
isolate model quality on a fixed harness.

```
agent harness (in sandbox)  ──API calls──►  localhost:13131 proxy  ──►  Inspect model provider
```

## Install

```bash
# Option A — from PyPI (quickest; bundles the framework AND the built-in challenges):
pip install agentsec-bench

# Option B — from source (for development or adding challenges):
git clone https://github.com/Santhosraj/AgentSec-Bench.git
cd AgentSec-Bench
pip install -e ".[dev]"        # add ",scan" on Linux/macOS for semgrep
```

> Running the live sandboxed challenges requires **Docker** (see [Requirements](#requirements)).
> `anthropic` ships in core; add a provider extra for others (`pip install "agentsec-bench[openai]"`).

## Quickstart

```bash
export ANTHROPIC_API_KEY=...    # or GOOGLE_API_KEY / OPENAI_API_KEY, etc.

# Easiest — the bundled CLI (works whether you pip-installed or cloned):
agentsec run --model anthropic/claude-opus-4-8 --axes offensive --difficulty easy
agentsec report ./logs          # comparison table across runs

# Or call Inspect directly (source checkout — gives full -T/-S control):
inspect eval src/agentsec/tasks/offensive.py --model anthropic/claude-opus-4-8

# Same challenge driven by the Claude Code harness via the bridge
# (--solver overrides the task default; -S passes solver args — note -S, not -T):
inspect eval src/agentsec/tasks/offensive.py \
    --solver src/agentsec/solvers/bridge_harness.py -S harness=claude-code \
    --model anthropic/claude-opus-4-8

inspect view   # browse logs / transcripts
```

## Providers 

AgentSec-Bench is **provider-agnostic** — the model under test, the LLM judge, and the
harness vendor are all swappable. `anthropic` ships in core so the examples above work out
of the box; install an extra for any other provider:

```bash
pip install -e ".[openai]"     # or .[google] / .[mistral] / .[groq] / .[all]
```

Then point `--model` (and optionally the judge role) at it — the task and scorer code is
unchanged:

```bash
inspect eval src/agentsec/tasks/offensive.py --solver builtin_agent --model openai/gpt-4o
inspect eval src/agentsec/tasks/defensive.py --solver builtin_agent \
    --model google/gemini-2.5-pro --model-role grader=openai/gpt-4o
# Local models work too: --model ollama/llama3.1  or  --model vllm/<model>
```

Because the **agent bridge** decouples a harness's API dialect from the backing model, you
can also run a harness on a *different* vendor's model — e.g. the Claude Code harness backed
by GPT-4o — to isolate scaffolding quality from the model:

```bash
inspect eval src/agentsec/tasks/offensive.py \
    --solver src/agentsec/solvers/bridge_harness.py -S harness=claude-code \
    --model openai/gpt-4o
```

Set the matching API key for whichever provider you use (`OPENAI_API_KEY`,
`GOOGLE_API_KEY`, `MISTRAL_API_KEY`, `GROQ_API_KEY`, …).

### Picking a model for the **agent** role (important)

The agent axes drive a multi-step **tool-calling** loop, so the agent model must reliably
emit *structured* tool calls. Two tiers:

- ✅ **Native tool-calling providers** — Anthropic, OpenAI, Google, Mistral. Rock-solid;
  use these for the agent role when you want clean runs.
- ⚠️ **Groq + open models (Llama/qwen/gpt-oss)** — Groq re-parses model output into tool
  calls server-side and is strict about it. Llama-3.3 tends to emit `<function=…>` text
  tags and reasoning models emit `<tool_call>` blocks; Groq rejects both with a fatal
  `400 tool_use_failed`. This is a **provider quirk, not a model-capability gap** (the model
  often solves the task right up to the bad tool call). `--retry-on-error` does **not** help
  because the format is deterministic, not flaky.

**Want fully open-weight with reliable tools?** Use **[Ollama](https://ollama.com)** locally
— it parses each model's native tool format itself, so there's none of Groq's
`tool_use_failed` and no API quotas.

```bash
# 1. Install Ollama (https://ollama.com/download)

# 2. Pull a tool-capable model. On CPU, prefer a smaller model for the multi-step agent loop:
ollama pull qwen2.5-coder:3b               # ~1.9 GB, decent tools, CPU-friendly
#   (qwen2.5-coder:7b is stronger but ~2-3x slower on CPU and needs ~8 GB RAM)

# 3. Run (--max-connections 1 keeps a local model from being flooded with parallel requests):
inspect eval src/agentsec/tasks/offensive.py -T difficulty=easy \
    --model ollama/qwen2.5-coder:3b --max-connections 1 --display plain
```

Notes for local models: inference is **CPU-bound without a GPU**, so each agent step takes
seconds and a hard challenge can run several minutes — cap it with `--message-limit 30`.
Small (3B) models legitimately score low on the hard/stretch tiers; that's a capability
result, not a bug.

The **grader** role does no tool-calling, so any open model is fine there — pair a native
agent model with an open-weight grader:

```bash
inspect eval src/agentsec/tasks/defensive.py \
    --model anthropic/claude-haiku-4-5 \
    --model-role grader=groq/llama-3.3-70b-versatile --display plain
```

## No-Docker smoke test (`sandbox_type=local`)

Don't have Docker set up yet? The benign coding axes (`defensive`, `secure_coding`) can run
in a **local** sandbox that executes the agent's tools directly on the host — handy for
verifying the dataset → solver → scorer pipeline before standing up Docker. Pair it with
Inspect's `mockllm` to run with **no Docker and no API key at all**:

```bash
inspect eval src/agentsec/tasks/secure_coding.py -T sandbox_type=local \
    --model mockllm/model --display plain
# ... or with a real local model:
inspect eval src/agentsec/tasks/defensive.py -T sandbox_type=local --model ollama/llama3.1
```

> ⚠️ Local mode runs agent-generated commands **on your machine** — it's a smoke-test
> convenience, gated to the benign axes only. The `offensive` and `harness_safety` axes are
> Docker-only by design (they need live targets and network isolation). For real scored runs,
> use Docker.

## Requirements

- Python ≥ 3.11
- **Docker** (Docker Desktop on Windows/macOS) running locally for real runs — every task
  provisions its target(s) in a sandbox. (See the local smoke-test above to try the pipeline
  without it.) Architected to scale to Kubernetes later (`eval.yaml`/`compose.yaml` are kept
  k8s-compatible).

## Layout

```
src/agentsec/
  tasks/        one Inspect @task per capability axis
  solvers/      builtin_agent (raw API baseline) + bridge_harness (external CLIs)
  scorers/      flag, test_suite, static_scan, judge, composite (hybrid)
  registry.py   harness registry (cmd, env, image) for -S harness=<name>
  cli.py        config presets over `inspect eval`
  challenges/   data: one subdir per task instance (challenge.json + compose.yaml + assets)
                (bundled in the wheel; override location with AGENTSEC_CHALLENGES_DIR)
tests/          pytest: scorer unit tests + judge calibration
```

## Interpreting results

Scores are per-axis means in `[0, 1]` (partial credit on the hard/stretch tiers). Browse
full transcripts with `inspect view`, and render a comparison table across runs with:

```bash
python -m agentsec.cli report ./logs   # axis x harness x model x difficulty x score
```

Compare **models** by fixing the harness and varying `--model`; compare **harnesses** by
fixing `--model` and varying the bridged harness (see the differentiator section). For a
stable mean on stochastic challenges, use `--epochs N`.

## Contributing

New challenges need **no code** — drop a directory under `src/agentsec/challenges/<axis>/<name>/` with a
`challenge.json` (+ `compose.yaml` and assets) and it's picked up automatically. See
**[CHALLENGES.md](CHALLENGES.md)** for the schema, the subtask/partial-credit format, and the
no-egress rules. Run `pytest -q` and `ruff check src tests` before opening a PR. Contributions
of harder challenges (especially `stretch`-tier) and additional harness definitions are
welcome.

## Safety & responsible use

- All **offensive** and **harness_safety** challenges are **self-contained,
  intentionally-vulnerable, network-isolated** sandbox targets — `internal: true` networks
  with no host or internet egress, enforced by [tests/test_no_egress.py](tests/test_no_egress.py).
- **`sandbox_type=local`** executes agent-generated commands **on your machine**; it is gated
  to the benign coding axes and intended only for pipeline smoke-testing. Use Docker for real
  runs.
- AgentSec-Bench is intended for **authorized evaluation of AI agents' security capabilities
  and safety** — measuring and improving model/harness behavior. It is not a tool for
  attacking systems you don't own. Use it accordingly.

## Acknowledgements

Built on **[Inspect AI](https://inspect.aisi.org.uk/)** by the UK AI Safety Institute. The
offensive axis and challenge layout are inspired by
**[Cybench](https://cybench.github.io/)** and the `inspect_evals` cyber suite.

## Citation

```bibtex
@software{agentsec_bench,
  title  = {AgentSec-Bench: A Security Benchmark for LLM Agents and Harnesses},
  author = {Santhosraj},
  year   = {2026},
  url    = {https://github.com/Santhosraj/AgentSec-Bench}
}
```

## License

[MIT](LICENSE) © 2026 Santhosraj. AgentSec-Bench is provided for research and authorized
security evaluation; you are responsible for using it lawfully.
