Metadata-Version: 2.4
Name: securevibebench
Version: 0.2.0
Summary: Generate and evaluate secure coding agents on SecureVibeBench
License-Expression: MIT
Project-URL: Homepage, https://github.com/iCSawyer/SecureVibeBench
Project-URL: Documentation, https://github.com/iCSawyer/SecureVibeBench/blob/main/docs/README.md
Project-URL: Issues, https://github.com/iCSawyer/SecureVibeBench/issues
Project-URL: Dataset, https://huggingface.co/datasets/iCSawyer/SecureVibeBench
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Requires-Dist: harbor<0.26.0,>=0.25.0
Requires-Dist: mini-swe-agent==2.4.6
Requires-Dist: huggingface-hub>=0.34.3
Requires-Dist: docker>=7.1.0
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: python-dotenv>=1.1.1
Dynamic: license-file

# SecureVibeBench

<p align="left">
    <img src="https://img.shields.io/badge/Version-v0.2-blue.svg?style=for-the-badge" alt="Version: v0.2">
    <a href="https://2026.aclweb.org/"><img src="https://img.shields.io/badge/Venue-ACL%202026-2ea44f.svg?style=for-the-badge" alt="Venue: ACL 2026"></a>
    <a href="https://arxiv.org/abs/2509.22097"><img src="https://img.shields.io/badge/arXiv-2509.22097-b31b1b.svg?style=for-the-badge" alt="arXiv"></a>
    <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/HF-SecureVibeBench-FFD21F.svg?style=for-the-badge&logo=huggingface" alt="Hugging Face: SecureVibeBench"></a>
    <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="License MIT"></a>
    <!-- <a href="https://chen-junkai.github.io/securevibebench-leaderboard/"><img src="https://img.shields.io/badge/Leaderboard-Online-2ea44f.svg?style=for-the-badge" alt="Leaderboard"></a> -->
</p>

> [!IMPORTANT]
> **SecureVibeBench is now v0.2!** Try `pip install securevibebench`!
>
> We refer to the original release and evaluation workflow as **v0.1**, and **results obtained with v0.2 may differ from v0.1.** The original task instructions, tests, and scoring logic remain unchanged. Please report the benchmark version and agent/model configuration when comparing results.

<p align="left">
    ✨&nbsp;<a href="#nav-news">News</a>
    | 🛠️&nbsp;<a href="#nav-quick-start">Quick Start</a>
    | 🔭&nbsp;<a href="#nav-overview">Overview</a>
    | 📚&nbsp;<a href="#nav-citation">Citation</a>
    | 🙏&nbsp;<a href="#nav-acknowledgments">Acknowledgments</a>
</p>

<a id="nav-news"></a>

## ✨ News

- **[2026-10-29]** We updated to **v0.2**, introducing a PyPI package, Harbor-format tasks, and unified model generation and evaluation. The original version is now referred to as **v0.1**; results may differ between versions.
<!-- - **[2026-06-06]** 🏆 We released the online [Leaderboard](https://chen-junkai.github.io/securevibebench-leaderboard/) for SecureVibeBench.
- **[2026-04-15]** 🤗 We support the usage via [Hugging Face Datasets](https://huggingface.co/datasets/iCSawyer/SecureVibeBench). 
- **[2026-04-11]** 🚀 We released code and data for SecureVibeBench. -->
- **[2026-04-07]** 🎉 Our paper has been accepted to **ACL 2026**.


<a id="nav-quick-start"></a>

## 🛠️ Quick Start

Requirements: Python 3.12+, Linux x86_64, Docker Engine with Compose, and a running Docker daemon.

### Install

We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments. pip also works; uv is not installed as a package dependency.

```bash
# Run without installing into your current environment
uvx securevibebench --help

# Or install into an active virtual environment
uv pip install securevibebench

# Or with pip
pip install securevibebench
```

Installation downloads all 105 tasks and the original evaluator from Hugging Face. Task containers are prepared when evaluation starts.

### For development

```bash
git clone https://github.com/iCSawyer/SecureVibeBench.git
cd SecureVibeBench
uv sync  # editable install + development tools
```

Use `uv run securevibebench eval ...` for the development checkout, or activate `.venv/bin/activate`.

Without uv, use `python tools/install.py --dev --installer pip` in an active virtual environment.

### Run evaluation

```bash
export OPENAI_API_KEY='<your-api-key>'
export MODEL='openai/<model-name>'

# With uv, without a persistent installation
uvx securevibebench eval --model "$MODEL" --task 992

# Or after installing the package
securevibebench eval --model "$MODEL" --task 992
```

Omit `--task 992` to run all 105 tasks. Static analysis is disabled by default. Enable it with `--run-sast TRUE` and configure Semgrep credentials such as `SEMGREP_APP_TOKEN`. Results are saved to `runs/<job>/summary.json` under the current working directory.

> [!NOTE]
> For custom endpoints, development setup, and more details, see the [Usage Guide](https://github.com/iCSawyer/SecureVibeBench/blob/main/docs/README.md). Run `securevibebench eval --help` for evaluation options.


<a id="nav-overview"></a>

## 🔭 Overview

SecureVibeBench is the **first** SWE-bench-level benchmark for secure vibe coding of agents, consisting of 105 C/C++ coding tasks sourced from real vulnerabilities ([OSS-Fuzz](https://github.com/google/oss-fuzz)/[ARVO](https://arxiv.org/abs/2408.02153)) covering various projects.

![Overview](https://raw.githubusercontent.com/iCSawyer/SecureVibeBench/main/assets/overview.png)

For each task in SecureVibeBench, we **reconstruct the real scenario where a human developer introduced a vulnerability** into the codebase, and then ask the agent to **implement the same requirements** and to see if the agent will also introduce the same vulnerability or not (and maybe new security issues as well).

To comprehensively evaluate the generated code of code agents, we conduct (i) **functional** correctness evaluation, (ii) PoV (proof-of-vulnerability) based **dynamic security** evaluation, and (iii) SAST-tool based **static security** evaluation.

> [!IMPORTANT]
> **Why SecureVibeBench?**
>
> - First SWE-bench-level, peer-reviewed benchmark for secure vibe coding.
> - Reconstruct coding scenarios where humans introduced vulnerabilities.
> - No other secure coding benchmark considering (i) functional correctness, (ii) PoV-based evaluation, and (iii) SAST tool-based new security issue detection.


<a id="nav-citation"></a>

## 📚 Citation

If you find SecureVibeBench useful, please cite:

```bibtex
@inproceedings{chen2026securevibebench,
  title     = "{S}ecure{V}ibe{B}ench: Benchmarking Secure Vibe Coding of {AI} Agents via Reconstructing Vulnerability-Introducing Scenarios",
  author    = "Chen, Junkai and Huang, Huihui and Lyu, Yunbo and An, Junwen and
               Shi, Jieke and Yang, Chengran and Zhang, Ting and Tian, Haoye and
               Li, Yikun and Li, Zhenhao and Zhou, Xin and Hu, Xing and Lo, David",
  editor    = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David",
  booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
  year      = "2026", month = jul, pages = "24144--24168",
  address   = "San Diego, California, United States",
  publisher = "Association for Computational Linguistics",
  url       = "https://aclanthology.org/2026.acl-long.1107/",
  ISBN      = "979-8-89176-390-6"
}
```

<a id="nav-acknowledgments"></a>

## 🙏 Acknowledgments

Our work builds on [OSS-Fuzz](https://github.com/google/oss-fuzz) and [ARVO](https://arxiv.org/abs/2408.02153).

```bibtex
@misc{mei2024arvoatlasreproduciblevulnerabilities,
  title         = {ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software},
  author        = {Xiang Mei and Pulkit Singh Singaria and Jordi Del Castillo and Haoran Xi and
                   Abdelouahab and Benchikh and Tiffany Bao and Ruoyu Wang and Yan Shoshitaishvili and
                   Adam Doupé and Hammond Pearce and Brendan Dolan-Gavitt},
  year          = {2024}, eprint = {2408.02153},
  archivePrefix = {arXiv}, primaryClass = {cs.CR},
  url           = {https://arxiv.org/abs/2408.02153}
}
```

```bibtex
@article{serebryany2017oss,
  title  = {$\{$OSS-Fuzz$\}$-Google's continuous fuzzing service for open source software},
  author = {Serebryany, Kostya},
  year   = {2017}
}
```
