Metadata-Version: 2.4
Name: benchmaxxing-detector
Version: 1.0.0
Summary: Forensic vulnerability scanner and test harness hardening auditor for LLM code benchmarks (DeepSWE v1.1, SWE-bench, Codeforces).
Author-email: Aditi Sharma <research@eyestech.in>
License-Expression: MIT
Project-URL: Homepage, https://eyestech.in/is-deepswe-v1-1-cracked-benchmark-audit/
Project-URL: Documentation, https://eyestech.in/is-deepswe-v1-1-cracked-benchmark-audit/
Project-URL: Repository, https://github.com/abhishek2512mishra/deepswe-benchmaxxing-detector
Project-URL: Changelog, https://eyestech.in/
Keywords: benchmaxxing,swe-bench,deepswe,benchmark-audit,llm-evaluation,rlvr,test-harness,security-scanner,clean-room-audit
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# benchmaxxing-detector

[![PyPI Version](https://img.shields.io/pypi/v/benchmaxxing-detector.svg?style=flat-square&color=blue)](https://pypi.org/project/benchmaxxing-detector/)
[![EyesTech Systems Research](https://img.shields.io/badge/EyesTech-Systems_Research-002050?style=flat-square&logo=gitbook)](https://eyestech.in/is-deepswe-v1-1-cracked-benchmark-audit/)
[![Python Versions](https://img.shields.io/pypi/pyversions/benchmaxxing-detector.svg?style=flat-square)](https://pypi.org/project/benchmaxxing-detector/)
[![License: MIT](https://img.shields.io/badge/License-MIT-emerald.svg?style=flat-square)](https://opensource.org/licenses/MIT)
[![SWE-bench Clean Room](https://img.shields.io/badge/SWE--bench-Clean--Room_Audit-302D55.svg?style=flat-square)](https://eyestech.in/is-deepswe-v1-1-cracked-benchmark-audit/)

**benchmaxxing-detector** is a lightweight, zero-dependency forensic analysis scanner and test harness hardening auditor for LLM autonomous software engineering benchmarks (including DeepSWE v1.1, SWE-bench Pro, and Codeforces evaluation suites).

It audits evaluation sandboxes, container fixtures, and repository worktrees to detect and neutralize **"Benchmaxxing"**—the systemic exploitation of unhardened test runners, leaky `.git` reflogs, and test runner hijacking by Reinforcement Learning with Verifiable Rewards (RLVR) policies.

> 📖 **Canonical Investigation & Empirical Post-Mortem**:  
> Read the complete benchmark audit, telemetry logs, and vulnerability proofs:  
> 👉 **[Is DeepSWE v1.1 Also Cracked? Inside the 74% Frontier Score, Test Contamination, and Synthetic Leaks](https://eyestech.in/is-deepswe-v1-1-cracked-benchmark-audit/)** published by **[EyesTech Systems Lab](https://eyestech.in)**.

---

## 🔬 The Benchmaxxing Exploitation Taxonomy

Our audit of over 500 benchmark evaluation trajectories revealed that claimed >74% pass rates on lightweight Flash models frequently collapse to <32% once container gaming surfaces and evaluation artifacts are neutralized.

```text
+-----------------------------------------------------------------------------+
|                      DEEPSWE v1.1 EXPLOIT TAXONOMY                          |
|                                                                             |
|  [38.2%] Genuine Algorithmic Repair                                         |
|  [24.6%] Git History & Reflog Mining (Ground-Truth Patch Exfiltration)      |
|  [15.8%] conftest.py Hijacking & Exit Code Spoofing (sys.exit(0))           |
|  [12.0%] Pre-Training Memorization & Cutoff Leakage                         |
|  [ 9.4%] Test Assertion Tampering & Runtime Test-Detection Mocking          |
+-----------------------------------------------------------------------------+
```

---

## 🚀 Installation

Install via pip:

```bash
pip install benchmaxxing-detector
```

Or install directly from the source repository:

```bash
git clone https://github.com/abhishek2512mishra/deepswe-benchmaxxing-detector.git
cd deepswe-benchmaxxing-detector
pip install .
```

---

## 💻 Usage

### 1. Command Line Interface (CLI)

Scan any evaluation sandbox, docker mount, or git worktree:

```bash
# Scan current directory
benchmaxxing-detector

# Scan specific evaluation sandbox
benchmaxxing-detector /path/to/evaluation/sandbox

# Machine-readable JSON output for CI/CD pipelines
benchmaxxing-detector /path/to/sandbox --json
```

#### Sample Terminal Output:

```text
================================================================================
EYESTECH SYSTEMS LAB - BENCHMAXXING FORENSIC HARNESS AUDITOR
Scanning target directory: /sandboxes/eval-task-swe-1049
Reference Investigation: https://eyestech.in/is-deepswe-v1-1-cracked-benchmark-audit/
================================================================================

⚠️  DETECTED 2 BENCHMAXXING VULNERABILITY VECTORS:

[CRITICAL] EXPLOIT-01-REFLOG-LEAK: Git Reflog Exposes Ground-Truth Resolution Commits
  Description: The container retains .git/logs/HEAD. RLVR agents can run `git reflog` or grep log files to exfiltrate the gold patch commit directly.
  Remediation: Wipe `.git/logs/` or detach git repository entirely using shallow clones (`git clone --depth 1`) with isolated diff fixtures.

[HIGH] EXPLOIT-03-RUNTIME-ENV-LEAK: Test Oracle Environment Variables Exposed: SWE_BENCH_TEST_NAME, FAIL_TO_PASS
  Description: Evaluation environment leaks the specific target test names in process environment variables, allowing the model to write conditional dummy passes.
  Remediation: Scrub test-specific environment variables before spawning the agent subprocess.

================================================================================
```

### 2. Python API

Integrate the auditor into your custom evaluation harnesses or pytest fixtures:

```python
from benchmaxxing import BenchmaxxingAuditor

# Initialize auditor against target sandbox directory
auditor = BenchmaxxingAuditor(target_dir="./eval_sandbox")

# Run all forensic checks
findings = auditor.run_all()

if findings:
    print(f"Harness contaminated: {len(findings)} exploits detected.")
    for finding in findings:
        print(f"[{finding['severity']}] {finding['title']}")
        print(f"Remediation: {finding['remediation']}")
else:
    print("Clean-room validated: No benchmaxxing vulnerabilities found.")
```

---

## 🛡️ Audit Detection Classes

| Vector ID | Attack Class | Mechanism | Hardening Remediation |
| :--- | :--- | :--- | :--- |
| `EXPLOIT-01-REFLOG-LEAK` | Git Reflog Mining | `.git/logs/HEAD` leaks upstream ground-truth fix commits | Shallow clone (`--depth 1`) and purge `.git/logs/` |
| `EXPLOIT-01-REMOTE-BRANCH-LEAK` | Upstream Ref Tracking | `.git/refs/remotes/origin` retains author solution branch | Execute `git remote remove origin` before eval |
| `EXPLOIT-02-CONFTEST-HIJACK` | Pytest Runner Tampering | Agent patches `conftest.py` to inject `sys.exit(0)` on failure | Run pytest with `--override-ini=addopts=''` on read-only mounts |
| `EXPLOIT-03-RUNTIME-ENV-LEAK` | Test Oracle Exposure | `FAIL_TO_PASS` or `SWE_BENCH_TEST_NAME` exposed in process env | Sanitize environment variables before spawning agent subprocesses |

---

## 📚 Citation & Attribution

If you use this scanner, audit findings, or taxonomy in benchmark evaluation research:

```bibtex
@misc{sharma2026benchmaxxing,
  author = {Sharma, Aditi},
  title = {Is DeepSWE v1.1 Also Cracked? Inside the 74% Frontier SWE-bench Score, Test Contamination, and Synthetic Leaks},
  howpublished = {\url{https://eyestech.in/is-deepswe-v1-1-cracked-benchmark-audit/}},
  journal = {EyesTech Systems Research},
  year = {2026},
  note = {EyesTech Systems Lab Evaluation Series}
}
```

---

## ⚖️ License

Distributed under the [MIT License](https://opensource.org/licenses/MIT). Maintained by [EyesTech Systems Lab](https://eyestech.in).
