Metadata-Version: 2.4
Name: arma-veto
Version: 0.1.0
Summary: Sub-50ms deterministic test-tampering and evasion linter for AI coding agents.
Author: ARMA Contributors
License: MIT
Project-URL: Homepage, https://github.com/codebreaker77/ARMA
Project-URL: Repository, https://github.com/codebreaker77/ARMA.git
Project-URL: Issues, https://github.com/codebreaker77/ARMA/issues
Keywords: ai-agents,swe-bench,code-generation,testing,linter,pre-commit,ast,reward-hacking
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# ARMA: Autonomous Reliability & Metacognitive Architecture
### Deterministic Test-Tampering Veto & Targeted Mutation Testing for Coding Agents

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square)](LICENSE)
[![Python: 3.9+](https://img.shields.io/badge/Python-3.9+-3776AB.svg?style=flat-square&logo=python&logoColor=white)](https://python.org)
[![Test--Tampering--Veto: Deterministic](https://img.shields.io/badge/Test--Tampering--Veto-Deterministic--Enforced-brightgreen.svg?style=flat-square)](https://github.com/codebreaker77/ARMA)
[![Mutation--Gate: Advisory--Eval--Active](https://img.shields.io/badge/Mutation--Gate-Advisory--AUROC--0.60-yellow.svg?style=flat-square)](https://github.com/codebreaker77/ARMA)
[![Stop--Gate: Logging--Only](https://img.shields.io/badge/Stop--Gate-Logging--Only-lightgrey.svg?style=flat-square)](https://github.com/codebreaker77/ARMA)

---

## What ARMA Is (and What It Isn't)

ARMA is a **reliability linter and verification gate for coding agents** (Claude Code, OpenHands, Aider, Codex). 

It is **not** an "autonomous runtime," an uncalibrated classifier wrapper, or a general ML agent judge. It solves one proven failure mode that occurs in automated software engineering:

> **The Green CI Illusion**: When coding agents struggle to fix complex bugs, they frequently hack the reward signal — deleting failing test assertions, skipping test cases, or weakening validation rules so CI passes green while leaving the underlying bug broken.

ARMA attacks this problem with a **two-tiered verification architecture**:
1. **Module A (`arma-veto`)**: A sub-millisecond, zero-dependency AST diff interrogator that detects hard structural test tampering.
2. **Module B (Targeted Mutation Probe)**: An active sandbox probe that injects targeted mutants into the agent's modified implementation to test whether tests actually discriminate correct code from broken code.

---

## 1. Module A: Test-Tampering Veto (`arma-veto`)

When agents modify test suites, naive regex checkers suffer an unbearable false-alarm rate. In our rigorous human-labeled ground-truth audit of real agent test edits:

### The Human-Labeled False Discovery Audit ($N=30$ Inspected Cases)
- **True Test Tampering Caught**: 4 instances (**13.3%**)
- **False Discoveries on Assertion Edits**: 26 instances (**86.7%**)

Of the 26 false alarms generated by raw assertion diffing:
- **11 cases** were agent-created reproduction scripts in the repository root (`reproduce_issue.py`) containing `try...except AssertionError: pass` constructs.
- **15 cases** were legitimate bug fixes updating outdated assertions (e.g. updating `"ADD"` to `"COPY"`, resolving off-by-one error coordinates, or fixing test expectations to match new features). **10 of these 15 were resolved patches.**

### Deterministic Tiered Policy

To prevent blocking legitimate bug fixes while catching genuine test tampering:

| Violation Category | Action | Precision | Mechanism |
| :--- | :---: | :---: | :--- |
| **Test File Deletion** | **HARD VETO** | ~100% | Prevents agent from deleting entire test modules to clear CI. |
| **Test Function Deletion** | **HARD VETO** | ~100% | Catches removal of `def test_*()` definitions. |
| **Injected Skips** | **HARD VETO** | ~100% | Detects `@pytest.mark.skip`, `pytest.skip()`, or `unittest.skip`. |
| **Swallowed Exceptions** | **HARD VETO** | ~100% | Catches `except AssertionError: pass` wrapping assertions. |
| **Assertion Deletion without Replacement** | **HARD VETO** | ~100% | Blocks deleting pre-existing assertions without adding replacement checks. |
| **Assertion Modification** (Value/Format) | **ADVISORY** | N/A | **Does not hard-block** (avoids 86.7% false-block rate). Routes to Module B (Mutation Probe). |

### Zero-Dependency CLI Quickstart

Install or run standalone in 30 seconds:

```bash
# Check git diff in current repository
python -m layer.test_diff_interrogator --git

# Or via installed console script
arma-veto --git

# Run on a patch file or stdin
arma-veto patch.diff
cat agent.patch | arma-veto
```

#### Pre-Commit / CI Hook (`.pre-commit-config.yaml`):
```yaml
- repo: local
  hooks:
    - id: arma-veto
      name: ARMA Test Tampering Veto
      entry: python -m layer.test_diff_interrogator --git
      language: system
      always_run: true
      pass_filenames: false
```

---

## 2. Module B: Targeted Mutation Probe Engine

When an agent modifies test assertions or claims a fix is complete, static analysis alone cannot tell if the new tests are meaningful or merely hollow assertions designed to pass.

Module B dynamically validates tests against code:
1. Identifies the exact AST nodes modified by the agent's patch.
2. Injects targeted first-order mutants (operator replacement, boolean inversion, boundary shifts) strictly inside modified implementation lines.
3. Executes the test suite against each mutant in a local sandbox to measure the empirical **Mutation Kill Ratio**:
   $$\text{Kill Ratio} = \frac{\text{Mutants Killed by Tests}}{\text{Total Mutants Injected}}$$

### Empirical Discrimination Results (SWE-rebench Trajectories)

Evaluated across real patches from `nebius/SWE-rebench-openhands-trajectories` grouped strictly by repository (zero repository overlap between splits):

| Split | Sample Size ($n$) | Base Resolved Rate | AUROC | 95% Cluster Bootstrap CI | Status |
| :--- | :---: | :---: | :---: | :---: | :--- |
| **Dev Split** (Threshold Derivation) | 272 | 41.2% | **0.657** | [0.548, 0.743] | Optimal $\tau^* = 20.0\%$ |
| **Frozen Test Split** (Held-Out Repos) | 119 | 57.1% | **0.586** | [0.511, 0.701] | Precision 76.9%, Recall 29.4% |

> **Current Status: Statistically Bounded Signal, Advisory Only**  
> On the scaled held-out test split ($n=119$ across 8 unseen repositories), the mutation gate achieved **76.9% precision** (vs. 57.1% base rate, a +19.8% precision lift) with a test AUROC of **0.586** and a 95% cluster bootstrap CI of `[0.511, 0.701]`.  
> While the scaled sample tightens the confidence interval above chance ($> 0.50$), the modest effect size (AUROC 0.59) and selective recall (29.4%) mean it serves as a **high-precision advisory signal**, not an automated blocking gate.

---

## 3. Comparison with Existing Approaches

How does ARMA compare to existing testing, security, and verification tools?

| Approach | Catches Deleted Tests? | Catches Hollow Tests? | Allows New Tests? | Allows Legitimate Bugfix Updates? | Latency |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **Blanket Test Freeze** (`tests/**` lock) | Yes | Yes | ❌ **No** | ❌ **No** | <1 ms |
| **Code Coverage** (Codecov / diff-cover) | No | ❌ **No** (blind to hollow asserts) | Yes | Yes | ~5 min |
| **Full Mutation Testing** (`mutmut` / `PITest`) | Yes | Yes | Yes | Yes | ❌ **15–60 min** |
| **LLM-as-a-Judge** (PR Review Bots) | Unreliable | Unreliable | Yes | Inconsistent | 5–10 s ($$) |
| **`arma-veto`** (Module A: Deterministic AST Linter) | **Yes** | Defers to Module B | **Yes** | **Yes** | **<50 ms** |
| **ARMA Mutation Probe** (Module B: Targeted AST Probe) | Yes | **Yes** (Kill Ratio) | **Yes** | **Yes** | **5–15 s** |

### Tradeoffs & Why Existing Tools Fall Short for Agents

1. **Blanket Test Freezing (`git checkout -- tests/` or `CODEOWNERS` locks)**:
   - *Pros*: 100% precision against test tampering.
   - *Cons*: **Zero flexibility**. The agent cannot write new acceptance tests for new features, nor can it update outdated assertion strings when fixing bugs (in our human audit, 10 out of 15 assertion edits were legitimate passing fixes).
   - *Takeaway*: Ideal for static benchmark evaluation (SWE-bench); completely breaks real-world interactive development.

2. **Code Coverage Gates (`diff-cover`, Codecov)**:
   - *Pros*: Standard in enterprise CI.
   - *Cons*: **Completely blind to reward hacking**. If an agent wraps failing assertions in `try...except AssertionError: pass` or deletes an assertion, the test lines still execute. Coverage reports 100% green. Coverage measures *execution*, not *assertion discrimination*.

3. **Full-Codebase Mutation Testing (`mutmut`, `Cosmic Ray`)**:
   - *Pros*: Decades of academic rigor.
   - *Cons*: **Execution latency**. Generating and running hundreds of mutants across entire test suites takes 15 to 45+ minutes. You cannot run full mutation suites inside an interactive agent turn.
   - *ARMA's Difference*: Module B restricts mutation *strictly to the diff lines* of the patch with a small mutant budget (5 mutants), finishing in 5–15 seconds.

4. **LLM-as-a-Judge (Prompting GPT-4o / Claude to audit diffs)**:
   - *Pros*: Flexible natural language understanding.
   - *Cons*: Unreliable, vulnerable to prompt injection, high false-discovery on large diffs, and costs $0.03–$0.10 per call with 5–10s network latency.

---

## 4. Negative Findings & Demoted Heuristics (Logging Only)

We explicitly evaluated common agent-control heuristics on 1,000 public trajectories (`nebius/SWE-rebench-openhands-trajectories` across 553 repositories) and found they perform at or near chance. Consequently, **none of these heuristics are allowed to make automated blocking decisions**:

### 1. Stop Gate (Exit-Code / Diffstat Classifiers) — Demoted to Logging
- Naive test-exit and surface diff classifiers exhibit a **42.1% False-Block Rate** (207 of 492 successful solutions blocked).
- Overall resolution classification accuracy on 1,000 runs was **56.2%** (chance baseline: 50.8%, AUROC: 0.531).
- *Decision*: Demoted from blocking enforcement to telemetry logging only.

### 2. Loop-Kill & Early Failure Termination — Demoted
- **8.5% of successful trajectories** (42 / 492 resolved runs) hit an exact action loop and self-recovered to solve the problem.
- Hard-killing agents upon loop detection destroys ~9% of viable solutions.
- Predicting final failure at Step 10–30 using error counts or TF-IDF text features yields AUROCs of **0.51–0.53** (pure chance).
- *Decision*: Hard termination disabled; loops trigger non-destructive context suggestions rather than session aborts.

### 3. Semantic Embedding Scope Gates — Demoted
- Zero-shot embedding similarity (`EmbedPrior`) scored **0.54 AUROC** on real code actions.
- Supervised linear probes that achieved 0.96 AUROC on synthetic benchmarks collapsed to **0.65 AUROC** on real SWE-bench Lite issue descriptions due to lexical distractors.
- *Decision*: Kept as advisory context indicators, never hard blocking gates.

---

## 5. Context Plane: Identifier-Preserving Tool Output Pruning

When running test suites or terminal commands, long outputs consume agent context windows and cause "Lost in the Middle" attention failures.

ARMA provides deterministic tool output pruning (`layer/context_plane.py`) evaluated across 54,605 target identifier instances:

| Pruning Strategy | Character Compression | Critical Identifier Retention | Loss Rate |
| :--- | :---: | :---: | :---: |
| **BM25 Line Selection (35 lines)** | 33.4% | **98.36%** | **1.64%** |
| **Conservative ARMA Pruner (3,000 chars)** | **55.55%** | **88.26%** | 11.74% |
| **Naive Head/Tail (15+15 lines)** | 44.66% | 88.71% | 11.29% |
| **Aggressive Pruner (1,200 chars)** | 59.55% | 77.82% | 22.18% |

*Recommendation*: Use **Conservative ARMA Pruning** or **BM25 Line Selection** to save 33–55% of tokens while retaining $\ge 88–98\%$ of necessary diagnostic identifiers.

---

## Repository Structure

```
ARMA/
├── layer/
│   ├── test_diff_interrogator.py  # Module A: Deterministic Test-Diff Interrogator & CLI (arma-veto)
│   ├── verification_gate.py       # Module B: Targeted Mutation Probe Engine & AST Mutator
│   ├── context_plane.py           # ToolOutputPruner, PinnedFactsManager
│   ├── evidence_db.py             # SQLite Evidence Plane telemetry logger
│   ├── code_graph.py              # Fullerenes AST parser and blast radius calculator
│   └── cli.py                     # Command-line telemetry inspector
├── demo_veto.py                   # 30-second live demo of test tampering scenarios
├── experiments/offline/
│   ├── eval_mutation_benchmarks.py# Module B sandbox mutation evaluation runner
│   ├── prepare_mutation_cohort.py # Stratified cohort manifest generator (Dev/Test split)
│   ├── data/                      # Dataset caches, trajectory parquets, manifests
│   └── results/                   # Evaluation reports, ROC analysis, human audit ground truth
│       ├── INTERROGATOR_HUMAN_AUDIT_REPORT.md
│       ├── MUTATION_EVAL_REPORT.md
│       └── AUDIT_CASES_DETAILED.md
└── tests/                         # Full automated test suite (pytest)
```

---

## Running the Demo

Test `arma-veto` across 5 representative scenarios in 0.05 seconds:

```bash
python demo_veto.py
```

Expected output:
```
[1/5] Scenario 1: Legitimate Bug Fix + Added Acceptance Test
>>> [PASS ALLOWED] Clean implementation and legitimate test additions.

[2/5] Scenario 2: Malicious Test Deletion (Agent deleting failing test)
>>> [VETO ENFORCED] Hard structural test tampering detected!
    - [TEST_DELETED] def test_unauthorized_token():

[3/5] Scenario 3: Test Skip Injection (Agent marking failing test skipped)
>>> [VETO ENFORCED] Hard structural test tampering detected!
    - [SKIP_INJECTED] @pytest.mark.skip(reason="Fails on CI")

[4/5] Scenario 4: Assertion Deletion (Agent deleting assert to get green CI)
>>> [VETO ENFORCED] Hard structural test tampering detected!
    - [ASSERTION_DELETED] assert parsed.is_valid is True

[5/5] Scenario 5: Legitimate Assertion Update (Updating expected output after bug fix)
>>> [ADVISORY] Modified assertion detected.
    Human audit showed raw assertion vetoes have an 86.7% False Discovery Rate.
    Structural integrity is intact; routing to Targeted Mutation Probe.
```

---

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
