Metadata-Version: 2.4
Name: crewai-no-vibes
Version: 0.1.0
Summary: CrewAI Task guardrail blocking verification claims without evidence — empirical F1 0.815 on MAST mode 3.3
Author-email: Fernando Lazzarin <fernando@waitdead.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/waitdeadai/crewai-no-vibes
Project-URL: Repository, https://github.com/waitdeadai/crewai-no-vibes
Project-URL: Issues, https://github.com/waitdeadai/crewai-no-vibes/issues
Project-URL: Empirical baseline (MAST-EVAL), https://github.com/waitdeadai/llm-dark-patterns/blob/main/evaluation/MAST-RESULTS.md
Project-URL: Upstream rule pack, https://github.com/waitdeadai/agent-closeout-bench/blob/main/rules/closeout/evidence_claims.yaml
Keywords: crewai,guardrail,ai-safety,agent-safety,verification,mast,multi-agent,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Provides-Extra: crewai
Requires-Dist: crewai>=0.86; extra == "crewai"
Dynamic: license-file

# crewai-no-vibes

CrewAI Task guardrail blocking verification claims without evidence. Pure-Python, zero runtime dependencies, drop-in.

Catches the canonical agent failure: tasks closing out with "done", "verified", "fixed", "shipped", "no issues" when no actual evidence (commands run, test output, files inspected) is present.

## Empirical baseline

`no-vibes` is the strongest measured detector for [MAST mode 3.3 ("No or Incorrect Verification")](https://github.com/multi-agent-systems-failure-taxonomy/MAST) — the highest-prevalence multi-agent failure mode in the canonical taxonomy.

| Dataset | n | F1 | 95% CI | Precision | Recall |
|---|---|---|---|---|---|
| MAD human-labelled | 19 | **0.815** | [0.615, 0.941] | 0.733 | 0.917 |
| MAD LLM-judge full | 954 | **0.308** | [0.264, 0.352] | 0.226 | 0.486 |

Inter-annotator Fleiss κ for mode 3.3 on the released human-labelled set: **1.000** (perfect agreement). Bootstrap: percentile method, B=10000 resamples, seed=42.

Full empirical report with methodology, per-MAS-framework breakdown, and limitations:
https://github.com/waitdeadai/llm-dark-patterns/blob/main/evaluation/MAST-RESULTS.md

## Install

```bash
pip install crewai-no-vibes
```

Or directly from this repository:

```bash
pip install git+https://github.com/waitdeadai/crewai-no-vibes.git
```

Pure Python, supports 3.9+, no external binary required. CrewAI itself is an **optional** dependency — the guardrail function works without it, you just won't have anywhere useful to attach it.

## Usage

```python
from crewai import Agent, Task
from crewai_no_vibes import verification_claim_evidence_guardrail

researcher = Agent(role="Researcher", goal="...", backstory="...")

task = Task(
    description="Run the benchmark and report results.",
    expected_output="A summary with command output and verification status.",
    agent=researcher,
    guardrail=verification_claim_evidence_guardrail,
)
```

When the agent's output claims completion without evidence, the guardrail returns `(False, feedback)` and CrewAI retries the task with the feedback as context. CrewAI's `max_retries` (default 3) caps the retry loop.

## What it catches

The guardrail blocks outputs that trigger `done_without_evidence`:

> claims_completion AND (NOT has_evidence_marker OR completion_with_negative_evidence)

Where:

- `claims_completion`: message contains any of `done`, `complete`, `completed`, `ready`, `implemented`, `fixed`, `resolved`, `verified`, `deployed`, `shipped`, `all set`, `no issues`
- `has_evidence_marker`: message contains command evidence (e.g. `pytest`, `cargo test`, `commands run: ...`) OR verification-pass evidence (`verification: passed`, `tests passed`) OR read-only-audit evidence (`files inspected:`, `sources reviewed:`)
- `completion_with_negative_evidence`: completion claim PLUS a negative evidence marker (`verification: not run`, `unverified`, `tests not run`, `could not verify`) — i.e. "I'm done" while explicitly admitting verification didn't happen

### Catches

```
"Done. All set." → BLOCKED
"Fixed. No issues found." → BLOCKED
"Implemented and ready, with no observable command or file evidence." → BLOCKED
"All tests passed, verified. (Tests not run.)" → BLOCKED  # negative-evidence override
```

### Passes

```
"Implemented. Commands run: `pytest tests/` → 8 passed. Verification: passed." → OK
"Read-only audit complete. Files inspected: src/parser.py, tests/test_parser.py. No changes." → OK
"Status: partial. The scaffolding is drafted. Verification: not run because no fixture exists yet. Next step: write fixture." → OK  # honest partial with no completion claim
"Reviewing the design — no decision yet." → OK
```

## Configuration

The default guardrail uses the upstream rule pack's settings. For customization, use the factory:

```python
from crewai_no_vibes import make_guardrail

# Custom feedback message
guardrail = make_guardrail(
    custom_feedback="Show test output, please."
)

# Stricter mode: block on ANY completion claim without command/verification evidence,
# regardless of whether a negative-evidence marker is present.
strict_guardrail = make_guardrail(require_command_or_verification=True)
```

## Source ledger

This package is a pure-Python port of the `evidence_claims` rule pack from [`agent-closeout-bench`](https://github.com/waitdeadai/agent-closeout-bench), the Rust YAML rule pack engine maintained alongside [`llm-dark-patterns`](https://github.com/waitdeadai/llm-dark-patterns).

- Rule pack source: [`rules/closeout/evidence_claims.yaml`](https://github.com/waitdeadai/agent-closeout-bench/blob/main/rules/closeout/evidence_claims.yaml)
- Engine source: [`engine/src/main.rs`](https://github.com/waitdeadai/agent-closeout-bench/blob/main/engine/src/main.rs) `extract_features` + helpers
- Engine sha256 at port time: `ffe3c4e5dce01505...`
- Rule pack hash at port time: `sha256:26fa8fd9999c055d...`

If the upstream rule pack updates, this package version-pins to its own snapshot. Re-port for new versions; do not assume drift-free behavior.

## Honest limitations

1. **Cross-surface caveat (inherited from MAST-EVAL)**: the underlying detector was tuned for Claude Code closeout text. MAD is multi-agent trajectory text. The F1 0.815 number is cross-surface transfer agreement with the MAD LLM-as-judge overseer, not in-surface gold-label accuracy on CrewAI-specific output text. CrewAI users may see different effective precision/recall depending on their agent's output style.
2. **Pure-Python port may diverge from Rust engine on edge cases**: regex flavor (Rust `regex` crate vs Python `re`) and Unicode handling can differ. Tests cover the rule pack's documented examples and core helper predicates; if you find a divergence, please open an issue with the input that exposes it.
3. **No LLM-judge fallback**: this is the deterministic floor. For semantic-reasoning catches the rule pack misses, layer an LLM-based guardrail downstream (CrewAI also supports those via string-description guardrails).
4. **Retry loop bounded**: CrewAI's `max_retries` (default 3) caps retries. If the agent persistently outputs `done_without_evidence`, the loop exits with the latest output despite the guardrail blocks. Set `max_retries` and the task's `description` accordingly.

## Related work

- **MAST taxonomy** (Cemri et al., NeurIPS 2025): https://arxiv.org/abs/2503.13657 — peer-reviewed catalogue of 14 multi-agent failure modes. Mode 3.3 is what this guardrail targets.
- **ARCF** (Arora & Singh, IJERT 2026): https://www.ijert.org/a-reliability-control-framework-for-robust-multi-agent-llm-systems-managing-workflows-in-large-language-model-systems-ijertv15is050114 — real-time MAS reliability framework citing MAST; flags rule-based detection as a limitation. This package is one empirical data point on what rule-based / deterministic detection can achieve.
- **DarkBench** (Kran et al., ICLR 2025): https://arxiv.org/abs/2503.10728 — dark pattern corpus; complementary chat-surface evaluation.
- **CrewAI Task guardrails docs**: https://docs.crewai.com/en/concepts/tasks
- **CrewAI quickstart for guardrails**: https://github.com/crewAIInc/crewAI-quickstarts/blob/main/Guardrails/task_guardrails.ipynb
- **CrewAI PR #1742** (feature that introduced task guardrails): https://github.com/crewAIInc/crewAI/pull/1742

## Development

```bash
git clone https://github.com/waitdeadai/crewai-no-vibes.git
cd crewai-no-vibes
pip install -e ".[dev]"
pytest
```

Or without installing (system Python with PEP 668 protection):

```bash
PYTHONPATH=src pytest tests/
```

## License

Apache-2.0. See [LICENSE](LICENSE).

## Author

Fernando Lazzarin · [waitdead.com](https://waitdead.com) · [restlessmachine.com](https://restlessmachine.com)
