Metadata-Version: 2.5
Name: lurebench
Version: 0.9.1
Summary: A maintained benchmark and evaluation harness for detecting AI-generated fraud lures (phishing, BEC, romance / pig-butchering).
Project-URL: Homepage, https://github.com/immu4989/lurebench
Project-URL: Repository, https://github.com/immu4989/lurebench
Project-URL: Issues, https://github.com/immu4989/lurebench/issues
Author: Imran Ahamed
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-security,benchmark,fraud-detection,llm,machine-generated-text,phishing,social-engineering
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.10
Provides-Extra: all
Requires-Dist: accelerate>=0.26; extra == 'all'
Requires-Dist: anthropic>=0.40; extra == 'all'
Requires-Dist: huggingface-hub>=0.23; extra == 'all'
Requires-Dist: joblib>=1.3; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: scikit-learn<1.7,>=1.6; extra == 'all'
Requires-Dist: torch>=2.0; extra == 'all'
Requires-Dist: transformers>=4.38; extra == 'all'
Provides-Extra: binoculars
Requires-Dist: torch>=2.0; extra == 'binoculars'
Requires-Dist: transformers>=4.38; extra == 'binoculars'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: jsonschema>=4.23; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff<0.17,>=0.15; extra == 'dev'
Requires-Dist: stix2-validator>=3.2; extra == 'dev'
Provides-Extra: generate
Requires-Dist: anthropic>=0.40; extra == 'generate'
Provides-Extra: hub
Requires-Dist: huggingface-hub>=0.23; extra == 'hub'
Provides-Extra: llamaguard
Requires-Dist: accelerate>=0.26; extra == 'llamaguard'
Requires-Dist: torch>=2.0; extra == 'llamaguard'
Requires-Dist: transformers>=4.38; extra == 'llamaguard'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: train
Requires-Dist: joblib>=1.3; extra == 'train'
Requires-Dist: scikit-learn<1.7,>=1.6; extra == 'train'
Description-Content-Type: text/markdown

<div align="center">

<img src="docs/assets/lurebench-atlas.gif" width="100%" alt="Animated LureBench research atlas moving from corpus construction through adversarial stress, calibration, and reporting.">

### The adversarial benchmark for AI-generated fraud

One schema. Three evaluation regimes. Honest answers about what survives deployment.

[![CI](https://github.com/immu4989/lurebench/actions/workflows/ci.yml/badge.svg)](https://github.com/immu4989/lurebench/actions/workflows/ci.yml)
![Version](https://img.shields.io/badge/version-0.9.1-57f2c1)
![License](https://img.shields.io/badge/license-Apache_2.0-2a78d6)
![Python](https://img.shields.io/badge/python-3.10%2B-1baf7a)
![Generators](https://img.shields.io/badge/generators-DeepSeek_·_GLM_·_Mistral-eda100)
![Status](https://img.shields.io/badge/status-research_pilot-e34948)
[![Code of Conduct](https://img.shields.io/badge/code%20of%20conduct-Contributor%20Covenant-5c6470)](CODE_OF_CONDUCT.md)
[![Security policy](https://img.shields.io/badge/security-policy-5c6470)](SECURITY.md)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21631777.svg)](https://doi.org/10.5281/zenodo.21631777)

</div>

<p align="center">
  <a href="#the-finding"><strong>See the finding</strong></a> ·
  <a href="#quickstart"><strong>Run the benchmark</strong></a> ·
  <a href="https://immu4989.github.io/lurescope/"><strong>Open the LureScope browser lab ↗</strong></a>
</p>

---

Fraud detectors that score well on classic spam corpora fall apart on lures written by modern language models. LureBench measures that gap on a common footing: one schema, one harness, one leaderboard, across fraud typologies and generator families. It runs out of the box with no model downloads or API keys, and it ships baseline detectors from a keyword heuristic up to a trained classifier.

More than a corpus, it is a **method for building the corpus honestly**. Getting a credible answer to "can you detect AI-generated fraud?" turned out to require finding, and removing, a dataset confound that makes the problem look far easier than it is. That story is below.

> **New — thresholds with evidence, not just point estimates.** LureBench can now
> export a policy only when exact finite-sample tests support a requested
> population false-positive rate. A 1% target at 95% confidence fails closed with
> fewer than 299 benign validation examples even after zero false alarms. See
> [risk-controlled deployment policies](docs/RISK_CONTROL.md).

### Who it's for

| You are… | Use LureBench to… |
|---|---|
| **A security engineer / vendor** | Benchmark your fraud detector on a common footing, then stress-test it against attacks a real fraudster would run (`robustness`) before you trust its clean-data score. |
| **A researcher** | Reproduce the provenance confound and its removal, add a detector in ~30 lines ([docs/adding-a-detector.md](docs/adding-a-detector.md)), or extend the corpus with new generators and typologies. |
| **A policy / threat-intel analyst** | Ground claims about "AI-generated fraud detection" in measured numbers — including where it works, and where it is close to a coin flip. |

Everything runs out of the box with no model downloads or API keys; provider keys are only needed to *generate* new lures or run LLM-based attacks, and never touch api.openai.com or api.anthropic.com.

## The finding

Train a classifier to tell AI-written fraud from human-written fraud on a naively assembled corpus, and it looks almost perfect: near-100% recall, a 0.1% false-alarm rate, and it even generalizes to generators it never trained on. That result is a trap.

<p align="center">
  <img src="docs/assets/provenance.svg" width="720" alt="Bar chart: cross-generator AI-fraud detection AUC drops from a perfect 1.00 on the naive corpus to 0.58 (DeepSeek), 0.57 (GLM) and 0.83 (Mistral) once the corpus is distribution-matched, with two of three near the 0.50 chance line.">
</p>

Inspecting the model showed it was separating **corpus-of-origin**, not authorship: the human phishing was older, longer, pre-tokenized, and defanged differently than the fresh LLM text. Once the two classes are distribution-matched (each human lure paired with an AI rewrite of the *same* lure, matched on length and defanged the same way), the separation falls apart. Cross-generator AUC drops from a perfect 1.00 to **0.58 and 0.57** for two of the three generators, barely above the 0.50 chance line. Only one model's output (Mistral) keeps a detectable signature at 0.83, and it does not transfer to the others. Distinguishing AI-authored fraud from human-authored fraud, across generators, turns out to be close to a coin flip. Full write-up in [docs/provenance_results.md](docs/provenance_results.md).

## Two tasks, and why the distinction matters

A fraud lure raises two separate questions, and most tools answer only one:

- **Is this a fraud lure?** (the `fraud` task, lure vs. benign)
- **Was this written by an LLM?** (the `provenance` task, AI vs. human)

The first is largely a solved classical problem. A trained bag-of-words baseline near-solves it, while a keyword heuristic fails on exactly the AI lures a keyword heuristic should fail on:

<p align="center">
  <img src="docs/assets/detection.svg" width="720" alt="Bar chart: detection rate by typology. heuristic-v0 scores 20% phishing, 83% BEC, 15% romance, 9% pig-butchering; tfidf-logreg scores 95-100% across all four.">
</p>

The second question, provenance, is where the real difficulty lives, and where the confound above had to be removed before the number meant anything.

## Clean accuracy is not deployment accuracy

A detector's score on clean test data is not the number that survives contact with a real fraudster. The adversary does not send the lure your model was trained on: they type `vеrifу` with a Cyrillic `е`, split a trigger word, or paraphrase the whole message. The `robustness` command measures what happens next. It takes the lures a detector **catches**, applies an attack, and reports the **attack success rate** — the fraction that now evade.

<p align="center">
  <img src="docs/assets/robustness.svg" width="720" alt="Bar chart: attack success rate by attack. heuristic-v0 collapses under every character attack (99% homoglyph, 99% leet, 100% zero-width, 52% whitespace); tfidf-logreg holds far better (38% homoglyph, 16% leet, 3% zero-width, 0% whitespace).">
</p>

Robustness is a *different axis* from clean accuracy and it ranks detectors differently. The keyword baseline looks cheap and interpretable until an attacker types one homoglyph and 99% of its catches walk through. The trained model degrades gracefully instead of collapsing. That gap — not the clean-data score — is what a buyer needs to see before deploying either. Attacks come in two tiers: free deterministic character tricks (`homoglyph`, `leet`, `zero-width`, `whitespace`) and stronger LLM rewrites (`llm-paraphrase`, `llm-keyword-evasion`, which targets a detector's own most-predictive words). Full write-up in [docs/adversarial-robustness.md](docs/adversarial-robustness.md).

> **Try it interactively** — [**live demo, no install**](https://huggingface.co/spaces/immu4989/lurescope): [**LureScope**](https://github.com/immu4989/lurescope) is the deployable companion — a small API and browser demo where you paste a message, score it, then watch an attack evade the detector live. It reuses these same detectors and attacks. Its [robustness scorecard writeup](https://github.com/immu4989/lurescope/blob/main/blog/2026-07-23-robustness-gap-fraud-detection.md) reports detector-by-attack evasion rates over this corpus and shows where a normalization defense recovers the catch (and where it cannot).

## How it works

```mermaid
flowchart LR
    S["Public corpora<br/>+ provider LLMs"] --> ING["Ingest<br/>defang · detokenize"]
    S --> GEN["Generate<br/>hard-mode · paired rewrite"]
    ING --> REV{{"Human review<br/>gate"}}
    GEN --> REV
    REV --> ASM["Assemble<br/>frozen train / test"]
    ASM --> EVAL["Evaluate<br/>leaderboard · cross-generator · robustness"]
    style REV fill:#fff3cd,stroke:#eda100,color:#0b0b0b
    style EVAL fill:#cde2fb,stroke:#2a78d6,color:#0b0b0b
    style GEN fill:#d6f2e6,stroke:#1baf7a,color:#0b0b0b
```

Every generated lure is defanged, provenance-logged, and held in a `review: pending` state until a human approves it. Nothing reaches a shard automatically. Train and test are split by a stable hash of each record id, so adding a new generator never reshuffles what was already in the test set.

## What's inside

The `lurebench-core` corpus (20,388 records):

| Class | Count | Detail |
|---|---|---|
| Human phishing + benign | 19,798 | `David-Egea/phishing-texts` (MIT), de-tokenized and defanged |
| AI-generated lures | 590 | across four typologies, three generators |
| — DeepSeek `deepseek-v4-pro` | 190 | |
| — GLM `glm-4.6` | 200 | |
| — Mistral `mistral-large-latest` | 200 | |

Typologies: phishing, BEC, romance, pig-butchering. The AI lures are hard-mode: written to persuade through plausibility and context rather than stock urgency and payment-demand markers.

## Quickstart

```bash
git clone https://github.com/immu4989/lurebench && cd lurebench
pip install -e .

# score the dependency-free heuristic on the sample shard (ships in the repo)
lurebench eval --dataset data/samples/lures.jsonl --detector heuristic-v0
```

The full `lurebench-core` corpus lives on the [Hugging Face Hub](https://huggingface.co/datasets/immu4989/lurebench-core). Load it in one call, no manual file placement (`v0.2`):

```python
from lurebench import load_core, run
from lurebench.detectors import HeuristicDetector

test = load_core("test")                       # downloads + caches from the Hub
print(run(HeuristicDetector(), test).metrics.mcc)
```

Reproduce the headline finding — the leave-one-generator-out provenance collapse — with one command. Point it at a naive corpus and AUC stays near 1.00 (the confound); point it at the distribution-matched set and it falls to the 0.50 chance line:

```bash
pip install -e ".[train]"
lurebench cross-generator -d data/full/paired/human.jsonl -d data/full/paired/deepseek-v4-pro.jsonl \
  -d data/full/paired/glm-4.6.jsonl -d data/full/paired/mistral-large-latest.jsonl
```

Stress-test a detector the way a real fraudster would — perturb the lures it catches and measure how many now evade (the **attack success rate**). Clean accuracy is not deployment accuracy:

```bash
lurebench robustness -d data/full/core/test.jsonl -m tfidf-logreg \
  -a homoglyph -a leet -a zero-width -a whitespace
```

The keyword baseline looks interpretable until an attacker types `vеrifу` once (ASR 0.99); the trained model degrades gracefully (homoglyph ASR 0.38). See [docs/adversarial-robustness.md](docs/adversarial-robustness.md).

Generation uses any OpenAI-compatible provider by name, with your own key:

```bash
export DEEPSEEK_API_KEY=...
lurebench generate --typology bec --n 50 --engine deepseek --hard --out staging/bec.jsonl
```

Export a dataset (or just the taxonomy) as a **STIX 2.1 bundle** for threat-intel sharing — validated against the official OASIS validator, with curated crosswalks to MITRE ATT&CK, FBI/IC3, and FinCEN:

```bash
lurebench stix -d data/full/core/test.jsonl -o lures.stix.json
lurebench stix --taxonomy-only -o taxonomy.stix.json
```

Measure the **cross-lingual gap** — how detectors hold up when the language shifts. Across eight languages the trained baseline posts a near-perfect ~1.00 recall, which looks like flawless multilingual detection; strip the defang placeholder and it splits along script lines — Latin-script survives, but every non-Latin script collapses (Chinese 1.00→0.09, Russian 0.94→0.06, Arabic 0.98→0.04), exposing the recall as a `<<link>>` artifact rather than detection (see [docs/multilingual.md](docs/multilingual.md)):

```bash
lurebench multilingual -d data/full/multilingual/eval.jsonl -m tfidf-logreg -m heuristic-v0
```

The fix for that gap is a detector that reads meaning, not tokens: **`llm-judge`** asks an LLM over the same provider plumbing as generation (your key, never OpenAI/Anthropic) and is the strongest detector you can run without a GPU or OpenAI credits. It closes the cross-lingual gap (non-Latin artifact-controlled recall ~0.06 → ~0.94) and shrugs off the character attacks that break the baselines (homoglyph attack-success 1.00 → 0.08), with a semantic paraphrase its one real weakness. Full write-up and honest limits in [docs/llm-detector.md](docs/llm-detector.md):

```bash
export DEEPSEEK_API_KEY=...   # or MISTRAL_API_KEY / ZHIPUAI_API_KEY
lurebench eval -d data/full/core/test.jsonl -m llm-judge
```

### Six models, four questions

With an aggregator key (`openrouter`) one run scores a whole panel, so the LLM rows are no longer a single provider's word. Measured across `gpt-5-nano`, `gemini-2.5-flash-lite`, `deepseek-v4-flash`, `qwen-2.5-7b`, `llama-3.1-8b` and `mistral-nemo`:

- **[Leaderboard](docs/leaderboard.md)** — every judge lands below the trained baseline on clean data (AUC 0.82–0.94 against tfidf's 0.99), and the `scored` column shows how many records each was willing to answer.
- **[Multilingual](docs/multilingual_llm.md)** — the practical win. Artifact-controlled, `deepseek-v4-flash` reads Arabic 0.93 / Russian 0.78 / Chinese 0.75 at a **1% false-positive rate**, where the trained baseline collapses to 0.04 / 0.06 / 0.09. Every row carries an FPR, because on an all-fraud shard a detector that flags everything scores a perfect 1.00 in nine languages.
- **[Provenance](docs/provenance_llm.md)** — the null result. Asked to tell AI-written fraud from human-written fraud on distribution-matched data, four of six models sit at or below chance (AUC 0.46–0.53), and three simply answer "AI" every time. The confound finding above is not something a bigger model reads its way out of.
- **[Adaptive robustness](docs/adaptive_robustness.md)** — the inversion. Let the attacker rewrite up to five times instead of once and the ordering flips: the trained baseline holds at 9% evasion while the judges reach 35–40%. Character attacks break token models and semantic attacks break LLM judges, which is an argument for running both. Rates are the mean of three replicates with the range reported, because a single run of this experiment moves by up to 17 points — hosted providers are not deterministic even at temperature 0.

```bash
export OPENROUTER_API_KEY=...
lurebench leaderboard -d data/full/core/test.jsonl \
  -m tfidf-logreg -m 'llm-judge@openrouter/deepseek/deepseek-v4-flash' \
  --cache-dir .cache/lb --workers 12
```

Fourteen commands cover the pipeline, including `audit-splits` for cross-split
near-duplicate detection and `calibrate` for validation-only policy export.
`assemble-core` produces frozen train/validation/test splits, and `eval
--bootstrap 2000` adds uncertainty intervals. See [benchmark validity](docs/VALIDITY.md),
the [finite-sample FPR control method](docs/RISK_CONTROL.md),
the [changelog](CHANGELOG.md), the [taxonomy & STIX guide](docs/taxonomy.md), and
[docs/adding-a-detector.md](docs/adding-a-detector.md) to contribute a detector.

## Why it matters

U.S. regulators and law enforcement have named this threat directly. FinCEN's Nov 2024 alert lists GenAI-generated **text** among its red-flag indicators and names BEC, spear phishing, elder exploitation, romance scams and virtual-currency investment ("pig-butchering") scams as active GenAI vectors. The FBI's Dec 2024 IC3 PSA warns that criminals use generative AI to produce fraudulent content at greater scale and believability. FS-ISAC cites a Deloitte projection of $40B in U.S. AI-enabled fraud losses by 2027.

LureBench maps its typologies onto exactly those frameworks. The [taxonomy](docs/taxonomy.md) carries curated crosswalks to MITRE ATT&CK, the FBI/IC3 crime categories, and FinCEN advisories, and the `stix` command emits standards-compliant STIX 2.1 — so a detection can travel from a detector to a fusion center, an ISAC, or a SAR narrative without being re-described.

## Responsible use

This is a defensive research project. The corpus exists to train and evaluate detectors. Controlled generation produces defanged, clearly-synthetic, review-gated text. It does not personalize lures to real targets, embed working links or payment rails, or deliver anything. See [DATA.md](DATA.md), [docs/SHARD_SPEC.md](docs/SHARD_SPEC.md) and [CONTRIBUTING.md](CONTRIBUTING.md).

## Honest limitations

LureBench is an early pilot, and the writeups say so plainly:

- The distribution-matched provenance result covers three generators and phishing only (the human data is phishing-only), with a few hundred paired rewrites per generator.
- The human corpus is older-era phishing. De-tokenization and rewriting remove the largest tells; the residual signal is register and style, which is arguably legitimate authorship signal, but a contemporary human-fraud source would be stronger.
- Audio and video deepfake fraud are out of scope. They are well served by existing benchmarks (ASVspoof 5, Deepfake-Eval-2024, VishGPT); LureBench covers text.

## Contributing

Detectors, data shards, attacks, and corrections to published numbers are all
welcome. Start with [CONTRIBUTING.md](CONTRIBUTING.md) and
[docs/adding-a-detector.md](docs/adding-a-detector.md); a detector is about thirty
lines. Community expectations are in [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

[SECURITY.md](SECURITY.md) covers both conventional vulnerabilities and what is
*not* one here: the corpus containing fraud lures, the attacks succeeding, and
detectors scoring badly are all intended. It also describes how to report a
measurement error, which for a benchmark matters as much as a memory-safety bug.

Release history is in [CHANGELOG.md](CHANGELOG.md).

## Citation

See [CITATION.cff](CITATION.cff). Archived releases carry a DOI: cite the concept DOI [10.5281/zenodo.21631777](https://doi.org/10.5281/zenodo.21631777), which always resolves to the latest version. Licensed under Apache-2.0.

[paper.md](paper.md) is an unsubmitted draft of a software paper, kept in the repo
so the framing evolves with the code. It has not been peer reviewed and should not
be cited as a publication.
