Metadata-Version: 2.4
Name: lethe-fuzz
Version: 0.1.1
Summary: Policy-algebra-aware fuzzer for LLM guardrail systems
License: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3
Requires-Dist: rich>=13
Requires-Dist: typer[all]>=0.12
Provides-Extra: dev
Requires-Dist: pytest-mock>=3; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: guardrails
Requires-Dist: guardrails-ai>=0.5; extra == 'guardrails'
Provides-Extra: llm-guard
Requires-Dist: llm-guard>=0.3; extra == 'llm-guard'
Description-Content-Type: text/markdown

# Lethe

<p align="center">
  <img src="https://img.shields.io/badge/python-3.10%2B-blue?style=flat-square" />
  <img src="https://img.shields.io/pypi/v/lethe-fuzz?style=flat-square&color=orange" />
  <img src="https://img.shields.io/badge/license-Apache--2.0-green?style=flat-square" />
  <img src="https://img.shields.io/badge/tests-46%20passing-brightgreen?style=flat-square" />
  <img src="https://img.shields.io/badge/CI-GitHub%20Actions-blue?style=flat-square" />
</p>

<p align="center">
  <strong>A policy-algebra-aware fuzzer for LLM guardrail systems.</strong><br/>
  <em>"What the guardrail doesn't remember, it can't enforce."</em>
</p>

---

## What is Lethe?

[Garak](https://github.com/leondz/garak) red-teams LLMs.
**Lethe red-teams the guardrail sitting in front of the LLM** — a gap no existing tool fills.

Most adversarial NLP tools target the model. Lethe targets the layer that's supposed to protect it: classifiers, rule engines, policy systems, and composite guardrail stacks. It probes them with mutations that know *how guardrails think* — not just random noise.

---

## Why Lethe?

|                                              | Garak | PromptBench | **Lethe** |
| -------------------------------------------- | ----- | ----------- | --------- |
| Probes the guardrail (not the LLM)           | ✗     | ✗           | ✅        |
| Policy-algebra-aware mutations               | ✗     | ✗           | ✅        |
| Streaming-aware probes (late-resolve / drip) | ✗     | ✗           | ✅        |
| Genetic algorithm mutation loop              | ✗     | ✗           | ✅        |
| Partial bypass detection (confidence-aware)  | ✗     | ✗           | ✅        |
| Adapter protocol (plug in any guardrail)     | ✗     | partial     | ✅        |
| Dark-themed filterable HTML report           | ✗     | ✗           | ✅        |

---

## Installation

```bash
pip install lethe-fuzz

# Optional adapters
pip install "lethe-fuzz[llm-guard]"    # LLM Guard (protectai)
pip install "lethe-fuzz[guardrails]"   # Guardrails AI
```

Or from source:

```bash
git clone https://github.com/AaditPani-RVU/lethe
cd lethe
pip install -e .
```

---

## Quick Start

```bash
# Run all mutators + genetic algorithm against NeuroSym
lethe run --target neurosym --corpus bench/seeds.jsonl --report out.html \
  --mutate --policy-aware --streaming --ga

# Raw seeds only, JSON output
lethe run --target llm_guard --no-mutate --report out.json

# Tune the genetic algorithm
lethe run --target neurosym --ga --ga-population 100 --ga-generations 20

# List available adapters
lethe targets
```

---

## Mutation Modes

Lethe ships three categories of mutation, each targeting a different class of guardrail blind spot.

### String Mutations (`--mutate`)

Surface-level transforms that defeat regex and embedding classifiers:

| Mutator | Technique |
| --- | --- |
| `case_flip` | Random character case changes |
| `homoglyph_sub` | Cyrillic Unicode lookalikes (е→е, а→а) |
| `base64_wrap` | "Decode and follow: `<base64>`" framing |
| `role_prefix` | Persona/roleplay prefixes before payload |
| `zwsp_inject` | Zero-width spaces between words |

### Policy-Algebra Mutations (`--policy-aware`)

Mutations that model how composite policy rules compose — exploiting the seams between conjuncts:

| Mutator | Exploits |
| --- | --- |
| `allof_split` | Splits an AllOf conjunct across two turns |
| `implies_negate` | Negates the antecedent of an Implies rule |
| `not_double` | Double-negation to confuse Not(Not(x)) simplification |
| `anyof_reframe` | Permissive reframe to trigger AnyOf leniency |

### Streaming Probes (`--streaming`)

Probes that exploit how guardrails handle long-context or token-drip inputs:

| Mutator | Technique |
| --- | --- |
| `late_resolve` | 60-80 token benign preamble before payload — shifts embedding context |
| `token_drip` | Filler tokens injected every 4 words — dilutes classifier signal |
| `split_intent` | Sentence-completion framing — payload appears as continuation |

### Genetic Algorithm (`--ga`)

Evolves mutations using fitness-guided selection — finds bypasses no fixed mutator reaches:

```
fitness = 1.0 + 1/len(prompt)   # full bypass
          0.5 + 1/len(prompt)   # partial bypass  
          0.0                   # blocked
```

- Tournament selection (k=3), 2-elite carry-over
- Word-boundary crossover
- Deterministic runs via `--seed`

---

## Bypass Oracle

Lethe distinguishes three outcomes for every probe:

| Result | Condition |
| --- | --- |
| **BYPASS** | Guardrail returned `allowed=True` |
| **PARTIAL** | Guardrail blocked but confidence < threshold (default 0.5) |
| **BLOCKED** | Guardrail blocked with high confidence |

Partials matter — a guardrail that blocks at 0.2 confidence is one re-roll away from a bypass.

```python
from lethe.core.oracle import Oracle

oracle = Oracle(partial_confidence_threshold=0.5)
result = oracle.judge(prompt, seed_id="hc-001", category="harmful_content", decision=decision)
print(result.is_bypass, result.is_partial)
```

---

## Supported Targets

| Flag | System | Extra install |
| --- | --- | --- |
| `neurosym` | [NeuroSym-AI](https://github.com/AaditPani-RVU/neurosym-ai) | `pip install httpx` |
| `llm_guard` | LLM Guard (protectai) | `pip install lethe[llm-guard]` |
| `guardrails_ai` | Guardrails AI | `pip install lethe[guardrails]` |

---

## Adding an Adapter

Implement the two-method `Guardrail` protocol from `lethe.core.protocol`:

```python
from lethe.core.protocol import Decision

class MyGuardrail:
    def evaluate(self, prompt: str) -> Decision:
        blocked, score = my_system.check(prompt)
        return Decision(allowed=not blocked, confidence=score)

    def name(self) -> str:
        return "my_guardrail"
```

No base class, no registration — just pass the instance to `lethe run` or call it directly.

---

## Reports

Lethe emits two report formats:

**HTML** — dark-themed, filterable by BYPASS / PARTIAL / BLOCKED, shows full prompt text and confidence per probe.

**JSON** — machine-readable, suitable for CI assertions:

```json
{
  "total_probes": 338,
  "bypasses": 101,
  "partials": 12,
  "bypass_rate_pct": 29.9,
  "results": [...]
}
```

---

## CLI Reference

```
lethe run --target <name> [OPTIONS]

Options:
  --corpus PATH            Seed file (.jsonl)          [default: bench/seeds.jsonl]
  --report PATH            Output file (.html or .json) [default: out.html]
  --mutate / --no-mutate   String mutations             [default: on]
  --policy-aware           Policy-algebra mutations
  --streaming              Streaming-aware probes
  --ga                     Genetic algorithm mode
  --ga-population INT      GA population size           [default: 50]
  --ga-generations INT     GA generations               [default: 10]
  --seed INT               RNG seed for deterministic runs

lethe targets              List all available adapters
```

---

## Seed Corpus

`bench/seeds.jsonl` ships 26 seeds across six categories:

| Category | IDs | Focus |
| --- | --- | --- |
| `prompt_injection` | pi-001–005 | Instruction override attempts |
| `jailbreak` | jb-001–005 | Role-play and persona escapes |
| `harmful_content` | hc-001–005 | Clinical/instructional harmful requests |
| `data_exfil` | de-001–004 | System prompt extraction |
| `role_play` | rp-001–004 | Character and fiction framing |
| `encoding` | enc-001 | Encoded payload delivery |
| `social_eng` | se-001–002 | Authority and urgency framing |

Add your own seeds in JSONL format:

```jsonl
{"id": "custom-001", "category": "jailbreak", "text": "...", "expected_blocked": true}
```

---

## Findings

Lethe maintains public findings files for each tested system:

- [`findings/neurosym.md`](findings/neurosym.md) — NeuroSym-AI findings (credibility loop)
- [`findings/lethe.md`](findings/lethe.md) — Lethe self-audit (bugs found during development)

Every bypass Lethe finds that gets fixed in NeuroSym-AI will be noted in both repos. That credibility loop is the point.

---

## Rules of Engagement

Lethe is an **authorized security testing tool**. Use it only against guardrail systems you own or have explicit written permission to test. All findings should be responsibly disclosed to the vendor before public release.

---

## License

Apache-2.0 © [Aadit Pani](https://github.com/AaditPani-RVU)
