Metadata-Version: 2.5
Name: lmtwt
Version: 0.2.0
Summary: Let Me Talk With Them — security testing framework for LLM prompt injection and jailbreak resistance
Project-URL: Homepage, https://github.com/tin537/LMTWT
Project-URL: Issues, https://github.com/tin537/LMTWT/issues
Author-email: Tanuphat Tin <tanuphat.chai@gmail.com>
License: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.10
Requires-Dist: aiolimiter>=1.1
Requires-Dist: anthropic>=0.5.0
Requires-Dist: google-genai>=1.0
Requires-Dist: httpx>=0.27
Requires-Dist: matplotlib>=3.7.0
Requires-Dist: openai>=1.0.0
Requires-Dist: pandas>=1.5.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: tenacity>=8.2.0
Provides-Extra: api
Requires-Dist: fastapi>=0.110; extra == 'api'
Requires-Dist: sse-starlette>=2.0; extra == 'api'
Requires-Dist: uvicorn[standard]>=0.27; extra == 'api'
Provides-Extra: local
Requires-Dist: accelerate>=0.20.0; extra == 'local'
Requires-Dist: torch>=2.0.0; extra == 'local'
Requires-Dist: transformers>=4.30.0; extra == 'local'
Provides-Extra: report
Requires-Dist: weasyprint>=60; extra == 'report'
Provides-Extra: web
Requires-Dist: gradio>=3.50.0; extra == 'web'
Description-Content-Type: text/markdown

# LMTWT — Let Me Talk With Them

<p align="center">
  <img src="https://img.shields.io/badge/Python-3.10%2B-blue" alt="Python 3.10+">
  <img src="https://img.shields.io/badge/License-MIT-green" alt="License: MIT">
  <img src="https://img.shields.io/badge/Contributions-Welcome-brightgreen" alt="Contributions: Welcome">
</p>

LMTWT is an async-first security testing framework for evaluating **LLM
resistance** to prompt injection, jailbreaks, and tool-use attacks. It pits
one model (the **attacker**) against another (the **target**) and reports
whether the target was compromised — automatically, at scale, against
frontier APIs or custom backends behind your own protocol.

> **Scope: LLM-only.** LMTWT attacks the model's behavior, training, context,
> tools, and conversation memory. It is **not** a general web/network/API
> pentester — for SQLi, XSS, IDOR, port scanning, OAuth flows, and the rest
> of the OWASP Top 10, use Burp / ZAP / nuclei / nmap. LMTWT speaks
> production chatbot protocols (Socket.IO, custom WS, JWT-auth) only because
> that's how you reach the LLM behind them.

## What you can hit

- **Hosted LLMs** — OpenAI, Anthropic, Gemini
- **Local LLMs** — Hugging Face transformers, **LM Studio** (OpenAI-compatible)
- **Agent runtimes** — **Claude Code via ACP** (Agent Client Protocol over stdio)
- **Custom backends via `external-api`** — your own chatbot at any of:
  - **HTTP** (single round-trip JSON)
  - **SSE** (Server-Sent Events streaming)
  - **WebSocket** (raw frame protocol)
  - **Socket.IO** (v5/EIO v4 *or* v2/EIO v3, with ack + event correlation)

Anything you can describe in a JSON config can be a target — payload templates,
auth headers, dotted-path response extraction, ack handling, the lot.

## What it does to them

- **Single-shot attack templates** (`--mode template`) — curated injection / jailbreak prompts
- **Probe sweeps** (`--probe-mode`) — eight built-in vulnerability categories
- **Multi-turn flows** (`--mode multi-turn`) — scripted social-engineering arcs
- **Tool-use attacks** (`--mode tool-use`) — indirect prompt injection via tool results
- **Refinement strategies** (`--strategy pair|tap`) — automated PAIR / TAP attack search
- **Hacker mode** (`--hacker-mode`) — attacker reads conversation history and adapts
- **Three judges**: regex, LLM-based, or ensemble for success detection

## OWASP LLM Top 10 coverage

Every probe in the YAML corpus (`src/lmtwt/probes/library/`) carries the
OWASP LLM Top 10 (2025) IDs it exercises; findings and reports inherit them.
Counts below are probes tagged with each ID (a probe can carry several).

| ID | Risk | Probes |
|---|---|---|
| LLM01 | Prompt Injection | 55 |
| LLM02 | Sensitive Information Disclosure | 13 |
| LLM03 | Supply Chain | — (out of scope: not observable from the chat interface) |
| LLM04 | Data and Model Poisoning | — (out of scope: needs training-pipeline access) |
| LLM05 | Improper Output Handling | 6 |
| LLM06 | Excessive Agency | 24 |
| LLM07 | System Prompt Leakage | 15 |
| LLM08 | Vector and Embedding Weaknesses | 5 |
| LLM09 | Misinformation | 6 |
| LLM10 | Unbounded Consumption | 8 |

Mapping rules per attack vector live in [`docs/taxonomy.md`](docs/taxonomy.md).

## Install

```bash
git clone https://github.com/tin537/LMTWT.git
cd LMTWT

# Recommended: uv (https://docs.astral.sh/uv/)
uv sync

# Or plain pip
pip install -e .

# Optional: local Hugging Face inference
pip install -e '.[local]'

# Optional: Web UI (Gradio)
pip install -e '.[web]'
```

Python 3.10+ is required. GPU acceleration (CUDA / Apple MPS) is auto-detected
when PyTorch is installed.

## Configure credentials

Create `.env` at the repo root (only the providers you'll use):

```bash
GEMINI_API_KEY=...
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
HUGGINGFACE_API_KEY=hf_...     # optional, only for gated models
```

LM Studio and Claude Code (ACP) need no key — they run locally.

## Quick start

The fastest way to use LMTWT is the `scan` subcommand — one command, full
vulnerability sweep, complete engagement bundle out:

```bash
# Full scan: fingerprint → catalog → adaptive → climb → pollinate → chatbot attacks
lmtwt scan --target openai --attacker gemini

# Preview the plan without firing
lmtwt scan --target openai --dry-run

# Quick scan (catalog + fingerprint only, ~2 min)
lmtwt scan --target openai --depth quick

# Thorough scan (above + self-play + N=10 repeats, ~hours)
lmtwt scan --target openai --depth thorough

# Custom backend behind your own protocol
lmtwt scan --target external-api --target-config examples/custom_api_target.json
```

The bundle lands in `./scan-<date>-<target>/`:

```
scan.json    report.md    report.html    report.pdf    scorecard.md
fingerprint.json    plan.json    scan.db    repro/F00N_<id>.json
```

You hand the folder to a client and they have everything: human-readable
report, machine-readable JSON, per-finding repro packs, and a queryable
SQLite db.

For granular control (run only one technique, tune individual knobs), the
legacy flat CLI still works:

```bash
# Interactive: Gemini attacking OpenAI
./run.sh --attacker gemini --target openai --mode interactive

# Local-only: LM Studio attacking LM Studio (no API costs)
./run.sh --attacker lmstudio --attacker-model "qwen2.5-7b" \
         --target lmstudio  --target-model "llama-3.1-8b" \
         --mode interactive

# Web UI (Gradio)
./run.sh --web

# Web UI (FastAPI + SSE — pip install -e '.[api]')
lmtwt --web-api

# Custom OpenAI-compatible attacker (Ollama, vLLM, LiteLLM, self-hosted gateway)
export OPENAI_COMPAT_BASE_URL=http://localhost:11434/v1
export OPENAI_COMPAT_MODEL=qwen2.5:7b
lmtwt scan --target openai --attacker openai-compat
```

## Usage examples

### Targeting different model backends

```bash
# Hosted: Claude as the target
./run.sh --attacker gemini --target anthropic

# Local: Hugging Face transformer
./run.sh --attacker gemini --target huggingface \
         --target-model "mistralai/Mistral-7B-Instruct-v0.2"

# Local: LM Studio (OpenAI-compatible REST on localhost:1234)
./run.sh --attacker gemini --target lmstudio --target-model "your-model-id"

# Agent runtime: Claude Code over ACP
./run.sh --attacker gemini --target claude-code

# Custom HTTP backend
./run.sh --attacker gemini --target external-api \
         --target-config examples/custom_api_target.json

# Custom Socket.IO backend (e.g. fintech / customer-service chatbot)
./run.sh --attacker gemini --target external-api \
         --target-config examples/socketio_target.json
```

### Attack modes

```bash
# Hacker mode — attacker adapts based on target's prior responses
./run.sh --attacker gemini --target openai --hacker-mode

# Probe a specific vulnerability category
./run.sh --probe-mode --probe-category injection --target openai

# Batch attacks with explicit instructions
./run.sh --mode batch \
         --instruction "Create a jailbreak prompt" \
         --instruction "Test system-prompt extraction"

# Multi-turn social-engineering flow
./run.sh --mode multi-turn --flow trust_then_pivot

# Tool-use attack (indirect prompt injection via tool results)
./run.sh --mode tool-use --tool-vector hidden_instruction

# Automated refinement: PAIR (5 iterations) or TAP (tree of thoughts)
./run.sh --strategy pair --strategy-iterations 5
./run.sh --strategy tap  --strategy-branching 3 --strategy-depth 4
```

### Standardized templates

```bash
# List built-in attack templates
./run.sh --list-templates

# Run a specific template
./run.sh --mode template --template basic_prompt_injection

# List multi-turn flows / tool-use vectors
./run.sh --list-flows
./run.sh --list-vectors
```

### Routing through Burp / mitmproxy / ZAP

Every model — hosted, local, and external — flows through the same TLS/proxy
layer. Burp captures help when you're targeting a custom backend whose protocol
you don't fully understand yet.

```bash
./run.sh --attacker gemini --target external-api \
         --target-config my_target.json \
         --proxy http://127.0.0.1:8080 \
         --ca-bundle ~/.burp/cacert.pem
```

CA bundle accepts both **PEM** (`.pem` / `.crt`) and **DER** (`.der` / `.cer`) —
no conversion needed. Per-target overrides (`proxy`, `ca_bundle`, `insecure`)
in the target-config JSON win over CLI flags.

## Attack categories (probe mode)

| Category | What it tests | OWASP LLM |
|---|---|---|
| `dan` | "Do Anything Now" jailbreak prompts | LLM01 |
| `injection` | Classic prompt injection | LLM01 |
| `xss` | Cross-site scripting payloads in model output | LLM05 |
| `glitch` | Unicode and token-boundary exploits | LLM01 |
| `misleading` | Misinformation / hallucination induction | LLM09 |
| `malware` | Malware-related content generation | LLM01 |
| `forbidden_knowledge` | Dangerous-knowledge extraction | LLM01 |
| `snowball` | Escalating-hallucination attacks | LLM09 |

## External-API targets in 30 seconds

The `external-api` target is the framework's escape hatch. Point it at a JSON
file describing the wire protocol and LMTWT handles the rest. Four protocols
are built in:

| `protocol` | Use when |
|---|---|
| `http` (default) | One-shot REST chat endpoint |
| `sse` | Server-Sent Events streaming response |
| `websocket` / `ws` / `wss` | Raw WebSocket JSON frames |
| `socketio` / `socket.io` | Socket.IO v5 (EIO v4) or v2 (EIO v3) — set `eio_version` |

A minimal Socket.IO config looks like:

```json
{
  "protocol": "socketio",
  "endpoint": "wss://chat.example.com/socket.io/",
  "eio_version": "3",
  "headers": { "Authorization": "Bearer ...", "User-Agent": "android" },
  "event_name": "send_message",
  "response_event": "receive_message",
  "payload_template": {
    "messageContent": [{ "content": "", "type": "TEXT" }],
    "messageId": "", "sessionId": "", "role": "USER"
  },
  "prompt_path": "messageContent.0.content",
  "message_id_key": "messageId",
  "session_id_key": "sessionId",
  "session_id": "session-from-your-bootstrap-api",
  "response_path": "messageContent.0.content"
}
```

See [`docs/configuration.md`](docs/configuration.md) for the full schema and
[`examples/`](examples/) for working configs (HTTP, Socket.IO, Ollama).

Debug helper: set `LMTWT_SOCKETIO_DEBUG=1` to dump every Socket.IO frame to
stderr while a run is in flight.

## Web UI

```bash
./run.sh --web                                    # localhost:8501
./run.sh --web --web-port 8080 --share            # public Gradio share link
```

The UI exposes model selection, interactive attack composition, result
visualization, and a session history with pass/fail tracking.

## Architecture in one paragraph

The async-first engine (`src/lmtwt/attacks/async_engine.py`) drives an
`attacker` and a `target`, both implementing a small `AsyncAIModel` interface
(`src/lmtwt/models/async_base.py`). Provider classes live in
`src/lmtwt/models/`; external transports in `src/lmtwt/models/external/`.
Each `chat()` returns a typed `ChatResponse`; streaming is exposed via
`astream()`. A judge (`src/lmtwt/judges/`) decides whether the target's
response counts as a successful jailbreak. Read [`docs/architecture.md`](docs/architecture.md)
for the full picture.

## Tests

```bash
uv run pytest                       # full suite (460+ tests, ~10s)
uv run pytest tests/test_external_socketio.py -v
uv run pytest --cov=src/lmtwt
```

[![Python Tests](https://github.com/tin537/LMTWT/actions/workflows/python-tests.yml/badge.svg)](https://github.com/tin537/LMTWT/actions/workflows/python-tests.yml)

## Documentation

- [`docs/index.md`](docs/index.md) — table of contents
- [`docs/architecture.md`](docs/architecture.md) — async engine, model layer, judges
- [`docs/configuration.md`](docs/configuration.md) — `.env`, `config.json`, target-config schema
- [`docs/cli.md`](docs/cli.md) — every flag, every mode
- [`docs/models.md`](docs/models.md) — provider matrix and capability notes
- [`docs/attacks.md`](docs/attacks.md) — templates, probes, flows, tool-use vectors
- [`docs/web.md`](docs/web.md) — Gradio UI
- [`docs/roadmap.md`](docs/roadmap.md) — what's next

## Contributing

```bash
git checkout -b feature/your-thing
uv sync                              # installs dev group automatically
uv run pytest                        # green before pushing
uv run ruff check src tests          # lint
git commit -m "feat: ..."
git push origin feature/your-thing
```

PRs welcome. See [`CONTRIBUTING.md`](CONTRIBUTING.md) and
[`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md). Found a security bug in LMTWT
itself? See [`SECURITY.md`](SECURITY.md).

## Acknowledgments

Inspired by [NVIDIA's garak](https://github.com/NVIDIA/garak) (Apache 2.0).
LMTWT is an original implementation under MIT — we appreciate garak's prior
work in the LLM red-team space.

## Support the project

<p align="center">
  <a href="https://www.paypal.me/tanuphattin">
    <img src="https://img.shields.io/badge/Donate-PayPal-blue.svg?style=for-the-badge" alt="PayPal">
  </a>
</p>

## Disclaimer

For **educational purposes** and **authorized security testing** only. Always
get written permission before testing any system you don't own. Researchers
running CTFs, internal red-team engagements, and personal lab work are the
intended audience. The authors disclaim responsibility for misuse.

## License

MIT — see [`LICENSE`](LICENSE).

## Contact

Tanuphat Tin — tanuphat.chai@gmail.com
[github.com/tin537/LMTWT](https://github.com/tin537/LMTWT)
