Metadata-Version: 2.4
Name: llm-pentest
Version: 0.1.0
Summary: Security testing toolkit for LLM-based systems
Project-URL: Homepage, https://github.com/jccazako/llm-pentest
Project-URL: Issues, https://github.com/jccazako/llm-pentest/issues
Author: Jean-Charles Cazako
License: MIT
License-File: LICENSE
Keywords: ai-safety,llm,owasp,pentest,prompt-injection,security
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Requires-Python: >=3.11
Requires-Dist: httpx>=0.28
Requires-Dist: jinja2>=3
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml>=6
Requires-Dist: rich>=13
Requires-Dist: typer>=0.15
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Requires-Dist: types-pyyaml>=6; extra == 'dev'
Description-Content-Type: text/markdown

# LLM Pentest

[![CI](https://github.com/jccazako/llm-pentest/actions/workflows/ci.yml/badge.svg)](https://github.com/jccazako/llm-pentest/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/llm-pentest)](https://pypi.org/project/llm-pentest/)
[![Python](https://img.shields.io/pypi/pyversions/llm-pentest)](https://pypi.org/project/llm-pentest/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![OWASP](https://img.shields.io/badge/OWASP-LLM%20Top%2010-blue)](https://owasp.org/www-project-top-10-for-large-language-model-applications/)

Security testing toolkit for LLM-based systems — chatbots, RAG pipelines, and AI agents.

Tests for prompt injection, data exfiltration, tool abuse, and jailbreaking.

## Install

```bash
pip install -e .
```

## Quick Start

```bash
# Interactive setup
llm-pentest init

# Or use a built-in profile
llm-pentest scan --profile openai

# Or provide a config file
cp examples/target.yaml my-target.yaml
llm-pentest scan -c my-target.yaml
```

## Usage

```bash
# List payloads and chains
llm-pentest payloads

# Scan specific category
llm-pentest scan -c target.yaml --category prompt_injection

# Scan single payload
llm-pentest scan -c target.yaml --payload pi-001

# Parallel scanning (5 concurrent requests)
llm-pentest scan -c target.yaml -n 5

# Only fail CI on high/critical findings
llm-pentest scan -c target.yaml --fail-on high

# Use a profile instead of config file
llm-pentest scan --profile openai --model gpt-4o

# HTML report
llm-pentest scan -c target.yaml --format html -o report.html

# SARIF report (GitHub Code Scanning compatible)
llm-pentest scan -c target.yaml --format sarif -o report.sarif

# Baseline comparison
llm-pentest scan -c target.yaml -o baseline.json
# ... fix vulnerabilities ...
llm-pentest scan -c target.yaml -o current.json --baseline baseline.json

# Standalone diff
llm-pentest diff -b baseline.json --current current.json

# Custom detection hooks
llm-pentest scan -c target.yaml --hooks my_hooks.py

# LLM-as-judge for better detection accuracy
llm-pentest scan -c target.yaml --judge-endpoint https://api.openai.com/v1/chat/completions --judge-key $OPENAI_API_KEY

# Dry run (no requests sent)
llm-pentest scan -c target.yaml --dry-run
```

## Target Profiles

Built-in profiles for common providers (`llm-pentest profiles`):

| Profile | Endpoint | Default Model |
|---|---|---|
| `openai` | api.openai.com | gpt-4o-mini |
| `anthropic` | api.anthropic.com | claude-sonnet-4-20250514 |
| `ollama` | localhost:11434 | llama3 |

## Attack Categories

| Category | Payloads | OWASP | Description |
|---|---|---|---|
| `prompt_injection` | pi-001 to pi-009 | LLM01 | System prompt extraction, instruction override, encoding bypass, indirect injection |
| `data_exfiltration` | de-001 to de-005 | LLM06 | RAG context leak, credential extraction, PII harvesting |
| `tool_abuse` | ta-001 to ta-004 | LLM07, LLM08 | Unauthorized tool use, parameter injection, CoT hijacking, DoS |
| `jailbreak` | jb-001 to jb-004 | LLM01 | Role-play, hypothetical framing, token smuggling, persona splitting |

## Attack Chains

Chains compose multiple payloads into multi-step attack sequences:

```yaml
- id: chain-001
  name: "Jailbreak then Extract Credentials"
  steps:
    - payload_id: jb-001
      prompt_index: 0
    - payload_id: de-002
      prompt_index: 0
```

## Custom Payloads

Drop YAML files into `payloads/` or pass `--payloads-dir`:

```yaml
- id: custom-001
  name: "My Custom Attack"
  category: prompt_injection
  severity: high
  owasp: ["LLM01"]
  prompts:
    - "Your attack prompt here"
  detection:
    - pattern: "expected response pattern"
      confidence: 0.8
```

## Custom Detection Hooks

Write Python functions for domain-specific detection:

```python
# my_hooks.py — any function named detect_* is auto-registered
def detect_internal_data(payload, prompt, response):
    if "internal-project-name" in response.lower():
        return "Leaked internal project name"
    return None
```

```bash
llm-pentest scan -c target.yaml --hooks my_hooks.py
```

## CI/CD Integration

```bash
# Fail only on high/critical findings
llm-pentest scan -c target.yaml --fail-on high || echo "Vulnerabilities found!"

# SARIF for GitHub Code Scanning
llm-pentest scan -c target.yaml --format sarif -o results.sarif

# Baseline regression check
llm-pentest scan -c target.yaml --baseline previous.json --fail-on high
```

## License

MIT
