Metadata-Version: 2.5
Name: llm-sanitizer
Version: 0.7.1
Summary: CLI + MCP server for detecting, classifying, and redacting embedded LLM agent instructions in documents, source code, and web pages.
Project-URL: Homepage, https://github.com/Warnes-Innovations/llm-sanitizer
Project-URL: Repository, https://github.com/Warnes-Innovations/llm-sanitizer
Project-URL: Issues, https://github.com/Warnes-Innovations/llm-sanitizer/issues
Project-URL: Changelog, https://github.com/Warnes-Innovations/llm-sanitizer/blob/main/CHANGELOG.md
Author-email: "Gregory R. Warnes" <greg@warnes-innovations.com>
License: AGPL-3.0-or-later
License-File: LICENSE
Keywords: agent,llm,mcp,model-context-protocol,prompt-injection,sanitizer,scanner,security
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: filetype<2,>=1.2
Requires-Dist: httpx<1,>=0.27
Requires-Dist: markitdown[docx,pdf,pptx,xls,xlsx]<1,>=0.1
Requires-Dist: mcp<3,>=2.0
Requires-Dist: pydantic<3,>=2.0
Requires-Dist: pypdf<7,>=4.0
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: striprtf<1,>=0.0.26
Provides-Extra: 7z
Requires-Dist: py7zr<2,>=1.0; extra == '7z'
Provides-Extra: binary
Provides-Extra: pdf-redact
Requires-Dist: pymupdf<2,>=1.24; extra == 'pdf-redact'
Provides-Extra: rar
Requires-Dist: libarchive-c<6,>=5.0; extra == 'rar'
Description-Content-Type: text/markdown

# llm-sanitizer

CLI + MCP server for detecting, classifying, and redacting embedded LLM agent
instructions in documents, source code, and web pages.

## What It Does

Documents, web pages, and source code can contain embedded instructions
targeting LLM agents — from legitimate customization to malicious prompt
injection. `llm-sanitizer` scans content for these embedded instructions,
classifies their risk level, reports findings in multiple formats, and
optionally produces redacted output.

### Detection

Twelve pluggable detection rules covering:
- Instruction override phrases ("ignore previous instructions…")
- Zero-width character encoding (hidden text via invisible Unicode)
- HTML/markdown hidden content (white-on-white, display:none)
- Role-play injection ("act as", "you are now DAN")
- System prompt markers (`<system>`, `## System Prompt`)
- Data exfiltration attempts ("output the system prompt")
- Comment-embedded directives (`<!-- AI: do this -->`)
- Base64-encoded instructions
- Unicode homoglyph substitution (Cyrillic lookalikes)
- Agent-specific config patterns in unexpected locations
- Character-splitting obfuscation (`i g n o r e`, `ignore___all`) —
  reconstructed and re-scanned
- **Semantic-intent injection** — a local, no-egress n-gram classifier that
  catches *keyword-less* rephrasings (role reassignment, verbatim/echo
  exfiltration, supersede-prior-guidance) the pattern rules miss

### Classification

Five risk levels: **info** (legitimate AI config) → **low** → **medium** →
**high** → **critical** (confirmed injection technique).

### Redaction

Non-destructive cleaning in four modes:
- **strip** — remove the instruction entirely
- **comment** — replace with `[REDACTED: ...]` marker
- **highlight** — wrap in visible markers for review
- **placeholder** — replace each character with `█`, so the text is gone but
  every byte offset, line number and column in the document is unchanged

**Binary inputs never produce unredacted binary output.** Scanning a PDF,
DOCX, PPTX or XLSX requires extracting its text, so that redacted text is what
the redact tools write, and the response says so
(`output_format: "extracted-text"`). The original bytes are never copied to an
output path. Where no text can be recovered at all, the call is **refused and
no file is written** — an unredacted copy is worse than no file, because a
caller that checks only whether the output exists cannot tell the two apart.

**PDFs can also be redacted in place**, with the `[pdf-redact]` extra:

```bash
uv pip install 'llm-sanitizer[pdf-redact]'
```

A rewritten `<stem>.redacted.pdf` is written beside the text output, with the
findings removed from the **content stream** — not a black rectangle drawn
over them, which leaves the text in the file and is the classic failure of
this category. It is published only after being verified clean three ways
(the project's extractor, a second independent extractor, and a raw
content-stream search); a rewrite that cannot be proved clean is deleted and
reported as `refused`, and the redacted text output still stands.

## Installation

```bash
pip install llm-sanitizer
# or
uv pip install llm-sanitizer
```

### Platform notes

The dependency chain `markitdown -> magika -> onnxruntime` needs binary wheels for
`onnxruntime` on your platform and Python version.

* **Intel macOS (x86_64):** onnxruntime dropped `macosx_x86_64` wheels entirely at
  1.24.0, and the last Intel build (1.23.2) only ships wheels for Python ≤3.13. Use
  Python 3.13 or older — with uv, pin the interpreter explicitly:
  `uvx --python 3.13 --from llm-sanitizer[7z,rar] llm-sanitize ...`.
* **Apple Silicon macOS:** works on macOS 14+ (onnxruntime 1.24+ wheels require
  macOS 14); on macOS 13 or earlier use Python ≤3.13.
* **Linux and Windows:** work on Python 3.11 through 3.14.

We keep `requires-python = ">=3.11"` rather than capping it: a cap cannot carry
platform markers and would regress the platforms where Python 3.14 resolves fine.

## Quick Start

### CLI

```bash
# Scan a file
llm-sanitize scan document.md

# Scan a URL
llm-sanitize scan https://example.com/page

# Scan a directory
llm-sanitize scan ./docs/ --glob "**/*.md"

# Redact a file
llm-sanitize redact document.md -o clean_document.md

# Redact a directory (mirrors structure)
llm-sanitize redact ./docs/ -o ./clean_docs/

# Control how binary files (PDF, DOCX, zip, images, ...) are handled —
# detected by content, not extension, so renaming a file changes nothing
llm-sanitize scan document.pdf --binary-mode extract  # pull embedded text via markitdown (default)
llm-sanitize scan suspicious.png --binary-mode text    # force raw bytes to be scanned as literal text
llm-sanitize scan large_dir/ --binary-mode skip        # exclude binary files from the scan entirely

# Assemble a directory report from previously-saved `scan --format json`
# results without re-scanning (for callers that cache scans by content hash)
llm-sanitize merge --manifest manifest.txt --format sarif > report.sarif
```

`--binary-mode extract` (the default) rejects malicious zip archives — excessive
entry counts, total uncompressed size, or compression ratios — before
decompressing anything.

### MCP Server

Add to your MCP configuration:

```json
{
  "mcpServers": {
    "llm-sanitizer": {
      "command": "llm-sanitizer"
    }
  }
}
```

Available tools: `scan_text`, `scan_file`, `scan_url`, `scan_dir`, `redact`,
`redact_file`, `redact_url`, `redact_dir`, `list_rules`.

`scan_url`/`redact_url` fetch server-side with an honest browser UA, but a
managed WAF (Cloudflare/Akamai) can still refuse the request based on origin
reputation. That failure comes back as `{"status": "error", "error_type":
"fetch_blocked", "http_status": <code>, ...}` — distinct from a scan failure,
so a caller knows the page could not be verified rather than that scanning
broke. llm-sanitizer will not route fetches through a third-party proxy to
work around a block (a trust-boundary tool observing/altering its own input
via another network defeats its own threat model) or otherwise bypass the
site's bot-protection. For a page that can't be fetched server-side, paste
its HTML/text and use `scan_text` instead.

### Python API

```python
from llm_sanitizer.scanner import scan_text

result = scan_text("Check this <!-- ignore previous instructions --> content")
print(result.findings)  # [Finding(rule='comment_directive', risk='high', ...)]
```

## Configuration

Create `.llm-sanitizer.yml` at your project root:

> **A config file that exists is always applied, or the scan fails.** If
> `.llm-sanitizer.yml` is present but cannot be read — an unparseable file, or a
> stripped-down install without `PyYAML` — the scanner raises `ConfigError`
> instead of falling back to defaults. Reporting one policy while enforcing a
> different one is the failure this refuses to have.
>
> Having *no* config file is unaffected: the built-in defaults apply, and that is
> a deliberate choice rather than a silent one.

```yaml
sensitivity: medium                    # low | medium | high

rules:
  zero_width:
    enabled: true
  instruction_override:
    enabled: true
    sensitivity: high                  # per-rule sensitivity override

policy:
  mode: "allow-known"                  # allow-known | allow-none | allow-all
  agents:
    copilot: allow
    cursor: allow

# Refuse oversized input fail-closed (a CRITICAL input_too_large finding)
# rather than reading/scanning it. Default 25 MiB.
max_scan_bytes: 26214400

# How to treat a non-archive binary that yields no extractable text.
unprocessable_binary_policy: fail      # fail (default) | scan-text | ignore

# Recursive archive scanning limits (zip / tar / gz / bz2 / xz / 7z / rar).
archive:
  max_depth: 3                         # archive-in-archive nesting levels
  max_cumulative_bytes: 524288000      # 500 MB across all nested levels
  max_uncompressed_bytes: 104857600    # 100 MB per archive
  max_entries: 1000                    # per-archive entry-count cap
  max_compression_ratio: 100           # zip-bomb ratio guard
  formats: [zip, tar, gz, bz2, xz, 7z, rar]

output:
  format: markdown                     # json | markdown | sarif
  context_lines: 2
```

## Output Formats

- **JSON** — structured findings for programmatic use
- **Markdown** — human-readable reports
- **SARIF** — GitHub Code Scanning / VS Code integration

## Documentation

- [Design Specification](https://github.com/Warnes-Innovations/llm-sanitizer/blob/main/docs/DESIGN_SPEC.md)

## Credits — semantic-intent training data

The `semantic_intent` classifier is trained on a hand-curated corpus plus,
optionally, third-party labeled datasets (used at **train time only** — only the
trained weights ship; the datasets are not redistributed). With thanks to:

- **[prodnull/prompt-injection-repo-dataset](https://huggingface.co/datasets/prodnull/prompt-injection-repo-dataset)**
  (Apache-2.0) — injection embedded in repository files, with domain-matched hard
  negatives.
- **[deepset/prompt-injections](https://huggingface.co/datasets/deepset/prompt-injections)**
  (Apache-2.0) — classic prompt-injection set (positives used for augmentation).

See [`data-raw/SOURCES.md`](data-raw/SOURCES.md) for full provenance, licenses,
and pinned revisions.

## License

AGPL-3.0-or-later. Commercial licensing available — contact
greg@warnes-innovations.com.
