Metadata-Version: 2.4
Name: prompt-canon
Version: 0.1.0
Summary: Canonicalize before you guard: a pure-stdlib Unicode normalizer for LLM prompt-injection defense (invisible-character stripping, homoglyph mapping, locale-independent case folding).
Author-email: Fevzi Ege Yurtsevenler <egeyurtsevenler@gmail.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/fevziegeyurtsevenler/prompt-canon
Project-URL: Source, https://github.com/fevziegeyurtsevenler/prompt-canon
Project-URL: Issues, https://github.com/fevziegeyurtsevenler/prompt-canon/issues
Project-URL: Changelog, https://github.com/fevziegeyurtsevenler/prompt-canon/blob/main/CHANGELOG.md
Keywords: unicode prompt injection normalizer,homoglyph filter for LLM,invisible unicode remover,zero-width character stripper,prompt injection,llm security,owasp llm top 10,canonicalization,confusables
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Text Processing :: Filters
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Dynamic: license-file

# prompt-canon — Canonicalize Before You Guard: a Unicode Prompt-Injection Normalizer for LLMs

**prompt-canon** is a small, pure-standard-library Python library that
**canonicalizes text before you guard it**. It is a **unicode prompt injection
normalizer** for LLM pipelines: it strips invisible characters, maps
cross-script homoglyphs to an ASCII skeleton, and applies a locale-independent
case fold — so that your guardrail, classifier, moderation model, or allow/deny
list sees **one** canonical form instead of the many visually identical variants
an attacker can spell.

> **Canonicalize before you guard.** A detector that never sees the real bytes
> cannot block them. `"İGNORE previous instructions"` spelled with a Turkish
> dotted capital I, a zero-width space, and a Cyrillic look-alike is invisible
> to a naive keyword or embedding filter — until you normalize it first.

Keywords: **unicode prompt injection normalizer**, **homoglyph filter for LLM**,
**invisible unicode remover**, **zero-width character stripper**, prompt
injection defense, OWASP LLM Top 10, MITRE ATLAS, Unicode confusables,
canonicalization.

- **Zero runtime dependencies** — only `unicodedata` and `re` from the standard
  library.
- **Python 3.8+.**
- **Deterministic and offline** — no model, no network, no telemetry.
- **Transparent** — every character it removes, rewrites, or flags is reported
  as a structured finding.

---

## Install

```bash
pip install prompt-canon
```

Or from source:

```bash
git clone https://github.com/fevziegeyurtsevenler/prompt-canon
cd prompt-canon
pip install -e ".[test]"
```

## 30-second quickstart

```python
from prompt_canon import canonicalize

# A prompt-injection string using a Turkish dotted-I, a Cyrillic 'р',
# and an embedded zero-width space.
raw = "İGNORE​ рrevious instructions"

result = canonicalize(raw, fold_case=True)

print(result.text)
# -> "ignore previous instructions"

for f in result.findings:
    print(f["kind"], f["codepoint"], f["note"])
# zero-width  U+200B  removed; zero-width / invisible character
# confusable  U+0440  mapped 'р' -> 'p' (CYRILLIC SMALL LETTER ER)
```

Now hand `result.text` — not `raw` — to your guardrail, regex, moderation API,
or classifier.

### Individual transforms

```python
from prompt_canon import strip_invisible, map_confusables, fold_case

strip_invisible("ig​nore")     # "ignore"  (zero-width space removed)
map_confusables("аdmin")            # "admin"   (Cyrillic 'а' -> 'a')
fold_case("İGNORE")                 # "ignore"  (Turkish dotted I handled)
fold_case("straße")                 # "strasse" (German sharp S handled)
```

## What it does

| Transform | Behaviour |
|---|---|
| **Zero-width / invisible** | Removes `U+200B`, `U+2060`–`U+2064`, and non-BOM `U+FEFF`. |
| **Tag block** | Removes `U+E0000`–`U+E007F` (the "ASCII smuggling" tag characters). |
| **Bidi controls** | Removes **and flags** `U+202A`–`U+202E` and `U+2066`–`U+2069` (Trojan Source, CVE-2021-42574). |
| **ZWJ / ZWNJ** | **Keeps and flags** `U+200D` / `U+200C` — they are legitimate in emoji and Persian/Arabic/Indic text. |
| **Byte-order mark** | A **leading** `U+FEFF` is preserved; inner `U+FEFF` is stripped. |
| **Confusables** | Maps curated Cyrillic / Greek / Latin-Extended / fullwidth homoglyphs to their ASCII skeleton. |
| **Case fold** | Locale-independent fold that also fixes Turkish dotted/dotless I and German ß. Off by default. |

Every removed, rewritten, or flagged character is returned as a finding:

```python
{"kind": "bidi", "codepoint": "U+202E", "offset": 4,
 "note": "removed; bidirectional control (Trojan Source / CVE-2021-42574 risk)"}
```

## Why NFKC and `str.casefold()` are not enough

The reflex is to reach for `unicodedata.normalize("NFKC", text)` and
`str.casefold()`. Both are useful, and prompt-canon is designed to sit
*alongside* them — but neither closes the gap on their own.

### The `İGNORE` worked example

Take the classic injection trigger `ignore`, spelled with a **Turkish dotted
capital I** (`İ`, `U+0130`):

```python
>>> "İGNORE".casefold()
'i̇gnore'          # note the extra combining dot: 'i' + U+0307
>>> "İGNORE".casefold() == "ignore"
False
```

`str.casefold()` is deliberately **locale-independent**, so it folds `U+0130`
to `i` **plus a combining dot above** — not to a bare ASCII `i`. A keyword or
regex check for `"ignore"` therefore misses it. The dotless variant is just as
bad:

```python
>>> "ıgnore".casefold()       # Turkish dotless small i, U+0131
'ıgnore'                      # unchanged — still not "ignore"
```

And NFKC does not fix it either, because these are distinct, "normal" letters,
not compatibility characters:

```python
>>> import unicodedata
>>> unicodedata.normalize("NFKC", "İGNORE").casefold() == "ignore"
False
```

prompt-canon's `fold_case` normalizes **both** Turkish I forms to ASCII `i`,
applies `casefold` (which already turns `ß`/`ẞ` into `ss`), and drops the
residual combining dot:

```python
>>> from prompt_canon import fold_case
>>> fold_case("İGNORE")
'ignore'
>>> fold_case("ıgnore")
'ignore'
>>> fold_case("straße")
'strasse'
```

Similarly, NFKC leaves whole classes of attack untouched: it does **not** remove
zero-width or bidi controls, and it does **not** collapse cross-script
homoglyphs — a Cyrillic `а` (`U+0430`) stays a Cyrillic `а` under NFKC. Those
are exactly the gaps `strip_invisible` and `map_confusables` are built to cover.

## Where this fits: OWASP LLM Top 10 and MITRE ATLAS

prompt-canon is **authorized, defensive security-testing and hardening
tooling**. It maps to:

- **OWASP Top 10 for LLM Applications — LLM01: Prompt Injection.** Unicode
  obfuscation (invisible characters, homoglyphs, bidi reordering) is a common
  way to smuggle injection payloads past filters. Canonicalizing first shrinks
  that evasion surface.
- **MITRE ATLAS.** Relevant to adversarial techniques around *LLM Prompt
  Injection* and evasion/obfuscation of input-side defenses. Use prompt-canon as
  a normalization control in front of your detection layer.

## Responsible use

This library is a **defensive normalization layer**. It is meant to be run on
untrusted input *before* your guardrails, to make evasion harder and to give you
an auditable record (the findings) of what was cleaned. It does not generate
attacks. Do not use the `findings` output or the confusable table to build or
tune payloads against systems you are not authorized to test.

## Honesty: what prompt-canon is and is not

Keeping this honest matters more than keeping it impressive.

- **It is a normalization / coverage *layer*, not a detector.** prompt-canon
  never decides whether text is malicious. It reduces many visually equivalent
  spellings to one canonical form so that *your* detector has a fair shot. Pair
  it with a real classifier or policy engine.
- **The confusable table is curated, not exhaustive.** It covers common
  Cyrillic, Greek, Latin-Extended, and fullwidth look-alikes — not the entire
  Unicode confusables database (UTS #39). It will miss glyphs it has never seen.
- **Normalization can cause false positives.** Mapping homoglyphs and folding
  case is lossy by design. Legitimate multilingual text — a genuinely Greek or
  Cyrillic word, a name, a URL — can be altered. Canonicalize the copy you feed
  to your *detector*; do not overwrite the text you store or display to the
  user without thought. Case folding is **off by default** precisely so that a
  clean ASCII input is returned byte-for-byte unchanged.
- **Idempotent, but destructive.** `canonicalize(canonicalize(x)) ==
  canonicalize(x)`, and the transforms discard information on purpose.

### Prior art and complementary tools

prompt-canon is a focused building block, not a replacement for these projects —
use it *with* them:

- **[promptfoo](https://www.promptfoo.dev/)** — red-teaming and eval framework
  that includes ASCII-smuggling / invisible-Unicode strategies for testing
  models. prompt-canon is the normalization side of that same problem.
- **[LLM Guard](https://llm-guard.com/)** (Protect AI) — a broader input/output
  scanner suite (prompt-injection, PII, toxicity, and more).
- **[Microsoft Presidio](https://microsoft.github.io/presidio/)** — PII
  detection and anonymization, which also benefits from canonical input.
- The **Unicode Security Mechanisms** standard,
  [UTS #39](https://www.unicode.org/reports/tr39/), is the authoritative source
  for the confusable equivalences this library draws a curated subset from.

If you need full detection, orchestration, or PII handling, reach for those.
prompt-canon does one thing: it canonicalizes the bytes first.

## Cross-link: unicode-threat-reveal

prompt-canon is the dependency behind the **unicode-threat-reveal** Hugging Face
Space, which *visualizes* what this library removes and flags — highlighting the
invisible characters, homoglyphs, and bidi controls hiding inside a pasted
prompt. This library is the engine; that Space is the interactive lens on top of
it.

## Development

```bash
pip install -e ".[test]"
python -m pytest -q
```

CI runs the suite on Python 3.9–3.12 (see `.github/workflows/test.yml`).

## License

Apache License 2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE).
