Metadata-Version: 2.4
Name: llm-scrub
Version: 0.1.0
Summary: Zero-dependency scrubbers for LLM chat output — strip leaked JSON state-op fences, backend scaffold blocks, and tool-preamble meta-narration from player-facing text
Project-URL: Homepage, https://github.com/felixchaos/llm-scrub
Project-URL: Issues, https://github.com/felixchaos/llm-scrub/issues
Author: Stellatrix Labs
License-Expression: MIT
License-File: LICENSE
Keywords: chat,llm,output-cleaning,postprocess,roleplay,sanitize,scrub
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Filters
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# llm-scrub

Zero-dependency scrubbers for LLM chat output. Weak models leak backend
scaffolding into the text shown to the user — JSON state-op fences, injected
`=== … ===` context blocks, and English "tool preamble" meta-narration.
`llm-scrub` strips that noise **deterministically**, using pure string/regex
logic instead of relying on the model to obey a prompt.

- **No dependencies.** Pure standard library (`json`, `re`).
- **Deterministic.** No model call, no network — a plain `str -> str` transform.
- **Conservative.** Tuned to under-strip rather than over-strip: legitimate
  fenced code blocks, JSON data, markdown tables, and `===` dividers survive.
- **Idempotent.** `scrub(scrub(x)) == scrub(x)`.

## Install

```bash
pip install llm-scrub
```

Requires Python 3.10+.

## Quickstart

```python
from llm_scrub import scrub

raw = (
    "好，开始输出。\n\n"
    "=== 时间线检索锚点 ===\n"
    "当前时间：序章 · 站台\n\n"
    "林夕睁开眼睛，废墟在昏黄的天光下铺展开来。\n\n"
    '```json\n[{"op": "set", "path": "world.location", "value": "废墟"}]\n```\n'
    "Let me mark the anchors now."
)

print(scrub(raw))
# 林夕睁开眼睛，废墟在昏黄的天光下铺展开来。
```

`scrub` chains the three scrubbers in the canonical order. Call them
individually when you only need one:

```python
from llm_scrub import (
    strip_json_state_ops,
    strip_meta_tool_preamble,
    strip_leaked_scaffold,
)

strip_json_state_ops('战斗结束。\n```json\n{"op": "append", "path": "bag", "value": "刀"}\n```')
# '战斗结束。'

strip_meta_tool_preamble("他转身离去。\nLet me update the anchors now.")
# '他转身离去。'

strip_leaked_scaffold("她走进废墟。\n\n=== 相关原文片段 ===\n某段原文\n\n她捡起刀。")
# '她走进废墟。\n\n她捡起刀。'
```

## API

| Function | Signature | Removes |
| --- | --- | --- |
| `scrub` | `scrub(text: str, *, extra_headers=None) -> str` | All three, chained in the canonical order. |
| `strip_json_state_ops` | `strip_json_state_ops(text: str) -> str` | JSON state-op fences — fenced (```` ```json […] ```` ), un-fenced bare `[{"op":…}]`, and truncated/unclosed ops left by a cut-off response. |
| `strip_meta_tool_preamble` | `strip_meta_tool_preamble(text: str) -> str` | A leaked English "tool preamble" sentence at the tail (e.g. *"Let me mark the anchors now."*). |
| `strip_leaked_scaffold` | `strip_leaked_scaffold(text: str, *, extra_headers=None) -> str` | Whole `=== … ===` backend scaffold blocks (retrieval anchors, world-line summaries, etc.), plus a trailing "begin output" reasoning preamble. |

`LEAKED_SCAFFOLD_KEYS` is the built-in tuple of scaffold header markers; pass
your own via `extra_headers` to extend it.

## What it strips

**1. JSON state-op fences.** Some pipelines let the model emit state changes as
JSON. Those blocks must never reach the reader:

````text
你推开门，走了进去。

```json
[{"op": "set", "path": "player.location", "value": "地窖"}]
```
````

→ `你推开门，走了进去。` — including malformed and truncated (streamed-and-cut)
variants. A fenced JSON block that is **not** ops (a plain object or array of
data) is preserved, so legitimate code examples survive.

**2. Leaked tool preamble.** Before a native tool call, a model may blurt one
line of meta-narration meant for itself:

> …晨光无声流转。**The scene has progressed naturally. Let me mark the anchors that have been satisfied through our gameplay.**

Only a trailing, unquoted, predominantly-English segment matching a preamble
phrase is removed. A character's English dialogue inside quotes is never
touched.

**3. Leaked scaffold blocks.** Weak models sometimes dump the backend's
invisible context straight into the prose:

```text
=== 时间线检索锚点 ===
当前时间：序章 · 站台
待确认跳跃：无
```

These headers are fixed strings the backend injects; normal narrative never
emits them verbatim, so the whole block is removed with zero collateral damage.
Extend the header list for your own backend:

```python
scrub(text, extra_headers=("我的自定义脚手架",))
```

## Design: under-strip, never over-strip

Every scrubber is tuned to leave the text alone when in doubt:

- Fenced/bare JSON is only stripped when it parses **and** carries state-op
  markers (`op` / `path` / `question`). Other JSON is preserved.
- A trailing English segment is only stripped when it is outside quotes **and**
  matches a known tool-preamble phrase.
- A `=== … ===` line is only stripped when its inner text matches a known
  scaffold header — a `=== 序章 ===` divider or a markdown setext underline is
  left intact.
- If stripping would empty the text (a whole turn of pure leakage), the
  original is returned rather than blanking the message.

## License

MIT — see [LICENSE](LICENSE).

## Acknowledgements

Extracted from the RPG Roleplay platform (https://github.com/felixchaos/rpg-roleplay-platform).
