Metadata-Version: 2.4
Name: langchain-scrubkit
Version: 0.1.0
Summary: LangChain middleware that cleans tool outputs: fixes broken encoding, HTML and invisible characters before your agent ever reads them.
Author-email: AI Data Tools <aidatatools@proton.me>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/aidatatools-dev/langchain-scrubkit
Project-URL: Source, https://github.com/aidatatools-dev/langchain-scrubkit
Project-URL: Issues, https://github.com/aidatatools-dev/langchain-scrubkit/issues
Project-URL: Changelog, https://github.com/aidatatools-dev/langchain-scrubkit/blob/main/CHANGELOG.md
Project-URL: Cleaning engine, https://pypi.org/project/scrubkit/
Keywords: langchain,langchain-middleware,agent,agents,tools,data-cleaning,clean-tool-output,encoding,mojibake,unicode,html,web-scraping,scraping,llm,deterministic
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: General
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: langchain>=1.0
Requires-Dist: scrubkit>=0.1.0
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Dynamic: license-file

# langchain-scrubkit

**Clean tool outputs in LangChain, before your agent ever reads them.** Fixes
broken encoding, leftover HTML and invisible characters — and tells the agent
when a tool came back with a bot wall instead of content.

[![PyPI](https://img.shields.io/pypi/v/langchain-scrubkit.svg)](https://pypi.org/project/langchain-scrubkit/)
[![Python](https://img.shields.io/pypi/pyversions/langchain-scrubkit.svg)](https://pypi.org/project/langchain-scrubkit/)
[![License](https://img.shields.io/pypi/l/langchain-scrubkit.svg)](LICENSE)

Your scraping tool returns `CafÃ© <b>Noir</b>&nbsp;`. Your model reads it, and
reasons about the damage as though it were content. There is no later stage
that undoes that.

```bash
pip install langchain-scrubkit
```

```python
from langchain.agents import create_agent
from langchain_scrubkit import ScrubkitMiddleware

agent = create_agent(model, tools=[scrape], middleware=[ScrubkitMiddleware()])
```

That is the whole setup. Every tool result is now repaired at the moment it is
produced — the conversation never contains the broken version.

---

## What your agent stops seeing

| Tool returns | Model reads |
|---|---|
| `CafÃ©` | `Café` |
| `<p>Warm light.</p><p>Fits any room.</p>` | `Warm light. Fits any room.` |
| `Solid oak &mdash; seats&nbsp;two.` | `Solid oak — seats two.` |
| `Tokyo​﻿ Ltd­` | `Tokyo Ltd` |

Deterministic, no LLM, no network call. The same input always produces the same
output, so a cached run stays a cached run.

## The part nobody else does

A scraper that hits a bot wall returns HTTP 200 and a page saying
`Access denied`. Every agent framework hands that to the model as a normal tool
result. The model has no way to know the extraction failed, so it treats the
error page as data and answers confidently from nothing.

**Without the middleware**, this is what reaches the model:

```
Please wait, verifying you are human
```

**With it**, the same result carries its diagnosis:

```
Please wait, verifying you are human

[scrubkit] This tool result looks like what a bot wall or error page returns
(matched 'please wait'), not the page's content. The extraction probably
failed: re-run the tool or try another source rather than treating the text
above as data. If the text is genuinely about that phrase, ignore this note.
```

The agent can now retry, switch source, or tell the user it failed — instead of
inventing an answer. The original text is kept, never replaced.

Three modes:

```python
ScrubkitMiddleware(on_flags="note")    # default: annotate, let the agent decide
ScrubkitMiddleware(on_flags="ignore")  # repair silently
ScrubkitMiddleware(on_flags="raise")   # stop the run (ExtractionFailedError)
```

## What this saves you writing

Without it, cleaning tool output means writing the middleware yourself:

```python
@wrap_tool_call
def clean_tool_output(request, handler):
    result = handler(request)
    result.content = ftfy.fix_text(result.content)   # and then?
    return result
```

That one line is the easy part. What is left:

- **HTML** — ftfy does not strip tags. `<p>A</p><p>B</p>` must not become `AB`.
- **`content` is not always a string.** It is typed `str | list[str | dict]`;
  the list form carries content blocks, and a text repairer must not touch an
  image payload.
- **JSON tool results.** Cleaning a whole JSON document as one opaque string
  works badly — and parsing it means you now own re-serialisation, and the
  decision not to reformat a document that was already fine.
- **CSV tool results** must not come back as JSON.
- **Knowing what not to touch.** `"None"` is a surname, `"NA"` is Namibia,
  `‌` is a required letter-shaping character in Persian, and the joiner inside
  `👨‍👩‍👧` holds a family emoji together.
- **Never breaking the agent.** A repair that throws must not end someone's run.
- **The bot-wall case above**, which no text repairer addresses at all.

This package is those decisions, tested. If your only problem is broken Unicode,
[ftfy](https://github.com/rspeer/python-ftfy) is excellent and you may not need
anything else.

## What it does *not* do

By default it runs only scrubkit's **AUTO tier**: seven rules that provably
cannot change a value's meaning, its type, or the shape of your data. It cannot
drop a row, retype a field, or rename a key unless you ask by name.

It does **not** touch user input or model output. Those did not come from a
scraper; repairing them would be overreach.

And it never destroys a legitimate value:

| Input | Kept as-is because |
|---|---|
| `"None"` | it is a common surname |
| `"NA"` | it is Namibia's ISO country code |
| `می‌تواند` | the zero-width non-joiner is required Persian orthography |
| `👨‍👩‍👧` | removing the joiner splits one family into three people |
| `AT&T`, `5 < 7` | a real HTML entity ends in `;` — these are not markup |
| `3 m²`, `½ cm` | Unicode NFKC would rewrite these to `3 m2` and `1/2` |

**A known limitation, stated rather than hidden.** Bot-wall detection matches
phrases, with no notion of context. A product genuinely called
`CAPTCHA Solver Pro`, or an article titled `A guide to CAPTCHA design`, trips
the same rule as a real captcha page. That is why the note quotes what triggered
it and hedges rather than asserts: the model sees both the text and the reason,
and can disagree with us. Nothing is deleted either way.

## Usage

**Clean only the tools that reach the open web.** A calculator or a SQL tool has
nothing to repair:

```python
ScrubkitMiddleware(tools=["scrape_products", "fetch_page"])
```

**Opt into the riskier repairs**, each off by default because each changes
something you may depend on:

```python
ScrubkitMiddleware(
    placeholder_policy="null_high_confidence",  # "N/A" -> None; "None" stays
    drop_exact_duplicates=True,                 # changes your row count
    coerce_numeric_text=True,                   # changes a value's type
    repair_keys=True,                           # changes your schema
)
```

**For chains rather than agents**, the same cleaning as a Runnable:

```python
from langchain_scrubkit import scrub

chain = scraper | scrub() | prompt | model
```

`scrub()` preserves shape — a string stays a string, a row stays a row, a list
keeps every element. It defaults to `on_flags="ignore"`, since a chain has no
conversation to annotate; `on_flags="raise"` is the useful one there, to stop
rather than write a captcha page into your vector store.

## Verify it yourself

```bash
pip install "langchain-scrubkit[test]"
python -m pytest --pyargs langchain_scrubkit
```

101 tests ship inside the package. They cover shape safety, the control set of
legitimate data above, both `content` forms, sync and async, and — through a
real `create_agent` — that a broken tool result reaches the model repaired.

## The engine

The cleaning itself is [scrubkit](https://pypi.org/project/scrubkit/): a
standalone, dependency-free Python library under Apache-2.0. Use it directly if
you are cleaning scraped data outside LangChain.

The same engine also runs as a hosted service at
[aidatatools.dev](https://aidatatools.dev), which adds dataset-level quality
scoring that neither library includes.

## Requirements

Python 3.10+, `langchain>=1.0`. The middleware hooks `wrap_tool_call`, checked
against langchain 1.0.0 and 1.3.x.

## License

Apache-2.0. See [LICENSE](LICENSE).
