Metadata-Version: 2.4
Name: langchain-redefiner
Version: 0.1.0
Summary: PII redaction and entity extraction for LangChain, using ReDefiNER and a plain-English rulebook.
Author: Sethuraman T A, Jungroo AI Labs
Author-email: Cibe Hariharan K <cibe@jungroo.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Jungroo-AI/langchain-redefiner
Project-URL: Repository, https://github.com/Jungroo-AI/langchain-redefiner
Project-URL: Issues, https://github.com/Jungroo-AI/langchain-redefiner/issues
Project-URL: ReDefiNER, https://github.com/jungroo-ai/redefiner
Keywords: langchain,middleware,pii,redaction,anonymization,named-entity-recognition,document-transformer,redefiner
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: langchain>=1.0
Requires-Dist: redefiner>=1.0.9
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# langchain-redefiner

Personal-data redaction and entity extraction for [LangChain](https://www.langchain.com), using
[ReDefiNER](https://github.com/jungroo-ai/redefiner): an open model (0.8B parameters, Apache 2.0) that finds what a
rulebook written in plain English defines. It runs locally on CPU or GPU, so text is redacted before it leaves your
machine. English only.

- **`ReDefiNERPIIMiddleware`**: agent middleware that redacts personal data in user messages (and optionally model
  output and tool results) before the model sees it. It is LangChain's own `PIIMiddleware` with ReDefiNER as the
  detector, so one middleware covers every type in the rulebook.
- **`redefiner_detector`**: the detector on its own, for `PIIMiddleware(..., detector=...)`.
- **`ReDefiNERRedactor`** and **`ReDefiNEREntityExtractor`**: document transformers that mask documents, or add the
  entities they mention to their metadata.

By default the PII rulebook of [`jungroo-ai/redefiner-pii`](https://huggingface.co/jungroo-ai/redefiner-pii) is used
(20 types: names, contact details, addresses, dates, ID and card numbers and more). Pass your own rulebook to redact
by your own policy.

## Install

```bash
pip install langchain-redefiner
```

The first call downloads the model weights (about 0.8 GB) from Hugging Face.

## Agent middleware

```python
from langchain.agents import create_agent
from langchain_redefiner import ReDefiNERPIIMiddleware

agent = create_agent(
    "openai:gpt-5",
    tools=[...],
    middleware=[ReDefiNERPIIMiddleware()],
)
agent.invoke({"messages": [{"role": "user", "content": "Refund Meera Nair, email meera.nair@example.org."}]})
# the model receives: "Refund [REDACTED_PERSON], email [REDACTED_EMAIL]."
```

Options:

| Option | Default | Meaning |
|---|---|---|
| `model` | `"jungroo-ai/redefiner-pii"` | A Hugging Face repo or local folder |
| `rulebook` | the model's default | Rulebook text, a path to a rulebook file, or a dict |
| `strategy` | `"redact"` | LangChain's strategies: `"redact"`, `"mask"`, `"hash"` or `"block"` |
| `types` | all | Only these rulebook types, e.g. `["PERSON", "EMAIL"]` |
| `apply_to_input` | `True` | Redact user messages before the model call |
| `apply_to_output` | `False` | Redact the model's messages |
| `apply_to_tool_results` | `False` | Redact tool results |
| `min_confidence` | `0.0` | Ignore spans below this confidence |
| `none_bias`, `threshold`, `max_span_tokens` | model defaults | ReDefiNER settings; see its README |
| `device` | GPU if available | `"cpu"`, `"cuda"` or `"cuda:N"` |

Each match's type is the rulebook type in lower case (`[REDACTED_PERSON]`). Emails, IP addresses, URLs and card numbers
use LangChain's own type names, so `strategy="mask"` handles them as it does its built-in types
(`meera.nair@****.org`).

To use LangChain's middleware directly:

```python
from langchain.agents.middleware import PIIMiddleware
from langchain_redefiner import redefiner_detector

PIIMiddleware("redefiner", detector=redefiner_detector(types=["PERSON"]), strategy="redact")
```

## Your own policy

Write the types you want masked, with a one-line description each, and anything that must stay:

```python
policy = """[TYPE] CUSTOMER_NAME
[DESCRIPTION] The name of a customer. Our own staff, who sign as "Agent <name>", are not customers.

[TYPE] ACCOUNT_NUMBER
[DESCRIPTION] A customer account number, such as "AC-449120".
"""

middleware = ReDefiNERPIIMiddleware(rulebook=policy)
# "Customer Daniel Brandt (account AC-771204) asked for a refund. Agent Rosa Diaz approved it." reaches the model as
# "Customer [REDACTED_CUSTOMER_NAME] (account [REDACTED_ACCOUNT_NUMBER]) asked for a refund. Agent Rosa Diaz approved it."
```

See the [rulebook guide](https://github.com/jungroo-ai/redefiner/blob/main/docs/writing-rules.md). Try a policy on a
few dozen of your own conversations before relying on it.

## Documents

```python
from langchain_core.documents import Document
from langchain_redefiner import ReDefiNEREntityExtractor, ReDefiNERRedactor

docs = [Document(page_content="Hi, this is Meera Nair from Brightwater Dental. Call me on 020 7946 0155.")]

ReDefiNERRedactor().transform_documents(docs)[0].page_content
# 'Hi, this is [PERSON] from [ORGANIZATION]. Call me on [PHONE].'

rulebook = """[TYPE] PERSON
[DESCRIPTION] A person's name.

[TYPE] ORGANIZATION
[DESCRIPTION] A company or institution.
"""
ReDefiNEREntityExtractor(rulebook=rulebook).transform_documents(docs)[0].metadata
# {'entities': {'PERSON': ['Meera Nair'], 'ORGANIZATION': ['Brightwater Dental']}}
```

- The redactor records what it masked in `metadata["redefiner_redactions"]`: type, confidence and offsets, but not
  the masked text (unless `keep_masked_text=True`).
- `ReDefiNERRedactor().redact_text` masks a single string; wrap it in `RunnableLambda` to use it in a chain.
- `ReDefiNEREntityExtractor(..., flat=True)` writes one key per type (`"entity_PERSON": "Meera Nair"`) for vector
  stores that only accept flat metadata.

## Limits

- **Not the only safeguard.** The model misses some personal data and marks some text that is not personal. Check it
  on your own data.
- **Speed.** On CPU the first call reads the rulebook (several seconds for the PII rulebook) and is then cached;
  later short messages take under a second. Each redacted message is one model call.
- **English only.**

## Development

```bash
pip install -e ".[dev]"
pytest              # fast tests with a stand-in model
pytest -m slow      # the real model on CPU
```

## Licence

Apache License 2.0. See [LICENSE](LICENSE). Made by [Jungroo AI Labs](https://jungroo.com).
