Metadata-Version: 2.4
Name: llama-index-redefiner
Version: 0.1.0
Summary: LlamaIndex components that mask personal data or extract entities defined in a plain-English rulebook, using ReDefiNER.
Author: Sethuraman T A, Jungroo AI Labs
Author-email: Cibe Hariharan K <cibe@jungroo.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Jungroo-AI/llama-index-redefiner
Project-URL: Repository, https://github.com/Jungroo-AI/llama-index-redefiner
Project-URL: Issues, https://github.com/Jungroo-AI/llama-index-redefiner/issues
Project-URL: ReDefiNER, https://github.com/jungroo-ai/redefiner
Keywords: llama-index,llamaindex,pii,redaction,rag,named-entity-recognition,redefiner
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: llama-index-core>=0.12
Requires-Dist: redefiner>=1.0.9
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# llama-index-redefiner

Personal-data masking and entity extraction for [LlamaIndex](https://www.llamaindex.ai), using
[ReDefiNER](https://github.com/jungroo-ai/redefiner): an open model (0.8B parameters, Apache 2.0) that finds what a
rulebook written in plain English defines. It runs locally on CPU or GPU. English only.

| Component | Where it goes | What it does |
|---|---|---|
| `ReDefiNERPIINodePostprocessor` | query engine `node_postprocessors` | Masks personal data in retrieved nodes before the LLM sees them, like LlamaIndex's `NERPIINodePostprocessor`, and keeps a mapping so answers can be mapped back |
| `ReDefiNERPIIMasker` | `IngestionPipeline` transformations | Masks personal data in node text before embedding and indexing, so it never reaches the embedding model or the vector store |
| `ReDefiNEREntityExtractor` | `IngestionPipeline` transformations | Adds the entities a rulebook defines to node metadata, like LlamaIndex's `EntityExtractor` |

The PII components use the rulebook of
[`jungroo-ai/redefiner-pii`](https://huggingface.co/jungroo-ai/redefiner-pii) by default (20 types: names, contact
details, addresses, dates, ID and card numbers and more). Pass your own rulebook to mask by your own policy.

## Install

```bash
pip install llama-index-redefiner
```

The first call downloads the model weights (about 0.8 GB) from Hugging Face.

## Mask retrieved nodes

```python
from llama_index_redefiner import ReDefiNERPIINodePostprocessor

query_engine = index.as_query_engine(node_postprocessors=[ReDefiNERPIINodePostprocessor()])
```

A node that reads "Hi, this is Meera Nair from Brightwater Dental. Call me on 020 7946 0155." reaches the LLM as
"Hi, this is [PERSON_12] from [ORGANIZATION_28]. Call me on [PHONE_59]." The mapping from tags to the original text is
kept in `node.metadata["__pii_node_info__"]`, which is excluded from what the LLM and the embedding model see, as with
LlamaIndex's own PII postprocessors.

## Mask before indexing

```python
from llama_index.core import Document
from llama_index.core.ingestion import IngestionPipeline
from llama_index_redefiner import ReDefiNERPIIMasker

pipeline = IngestionPipeline(transformations=[ReDefiNERPIIMasker()])
nodes = pipeline.run(documents=[Document(text="Hi, this is Meera Nair from Brightwater Dental. Call me on 020 7946 0155.")])
nodes[0].get_content()
# 'Hi, this is [PERSON] from [ORGANIZATION]. Call me on [PHONE].'
```

Nothing of the masked text is kept. Put a text splitter and an embedding model after it in the same pipeline as usual.

## Extract entities into metadata

```python
from llama_index_redefiner import ReDefiNEREntityExtractor

rulebook = """[TYPE] PERSON
[DESCRIPTION] A person's name.

[TYPE] ORGANIZATION
[DESCRIPTION] A company.
"""
pipeline = IngestionPipeline(transformations=[ReDefiNEREntityExtractor(rulebook=rulebook)])
pipeline.run(documents=[Document(text="Hi, this is Meera Nair from Brightwater Dental.")])[0].metadata
# {'entities': ['Meera Nair', 'Brightwater Dental']}
```

With `label_entities=True` each type gets its own key (`{"PERSON": [...], "ORGANIZATION": [...]}`), renamed through
`entity_map` if given.

## Rulebooks and settings

A rulebook lists the types you want, with a one-line description each, and can add rules and a list of what must
never be found. See the [rulebook guide](https://github.com/jungroo-ai/redefiner/blob/main/docs/writing-rules.md).
`rulebook` takes the text or the path of a rulebook file.

Every component also takes `model`, `min_confidence`, `none_bias`, `threshold`, `max_span_tokens`, `device`,
`revision` and `batch_size`; see the ReDefiNER README for what the model settings do.

## Limits

- **Not the only safeguard.** The model misses some personal data and marks some text that is not personal. Check it
  on your own documents.
- **Speed.** On CPU the first call reads the rulebook (several seconds for the PII rulebook) and is then cached;
  later short texts take under a second. All nodes of a call go to the model in one batch.
- **English only.**

## Development

```bash
pip install -e ".[dev]"
pytest              # fast tests with a stand-in model
pytest -m slow      # the real model on CPU
```

## Licence

Apache License 2.0. See [LICENSE](LICENSE). Made by [Jungroo AI Labs](https://jungroo.com).
