Metadata-Version: 2.4
Name: anonymizer-ner
Version: 1.0.0
Summary: A zero-shot NER and document anonymization package.
Author-email: PrathamK <kalwani.pratham@gmail.com>
Project-URL: Homepage, https://github.com/k-pratham/anonymizer
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: gliner2
Requires-Dist: PyMuPDF
Requires-Dist: python-docx
Requires-Dist: pydantic
Requires-Dist: faker

# Anonymizer

A lightweight Python library for **Zero-Shot Named Entity Recognition (NER)** and **Document Anonymization**, powered by GLiNER 2. 

Anonymizer allows you to extract custom entities from PDFs and Word documents, dynamically redact them, and most importantly, it includes a **built-in memory override** so you can easily correct the AI when it makes mistakes.

## Installation

```bash
pip install anonymizer_ner
```
*(Note: Not actually published to PyPI yet. Install from source).*

## Features

- **Zero-Shot Extraction**: Define what you want to extract using natural language descriptions (e.g., `"address": "street addresses"`).
- **Document Anonymization**: Physically redact the extracted entities from PDFs (using secure PyMuPDF redaction) or DOCX files.
- **Feedback Memory Loop**: If the AI mislabels something, you can add a correction to the memory database. On all future extractions, the package will automatically override the AI and correctly label that specific text.

## Usage Guide

### 1. Extracting Entities

```python
import anonymizer_ner

# 1. Define AI entities using natural language
ai_entities = {
    "person": "Names of individuals",
    "address": "Street addresses and locations",
    "organization": "Companies or corporations"
}

# 2. Define strict pattern matching using Regex
regex_rules = {
    "aadhaar": r"\b\d{4}\s?\d{4}\s?\d{4}\b",
    "pan_card": r"\b[A-Z]{5}[0-9]{4}[A-Z]{1}\b"
}

with open("document.pdf", "rb") as f:
    matches = anonymizer_ner.extract(
        file_bytes=f.read(),
        filename="document.pdf",
        entity_definitions=ai_entities,
        regex_rules=regex_rules, # Pass the regex rules here!
        threshold=0.5
    )

for match in matches:
    print(f"[{match.label}] {match.text} (Page: {match.page}, Line: {match.line})")
```

### 2. Anonymizing Documents

You can securely redact all the entities you found. You can choose different strategies for how the text is hidden:

```python
import anonymizer_ner

with open("document.pdf", "rb") as f:
    pdf_bytes = f.read()

# Default strategy: "redact" (blackout boxes)
redacted_bytes = anonymizer_ner.anonymize(
    file_bytes=pdf_bytes, 
    filename="document.pdf", 
    matches=matches,
    strategy="redact"
)

# Other strategies available:
# strategy="replace_with_tags" -> Replaces "John" with "PERSON"
# strategy="replace_with_fake" -> Replaces "John" with a fake name like "Michael"
# strategy="replace_with_indexed_tags" -> Replaces "HDFC Bank" with "ORGANIZATION_1" consistently

with open("redacted_document.pdf", "wb") as f:
    f.write(redacted_bytes)
```

### 3. Using the Feedback Loop (Memory)

If the AI makes a mistake (e.g., it tags "John Doe" as an `address`), you can add a correction. By default, this is saved to `~/.anonymizer_ner/memory.json`, but you can isolate it per project:

```python
import anonymizer_ner

# Set a custom memory path for this specific project (optional)
anonymizer_ner.set_memory_path("./my_project_memory.json")

# Save a correction to the local memory database
anonymizer_ner.add_correction("John Doe", "person")
```

This correction is saved locally to `~/.anonymizer_ner/memory.json`. 

The next time you call `anonymizer_ner.extract()`, two things happen automatically:
1. The AI's internal prompt is augmented with your examples to help it generalize.
2. If the text "John Doe" is found anywhere in the document, it is **guaranteed** to be tagged as a `person` with 100% confidence, overriding any AI predictions.

### 4. Using a Custom Fine-Tuned Model

By default, the package uses `fastino/gliner2-base-v1`. If you have fine-tuned your own GLiNER 2 model on specific domain data, you can easily load it either globally or per-extraction.

**Set it globally:**
```python
import anonymizer_ner

# Can be a Hugging Face repo ID or a local path to the model weights
anonymizer_ner.set_model("path/to/my/finetuned/gliner_model")

# All subsequent extract() calls will now use your model
matches = anonymizer_ner.extract(...)
```

**Set it per extraction:**
```python
matches = anonymizer_ner.extract(
    file_bytes=f.read(),
    filename="document.pdf",
    entity_definitions=entities,
    model_name="path/to/my/finetuned/gliner_model"
)
```

### 5. Processing Multiple Documents (Low RAM)

The `anonymizer_ner` package is designed to be highly memory efficient. It breaks large documents into small chunks so a 10,000-page document uses the exact same amount of RAM as a 5-page document.

However, if you are running this on a standard laptop and need to process hundreds of documents, **do not use multiprocessing**. Loading multiple AI models at the same time will crash your RAM. Instead, use a simple `for` loop to process them sequentially in complete safety:

```python
import os
import anonymizer_ner

# Completely safe for low-RAM machines!
for filename in os.listdir("my_pdfs/"):
    if not filename.endswith(".pdf"):
        continue
        
    with open(f"my_pdfs/{filename}", "rb") as f:
        matches = anonymizer.extract(f.read(), filename, ai_entities)
        
    # Anonymize and save...
```

---

## Technical Details

- **Model**: `fastino/gliner2-base-v1` (runs locally, downloaded automatically on first use)
- **PDF Parsing**: `PyMuPDF`
- **DOCX Parsing**: `python-docx`
