Metadata-Version: 2.4
Name: aicurt
Version: 0.1.1
Summary: AICURT is a lightweight, dependency-free Python package for building RAG and retrieval-powered AI applications with PII protection, chunking, token analysis, and document search workflows.
Home-page: https://github.com/krishnatadi/aicurt
Author: krishna Tadi
License: Custom License
Project-URL: Documentation, https://github.com/krishnatadi/aicurt#readme
Project-URL: Source, https://github.com/krishnatadi/aicurt
Project-URL: Issue Tracker, https://github.com/krishnatadi/aicurt/issues
Keywords: AI,PII,redaction,privacy,text chunking,tokenization,NLP,data protection,Generative AI,Text Masking,retrieval,RAG,embeddings,vector search
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing
Classifier: Topic :: Security
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-python
Dynamic: summary

﻿[![PyPI Version](https://img.shields.io/pypi/v/aicurt?style=flat-square)](https://pypi.org/project/aicurt/)
[![Python Version](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue?style=flat-square)](https://www.python.org/)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen?style=flat-square)]()


# aicurt

`aicurt` is a lightweight, dependency-free Python package for building retrieval and RAG-powered AI applications.

It combines practical capabilities for modern AI systems:

- selective PII detection and redaction through explicit regex or rule registration
- deterministic chunking for downstream model input windows and retrieval pipelines
- lightweight tokenization and token statistics for embedding and search workflows
- retrieval-ready document indexing and search flows for RAG applications

## Why AICURT matters for AI search and RAG

AICURT is designed to help teams build the retrieval layer behind AI systems without bringing in a heavy dependency stack.

The package is useful when you want to:

- protect sensitive content before indexing documents
- split large text into clean, reusable chunks
- prepare queryable content for embedding and retrieval pipelines
- search documents by keyword, semantic similarity, or hybrid retrieval patterns
- keep runtime dependencies minimal and production-friendly

## Package overview

| Area | Capability |
| --- | --- |
| PII Engine | Register your own patterns and replace only the matches you want to protect |
| Chunking Engine | Split text with word, sentence, paragraph, sliding-window, recursive, and token strategies |
| Tokenizer | Tokenize text and compute token, word, and character statistics |
| Retrieval | Index chunked text and search it with keyword, semantic, and hybrid retrieval flows |
| Storage | Use `AICurtStore` for persistent document storage or `AICurtMemoryStore` for lightweight testing |
| CLI | Read from stdin or a file, then detect, redact, chunk, tokenize, or print stats |
| Packaging | Installable as a standard Python package with an `aicurt` console script |

## Retrieval overview

AICURT includes a lightweight retrieval layer built around:

- `AICurtEmbedder`: provider-neutral embedding abstraction
- `AICurtStore`: persistent document storage for indexing workflows
- `AICurtMemoryStore`: in-memory store for testing and quick prototypes
- `AICurtRAG`: indexing and retrieval workflow for chunked content
- `AICurtSearchResult`: structured result object with text, score, source, and metadata

This makes it a practical foundation for document search, retrieval pipelines, and RAG-based AI applications without vendor lock-in.

## What the current PII model does

The current design is intentionally selective:

- `PiiRedactor()` starts as a passive redactor with no built-in detection enabled
- you explicitly register the patterns you want to redact by calling `register_rule()` or `register_pattern()`
- `detect()` returns only the matches from the patterns you have registered
- `redact()` replaces only those matches, leaving unrelated text untouched

That makes the engine safer, more predictable, and easier to embed into production pipelines.

## Core API

### `PiiRedactor`

Public methods:

- `detect(text)` → returns a list of `PiiMatch` objects
- `mask(text)` → returns a `MaskingResult` with `original_text`, `masked_text`, `matches`, and `mapping`
- `redact(text, replacement=None)` → same redaction flow, with optional global replacement override
- `register_rule(name, pattern, replacement=...)` → register a named rule that will be detected and replaced
- `register_pattern(name, pattern)` → register a custom regex pattern for detection
- `addRegex(pattern)` / `addRegexPatterns(patterns)` → register more regexes
- `addWord(word, mask_length=None, case_sensitive=None)` / `addWords(...)` → register word-based masking rules
- `configureEmailMasking(...)` and `configurePhoneMasking(...)` → tweak the built-in masking settings when needed
- `setMaskCharacter(...)` and `setMaskLength(...)` → adjust the mask output style

### `PiiConfig`

`PiiConfig` is the configuration object that carries replacement, mask style, and masking-policy settings.

Common fields:

- `mask_strategy`
- `mask_char`
- `mask_visible_prefix`
- `mask_visible_suffix`
- `preserve_length`
- `default_replacement`
- `masking_policies`
- `enable_reversible`

## Installation

This package is available through the [PyPI registry](https://pypi.org/project/aicurt/).

Before installing, ensure you have Python 3.9 or higher installed. You can download and install Python from [python.org](https://www.python.org/downloads/__).

You can install the package using `pip`:

```bash
pip install aicurt

```

### Editable development install

```bash
python -m pip install -e .
```

### Verify the install

```bash
aicurt --help
aicurt --version
```

## Quick start

### 1) Protect sensitive information before indexing

This example shows the simplest and safest pattern: redact email addresses and phone numbers before you store or retrieve documents.

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)
redactor.register_rule(
    "PHONE",
    r"(?<!\w)(?:\+?\d[\d\s().-]{6,}\d)(?!\w)",
    replacement="[PHONE]",
)

text = "Contact alice@example.com or call +1 555 123 4567 for support."
print(redactor.redact(text).text)
```

Output will look like:

```text
Contact [EMAIL] or call [PHONE] for support.
```

This is useful when you want to keep private values out of your search index or retrieval store.

### 2) Chunk long text into smaller retrieval units

Large documents should be split into meaningful chunks before indexing. This improves retrieval quality and keeps context manageable for downstream AI workflows.

```python
from aicurt.chunking import ChunkStrategy, TextChunker

text = """
Artificial Intelligence is transforming the way teams build software.
Retrieval systems improve response quality by finding the right context.
Chunking helps keep each search unit small and relevant.
"""

chunks = TextChunker().chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=35,
)

for chunk in chunks:
    print(f"Chunk {chunk.index}: {chunk.content}")
```

This produces smaller text blocks that can be indexed individually and searched more accurately.

### 3) Build a RAG-style search index

The pattern below creates a tiny in-memory index, adds a few document chunks, and then searches them by keyword or semantic similarity.

```python
from aicurt import AICurtEmbedder, AICurtMemoryStore, AICurtRAG


class DemoEmbedder(AICurtEmbedder):
    def __init__(self):
        super().__init__(dimension=3)

    def embed(self, text: str):
        lower = text.lower()
        if "refund" in lower:
            return [1.0, 0.0, 0.0]
        if "shipping" in lower:
            return [0.0, 1.0, 0.0]
        if "support" in lower:
            return [0.0, 0.0, 1.0]
        return [0.4, 0.4, 0.4]

    def embed_many(self, texts):
        return [self.embed(text) for text in texts]


rag = AICurtRAG(
    store=AICurtMemoryStore(),
    embedder=DemoEmbedder(),
    protect=True,
    chunk_size=80,
    overlap=10,
)

rag.index_text(
    "Refunds are available within 30 days for eligible purchases.",
    source="refund_policy.txt",
    metadata={"team": "support"},
)
rag.index_text(
    "Shipping takes three to five business days for domestic delivery.",
    source="shipping_policy.txt",
    metadata={"team": "ops"},
)

results = rag.retrieve("refund policy", mode="keyword", top_k=3)
for item in results:
    print(item.text)
    print(item.score)
```

This demonstrates the main retrieval flow: index clean text, then quickly fetch the best matching chunks for a user query.

### 4) Search with semantic, keyword, and hybrid modes

AICURT supports different retrieval styles depending on the search problem.

```python
semantic = rag.retrieve("shipping delivery", mode="semantic", top_k=3)
keyword = rag.retrieve("refund", mode="keyword", top_k=3)
hybrid = rag.retrieve("refund and shipping", mode="hybrid", top_k=3)

print("SEMANTIC:", [item.text for item in semantic])
print("KEYWORD:", [item.text for item in keyword])
print("HYBRID:", [item.text for item in hybrid])
```

- `semantic` is best when you want similarity-based matching.
- `keyword` is best for direct literal term matches.
- `hybrid` blends both behaviors for more balanced ranking.

### 5) Filter results by metadata

Metadata filters help narrow retrieval to the right domain or team.

```python
filtered = rag.retrieve(
    "refund",
    mode="keyword",
    top_k=5,
    filters={"team": "support"},
)

for item in filtered:
    print(item.text, item.metadata)
```

This keeps the results relevant when your corpus contains multiple departments or document categories.

### 6) Persist data across runs with a local store

For long-lived projects, use a persistent store instead of a memory-only store.

```python
from aicurt import AICurtRAG, AICurtStore

store = AICurtStore("demo_store")
rag = AICurtRAG(store=store, embedder=DemoEmbedder(), protect=True)
rag.index_text("Support can help with billing and refunds.", source="support_guide.txt")
print(rag.retrieve("billing refund", mode="hybrid", top_k=2))
rag.close()
```

This makes it easier to reuse indexed knowledge between runs without needing a third-party vector database.

### Python API

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)

result = redactor.redact("Contact alice@example.com now")
print(result.text)
```

### Custom replacement callback

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)

result = redactor.redact(
    "alice@example.com",
    replacement=lambda match: f"<{match.entity_type}>",
)
print(result.text)
```

### Selective partial masking policy

```python
from aicurt.pii import PiiConfig, PiiRedactor

config = PiiConfig(
    mask_strategy="partial",
    mask_char="*",
    mask_visible_prefix=1,
    mask_visible_suffix=1,
    preserve_length=True,
)

redactor = PiiRedactor(config)
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)
redactor.register_rule(
    "PHONE",
    r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})\b",
    replacement="[PHONE]",
)

result = redactor.redact("Email: john@example.com, Phone: 9876543210")
print(result.text)
```

This keeps labels like `Email:` and `Phone:` intact while only replacing the registered sensitive values.

## End-to-end examples

### Example 1: Detect a custom email rule

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_pattern("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b")

matches = redactor.detect("Contact alice@example.com now")
for match in matches:
    print(match.entity_type, match.value, match.start, match.end)
```

### Example 2: Redact a custom organization name

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule("ORG", r"Acme", replacement="[COMPANY]")

result = redactor.redact("Acme is here")
print(result.text)
```

### Example 3: Register a custom word and mask it with a specific length

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.addWord("secret-token", mask_length=6)

result = redactor.redact("The secret-token value must be hidden")
print(result.text)
```

### Example 4: Chunk text

```python
from aicurt.chunking import ChunkStrategy, TextChunker

text = "Paragraph one. Paragraph two."
chunks = TextChunker().chunk(text, strategy=ChunkStrategy.SENTENCE, chunk_size=20)
for chunk in chunks:
    print(chunk.index, chunk.content)
```

### Example 5: Tokenize and compute stats

```python
from aicurt.tokenizer import SimpleTokenizer

text = "hello world"
tokenizer = SimpleTokenizer()
print(tokenizer.tokenize(text))
print(tokenizer.count_tokens(text))
print(tokenizer.stats(text))
```

### Example 6: Embedding-ready payload

```python
import json

result = redactor.redact("Contact alice@example.com now")
stats = SimpleTokenizer().stats("Contact alice@example.com now")
payload = {
    "masked_text": result.text,
    "token_count": stats.token_count,
    "word_count": stats.word_count,
    "character_count": stats.character_count,
    "matched_entities": [
        {"entity_type": match.entity_type, "value": match.value}
        for match in result.matches
    ],
}

print(json.dumps(payload, ensure_ascii=False, indent=2))
```

## PII Examples

Refer to the PII examples below and use them as a guide when implementing the PII masking in your code.

```python
from aicurt.pii import PiiConfig, PiiRedactor

paragraph_text = """Customer Information Report

Krishna Tadi is a product manager based in Bengaluru, Karnataka. His work email is krishna.t@example.com and his contact number is +91 90000000000.
During onboarding, Krishna shared his Aadhaar number 234567891234, PAN number ABCDE1234F, and passport number N1234567. The same profile also included a driver's license number DL-0420110012345.
The account team reviewed the customer's credit card number 4111 1111 1111 1111 and bank account number 123456789012. The date of birth listed in the record was 15-08-1995, and the latest login IP address was 192.168.1.105.
The account API key used for integration testing was sk_live_51N8example123456789, and the username associated with the account was krishna_t_95.
This document should remain confidential and should only be shared with authorized personnel for secure verification steps.
"""

mask_config = PiiConfig(
    mask_strategy="partial",
    mask_char="*",
    mask_visible_prefix=1,
    mask_visible_suffix=1,
    preserve_length=True,
)
redactor = PiiRedactor(mask_config)
redactor.register_rule("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b", replacement="[EMAIL]")
redactor.register_rule("PHONE", r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})\b", replacement="[PHONE]")
redactor.register_rule("AADHAAR", r"\b\d{12}\b", replacement="[AADHAAR]")
redactor.register_rule("PAN", r"\b[A-Z]{5}[0-9]{4}[A-Z]\b", replacement="[PAN]")
redactor.register_rule("BANK_ACCOUNT", r"\b\d{9,18}\b", replacement="[BANK_ACCOUNT]")
redactor.register_rule("DOB", r"\b(?:0?[1-9]|[12]\d|3[01])[-/](?:0?[1-9]|1[0-2])[-/](?:\d{4})\b", replacement="[DOB]")
redactor.register_rule("IP", r"\b(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}\b", replacement="[IP]")

redactor.addWord("KRISHNA")
redactor.addWord("4111 1111 1111 1111")

result = redactor.redact(paragraph_text)
matches = redactor.detect(paragraph_text)

print("Masked paragraph preview:")
print(result.text)
print("\nDetected matches:")
for match in matches[:6]:
    print(f"- {match.entity_type}: {match.value}")
```

## Chunking Examples
Refer to the chunking examples below and use them as a guide when implementing the chunking strategy in your code.
``` python

text = """
Artificial Intelligence is transforming the way developers build applications.
Large Language Models can understand and generate human-like text.
Retrieval Augmented Generation combines search with AI models.
Chunking is an important step because large documents need to be split into smaller pieces.
Good chunking improves embeddings, retrieval accuracy, and response quality.
This library provides deterministic text preprocessing utilities for AI workflows.
"""

chunker = TextChunker()

def print_chunks(title, chunks):
    print("\n")
    print("=" * 80)
    print(title)
    print("=" * 80)

    for chunk in chunks:
        print("\nChunk Index:", chunk.index)
        print("Content:")
        print(chunk.content)
        print("Start:", chunk.start)
        print("End:", chunk.end)
        print("Characters:", chunk.character_count)
        print("Words:", chunk.word_count)
        print("Tokens:", chunk.token_count)


# ============================================================
# 1. STANDARD CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.STANDARD,
    chunk_size=100
)

print_chunks(
    "STANDARD CHUNKING",
    chunks
)


# ============================================================
# 2. CHARACTER CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.CHARACTER,
    chunk_size=80
)

print_chunks(
    "CHARACTER CHUNKING",
    chunks
)


# ============================================================
# 3. WORD CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.WORD,
    chunk_size=20
)

print_chunks(
    "WORD CHUNKING",
    chunks
)


# ============================================================
# 4. SENTENCE CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=150
)

print_chunks(
    "SENTENCE CHUNKING",
    chunks
)


# ============================================================
# 5. PARAGRAPH CHUNKING
# ============================================================

paragraph_text = """
Artificial Intelligence is transforming applications.

Large Language Models are powerful AI systems.

Chunking improves retrieval performance.
"""


chunks = chunker.chunk(
    paragraph_text,
    strategy=ChunkStrategy.PARAGRAPH,
    chunk_size=100
)

print_chunks(
    "PARAGRAPH CHUNKING",
    chunks
)


# ============================================================
# 6. SLIDING WINDOW CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SLIDING_WINDOW,
    window_size=100,
    stride=50
)

print_chunks(
    "SLIDING WINDOW CHUNKING",
    chunks
)


# ============================================================
# 7. RECURSIVE CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.RECURSIVE,
    chunk_size=120
)

print_chunks(
    "RECURSIVE CHUNKING",
    chunks
)


# ============================================================
# 8. TOKEN CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.TOKEN,
    chunk_size=30
)

print_chunks(
    "TOKEN CHUNKING",
    chunks
)


# ============================================================
# 9. WORD CHUNK WITH OVERLAP
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.WORD,
    chunk_size=15,
    overlap=5
)

print_chunks(
    "WORD CHUNKING WITH OVERLAP",
    chunks
)


# ============================================================
# 10. CUSTOM SEPARATOR TEST
# ============================================================

custom_text = """
AI|Machine Learning|Deep Learning|Generative AI
"""


chunks = chunker.chunk(
    custom_text,
    strategy=ChunkStrategy.WORD,
    chunk_size=2,
    separators=["|"]
)

print_chunks(
    "CUSTOM SEPARATOR CHUNKING",
    chunks
)


# ============================================================
# 11. DISABLE SMALL CHUNK MERGING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=200,
    merge_small=False
)

print_chunks(
    "SENTENCE CHUNK WITHOUT MERGING",
    chunks
)


# ============================================================
# 12. INVALID INPUT TESTS
# ============================================================

print("\n")
print("=" * 80)
print("ERROR HANDLING TESTS")
print("=" * 80)


try:
    chunker.chunk(
        text,
        strategy=ChunkStrategy.WORD,
        chunk_size=0
    )

except Exception as e:
    print("Chunk size error:")
    print(type(e).__name__, e)


try:
    chunker.chunk(
        text,
        strategy=ChunkStrategy.WORD,
        overlap=-1
    )

except Exception as e:
    print("Overlap error:")
    print(type(e).__name__, e)


# ============================================================
# 13. ENUM TEST
# ============================================================

print("\n")
print("=" * 80)
print("SUPPORTED STRATEGIES")
print("=" * 80)


for strategy in ChunkStrategy:
    print(strategy.value)

`````

## RAG and retrieval examples

The sections below add the retrieval and RAG patterns used in real-world workflows. They are designed to be added on top of the existing preprocessing features without changing the rest of the README.

### Example 1: Protect text before indexing

Use PII redaction before storing documents in a retrieval layer so private values do not remain in your index.

```python
from aicurt.pii import PiiRedactor

text = "Contact alice@example.com for refund policy questions and call +1 555 123 4567."
redactor = PiiRedactor()
redactor.register_rule("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b", replacement="[EMAIL]")
redactor.register_rule("PHONE", r"(?<!\w)(?:\+?\d[\d\s().-]{6,}\d)(?!\w)", replacement="[PHONE]")

protected_text = redactor.redact(text).text
print(protected_text)
```

Example output:

```text
Contact [EMAIL] for refund policy questions and call [PHONE].
```

This keeps your knowledge base safe before search and retrieval are performed.

### Example 2: Index policy documents with metadata

This shows a realistic workflow where each document is stored with metadata such as department and language.

```python
from aicurt import AICurtEmbedder, AICurtRAG, AICurtStore


class MyEmbedder(AICurtEmbedder):
    def __init__(self):
        super().__init__(dimension=3)

    def embed(self, text):
        lower = text.lower()
        if "refund" in lower:
            return [1.0, 0.0, 0.0]
        if "policy" in lower:
            return [0.0, 1.0, 0.0]
        if "shipping" in lower:
            return [0.0, 0.0, 1.0]
        return [0.1, 0.1, 0.1]

    def embed_many(self, texts):
        return [self.embed(text) for text in texts]


store = AICurtStore("aicurtstore")
rag = AICurtRAG(store=store, embedder=MyEmbedder(), protect=True)

rag.index_text(
    "Contact alice@example.com for refund policy questions and call +1 555 123 4567.",
    source="customer_support.txt",
    metadata={"department": "support", "lang": "en"},
)
rag.index_text(
    "Shipping updates and delivery status are available in the order policy guide.",
    source="shipping_guide.txt",
    metadata={"department": "ops", "lang": "en"},
)

print(rag.store.count())
print(rag.store.list_ids())
```

This is the basic indexing pattern for RAG applications: store chunked text with useful metadata, then query it later.

### Example 3: Run semantic, keyword, and hybrid retrieval

Each retrieval mode serves a different purpose.

```python
semantic = rag.retrieve("refund", mode="semantic", top_k=5)
keyword = rag.retrieve("shipping status", mode="keyword", top_k=5)
hybrid = rag.retrieve("refund policy", mode="hybrid", top_k=5, filters={"department": "support"})

print("SEMANTIC:")
for item in semantic:
    print({
        "source": item.source,
        "text": item.text,
        "score": item.score,
        "metadata": item.metadata,
    })

print("KEYWORD:")
for item in keyword:
    print({
        "source": item.source,
        "text": item.text,
        "score": item.score,
        "metadata": item.metadata,
    })

print("HYBRID:")
for item in hybrid:
    print({
        "source": item.source,
        "text": item.text,
        "score": item.score,
        "metadata": item.metadata,
    })
```

What each mode is best for:

- `semantic`: similarity-based retrieval when wording differs from the source text
- `keyword`: direct match retrieval when exact terms matter most
- `hybrid`: balanced results that combine similarity and direct term relevance

### Example 4: Update an indexed record

Sometimes the content or metadata changes after indexing. AICURT allows updating a stored chunk record inline.

```python
first_id = rag.store.list_ids()[0]
record = rag.store.get(first_id)
print(record["text"])

rag.store.update(first_id, {
    "text": "Updated support policy: contact support@example.com for refund assistance.",
    "metadata": {"department": "support", "lang": "en", "status": "updated"},
})

updated = rag.store.get(first_id)
print(updated["text"])
print(updated["metadata"])
```

This is useful when a document changes and you want the retrieval index to reflect the new version.

### Example 5: Delete a record from the index

This pattern is useful for housekeeping when a document is no longer valid or should be removed from search.

```python
first_id = rag.store.list_ids()[0]
rag.store.delete(first_id)
print(rag.store.list_ids())
print(rag.store.count())
```

This keeps the index in sync with your actual source documents.

### Example 6: Close the store cleanly

On Windows and in long-lived apps, it is good practice to close the SQLite-backed store when you finish with it.

```python
rag.store.close()
print("Database closed successfully.")
```

This helps avoid stale file locks when deleting temporary directories or rerunning tests.

### Example 7: Full end-to-end RAG pattern

This is the full, realistic pattern used in real applications: redact content, index documents, search them, and narrow the results with metadata.

```python
from aicurt import AICurtEmbedder, AICurtRAG, AICurtStore
from aicurt.pii import PiiRedactor


class MyEmbedder(AICurtEmbedder):
    def __init__(self):
        super().__init__(dimension=3)

    def embed(self, text):
        lower = text.lower()
        if "refund" in lower:
            return [1.0, 0.0, 0.0]
        if "policy" in lower:
            return [0.0, 1.0, 0.0]
        if "shipping" in lower:
            return [0.0, 0.0, 1.0]
        return [0.1, 0.1, 0.1]

    def embed_many(self, texts):
        return [self.embed(text) for text in texts]


store = AICurtStore("aicurtstore")
rag = AICurtRAG(store=store, embedder=MyEmbedder(), protect=True)

redactor = PiiRedactor()
redactor.register_rule("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b", replacement="[EMAIL]")
redactor.register_rule("PHONE", r"(?<!\w)(?:\+?\d[\d\s().-]{6,}\d)(?!\w)", replacement="[PHONE]")

policy_text = "Contact alice@example.com for refund policy questions and call +1 555 123 4567."
protected_text = redactor.redact(policy_text).text

rag.index_text(protected_text, source="customer_support.txt", metadata={"department": "support", "lang": "en"})
rag.index_text(
    "Shipping updates and delivery status are available in the order policy guide.",
    source="shipping_guide.txt",
    metadata={"department": "ops", "lang": "en"},
)

results = rag.retrieve("refund policy", mode="hybrid", top_k=5, filters={"department": "support"})
for item in results:
    print(item.text)
    print(item.score)

rag.close()
```

## Summary

`aicurt` is a secure, lightweight, dependency-free toolkit for AI text preprocessing. The PII engine intentionally favors explicit, user-controlled matching and replacement so that only the patterns the caller chooses are masked. That makes it suitable for privacy-sensitive AI pipelines, embedded retrieval systems, and deterministic text cleaning workflows.


## CLI usage

The package installs one console entry point named `aicurt`.

### CLI command table

| Command | Purpose |
| --- | --- |
| `aicurt --help` | Show CLI help |
| `aicurt --version` | Show the package version |
| `aicurt detect <file>` | Detect registered entities from a file or stdin |
| `aicurt redact <file>` | Redact registered entities from a file or stdin |
| `aicurt chunk <file>` | Chunk the input text |
| `aicurt tokenize <file>` | Tokenize the input text |
| `aicurt stats <file>` | Show token, word, and character statistics |
| `aicurt rag index --input file.txt --store rag.db` | Index text into the local SQLite-backed RAG store |
| `aicurt rag search --query "refund policy" --store rag.db --mode hybrid --top-k 5` | Search the indexed content |

### CLI examples

```bash
aicurt detect sample.txt
aicurt redact sample.txt --output redacted.txt
aicurt chunk sample.txt --strategy word --chunk-size 50
aicurt tokenize sample.txt
aicurt rag index --input sample.txt --store rag.db --source support.txt
aicurt rag search --query "refund policy" --store rag.db --mode hybrid --top-k 5
```

### RAG CLI details

The built-in CLI includes a lightweight retrieval flow that does not require any third-party dependencies.

```bash
# Index plain text from stdin into a local SQLite store
printf "Refund policy allows returns within 30 days." | aicurt rag index --input - --store rag.db --source refund_policy.txt

# Search by keyword, semantic similarity, or hybrid ranking
 aicurt rag search --query "return refund" --store rag.db --mode hybrid --top-k 3
```

The `search` command prints JSON output like:

```json
{
  "query": "return refund",
  "mode": "hybrid",
  "top_k": 3,
  "results": [
    {
      "text": "Refund policy allows returns within 30 days.",
      "score": 0.95,
      "source": "refund_policy.txt",
      "document_id": "refund_policy.txt",
      "chunk_id": "...",
      "metadata": {}
    }
  ]
}
```

## Required inputs

The caller should provide:

| Workflow | Required input |
| --- | --- |
| PII detect | a text string, file path, or stdin stream |
| PII redact | a text string, file path, or stdin stream plus a rule or regex to register |
| Chunking | a text string, file path, or stdin stream plus a chunk strategy and chunk size |
| Tokenization | a text string, file path, or stdin stream |
| Custom masking | a regex pattern, a rule name, and a replacement token |

## Contributions

Contributions are welcome through the normal repository maintainer process.

For formal contribution or review requests:

1. open a pull request or review request through the repository workflow
2. keep changes aligned with the package’s security, data-privacy, and dependency-free design goals
3. preserve the selective, explicit registration model for PII redaction

For more details on contribution process please vist - [Contribution Guidelines](https://github.com/Krishnatadi/aicurt/blob/main/CONTRIBUTING.md)

## Code of Conduct

Please review our [Code of Conduct](https://github.com/Krishnatadi/aicurt/blob/main/CODE_OF_CONDUCT.md) before contributing to this project.


## Testing
This project uses Python's built-in `unittest` framework. All test cases are located inside the `tests/` directory.


## Security

Security-sensitive workflows should keep all processing local to the runtime environment.

Please review [SECURITY.md](https://github.com/Krishnatadi/aicurt/blob/main/SECURITY.md) for vulnerability disclosure guidance.

## License

This package is distributed under a restrictive proprietary all-rights-reserved license. See [LICENSE](https://github.com/Krishnatadi/aicurt/blob/main/LICENSE) and [NOTICE](https://github.com/Krishnatadi/aicurt/blob/main/NOTICE) for the exact legal terms.

## Best practices

- Use `PiiRedactor` with explicit `register_rule()` and `register_pattern()` calls for sensitive text handling.
- Use `mask()` when you want a structured `MaskingResult` object.
- Use `redact()` when you want a simple text replacement flow.
- Use `SimpleTokenizer` for lightweight token-aware chunking and embedding payload generation.
- Keep chunk sizes deterministic for downstream model input limits.
- Prefer explicit user-controlled redaction over implicit broad masking.
