Metadata-Version: 2.4
Name: aicurt
Version: 0.1.0
Summary: `AICurt` is a lightweight, Python AI toolkit for AI text preprocessing that provides customizable PII redaction, deterministic text chunking, and token analysis utilities for LLM, embedding, and retrieval workflows.
Home-page: https://github.com/krishnatadi/aicurt
Author: krishna Tadi
License: Custom License
Project-URL: Documentation, https://github.com/krishnatadi/aicurt#readme
Project-URL: Source, https://github.com/krishnatadi/aicurt
Project-URL: Issue Tracker, https://github.com/krishnatadi/aicurt/issues
Keywords: AI,PII,redaction,privacy,text chunking,tokenization,NLP,data protection,Generative AI,Text Masking
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing
Classifier: Topic :: Security
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-python
Dynamic: summary

# aicurt

`aicurt` is a lightweight, dependency-free Python toolkit for AI text preprocessing.

It focuses on three things:

- selective PII detection and redaction through explicit regex or rule registration
- deterministic chunking for downstream model input windows
- lightweight tokenization and token statistics for embedding and retrieval workflows

## Package overview

| Area | Capability |
| --- | --- |
| PII Engine | Register your own patterns and replace only the matches you want to protect |
| Chunking Engine | Split text with word, sentence, paragraph, sliding-window, recursive, and token strategies |
| Tokenizer | Tokenize text and compute token, word, and character statistics |
| CLI | Read from stdin or a file, then detect, redact, chunk, tokenize, or print stats |
| Packaging | Installable as a standard Python package with an `aicurt` console script |

## What the current PII model does

The current design is intentionally selective:

- `PiiRedactor()` starts as a passive redactor with no built-in detection enabled
- you explicitly register the patterns you want to redact by calling `register_rule()` or `register_pattern()`
- `detect()` returns only the matches from the patterns you have registered
- `redact()` replaces only those matches, leaving unrelated text untouched

That makes the engine safer, more predictable, and easier to embed into production pipelines.

## Core API

### `PiiRedactor`

Public methods:

- `detect(text)` â†’ returns a list of `PiiMatch` objects
- `mask(text)` â†’ returns a `MaskingResult` with `original_text`, `masked_text`, `matches`, and `mapping`
- `redact(text, replacement=None)` â†’ same redaction flow, with optional global replacement override
- `register_rule(name, pattern, replacement=...)` â†’ register a named rule that will be detected and replaced
- `register_pattern(name, pattern)` â†’ register a custom regex pattern for detection
- `addRegex(pattern)` / `addRegexPatterns(patterns)` â†’ register more regexes
- `addWord(word, mask_length=None, case_sensitive=None)` / `addWords(...)` â†’ register word-based masking rules
- `configureEmailMasking(...)` and `configurePhoneMasking(...)` â†’ tweak the built-in masking settings when needed
- `setMaskCharacter(...)` and `setMaskLength(...)` â†’ adjust the mask output style

### `PiiConfig`

`PiiConfig` is the configuration object that carries replacement, mask style, and masking-policy settings.

Common fields:

- `mask_strategy`
- `mask_char`
- `mask_visible_prefix`
- `mask_visible_suffix`
- `preserve_length`
- `default_replacement`
- `masking_policies`
- `enable_reversible`

## Installation

This package is available through the [PyPI registry](__https://pypi.org/project/random-password-toolkit/__).

Before installing, ensure you have Python 3.9 or higher installed. You can download and install Python from [python.org](__https://www.python.org/downloads/__).

You can install the package using `pip`:

```bash
pip install aicurt

```

### Editable development install

```bash
python -m pip install -e .
```

### Verify the install

```bash
aicurt --help
aicurt --version
```

## Quick start

### Python API

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)

result = redactor.redact("Contact alice@example.com now")
print(result.text)
```

### Custom replacement callback

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)

result = redactor.redact(
    "alice@example.com",
    replacement=lambda match: f"<{match.entity_type}>",
)
print(result.text)
```

### Selective partial masking policy

```python
from aicurt.pii import PiiConfig, PiiRedactor

config = PiiConfig(
    mask_strategy="partial",
    mask_char="*",
    mask_visible_prefix=1,
    mask_visible_suffix=1,
    preserve_length=True,
)

redactor = PiiRedactor(config)
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)
redactor.register_rule(
    "PHONE",
    r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})\b",
    replacement="[PHONE]",
)

result = redactor.redact("Email: john@example.com, Phone: 9876543210")
print(result.text)
```

This keeps labels like `Email:` and `Phone:` intact while only replacing the registered sensitive values.

## End-to-end examples

### Example 1: Detect a custom email rule

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_pattern("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b")

matches = redactor.detect("Contact alice@example.com now")
for match in matches:
    print(match.entity_type, match.value, match.start, match.end)
```

### Example 2: Redact a custom organization name

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule("ORG", r"Acme", replacement="[COMPANY]")

result = redactor.redact("Acme is here")
print(result.text)
```

### Example 3: Register a custom word and mask it with a specific length

```python
from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.addWord("secret-token", mask_length=6)

result = redactor.redact("The secret-token value must be hidden")
print(result.text)
```

### Example 4: Chunk text

```python
from aicurt.chunking import ChunkStrategy, TextChunker

text = "Paragraph one. Paragraph two."
chunks = TextChunker().chunk(text, strategy=ChunkStrategy.SENTENCE, chunk_size=20)
for chunk in chunks:
    print(chunk.index, chunk.content)
```

### Example 5: Tokenize and compute stats

```python
from aicurt.tokenizer import SimpleTokenizer

text = "hello world"
tokenizer = SimpleTokenizer()
print(tokenizer.tokenize(text))
print(tokenizer.count_tokens(text))
print(tokenizer.stats(text))
```

### Example 6: Embedding-ready payload

```python
import json

result = redactor.redact("Contact alice@example.com now")
stats = SimpleTokenizer().stats("Contact alice@example.com now")
payload = {
    "masked_text": result.text,
    "token_count": stats.token_count,
    "word_count": stats.word_count,
    "character_count": stats.character_count,
    "matched_entities": [
        {"entity_type": match.entity_type, "value": match.value}
        for match in result.matches
    ],
}

print(json.dumps(payload, ensure_ascii=False, indent=2))
```

## PII Examples

Refer to the PII examples below and use them as a guide when implementing the PII masking in your code.

```python
from aicurt.pii import PiiConfig, PiiRedactor

paragraph_text = """Customer Information Report

Krishna Tadi is a product manager based in Bengaluru, Karnataka. His work email is krishna.t@example.com and his contact number is +91 90000000000.
During onboarding, Krishna shared his Aadhaar number 234567891234, PAN number ABCDE1234F, and passport number N1234567. The same profile also included a driver's license number DL-0420110012345.
The account team reviewed the customer's credit card number 4111 1111 1111 1111 and bank account number 123456789012. The date of birth listed in the record was 15-08-1995, and the latest login IP address was 192.168.1.105.
The account API key used for integration testing was sk_live_51N8example123456789, and the username associated with the account was krishna_t_95.
This document should remain confidential and should only be shared with authorized personnel for secure verification steps.
"""

mask_config = PiiConfig(
    mask_strategy="partial",
    mask_char="*",
    mask_visible_prefix=1,
    mask_visible_suffix=1,
    preserve_length=True,
)
redactor = PiiRedactor(mask_config)
redactor.register_rule("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b", replacement="[EMAIL]")
redactor.register_rule("PHONE", r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})\b", replacement="[PHONE]")
redactor.register_rule("AADHAAR", r"\b\d{12}\b", replacement="[AADHAAR]")
redactor.register_rule("PAN", r"\b[A-Z]{5}[0-9]{4}[A-Z]\b", replacement="[PAN]")
redactor.register_rule("BANK_ACCOUNT", r"\b\d{9,18}\b", replacement="[BANK_ACCOUNT]")
redactor.register_rule("DOB", r"\b(?:0?[1-9]|[12]\d|3[01])[-/](?:0?[1-9]|1[0-2])[-/](?:\d{4})\b", replacement="[DOB]")
redactor.register_rule("IP", r"\b(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}\b", replacement="[IP]")

redactor.addWord("KRISHNA")
redactor.addWord("4111 1111 1111 1111")

result = redactor.redact(paragraph_text)
matches = redactor.detect(paragraph_text)

print("Masked paragraph preview:")
print(result.text)
print("\nDetected matches:")
for match in matches[:6]:
    print(f"- {match.entity_type}: {match.value}")
```

## Chunking Examples
Refer to the chunking examples below and use them as a guide when implementing the chunking strategy in your code.
``` python

text = """
Artificial Intelligence is transforming the way developers build applications.
Large Language Models can understand and generate human-like text.
Retrieval Augmented Generation combines search with AI models.
Chunking is an important step because large documents need to be split into smaller pieces.
Good chunking improves embeddings, retrieval accuracy, and response quality.
This library provides deterministic text preprocessing utilities for AI workflows.
"""

chunker = TextChunker()

def print_chunks(title, chunks):
    print("\n")
    print("=" * 80)
    print(title)
    print("=" * 80)

    for chunk in chunks:
        print("\nChunk Index:", chunk.index)
        print("Content:")
        print(chunk.content)
        print("Start:", chunk.start)
        print("End:", chunk.end)
        print("Characters:", chunk.character_count)
        print("Words:", chunk.word_count)
        print("Tokens:", chunk.token_count)


# ============================================================
# 1. STANDARD CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.STANDARD,
    chunk_size=100
)

print_chunks(
    "STANDARD CHUNKING",
    chunks
)


# ============================================================
# 2. CHARACTER CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.CHARACTER,
    chunk_size=80
)

print_chunks(
    "CHARACTER CHUNKING",
    chunks
)


# ============================================================
# 3. WORD CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.WORD,
    chunk_size=20
)

print_chunks(
    "WORD CHUNKING",
    chunks
)


# ============================================================
# 4. SENTENCE CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=150
)

print_chunks(
    "SENTENCE CHUNKING",
    chunks
)


# ============================================================
# 5. PARAGRAPH CHUNKING
# ============================================================

paragraph_text = """
Artificial Intelligence is transforming applications.

Large Language Models are powerful AI systems.

Chunking improves retrieval performance.
"""


chunks = chunker.chunk(
    paragraph_text,
    strategy=ChunkStrategy.PARAGRAPH,
    chunk_size=100
)

print_chunks(
    "PARAGRAPH CHUNKING",
    chunks
)


# ============================================================
# 6. SLIDING WINDOW CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SLIDING_WINDOW,
    window_size=100,
    stride=50
)

print_chunks(
    "SLIDING WINDOW CHUNKING",
    chunks
)


# ============================================================
# 7. RECURSIVE CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.RECURSIVE,
    chunk_size=120
)

print_chunks(
    "RECURSIVE CHUNKING",
    chunks
)


# ============================================================
# 8. TOKEN CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.TOKEN,
    chunk_size=30
)

print_chunks(
    "TOKEN CHUNKING",
    chunks
)


# ============================================================
# 9. WORD CHUNK WITH OVERLAP
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.WORD,
    chunk_size=15,
    overlap=5
)

print_chunks(
    "WORD CHUNKING WITH OVERLAP",
    chunks
)


# ============================================================
# 10. CUSTOM SEPARATOR TEST
# ============================================================

custom_text = """
AI|Machine Learning|Deep Learning|Generative AI
"""


chunks = chunker.chunk(
    custom_text,
    strategy=ChunkStrategy.WORD,
    chunk_size=2,
    separators=["|"]
)

print_chunks(
    "CUSTOM SEPARATOR CHUNKING",
    chunks
)


# ============================================================
# 11. DISABLE SMALL CHUNK MERGING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=200,
    merge_small=False
)

print_chunks(
    "SENTENCE CHUNK WITHOUT MERGING",
    chunks
)


# ============================================================
# 12. INVALID INPUT TESTS
# ============================================================

print("\n")
print("=" * 80)
print("ERROR HANDLING TESTS")
print("=" * 80)


try:
    chunker.chunk(
        text,
        strategy=ChunkStrategy.WORD,
        chunk_size=0
    )

except Exception as e:
    print("Chunk size error:")
    print(type(e).__name__, e)


try:
    chunker.chunk(
        text,
        strategy=ChunkStrategy.WORD,
        overlap=-1
    )

except Exception as e:
    print("Overlap error:")
    print(type(e).__name__, e)


# ============================================================
# 13. ENUM TEST
# ============================================================

print("\n")
print("=" * 80)
print("SUPPORTED STRATEGIES")
print("=" * 80)


for strategy in ChunkStrategy:
    print(strategy.value)

`````

## CLI usage

The package installs one console entry point named `aicurt`.

### CLI command table

| Command | Purpose |
| --- | --- |
| `aicurt --help` | Show CLI help |
| `aicurt --version` | Show the package version |
| `aicurt detect <file>` | Detect registered entities from a file or stdin |
| `aicurt redact <file>` | Redact registered entities from a file or stdin |
| `aicurt chunk <file>` | Chunk the input text |
| `aicurt tokenize <file>` | Tokenize the input text |
| `aicurt stats <file>` | Show token, word, and character statistics |

### CLI examples

```bash
aicurt detect sample.txt
aicurt redact sample.txt --output redacted.txt
aicurt chunk sample.txt --strategy word --chunk-size 50
aicurt tokenize sample.txt
```

## Required inputs

The caller should provide:

| Workflow | Required input |
| --- | --- |
| PII detect | a text string, file path, or stdin stream |
| PII redact | a text string, file path, or stdin stream plus a rule or regex to register |
| Chunking | a text string, file path, or stdin stream plus a chunk strategy and chunk size |
| Tokenization | a text string, file path, or stdin stream |
| Custom masking | a regex pattern, a rule name, and a replacement token |

## Contributions

Contributions are welcome through the normal repository maintainer process.

For formal contribution or review requests:

1. open a pull request or review request through the repository workflow
2. keep changes aligned with the packageâ€™s security, data-privacy, and dependency-free design goals
3. preserve the selective, explicit registration model for PII redaction

For more details on contribution process please vist - [Contribution Guidelines](https://github.com/Krishnatadi/aicurt/blob/main/CONTRIBUTING.md)

## Code of Conduct

Please review our [Code of Conduct](https://github.com/Krishnatadi/aicurt/blob/main/CODE_OF_CONDUCT.md) before contributing to this project.

## Security

Security-sensitive workflows should keep all processing local to the runtime environment.

Please review [SECURITY.md](https://github.com/Krishnatadi/aicurt/blob/main/SECURITY.md) for vulnerability disclosure guidance.

## License

This package is distributed under a restrictive proprietary all-rights-reserved license. See [LICENSE](https://github.com/Krishnatadi/aicurt/blob/main/LICENSE) and [NOTICE](https://github.com/Krishnatadi/aicurt/blob/main/NOTICE) for the exact legal terms.

## Best practices

- Use `PiiRedactor` with explicit `register_rule()` and `register_pattern()` calls for sensitive text handling.
- Use `mask()` when you want a structured `MaskingResult` object.
- Use `redact()` when you want a simple text replacement flow.
- Use `SimpleTokenizer` for lightweight token-aware chunking and embedding payload generation.
- Keep chunk sizes deterministic for downstream model input limits.
- Prefer explicit user-controlled redaction over implicit broad masking.

## Summary

`aicurt` is a secure, lightweight, dependency-free toolkit for AI text preprocessing. The PII engine intentionally favors explicit, user-controlled matching and replacement so that only the patterns the caller chooses are masked. That makes it suitable for privacy-sensitive AI pipelines, embedded retrieval systems, and deterministic text cleaning workflows.

