Metadata-Version: 2.4
Name: mask-llm
Version: 0.1.1
Summary: Privacy-safe LLM data cleaning for fine-tuning datasets
Home-page: https://github.com/8418145/mask-llm
Author: 8418145
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: spacy>=3.7.0
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: python-dotenv>=1.0.0
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# mask-llm

> Privacy-safe data cleaning for LLM fine-tuning datasets.

Your data never leaves your machine unprotected. Names, emails, and phone 
numbers are masked **locally** before being sent to any LLM API. Results 
are automatically restored after cleaning.

## The Problem

When preparing fine-tuning datasets, your data often contains sensitive 
information: customer names, emails, internal project codenames. Sending 
this raw data to OpenAI or any cloud LLM is a privacy risk and may violate 
GDPR, HIPAA, or your company's data policy.

## How It Works

Your Data → [Local PII Masking] → Anonymized Text → [Cloud LLM Cleaning] → [Local Restore] → Clean Data

1. PII detected and replaced locally (`John Smith` → `[PERSON_1]`)
2. Only anonymized text is sent to the cleaning API
3. Cleaned result is restored with original values locally
4. Output as standard JSONL ready for fine-tuning

## Install
````bash
pip install mask-llm
python -m spacy download en_core_web_sm
````

## Quick Start

Prepare your input as a JSONL file (one JSON object per line):
````jsonl
{"text": "John Smith at john@company.com needs this dataset cleaned."}
{"text": "Call Sarah on 123-456-7890 to confirm the order."}
````

Run the cleaner:
````bash
python -m mask_llm \
  --input your_data.jsonl \
  --output cleaned_data.jsonl \
  --key YOUR_API_KEY
````

Get your free API key at: **https://nexus.xiaoku88.com**

## Output Fine-Tuning Format

Add `--finetune` to output standard instruction/input/output format:
````bash
python -m mask_llm \
  --input your_data.jsonl \
  --output cleaned_data.jsonl \
  --key YOUR_API_KEY \
  --finetune
````

Output:
````jsonl
{"instruction": "Clean and format the following text for fine-tuning.", "input": "John Smith at john@company.com...", "output": "John Smith at john@company.com..."}
````

## Options

| Option | Default | Description |
|--------|---------|-------------|
| `--input` | required | Input JSONL file |
| `--output` | required | Output JSONL file |
| `--key` | required | Your API key |
| `--model` | deepseek-chat | Model to use |
| `--concurrency` | 10 | Parallel requests |
| `--finetune` | false | Output fine-tuning format |
| `--prompt` | built-in | Custom system prompt |

## What Gets Masked

| Type | Example | Masked As |
|------|---------|-----------|
| Person names | John Smith | [PERSON_1] |
| Organizations | Apple Inc | [ORG_1] |
| Email addresses | john@example.com | [EMAIL_1] |
| Phone numbers | 123-456-7890 | [PHONE_1] |
| IP addresses | 192.168.1.1 | [IP_1] |
| SSN | 123-45-6789 | [SSN_1] |

## License

MIT
````

