Metadata-Version: 2.5
Name: credgate
Version: 0.5.0
Summary: Governance layer for LLM requests with Indian PII detection
Project-URL: Homepage, https://github.com/dwarpal/credgate
Project-URL: Issues, https://github.com/dwarpal/credgate/issues
Project-URL: Changelog, https://github.com/dwarpal/credgate/blob/main/pkg/python/CHANGELOG.md
Author-email: Bharat Sikka <bharat8sikka@gmail.com>, Aishwarya Bais <aishwarya.bais00@gmail.com>
Maintainer-email: Bharat Sikka <bharat8sikka@gmail.com>, Aishwarya Bais <aishwarya.bais00@gmail.com>
License-Expression: Apache-2.0
License-File: AUTHORS
License-File: LICENSE
License-File: NOTICE
Keywords: aadhaar,compliance,dpdp,india,llm,pii,privacy
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Security
Classifier: Topic :: Text Processing
Requires-Python: >=3.11
Requires-Dist: pyyaml>=6.0
Requires-Dist: regex>=2023.0
Provides-Extra: all
Requires-Dist: duckdb>=1.1; extra == 'all'
Requires-Dist: numpy; extra == 'all'
Requires-Dist: polars>=1.0; extra == 'all'
Requires-Dist: pyarrow>=14; extra == 'all'
Requires-Dist: pynacl>=1.5; extra == 'all'
Provides-Extra: arrow
Requires-Dist: pyarrow>=14; extra == 'arrow'
Provides-Extra: duckdb
Requires-Dist: duckdb>=1.1; extra == 'duckdb'
Requires-Dist: numpy; extra == 'duckdb'
Requires-Dist: pyarrow>=14; extra == 'duckdb'
Provides-Extra: polars
Requires-Dist: polars>=1.0; extra == 'polars'
Provides-Extra: test
Requires-Dist: pynacl>=1.5; extra == 'test'
Requires-Dist: pytest>=7; extra == 'test'
Provides-Extra: vault
Requires-Dist: pynacl>=1.5; extra == 'vault'
Description-Content-Type: text/markdown

# credgate

Detects personal data before it leaves the machine, with Indian identifiers
built in: Aadhaar (with Verhoeff checksum), Aadhaar Virtual ID, PAN, GSTIN,
voter ID, passport, driving licence, UPI IDs, IFSC, bank accounts, payment
cards, +91 mobile numbers, ABHA, and names and addresses in Latin and Indic
scripts.

The detection rules live in a shared, language-neutral spec, so this package
and the R package `credgate` find exactly the same matches. Part of the
[CredGate](https://github.com/dwarpal/credgate) project; early development.

## Use it with local LLMs (Ollama, vLLM)

Redact before the prompt reaches the model; the model only ever sees
placeholders such as `[PAN]` and `[PERSON_NAME]`.

```python
import credgate, ollama

note = "Customer name: Priya Nair. PAN ABCPN1234K, mobile +91 98765 43210, wants the EMI date moved."
print(credgate.preview(note, min_confidence=0.5))   # what will be masked, and why
safe = credgate.redact(note, min_confidence=0.5)    # "Customer name: [PERSON_NAME]. PAN [PAN], mobile [MOBILE_IN], ..."
reply = ollama.chat(model="gemma3:1b", messages=[{"role": "user", "content": f"Summarise: {safe}"}])
```

With **vLLM** (or Ollama's `/v1`, LM Studio, any OpenAI-compatible server):

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
client.chat.completions.create(model="<served model>",
                               messages=[{"role": "user", "content": credgate.redact(note, min_confidence=0.5)}])
```

From the shell: `credgate redact notes.txt | ollama run gemma3:1b "Summarise:"`.
More runnable examples (ticket classification, RAG ingestion) are in the
[examples folder](https://github.com/dwarpal/credgate/tree/main/examples).

> **Check the preview.** Names are found next to an honorific, relation or
> field label (`Shri`, `S/o`, `Name:`, `नाम:`); a name in running text
> without such a cue ("Customer Priya Nair called") is not detected yet.
> Redaction is irreversible; to get the real values back in the model's
> answer, tokenise instead (see below). An automatic proxy is on the roadmap.

## How it compares

| On 700 held-out synthetic documents | Precision | Recall | **F1** | Time |
|---|---:|---:|---:|---:|
| **credgate** | 0.999 | 0.957 | **0.977** | 0.2 s |
| Presidio, India-configured | 0.600 | 0.748 | 0.666 | 11.4 s |
| Presidio, default | 0.555 | 0.472 | 0.510 | 11.9 s |

<sub>**About the data:** the 700 documents are **synthetic** - KYC forms, bank
messages, call-centre notes, Hinglish chats, Hindi text and hospital
summaries generated by
[`benchmark/generate_corpus.py`](https://github.com/dwarpal/credgate/blob/main/benchmark/generate_corpus.py)
and included in the repository under
[`benchmark/corpus/`](https://github.com/dwarpal/credgate/tree/main/benchmark/corpus).
No real personal data was used.
Types both tools cover, exact span matching, thresholds tuned on a
separate dev split for every system; time is for 1,400 documents. Synthetic
data generated by us, so read with that in mind.</sub>

**Where Presidio does better.** credgate is built for Indian identifiers
and Indian-language context, and that is what this benchmark measures.
Presidio is broader and more mature:

- **Voter IDs without context**: Presidio scores 1.000 here, credgate 0.677
  (credgate reports a bare voter ID format at low confidence, 0.3, because
  it could equally be a product code).
- **Many more entity types**: international IDs (US, UK, EU, ...), IBAN, IP
  and MAC addresses, URLs, crypto wallets, dates, nationality and religion -
  credgate detects none of these.
- **Names in running text**: Presidio's NER finds names without an honorific,
  field label or "ji"; credgate does not yet.
- **Anonymisation**: Presidio can mask, hash or encrypt with a key, and
  add custom operators; credgate redacts, or tokenises with an in-memory
  vault that can be saved encrypted (no hashing or encryption operators).
- **Beyond text**: image (OCR) and structured-data redaction, a REST service
  and Docker images.
- **Maturity and extensibility**: years in production, many languages and
  NLP back-ends, custom recognisers; credgate is pre-alpha.
- **The corpus favours credgate**: we generated it; on general English text
  Presidio may well do better.

Full method, per-type results and how to reproduce:
[benchmark/README.md](https://github.com/dwarpal/credgate/blob/main/benchmark/README.md).
The comparison is independent and not affiliated with Microsoft.

## Installation

Python 3.11 or later; depends only on `regex` and `PyYAML`.

```sh
pip install credgate
```

The development version: `pip install "git+https://github.com/dwarpal/credgate#subdirectory=pkg/python"`.

With Nix: `nix build github:dwarpal/credgate#credgate`.

## Usage

```python
import credgate

spans = credgate.detect_pii("Name: Priya Nair; PAN: ABCPN1234K; UPI: priya.n@oksbi")
for s in spans:
    print(s)
# Span(doc=0, start=6, end=16, type='person_name', value='XXXXX XXXX', confidence=0.8)
# Span(doc=0, start=23, end=33, type='pan', value='XXXXXX234K', confidence=0.97)
# Span(doc=0, start=40, end=53, type='upi_id', value='XXXXX.X@Xksbi', confidence=0.99)

spans[1].value    # 'ABCPN1234K' - printing masks values, the attribute does not
spans[1].reasons  # "matched PAN pattern; context 'PAN'; holder type: individual"
```

`detect_pii()` takes a string or a list of strings and returns `Span`
objects with the document index, 0-based end-exclusive positions, type, data
class, value, canonical form, confidence and reasons. `span.to_dict()` gives
a JSON-ready dict with the value masked (`reveal=True` to include it).

Redact identifiers, or preview what would be redacted and why:

```python
credgate.redact("Name: Priya Nair; PAN: ABCPN1234K")
# 'Name: [PERSON_NAME]; PAN: [PAN]'

print(credgate.preview("Name: Priya Nair; PAN: ABCPN1234K; order no 878245671885"))
# 3 items would be redacted:
#   Name: [PERSON_NAME]; PAN: [PAN]; order no [AADHAAR]
#
#   [PERSON_NAME]  XXXXX XXXX    0.80  matched Person name pattern
#   [PAN]          XXXXXX234K    0.97  matched PAN pattern; context 'PAN'; ...
#   [AADHAAR]      XXXXXXXX1885  0.42  ...; negative context 'order'
```

Tokenise instead, to get the real values back in a model's answer: the
vault keeps the mapping, and `restore()` puts the values back:

```python
vault = credgate.Vault()
credgate.tokenise([
    "Shri Ramesh Kumar, PAN ABCPE1234F, mobile +91 98765 43210",
    "pan abcpe1234f called again from 9876543210",
], vault)
# ['Shri [PERSON_NAME_1], PAN [PAN_1], mobile [MOBILE_IN_1]',
#  'pan [PAN_1] called again from [MOBILE_IN_1]']

credgate.restore("[PERSON_NAME_1] (PAN [pan_1]) will be called on MOBILE_IN_1.", vault)
# 'Ramesh Kumar (PAN ABCPE1234F) will be called on +91 98765 43210.'
```

The same identifier gets the same token however it is written, so the model
can still tell customers apart. `restore()` accepts tokens as models tend to
copy them (other case, spaces, missing brackets); `strict=True` raises
`UnknownTokenError` on a token the vault does not know. The vault prints
with values masked.

To restore tokens in a later session - or in R - save the vault to a file
encrypted with a passphrase (at least 12 characters; from the argument, the
`CREDGATE_VAULT_PASSPHRASE` environment variable, or a prompt):

```python
vault.save("loans.cgvault")                    # pip install 'credgate[vault]'
vault = credgate.Vault.open("loans.cgvault")   # later: same tokens, numbering continues
```

The passphrase is never stored: without it the file cannot be opened. Keep
`*.cgvault` files out of version control.

Scan a table (a pandas DataFrame, or a dict of column -> values) to see
which columns hold PII; column names count as context:

```python
report = credgate.scan_pii({
    "cust_pan": ["ABCPE1234F", "AAACR5055K", "XYZPQ9876R"],
    "notes": ["Called, Aadhaar 542229371182", "no issues", "mobile 9876543210"],
})
print(report)
# Scanned 2 column(s); PII found in 2: cust_pan, notes
#   cust_pan  pan              100% of 3 (3 whole value)  confidence 0.97  e.g. XXXXXX234F
#   notes     aadhaar           33% of 3 (0 whole value)  confidence 0.97  e.g. XXXXXXXX1182
#   notes     mobile_in         33% of 3 (0 whole value)  confidence 0.93  e.g. XXXXXX3210
pandas.DataFrame(report.to_records())  # as a table
```

Context makes generic patterns trustworthy; pass a column name or similar:

```python
credgate.detect_pii("ABC1234567")                      # [] - could be anything
credgate.detect_pii("ABC1234567", context="voter_id")  # voter ID, confidence 0.97
```

Checksums, synthetic test data and your own detectors:

```python
credgate.verhoeff_valid("542229371182")
credgate.synth_indian_pii(5, seed=1)   # synthetic records; never use as anonymised data

credgate.register_detector(credgate.Detector(
    id="employee_id", data_class="internal", label="Employee ID",
    patterns=credgate.bounded("EMP[0-9]{6}"),
))
```

## Large data: pandas, Polars, Arrow, DuckDB, Parquet

`redact()`, `tokenise()` and `detect_pii()` accept a pandas or Polars Series
or an Arrow array and return the same type (a Series keeps its index and
name; missing values stay missing). For tables, install the extra you need:
`pip install 'credgate[polars]'`, `'credgate[arrow]'` or `'credgate[duckdb]'`,
or everything (these, plus encrypted vaults) with `pip install 'credgate[all]'`.

```python
import polars as pl
import credgate.polars  # adds the .credgate namespace

df.with_columns(pl.col("notes").credgate.redact(min_confidence=0.5))
df.filter(pl.col("notes").credgate.has_pii())
```

```python
import duckdb, credgate.duckdb

con = duckdb.connect()
credgate.duckdb.register(con, min_confidence=0.5)   # adds cg_redact() and cg_has_pii()
con.sql("COPY (SELECT id, cg_redact(notes) AS notes FROM 'calls.parquet') TO 'safe.parquet'")
```

```python
import credgate.arrow
credgate.arrow.redact_parquet("calls.parquet", "safe.parquet", columns=["notes"])
```

```sh
credgate redact --parquet calls.parquet --column notes -o safe.parquet
credgate scan calls.parquet
```

See the [large-data guide](https://github.com/dwarpal/credgate/blob/main/docs/large-data.md)
for the approach and runnable examples. Parquet files, DuckDB queries and
Polars lazy frames are processed in
batches, so the data may be larger than memory. `scan_pii()` also accepts
Polars frames (eager or lazy), Arrow tables and datasets, and DuckDB
relations; it reads one column at a time and samples exactly as it does in
memory, so the report is the same at any size. Numbered tokens are not
offered inside Polars and DuckDB, which process batches in parallel and in
no fixed order; tokenise a Series or Arrow array instead.

## Command line

```sh
credgate detect notes.txt                      # masked report
credgate detect --lines --json customers.csv   # one JSON object per match
git diff --cached | credgate detect --fail     # non-zero exit if PII is found
credgate redact notes.txt > notes.redacted.txt  # replace PII with [PAN] etc.
credgate preview notes.txt                     # what would be redacted, and why
credgate scan customers.csv                    # which columns hold PII
credgate detectors                             # list detectors
```

Values are masked unless you pass `--reveal`.

## Masked output

Wherever credgate *shows* a detected value (printed matches, `preview()`,
`scan_pii()` examples, the command line), it masks it:

| Value | Shown as |
|---|---|
| Identifiers (Aadhaar, PAN, mobile, account, card, ...) | last 4 characters kept: `XXXX XXXX 1182`, `XXXXXX234F` |
| Names and addresses | masked in full: `XXXXX XXX` |

Keeping the last 4 characters lets you check which value matched, and stays
within the display limits of UIDAI's masked Aadhaar and PCI DSS (cards). If
your policy allows no part of a value to be shown, mask everything:

```python
credgate.set_mask("full")   # or the environment variable CREDGATE_MASK=full
```

Masking changes only what is displayed; detection and `redact()`
(`[AADHAAR]`, `[PERSON_NAME]`) are unaffected.

## Limits

Names and addresses are found only next to a cue: an honorific (Shri, Smt,
श्रीमती, or "ji" after the name), a relation (S/o, D/o), a field label
(`Name:`, `पता:`) or a house/village keyword followed by a PIN code. A deity
addressed with "ji" ("Ram ji") is taken as a name. A bare 12-digit number may pass
the Aadhaar checksum by chance and is reported with lower confidence.

## Citing credgate

If you use credgate in your work, please cite it, with the version you
used (`credgate.__version__`):

Sikka B, Bais A (2026). *credgate: Detect Indian Personal Data Before It
Reaches Language Models.* Python package version 0.5.0.
https://github.com/dwarpal/credgate

```bibtex
@Manual{credgate,
  title  = {{credgate}: Detect Indian Personal Data Before It Reaches Language Models},
  author = {Bharat Sikka and Aishwarya Bais},
  year   = {2026},
  note   = {Python package version 0.5.0},
  url    = {https://github.com/dwarpal/credgate},
}
```

## Licence

Apache License 2.0.
