Metadata-Version: 2.5
Name: rahasya
Version: 0.0.4
Summary: thirty-four kinds of personal identifier, found by arithmetic: checksummed patterns read in context, no model and no network
Project-URL: Repository, https://github.com/vedicreader/rahasya
Project-URL: Documentation, https://vedicreader.github.io/rahasya/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: checksum,dlp,gdpr,nbdev,ner,pii,privacy,redaction
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Text Processing
Requires-Python: >=3.12
Requires-Dist: fastcore>=2.2.15
Description-Content-Type: text/markdown

# rahasya


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

rahasya (रहस्य, “secret”) tells you whether a piece of text contains personal information, such as an email address, a card number, a passport number or a home address, and can mask it. It does this with regular expressions and the check digits that many identifiers carry, not with a machine-learning model. It needs no network connection and no model files, and it runs in plain Python with only fastcore as a dependency.

It finds 34 kinds of identifier, listed in `PATTERNS`, and people’s names when you ask for them. Everything runs in your own process. rahasya does not send text anywhere, does not write it to disk, and keeps nothing between calls apart from the numbering dictionary you can choose to pass to [`redact`](https://vedicreader.github.io/rahasya/core.html#redact).

On the author’s test corpora, which are not part of this repository, the patterns scored a precision of 1.000 at a recall of 1.000 on a held-out set, and 0.996 on the set they were tuned against. Precision is the share of findings that were real; recall is the share of real identifiers that were found. These are measurements on those corpora, not a guarantee for your text.

## Install

``` sh
uv add rahasya
```

## Checking a text

[`pii_report`](https://vedicreader.github.io/rahasya/core.html#pii_report) scans a text and returns what it found. `has_pii` is True when the text contains at least one identifying kind, `identifying` counts those kinds, and `kinds` counts every kind found. In the example below an email address, a card number and an API key are found:

``` python
r = pii_report('Invoice for ada@example.com. Card 4111 1111 1111 1111. '
                'KEY=sk-abcdefghijklmnopqrstuvwxyz123456')
r.has_pii, r.identifying
```

    (True, {'email': 1, 'card': 1, 'secret': 1})

Every kind counts towards `has_pii` except IP addresses. An IP address is still reported in `kinds`, but a server log full of them is not usually somebody’s private information. `IDENTIFYING` lists the kinds that count.

[`redact`](https://vedicreader.github.io/rahasya/core.html#redact) replaces each finding with a label naming its kind:

``` python
redact('Card 4111 1111 1111 1111 charged to ada@example.com')
```

    'Card [CARD] charged to [EMAIL]'

## Numbers are read in context

Many identifiers are long runs of digits, and so are order numbers, page ids and product codes. rahasya uses two tests to tell them apart. First, an identifier with a check digit must pass it: a card number must pass the Luhn check that every payment card carries, so a sixteen-digit number that fails it is not reported. Second, a number introduced by a word that names it as a reference, such as “Order”, “ISO” or “page”, is not an identifier.

``` python
[pii_report(t).has_pii for t in
 ('Order 4111 1111 1111 1112 shipped',   # fails Luhn
  'Conforms to EN 60601-1',              # a standard, not a ZIP
  'Card 4111 1111 1111 1111')]           # this one is a card
```

    [False, False, True]

`DESIGNATOR` holds those reference words: standards bodies such as `ISO`, `EN` and `RFC`, and words such as `invoice`, `page`, `commit` and `order`, 76 in all. rahasya checks the 48 characters before a match that starts with a digit, and drops the match when one of those words ends just before it.

## Regional identifiers

Nineteen of the 34 kinds are national identifiers. Most carry a checksum that the match must pass. A few have no usable check, such as the UK National Insurance number, or only a weak one, and rahasya finds those only when a label such as “NINO” or “TFN” comes before the number.

| where | kinds |
|----|----|
| India | Aadhaar, PAN, GSTIN |
| Australia | TFN, ABN, Medicare |
| Europe | Dutch BSN, French NIR, Spanish DNI and NIE, Italian codice fiscale, Polish PESEL, Swedish personnummer, Norwegian fødselsnummer, German Steuer-IdNr, UK NINO |
| Asia-Pacific | Singapore NRIC, Thai national ID |
| anywhere | IMEI, the serial number of a mobile phone |

Each check is also exported on its own, such as [`aadhaar_ok`](https://vedicreader.github.io/rahasya/core.html#aadhaar_ok), [`pesel_ok`](https://vedicreader.github.io/rahasya/core.html#pesel_ok) and [`nino_ok`](https://vedicreader.github.io/rahasya/core.html#nino_ok). [`gen_aadhaar`](https://vedicreader.github.io/rahasya/core.html#gen_aadhaar) makes a valid Aadhaar number from a random generator, for test data:

``` python
import random
a = gen_aadhaar(random.Random(0))            # a valid one, for a fixture
a, aadhaar_ok(a), pesel_ok('44051401359'), nino_ok('AB123456C')
```

    ('860487647593', True, True, True)

## Names are opt-in

No pattern can tell a person’s name from a place, a product or a heading, so names are not looked for unless you pass `ner=True`. Even then, rahasya finds only names that follow an honorific such as Mr, Mrs, Dr or Prof. `Dr Charles Babbage` is found and a bare `Ada Lovelace` is not, and names are looked for only in the first 20,000 characters (`NER_CHARS`). The report’s `scanned_ner` field says whether names were looked for, so that “no names found” can be told apart from “names not looked for”.

``` python
(pii_report('Dr Charles Babbage signed it.').has_pii,
 pii_report('Dr Charles Babbage signed it.', ner=True).identifying,
 redact('Dr Charles Babbage signed it.', ner=True))
```

    (False, {'person': 1}, 'Dr [PERSON] signed it.')

## What is here

| name | what it does |
|----|----|
| [`pii_report`](https://vedicreader.github.io/rahasya/core.html#pii_report) | the findings in a text, and whether they make `has_pii` True |
| [`pii_spans`](https://vedicreader.github.io/rahasya/core.html#pii_spans) | each finding as `(start, end, kind, text)`, with overlapping matches resolved in favour of the longest |
| [`redact`](https://vedicreader.github.io/rahasya/core.html#redact) | the text with each finding replaced by `[EMAIL]`, `[CARD]` and so on, or by a mask you choose |
| [`redact_obj`](https://vedicreader.github.io/rahasya/core.html#redact_obj) | [`redact`](https://vedicreader.github.io/rahasya/core.html#redact) applied to every string inside a nested dictionary or list |
| `redact(..., numbered=True)` | labels numbered per value, such as `[EMAIL_1]` and `[EMAIL_2]`; pass a dictionary to keep the numbers across calls |
| [`block_spans`](https://vedicreader.github.io/rahasya/core.html#block_spans) | findings across a list of text blocks, such as the cells of a table, as `(block, start, end, kind, text)` |
| [`fingerprint`](https://vedicreader.github.io/rahasya/core.html#fingerprint) | a short id for a finding that stays the same however the value is spaced or capitalised |
| [`mask_preview`](https://vedicreader.github.io/rahasya/core.html#mask_preview) | a partly hidden form of a finding, such as `a••••@g••••.com`, for showing a person which one is meant |
| [`person_names`](https://vedicreader.github.io/rahasya/core.html#person_names) / [`person_spans`](https://vedicreader.github.io/rahasya/core.html#person_spans) | the names that follow an honorific |
| `PATTERNS` | every kind, as `kind: (regular expression, check function)` |
| `IDENTIFYING` | the kinds that make `has_pii` True |

[`luhn`](https://vedicreader.github.io/rahasya/core.html#luhn), [`aadhaar_ok`](https://vedicreader.github.io/rahasya/core.html#aadhaar_ok), [`pesel_ok`](https://vedicreader.github.io/rahasya/core.html#pesel_ok) and the other check functions are exported too, since a checksum is useful on its own.

## Why there is no model

The author compared two machine-learning detectors with these patterns. Both had lower precision and recall and ran about 200 times slower. A DeBERTa-v3 classifier in ONNX format, which needs about a gigabyte of model weights, did better only on names without an honorific, finding 5 of 8 test names against 2 of 8. A 350-million-parameter encoder found no names at all. These comparisons were run outside this repository.

## Development

The notebooks in `nbs/` are the source, and the Python modules are generated from them.

``` sh
pip install -e .
nbdev-prepare
```
