Metadata-Version: 2.5
Name: rahasya
Version: 0.0.6
Summary: thirty-four kinds of personal identifier, found by arithmetic: checksummed patterns read in context, no model and no network
Project-URL: Repository, https://github.com/vedicreader/rahasya
Project-URL: Documentation, https://vedicreader.github.io/rahasya/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: checksum,dlp,gdpr,nbdev,ner,pii,privacy,redaction
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Text Processing
Requires-Python: >=3.12
Requires-Dist: fastcore>=2.2.15
Description-Content-Type: text/markdown

# rahasya


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

rahasya (रहस्य, “secret”) tells you whether a text contains personal information, such as an email address, a card number, a passport number or a home address. It can also mask that information. It uses regular expressions and the check digits that many identifiers carry. It does not use a machine-learning model. It needs no network connection. It needs no model files. It runs in plain Python and depends only on fastcore.

rahasya finds 34 kinds of identifier, listed in `PATTERNS`. It finds people’s names when you ask for them.

Everything runs in your own process. rahasya does not send text anywhere and does not write it to disk. It keeps no state between calls, except the numbering dictionary that you can pass to [`redact`](https://vedicreader.github.io/rahasya/core.html#redact).

The author measured the patterns on test corpora that are not part of this repository. Precision was 1.000 and recall was 1.000 on a held-out set. Precision was 0.996 on the set that the author tuned the patterns on. Precision is the share of findings that were real. Recall is the share of real identifiers that were found. These numbers describe those corpora. They do not guarantee the same results on your text.

## Install

``` sh
uv add rahasya
```

## Checking a text

[`pii_report`](https://vedicreader.github.io/rahasya/core.html#pii_report) scans a text and returns what it found. `has_pii` is True when the text contains at least one identifying kind. `identifying` maps each identifying kind to its finding count. `kinds` maps every detected kind to its finding count. The example below finds an email address, a card number and an API key:

``` python
r = pii_report('Invoice for ada@example.com. Card 4111 1111 1111 1111. '
                'KEY=sk-abcdefghijklmnopqrstuvwxyz123456')
r.has_pii, r.identifying
```

    (True, {'email': 1, 'card': 1, 'secret': 1})

Every kind counts towards `has_pii` except IP addresses. `IDENTIFYING` lists the kinds that count. An IP address is still reported in `kinds`. A server log full of IP addresses is not usually private information about a person.

[`redact`](https://vedicreader.github.io/rahasya/core.html#redact) replaces each finding with a label that names its kind:

``` python
redact('Card 4111 1111 1111 1111 charged to ada@example.com')
```

    'Card [CARD] charged to [EMAIL]'

## Numbers are read in context

Many identifiers are long runs of digits. Order numbers, page ids and product codes are also long runs of digits. rahasya uses two tests to tell them apart.

First, an identifier with a check digit must pass the check. A card number must pass the Luhn check that every payment card number passes. rahasya does not report a sixteen-digit number that fails it.

Second, a number that follows a word that marks it as a reference, such as “Order”, “ISO” or “page”, is not an identifier.

``` python
[pii_report(t).has_pii for t in
 ('Order 4111 1111 1111 1112 shipped',   # fails Luhn
  'Conforms to EN 60601-1',              # a standard, not a ZIP
  'Card 4111 1111 1111 1111')]           # this one is a card
```

    [False, False, True]

`DESIGNATOR` holds 76 reference words. They include standards bodies such as `ISO`, `EN` and `RFC`, and words such as `invoice`, `page`, `commit` and `order`. For a match that starts with a digit, rahasya reads the 48 characters before the match. It drops the match when one of these words ends just before it.

## Regional identifiers

Nineteen of the 34 kinds are national identifiers. For most of them, a match must pass a check. A few kinds have no usable check, such as the UK National Insurance number. Others have only a weak check. rahasya finds these kinds only when a label such as “NINO” or “TFN” comes before the number.

| where | kinds |
|----|----|
| India | Aadhaar, PAN, GSTIN |
| Australia | TFN, ABN, Medicare |
| Europe | Dutch BSN, French NIR, Spanish DNI and NIE, Italian codice fiscale, Polish PESEL, Swedish personnummer, Norwegian fødselsnummer, German Steuer-IdNr, UK NINO |
| Asia-Pacific | Singapore NRIC, Thai national ID |
| anywhere | IMEI, the serial number of a mobile phone |

Each check is also exported, for example [`aadhaar_ok`](https://vedicreader.github.io/rahasya/core.html#aadhaar_ok), [`pesel_ok`](https://vedicreader.github.io/rahasya/core.html#pesel_ok) and [`nino_ok`](https://vedicreader.github.io/rahasya/core.html#nino_ok). [`gen_aadhaar`](https://vedicreader.github.io/rahasya/core.html#gen_aadhaar) takes a `random.Random` and returns a valid Aadhaar number for test data:

``` python
import random
a = gen_aadhaar(random.Random(0))
a, aadhaar_ok(a), pesel_ok('44051401359'), nino_ok('AB123456C')
```

    ('860487647593', True, True, True)

## Names are opt-in

No pattern can tell a person’s name from a place, a product or a heading. rahasya looks for names only when you pass `ner=True`. It then finds only names that follow an honorific such as Mr, Mrs, Dr or Prof. `Dr Charles Babbage` is found and a bare `Ada Lovelace` is not.

rahasya looks for names only in the first 20,000 characters (`NER_CHARS`) of each scanned part of the text. A text of up to 200,000 characters is one part.

The `scanned_ner` field of the report says whether rahasya looked for names. It separates “no names found” from “names not looked for”.

``` python
(pii_report('Dr Charles Babbage signed it.').has_pii,
 pii_report('Dr Charles Babbage signed it.', ner=True).identifying,
 redact('Dr Charles Babbage signed it.', ner=True))
```

    (False, {'person': 1}, 'Dr [PERSON] signed it.')

## What is here

| name | what it does |
|----|----|
| [`pii_report`](https://vedicreader.github.io/rahasya/core.html#pii_report) | the findings in a text, their counts, and `has_pii` |
| [`pii_spans`](https://vedicreader.github.io/rahasya/core.html#pii_spans) | each finding as `(start, end, kind, text)`. The longest of two overlapping matches wins. |
| [`redact`](https://vedicreader.github.io/rahasya/core.html#redact) | the text with each finding replaced by `[EMAIL]`, `[CARD]` and so on, or by a mask you choose |
| [`redact_obj`](https://vedicreader.github.io/rahasya/core.html#redact_obj) | [`redact`](https://vedicreader.github.io/rahasya/core.html#redact) applied to every string inside a nested dictionary or list |
| `redact(..., numbered=True)` | labels numbered per value, such as `[EMAIL_1]` and `[EMAIL_2]`. A dictionary passed as `numbered` keeps the numbers across calls. |
| [`block_spans`](https://vedicreader.github.io/rahasya/core.html#block_spans) | findings across a list of text blocks, such as the cells of a table, as `(block, start, end, kind, text)` |
| [`fingerprint`](https://vedicreader.github.io/rahasya/core.html#fingerprint) | a 16-character id for a finding. The id does not change with spacing or capitalisation. |
| [`mask_preview`](https://vedicreader.github.io/rahasya/core.html#mask_preview) | a partly hidden finding, such as `a••••@g••••.com`, to show a person which finding is meant |
| [`person_names`](https://vedicreader.github.io/rahasya/core.html#person_names) / [`person_spans`](https://vedicreader.github.io/rahasya/core.html#person_spans) | the names that follow an honorific |
| `PATTERNS` | every kind, as `kind: (regular expression, check function)` |
| `IDENTIFYING` | the kinds that make `has_pii` True |

[`luhn`](https://vedicreader.github.io/rahasya/core.html#luhn), [`nhs_ok`](https://vedicreader.github.io/rahasya/core.html#nhs_ok), [`aadhaar_ok`](https://vedicreader.github.io/rahasya/core.html#aadhaar_ok), [`pesel_ok`](https://vedicreader.github.io/rahasya/core.html#pesel_ok) and the other check functions are also exported. The tables behind them are exported too, such as `DNI_LETTERS` and `NRIC_LETTERS`, with [`luhn_digits`](https://vedicreader.github.io/rahasya/core.html#luhn_digits). A test can use them to build a number that passes. `COMPILED` holds every pattern in compiled form, which [`pii_spans`](https://vedicreader.github.io/rahasya/core.html#pii_spans) uses.

## Why there is no model

The author compared two machine-learning detectors with these patterns. Both had lower precision and lower recall. Both ran about 200 times slower. One detector was a DeBERTa-v3 classifier in ONNX format, which needs about a gigabyte of model weights. It did better only on names without an honorific. It found 5 of 8 test names, and rahasya found 2 of 8. The other detector was an encoder with 350 million parameters. It found no names. The author ran these comparisons outside this repository.

## Development

The notebooks in `nbs/` are the source, and the Python modules are generated from them.

``` sh
pip install -e .
nbdev-prepare
```
