Metadata-Version: 2.4
Name: deidkit
Version: 0.1.5
Summary: Schema-driven pseudonymization for clinical/tabular datasets: synthetic names, interval-preserving date shifting, multi-stage free-text PII detection, and before/after audit exports.
Author: Meridian Data
License: MIT
Project-URL: Homepage, https://github.com/Meridian-Int/meridian-deid-engine
Project-URL: Documentation, https://github.com/Meridian-Int/meridian-deid-engine#readme
Project-URL: Issues, https://github.com/Meridian-Int/meridian-deid-engine/issues
Keywords: de-identification,pseudonymization,hipaa,phi,ner,clinical,healthcare,privacy
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Topic :: Security
Classifier: Intended Audience :: Healthcare Industry
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=1.3
Requires-Dist: openpyxl>=3.0
Requires-Dist: PyYAML>=5.4
Requires-Dist: SQLAlchemy>=1.4
Requires-Dist: pyodbc>=4.0
Requires-Dist: spacy>=3.4
Provides-Extra: parquet
Requires-Dist: pyarrow>=7.0; extra == "parquet"
Provides-Extra: sql
Requires-Dist: SQLAlchemy>=1.4; extra == "sql"
Requires-Dist: pyodbc>=4.0; extra == "sql"
Provides-Extra: ner
Requires-Dist: spacy>=3.4; extra == "ner"
Provides-Extra: all
Requires-Dist: spacy>=3.4; extra == "all"
Requires-Dist: pyarrow>=7.0; extra == "all"
Requires-Dist: SQLAlchemy>=1.4; extra == "all"
Requires-Dist: pyodbc>=4.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# deidkit

**Schema-driven de-identification for clinical & tabular data.** Point it at your
tables; get back a *usable*, pseudonymized copy plus a full audit of every change.

It replaces names with realistic fakes (in columns *and* inside free text), shifts
dates while preserving every interval, swaps IDs/phones/emails for stable
surrogates, and sends anything uncertain to a human-review sheet instead of
guessing. It is deterministic (same input + same secret ⇒ same output) and
re-runnable.

---

## Contents

1. [Install](#1-install)
2. [Quick start](#2-quick-start)
3. [Inputs — what you feed it](#3-inputs--what-you-feed-it)
4. [What it does to each column](#4-what-it-does-to-each-column)
5. [Outputs — what you get back](#5-outputs--what-you-get-back)
6. [Intermediate outputs — inspect before you trust](#6-intermediate-outputs--inspect-before-you-trust)
7. [Training it — the review → learn loop](#7-training-it--the-review--learn-loop)
8. [Tuning recall & precision](#8-tuning-recall--precision)
9. [Reading from a database (Fabric / Azure SQL)](#9-reading-from-a-database-fabric--azure-sql)
10. [Database connection parameters — exactly what to fill in](#10-database-connection-parameters--exactly-what-to-fill-in)
11. [What to look out for](#11-what-to-look-out-for)
12. [Full option reference (CLI + Python)](#12-full-option-reference-cli--python)
13. [Troubleshooting](#13-troubleshooting)

---

## 1. Install

```bash
pip install deidkit
```

One system-level thing is required **once**, on the machine that will connect:
the Microsoft **ODBC Driver 18 for SQL Server** — the actual database driver that
`pyodbc` talks to:

```bash
# macOS
brew tap microsoft/mssql-release https://github.com/microsoft/homebrew-mssql-release
brew install msodbcsql18
# Linux (Debian/Ubuntu): install Microsoft's "msodbcsql18" package
# Windows: install the "ODBC Driver 18 for SQL Server" MSI
```

Requires Python 3.9+. Confirm it:

```bash
deidkit --version
```

---

## 2. Quick start

Your data is the **Azure SQL Database mirrored into Microsoft Fabric**, so the
run connects to that endpoint, de-identifies the six tables together, and writes
a clean copy out. That's the whole thing:

```bash
deidkit init-secret --out secret.key       # do this once; keep the file private
```

```python
import deidkit as dk
from deidkit.io import read_sql_table

# connection string built from the values in section 10 (nothing writes back)
CONN = (
    "mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>"
    "?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"
)
TABLES = ["tbl_patient", "tbl_encounter", "tbl_notes",
          "tbl_lab", "tbl_imaging", "tbl_medication"]

tables = {t: read_sql_table(CONN, t) for t in TABLES}     # read the DB (read-only)
result = dk.deidentify(
    tables, out="deid_out",                               # de-identify all six together
    secret_file="secret.key",
    lang="es",
    entity_key="patient_id",
    known_id_columns=["patient_id"],
)
print(result.summary)
# -> deid_out/ (ship this)  +  deid_out-PRIVATE/ (keep private — see section 5)
```

That reads the tables from Fabric, never writes back, and produces the two output
folders. Section 9 walks through the connection (plus a database-to-database
variant), and section 10 lists the exact connection values to put in `CONN`.

> **Handed a file export instead of database access?** If you receive the tables
> as files (CSV etc.) rather than a live endpoint, the same run is one CLI line —
> `deidkit run data/` — writing `data_deid/` and `data_deid-PRIVATE/`. See
> section 3.

> **CLI or code — same engine.** Everything below is shown for **both**
> interfaces. The rule of thumb: every CLI `--flag` is a field on the Python
> `Policy` object (e.g. `--mode balanced` ⇄ `Policy(mode="balanced")`,
> `--ignore blood_type` ⇄ `Policy(ignore={"blood_type"})`). Pick whichever fits
> your workflow — a one-off delivery (CLI) or a pipeline/notebook (code).

---

## 3. Inputs — what you feed it

### Your source: the Fabric / Azure SQL database

For this project the input is **your live database** — the Azure SQL Database
mirrored into Microsoft Fabric. deidkit connects to the Fabric **SQL analytics
endpoint** (read-only), reads the six tables directly, de-identifies them, and
writes the clean copy out. It **never writes back** to your database.

deidkit connects on its own — it only needs the connection details for that
endpoint (the SQL analytics endpoint host, the database name, and how to
authenticate). Those go into one connection string; section 9 shows how a run
works and section 10 lists the exact values. Start there.

### Or: a data export as files

If instead you hand over a **data extract as files** (one table per file in a
folder), deidkit reads those too — `.csv`, `.tsv`, or `.json`. One file = one
table, and the filename becomes the table name. CSVs are read as text (so IDs
and date formats aren't mangled) and tolerate non-UTF-8 exports.

```bash
deidkit run data/            # a folder of table files
```

### Two things that matter about your columns

- **`patient_id` is the anchor.** By default `entity_key = patient_id`. Every
  date for a patient is shifted by the *same* offset, and every real value maps
  to the *same* surrogate — **across all tables** — so joins and time-intervals
  survive. If your patient column has a different name, set `--entity-key <col>`.
  All six Meridian tables share `patient_id`, so this works out of the box.
- **Column names drive the plan.** deidkit recognises clinical column names in
  **English and Spanish** (e.g. `note_full`, `result_interpretation`,
  `report_text`, `clinical_justification_pbs`) and classifies each one
  automatically. You can override any decision (section 12). An optional
  **data-dictionary JSON** (`--schema dict.json`) makes the classification
  authoritative instead of heuristic.

---

## 4. What it does to each column

Every column gets exactly one **strategy**. Most are chosen automatically; you can
force any of them with a rule.

| strategy | effect | typical columns |
| --- | --- | --- |
| `passthrough` | left untouched (the safe default) | `sex`, `hemoglobin`, `diagnosis`, `insurer` |
| `date_shift` | move by the per-patient offset — **intervals preserved** | `note_datetime`, `encounter_start/end`, `birth_date` |
| `freetext` | detector scrubs names/IDs/phones/emails/dates **inside the text** | `note_full`, `report_text`, `result_interpretation` |
| `synthetic_name` | a name value → realistic fake name | a `patient_name` / `provider_name` column |
| `identifier` | an ID value → stable shape-preserving surrogate | a raw cédula / document number |
| `age_band` | bucket an age (e.g. 10-yr bands, 90+) | `age_years` |
| `generalize_year` | reduce a date to its year | |
| `redact` | blank the value | |
| `drop` | remove the column entirely | |

Preview the full plan before running anything with `deidkit plan` (section 6).

---

## 5. Outputs — what you get back

A run always produces **two folders**, kept separate on purpose so that zipping
the deliverable can never leak the key:

```
deid_out/                         ← SHIP THIS — de-identified data only
    tbl_patient.csv
    tbl_encounter.csv
    tbl_notes.csv
    tbl_lab.csv
    tbl_imaging.csv
    tbl_medication.csv

deid_out-PRIVATE/                 ← NEVER SHARE — the re-identification key
    deidkit-secret.key            ← the secret (only if it was auto-generated)
    mapping.private.json          ← value → surrogate map (reverses/extends the run)
    deid_audit.PRIVATE.xlsx       ← before/after audit workbook (see section 6)
    deid_audit.PRIVATE_freetext_changes.csv
    review.csv                    ← uncertain detections for a human (see section 7)
```

- **`deid_out/`** — the pseudonymized tables, same structure as the input. This
  is the *only* thing you deliver.
- **`deid_out-PRIVATE/`** — the secret + mapping + audit. Whoever holds these can
  re-identify. Store them somewhere else and never include them in a delivery.

**Output format** defaults to CSV; use `--format parquet|tsv|json` (CLI) or
`fmt=` (Python) to change it. Reading from a DB and want the result written back
to a DB? See `run-db` in section 9.

---

## 6. Intermediate outputs — inspect before you trust

Don't run blind. deidkit gives you three ways to look *between* input and final
output.

### `deidkit plan` — the decision for every column (changes nothing)

```bash
deidkit plan data/
```
```
table            column                           strategy         decided_by
----------------------------------------------------------------------------------------
tbl_notes        note_full                        freetext         heuristic:freetext  *
tbl_lab          result_interpretation            freetext         heuristic:freetext  *
tbl_lab          order_datetime                   date_shift       heuristic:date  *
tbl_patient      sex                              passthrough      default
```
Columns marked `*` will change. If something is wrong, fix it with `--ignore`,
`--only`, `--name-column`, or a rule, then re-run `plan` until it's right.

*In code:* `deid.plan_dataframe(df, "tbl_notes")` returns the same
`{column: PlanItem}` decision for one table.

### `deidkit scan` — what the free-text detector sees on a snippet

```bash
deidkit scan --text "Paciente Juan Pérez, CC 1.032.456.789, tel 300 123 4567"
```
```
accepted  type         conf  detectors              text
--------------------------------------------------------------------------------
REPLACE   PERSON       0.99  context+gazetteer      'Juan Pérez'
REPLACE   NATIONAL_ID  0.99  regex                  '1.032.456.789'
REPLACE   PHONE        0.99  regex                  '300 123 4567'
```
`REPLACE` = will be scrubbed. `review` = queued for a human instead.

*In code:* `dk.Detector(lang="es").detect(text)` returns the same detections.

### The audit workbook — evidence of every change

`deid_out-PRIVATE/deid_audit.PRIVATE.xlsx` has:

| sheet | what it shows |
| --- | --- |
| **Summary** | counts by entity type and column |
| **Field plan** | the strategy chosen for every column, and why |
| **Free-text changes** | every before → after edit inside notes/reports, and which detector fired |
| **Structured changes** | before → after for structured columns (dates, IDs, names) |
| **Mapping** | the private value → surrogate map |
| **Review queue** | detections that were *not* auto-replaced |

**Verify intervals survived** (they should, exactly):

```python
import pandas as pd
enc = pd.read_csv("deid_out/tbl_encounter.csv")
los = pd.to_datetime(enc["encounter_end"]) - pd.to_datetime(enc["encounter_start"])
print(los.describe())   # identical to the source — only absolute dates moved
```

---

## 7. Training it — the review → learn loop

deidkit **never guesses** on an ambiguous name. Anything it detects but isn't sure
about (for example a lone surname that is also a common word) is left untouched
and listed in **`review.csv`** for a person to rule on. Confirm or reject those,
and the next run does better — deterministically, with no model.

`review.csv` holds only that **uncertain tail**; the confident detections are
already scrubbed in the output and never appear here. It is written to the
**private** folder on every run (`deid_out-PRIVATE/review.csv`) and contains raw
candidate text, so treat it as sensitive — never ship it.

### What the reviewer does

Open `deid_out-PRIVATE/review.csv` (Excel, Google Sheets, any editor). Each row is
one uncertain candidate, with a `context` snippet so you can judge it. Fill in the
**first column only** — `decision(y/n)` — and save:

| decision(y/n) | entity_type | candidate | context | table | column | row_id | confidence | detectors |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `y` | PERSON | `Bello` | `…paciente Bello refiere dolor…` | tbl_notes | note_full | 87 | 0.6 | context |
| `n` | PERSON | `Cruz` | `…la Cruz Roja atendió…` | tbl_notes | note_full | 42 | 0.55 | gazetteer |
| *(blank)* | PERSON | `Sur` | `…zona Sur del hospital…` | tbl_notes | note_full | 91 | 0.5 | gazetteer |

- **`y`** — yes, this **is** a real person's name (it should be redacted).
- **`n`** — no, it's a **false positive** — not a name (e.g. "Cruz" in "Cruz Roja").
- **blank** — undecided; that row is skipped and simply stays for a later pass.

That is the whole reviewer job: read the `context`, type `y` or `n`, save the
file. (Lenient spellings work too: `yes` / `1` / `true` = yes; `no` / `0` /
`false` / `fp` = no.)

### The four steps

```bash
# 1. Run. Uncertain detections are written to deid_out-PRIVATE/review.csv.
deidkit run <input> deid_out

# 2. A reviewer opens deid_out-PRIVATE/review.csv and marks each row y / n (above).

# 3. Fold those decisions into a "learn" directory.
deidkit learn --review deid_out-PRIVATE/review.csv --learn-dir learned/

# 4. Re-run pointing at what it learned: confirmed names are now caught
#    automatically, and rejected candidates are suppressed.
deidkit run <input> deid_out --learn-dir learned/
```

Step 3 appends to two plain-text files that `learned/` **accumulates** across
cycles (so it keeps improving):

- `learned_names.txt` — words from every `y` → added to the name list (**higher recall**),
- `learned_stoplist.txt` — words from every `n` → added to the stoplist (**higher precision**).

Both are one token per line and human-editable. *In code*, load them with
`Policy(extra_name_files=["learned/learned_names.txt"],
extra_stoplist_files=["learned/learned_stoplist.txt"])`.

**Good to know:**
- `review.csv` is produced on **every** run — `--learn-dir` is only needed to
  *load* what was learned (steps 3–4), not to create the queue.
- Learning is **token-level**: marking "María Gómez" as `y` teaches both `maría`
  and `gómez` (lowercased, accent-folded), so they are caught elsewhere too.
- The loop tunes the **borderline** cases. A name the detector missed entirely
  will not appear in `review.csv` — for those, use the known-PHI columns below.

### Two stronger levers (deterministic, near-100% recall)

- **Known-PHI columns** — if you already know a row's real name or ID (from a
  structured column), force-scrub those exact strings from *that row's* free text.
  No guessing:
  ```bash
  # CLI
  deidkit run data/ deid_out \
      --known-name-column patient_name \
      --known-id-column patient_id
  ```
  ```python
  # code
  dk.Policy(known_name_columns=["patient_name"], known_id_columns=["patient_id"])
  ```
- **Extend the name lists** — feed your own gazetteer / stoplist:
  ```bash
  # CLI
  deidkit run data/ deid_out \
      --extra-name-file my_surnames.txt \
      --extra-stoplist-file never_redact.txt
  ```
  ```python
  # code
  dk.Policy(extra_name_files=["my_surnames.txt"],
            extra_stoplist_files=["never_redact.txt"])
  ```

---

## 8. Tuning recall & precision

**By default deidkit auto-redacts** (`balanced` mode) **with the statistical NER
stage on**, so it catches names automatically — including novel names with no
surrounding cue — without any flags. The only setup is downloading the NER model
once (it's ~500 MB, so it can't ship inside the pip package):

```bash
deidkit download-model      # one-time; grabs es_core_news_lg. spaCy is already
                            # installed with deidkit — no extra install, no brackets.
```

That's it — `deidkit run <input> deid_out` then uses NER + auto-redact. If you
never download the model, deidkit still runs on the deterministic rules and prints
one clear warning; it never fails, but recall on bare/novel names is lower.

Knobs, only if you want to change the defaults:

| Lever | What it does |
| --- | --- |
| `--mode balanced` (default) | auto-redact confident detections, incl. single NER hits |
| `--mode conservative` | stricter — only clear names; send the ambiguous tail to `review.csv` |
| `--mode aggressive` | highest recall (also lone dictionary tokens); more to review |
| `--no-ner` | turn the NER stage off (deterministic rules only) |
| `--spacy-model NAME` | use a different spaCy model (default `es_core_news_lg`) |
| `--id-checksums DNI,NIE` | catch *bare* national IDs (no cue word) with near-zero false positives |
| `--name-style tagged\|placeholder\|token\|realistic` | how fakes look — `tagged` = "Juan Gómez [SINTÉTICO]" (obvious); `realistic` = unmarked |
| `--no-medical-vocab` | (advanced) stop treating ~21k drug/disease terms as safe non-names |

Work the `review.csv` queue for whatever the automatic pass leaves uncertain — the
audit is how you prove the result.

*In code*, the same knobs are `Policy` fields:

```python
dk.Policy(mode="balanced", spacy_model="es_core_news_lg",
          id_checksums=["DNI", "NIE"], name_style="tagged")
```

---

## 9. Reading from a database (Fabric / Azure SQL)

### How the connection works

deidkit talks only to the Fabric **SQL analytics endpoint** — it never logs into
the Azure portal or the production database. The setup is the standard Fabric
mirror:

1. The Azure SQL Database is mirrored into a Microsoft Fabric workspace, and the
   mirrored database is shared **read-only** — *Read* + *Read all SQL analytics
   endpoint data*, nothing else. No write or admin access is ever required.
2. deidkit is pointed at that endpoint with a connection string built from four
   values (see section 10): the SQL analytics endpoint, the database name, the
   authentication method, and the workspace URL.
3. It connects to the read-only endpoint, reads the six tables, de-identifies
   them, and writes out the clean copy. It **never writes back** — it only reads.

All that's needed is read-only access to the shared endpoint plus those values.

### Then it runs — two patterns

The database drivers ship with deidkit (section 1). Pick based on where you want
the output.

### Pattern 1 — Database → files (recommended, simplest)

Read the tables from Fabric into memory, de-identify them together, write clean
files. Python only:

```python
import deidkit as dk
from deidkit.io import read_sql_table

CONN = (
    "mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>"
    "?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"
)

TABLES = ["tbl_patient", "tbl_encounter", "tbl_notes",
          "tbl_lab", "tbl_imaging", "tbl_medication"]

# Read all tables so they share one map + one set of per-patient date offsets
tables = {t: read_sql_table(CONN, t) for t in TABLES}

result = dk.deidentify(
    tables,
    out="deid_out",
    secret_file="secret.key",
    lang="es",
    entity_key="patient_id",
    known_id_columns=["patient_id"],
)
print(result.summary)
# -> deid_out/*.csv  (ship)   +   deid_out-PRIVATE/  (keep)
```

### Pattern 2 — Database → database (`run-db` / `deidentify_database`)

Write the de-identified result into a **separate output database that mirrors the
source structure** (same tables, columns, types). You must **provision the output
DB yourself** — deidkit refuses to run without it and never edits the source.

```bash
# CLI
deidkit run-db \
    --source "mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive" \
    --out-db "postgresql://deid_user:PASSWORD@your-host:5432/deid_db" \
    --policy examples/policy.meridian.yaml \
    --secret-file secret.key
```

```python
# code
import deidkit as dk

dk.deidentify_database(
    source="mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>"
           "?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive",
    out_db="postgresql://deid_user:PASSWORD@your-host:5432/deid_db",
    secret_file="secret.key",
    lang="es",
    entity_key="patient_id",
    known_id_columns=["patient_id"],
)
```

- `--out-db` / `out_db` is **required** and must differ from the source.
- A column whose de-identified value no longer fits the source type (e.g. an age
  turned into a band) is widened to text; every other column keeps its structure.
- The secret + mapping + audit go to `./deidkit-db-PRIVATE/` (never the output DB).

---

## 10. Database connection parameters — exactly what to fill in

### The four connection values

Once the mirrored database is shared read-only, gather these **four values** (they
are exactly what the Fabric "share" step produces):

| # | Item | Example | Used for |
| --- | --- | --- | --- |
| 1 | **SQL Analytics Endpoint** (the host) | `abc123....datawarehouse.fabric.microsoft.com` | connecting |
| 2 | **Database name** | `ProductionMirror` | connecting |
| 3 | **Authentication method** — Microsoft Entra ID *or* SQL Login | `Microsoft Entra ID` | connecting |
| 4 | **Fabric Workspace URL** | `https://app.fabric.microsoft.com/groups/...` | locating & confirming the shared database |

Items 1–3 go into the connection string; item 4 (the workspace URL) only locates
and confirms the shared database in Fabric — it is not part of the connection
string.

### The connection string

Connections are **SQLAlchemy URLs**. For the Fabric source, the shape is:

```
mssql+pyodbc://@<HOST>/<DATABASE>?driver=<DRIVER>&authentication=<AUTH>
```

| Placeholder | Filled from | Example |
| --- | --- | --- |
| `<HOST>` | item 1 — the SQL analytics endpoint | `abc123....datawarehouse.fabric.microsoft.com` |
| `<DATABASE>` | item 2 — the database name | `ProductionMirror` |
| `<AUTH>` | item 3 — the authentication method | `ActiveDirectoryInteractive` |
| `<DRIVER>` | installed locally (section 1), not a Fabric value | `ODBC+Driver+18+for+SQL+Server` |

> Note the `@` before `<HOST>`: `...pyodbc://@abc123...`. With Entra ID auth there
> is no username/password in the URL — the `@` with an empty user is intentional.
> Spaces in the driver name are written as `+` (URL-encoded).

**Authentication options for `<AUTH>`:**

- **Microsoft Entra ID, interactive** (opens a browser to sign in) — best for a
  person running it manually:
  ```
  ...?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive
  ```
- **Microsoft Entra ID, service principal** (unattended jobs / servers):
  ```
  mssql+pyodbc://<CLIENT_ID>:<CLIENT_SECRET>@<HOST>/<DATABASE>?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryServicePrincipal
  ```
- **SQL login** (only if they gave you a SQL username/password instead of Entra):
  ```
  mssql+pyodbc://<USER>:<PASSWORD>@<HOST>/<DATABASE>?driver=ODBC+Driver+18+for+SQL+Server
  ```

**Install the ODBC driver** (`<DRIVER>` above must match what’s installed):

```bash
# macOS
brew tap microsoft/mssql-release https://github.com/microsoft/homebrew-mssql-release
brew install msodbcsql18

# Debian/Ubuntu — follow Microsoft’s "ODBC Driver 18 for SQL Server" apt instructions
# Windows — install the "ODBC Driver 18 for SQL Server" MSI
```

**The output DB URL** (Pattern 2 only) is any SQLAlchemy URL for a database
**you created and can write to**:

| Target | URL shape |
| --- | --- |
| PostgreSQL | `postgresql://user:password@host:5432/deid_db` |
| Azure SQL / SQL Server | `mssql+pyodbc://user:password@host/deid_db?driver=ODBC+Driver+18+for+SQL+Server` |
| Local SQLite (quick test) | `sqlite:///deid_local.db` |

Quick connectivity check before a full run:

```python
from sqlalchemy import create_engine, text
CONN = "mssql+pyodbc://@<HOST>/<DATABASE>?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"
with create_engine(CONN).connect() as c:
    print(c.execute(text("SELECT TOP 1 patient_id FROM tbl_patient")).fetchone())
```

---

## 11. What to look out for

- **Is `patient_id` a raw ID or already hashed?** By default it's passed through
  (it's the join key). If it's actually the **cédula / national ID**, that's a
  leak — add `FieldRule("patient_id", "identifier")` (or uncomment the rule in
  `examples/policy.meridian.yaml`). Surrogating it still preserves joins.
- **Filenames & paths can hide PHI.** Columns like `result_file_name` and
  `image_relative_uri` often embed a name or MRN. They pass through by default —
  `redact` them if you don't need them.
- **Keep the secret and mapping private.** They *are* the re-identification key.
  Never put `*-PRIVATE/` in a delivery. Losing the secret means you can't re-run
  consistently; leaking it undoes the de-identification.
- **Determinism needs the same secret.** Pass the same `--secret-file` every time,
  or every run produces different surrogates.
- **No detector is perfect.** Pseudonymization reduces but does not eliminate
  re-identification risk (per NIST SP 800-188 / HIPAA expert-determination).
  Review the audit, consider quasi-identifier combinations (age + stratum +
  location …), and treat outputs accordingly.

---

## 12. Full option reference (CLI + Python)

### CLI commands

| Command | Purpose |
| --- | --- |
| `deidkit init-secret --out secret.key` | generate a keying secret (chmod 600) |
| `deidkit plan <input>` | show the strategy for every column; change nothing |
| `deidkit run <input> [out]` | de-identify files → files |
| `deidkit run-db --source <url> --out-db <url>` | de-identify a DB → a separate DB |
| `deidkit scan --text "…"` | show detections on a snippet (tuning/debug) |
| `deidkit learn --review review.csv --learn-dir learned/` | fold review decisions into learned dicts |
| `deidkit map --mapping mapping.private.json` | dump the private value→surrogate map |

Run `deidkit <command> --help` for any command's flags.

### CLI flags shared by `run`, `run-db`, `plan`, `scan`

| Flag | Meaning |
| --- | --- |
| `--policy FILE` | YAML policy file; CLI flags override it |
| `--lang {es,en,multi}` | language for names/detection (`multi` = ES+EN) |
| `--mode {conservative,balanced,aggressive}` | free-text detection aggressiveness |
| `--name-style {tagged,placeholder,token,realistic}` | how fake names look (default `tagged` = obviously synthetic) |
| `--entity-key COL` | column that groups date shifts (default `patient_id`) |
| `--date-max-days N` | date-shift window in days (default 365; intervals preserved regardless) |
| `--ignore a,b,c` | never transform these columns |
| `--only a,b` | transform *only* these columns |
| `--name-column COL` | force a column to synthetic-name treatment (repeatable) |
| `--known-name-column COL` | column holding a known name to force-scrub from that row’s text (repeatable) |
| `--known-id-column COL` | column holding a known ID to force-scrub from that row’s text (repeatable) |
| `--known-name "NAME"` | a known name (e.g. clinic staff) scrubbed from all text (repeatable) |
| `--id-checksums LIST` | checksum-validated bare-ID detection: `DNI,NIE` (ES), `CPF,CNPJ,CNS` (BR), `NHS` (UK), `CURP,RFC` (MX), `SSN`, or `ALL` |
| `--spacy-model NAME` | enable the spaCy NER stage (e.g. `es_core_news_lg`) |
| `--names-token` | use `[PERSON_x]` tokens instead of realistic fake names |
| `--no-date-text` | do not shift date strings found inside free text |
| `--no-medical-vocab` | do not load the bundled medical safe-vocabulary |
| `--extra-name-file FILE` / `--extra-stoplist-file FILE` | merge extra gazetteer / stoplist files (repeatable) |
| `--learn-dir DIR` | auto-load `learned_names.txt` + `learned_stoplist.txt` from a previous `learn` |

### `run`-specific flags

| Flag | Meaning |
| --- | --- |
| `input` / `out` (positional) | input path/glob/dir, and output dir (default `<input>_deid`) |
| `--secret-file FILE` / `--secret STR` | the keying secret (or set `DEIDKIT_SECRET`) |
| `--schema FILE` | data-dictionary JSON (auto-detected from the input folder if omitted) |
| `--mapping FILE` | private mapping JSON (default `<out>-PRIVATE/mapping.private.json`) |
| `--report FILE` | audit workbook path (default `<out>-PRIVATE/deid_audit.PRIVATE.xlsx`) |
| `--format {csv,parquet,tsv,json}` | output format (default `csv`) |

### `run-db`-specific flags

| Flag | Meaning |
| --- | --- |
| `--source URL` | source DB SQLAlchemy URL (read-only) — **required** |
| `--out-db URL` | output DB SQLAlchemy URL you created — **required**, must differ from source |
| `--tables a,b,c` | limit to these tables (default: all) |
| `--out-schema NAME` | schema/namespace in the output DB |
| `--if-exists {fail,replace,append}` | behaviour if an output table exists (default `fail`) |
| `--private-dir DIR` | where secret+mapping+audit go (default `./deidkit-db-PRIVATE`) |

### Python `Policy` fields

`entity_key` (default `"patient_id"`), `date_max_days` (365), `lang` (`"es"`),
`mode` (`"balanced"`), `name_style` (`"tagged"`), `spacy_model`
(`"es_core_news_lg"`, NER on by default), `shift_dates_in_text`
(True), `only`, `ignore`, `rules` (`list[FieldRule]`), `spacy_model`,
`known_name_columns`, `known_id_columns`, `extra_known_names`,
`known_token_min_len` (5), `id_checksums`, `use_medical_vocab` (True),
`extra_name_files`, `extra_stoplist_files`.

`FieldRule(column, strategy, table=None, options={})` — `column` and `table`
accept `*` globs; the first matching rule wins.

### Python functions

```python
import deidkit as dk

# one call: files / DataFrame / dict of DataFrames -> Result
dk.deidentify(data, out=None, *, policy=None, schema=None, secret=None,
              secret_file=None, mapping=None, report=None, private_dir=None,
              fmt=None, table="table", **policy_kwargs)

# one DataFrame in, one clean DataFrame out
dk.deidentify_dataframe(df, table="table", **kwargs)

# database -> a separate database (mirrors source structure)
dk.deidentify_database(source, out_db, *, tables=None, schema=None,
                       out_schema=None, if_exists="fail", policy=None,
                       secret=None, secret_file=None, mapping=None,
                       private_dir=None, report=None, **policy_kwargs)
```

`deidentify(...)` returns a `Result` with `.tables`, `.table` (the single
DataFrame when one went in), `.summary`, `.out`, `.report_path`, `.mapping_path`,
`.private_dir`, and `.secret` (set only when a secret was auto-generated — save it
to stay reproducible).

**Full control** — drive many tables through one shared map + one set of
per-patient date offsets, and write the audit yourself:

```python
import deidkit as dk

policy = dk.Policy(lang="es", entity_key="patient_id",   # mode defaults to balanced + NER
                   known_id_columns=["patient_id"], ignore={"blood_type"})

deid = dk.Deidentifier(
    policy,
    secret_file="secret.key",                              # or secret="…", or $DEIDKIT_SECRET
    mapping_path="deid_out-PRIVATE/mapping.private.json",  # load+update for consistency
    # dict_index=dk.load_dictionary("dictionary.json"),   # optional data dictionary
)

tables = dk.io.read_tables("data/")          # {name: DataFrame}
clean = deid.run_dataset(tables)             # {name: DataFrame}, all sharing state
# or a single table:  deid.run_table(df, table="tbl_notes")

deid.save_mapping()
deid.write_report("deid_out-PRIVATE/deid_audit.PRIVATE.xlsx")
deid.write_review("deid_out-PRIVATE/review.csv")
print(deid.summary())
```

---

## 13. Troubleshooting

| Symptom | Fix |
| --- | --- |
| `ModuleNotFoundError: pyodbc` / SQLAlchemy | Reinstall: `pip install --force-reinstall deidkit` (the DB drivers ship with it) |
| `Data source name not found` / driver error | Install **ODBC Driver 18 for SQL Server** and make the `driver=` value match exactly (section 10) |
| Browser never opens for `ActiveDirectoryInteractive` | Run from an interactive session; for servers use `ActiveDirectoryServicePrincipal` |
| `cannot reach the output database` (run-db) | Create the empty output DB first and pass a reachable `--out-db` URL |
| A column was changed that shouldn’t be | Add it to `--ignore`, or add a `FieldRule(..., "passthrough")`; confirm with `deidkit plan` |
| A free-text name slipped through | Try `--mode balanced`, enable `--spacy-model`, or add the name via the review→learn loop (section 7) |
| Non-UTF-8 CSV export | Handled automatically (deidkit falls back to latin-1/cp1252) |
| Different output each run | You’re not pinning the secret — pass the same `--secret-file` every time (section 2) |

---

## License

MIT.
