Metadata-Version: 2.4
Name: health-deid
Version: 1.0.0
Summary: A local, auditable, and resumable clinical-text de-identification pipeline.
Keywords: clinical-nlp,de-identification,natural-language-processing,phi,research-software
Author: Omid Jafari
License-Expression: AGPL-3.0-or-later
License-File: LICENSE
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Framework :: Flask
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Dist: boto3[crt]>=1.43.11,<2
Requires-Dist: faker>=37.12.0,<38
Requires-Dist: filelock>=3.20.0,<4
Requires-Dist: flask>=3.1.2,<4
Requires-Dist: ftfy>=6.3.1,<7
Requires-Dist: polars>=1.40.1,<2
Requires-Dist: pydantic>=2.13.4,<3
Requires-Dist: pyyaml>=6.0.3,<7
Requires-Dist: typer>=0.25.1,<1
Requires-Python: >=3.12, <3.15
Project-URL: Homepage, https://github.com/omid-jf/Health-DeID
Project-URL: Documentation, https://omid-jf.github.io/Health-DeID/
Project-URL: Issues, https://github.com/omid-jf/Health-DeID/issues
Project-URL: Repository, https://github.com/omid-jf/Health-DeID
Description-Content-Type: text/markdown

# Health-DeID

[![CI](https://github.com/omid-jf/Health-DeID/actions/workflows/ci.yml/badge.svg)](https://github.com/omid-jf/Health-DeID/actions/workflows/ci.yml)
[![Codecov](https://codecov.io/gh/omid-jf/Health-DeID/graph/badge.svg)](https://codecov.io/gh/omid-jf/Health-DeID)
[![PyPI](https://img.shields.io/pypi/v/health-deid.svg)](https://pypi.org/project/health-deid/)
![Python](https://img.shields.io/badge/Python-3.12%20%7C%203.13%20%7C%203.14-blue)
![License](https://img.shields.io/badge/license-AGPLv3-blue)

Health-DeID is local research software for auditable, resumable de-identification of
clinical text and selected structured fields. Its desktop web UI, CLI, and Python API
all use the same SQLite run database.

> ⚠️ **Important:**
> Health-DeID has not been validated or approved for diagnosis, treatment, clinical decisions, or as a guarantee of legal or regulatory compliance. Validate it for your data and use case before sharing output.

## Features

- Imports JSONL or Parquet while preserving input order and selected metadata.
- Detects PHI with AWS Comprehend Medical, Claude Sonnet 4.6 on Amazon Bedrock,
  local rules, or a combination.
- Retains, redacts, generalizes, date-shifts, or replaces each PHI category.
- Generates replacements with Faker or a custom list, consistently by entity, record,
  or occurrence.
- Applies the same replacement policy to unstructured text and mapped structured fields.
- Optionally validates de-identified text with GPT-OSS Safeguard 120B and includes a human review user interface.
- Recovers interrupted work and retains completed backend requests.
- Creates revised runs that reuse detector results when the input and detector setup
  are identical, while rerunning rules and downstream de-identification.
- Exports explicitly selected columns to JSONL or Parquet.

## Requirements

- Python 3.12 through 3.14
- AWS credentials and access to the configured services for AWS-backed runs

## Installation

```bash
pip install health-deid
health-deid --version
```

AWS services use boto3's standard credential and region resolution; credentials are not
entered or stored in the UI.

## Start with the UI

```bash
health-deid ui --reviewer "Reviewer Name" --runs-dir runs
```

Open one run directly:

```bash
health-deid ui runs/RUN_ID --reviewer "Reviewer Name"
```

## CLI

The CLI intentionally has eight commands:

```bash
health-deid check config.yaml
health-deid run config.yaml
health-deid status runs/RUN_ID
health-deid resume runs/RUN_ID
health-deid retry runs/RUN_ID --failed
health-deid revise runs/RUN_ID revised.yaml --reason "change date handling"
health-deid export runs/RUN_ID final.parquet \
  --mode ready_only --column record_id --column final_text
health-deid ui --reviewer "Reviewer Name"
```

`check` is a precheck: it validates setup and shows an estimated AWS cost without
making paid model calls. After processing starts, status and UI usage cards show actual costs.

## Run data

Normal operation creates:

```text
runs/<run-id>/
├── run.sqlite
└── exports/
```

The database contains source clinical text, identifiers, backend responses, findings,
transformations, review decisions, usage, errors, and export history. Treat the complete
run directory as PHI even after creating a de-identified export.

The processing sequence is:

```text
input → detection and rules → draft de-identification → validation → review → final de-identification
```

## Configuration

A minimal Comprehend configuration is:

```yaml
config_version: 1
run:
  name: example
  output_dir: runs
input:
  path: notes.jsonl
  format: jsonl
  record_id_column: note_id
  entity_id:
    source: column
    column: patient_id
  text_column: note_text
detection:
  enabled: true
  detectors:
    - backend: aws_comprehend_medical
      region_name: us-east-1
      cost_per_100_characters_usd: 0.0014
  execution:
    workers: 2
```

The supported Bedrock detector and validator models are fixed:

- `us.anthropic.claude-sonnet-4-6`
- `openai.gpt-oss-safeguard-120b`

AWS calls use botocore's standard retry mode. Detection and validation each expose only
the useful local concurrency setting, `workers`, which defaults to 2. Validator responses
that are explicitly truncated are retried with the next fixed output-token budget.

Non-date synthetic replacements use one of two direct methods:

- `faker`: realistic values generated by Faker.
- `custom_list`: values supplied in configuration or the UI.

Both support `entity`, `record`, and `occurrence` consistency. Date replacement uses
`date_shift`, with one whole-week offset per entity. Dates can also be generalized to the
year, and ages can be generalized to `90+`.

## Python API

```python
from health_deid import create_run, open_run

run = create_run("config.yaml").execute()
print(run.status())

reopened = open_run(run.run_dir).resume()
revised = reopened.revise(
    "revised.yaml",
    reason="change replacement policy",
)
revised.export(
    output_path=revised.context.exports_dir / "final.jsonl",
    format="jsonl",
    mode="ready_only",
    selected_columns=["record_id", "final_text"],
)
```

A revised run reuses paid detection only when the imported input and detector definitions
match. Use `rerun_detection=True` (or CLI `--rerun-detection`) to authorize a new detector
pass when they do not.

## Examples

See the [examples](https://github.com/omid-jf/Health-DeID/blob/main/examples/README.md)
for thirteen configurations using fictional records.

## Documentation

See the [documentation](https://omid-jf.github.io/health-deid/) for the full user guide,
API, and module reference (generated with Sphinx).

## Acknowledgements

- **Developed by:** [Omid Jafari](https://omidjafari.com/)
- **Research affiliation and acknowledgement:** [Ang Li Lab](https://angli-lab.com/)

## License

Health-DeID is licensed under
the [GNU Affero General Public License v3.0 or later](https://github.com/omid-jf/Health-DeID/blob/main/LICENSE).
