Metadata-Version: 2.4
Name: medical-data-validator
Version: 1.3.0
Summary: A comprehensive medical data validation library for healthcare datasets
Author-email: Rana Ehtasham Ali <ranaehtashamali1@gmail.com>
Maintainer-email: Rana Ehtasham Ali <ranaehtashamali1@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/RanaEhtashamAli/medical-data-validator
Project-URL: Documentation, https://medical-data-validator.readthedocs.io/
Project-URL: Repository, https://github.com/RanaEhtashamAli/medical-data-validator
Project-URL: Bug Tracker, https://github.com/RanaEhtashamAli/medical-data-validator/issues
Project-URL: PyPI, https://pypi.org/project/medical-data-validator/
Keywords: medical,healthcare,data,validation,phi,pii,quality
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Filters
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=1.5.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: numpy>=1.21.0
Requires-Dist: openpyxl>=3.0.0
Requires-Dist: requests>=2.25.0
Requires-Dist: urllib3>=1.26.0
Requires-Dist: PyJWT>=2.8.0
Requires-Dist: reportlab>=4.0.0
Provides-Extra: cli
Requires-Dist: click>=8.0.0; extra == "cli"
Requires-Dist: rich>=13.0.0; extra == "cli"
Requires-Dist: tabulate>=0.9.0; extra == "cli"
Provides-Extra: web
Requires-Dist: flask>=2.3.0; extra == "web"
Requires-Dist: flask-cors>=4.0.0; extra == "web"
Requires-Dist: plotly>=5.15.0; extra == "web"
Requires-Dist: dash>=2.14.0; extra == "web"
Requires-Dist: dash-bootstrap-components>=1.5.0; extra == "web"
Requires-Dist: gunicorn>=21.0.0; extra == "web"
Requires-Dist: python-dotenv>=1.0.0; extra == "web"
Requires-Dist: werkzeug>=2.3.0; extra == "web"
Provides-Extra: api
Requires-Dist: flask>=2.3.0; extra == "api"
Requires-Dist: flask-cors>=4.0.0; extra == "api"
Requires-Dist: python-multipart>=0.0.6; extra == "api"
Requires-Dist: flask-restx>=1.1.0; extra == "api"
Requires-Dist: apispec>=6.0.0; extra == "api"
Requires-Dist: marshmallow>=3.19.0; extra == "api"
Requires-Dist: markdown>=3.4.0; extra == "api"
Provides-Extra: web-all
Requires-Dist: flask>=2.3.0; extra == "web-all"
Requires-Dist: flask-cors>=4.0.0; extra == "web-all"
Requires-Dist: plotly>=5.15.0; extra == "web-all"
Requires-Dist: dash>=2.14.0; extra == "web-all"
Requires-Dist: dash-bootstrap-components>=1.5.0; extra == "web-all"
Requires-Dist: gunicorn>=21.0.0; extra == "web-all"
Requires-Dist: python-dotenv>=1.0.0; extra == "web-all"
Requires-Dist: werkzeug>=2.3.0; extra == "web-all"
Requires-Dist: flask-restx>=1.1.0; extra == "web-all"
Requires-Dist: apispec>=6.0.0; extra == "web-all"
Requires-Dist: marshmallow>=3.19.0; extra == "web-all"
Requires-Dist: markdown>=3.4.0; extra == "web-all"
Provides-Extra: all
Requires-Dist: click>=8.0.0; extra == "all"
Requires-Dist: rich>=13.0.0; extra == "all"
Requires-Dist: tabulate>=0.9.0; extra == "all"
Requires-Dist: flask>=2.3.0; extra == "all"
Requires-Dist: flask-cors>=4.0.0; extra == "all"
Requires-Dist: plotly>=5.15.0; extra == "all"
Requires-Dist: dash>=2.14.0; extra == "all"
Requires-Dist: dash-bootstrap-components>=1.5.0; extra == "all"
Requires-Dist: gunicorn>=21.0.0; extra == "all"
Requires-Dist: python-dotenv>=1.0.0; extra == "all"
Requires-Dist: werkzeug>=2.3.0; extra == "all"
Requires-Dist: flask-restx>=1.1.0; extra == "all"
Requires-Dist: apispec>=6.0.0; extra == "all"
Requires-Dist: marshmallow>=3.19.0; extra == "all"
Requires-Dist: markdown>=3.4.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pytest-mock>=3.10.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: isort>=5.12.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: bandit>=1.7.0; extra == "dev"
Requires-Dist: safety>=2.0.0; extra == "dev"
Requires-Dist: pre-commit>=3.0.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=6.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.2.0; extra == "docs"
Requires-Dist: myst-parser>=1.0.0; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.23.0; extra == "docs"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: pytest-mock>=3.10.0; extra == "test"
Requires-Dist: pytest-benchmark>=4.0.0; extra == "test"
Requires-Dist: selenium>=4.0.0; extra == "test"
Requires-Dist: webdriver-manager>=4.0.0; extra == "test"
Requires-Dist: matplotlib>=3.3.0; extra == "test"
Requires-Dist: seaborn>=0.11.0; extra == "test"
Requires-Dist: xlrd>=2.0.0; extra == "test"
Provides-Extra: dashboard
Requires-Dist: flask>=2.3.0; extra == "dashboard"
Requires-Dist: flask-cors>=4.0.0; extra == "dashboard"
Requires-Dist: plotly>=5.15.0; extra == "dashboard"
Requires-Dist: dash>=2.14.0; extra == "dashboard"
Requires-Dist: dash-bootstrap-components>=1.5.0; extra == "dashboard"
Requires-Dist: gunicorn>=21.0.0; extra == "dashboard"
Requires-Dist: python-dotenv>=1.0.0; extra == "dashboard"
Requires-Dist: werkzeug>=2.3.0; extra == "dashboard"
Provides-Extra: real-data
Requires-Dist: requests>=2.25.0; extra == "real-data"
Requires-Dist: matplotlib>=3.3.0; extra == "real-data"
Requires-Dist: seaborn>=0.11.0; extra == "real-data"
Requires-Dist: xlrd>=2.0.0; extra == "real-data"
Requires-Dist: urllib3>=1.26.0; extra == "real-data"
Dynamic: license-file

# Medical Data Validator

[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.15815620.svg)](https://doi.org/10.5281/zenodo.15815620)
[![CI](https://github.com/RanaEhtashamAli/medical-data-validator/actions/workflows/ci.yml/badge.svg)](https://github.com/RanaEhtashamAli/medical-data-validator/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/RanaEhtashamAli/medical-data-validator/branch/master/graph/badge.svg)](https://codecov.io/gh/RanaEhtashamAli/medical-data-validator)

A Python library and REST API for validating healthcare datasets — PHI/PII detection, HIPAA/GDPR/FDA compliance scoring, data quality checks, and interactive dashboards.

> **Scope notice:** This tool assists research and data engineering teams in assessing data quality and identifying compliance risks. It is **not a substitute for a certified HIPAA/GDPR audit**, legal counsel, or a Business Associate Agreement. Regulatory compliance requires organisational controls, staff training, and third-party assessment that go beyond software.

## Features

### Core Validation
- **Multi-format support**: CSV, Excel, JSON, Parquet
- **PHI/PII detection**: All 18 HIPAA Safe Harbor identifiers
- **Medical code validation**: ICD-10, LOINC, CPT
- **Data quality checks**: Completeness, accuracy, consistency, timeliness
- **Extensible rule system**: Custom `ValidationRule` subclasses + setuptools entrypoints

### Compliance & Analytics (v1.2)
- **Compliance scoring**: HIPAA, GDPR, FDA 21 CFR Part 11 (heuristic, not certified)
- **FHIR R4 & SNOMED CT plugins**: via `load_compliance_plugins()`
- **Real-time monitoring**: Background thread, alert deduplication
- **Audit trail**: Append-only SQLite log, JWT authentication, multi-tenancy
- **Async jobs**: SQLite-backed job queue with background worker
- **Report export**: PDF and CSV via `/api/report` endpoints

### Operations
- **REST API**: Flask + Gunicorn, `/api/validate`, `/api/anonymize`, `/api/jobs`, `/api/report`
- **Web dashboard**: Dash + Bootstrap 5, interactive charts
- **Docker**: Multi-stage image, docker-compose with Redis

## Quick Start

```bash
git clone https://github.com/RanaEhtashamAli/medical-data-validator.git
cd medical-data-validator

# Install with all extras
pip install -e ".[all,dev]"

# Run the REST API (port 8000)
python api.py

# Run the web dashboard (port 5000)
SECRET_KEY=$(python -c "import secrets; print(secrets.token_hex(32))") python launch_dashboard.py
```

### Python Library

```python
from medical_data_validator import MedicalDataValidator
from medical_data_validator.validators import PHIDetector, DataQualityChecker
import pandas as pd

validator = MedicalDataValidator(
    enable_compliance=True,
    compliance_template='clinical_trials',
)
validator.add_rule(PHIDetector())
validator.add_rule(DataQualityChecker())

data = pd.read_csv('your_medical_data.csv')
result = validator.validate(data)

print(f"Valid: {result.is_valid}")
print(f"Issues: {len(result.issues)}")

compliance = result.summary.get('compliance_report', {})
print(f"Overall Score: {compliance.get('overall_score', 'N/A')}")
print(f"Risk Level:    {compliance.get('risk_level', 'N/A')}")
```

### FHIR R4 + SNOMED CT

```python
from medical_data_validator.plugins import load_compliance_plugins

engine = load_compliance_plugins()          # auto-discovers installed plugins
report  = engine.comprehensive_compliance_validation(df)
```

### Anonymization

```python
anon_df = validator.anonymize(
    df,
    columns=['patient_name', 'ssn', 'email'],
    method='hipaa_safe_harbor',
)
```

## API Endpoints

| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/health` | Health check |
| POST | `/api/validate/file` | Validate uploaded file |
| POST | `/api/validate/data` | Validate JSON data |
| POST | `/api/anonymize` | Anonymize columns |
| POST | `/api/jobs` | Submit async validation job |
| GET | `/api/jobs/<id>` | Poll job status |
| POST | `/api/report/inline/pdf` | Generate PDF report |
| POST | `/api/report/inline/csv` | Generate CSV report |
| GET | `/api/audit` | Query audit log |

## Docker

```bash
cp env.example .env   # fill in SECRET_KEY, JWT_SECRET, ADMIN_PASSWORD
docker-compose up -d
# API: http://localhost:8000/api/health
# Dashboard: http://localhost:8000/home
```

## Development

```bash
# Run all tests
pytest

# Skip slow/integration tests
pytest -m "not slow and not integration"

# Coverage report
pytest --cov=medical_data_validator --cov-report=term-missing

# Security scan
bandit -r medical_data_validator/ -ll

# Lint
flake8 medical_data_validator/
black medical_data_validator/
isort medical_data_validator/
```

## Supported Standards

| Standard | What is checked |
|---|---|
| HIPAA Safe Harbor | 18 PHI identifier patterns |
| GDPR | Personal and sensitive data presence |
| FDA 21 CFR Part 11 | Audit trail fields, electronic signature fields |
| ICD-10 / LOINC / CPT | Code format validity |
| FHIR R4 | Resource structure (plugin) |
| SNOMED CT | Terminology codes (plugin) |

## Configuration

Copy `env.example` to `.env` and set at minimum:

```bash
SECRET_KEY=<32-byte hex>
JWT_SECRET=<32-byte hex>
ADMIN_PASSWORD=<strong password>
FLASK_ENV=production
```

See `docs/compliance_checklist.md` for the full pre-production checklist.

## Researcher Documentation

- `docs/quickstart.ipynb` — Jupyter notebook: validate → inspect compliance → anonymize → export reports → async jobs
- `docs/compliance_checklist.md` — HIPAA, GDPR, FDA 21 CFR Part 11, GCP feature mapping

## License

MIT — see [LICENSE](LICENSE).

## Support

- Issues: [GitHub Issues](https://github.com/RanaEhtashamAli/medical-data-validator/issues)
- Discussions: [GitHub Discussions](https://github.com/RanaEhtashamAli/medical-data-validator/discussions)
