Metadata-Version: 2.4
Name: pamola-core
Version: 1.0.0.dev3
Summary: Pamola Core library for data anonymization, privacy models, metrics, and utilities
Author-email: Author <author@example.com>
Maintainer-email: Maintainer <maintainer@example.com>
License: BSD-3-Clause
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: python-decouple==3.8
Requires-Dist: pydantic[email]==2.12.4
Requires-Dist: matplotlib==3.8.2
Requires-Dist: seaborn==0.13.2
Requires-Dist: pandas==2.2.2
Requires-Dist: torch==2.8.0
Requires-Dist: scikit-learn==1.7.2
Requires-Dist: deepdiff==8.6.1
Requires-Dist: diff-match-patch==20241021
Requires-Dist: sdv==1.18.0
Requires-Dist: faker==33.3.1
Requires-Dist: scipy==1.15.3
Requires-Dist: uvicorn[standard]==0.38.0
Requires-Dist: openpyxl==3.1.5
Requires-Dist: xlrd==2.0.1
Requires-Dist: rstr==3.2.2
Requires-Dist: recordlinkage==0.16
Requires-Dist: plotly==6.4.0
Requires-Dist: kaleido==1.2.0
Requires-Dist: pillow==11.3.0
Requires-Dist: multidict==6.7.0
Requires-Dist: bcrypt==4.3.0
Requires-Dist: wordcloud==1.9.4
Requires-Dist: langdetect==1.0.9
Requires-Dist: fasttext-wheel==0.9.2
Requires-Dist: datasketch==1.7.0
Requires-Dist: nltk==3.9.2
Requires-Dist: dask[complete]==2025.11.0
Requires-Dist: pyarrow==14.0.2
Requires-Dist: numpy==1.26.4
Requires-Dist: matplotlib-venn==1.1.2
Requires-Dist: psutil==5.9.8
Requires-Dist: ijson==3.4.0.post0
Requires-Dist: chardet==5.2.0
Requires-Dist: rapidfuzz==3.14.3
Requires-Dist: cachetools==6.2.2
Requires-Dist: faiss-cpu==1.12.0
Requires-Dist: spacy==3.8.9
Requires-Dist: phonenumbers==9.0.10
Requires-Dist: base58==2.1.1
Requires-Dist: cryptography==46.0.3
Requires-Dist: PyYAML==6.0.2
Requires-Dist: typer[all]==0.24.1
Requires-Dist: jsonschema==4.25.1
Provides-Extra: test
Requires-Dist: pytest==8.4.2; extra == "test"
Requires-Dist: pytest-cov>=7.0; extra == "test"
Dynamic: license-file

# PAMOLA.CORE

<p align="center">
  <img src="https://realmdata.io/assets/img/logos/pamola-logo.png" alt="PAMOLA Logo" width="300"/>
</p>

<p align="center">
  <a href="https://github.com/DGT-Network/PAMOLA/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/license-BSD%203--Clause-blue.svg"></a>
  <a href="https://www.python.org/downloads/"><img alt="Python" src="https://img.shields.io/badge/python-3.10--3.12-blue.svg"></a>
  <a href="https://pypi.org/project/pamola-core/"><img alt="PyPI" src="https://img.shields.io/badge/pypi-1.0.0.dev3-orange.svg"></a>
  <img alt="Status" src="https://img.shields.io/badge/status-active%20development-orange.svg">
</p>

---

## Privacy Engineering for Python. Finally.

**PAMOLA.CORE** is the open-source foundation of the PAMOLA platform — a Python library for **privacy-preserving data operations** with reproducible pipelines, structured artifacts, and an audit trail per operation.

Developed by **[Realm Inveo Inc.](https://realmdata.io)**

---

## The Problem

You need to anonymize sensitive data. You've tried:

- **ARX:** Powerful, but Java, GUI-focused, opaque operations.
- **Faker + Presidio + custom scripts:** Fragmented, no pipeline, no proof.
- **DP libraries:** Great math, but narrow scope.

You're still missing:

- Direct operations (mask, generalize, pseudonymize, suppress) — not just "achieve k-anonymity".
- Risk measurement (privacy/fidelity/utility metrics on real outputs).
- Reproducibility (config + metrics + artifacts written to disk per task).

---

## The Solution

PAMOLA.CORE: operations-first privacy engineering with a per-operation lifecycle (validate → load → process → save → metrics → visualize → cache).

```python
from pathlib import Path

import pandas as pd

from pamola_core import HashBasedPseudonymizationOperation, FullMaskingOperation
from pamola_core.utils.ops.op_data_source import DataSource
from pamola_core.utils.tasks.task_reporting import TaskReporter
from pamola_core.utils.progress import HierarchicalProgressTracker

df = pd.read_csv("customers.csv")
data_source = DataSource(dataframes={"main": df})
task_dir = Path("./anonymize_customers")
reporter = TaskReporter(task_dir=task_dir, task_name="anonymize")
tracker = HierarchicalProgressTracker(total=2, description="Anonymize")

# 1) Irreversible hash-based pseudonymization of the email column
HashBasedPseudonymizationOperation(
    field_name="email",
    algorithm="sha3_256",
    salt_config={"source": "parameter", "value": "ab" * 32},
    use_pepper=True,
).execute(data_source, task_dir, reporter, tracker)

# 2) Mask the phone column entirely
FullMaskingOperation(
    field_name="phone",
    mask_char="*",
).execute(data_source, task_dir, reporter, tracker)
```

**Output structure (`task_dir/`):**

```
anonymize_customers/
├── config.json           # Operation configuration (secrets redacted)
├── output/               # Anonymized data (csv/parquet)
├── metrics/              # Privacy & quality metrics (JSON)
├── visualizations/       # Generated charts (PNG)
└── logs/                 # Per-task execution log
```

> **Security note (1.0.0.dev3):** `config.json` now redacts sensitive parameters (e.g. AES-256 mapping encryption keys) via `OperationConfig.SENSITIVE_KEYS`. See [CHANGELOG.md](CHANGELOG.md) for the full release notes.

---

## PAMOLA Ecosystem

PAMOLA.CORE is part of a comprehensive privacy engineering stack:

| Component | Description | Availability |
|-----------|-------------|--------------|
| **PAMOLA.CORE** | Anonymization, profiling, transformation, metrics, and shared op framework | **Open Source (this repo)** |
| **PAMOLA.STUDIO** | Visual environment for data transformation and privacy management | Commercial |
| **PAMOLA.SYNT** | Synthetic data generation, including formal DP-SGD-based generators | Commercial |
| **PAMOLA.BEST** | Best-practice policy modules and DP accounting | Commercial |
| **PAMOLA.TEXT** | Long text and document anonymization (NLP/LLM-based) | Commercial |
| **PAMOLA.INSIGHT** | Agent modules for LLM integration | Commercial |

> **Scope of CORE:** This package provides anonymization and pseudonymization primitives, classical privacy metrics, and reproducibility plumbing. It does **not** implement formal differential-privacy synthetic-data generation — that lives in PAMOLA.SYNT / PAMOLA.BEST.

---

## What's In CORE

All classes below are exported from the top-level `pamola_core` package.

| Category | Operations |
|----------|------------|
| **Anonymization — Masking** | `FullMaskingOperation`, `PartialMaskingOperation` |
| **Anonymization — Generalization** | `CategoricalGeneralizationOperation`, `NumericGeneralizationOperation`, `DateTimeGeneralizationOperation` |
| **Anonymization — Suppression** | `AttributeSuppressionOperation`, `CellSuppressionOperation`, `RecordSuppressionOperation` |
| **Anonymization — Pseudonymization** | `HashBasedPseudonymizationOperation` (SHA3-256/512 + salt + pepper), `ConsistentMappingPseudonymizationOperation` (AES-256-GCM reversible mapping) |
| **Anonymization — Noise** | `UniformNumericNoiseOperation`, `UniformTemporalNoiseOperation` |
| **Profiling — Field analyzers** | `CategoricalOperation`, `CorrelationOperation`, `CorrelationMatrixOperation`, `CurrencyOperation`, `DateOperation`, `EmailOperation`, `GroupAnalyzerOperation`, `IdentityAnalysisOperation`, `MVFOperation`, `NumericOperation`, `PhoneOperation`, `TextSemanticCategorizerOperation` |
| **Profiling — Dataset-level** | `KAnonymityProfilerOperation`, `DataAttributeProfilerOperation` |
| **Transformation** | `AddOrModifyFieldsOperation`, `RemoveFieldsOperation`, `CleanInvalidValuesOperation`, `ImputeMissingValuesOperation`, `AggregateRecordsOperation`, `MergeDatasetsOperation`, `SplitByIDValuesOperation`, `SplitFieldsOperation` |
| **Fake Data** | `FakeNameOperation`, `FakeEmailOperation`, `FakePhoneOperation`, `FakeOrganizationOperation` |
| **Metrics** | `FidelityOperation` (KS, KL-divergence), `PrivacyMetricOperation` (DCR, NNDR, uniqueness, k-anonymity, l-diversity), `UtilityMetricOperation` (classification, regression) |
| **Analysis helpers** | `analyze_dataset_summary`, `analyze_descriptive_stats`, `analyze_correlation`, `visualize_distribution_df`, `calculate_full_risk` |

> **Note on attack simulation:** Internal modules under `pamola_core/attacks/` (linkage, attribute inference, membership inference, DCR/NNDR helpers) are used by `PrivacyMetricOperation`, but there are no public, registered `Attack*Operation` classes in CORE 1.0.0.dev3. Future releases may expose them.

---

## Pseudonymization Spotlight (1.0.0.dev3)

The 1.0.0.dev3 release hardened the pseudonymization stack:

```python
from pamola_core import ConsistentMappingPseudonymizationOperation

op = ConsistentMappingPseudonymizationOperation(
    field_name="customer_id",
    mapping_encryption_key="ab" * 32,         # 256-bit hex key
    pseudonym_type="uuid",                    # or "sequential" / "random_string"
    mapping_format="csv",                     # encrypted at rest with AES-256-GCM
    persist_frequency=1000,
)

# Outputs:
#  - {task_dir}/output/         anonymized data
#  - {task_dir}/output/         encrypted mapping file (re-identification key)
#  - {task_dir}/metrics/        operation metrics
#  - {task_dir}/config.json     mapping_encryption_key is replaced with "*REDACTED*"
```

Highlights:

- **AES-256-GCM mapping encryption keys are never persisted to disk.** A new `OperationConfig.SENSITIVE_KEYS` declaration + `to_safe_dict()` redacts secrets before any `save_config()` call.
- **Hash-based op rejects weak salts** (all-zero or empty) when `use_pepper=False`.
- **Per-run session id** invalidates stale disk cache when `use_pepper=True`, so previous-run pseudonyms cannot be served back.
- **Compound identifiers**, **ENRICH/REPLACE modes**, **reverse mapping**, and **Dask pickle safety** are covered by 41 dedicated tests.

---

## Metrics

Metric operations write JSON artifacts under `{task_dir}/metrics/`:

```python
from pamola_core import FidelityOperation, PrivacyMetricOperation

FidelityOperation(
    fidelity_metrics=["ks", "kl"],
    columns=["age", "income"],
).execute(data_source, task_dir, reporter, tracker)

PrivacyMetricOperation(
    privacy_metrics=["dcr", "nndr", "uniqueness"],
    quasi_identifiers=["age", "gender", "zipcode"],
).execute(data_source, task_dir, reporter, tracker)
```

For dataset-level utility scoring (classification / regression downstream models) use `UtilityMetricOperation`.

---

## Installation

**From PyPI:**

```bash
pip install pamola-core==1.0.0.dev3
```

**From source:**

```bash
git clone https://github.com/DGT-Network/PAMOLA.git
cd PAMOLA
pip install -e .
```

**Test extras:**

```bash
pip install -e ".[test]"   # adds pytest, pytest-cov
```

> **Heads-up:** All scientific dependencies (numpy, pandas, scikit-learn, scipy, torch, dask, spacy, sdv, faker, cryptography, etc.) are pinned in the main `[project.dependencies]` table — no separate `[fast]/[ner]/[dp]` extras in this release.

---

## Supported Python Versions

PAMOLA.CORE supports Python **3.10, 3.11, and 3.12** (`requires-python = ">=3.10,<3.13"`).

| Python Version | Supported |
|---|---|
| 3.10 | ✅ |
| 3.11 | ✅ |
| 3.12 | ✅ |
| 3.9 and below | ❌ |
| 3.13 and above | ❌ (blocked by third-party dependencies) |

---

## Core Dependencies

A non-exhaustive view of the heaviest third-party packages (full list in `pyproject.toml`):

| Package | Pin | Purpose |
|---|---|---|
| **numpy** | `1.26.4` | Numerical computation across metrics, attacks, statistical analysis |
| **pandas** | `2.2.2` | The DataFrame container for every CORE operation |
| **scikit-learn** | `1.7.2` | Classification/regression metrics, nearest-neighbor distance, model-based utility |
| **scipy** | `1.15.3` | Statistical tests (KS, KL divergence) used by `FidelityOperation` |
| **cryptography** | `46.0.3` | AES-256-GCM mapping encryption for `ConsistentMappingPseudonymizationOperation` |
| **dask[complete]** | `2025.11.0` | Optional out-of-core / distributed execution path |
| **pyarrow** | `14.0.2` | Parquet I/O |
| **typer** | `0.24.1` | CLI entry point |

---

## Versioning

PAMOLA.CORE follows [Semantic Versioning](https://semver.org/) and [PEP 440](https://peps.python.org/pep-0440/).

```python
import pamola_core
print(pamola_core.__version__)   # e.g. "1.0.0.dev3"
```

| Phase | Version | Branch | Tag | Install |
|-------|---------|--------|-----|---------|
| Dev (current) | `1.0.0.dev3` | `develop` | `v1.0.0.dev3` | `pip install pamola-core==1.0.0.dev3` |
| Stable (planned) | `1.0.0` | `main` | `v1.0.0` | `pip install pamola-core` |

- **Source of truth:** `pyproject.toml` → `version`
- **Changelog:** [CHANGELOG.md](CHANGELOG.md)
- **CI/CD:** GitHub Actions — lint (ruff), test (3.10/3.11/3.12, pytest), build (sdist+wheel), PyPI publish on tag `v*`
- **Release rules:** Dev tags (`v*dev*`) must be on `develop`; stable tags on `main`.

---

## CLI

The `pamola-core` console script is installed automatically:

```bash
pamola-core --version
pamola-core list-ops                              # discover registered operations
pamola-core run --task task.json                  # run a task definition
pamola-core run --op FullMaskingOperation --config config.json --input data.csv
pamola-core schema FullMaskingOperation           # show parameter schema
pamola-core validate-config --config config.json  # validate a config file
```

Run `pamola-core --help` for the full command list.

---

## Examples

Hands-on notebooks live under [`examples/`](https://github.com/DGT-Network/PAMOLA/tree/main/examples):

- `examples/anonymization/pseudonymization/` — simple + advanced for hash-based and consistent-mapping pseudonymization
- `examples/anonymization/` — masking, generalization, noise, suppression
- `examples/profiling/` — field-level and dataset-level profilers
- `examples/transformations/` — merge, split, aggregate, clean, impute
- `examples/fake_data/` — synthetic identity, email, phone, organization
- `examples/metrics/` — fidelity, privacy, utility metrics
- `examples/data_examples/sample.csv` — non-PII synthetic sample used by the notebooks

> **No real personal data** is included in this repository. All sample records are artificially generated.

---

## Philosophy

- **Operations-first:** Direct transforms with a well-defined 7-step lifecycle, not constraint optimization.
- **Measure everything:** Privacy, fidelity, and utility metrics persisted as JSON artifacts.
- **Reproducibility by default:** Each operation writes a `config.json` (with secrets redacted) alongside its output.
- **Secret hygiene:** `OperationConfig.SENSITIVE_KEYS` provides a single place to declare parameters that must never reach disk.

---

## API Documentation

The project uses **Sphinx** to generate API reference documentation from Python docstrings.

```bash
cd docs
make html        # output: docs/_build/html/index.html
```

---

## Documentation

| Resource | Link |
|----------|------|
| **PET Knowledge Base** | [realmdata.io/kb](https://realmdata.io/kb/index.html) |
| **Technical Documentation** | [docs/en/index.md](https://github.com/DGT-Network/PAMOLA/blob/main/docs/en/index.md) |
| **Glossary** | [realmdata.io/glossary](https://realmdata.io/pages/glossary.html) |
| **Examples** | [`examples/`](https://github.com/DGT-Network/PAMOLA/tree/main/examples) |
| **Changelog** | [CHANGELOG.md](CHANGELOG.md) |

---

## Use Cases

- **Data Engineering:** Prepare privacy-safe datasets for ML training.
- **Healthcare:** HIPAA-oriented de-identification workflows (Safe Harbor support).
- **Finance:** Privacy engineering aligned with PCI/GDPR considerations.
- **Compliance:** Audit-ready evidence with structured per-operation artifacts.
- **Data Sharing:** Risk-assessed data exchange between organizations.

---

## Regulatory Context

PAMOLA.CORE provides technical building blocks for privacy compliance programs:

| Regulation | Relevant Capabilities |
|------------|----------------------|
| **GDPR** | Pseudonymization (reversible / irreversible), data minimization (Art. 25, 32) |
| **HIPAA** | Safe Harbor de-identification support |
| **CCPA/CPRA** | Data suppression, masking, anonymization workflows |

> **Important:** PAMOLA.CORE provides technical capabilities only. Legal compliance requires organizational policies, procedures, and legal guidance beyond software tools.

---

## Contributing

```bash
git clone https://github.com/DGT-Network/PAMOLA.git
cd PAMOLA
pip install -e ".[test]"
pytest tests/ -v
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

---

## Ownership & Licensing

**PAMOLA.CORE** is developed and owned exclusively by **[Realm Inveo Inc.](https://realmdata.io)**

This repository is hosted under the DGT-Network GitHub organization, which provides shared development infrastructure for Realm Inveo projects. **DGT-Network does not claim ownership of this intellectual property.** All IP rights belong exclusively to Realm Inveo Inc.

**License:** BSD 3-Clause — see [LICENSE](https://github.com/DGT-Network/PAMOLA/blob/main/LICENSE).

---

## Contact

| Purpose | Contact |
|---------|---------|
| **General inquiries** | [contact@realmdata.io](mailto:contact@realmdata.io) |
| **Commercial / Sales** | [sales@realmdata.io](mailto:sales@realmdata.io) |
| **Due diligence / Legal** | [legal@realmdata.io](mailto:legal@realmdata.io) |
| **Website** | [realmdata.io](https://realmdata.io) |

---

<p align="center">
  <sub>Built by <a href="https://realmdata.io">Realm Inveo Inc.</a></sub>
</p>
