Metadata-Version: 2.4
Name: pamola-core
Version: 1.0.0
Summary: Pamola Core library for data anonymization, privacy models, metrics, and utilities
Author-email: "REALM Inveo Inc." <info@realmdata.ai>
Maintainer-email: "REALM Inveo Inc." <info@realmdata.ai>
License-Expression: MIT
Project-URL: Homepage, https://realmdata.ai
Project-URL: Repository, https://github.com/realmdata-ai/PAMOLA
Project-URL: Changelog, https://github.com/realmdata-ai/PAMOLA/blob/main/CHANGELOG.md
Project-URL: Bug Tracker, https://github.com/realmdata-ai/PAMOLA/issues
Keywords: privacy,anonymization,data-privacy,k-anonymity,l-diversity,pseudonymization,pii,gdpr,data-protection,re-identification-risk
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Legal Industry
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: <3.13,>=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: python-decouple==3.8
Requires-Dist: pandas==2.2.2
Requires-Dist: scikit-learn==1.7.2
Requires-Dist: faker==33.3.1
Requires-Dist: scipy==1.15.3
Requires-Dist: openpyxl==3.1.5
Requires-Dist: xlrd==2.0.1
Requires-Dist: recordlinkage==0.16
Requires-Dist: langdetect==1.0.9
Requires-Dist: fasttext-wheel==0.9.2
Requires-Dist: datasketch==1.7.0
Requires-Dist: nltk==3.10.3
Requires-Dist: dask[complete]==2026.1.1
Requires-Dist: pyarrow==23.0.1
Requires-Dist: numpy==1.26.4
Requires-Dist: psutil==5.9.8
Requires-Dist: ijson==3.4.0.post0
Requires-Dist: chardet==5.2.0
Requires-Dist: rapidfuzz==3.14.3
Requires-Dist: cachetools==6.2.2
Requires-Dist: faiss-cpu==1.12.0
Requires-Dist: spacy==3.8.9
Requires-Dist: phonenumbers==9.0.10
Requires-Dist: base58==2.1.1
Requires-Dist: cryptography==50.0.0
Requires-Dist: PyYAML==6.0.2
Requires-Dist: typer[all]==0.24.1
Requires-Dist: jsonschema==4.25.1
Requires-Dist: python-dotenv==1.2.3
Requires-Dist: filelock==4.2.1
Requires-Dist: joblib==1.5.3
Requires-Dist: rich==15.0.0
Requires-Dist: tqdm==4.70.0
Requires-Dist: packaging==26.3
Requires-Dist: jinja2==3.1.6
Requires-Dist: regex==2026.7.19
Provides-Extra: test
Requires-Dist: pytest==8.4.2; extra == "test"
Requires-Dist: pytest-cov>=7.0; extra == "test"
Provides-Extra: viz
Requires-Dist: matplotlib==3.8.2; extra == "viz"
Requires-Dist: seaborn==0.13.2; extra == "viz"
Requires-Dist: plotly==6.4.0; extra == "viz"
Requires-Dist: kaleido==1.2.0; extra == "viz"
Requires-Dist: pillow==11.3.0; extra == "viz"
Requires-Dist: wordcloud==1.9.4; extra == "viz"
Requires-Dist: matplotlib-venn==1.1.2; extra == "viz"
Dynamic: license-file

# PAMOLA.CORE

<p align="center">
  <img src="https://realmdata.ai/assets/img/logos/pamola-logo.png" alt="PAMOLA Logo" width="300"/>
</p>

<p align="center">
  <a href="https://github.com/realmdata-ai/PAMOLA/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
  <a href="https://www.python.org/downloads/"><img alt="Python" src="https://img.shields.io/badge/python-3.11--3.12-blue.svg"></a>
  <a href="https://pypi.org/project/pamola-core/"><img alt="PyPI" src="https://img.shields.io/pypi/v/pamola-core?label=pypi"></a>
  <img alt="Status" src="https://img.shields.io/badge/status-active%20development-orange.svg">
</p>

---

## Privacy Engineering for Python. Finally.

**PAMOLA.CORE** is the open-source foundation of the PAMOLA platform — a Python library for **privacy-preserving data operations** with reproducible pipelines, structured artifacts, and an audit trail per operation.

Developed by **[Realm Inveo Inc.](https://realmdata.ai)**

---

## The Problem

You need to anonymize sensitive data. You've tried:

- **ARX:** Powerful, but Java, GUI-focused, opaque operations.
- **Faker + Presidio + custom scripts:** Fragmented, no pipeline, no proof.
- **DP libraries:** Great math, but narrow scope.

You're still missing:

- Direct operations (mask, generalize, pseudonymize, suppress) — not just "achieve k-anonymity".
- Risk measurement (privacy/fidelity/utility metrics on real outputs).
- Reproducibility (config + metrics + artifacts written to disk per task).

---

## The Solution

PAMOLA.CORE: operations-first privacy engineering with a per-operation lifecycle (validate → load → process → save → metrics → visualize → cache).

```python
from pathlib import Path

import pandas as pd

from pamola_core import DataSource, FullMaskingOperation, HashBasedPseudonymizationOperation

df = pd.DataFrame({
    "customer_id": [101, 102, 103],
    "email": ["ann@example.com", "bob@example.com", "cy@example.com"],
    "phone": ["+1 416 555 0101", "+1 416 555 0102", "+1 416 555 0103"],
})
task_dir = Path("anonymize_customers")


def output_of(result):
    """Each operation writes its own output file; read it back to feed the next step."""
    return pd.read_csv(result.get_artifacts_by_category("output")[0].path)


# 1) Keyed, irreversible hash of the e-mail column
hashed = HashBasedPseudonymizationOperation(field_name="email").execute(
    DataSource(dataframes={"main": df}), task_dir / "01_hash_email", reporter=None
)

# 2) Mask the phone column of the hashed output
masked = FullMaskingOperation(field_name="phone").execute(
    DataSource(dataframes={"main": output_of(hashed)}), task_dir / "02_mask_phone", reporter=None
)

print(masked.status)        # OperationStatus.SUCCESS
print(output_of(masked))    # e-mail hashed, phone masked, customer_id unchanged
```

> **One operation, one field, one output file.** An operation reads the dataset you pass in its
> `DataSource` and writes a complete copy with *its* field transformed. It does not see what another
> operation did to the same dataset. To protect several fields, feed each operation the output of
> the previous one, as above — otherwise every output file still contains the other fields in
> clear text. Give each operation its own directory so its `config.json` is kept.
>
> The hash uses a fresh random pepper per run (`use_pepper=True` by default), so it cannot be
> recomputed from the e-mail alone; on three e-mails the operation also warns that hashing a
> low-entropy column is reversible by enumeration.

**Output structure (per operation directory):**

```
anonymize_customers/01_hash_email/
├── config.json           # Operation configuration (secrets redacted)
├── output/               # The transformed dataset (CSV by default, Parquet with output_format="parquet")
└── metrics/              # Operation metrics (JSON)
```

Charts (`visualizations/`) are written only with `generate_visualization=True` and the `viz` extra;
the consistent-mapping operation adds an encrypted `maps/` directory.

> **Security note:** `config.json` redacts sensitive parameters (salts, mapping encryption keys, fake-data keys) declared in `OperationConfig.SENSITIVE_KEYS`. See [CHANGELOG.md](CHANGELOG.md) for the full release notes.

---

## PAMOLA Ecosystem

PAMOLA.CORE is part of a comprehensive privacy engineering stack:

| Component | Description | Availability |
|-----------|-------------|--------------|
| **PAMOLA.CORE** | Anonymization, profiling, transformation, metrics, and shared op framework | **Open Source (this repo)** |
| **PAMOLA.STUDIO** | Visual environment for data transformation and privacy management | Commercial |
| **PAMOLA.SYNT** | Synthetic data generation, including formal DP-SGD-based generators | Commercial |
| **PAMOLA.BEST** | Best-practice policy modules and DP accounting | Commercial |
| **PAMOLA.TEXT** | Long text and document anonymization (NLP/LLM-based) | Commercial |
| **PAMOLA.INSIGHT** | Agent modules for LLM integration | Commercial |

> **Scope of CORE:** This package provides anonymization and pseudonymization primitives, classical privacy metrics, and reproducibility plumbing. It does **not** implement formal differential-privacy synthetic-data generation — that lives in PAMOLA.SYNT / PAMOLA.BEST.
>
> CORE also does not *search* for a k-anonymous generalization the way ARX or Mondrian-style tools do. You choose the generalization levels, apply them with the generalization and suppression operations, and verify the result with `KAnonymityProfilerOperation` (its ENRICH mode adds a per-record k that `RecordSuppressionOperation(suppression_condition="risk")` can act on).

---

## What's In CORE

All classes below are exported from the top-level `pamola_core` package.

| Category | Operations |
|----------|------------|
| **Anonymization — Masking** | `FullMaskingOperation`, `PartialMaskingOperation` |
| **Anonymization — Generalization** | `CategoricalGeneralizationOperation`, `NumericGeneralizationOperation`, `DateTimeGeneralizationOperation` |
| **Anonymization — Suppression** | `AttributeSuppressionOperation`, `CellSuppressionOperation`, `RecordSuppressionOperation` |
| **Anonymization — Pseudonymization** | `HashBasedPseudonymizationOperation` (SHA3-256/512 + salt + pepper), `ConsistentMappingPseudonymizationOperation` (AES-256-GCM reversible mapping) |
| **Anonymization — Noise** | `UniformNumericNoiseOperation`, `UniformTemporalNoiseOperation` |
| **Profiling — Field analyzers** | `CategoricalOperation`, `CorrelationOperation`, `CorrelationMatrixOperation`, `CurrencyOperation`, `DateOperation`, `EmailOperation`, `GroupAnalyzerOperation`, `IdentityAnalysisOperation`, `MVFOperation`, `NumericOperation`, `PhoneOperation`, `TextSemanticCategorizerOperation` |
| **Profiling — Dataset-level** | `KAnonymityProfilerOperation`, `DataAttributeProfilerOperation` |
| **Transformation** | `AddOrModifyFieldsOperation`, `RemoveFieldsOperation`, `CleanInvalidValuesOperation`, `ImputeMissingValuesOperation`, `AggregateRecordsOperation`, `MergeDatasetsOperation`, `SplitByIDValuesOperation`, `SplitFieldsOperation` |
| **Fake Data** | `FakeNameOperation`, `FakeEmailOperation`, `FakePhoneOperation`, `FakeOrganizationOperation` |
| **Metrics** | `FidelityOperation` (KS, KL-divergence), `PrivacyMetricOperation` (DCR, NNDR, uniqueness, k-anonymity, l-diversity), `CVPLLinkageOperation` (blocked, projected linkage exposure of a release against its source; see [docs](docs/en/core/metrics/privacy/cvpl.md)), `UtilityMetricOperation` (classification, regression) |
| **Analysis helpers** | `analyze_dataset_summary`, `analyze_descriptive_stats`, `analyze_correlation`, `visualize_distribution_df`, `calculate_full_risk` |

> **Note on attack simulation:** `pamola_core/attacks/` is an **internal** package with no stability guarantee, and it is deliberately narrow — it carries only the attacks that measure *anonymization* quality: `LinkageAttack` (its TF-IDF / SVD "CVPLA" variant, which is **not** the CVPL metric below) and `AttributeInference`. The CVPL linkage metric lives in `pamola_core.metrics.privacy.cvpl` and is run through `CVPLLinkageOperation`. There are no public, registered `Attack*Operation` classes.
>
> Two corrections to earlier releases of this note. These modules are **not** used by `PrivacyMetricOperation` — that operation imports `pamola_core.metrics.privacy`. And DCR/NNDR live in `pamola_core.metrics.privacy.distance` and `.neighbor`; duplicate copies that had accumulated under `attacks/` were removed, as was membership inference, which measures model privacy rather than anonymization.
>
> **Singling-out is not implemented.** It is the third anonymization-facing criterion, and no report produced from this package covers it.

---

## Pseudonymization Spotlight

```python
from pamola_core import ConsistentMappingPseudonymizationOperation

op = ConsistentMappingPseudonymizationOperation(
    field_name="customer_id",
    mapping_encryption_key="ab" * 32,         # 256-bit hex key - keep it out of source control
    pseudonym_type="uuid",                    # or "sequential" / "random_string"
    mapping_format="csv",                     # encrypted at rest with AES-256-GCM
    persist_frequency=1000,
)

# Outputs:
#  - {task_dir}/output/         pseudonymized data
#  - {task_dir}/maps/           encrypted mapping file (*.enc) - the re-identification key
#  - {task_dir}/metrics/        operation metrics
#  - {task_dir}/config.json     mapping_encryption_key is replaced with "*REDACTED*"
```

Highlights:

- **Mapping encryption keys are never persisted to disk.** `OperationConfig.SENSITIVE_KEYS` + `to_safe_dict()` redact secrets before any `save_config()` call.
- **No plain-text mapping file is ever written**: mappings and their backups are encrypted in memory and replaced atomically.
- **Hash-based op rejects weak salts** (all-zero or empty) when `use_pepper=False`, and warns when the source column has so few distinct values that the hash is reversible by enumeration.
- **Per-run session id** invalidates stale disk cache when `use_pepper=True`, so previous-run pseudonyms cannot be served back.
- **Compound identifiers**, **ENRICH/REPLACE modes**, **reverse mapping**, and **Dask pickle safety** have dedicated tests.

---

## Metrics

Metric operations compare an original and a transformed dataset and write JSON artifacts under `{task_dir}/metrics/`:

```python
from pathlib import Path

import numpy as np
import pandas as pd

from pamola_core import DataSource, FidelityOperation, PrivacyMetricOperation

rng = np.random.default_rng(0)
original_df = pd.DataFrame({
    "age": rng.integers(18, 90, 1000),
    "gender": rng.choice(["F", "M"], 1000),
    "income": rng.normal(50_000, 15_000, 1000).round(),
})
anonymized_df = original_df.assign(age=(original_df["age"] // 10) * 10)  # 10-year bands

pair = DataSource(dataframes={"original": original_df, "transformed": anonymized_df})
names = {"original_dataset_name": "original", "transformed_dataset_name": "transformed"}
task_dir = Path("metrics_task")

fidelity = FidelityOperation(fidelity_metrics=["ks", "kl"], columns=["age", "income"]).execute(
    pair, task_dir / "fidelity", reporter=None, **names
)
privacy = PrivacyMetricOperation(
    privacy_metrics=["dcr", "nndr", "uniqueness"],
    columns=["age", "gender", "income"],
    metric_params={"uniqueness": {"quasi_identifiers": ["age", "gender"], "sensitives": ["income"]}},
).execute(pair, task_dir / "privacy", reporter=None, **names)
```

DCR/NNDR run an exact, block-wise search with bounded memory; above 10K × 10K records FAISS is selected automatically where it returns the same result as the exact search. A comparison that would exceed `max_comparisons` fails with a typed `ResourceError` instead of exhausting memory.

For dataset-level utility scoring (classification / regression downstream models) use `UtilityMetricOperation`.

---

## Installation

**From PyPI:**

```bash
pip install pamola-core
```

> The base install is large (about 0.9 GB of dependencies: pandas, scikit-learn, Dask, spaCy,
> FAISS, PyArrow and others). Install it into a virtual environment.

**From source:**

```bash
git clone https://github.com/realmdata-ai/PAMOLA.git
cd PAMOLA
pip install -e .
```

**Visualization is optional:**

```bash
pip install "pamola-core[viz]"
```

The base install writes JSON, CSV and Parquet, and never draws. Charts need
matplotlib, plotly, seaborn, kaleido, wordcloud and matplotlib-venn, which are
large, so they ship in the `viz` extra instead of the base install. Everything
else — profiling, anonymization, transformations, metrics — works without them.

Call a chart without the extra and you get a message naming what to install,
not a `ModuleNotFoundError` about a package you never asked for:

```text
DependencyMissingError: no visualization backend is installed ...
They ship with the viz extra: pip install 'pamola-core[viz]'
```

Visualization is **off by default**: operations only draw when you pass
`generate_visualization=True`. A default run starts no render thread, writes no
plot files and logs nothing about charts, with or without the `viz` extra. With
`generate_visualization=True` the charts are written and registered as
artifacts before `execute()` returns; on a base install that request returns a
single `ERROR` result naming `pip install 'pamola-core[viz]'` instead of running
and logging a traceback from every layer of the plotting stack.

> **Behaviour change before `1.0.0`:** `generate_visualization` used to default to
> `True`. Pass `generate_visualization=True` to keep the old behaviour.

**Test extras:**

```bash
pip install -e ".[test]"   # adds pytest, pytest-cov
pip install -e ".[viz]"    # adds the plotting stack
```

> **Heads-up:** Apart from `viz`, all scientific dependencies (numpy, pandas, scikit-learn, scipy, dask, spacy, faker, cryptography, etc.) are pinned in the main `[project.dependencies]` table — no separate `[fast]/[ner]/[dp]` extras in this release. Splitting the remaining heavy stacks into `[nlp]` and `[bigdata]` is planned; see FP-02b.
>
> As of `1.0.0.dev4`, `torch` and `sdv` are **no longer installed**: nothing in
> the library imported them. If you relied on them being pulled in as a side
> effect, install them explicitly. Splitting the remaining heavy dependencies
> (spacy, nltk, faiss, dask, the plotting stack) into optional extras is
> planned for a later release.

---

## Supported Python Versions

PAMOLA.CORE supports Python **3.11 and 3.12** (`requires-python = ">=3.11,<3.13"`). Python 3.10 is not supported: the fixed `filelock` 4.2.1 requires Python 3.11 or newer.

| Python Version | Supported |
|---|---|
| 3.11 | ✅ |
| 3.12 | ✅ |
| 3.10 and below | ❌ |
| 3.13 and above | ❌ (not yet: the pinned `numpy 1.26.4` / `pandas 2.2.2` and `fasttext-wheel 0.9.2` publish no 3.13 wheels) |

---

## Core Dependencies

A non-exhaustive view of the heaviest third-party packages (full list in `pyproject.toml`):

| Package | Pin | Purpose |
|---|---|---|
| **numpy** | `1.26.4` | Numerical computation across metrics, attacks, statistical analysis |
| **pandas** | `2.2.2` | The DataFrame container for every CORE operation |
| **scikit-learn** | `1.7.2` | Classification/regression metrics, nearest-neighbor distance, model-based utility |
| **scipy** | `1.15.3` | Statistical tests (KS, KL divergence) used by `FidelityOperation` |
| **cryptography** | `50.0.0` | AES-256-GCM mapping encryption for `ConsistentMappingPseudonymizationOperation` |
| **dask[complete]** | `2026.1.1` | Optional out-of-core / distributed execution path |
| **pyarrow** | `23.0.1` | Parquet I/O |
| **typer** | `0.24.1` | CLI entry point |

---

## Versioning

PAMOLA.CORE follows [Semantic Versioning](https://semver.org/) and [PEP 440](https://peps.python.org/pep-0440/).

```python
import pamola_core
print(pamola_core.__version__)   # e.g. "1.0.0"
```

| Phase | Version | Branch | Tag | Install |
|-------|---------|--------|-----|---------|
| Stable (current) | `1.0.0` | `main` | `v1.0.0` | `pip install pamola-core` |
| Pre-releases | `X.Y.Z.devN`, `X.Y.ZrcN` | `develop` | `vX.Y.Z.devN`, `vX.Y.ZrcN` | `pip install --pre pamola-core` |

The badge at the top shows the newest version published on PyPI; the version on `develop` can be ahead of it.

- **Source of truth:** `pyproject.toml` → `version`
- **Changelog:** [CHANGELOG.md](CHANGELOG.md)
- **CI/CD:** GitHub Actions — lint (ruff), test (3.11/3.12, pytest), build (sdist+wheel), PyPI publish on tag `v*`
- **Release rules:** Dev tags (`v*dev*`) must be on `develop`; stable tags on `main`.

---

## CLI

The `pamola-core` console script is installed automatically:

```bash
pamola-core --version
pamola-core list-ops                              # discover registered operations
pamola-core run --task task.json                  # run a task definition
pamola-core run --op FullMaskingOperation --config config.json --input data.csv
pamola-core schema FullMaskingOperation           # show parameter schema
pamola-core validate-config --config config.json  # validate a config file
```

Run `pamola-core --help` for the full command list.

---

## Examples

Hands-on notebooks live under [`examples/`](https://github.com/realmdata-ai/PAMOLA/tree/main/examples):

- `examples/anonymization/pseudonymization/` — simple + advanced for hash-based and consistent-mapping pseudonymization
- `examples/anonymization/` — masking, generalization, noise, suppression
- `examples/profiling/` — field-level and dataset-level profilers
- `examples/transformations/` — merge, split, aggregate, clean, impute
- `examples/fake_data/` — fake-value substitution for names, email, phone, organization (replacement values, reproducible when a key is given; not synthetic data)
- `examples/metrics/` — fidelity, privacy, utility metrics
- `examples/data_examples/sample.csv` — non-PII synthetic sample used by the notebooks

> **No real personal data** is included in this repository. All sample records are artificially generated; the origin and licence of every data asset is recorded in [DATA_PROVENANCE.md](DATA_PROVENANCE.md).

---

## Philosophy

- **Operations-first:** Direct transforms with a well-defined 7-step lifecycle, not constraint optimization.
- **Measure everything:** Privacy, fidelity, and utility metrics persisted as JSON artifacts.
- **Reproducibility by default:** Each operation writes a `config.json` (with secrets redacted) alongside its output.
- **Secret hygiene:** `OperationConfig.SENSITIVE_KEYS` provides a single place to declare parameters that must never reach disk.

---

## API Documentation

The project uses **Sphinx** to generate API reference documentation from Python docstrings.

```bash
pip install sphinx sphinx-autodoc-typehints
sphinx-build -b html docs docs/_build    # output: docs/_build/index.html
```

See [docs/README.md](docs/README.md) for build options.

---

## Documentation

| Resource | Link |
|----------|------|
| **PET Knowledge Base** | [realmdata.io/kb](https://realmdata.io/kb/index.html) |
| **Technical Documentation** | [docs/en/index.md](https://github.com/realmdata-ai/PAMOLA/blob/main/docs/en/index.md) |
| **Glossary** | [realmdata.io/glossary](https://realmdata.ai/pages/glossary.html) |
| **Examples** | [`examples/`](https://github.com/realmdata-ai/PAMOLA/tree/main/examples) |
| **Changelog** | [CHANGELOG.md](CHANGELOG.md) |
| **Data provenance & licences** | [DATA_PROVENANCE.md](DATA_PROVENANCE.md) |

---

## Use Cases

- **Data Engineering:** Prepare pseudonymized or masked datasets for ML training pipelines.
- **Healthcare:** Building blocks (masking, generalization, suppression) for de-identification workflows; whether a result meets HIPAA Safe Harbor or Expert Determination is for the data owner to establish.
- **Finance:** Privacy-engineering building blocks that can support PCI/GDPR programs.
- **Compliance:** Structured per-operation artifacts that can serve as technical evidence in a compliance process.
- **Data Sharing:** Risk-assessed data exchange between organizations.

---

## What the terms mean here

Privacy-engineering vocabulary is used loosely in the industry. This is what PAMOLA.CORE 1.0 means by each term, what it provides for it, and what it does **not** claim. The definitions follow the 1.0 specification.

| Term | What it means in CORE | What CORE provides | What CORE does **not** claim |
|---|---|---|---|
| **Anonymization process** | A workflow you run and evidence you collect: profile the data, transform it, measure the residual risk, repeat. | Operations for each step (profiling, generalization, suppression, masking, pseudonymization, noise, metrics) and per-run artifacts (`config.json`, metrics, reports). | That any single operation, or the whole pipeline, makes data anonymous. The decision that a dataset is anonymous belongs to you and your organization, on the evidence you collect. |
| **De-identification** | Removing or replacing direct identifiers (names, e-mails, phone numbers, account numbers) so a record no longer names a person. | Suppression, masking, pseudonymization and generalization operations; profilers that help find direct identifiers and quasi-identifiers. | That removing direct identifiers is enough. Quasi-identifiers (age, geography, dates, occupation) can still single a person out; the k-anonymity profiler and the privacy metrics measure that, they do not prove it away. |
| **Masking** | Hiding all or part of a value's original representation (`FullMaskingOperation`, `PartialMaskingOperation`). | Full masking and partial masking (keep a configurable prefix and suffix, mask the rest). | Irreversibility against a motivated attacker: a masked value that keeps its length, format or a prefix still carries information. |
| **Pseudonymization** | Replacing an identifier with a consistent substitute while keeping the ability to link records. `HashBasedPseudonymizationOperation` (salted hash) and `ConsistentMappingPseudonymizationOperation` (encrypted, reversible mapping file). | Consistent substitutes; for the mapping variant, a mapping file encrypted with a key you supply. | That pseudonymized data is anonymous. It is not: the linkage capability is the point, and the mapping file (or the hash key) is the re-identification key. Protect it as such. |
| **Uniform (non-DP) noise** | Random perturbation drawn from a uniform distribution to reduce precision (`UniformNumericNoiseOperation`, `UniformTemporalNoiseOperation`). | Bounded, seeded perturbation of numeric and temporal fields. | Any differential-privacy guarantee. There is no sensitivity analysis, no epsilon, no privacy budget. Uniform noise reduces precision; it does not give a formal bound on what an attacker learns. |
| **Differential Privacy (DP)** | A formal guarantee, defined by a privacy parameter epsilon, on the output of a randomized mechanism. | Nothing. CORE 1.0 does not ship a differential-privacy mechanism, epsilon accounting or DP training. | Any DP property. Do not describe CORE output as differentially private. |
| **Fake-value substitution** | Replacing values with plausible generated ones (`FakeNameOperation`, `FakeEmailOperation`, `FakePhoneOperation`, `FakeOrganizationOperation`). | Deterministic generators (seeded, reproducible), consistent per source value when requested. | That the substituted dataset is a statistically faithful or anonymous version of the original. It is the original rows with some fields replaced. |
| **Synthetic data** | A new dataset produced by a model trained on the original (GAN, copula, diffusion and similar). | Nothing for generation: CORE 1.0 does not generate synthetic data. It does measure how close a synthetic dataset is to its source (`PrivacyMetricOperation`: DCR, NNDR; `FidelityOperation`; `UtilityMetricOperation`) and a basic linkability measure (CVPL), so you can evaluate data produced by other tools. | That a dataset is private because it is synthetic. The metrics are evidence, not a verdict. |

### Privacy metrics are evidence, not determinations

The metrics in CORE (k-anonymity and uniqueness profilers, distance to closest record, nearest-neighbour distance ratio, fidelity and utility metrics, and the CVPL linkability measure) report measurements. None of them decides that data is legally anonymous, compliant with a regulation, or safe to release, and none of their wording should be read that way. See the [regulatory evidence map](docs/en/core/regulatory_evidence_map.md) for what each measurement does and does not establish.

---

## Regulatory Context

PAMOLA.CORE provides technical building blocks for privacy compliance programs:

| Regulation | Relevant Capabilities |
|------------|----------------------|
| **GDPR** | Pseudonymization (reversible / irreversible) and data-minimization techniques relevant to Art. 25 and 32 |
| **HIPAA** | Masking, generalization and suppression operations that can be used in a de-identification process |
| **CCPA/CPRA** | Data suppression, masking, anonymization workflows |

> **Important:** PAMOLA.CORE provides technical capabilities only. It never certifies that a dataset is anonymous, de-identified, or compliant with GDPR, HIPAA, CCPA/CPRA or any other regime, and a successful run is not such a determination. Privacy metrics are evidence inputs to a regulated process; legal compliance requires organizational policies, procedures, and legal guidance beyond software tools. Pseudonymized data remains personal data under GDPR.

---

## Contributing

```bash
git clone https://github.com/realmdata-ai/PAMOLA.git
cd PAMOLA
pip install -e ".[test]"
pytest tests/ -v
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

---

## Ownership & Licensing

PAMOLA.CORE is developed and maintained by Realm Inveo Inc. (REALMDATA) and distributed under the [MIT License](LICENSE).

---

## Contact

| Purpose | Contact |
|---------|---------|
| **General inquiries** | [contact@realmdata.ai](mailto:contact@realmdata.ai) |
| **Commercial / Sales** | [sales@realmdata.ai](mailto:sales@realmdata.ai) |
| **Due diligence / Legal** | [legal@realmdata.ai](mailto:legal@realmdata.ai) |
| **Website** | [realmdata.ai](https://realmdata.ai) |

---

<p align="center">
  <sub>Built by <a href="https://realmdata.ai">Realm Inveo Inc.</a></sub>
</p>
