Metadata-Version: 2.4
Name: profilefoundry
Version: 1.0.0
Summary: An open-source generator for structured, internally-consistent, linked, temporally coherent synthetic Person Objects.
Author: Sriram Selvam
License-Expression: LicenseRef-ProfileFoundry-Citation-1.0
Project-URL: Homepage, https://github.com/selvamsriram/ProfileFoundry
Project-URL: Issues, https://github.com/selvamsriram/ProfileFoundry/issues
Keywords: synthetic-data,pii,person-object,linkage,evaluation,privacy
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.6
Requires-Dist: faker>=24.0
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.1
Requires-Dist: pyarrow>=15.0
Requires-Dist: click>=8.1
Requires-Dist: rapidfuzz>=3.6
Requires-Dist: pybloom-live>=4.0
Requires-Dist: requests>=2.31
Requires-Dist: tqdm>=4.66
Requires-Dist: python-dotenv>=1.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: ipython>=8.20; extra == "dev"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.8; extra == "viz"
Requires-Dist: seaborn>=0.13; extra == "viz"
Requires-Dist: plotly>=5.20; extra == "viz"
Provides-Extra: hf
Requires-Dist: datasets>=2.18; extra == "hf"
Requires-Dist: huggingface-hub>=0.23; extra == "hf"
Dynamic: license-file

# ProfileFoundry

> A structured, internally-consistent, linked, temporally coherent synthetic
> Person Object - and an open-source generator that produces them at scale.

This repository is the implementation home for ProfileFoundry, which builds
on the problem setting exposed by [PANORAMA](https://arxiv.org/abs/2505.12238)
(Selvam & Ghosh, 2025). It provides a deterministic generator, schema,
validation suite, release reports, and Hugging Face export tooling for the
`ProfileFoundry-Core-100K` dataset.

The ACL/NeurIPS-style paper draft lives in [`Paper/`](Paper/).
The vetted 100K release package is staged for Hugging Face at
[`srirxml/ProfileFoundry-Core-100K`](https://huggingface.co/datasets/srirxml/ProfileFoundry-Core-100K);
run `python scripts/verify_hf_release_current.py` with an `HF_TOKEN` before
calling the remote artifact current. The default HF viewer table is
`person_objects.parquet`, a complete one-row-per-person view with nested
sections encoded as JSON strings; the remaining parquet files expose the
normalized relational schema.

## Status (2026-05-24) — v1.0 release sprint

| Phase | Status |
|---|---|
| **0 — Foundation** (schema and package scaffolding) | ✅ complete |
| **1 — Reference data** (manifest + bootstrap + loader) | ✅ ref data shipped for 5 validation locales |
| **2 — Generator factory** (port + joint constraints + age-gating) | ✅ deterministic across processes (set-iteration bug fixed) |
| **3 — Linkage** (households, employers, family graph) | ✅ household orchestrator + family edges + employer pool |
| **4 — Temporal** (event taxonomy, backfill, replay) | ✅ replay-valid non-credit timeline + typed payload export |
| **5a — Validation** (KS gaps, consistency) | ✅ disclosed KS gaps (max 0.124) · 100% consistency |
| **5b — Leakage audit** (Wikidata Bloom, reserved-domain email audit, self-collision) | ✅ Wikidata Bloom + self-collision + reserved-domain email syntax audit |
| **6 — Scale + publish** (100K + HF export) | ✅ 100K release · complete viewer table · normalized parquet star schema · MANIFEST.json |
| **7 — Reproducibility** (verify fixture, manifest hash) | ✅ normalized fixture stable across processes; verify script + fixture |
| **8 — Paper draft** | ✅ submission draft revised |
| **9 — HF push + tag v1.0.0** | ⏳ HF upload target documented; remote current check requires `HF_TOKEN`; tag pending |

## Quick start

```bash
pip install -e ".[dev]"
profilefoundry verify                # smoke-test all 8 locales
profilefoundry person --locale US    # one US Person as JSON
profilefoundry household --locale US # one linked household (multiple Persons)
profilefoundry scale --n 1000 --locale UK --out /tmp/uk.jsonl
profilefoundry scale-households --n 500 --locale CA --out /tmp/ca_households.jsonl
profilefoundry validate --n 300                     # KS gaps + leakage + consistency
profilefoundry export --out /tmp/pf_core --n-per-locale 1000 --generation-date 2026-05-24 --exported-at 2026-05-24 --skip-hibp  # normalized parquet star schema
profilefoundry scale-smoke --sizes 1000,10000,100000           # timings

# Full v1.0 release run (100K profiles in ~4-5 minutes on a laptop):
python scripts/run_full_core.py --generation-date 2026-05-24 --skip-hibp --verbose
# Reproducibility check:
python scripts/verify_reproducibility.py
# Hugging Face upload preflight / push:
python scripts/push_to_hf.py --dry-run
python scripts/push_to_hf.py --repo srirxml/ProfileFoundry-Core-100K
# Confirm the public HF manifest matches the vetted local release:
python scripts/verify_hf_release_current.py
# Build the Wikidata bloom (multi-hour due to WDQS rate limit):
python scripts/ingest_wikidata_persons.py --scope sample --sleep 65 --verbose
```

Or without installing:

```bash
PYTHONPATH=src python3 -m profilefoundry.cli scale --n 1000 --locale US --out /tmp/us.jsonl
```

## CLI reference

All commands are available through `profilefoundry` after installation, or via
`PYTHONPATH=src python3 -m profilefoundry.cli` from a checkout.

| Command | Purpose | Common example |
|---|---|---|
| `profilefoundry verify` | Generate one profile per supported locale as a smoke test. | `profilefoundry verify` |
| `profilefoundry person` | Print one deterministic Person Object as JSON. | `profilefoundry person --locale US --seed 4321 --profile-seq 1` |
| `profilefoundry household` | Print one linked household as JSON. | `profilefoundry household --locale UK --seed 4321 --seq 7` |
| `profilefoundry scale` | Generate flat JSONL profiles for one locale. | `profilefoundry scale --n 1000 --locale CA --out /tmp/ca.jsonl` |
| `profilefoundry scale-households` | Generate linked household members as JSONL. | `profilefoundry scale-households --n 500 --locale AU --out /tmp/au_households.jsonl` |
| `profilefoundry validate` | Run distributional, leakage, replay, and consistency checks. | `profilefoundry validate --n 300 --locales US,UK,IN --skip-hibp` |
| `profilefoundry export` | Write JSONL, parquet tables, manifest, and dataset card. | `profilefoundry export --out /tmp/pf_core --n-per-locale 1000 --generation-date 2026-05-24 --exported-at 2026-05-24T00:00:00Z --skip-hibp` |
| `profilefoundry scale-smoke` | Time the generator at several sizes. | `profilefoundry scale-smoke --sizes 1000,10000,100000` |

Script-level release utilities:

| Script | Purpose |
|---|---|
| `python scripts/run_full_core.py --generation-date 2026-05-24 --skip-hibp --verbose` | Rebuild the 100K release artifact under `data/raw/profilefoundry-core-v1/` and reports under `reports/v1_release/`. |
| `python scripts/verify_reproducibility.py` | Confirm the pinned reproducibility fixture still matches. |
| `python scripts/verify_package_reference_data.py` | Confirm reference data is available from an installed wheel. |
| `python scripts/push_to_hf.py --dry-run` | Preview the Hugging Face upload. |
| `python scripts/push_to_hf.py --repo srirxml/ProfileFoundry-Core-100K` | Upload the vetted release artifact. |
| `python scripts/verify_hf_release_current.py` | Compare the local release manifest with the Hugging Face dataset. Requires `HF_TOKEN` for private or gated repos. |
| `python scripts/export_schema.py --check` | Verify the checked-in JSON Schema matches the Pydantic model. |
| `python scripts/ingest_us_acs.py` / `python scripts/ingest_uk_ons.py` | Refresh reference-data inputs from upstream sources. |

## Project layout

```
.
├── README.md                  # this file
├── pyproject.toml             # package metadata, deps, ruff/pytest config
├── notes/
│   ├── BUGS.tsv               # resolved/known issue ledger
│   └── RELEASE_PLAN.md        # v1.0 release plan
├── schemas/
│   ├── README.md
│   └── person_v0_1.schema.json  # auto-exported JSON Schema
├── src/profilefoundry/
│   ├── schema/                # Pydantic Person Object v0.1 (source of truth)
│   ├── data/                  # reference-data loader (bootstrap ↔ derived)
│   ├── generate/              # factory + sampling + locale providers
│   ├── linkage/               # (phase 3) households, employers, families
│   ├── validate/              # (phase 5) invariants, distributions, leakage
│   ├── io/                    # (phase 6) Hugging Face export
│   └── cli.py                 # `profilefoundry` entry point
├── data/reference/
│   ├── MANIFEST.md            # every external source enumerated
│   └── bootstrap/             # committed minimum reference marginals
├── scripts/
│   ├── export_schema.py       # regenerate JSON Schema
│   └── ingest_us_acs.py       # pull richer ACS tables live (needs API key)
├── tests/                     # pytest invariants & unit tests
└── Paper/                     # ACL-style paper draft + figures
```

## Testing

```bash
PYTHONPATH=src python3 -m pytest tests/
```

The full pytest suite covers schema invariants (Pydantic model), reference-data
loading, factory smoke checks, determinism, plausibility (age, employment,
addresses, education), **adults-only enforcement** (no profiles under 18),
linkage (households, employers, family graph), validation (KS gaps, replay,
consistency, leakage), release-report freshness, normalized export quality, and
reproducibility.

## License

- Code and SDK: **ProfileFoundry Citation License 1.0** (see `LICENSE`).
  Public uses and redistributions must cite the ProfileFoundry paper when
  available, or this repository until then. Machine-readable citation metadata
  is in `CITATION.cff`.
- Generated dataset: **CC-BY-4.0**.
- Embedded reference data retains its upstream license; see
  `data/reference/MANIFEST.md`.
