Metadata-Version: 2.4
Name: ddharmon
Version: 0.6.0
Summary: Data Dictionary Harmonization Tool: cluster equivalent variables across data dictionaries and anchor them to Common Data Elements (CDEs)
Project-URL: Homepage, https://github.com/Phenome-Health/ddharmon
Project-URL: Repository, https://github.com/Phenome-Health/ddharmon
Project-URL: Documentation, https://github.com/Phenome-Health/ddharmon#readme
Project-URL: Issues, https://github.com/Phenome-Health/ddharmon/issues
Project-URL: Changelog, https://github.com/Phenome-Health/ddharmon/blob/main/CHANGELOG.md
Author-email: Trent Leslie <trent.leslie@phenomehealth.org>, Bhargav Vemuri <bhargav.vemuri@phenomehealth.org>
License: MIT
License-File: LICENSE
Keywords: CDE,biomedical,clustering,common data elements,data dictionary,data harmonization,embeddings,metadata
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.12
Requires-Dist: click>=8.1.0
Requires-Dist: ftfy>=6.1.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: numpy>=1.26.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pandas>=2.1.0
Requires-Dist: pyarrow>=15.0.0
Requires-Dist: python-dotenv>=1.2.1
Requires-Dist: rapidfuzz>=3.0.0
Provides-Extra: all
Requires-Dist: anthropic>=0.39.0; extra == 'all'
Requires-Dist: bertopic>=0.16.0; extra == 'all'
Requires-Dist: faiss-cpu>=1.8.0; extra == 'all'
Requires-Dist: kaleido>=0.2.1; extra == 'all'
Requires-Dist: matplotlib>=3.8.0; extra == 'all'
Requires-Dist: nest-asyncio>=1.6.0; extra == 'all'
Requires-Dist: openai>=1.0.0; extra == 'all'
Requires-Dist: plotly>=5.18.0; extra == 'all'
Requires-Dist: scikit-learn>=1.4.0; extra == 'all'
Requires-Dist: scipy>=1.11.0; extra == 'all'
Requires-Dist: seaborn>=0.13.0; extra == 'all'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'all'
Requires-Dist: umap-learn>=0.5.6; extra == 'all'
Provides-Extra: bertopic
Requires-Dist: bertopic>=0.16.0; extra == 'bertopic'
Provides-Extra: clustering
Requires-Dist: kaleido>=0.2.1; extra == 'clustering'
Requires-Dist: plotly>=5.18.0; extra == 'clustering'
Requires-Dist: scikit-learn>=1.4.0; extra == 'clustering'
Requires-Dist: umap-learn>=0.5.6; extra == 'clustering'
Provides-Extra: dev
Requires-Dist: anthropic>=0.39.0; extra == 'dev'
Requires-Dist: bertopic>=0.16.0; extra == 'dev'
Requires-Dist: black>=24.0.0; extra == 'dev'
Requires-Dist: faiss-cpu>=1.8.0; extra == 'dev'
Requires-Dist: kaleido>=0.2.1; extra == 'dev'
Requires-Dist: matplotlib>=3.8.0; extra == 'dev'
Requires-Dist: nest-asyncio>=1.6.0; extra == 'dev'
Requires-Dist: openai>=1.0.0; extra == 'dev'
Requires-Dist: plotly>=5.18.0; extra == 'dev'
Requires-Dist: pyright>=1.1.350; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: scikit-learn>=1.4.0; extra == 'dev'
Requires-Dist: scipy>=1.11.0; extra == 'dev'
Requires-Dist: seaborn>=0.13.0; extra == 'dev'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'dev'
Requires-Dist: umap-learn>=0.5.6; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: faiss-cpu>=1.8.0; extra == 'embeddings'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'embeddings'
Provides-Extra: llm
Requires-Dist: anthropic>=0.39.0; extra == 'llm'
Requires-Dist: openai>=1.0.0; extra == 'llm'
Provides-Extra: notebooks
Requires-Dist: anthropic>=0.39.0; extra == 'notebooks'
Requires-Dist: bertopic>=0.16.0; extra == 'notebooks'
Requires-Dist: faiss-cpu>=1.8.0; extra == 'notebooks'
Requires-Dist: ipywidgets>=8.0.0; extra == 'notebooks'
Requires-Dist: jupyter>=1.0.0; extra == 'notebooks'
Requires-Dist: jupyterlab>=4.0.0; extra == 'notebooks'
Requires-Dist: kaleido>=0.2.1; extra == 'notebooks'
Requires-Dist: matplotlib>=3.8.0; extra == 'notebooks'
Requires-Dist: nest-asyncio>=1.6.0; extra == 'notebooks'
Requires-Dist: openai>=1.0.0; extra == 'notebooks'
Requires-Dist: plotly>=5.18.0; extra == 'notebooks'
Requires-Dist: scikit-learn>=1.4.0; extra == 'notebooks'
Requires-Dist: scipy>=1.11.0; extra == 'notebooks'
Requires-Dist: seaborn>=0.13.0; extra == 'notebooks'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'notebooks'
Requires-Dist: umap-learn>=0.5.6; extra == 'notebooks'
Provides-Extra: viz
Requires-Dist: matplotlib>=3.8.0; extra == 'viz'
Requires-Dist: scipy>=1.11.0; extra == 'viz'
Requires-Dist: seaborn>=0.13.0; extra == 'viz'
Description-Content-Type: text/markdown

# ddharmon — Data Dictionary Harmonization Tool

ddharmon harmonizes biomedical **data dictionaries**: it identifies clusters of equivalent
variables across studies and recommends a Common Data Element (CDE) anchor for each — routing
every recommendation to expert review.

![ddharmon in the Phenome Health data-harmonization ecosystem](docs/ph-ecosystem-v1.png)

*Where ddharmon sits: cluster variables + CDEs → anchor a CDE per sub-cluster (or generate a novel
one) → expert review — alongside related public tools and the broader Phenome Health stack.*

## Quick start

Requirements: **Python 3.12+** and [uv](https://github.com/astral-sh/uv).

```bash
git clone https://github.com/Phenome-Health/ddharmon.git
cd ddharmon
uv sync --extra all
cp .env.example .env        # set ANTHROPIC_API_KEY for the classify pass (sync/batch)
```

Then open the end-to-end notebook **`notebooks/clustering/v1_harmonization_pipeline.ipynb`** — it
runs the full pipeline (ingest → embed → cluster → value sub-cluster → CDE anchor → adopt/refine/novel
→ review) on example data and is the best place to start.

> The NIH CDE catalog is not bundled. To anchor against CDEs, flatten the CDE repository locally with
> `scripts/flatten_cde_repo.py <All-CDEs.json> <out.tsv>`; without it, the pipeline still clusters and
> sub-clusters cohort variables (`cdeSet = none`).

## Related work

Framing biomedical variable/CDE harmonization as an **embedding → clustering → optional LLM**
problem is an active line of work; ddharmon builds directly on it. The closest tools (full lineage
and citations in [`docs/v1_methods.md`](docs/v1_methods.md)):

- **CDEMapper** (Wang et al., *JAMIA* 2025) — LLM-powered mapping of local data elements to NIH CDEs
  via semantic indexing + BM25 + GPT candidates + human review. *Per-element lookup.*
- **Krishnamurthy et al., 2025** (arXiv:2506.02160) — embeds ~24k NIH CDEs and clusters them with
  HDBSCAN, then LLM-labels the clusters. *Clusters the target CDE repository.*
- **Salimi et al., 2025** (*Sci Rep*) — the **PASSIONATE** Parkinson's variable-mapping ground truth;
  shows language-model embeddings beat fuzzy string matching for **pairwise** cohort harmonization
  (frames matching as a clustering task, but stops at t-SNE visualization).
- **DataTecnica — DIVER / RoP** (Long et al., medRxiv 2024) — LLMs generate and audit CDEs at scale
  (**GenCDEs**); the RoP release ships ~1.33M harmonized CDEs with embeddings. *Closest to generating
  novel CDEs.*
- **Harmony** (McElroy et al., *BMC Psychiatry* 2024), **Semantic Search Helper** (Gottfried 2025),
  **datastew**, and **BDI-Kit** (Lopez et al., *Patterns* 2026) — embedding/LLM harmonization and
  schema/value-matching siblings.

## What ddharmon adds

These tools tend to cluster *either* a CDE repository *or* a single cohort, use a single semantic
vector, and map element-by-element. ddharmon targets the gaps that matter for harmonizing **many
studies at once**:

- **Multi-cohort, source *and* target together.** Cohort variables and the NIH CDE catalog are
  embedded and clustered in one space, so equivalent variables across many studies surface as a
  single cluster — not one pairwise lookup at a time.
- **Dual-vector, value-aware sub-clustering.** Each field gets a **semantic** vector (what it
  measures) *and* a **value** vector (how it is answered). Semantic vectors cluster the concept;
  **value vectors sub-cluster within a concept by encoding shape** — so "age in years" and "age
  bracket" land in different sub-clusters. Antecedents use a single vector and flat clusters.
- **A recommended CDE anchor per sub-cluster — with a novel-CDE path.** Each value sub-cluster is
  anchored to the best in-cluster CDE (medoid → canonicalness → metadata richness). When **no
  existing CDE fits**, the sub-cluster is flagged for a **generated (GenCDE) novel CDE** instead of
  being forced onto a poor match.
- **adopt / refine / novel → expert review.** One classify-only LLM call per sub-cluster proposes
  adopt / refine / novel against the anchor and routes every recommendation to **expert-in-the-loop
  (EITL)** review — nothing is auto-applied.

## How it works

```
ingest (cohorts + CDE)
  → dual-vector embed (semantic + value)
    → semantic cluster (BERTopic)
      → value sub-cluster (HDBSCAN on value vectors, per topic)
        → CDE anchor per sub-cluster (medoid → best in-cluster CDE; GenCDE fallback)
          → adopt / refine / novel  (single classify-only LLM call)
            → EITL review queue
```

Full algorithm, parameters, and lineage are in [`docs/v1_methods.md`](docs/v1_methods.md). The
research contributions held for a pending publication — LLM coherence judging, concept labeling,
transformation-spec authoring, granularity-loss detection, deep recursive clustering, and a CDE
common data model — are **not in v1**; see [`CHANGELOG.md`](CHANGELOG.md).

## Installation

The core install is lightweight; optional extras unlock additional capabilities.

| Extra | Use case |
|-------|----------|
| *(none)* | Core ingestion + lexical matching |
| `embeddings` | sentence-transformers + faiss-cpu — semantic embedding + vector search |
| `clustering` | scikit-learn, UMAP, plotly |
| `bertopic` | BERTopic topic modeling |
| `llm` | openai, anthropic — LLM classify / rerank |
| `all` | everything above (required to run the full pipeline + notebook) |
| `dev` | `all` + pytest, ruff, black, pyright |

```bash
pip install "ddharmon[all]"          # full pipeline
# or, with uv (recommended for development):
uv sync --extra all
```

## Development

```bash
./scripts/check.sh     # lint, format, typecheck, test
./scripts/fix.sh       # auto-fix lint + format
pytest                 # tests
```

### Project structure

```
src/ddharmon/
├── models/          # data models (plain dataclasses)
├── ingestion/       # multi-cohort + CDE dictionary parsers
├── embedding/       # dual-vector embedding (semantic + value), SQLite cache
├── clustering/      # BERTopic semantic clustering + value sub-clustering
├── harmonization/   # v1: CDE anchoring + adopt/refine/novel + EITL export
├── matching/        # pairwise 1:1 matching (built; not in the v1 surface)
├── llm/             # LLM clients + Anthropic Batch API
├── values/          # value-encoding parsing
└── export/          # visualization (dendrograms, UMAP, Plotly)
notebooks/clustering/v1_harmonization_pipeline.ipynb
scripts/             # flatten_cde_repo, build_clsa_csv, prompt runners, check/fix
tests/
```

## Roadmap (beyond v1)

- **Web GUI** — under development; the point-and-click app (upload dictionaries → run with live
  progress → review recommendations → export) will ship as added functionality of **biomapper-ui**.
- Pairwise **1:1 mapping** as a first-class surface (the engine is built).
- **Standards mapping** (LOINC / SNOMED / OMOP).
- LLM coherence judging, concept labeling, and transformation-spec authoring (publication-pending).

## License

MIT — see [LICENSE](LICENSE). © 2026 Phenome Health.
