Metadata-Version: 2.5
Name: py-phenoforge
Version: 0.1.0
Summary: Semantic value set assembly for clinical cohort definitions, over MCP
Project-URL: Homepage, https://github.com/colbyw5/phenoforge
Project-URL: Repository, https://github.com/colbyw5/phenoforge
Project-URL: Issues, https://github.com/colbyw5/phenoforge/issues
Author: Colby Wilkinson
License: Apache-2.0
License-File: LICENSE
Keywords: clinical-research,icd-10-cm,mcp,omop,phenotype
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: duckdb>=1.0
Requires-Dist: httpx>=0.27
Requires-Dist: lancedb>=0.13
Requires-Dist: mcp>=1.2
Requires-Dist: pydantic>=2.7
Requires-Dist: rank-bm25>=0.2.2
Requires-Dist: sentence-transformers>=3.0
Requires-Dist: typer>=0.12
Provides-Extra: agent
Requires-Dist: langchain-anthropic>=0.2; extra == 'agent'
Requires-Dist: langgraph>=0.2; extra == 'agent'
Requires-Dist: langsmith>=0.1; extra == 'agent'
Requires-Dist: python-dotenv>=1.0; extra == 'agent'
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: ipykernel>=6.29; extra == 'dev'
Requires-Dist: ipywidgets>=8.0; extra == 'dev'
Requires-Dist: jupyterlab>=4.0; extra == 'dev'
Requires-Dist: matplotlib>=3.8; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# phenoforge

[![CI](https://github.com/colbyw5/phenoforge/actions/workflows/ci.yml/badge.svg)](https://github.com/colbyw5/phenoforge/actions/workflows/ci.yml)

> Semantic value set assembly for clinical cohort definitions, over MCP.

**Status: early prototype.** The vocabulary loader, hierarchy expansion, hybrid (BM25 + dense)
search, curated OHDSI Phenotype Library matching, an eval harness scoring all of it against
curated ground truth, a thin MCP server exposing them all, and a LangGraph agent that decomposes
a population description, checks curated first, and pauses for human confirmation before
including anything generated — all work end-to-end against a real Athena download. See Roadmap.

## Try it now, no setup

```bash
git clone https://github.com/colbyw5/phenoforge && cd phenoforge
uv run python scripts/run_demo.py "diabetic nephropathy"   # or: make demo
```

Runs real hybrid search and hierarchy expansion against a small, bundled, redistributable
ICD-10-CM-only demo vocabulary (see `demo/README.md`) — no Athena account, no API key, no model
download. It only exercises the `generated` tier (BM25 + hierarchy expansion); the `curated`
tier, dense retrieval, and the full agent need the real Setup below.

## Setup

Published to PyPI as `py-phenoforge` (the bare `phenoforge` name is taken by an unrelated
package) — `pip install py-phenoforge` or `uv add py-phenoforge` gets you the library and the
`phenoforge-mcp` CLI, importable as `phenoforge`. That alone isn't runnable yet, though: every
tool needs the vocabulary database built below from your own Athena download, which can't be
redistributed on PyPI.

Requires your own [OHDSI Athena](https://athena.ohdsi.org/) bulk download — vocabulary content
carries its own license terms, so it can't be bundled with this repo.

1. Create a free account at [athena.ohdsi.org](https://athena.ohdsi.org/).
2. Under Search, select at least **ICD10CM** and **SNOMED** as vocabularies, then click
   Download. SNOMED is used locally only — to resolve curated-set matching (`find_curated_definition`)
   — and is never shipped or exposed through any tool surface.
3. Unzip the download into `data/athena/` (this directory is gitignored — nothing under `data/`
   is ever committed or shipped). It contains OMOP CDM vocabulary tables (`CONCEPT.csv`,
   `CONCEPT_RELATIONSHIP.csv`, etc.) — Athena ships vocabulary content pre-shaped as OMOP, so no
   separate mapping step is needed.

```bash
uv sync
python scripts/load_vocab.py data/athena --output data/vocab.duckdb
phenoforge-mcp  # stdio MCP server; add to Claude Desktop's mcpServers config to try it
```

Or, once the Athena download is in place, `make setup` chains install + every optional data-build
step below (vocab, phenotype library, dense index) in one call. `make check` runs lint, type
checks, and the full test suite — the same checks CI runs on every push. See the `Makefile` for
the full target list.

### Optional: curated phenotype library

`find_curated_definition` needs a small local cache of OHDSI Phenotype Library cohort
definitions. Like the Athena download, this content carries its own terms — the OHDSI
PhenotypeLibrary GitHub repo has no confirmed LICENSE file (its R package `DESCRIPTION` claims
Apache, but that's a manifest claim, not a verified grant for the cohort content itself) — so
it's fetched into a gitignored local directory, never committed to this repo. Without this step,
`find_curated_definition` still runs and reports why nothing matched.

```bash
python scripts/fetch_phenotype_library.py  # writes data/phenotype_library/
```

Bundles 8 hand-picked, diabetes/kidney-relevant cohorts (not the full library): Type 2/Type 1/
gestational diabetes, diabetic ketoacidosis, retinopathy, and chronic kidney disease.

### Optional: dense (semantic) search

`search_concepts` fuses lexical and semantic matching when a dense index has been built;
without one, it falls back to lexical-only search automatically.

```bash
python scripts/build_index.py  # writes data/concept_index.lance; downloads BioLORD-2023 on first run
```

### Optional: eval harness

Scores each retrieval method (BM25, dense, hybrid, hierarchy expansion) against the curated
demo cohorts as ground truth — hierarchical distance-weighted scoring, set-level coverage, and
an over-inclusion penalty (partial credit for near-misses under the same hierarchy parent, not
exact-match recall). Requires the phenotype library fetch step above; dense/hybrid scoring also
needs the built index.

```bash
python scripts/run_eval.py                                 # bm25 + expand_descendants
python scripts/run_eval.py --index data/concept_index.lance # + dense + hybrid
```

### Optional: the agent

The LangGraph agent decomposes a plain-English population description into seed terms, checks
each against the curated phenotype library first, and pauses on the command line for you to
accept or reject any generated (unverified) candidates before they're included. Requires an
Anthropic API key (decomposition is a real Claude call) and the `agent` extra.
`scripts/run_agent.py` loads a local `.env` automatically (gitignored — never commit real keys),
or export the variable directly.

```bash
uv sync --extra agent
echo 'ANTHROPIC_API_KEY=...' > .env   # or: export ANTHROPIC_API_KEY=...
python scripts/run_agent.py "adults with type 2 diabetes and diabetic nephropathy"
python scripts/run_agent.py "..." --index data/concept_index.lance  # + dense retrieval for generated candidates
```

### Interactive exploration

Both notebooks are exploration only, never pushed to production — reusable logic stays in
`src/phenoforge/`.

- `notebooks/explore.ipynb` calls the engine directly (no MCP transport) against your real
  built `data/vocab.duckdb`.
- `notebooks/evaluate.ipynb` runs the eval harness and walks through the metrics with
  explanatory text, a method-comparison chart, and a sortable per-cohort results table.

```bash
uv sync --extra dev
jupyter lab notebooks/explore.ipynb
```

## What it does

Turns a plain-English patient population description into a defensible set of ICD-10-CM codes,
where every code carries provenance — whether it came from a peer-reviewed phenotype
definition, from hierarchy expansion, or from semantic retrieval that a human should check.

```
"adults with type 2 diabetes and diabetic nephropathy"
  → decomposes into seed clinical terms
  → checks OHDSI Phenotype Library for a validated definition for each
  → falls back to hybrid retrieval for terms with no curated match,
    pausing for human confirmation before including anything generated
  → returns a ConceptSet with per-code provenance and citations
```

Run it for real: `python scripts/run_agent.py "adults with type 2 diabetes and diabetic
nephropathy"` (see Setup).

## Why not an existing terminology server

Several good MCP terminology servers exist. They solve *lookup* — "what is code X", "map X to
Y". This solves *set assembly*, which is the actual task in cohort definition. It also covers
US ICD-10-CM, which the existing servers do not, and uses semantic retrieval rather than
proxied keyword search, which fails when a population description and a code description share
no vocabulary.

## Architecture

Three layers — a retrieval engine, a thin MCP server, and a LangGraph agent. The MCP server
and the agent are independent consumers of the same engine.

## Roadmap

- [x] **v0.1 — vocabulary layer.** Athena loader, DuckDB schema, hierarchy queries
- [x] **v0.2 — retrieval.** BM25, BioLORD-2023 embeddings, LanceDB index, RRF hybrid scoring
- [x] **v0.3 — expansion + provenance.** `ConceptSet` model, descendant expansion, and OHDSI PL
      curated matching (small hand-picked demo set — see Setup)
- [x] **v0.4 — MCP server.** stdio transport, four tools (see below), Claude Desktop config
- [x] **v0.5 — eval harness.** Distance-weighted scoring, coverage, over-inclusion penalty;
      curated demo cohorts as ground truth. Decomposition accuracy deferred to `v0.7` (see
      `phenoforge.eval`) — nothing decomposes a population description yet
- [x] **v0.7 — LangGraph agent.** Decompose → check curated first → generate only for
      unresolved terms → human confirmation gate on anything generated → assemble. Real
      interactive CLI (`scripts/run_agent.py`)
- [x] **v1.0 — packaging.** One-command setup (`make setup`), CI (lint/typecheck/test on every
      push), a PyPI release pipeline (tag-triggered, trusted publishing) as `py-phenoforge`, a
      zero-setup demo (`make demo`, no account/key/download needed), and a validation and
      limitations section (see below)

Skipped: `v0.6` (encoder benchmark — BioLORD vs SapBERT vs MedCPT). The agent was the more
demonstrable deliverable, so `v0.7` was built first; the encoder benchmark may return later.

Deferred: literature-derived phenotype algorithms (`published` tier), RxNorm and LOINC domains,
hosted API reference docs (Sphinx/mkdocs) — docstrings are complete throughout `src/`, just not
yet published anywhere

Watching: [TypeSafe AI's Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev), a
schema-constrained "System One Model" that could fit the `decompose` node's structured-output
call. Early access/waitlist, proprietary, no independent benchmarks yet as of 2026-10 — not
adopted; revisit if it leaves early access and the latency/cost tradeoff is verified
independently.

## Tools

| Tool | Status | Purpose |
|------|--------|---------|
| `lookup_concept` | done | Exact ICD-10-CM code → name and metadata |
| `expand_hierarchy` | done | Seed code → full descendant expansion (`generated` provenance) |
| `search_concepts` | done | Hybrid BM25 + dense search over ICD-10-CM names, RRF-fused (`generated` provenance); falls back to BM25-only if no dense index is built |
| `find_curated_definition` | done | Search the bundled OHDSI Phenotype Library demo set before generating anything (`curated` provenance) |
| `explain_inclusion` | planned | Why is this code in this set, via which path, with what evidence |

`find_curated_definition` is the only source of `curated` provenance today, and only for the 8
bundled demo cohorts. Everything from `expand_hierarchy` and `search_concepts` is `generated`
provenance — ungrounded, structural or lexical/semantic only, and meant to be confirmed by a
human before use in a cohort definition.

## Validation and limitations

The only quantitative evaluation that exists is `scripts/run_eval.py` scoring each retrieval
method against the 8 bundled OHDSI Phenotype Library demo cohorts (diabetes/kidney-relevant) as
ground truth. Current numbers on that set:

| method | coverage | over-inclusion | hierarchical |
|---|---|---|---|
| `bm25` | 0.625 | 0.367 | 0.535 |
| `expand_descendants` | 0.645 | 0.138 | 0.697 |
| `dense` | 0.850 | 0.528 | 0.586 |
| `hybrid` (0.5/0.5) | 0.858 | 0.399 | 0.692 |

What this does and doesn't establish:

- **8 cohorts, one disease area.** This is not a general benchmark. It says nothing about
  performance outside diabetes/kidney conditions, rare diseases, or codes with sparse/ambiguous
  natural-language descriptions. Treat every number above as specific to this narrow slice, not
  as a claim about ICD-10-CM retrieval in general.
- **Over-inclusion is the dominant error mode, not coverage.** Every method pulls in more
  codes than the curated ground truth (dense especially: 0.528 over-inclusion against 0.850
  coverage) — the system errs toward casting a wide net and relying on human review to reject
  false positives, not toward silently missing codes. Anyone using the `generated` tier should
  expect to reject a meaningful fraction of what it returns, not rubber-stamp it.
- **Decomposition has no quantitative evaluation at all.** The agent's `decompose` node (a real
  Claude call turning a population description into seed terms) has been exercised on exactly
  one worked example end-to-end (see `notebooks/agent_walkthrough.ipynb`), not benchmarked
  against a labeled set of population descriptions. Its error rate on descriptions unlike that
  example — ambiguous phrasing, multiple unrelated conditions, qualifiers it should drop — is
  unknown.
- **Curated-tier coverage is narrow by construction.** Only 8 cohorts are bundled; any
  population description outside them falls through entirely to the unverified `generated`
  tier. Separately, SNOMED-to-ICD-10-CM resolution deliberately drops concepts whose fan-out
  exceeds a configurable threshold (`find_high_fanout_snomed_concepts`) rather than including
  everything — a real cohort definition can lose codes this way, and the demo cohorts above may
  not exercise that path.
- **No scale or concurrency testing.** Built and measured against one developer's own Athena
  download on one machine. No data on index build time, query latency, or memory at a larger
  vocabulary, or under concurrent MCP clients.
- **English, US ICD-10-CM only.** No other language, no RxNorm/LOINC/other domains (see Roadmap).
- **The human-confirmation gate has no audit trail.** `confirm` pauses for review, but nothing
  records who approved what or persists decisions across runs — each CLI invocation starts a
  fresh thread with no memory of past confirmations.

## Releasing

`.github/workflows/publish.yml` builds and publishes to PyPI on any `v*.*.*` tag push, via
[PyPI trusted publishing](https://docs.pypi.org/trusted-publishers/) (OIDC — no stored API
token). One-time setup, done outside this repo:

1. On PyPI: Account settings → Publishing → add a pending publisher for `py-phenoforge`,
   repository `colbyw5/phenoforge`, workflow `publish.yml`, environment `pypi`.
2. On GitHub: Settings → Environments → create an environment named `pypi` (optionally with
   required reviewers, for a manual approval gate before each publish).

To cut a release: bump `version` in `pyproject.toml`, commit, then `git tag v0.1.0 && git push
origin v0.1.0`.

## License

Apache 2.0. Vocabulary content (ICD-10-CM, SNOMED) carries its own license terms from OHDSI/
Athena, separate from this repo's license — nothing from `data/` is redistributed.

## Not a clinical decision tool

This produces code sets for research and analytics. It does not make clinical determinations
and has not been validated for patient care.
