Metadata-Version: 2.4
Name: hponorm
Version: 1.1.0
Summary: Interactively validate and normalise GA4GH phenopacket phenotypic-feature terms against the HPO ontology
Author-email: Tim Hearn <tjh70@cam.ac.uk>
License-Expression: MIT
Project-URL: Homepage, https://github.com/comparativechrono/hponorm
Project-URL: Issues, https://github.com/comparativechrono/hponorm/issues
Project-URL: Changelog, https://github.com/comparativechrono/hponorm/blob/main/CHANGELOG.md
Keywords: HPO,Human Phenotype Ontology,phenopackets,GA4GH,ontology,normalisation
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Healthcare Industry
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: recommended
Requires-Dist: rapidfuzz>=3.0; extra == "recommended"
Requires-Dist: rich>=13.0; extra == "recommended"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# hponorm

Interactively **validate and normalise the phenotypic-feature terms** in GA4GH
phenopacket / family JSON files against the **HPO** (Human Phenotype Ontology).

It's the sibling of `mondonorm` (which does disease terms → MONDO) and shares the
same look and feel. This one targets `phenotypicFeatures[].type` and HPO.

If your pedigree conversion currently emits phenotype terms with an empty `id` and a
free-text label:

```json
"phenotypicFeatures": [ { "type": { "id": "", "label": "long qt interval" } } ]
```

`hponorm` walks the file, finds every phenotypic feature, suggests the most
likely HPO terms for each label, lets you pick one (or type an HP id yourself),
and writes a normalised copy:

```json
"phenotypicFeatures": [ { "type": { "id": "HP:0001657", "label": "Prolonged QT interval" } } ]
```

## How suggestions are sourced

Two interchangeable backends; pick with `--backend`:

| Backend | What it uses | Network |
|---------|--------------|---------|
| `ols`   | EBI Ontology Lookup Service v4 (live HPO) | required |
| `local` | A local `hp.obo` / `hp.json`, downloaded + cached on first use | only for the one-time download |
| `auto`  | OLS if reachable, otherwise local (the default) | preferred |

The official ontology file can be fetched automatically (from
`https://purl.obolibrary.org/obo/hp.obo`) or supplied with `--hpo-file`.
A tiny **illustrative** offline sample (`hponorm/data/hpo-sample.obo`, 8 terms)
ships with the package so you can try the tool with no network — it is **not** a
substitute for the real ontology. In Python its path is `hponorm.SAMPLE_OBO`.

If OLS does not answer (no connection, a time-out, or an error from EBI's
server), hponorm says so, rather than showing "no suggestions" as if the name
had no match. In `auto` mode, when a copy of the HPO is already on the
computer, the suggestions then come from that copy, and hponorm says that too.

## How suggestions are scored

Both backends score a suggestion in the same way, from 0 to 100:

* **100**: the name is the term's label, once case, accents, possessives and
  punctuation are set aside ("long-QT interval" and "Long QT interval" are the
  same name, as are "Down's syndrome" and "Down syndrome");
* **98**: the name is one of the term's synonyms;
* **97**: the same words in another order;
* **96 or less**: otherwise, how closely the name matches the label or the
  closest synonym. This is the mean of how alike they are letter by letter and
  the share of their words they have in common, where a plural, a word with an
  ending ("abnormal", "abnormality") or a close spelling of a longer word
  ("anaemia", "anemia") counts as nearly the same word. So "learning
  disability" puts *Specific learning disability* (80) before *Dyslexia*,
  whose synonym "Reading disability" scores 68, and a name inside a longer one
  is not taken for it: "hearing loss from birth" scores 74 against *Hearing
  impairment*, whose synonym is "Hearing loss".

Names that are alike letter by letter but are not the same thing are scored
by the words they share alone, so they never look like close spellings of each
other:

* words that differ by their prefixes: "hypertension" and "hypotension",
  "microcephaly" and "macrocephaly", "bradycardia" and "tachycardia",
  "polyhydramnios" and "oligohydramnios";
* a word and the same word with a prefix added ("hypothyroidism",
  "hypoparathyroidism") or a negating one ("symptomatic", "asymptomatic";
  "syndromic", "nonsyndromic"; "normal", "abnormal");
* names of different types or numbers: "type 1" and "type 2", "long QT
  syndrome 1" and "2", "hepatitis A" and "B". "Type I" and "type 1" are the
  same.

So "Maternal hypothyroidism" scores 25 against *Maternal hyperthyroidism* (in
1.0's local search it scored 93.6), and "Hypotension" 0 against *Hypertension*.
British and American spellings are still the same name: "Oesophageal atresia"
and "Esophageal atresia" score 96.

OLS returns its results in its own order of relevance; hponorm scores each of
them as above and shows the closest first. (Version 1.0 scored OLS's results
by their place in that order, 100, 96, 92 and so on, so the first always looked
like an exact match.) A threshold therefore means the same whichever backend
found the term. [phenopacket-toolkit](https://pypi.org/project/phenopacket-toolkit/)'s
`phenotk normalise`, for one, by default accepts only 0.98 or more (98 here),
which only the label or a synonym reaches.

The local backend finds the names that share a word with the one searched for,
or a word spelt nearly the same, and scores those: a fraction of a second per
name for the whole HPO, with or without `rapidfuzz`. With `rapidfuzz`
installed it also looks at names that are alike letter by letter but share no
word (a name written as one word, say); a term found either way gets the same
score.

## Install

```bash
pip install -e .                 # installs the `hponorm` command
pip install -e ".[recommended]"  # + rapidfuzz (better matching) and rich (nicer UI)
```

Both extras are optional: without `rapidfuzz` the local search uses only its
own word index and the standard library's `difflib`; without `rich` it prints
plain text.

## Quick start

```bash
# Online (auto-detects OLS):
hponorm test3_phenopackets.json

# Fully offline against a downloaded ontology:
hponorm myfile.json --backend local --hpo-file /path/to/hp.obo

# Try it offline with the bundled sample (diabetes / long QT / a few others):
hponorm myfile.json --backend local --hpo-file hponorm/data/hpo-sample.obo

# Equivalent without installing:
python -m hponorm myfile.json
```

## The interactive review

Each **distinct** label is reviewed once and the decision is applied to every
occurrence (so "diabetes" appearing 8 times is one question, not eight). For
each label you see a ranked table and a prompt:

```
Select [#, s words, h HP:id, r, k, x, ?, q]:
  <number>     select that suggestion
  s <words>    search again with different words
  h HP:id      enter an HPO id manually (validated, canonical label fetched)
  r            reuse a decision remembered from earlier in this session
  k            keep the current term unchanged
  x            skip this label (leave it unmapped)
  ?            help
  q            finish now: apply decisions made so far and save
```

If a feature already has an `id`, it is validated: the tool reports whether the
id resolves in HPO and whether its canonical label matches the existing label
(or, if OLS did not answer, that the id could not be checked).
A feature's `excluded` flag (a negated phenotype) is preserved untouched — only
`type.id` / `type.label` are changed.

## Output

For `input.json` (unless you pass `--out` or `--in-place`):

* `input.normalized.json` — the normalised phenopacket (only `type.id` /
  `type.label` are changed; everything else, including the pedigree and any
  `diseases` terms, is left exactly as-is).
* `input.normalized.json.mapping.json` — a decision log (label → HP id, count).

Because `hponorm` only touches `phenotypicFeatures`, you can run it alongside
`mondonorm` (which only touches `diseases`) on the same file without conflict.

## Useful options

```
--out PATH            output path (single input file only)
--in-place            overwrite the input file(s)
--backend {auto,ols,local}
--hpo-file PATH       local hp.obo or hp.json
--cache-dir DIR       where the downloaded HPO + parsed index are cached
--update              re-download / re-parse HPO
--no-download         never download (local backend must find a file)
--no-online           never use OLS, even in auto mode
--limit N             suggestions shown per label (default 8)
--mapping FILE        load/save remembered label->term decisions across runs
--auto-remembered     auto-apply remembered labels without prompting
--no-color            plain text output
--version             the version of hponorm
```

Process several files in one go; remembered decisions carry across them:

```bash
hponorm *.json --mapping team-hpo-map.json
# next time, reuse without re-typing:
hponorm new/*.json --mapping team-hpo-map.json --auto-remembered
```

### The mapping file

The mapping file is a JSON object of names and the terms chosen for them.
hponorm writes each new entry as `"long qt interval": {"id": "HP:0001657",
"label": "Prolonged QT interval"}`, and since version 1.1 it reads the names
without regard to case, so you can write one by hand as you like:

```json
{
  "Long QT interval": {"id": "HP:0001657", "label": "Prolonged QT interval"},
  "Deafness": "HP:0000365",
  "Breast cancer": "MONDO:0007254"
}
```

An entry may give the id alone (hponorm then looks up its label), and entries
for other ontologies are left as they are, so the same file can serve
`mondonorm`, studio2ga4gh's `--terms` and `phenotk normalise --mapping`. A
decision log from `phenotk normalise --log` (`{"resolved": {...}}`) can be
read too. When hponorm saves its decisions, an entry already in the file keeps
its own form; and a file it cannot read is left untouched.

## Use as a library

```python
from hponorm import HpoIndex, OlsClient, Suggester
from hponorm import phenopackets as pp

sug = Suggester(OlsClient(), name="ols")          # or Suggester(HpoIndex.load(...))
for c in sug.suggest("long qt interval", limit=5):
    print(c.id, c.label, c.score, c.matched_on)
if sug.last_error:                                 # e.g. "EBI's Ontology Lookup Service did not answer in time"
    print(sug.last_error)

data = pp.load("myfile.json")
for ref in pp.find_phenotypic_features(data):
    print(ref.path, ref.label, ref.id, "excluded" if ref.excluded else "")
```

## Getting the full HPO ontology

* Browser / search: https://hpo.jax.org
* Direct files: `https://purl.obolibrary.org/obo/hp.obo` or `.../hp.json`
* Releases: https://github.com/obophenotype/human-phenotype-ontology/releases

Phenopackets reference: https://phenopacket-schema.readthedocs.io
(phenotypic features use HPO `OntologyClass` values in `PhenotypicFeature.type`).

## A note on the bundled sample's ids

The HP ids in `hpo-sample.obo` were checked against HPO browsers (e.g. Diabetes
mellitus HP:0000819, Prolonged QT interval HP:0001657, Seizure HP:0001250,
Microcephaly HP:0000252, Short stature HP:0004322, Global developmental delay
HP:0001263). It is still only a tiny demo fixture — verify against the live HPO
for real curation work.

## Development

```bash
pip install -e ".[test]"
pytest
```

The tests never go online: they use the bundled sample, a few terms copied
from the HPO into `tests/data`, and a stand-in for OLS that answers with JSON
written for the tests.

## Licence

MIT. The test ontology in `tests/data` is an excerpt of the HPO, used under the
HPO's licence (https://hpo.jax.org/app/license).
