Metadata-Version: 2.5
Name: pubmed-proto
Version: 0.5.0
Summary: Typed protobuf messages and an XML converter for NLM PubMed XML.
Project-URL: Repository, https://github.com/populationgenomics/pubmed-proto
Project-URL: Issues, https://github.com/populationgenomics/pubmed-proto/issues
Author: Centre for Population Genomics
License-Expression: MIT
License-File: LICENSE
Keywords: bioinformatics,ncbi,protobuf,pubmed,xml
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: protobuf>=6.31.1
Description-Content-Type: text/markdown

# pubmed-proto

Generates **`pubmed_proto`** — a typed Python package for parsing NLM PubMed XML into protobuf messages — from the
PubMed DTD.

This repository is the **generator**, not the package. It holds the inputs (`pubmed.dtd`, `pubmed_transforms.yaml`) and
drives [`xsd-former`](https://github.com/populationgenomics/xsd-former) (the `xsdformer` CLI) to emit the `pubmed_proto`
source tree, which is then built into a wheel and published to PyPI. The generated tree (`generated/`) and build outputs
(`dist/`) are gitignored — only the inputs are version-controlled.

## Consuming `pubmed_proto`

Depend on the published wheel, not this repo:

```
pip install pubmed_proto      # or: uv add pubmed_proto
```

```python
from lxml import etree
from pubmed_proto import xml_converter

tree = etree.parse("efetch_output.xml")
record_set = xml_converter.PubmedArticleSet(tree.getroot())  # XML -> protobuf

article = record_set.pubmed_article[0]  # or .pubmed_book_article
data = article.SerializeToString()      # protobuf wire bytes, for storage
```

An `efetch` response is a `PubmedArticleSet` of journal articles (`PubmedArticle`) and Bookshelf records
(`PubmedBookArticle` — a GeneReviews chapter, say); both are modelled, with one converter per message.

The package exposes two modules (both typed; ships `py.typed`):

| module          | purpose                                             |
| --------------- | --------------------------------------------------- |
| `pubmed_pb2`    | compiled protobuf messages (`Article`, `Author`, …) |
| `xml_converter` | PubMed XML → protobuf (per-message factory funcs)   |

### Rich-text fields

`ArticleTitle`, `AbstractText`, `Keyword`, `CollectiveName`, `Citation`, `BookTitle`, `CollectionTitle` and
`SectionTitle` carry their inline markup as markdown in `value`: `<b>` becomes `**…**`, `<i>` becomes `*…*`, `<u>`
becomes `__…__`, `<sup>` becomes `^(…)` and `<sub>` becomes `~(…)`.

Literal characters in the text that could be read as one of those markers are backslash-escaped: `\`, `*`, `_`, `^` and
`~` everywhere, and `)` inside a `^(…)` or `~(…)`, where an unescaped `)` closes it. The markers themselves are never
escaped. The title of PMID 34207366 comes out as

```
*Cis*-Segregation of c.1171C>T Stop Codon (p.R391\*) in *SERPINC1* Gene …
```

The escapes are CommonMark backslash escapes, so a CommonMark renderer shows the stop codon as `p.R391*`. Other markdown
syntax in the text is not neutralised: a pair of backticks still forms a code span, and `<…>` can still form raw HTML.

For plain text, call `xml_converter.markdown_to_plain_text(value)`, which unescapes while it drops the markers and
returns `Cis-Segregation of c.1171C>T Stop Codon (p.R391*) in SERPINC1 Gene …`. In another language, the same thing is
one left-to-right scan: a `\` and the character after it stand for that character; an unescaped `*` or `_` is dropped;
`^(` or `~(` is dropped and opens a script; an unescaped `)` is dropped if a script is open (closing the innermost one)
and kept otherwise. A regex that strips the markers and ignores the escapes (`re.sub(r'[*_]', '', value)`, say) turns
the stop codon into `p.R391\`. Releases before 0.4.0 did not escape, so their output cannot be stripped losslessly.

## Developing the generator

Requires [`uv`](https://docs.astral.sh/uv/).

```
make generate   # DTD + transforms -> generated/pubmed_proto/
make build      # generate, then build the wheel into dist/
make clean      # remove generated/ and dist/
uv run --group test pytest   # record gate over real PubMed records
```

Shaping the output is done in **`pubmed_transforms.yaml`** — dropping admin types, flattening list wrappers, coercing
booleans/timestamps, and serializing rich-text fields to markdown. See the
[`xsd-former`](https://github.com/populationgenomics/xsd-former) docs for the transform reference.

## Provenance & attribution

`pubmed.dtd` is derived from the **U.S. National Library of Medicine PubMed DTD**, version `pubmed_250101` (dated
2024-08-28):

<https://dtd.nlm.nih.gov/ncbi/pubmed/out/pubmed_250101.dtd>

NLM DTDs are U.S. Government works and public domain in the United States; the MIT `LICENSE` in this repo covers CPG's
own files (transforms, generator wiring, tests), not the NLM DTD.

Courtesy of the U.S. National Library of Medicine. NLM does not endorse this package. The vendored DTD and the PubMed
records under `tests/records/` are pinned snapshots and do not necessarily reflect the most current data available from
NLM — fetch from NLM directly for current data. The records are `efetch` output included solely as parser fixtures. The
two journal records are verbatim; their abstracts remain the copyright of their publishers or authors. Of the two
Bookshelf records, `pmid_25834910.xml` (an NIH Molecular Libraries probe report) is verbatim: the collection's Copyright
and Permissions statement reads "This publication is in the public domain", and the record carries no copyright
statement. `pmid_20301425.xml` (a GeneReviews chapter) is trimmed: each `AbstractText` paragraph is replaced by a
placeholder sentence and `CopyrightInformation` is removed, so the fixture carries only bytes we may redistribute — the
gate exercises the record's structure, and its prose is not ours. Everything else in it is verbatim.

**Local modification:** the external MathML module include (`<!ENTITY % mathml-in-pubmed SYSTEM "mathml-in-pubmed.mod">`
and its reference) was removed so the DTD is self-contained for schema generation — MathML markup in titles/abstracts is
not modelled. With that one include removed, the file is byte-identical to upstream `pubmed_250101`.

**Inline markup in attribute-less text elements:** `Affiliation`, `PublisherName`, `Suffix` and `VolumeTitle` are
declared as text with inline markup (`b`, `i`, `sub`, `sup`, `u`) and nothing else — no attributes, no MathML
alternative (they are exactly the `%text;` elements of that shape not otherwise dropped; `CoiStatement` is the fifth and
is in `drop_types`). Two transforms in `pubmed_transforms.yaml` collapse them: `drop_types` removes the inline-markup
types (`B`, `I`, `Sub`, `Sup`, `U`), which leaves each element a bare text wrapper, and `inline_wrappers: true` then
folds it into its parent as a plain string. Both run before `serialize_content`, so an entry for these names has no
effect, and the converter keeps only the text before the first inline element
(`<PublisherName>Univ of <i>Washington</i></PublisherName>` parses as `Univ of `). Elements with attributes or a MathML
alternative stay messages and are the ones `serialize_content` renders to markdown.

The DTD is **vendored deliberately, not fetched at build time**: it's a modified derivative (so a fetch wouldn't
reproduce it), and pinning the exact bytes keeps the generated schema reproducible. The full provenance also lives in a
comment at the top of `pubmed.dtd`.

## Releasing

The published version is **`build.version` in `pubmed_transforms.yaml`** (what `xsdformer` stamps into the wheel). To
release:

1. Bump `build.version` in `pubmed_transforms.yaml`.
1. Publish a GitHub Release tagged `vX.Y.Z` matching that version.

The `release` workflow generates, builds, and publishes to PyPI via Trusted Publishing (OIDC). It fails if the tag and
`build.version` disagree.
