Metadata-Version: 2.4
Name: cdc-nhanes-dl
Version: 0.1.0
Summary: Snapshot the public NHANES dataset from CDC into analysis-ready Parquet
License-Expression: MIT
Project-URL: Homepage, https://github.com/ink-waffle/cdc-nhanes-dl
Keywords: nhanes,cdc,nchs,epidemiology,parquet,dataset
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: lxml>=5.0
Requires-Dist: pyreadstat>=1.2
Requires-Dist: pyarrow>=14.0
Requires-Dist: pandas>=2.0
Provides-Extra: types
Requires-Dist: pandera>=0.20; extra == "types"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pandera>=0.20; extra == "dev"
Dynamic: license-file

# cdc-nhanes-dl

Takes a local snapshot of the entire public [NHANES](https://www.cdc.gov/nchs/nhanes/) dataset — every table, every survey cycle — and leaves you with Parquet files you can open in pandas.

This is not a query API. It mirrors what CDC publishes to your disk in one shot: no API key, no auth, no rate limits to negotiate. Run it once to get the dataset, re-run it later to pick up new cycles. Every step skips work already on disk, so an interrupted run resumes where it stopped.

Expect roughly **6 GB** and 15–40 minutes on a decent connection for the full set.

## Install

```bash
pip install git+https://github.com/ink-waffle/cdc-nhanes-dl
```

## Get the data

```bash
cdc-nhanes-dl
```

That writes to `output/nhanes/` relative to your current directory. The part you'll actually use is `output/nhanes/nhanes_tables/`:

```python
import pandas as pd

bmx = pd.read_parquet("output/nhanes/nhanes_tables/BMX — Body Measures.parquet")
bmx[["SEQN", "CYCLE", "BMXWT — Weight (kg)", "BMXBMI — Body Mass Index (kg/m**2)"]].head()
```

One file per table, with all cycles stacked into it. `SEQN` is the respondent ID (join key across tables), `CYCLE` tells you which survey wave a row came from.

Don't need everything? Narrow it:

```bash
cdc-nhanes-dl --tables BMX,DEMO,GHB          # just these tables, all cycles
cdc-nhanes-dl --cycles 2017-2018,2021-2023   # just these cycles, all tables
cdc-nhanes-dl -o ~/data/nhanes -w 16         # elsewhere, more parallelism
```

## Column names

Raw NHANES variable codes are unreadable, so columns are renamed to `CODE — Label` using the labels embedded in CDC's SAS files:

```
BMXWT — Weight (kg)
LBXGH — Glycohemoglobin (%)
RIDAGEYR — Age in years at screening
```

The bare code stays available in Parquet metadata if you need to map back — per column under the `nhanes_code` field key, or for the whole table as a JSON map under `nhanes_columns`.

## Typed schemas

The final step generates [pandera](https://pandera.readthedocs.io/) models under `output/nhanes/types/`, one module per component, so you get autocomplete over those long column names instead of typing them by hand:

```python
from output.nhanes.types.examination.types import BmxSchema

BmxSchema.validate(bmx)
bmx[BmxSchema.bmxbmi]   # -> "BMXBMI — Body Mass Index (kg/m**2)"
```

Requires `pip install pandera`. Pass `--import-prefix` to match wherever the generated package ends up on your import path (`--import-prefix output.nhanes.types` for the layout above).

## What it produces

```
output/nhanes/
├── xpt/<cycle>/                 raw SAS transport files, as downloaded
├── parquet/<cycle>/             one-to-one Parquet conversion
├── nhanes_tables/               cycles stacked per table  ← use these
├── types/                       generated pandera schemas
├── table_categories.json        table → component
└── table_descriptions.json      table → plain-English name
```

`xpt/` and `parquet/` are intermediates; they exist so re-runs don't re-download and so you can go back to the source if a conversion looks wrong. Delete them once you have `nhanes_tables/` and you save about 4 GB.

## Pipeline steps

| Step | Does |
|---|---|
| `download` | scrapes CDC's table manifest, fetches every XPT |
| `convert` | XPT → Parquet, preserving SAS column labels |
| `metadata` | scrapes component + description for every table |
| `combine` | strips cycle suffixes, stacks rows, renames columns |
| `types` | writes pandera schemas |

Run one on its own with `--only`, or exclude one with `--skip-<step>`:

```bash
cdc-nhanes-dl --only download
cdc-nhanes-dl --skip-types
```

Individual steps are also importable and directly runnable:

```bash
python -m cdc_nhanes_dl.download --cycles 2017-2018
python -m cdc_nhanes_dl.combine --tables BMX
```

## Caveats

- Tables with no `SEQN` column — format lookups, pooled-sample files — are skipped by `combine`, since they have no respondent to join on. They're still in `parquet/`.
- Multi-cycle aggregate releases (`1999-2004`, `1988-2020`, …) are downloaded but excluded from stacking; their rows already appear in the individual cycles.
- When a variable is text in one cycle and numeric in another, the stacked column becomes text. Cast it yourself.
- When a label was reworded between cycles, the most recent wording is used for the whole column.
- Limited Access tables are listed in the metadata but their data is not public — CDC gates it behind the [RDC](https://www.cdc.gov/rdc/), so those downloads fail and are reported as such.

## Tests

```bash
pip install -e ".[dev]"
pytest                  # runs the real pipeline over 2 small tables
pytest -m "not network" # offline only
```

## Licence

MIT. The data itself is public domain, published by the US CDC / NCHS.
