Metadata-Version: 2.5
Name: pynhts
Version: 0.6.0
Summary: Typed, survey-weight-aware access to US National Household Travel Survey microdata.
Project-URL: Homepage, https://github.com/sabernaseralavi-60/pynhts
Project-URL: Issues, https://github.com/sabernaseralavi-60/pynhts/issues
Project-URL: Changelog, https://github.com/sabernaseralavi-60/pynhts/blob/main/CHANGELOG.md
License: BSD-3-Clause
License-File: LICENSE
Keywords: NHTS,jackknife,replicate weights,transportation,travel survey
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: platformdirs>=3.10
Requires-Dist: pooch>=1.8
Requires-Dist: pyarrow>=14
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: matplotlib>=3.7; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: openpyxl; extra == 'dev'
Requires-Dist: pandas-stubs; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: graph
Requires-Dist: torch-geometric>=2.4; extra == 'graph'
Requires-Dist: torch>=2.0; extra == 'graph'
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == 'torch'
Provides-Extra: viz
Requires-Dist: matplotlib>=3.7; extra == 'viz'
Description-Content-Type: text/markdown

# pynhts

[![ci](https://github.com/sabernaseralavi-60/pynhts/actions/workflows/ci.yml/badge.svg)](https://github.com/sabernaseralavi-60/pynhts/actions/workflows/ci.yml)
![status](https://img.shields.io/badge/status-alpha-orange)
![python](https://img.shields.io/badge/python-3.10%2B-blue)
![license](https://img.shields.io/badge/license-BSD--3--Clause-green)

Typed, survey-weight-aware Python access to US National Household Travel Survey (NHTS)
microdata.

> **Scope and status (0.6.0, alpha).** The **2001**, **2009**, **2017** and **2022** (NextGen)
> waves are supported. The API may still change. Results are validated against published tables:
> for 2009, 35 of 40 figures of the *Summary of Travel Trends* reproduce completely (the other 5
> reproduce the estimate; the report's margin of error uses another multiplier), for 2001 40 of
> 47 (the 7 others are within one unit or come from an older data version); for 2017, 31 of 37
> figures of FHWA's *Summary of Travel Trends* reproduce completely (3 more reproduce the estimate
> but not the margin of error, 3 are gaps); for 2022, 15 of 18 control totals of the User Guide,
> and the Taylor-series errors equal R's `survey` package. Every gap is reported openly in
> [docs/validation.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md) (tours: 21 of 33).

## Install

```
pip install pynhts
pynhts fetch --replicates        # download, verify (SHA-256) and convert; about 0.4 GB of Parquet, plus 1.8 GB of raw CSV that you may delete afterwards
```

`pynhts fetch --wave 2022` needs no replicate weights (4.5 MB); 2009 and 2001 need a separate
replicate archive (400 and 440 MB) for standard errors. Already have the NHTS CSV files?
`pynhts register <folder> [--wave 2022]` adopts them after checking their hashes, and
`pynhts docs --wave 2001|2009|2017|2022` downloads the official documents (also hash-checked).

## What it does

```python
import pynhts
data = pynhts.load(2017)

# lazy, validated tables: columns are joined from the table that owns them
trips = data.trips(["TRPMILES", "TRPTRANS", "HHFAMINC", "R_AGE"])

# weighted statistics with jackknife (JKn) standard errors and 95 % margins of error
s = data.survey("person", ["R_AGE_IMP", "DRIVER"])
s.total(domain=s.rows["DRIVER"] == "01")        # 223,277 thousand drivers, MOE 827 thousand
data.survey("trip", ["TRPMILES"]).mean("TRPMILES")   # 10.70 miles, MOE 0.40

# replicate a published table and see every gap
from pynhts import recipes
print(recipes.to_markdown(recipes.run("stt2017", data)))

# machine-learning bridge
from pynhts import ml
arr = ml.to_arrays(trips, ["TRPMILES", "R_AGE"], target="TRPTRANS", weight="WTTRDFIN")

# NHTS 2022: same interface, Taylor-series errors (strata + households) instead of replicates
d22 = pynhts.load(2022)
d22.survey("trip", ["TRPMILES"]).mean("TRPMILES")          # weights="weekday" | "weekend" too

# 2009 and 2001: 100 and 99 jackknife replicates, same interface (children 0-4 are in the 2001 file)
d09 = pynhts.load(2009, replicates=True)
d09.survey("person", ["R_AGE"]).total()                     # 283,054 thousand persons

# daily rates by the official route, with the unit in the result
from pynhts import rates
rates.daily_rate(d09, "trips", per="person")                # 3.79 person trips per person and day

# what a variable means in each wave, and what to watch for
pynhts.concepts.find("VEHAGE", 2009)                        # top-coded (35 = 25+), grade H, pitfalls
pynhts.concepts.table()                                     # concepts x 4 waves, grade C / H / N

# the waves use different codes: compare and harmonize explicitly (TRPTRANS 02 is a bike in 2017, a van in 2022)
from pynhts import harmonize
harmonize.compare_waves()                                   # 386 variable names: 19 identical, 47 with changed codes
trips22 = d22.trips(["TRPTRANS"], missing="keep")
trips22["mode"] = harmonize.apply(trips22["TRPTRANS"], "mode", wave=2022)
pynhts.dictionary(2022)                                     # every variable with owner table, type, codes
```

| Module | Purpose |
|---|---|
| `pynhts.io` | verified download, offline mode, deterministic Parquet cache |
| `pynhts.core` | codebook-derived schema, `NHTSData` loader, missing-code handling |
| `pynhts.stats` | totals, means, proportions, ratios with jackknife (2001, 2009, 2017) or Taylor-series (2022) errors and margins of error |
| `pynhts.harmonize` | variable inventory of two waves; crosswalks of trip mode (four waves) and trip purpose (2017/2022; summary purpose for all four) |
| `pynhts.concepts` | machine-readable dictionary: variable, universe, unit, comparability and pitfalls of each concept in each wave |
| `pynhts.rates` | daily trips and miles per person or household by the official route, with units |
| `pynhts.recipes` | replication of published tables with honest status per figure |
| `pynhts.tours` | tour formation (HBW/HBO/NHB, work-tour patterns), FHWA-style chains; flexible home/work rules |
| `pynhts.ml` | NumPy / PyTorch / graph export (PyTorch optional) |

## Why a library

Correct NHTS analysis has many silent traps (a data server that returns HTTP 200 for missing
files, string codes with leading zeros, `-1` skips that are not missing data, 107 copied
parent columns in the child tables, a vehicle `PERSONID` that is not the person key, ...).
They are documented in [docs/data_notes.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/data_notes.md) and each is covered by a test.

## Validation

* All coded values of the official codebooks (counts and weighted totals) reproduce: 3,807 in
  2017, 2,639 in 2022, 3,435 in 2009 and 6,327 in 2001 (three documented codebook errors in 2009).
* 37 figures from Tables 1 and 3 of FHWA's *Summary of Travel Trends: 2017 NHTS*, including the
  adjusted-distance rows and the annual totals, with margins of error: 31 reproduce completely, 3
  reproduce the estimate but not the margin of error, 3 do not reproduce.
* 2022: 15 of the 18 control totals of the V2.x User Guide (Tables 10 and 11); the 3 that do not
  reproduce are documented. Estimates, standard errors and degrees of freedom of 24 estimands
  equal R's `survey` package to 1e-9.
* 2009: 40 figures of the *Summary of Travel Trends* (35 reproduce completely; all 27 printed
  margins of error of its Table 1 match t(0.975, 100) x the jackknife standard error to rounding).
  2001: 47 figures (40 reproduce; the 2004 report's per-person rates are reproduced only with its
  own definition, documented). The 2001 User's Guide worked example (202,586,200 vehicles, SE
  672,072) reproduces.
* The transit-user sample of McNally & Rafiq's tour study (4,994 persons, 20,222 trips, 10,011
  transit trips) exactly, plus 33 of their figures in all: 21 reproduce, 12 gaps are documented
  (`pynhts.tours`). See [docs/validation.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md).

## Documentation

[Comparing waves](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/waves.md) · [Defaults and guards](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/guards.md) · [AGENTS.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/AGENTS.md) · [Methods](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/methods.md) · [Data notes](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/data_notes.md) · [Validation](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md)
· [Roadmap](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/roadmap.md) · [Contributing](https://github.com/sabernaseralavi-60/pynhts/blob/main/CONTRIBUTING.md) · [Changelog](https://github.com/sabernaseralavi-60/pynhts/blob/main/CHANGELOG.md)

## Data

The NHTS microdata are public data of the US Federal Highway Administration
(<https://nhts.ornl.gov>). They are **never** bundled with or committed to this repository.

## Development

```
uv venv && uv pip install -e ".[dev]"
pytest                             # unit tests, no data needed
pytest -m data                     # real-data tests (files registered in the pynhts cache)
pytest -m network -o addopts=-ra   # live ORNL download, manual only
```

## Citation

See [CITATION.cff](https://github.com/sabernaseralavi-60/pynhts/blob/main/CITATION.cff). License: BSD-3-Clause.
