Metadata-Version: 2.4
Name: logit-stream
Version: 0.1.0
Summary: Out-of-core logit models by streamed Newton: binary, multinomial, ordered and conditional logit, with absorbed fixed effects
Author: Lee Tucker
License-Expression: MIT
Project-URL: Homepage, https://github.com/leetucker/logit-stream
Project-URL: Source, https://github.com/leetucker/logit-stream
Project-URL: Issues, https://github.com/leetucker/logit-stream/issues
Project-URL: Documentation, https://leetucker.github.io/logit-stream/
Project-URL: Changelog, https://github.com/leetucker/logit-stream/blob/main/CHANGELOG.md
Keywords: econometrics,logit,multinomial logit,ordered logit,conditional logit,discrete choice,out-of-core,fixed effects,imputation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: hdfe-stream[formula]<0.7,>=0.6
Requires-Dist: polars!=1.41.*,>=1.37
Requires-Dist: numpy>=1.24
Requires-Dist: numba>=0.59
Requires-Dist: pyarrow>=14.0
Requires-Dist: scipy>=1.11
Requires-Dist: threadpoolctl>=3.1
Provides-Extra: benchmark
Requires-Dist: scikit-learn>=1.5; extra == "benchmark"
Requires-Dist: statsmodels>=0.14; extra == "benchmark"
Requires-Dist: xlogit>=0.2.7; extra == "benchmark"
Requires-Dist: pylogit>=1.0.1; extra == "benchmark"
Requires-Dist: mord>=0.7; extra == "benchmark"
Requires-Dist: matplotlib>=3.7; extra == "benchmark"
Requires-Dist: pandas>=2.0; extra == "benchmark"
Provides-Extra: docs
Requires-Dist: sphinx>=8.0; extra == "docs"
Requires-Dist: myst-parser>=4.0; extra == "docs"
Requires-Dist: furo>=2024.8.6; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5; extra == "docs"
Requires-Dist: sphinx-design>=0.6; extra == "docs"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: scikit-learn>=1.5; extra == "test"
Requires-Dist: statsmodels>=0.14; extra == "test"
Requires-Dist: pandas>=2.0; extra == "test"
Dynamic: license-file

# logit-stream

Logit models for data that do not fit in memory: binary, multinomial,
ordered, conditional, nested and mixed logit, fitted by Newton's method with
the data streamed off disk, from
[pyfixest](https://github.com/py-econometrics/pyfixest)-style formulas, with
absorbed fixed effects or random intercepts.

```python
from logit_stream import mlogit_stream

fit = mlogit_stream(
    "race ~ C(agecat) + female + C(inccat) + C(cit) | puma",
    "acs/*.parquet",               # a Parquet path or glob, never fully loaded
    vcov={"CRV1": "puma"},
)
fit.summary()
fit.predict("people/*.parquet").sink_parquet("imputed.parquet")   # lazily
```

| | |
|---|---|
| `logit_stream`, `probit_stream` | binary logit and probit with any number of fixed effects ([hdfe-stream](https://github.com/leetucker/hdfe-stream)'s `feglm_stream`) |
| `mlogit_stream` | multinomial logit (any number of fixed effects; several through a Poisson regression) |
| `ologit_stream`, `oprobit_stream` | ordered logit and probit |
| `clogit_stream` | conditional (McFadden) logit on long-format choice data |
| `cfelogit_stream` | Chamberlain's conditional fixed-effects logit for panels |
| `mixlogit_stream`, `nlogit_stream` | mixed and nested logit (prototypes) |

Newton's method for a logit needs the data only through sums over rows -- the
log-likelihood, its gradient and the information matrix -- so each step is one
read of the data, block by block, and one dense solve; the likelihoods are
concave, so a handful of steps suffice. Memory grows with the number of
coefficients, not of rows. The multinomial and ordered logit absorb one fixed
effect exactly, by eliminating its per-level blocks of the information matrix
in every step.

Results share [hdfe-stream](https://github.com/leetucker/hdfe-stream)'s
reporting: `summary()`, `tidy()`, `summary_dict()`, `summary_json()`,
`confint()`, `wald_test()`, `with_vcov()`, `etable()`, and lazy `predict()`,
`resid()`, `sample()` and `fixef()` frames. Estimates and standard errors
match statsmodels and scikit-learn (see `tests/`).

Documentation: <https://leetucker.github.io/logit-stream/> (build locally with
`pip install -e ".[docs]"` and `sphinx-build -b html docs docs/_build/html`).
`examples/` has a runnable script for each page of its guide, on simulated
data.

## Install

```bash
pip install logit-stream
```

Python 3.10 or later. On Python 3.15, pyfixest 0.60 (which hdfe-stream's
formula extra brings in) has no wheels yet: it builds from source with a Rust
toolchain and `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` set; everything else
installs from wheels.

## Benchmarks

`benchmarks/` compares logit-stream with scikit-learn, statsmodels, mord and
R's glm, fixest, nnet, MASS and ordinal on American Community Survey
microdata (10,000 to 12.9 million adults); with xlogit, pylogit,
statsmodels, jaxlogit, Biogeme and R's mlogit and logitr on the Swissmetro
choice survey, resampled to 10 million choice sets; on short panels with
statsmodels and R's survival, fixest and alpaca; and on a GPU. See
`benchmarks/README.md` and the documentation's benchmark pages.

## License

MIT.
