Metadata-Version: 2.4
Name: fast-feature-transform
Version: 0.0.1
Summary: Feature preprocessing for machine learning, written in C++ with a Python API
Keywords: feature engineering,preprocessing,machine learning,category_encoders,scikit-learn,parquet
Author: Max Lei
License-Expression: MIT AND Apache-2.0 AND BSD-3-Clause AND BSD-2-Clause AND Zlib AND bzip2-1.0.6 AND BSD-Attribution-HPND-disclaimer AND BSL-1.0
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.txt
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Free Threading :: 3 - Stable
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Project-URL: Source, https://github.com/mxlei01/fast-feature-transform
Project-URL: Issues, https://github.com/mxlei01/fast-feature-transform/issues
Project-URL: Changelog, https://github.com/mxlei01/fast-feature-transform/blob/master/CHANGELOG.md
Requires-Python: >=3.14
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/mxlei01/fast-feature-transform/v0.0.1/images/icon.svg" alt="fast-feature-transform" width="104">
</p>

<h1 align="center">fast-feature-transform</h1>

Feature preprocessing for machine learning, written in C++ with a Python API. Fit a pipeline on Parquet files, save
it, then transform one record at a time when serving or whole files when training.

- **Feature parity** with category_encoders and scikit-learn's preprocessors.
- **Fast:** one record takes 0.13 to 1.2 µs, where scikit-learn takes 32 to 219 µs and category_encoders 0.31 to
  1.2 ms.
- **Out-of-core execution:** Parquet is read in batches, and fits that need every value spill to disk.

## Install

```sh
pip install fast-feature-transform
```

Wheels cover CPython 3.14, standard and free-threaded (`python3.14t`), on Linux x86_64 and arm64 (glibc 2.28 or
later) and on macOS 26 or later on Apple silicon. Elsewhere, build from source as [Development](https://github.com/mxlei01/fast-feature-transform/blob/v0.0.1/README.md#development)
describes.

## Usage

```python
from fast_feature_transform import OrdinalEncoder, Pipeline, StandardScaler

pipeline = Pipeline(
    [
        ("country", OrdinalEncoder(columns=["country"])),
        ("numbers", StandardScaler(columns=["age", "income"])),
    ]
).fit(["train.parquet"])

pipeline.transform_dict({"country": "CA", "age": 42, "income": 75000.0})  # serving: one record
pipeline.transform_parquet(["train.parquet"], output_dir="train-features")  # batches: Parquet in, Parquet out
pipeline.transform_memmap(["train.parquet"], output_dir="train-mmap")  # training: one packed file for NumPy

pipeline.save("artifacts/preprocessing")
serving = Pipeline.load("artifacts/preprocessing")
```

## Transform latency

Time to transform one record (a `dict` in, a `dict` out) and batches of 100,000 and 1,000,000 rows, after fitting on
100,000 rows. For one record, category_encoders and scikit-learn also pay to build their DataFrame or array and to
turn the result back into a `dict`. A batch goes from one Parquet file to another on both sides. This package reads
the file's strings as stored, a dictionary of distinct values plus an index per row; category_encoders and
scikit-learn read the file with pyarrow into pandas or NumPy, transform it, and write the result with this package's
Parquet settings. Medians of seven rounds on an Apple M4, one thread; lower is better.

![Transform time for 20 encoders, fast-feature-transform against category_encoders 2.10.0](https://raw.githubusercontent.com/mxlei01/fast-feature-transform/v0.0.1/images/transformer-comparison-category-encoders.png)

![Transform time for 11 preprocessors, fast-feature-transform against scikit-learn 1.9.0](https://raw.githubusercontent.com/mxlei01/fast-feature-transform/v0.0.1/images/transformer-comparison-scikit-learn.png)

At a million rows scikit-learn took 1.14 to 5.3 times as long and category_encoders 1.2 to 30 times. Reading and
writing Parquet adds time on both sides, which narrows the gap where the transform is cheap or its output wide; it
stays widest where the other library's transform is slow. Matching names don't always mean matching behavior:
categories are sorted, output types are fixed at fit, and `PolynomialEncoder` differs from category_encoders by about
1e-5 at 32 categories and by more than 0.5 from 40 to 128, because category_encoders' basis is the less accurate one.

## License

MIT. The wheels also contain Apache Arrow and the other libraries that
[THIRD_PARTY_NOTICES.txt](https://github.com/mxlei01/fast-feature-transform/blob/v0.0.1/THIRD_PARTY_NOTICES.txt) lists, and code adapted from SciPy, each under its own license.
