Metadata-Version: 2.4
Name: pytorch-normalizers
Version: 0.1.3
Summary: PyTorch normalization layers for numerical features, including SmoothBuckets and QuantileNorm
Author-email: Alon Rozental <alonzorz@gmail.com>
License: MIT License
        
        Copyright (c) 2025 Alon Rozental
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/yourusername/pytorch_normalizers
Project-URL: Repository, https://github.com/yourusername/pytorch_normalizers.git
Project-URL: Issues, https://github.com/yourusername/pytorch_normalizers/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0.0
Requires-Dist: numpy>=1.20.0
Provides-Extra: test
Requires-Dist: pandas>=1.0.0; extra == "test"
Requires-Dist: catboost>=1.0.0; extra == "test"
Requires-Dist: pytest>=7.0.0; extra == "test"
Dynamic: license-file

# PyTorch Normalizers: Useful PyTorch Layers for Feature Normalization

This package provides several useful PyTorch layers for normalizing and processing numerical features in deep learning models, with a focus on differentiability, robustness, and handling real-world data issues like outliers and missing values. Currently, it includes SmoothBuckets and QuantileNorm, with more layers (such as DefaultEmbeddings and DetachFrom) planned for future additions.

## Overview of Layers

### SmoothBuckets
SmoothBuckets is a PyTorch module designed to normalize numerical features using a smooth, differentiable bucketing approach. It generalizes traditional deterministic bucketing and percentile-based normalization, making it suitable for deep learning models where gradients need to flow through the normalization step. Key features include:

- **Soft Bucketing with Gaussians**: Instead of hard assignments to buckets, it uses Gaussian distributions to compute soft probabilities (bucket shares) for each feature value, ensuring differentiability.
- **Handling Missing Values**: Gracefully manages NaNs by learning replacements during training, moving imputation logic into the model.
- **Robustness**: Demonstrates improved performance over standard LayerNorm in scenarios with data corruption or outliers, as shown in the included tests.
- **Additional Percentile Feature**: Appends an approximated percentile for each feature, enhancing the representation.
- **Trainable Parameters**: Allows fine-tuning of multipliers for means and standard deviations while keeping initial estimates fixed to avoid issues with large value ranges.

This layer is particularly useful in tabular data modeling, such as regression or classification tasks, where numerical features vary widely in scale and distribution.

### QuantileNorm
QuantileNorm is a PyTorch module that performs quantile normalization with incremental updates to adapt to data distribution shifts during training. It initializes quantiles from a sample and updates them gradually, handling outliers and providing smooth extrapolation for extreme values. This makes it ideal for streaming data or environments with evolving distributions.

## Installation

Install via pip:

```bash
pip install pytorch-normalizers
```

## Usage

To integrate the layers into your PyTorch model, import them from the package. Both layers support initialization with a sample of numerical features (as a `torch.Tensor`, `np.ndarray`, or `pd.DataFrame`).

### SmoothBuckets Example

```python
import torch.nn as nn
from pytorch_normalizers import SmoothBuckets

class MyModel(nn.Module):
    def __init__(self, numerical_features_sample, *args, **kwargs):
        super().__init__()
        self.input_normalizer = SmoothBuckets(numerical_features_sample) # num_buckets=5, eps=1e-5, dropout=0.1 are optional here
        # Add other layers as needed...

    def forward(self, x):
        x = self.input_normalizer(x)
        # Rest of the forward pass...
        return x
```

- `num_buckets`: Number of buckets per feature (e.g., 5).
- `numerical_features_sample`: A sample containing only the numerical features, used to initialize means and stds based on quantiles.
- `eps`: Small value to prevent division by zero.
- `dropout`: Dropout rate for regularization.

The layer outputs a tensor of shape `(batch_size, num_features)`, after concatenating bucket shares and percentiles, applying layer norm, dropout, and a linear projection back to the original feature dimensionality.

### QuantileNorm Example

```python
import torch.nn as nn
from pytorch_normalizers import QuantileNorm

class MyModel(nn.Module):
    def __init__(self, numerical_features_sample, num_buckets=9):
        super().__init__()
        self.input_normalizer = QuantileNorm(numerical_features_sample, num_buckets)
    
    def forward(self, x):
        x = self.input_normalizer(x)
        # Rest of the model...
        return x
```

- `num_buckets`: Number of quantiles/buckets (e.g., 9 for deciles).
- `numerical_features_sample`: A sample for initial quantile estimation.

The layer outputs normalized values in [0, 1] range, with incremental updates during training.

## Testing

The package includes self-contained tests for each layer:

- For SmoothBuckets: Run `python layers/smooth_buckets.py` (or import and run `test_me()`). It uses the [House Prices dataset from Kaggle](https://www.kaggle.com/competitions/house-prices-advanced-regression-techniques/overview) to compare against LayerNorm and CatBoost in a binary classification task.

Example output (results may vary slightly due to randomness):

```
Corrupt batch:  False
LayerNorm Classifier Accuracy: 0.8101 
SmoothBuckets Classifier Accuracy: 0.9337 # a single (features, 1) layer over the normalized features
CatBoost Classifier Accuracy: 0.9281

Corrupt batch:  True
LayerNorm Classifier Accuracy: 0.6998  # Degrades due to corruption
SmoothBuckets Classifier Accuracy: 0.9219 # More robust
CatBoost Classifier Accuracy: 0.9281 # Deterministic result, unaffected by a tensor corruption
```

- For QuantileNorm: Run `python tests/quantile_norm_test.py` to verify quantile adaptation on synthetic data.

## Dependencies

- Runtime: PyTorch, NumPy
- Tests: Pandas, CatBoost

Install core dependencies:

```bash
pip install torch numpy
```

For tests:

```bash
pip install pandas catboost
```

## License

MIT License. Feel free to use, modify, and distribute. See [LICENSE](LICENSE) for details.

## Future Plans

Additional layers like DefaultEmbeddings (for efficient categorical embeddings) and DetachFrom (for selective gradient detachment) will be added in upcoming releases. Contributions are welcome!
