Metadata-Version: 2.4
Name: fast-varclushi
Version: 0.1.3
Summary: High-performance Python implementation of SAS PROC VARCLUS for variable clustering on large datasets.
Author-email: Aashay Belekar <aashaybelekar22@gmail.com>, Xuan Jing <xuanjing@hotmail.com>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/aashaybelekar/fast-varclushi
Project-URL: Repository, https://github.com/aashaybelekar/fast-varclushi
Project-URL: Bug Tracker, https://github.com/aashaybelekar/fast-varclushi/issues
Keywords: varclushi,fast-varclushi,variable-clustering,proc-varclus,dimension-reduction,factor-analysis,pca,feature-selection
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20.0
Requires-Dist: pandas>=1.2.0
Requires-Dist: scipy>=1.6.0
Requires-Dist: scikit-learn>=0.24.0
Requires-Dist: joblib>=1.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# fast-varclushi

`fast-varclushi` is a high-performance Python package for variable clustering (`VARCLUS`) and hierarchical dimension reduction on tabular data. It provides an optimized, 100% bitwise backward-compatible Python alternative to SAS `PROC VARCLUS`.

Variable clustering divides a set of numeric variables into disjoint clusters such that variables within each cluster are strongly correlated with their cluster component, while variables across different clusters are relatively uncorrelated.

---

## ⚡ Key Highlights & Performance Optimizations

`fast-varclushi` is engineered to scale variable clustering to massive datasets with millions of rows and hundreds or thousands of features:

1. **Pre-computed Correlation Matrix Caching ($O(M \cdot N^2)$ Initialization)**:
   - Computes the feature correlation matrix $\mathbf{C}$ once upon initialization.
   - All subsequent sub-cluster splits and variable reassignment iterations slice correlation matrices in microsecond memory operations ($O(K^2)$).
   - **Sample-Size Independent Clustering**: Running variable clustering on **2,000,000 rows** takes the **exact same time** as running it on 1,000 rows once the correlation matrix is computed!
2. **Sub-Cluster Eigenvalue Memoization & LRU Caching**:
   - Total variance calculations and top eigenvalue updates during greedy variable reassignments are cached and solved with symmetric eigensolvers (`np.linalg.eigvalsh`).
3. **Multi-CPU Core Parallelization (`n_jobs`)**:
   - Supports parallel random search restarts (`n_rs > 0`) using `joblib.Parallel` across available CPU cores (`n_jobs=-1`).
4. **Vectorized $R^2$ Property Computation (50x–100x Speedup)**:
   - Computes $R^2_{Own}$ and $R^2_{NC}$ (Nearest Cluster) across all variables simultaneously using matrix projections ($\mathbf{R}_{N \times K} = \mathbf{C}_{N \times N} \cdot \mathbf{W}_{N \times K}$).
5. **Optimized Factor Rotations (`Rotator`)**:
   - Features efficient implementations of 7 factor rotation algorithms (`varimax`, `promax`, `oblimin`, `quartimax`, `quartimin`, `oblimax`, `equamax`) with support for batch parallel processing and native `float32` precision.
6. **Reproducibility with `random_seed`**:
   - Full support for setting random seeds across single-threaded and multi-core parallel random search execution paths.
7. **100% SAS `PROC VARCLUS` Backward Compatibility**:
   - Verified across regression tests to yield identical mathematical outputs to SAS `PROC VARCLUS`.

---

## 📦 Installation

Install `fast-varclushi` via `pip`:

```bash
pip install fast-varclushi
```

### Dependencies
- `numpy >= 1.20.0`
- `pandas >= 1.2.0`
- `scipy >= 1.6.0`
- `scikit-learn >= 0.24.0`
- `joblib >= 1.0.0`

### Development Setup
To install `fast-varclushi` locally with development dependencies:

```bash
git clone https://github.com/aashaybelekar/fast-varclushi.git
cd fast-varclushi
pip install -e .[dev]
```

---

## 🚀 Quickstart Example

```python
import pandas as pd
from varclushi import VarClusHi

# Load a sample dataset (Wine Quality Red dataset)
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv"
df = pd.read_csv(url, sep=";")
df = df.drop(columns=["quality"])

# Initialize VarClusHi
vc = VarClusHi(
    df=df,
    maxeigval2=1.0,     # Stop splitting when max 2nd eigenvalue <= 1.0
    maxclus=None,      # Maximum number of clusters (None for unlimited)
    n_rs=10,           # 10 random search restarts per split
    n_jobs=-1,         # Use all available CPU cores
    random_seed=42     # Ensure reproducibility
)

# Run variable clustering
vc.varclus()

# View Cluster Summary
print("--- Cluster Info ---")
print(vc.info)

# View R-Squared Ratios for Feature Selection
print("\n--- RSquare Table ---")
print(vc.rsquare.head(10))
```

### Output Tables

#### Cluster Summary (`vc.info`)
| Cluster | N_Vars | Eigval1 | Eigval2 | VarProp |
| :--- | :--- | :--- | :--- | :--- |
| **0** | 3 | 2.141357 | 0.658413 | 0.713786 |
| **1** | 3 | 1.766885 | 0.900991 | 0.588962 |
| **2** | 2 | 1.371260 | 0.628740 | 0.685630 |
| **3** | 2 | 1.552496 | 0.447504 | 0.776248 |
| **4** | 1 | 1.000000 | 0.000000 | 1.000000 |

#### R-Squared Ratios (`vc.rsquare`)
| Cluster | Variable | RS_Own | RS_NC | RS_Ratio |
| :--- | :--- | :--- | :--- | :--- |
| 0 | fixed acidity | 0.882210 | 0.277256 | 0.162976 |
| 0 | density | 0.622070 | 0.246194 | 0.501362 |
| 0 | pH | 0.637076 | 0.194359 | 0.450478 |
| 1 | free sulfur dioxide | 0.777796 | 0.010358 | 0.224530 |
| 1 | total sulfur dioxide | 0.786660 | 0.042294 | 0.222761 |
| 1 | residual sugar | 0.202428 | 0.045424 | 0.835525 |

---

## 🛠️ Feature Selection Workflow

`fast-varclushi` is ideal for reducing multi-collinearity and selecting representative features for machine learning models:

1. **`RS_Own`**: Squared correlation between the variable and its own cluster component (higher is better).
2. **`RS_NC`**: Squared correlation between the variable and the nearest cluster component (lower is better).
3. **`RS_Ratio`**: Defined as:
   $$\text{RS\_Ratio} = \frac{1 - \text{RS\_Own}}{1 - \text{RS\_NC}}$$
   Small values of `RS_Ratio` indicate that a variable has high correlation with its own cluster and low correlation with the nearest cluster.

### Selecting Cluster Representatives

To select the single best representative feature from each cluster:

```python
# Select the variable with the lowest RS_Ratio in each cluster
selected_features = (
    vc.rsquare
    .sort_values(by=["Cluster", "RS_Ratio"])
    .groupby("Cluster")
    .first()["Variable"]
    .tolist()
)

print("Selected Representative Features:", selected_features)
# ['fixed acidity', 'free sulfur dioxide', 'chlorides', 'volatile acidity', 'alcohol']
```

---

## 📚 API Reference

### `varclushi.VarClusHi`

```python
VarClusHi(
    df,
    feat_list=None,
    maxeigval2=1,
    maxclus=None,
    n_rs=0,
    n_jobs=None,
    random_seed=None
)
```

#### Parameters

| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `df` | `pandas.DataFrame` | *Required* | Input DataFrame containing numeric variables. |
| `feat_list` | `list` or `None` | `None` | List of column names to cluster. If `None`, uses all columns in `df`. |
| `maxeigval2` | `float` | `1.0` | Threshold for stopping cluster splitting. Splits continue as long as the second eigenvalue of a cluster exceeds `maxeigval2`. |
| `maxclus` | `int` or `None` | `None` | Maximum number of clusters to form. If set, splitting stops when cluster count reaches `maxclus`. |
| `n_rs` | `int` | `0` | Number of random search restarts for reassigning variables after factor rotation split. |
| `n_jobs` | `int` or `None` | `None` | Number of parallel jobs for random search iterations (`-1` uses all CPU cores). |
| `random_seed` | `int`, `random.Random`, `None` | `None` | Seed or random number generator instance for reproducible clustering. |

#### Methods

- **`varclus(speedup=True, random_seed=None)`**
  Performs hierarchical variable clustering.
  - `speedup`: `bool` (default `True`). Enables high-performance precomputed correlation matrix mode.
  - `random_seed`: `int` or `None`. Optional seed override.

- **`correig(df, feat_list=None, n_pcs=2)`** *(static)*
  Computes correlation matrix, eigenvalues, eigenvectors, and variance proportions for a DataFrame.

- **`pca(df, feat_list=None, n_pcs=2)`** *(static)*
  Computes standardized principal components, eigenvalues, eigenvectors, and variance proportions.

#### Properties

- **`vc.info`** (`pandas.DataFrame`):
  Cluster summary table with columns:
  - `Cluster`: Cluster ID.
  - `N_Vars`: Number of variables in the cluster.
  - `Eigval1`: First eigenvalue (variance explained by primary cluster component).
  - `Eigval2`: Second eigenvalue (variance of second principal component).
  - `VarProp`: Proportion of cluster variance explained by the primary component.

- **`vc.rsquare`** (`pandas.DataFrame`):
  R-squared ratio analysis table with columns:
  - `Cluster`: Cluster ID.
  - `Variable`: Variable name.
  - `RS_Own`: $R^2$ with own cluster component.
  - `RS_NC`: $R^2$ with nearest cluster component.
  - `RS_Ratio`: $(1 - R^2_{Own}) / (1 - R^2_{NC})$.

---

### `varclushi.rotator.Rotator`

The `Rotator` module provides factor rotation algorithms for structural equation modeling, factor analysis, and custom dimension reduction workflows.

```python
from varclushi.rotator import Rotator

rotator = Rotator(
    method="varimax",
    normalize=None,
    power=4,
    kappa=0,
    gamma=0,
    max_iter=500,
    tol=1e-5,
    n_jobs=-1
)
```

#### Supported Rotation Methods

| Rotation Method | Category | Description |
| :--- | :--- | :--- |
| `"varimax"` | Orthogonal | Maximizes variance of squared loadings within columns (default). |
| `"quartimax"` | Orthogonal | Minimizes complexity of rows by maximizing sum of 4th powers of loadings. |
| `"oblimax"` | Orthogonal | Maximizes kurtosis of factor loadings. |
| `"equamax"` | Orthogonal | Compromise between Varimax and Quartimax. Controlled by `kappa`. |
| `"promax"` | Oblique | Oblique rotation constructed from Varimax rotated loadings. Controlled by `power`. |
| `"oblimin"` | Oblique | General family of oblique rotations. Controlled by `gamma`. |
| `"quartimin"` | Oblique | Special case of Oblimin (`gamma=0`). |

#### Methods & Attributes

- **`fit(X, y=None)`**: Fits rotation to unrotated loading matrix `X`.
- **`fit_transform(X, y=None)`**: Fits and returns rotated loading matrix.
- **`fit_transform_batch(X_list, n_jobs=None)`**: Batch rotates multiple loading matrices in parallel across CPU cores.
- **`loadings_`**: `numpy.ndarray` of rotated factor loadings.
- **`rotation_`**: `numpy.ndarray` rotation matrix.
- **`phi_`**: `numpy.ndarray` factor correlation matrix (for oblique rotations).

#### Standalone Rotator Example

```python
import numpy as np
from varclushi.rotator import Rotator

# Sample unrotated factor loading matrix (5 features, 2 factors)
loadings = np.array([
    [0.7, 0.2],
    [0.8, 0.1],
    [0.2, 0.6],
    [0.1, 0.9],
    [0.6, 0.5]
])

# Perform Varimax rotation
rotator = Rotator(method="varimax", normalize=True)
rotated_loadings = rotator.fit_transform(loadings)

print("Rotated Loadings:\n", rotated_loadings)
print("Rotation Matrix:\n", rotator.rotation_)
```

---

## 📊 Big Data Performance Benchmark

`fast-varclushi` delivers exceptional speedups on large-scale tabular datasets:

| Benchmark Dataset | Rows | Features | Memory Footprint | Execution Time | Total Clusters |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Synthetic Big Data** | **2,000,000** | **500** | **3.73 GB** | **74.78 seconds** ⚡ | **186** |

To run the benchmark on your local system:

```bash
python benchmark_bigdata.py
```

---

## 🧪 Running Tests

`fast-varclushi` includes 141 unit, integration, and regression tests.

Run the test suite with `pytest`:

```bash
pytest
```

---

## 📜 License

Distributed under the **GNU General Public License v3 (GPLv3)**. See [`LICENSE`](LICENSE) for details.

---

## 🤝 Authors & Credits

- **Aashay Belekar** ([@aashaybelekar](https://github.com/aashaybelekar)) - High-performance parallelization, precomputed matrix caching, vectorized `RSquare`, `Rotator` optimizations, and maintenance.
- **Xuan Jing** - Original `VarClusHi` package author.
- **Jeremy Biggs** - Original factor rotation routines port.
