Metadata-Version: 2.4
Name: dimkit
Version: 0.1.0
Summary: A unified dimensionality reduction toolkit with a consistent sklearn-like API.
Author: Aeron Zentner
License: MIT License
        
        Copyright (c) 2025 dimkit contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/aeron100/dimkit
Project-URL: Issues, https://github.com/aeron100/dimkit/issues
Keywords: dimensionality-reduction,machine-learning,pca,tsne,umap,sklearn,data-science
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: scikit-learn>=1.3
Requires-Dist: pandas>=1.5
Requires-Dist: matplotlib>=3.7
Provides-Extra: umap
Requires-Dist: umap-learn>=0.5; extra == "umap"
Provides-Extra: plot
Requires-Dist: plotly>=5.0; extra == "plot"
Provides-Extra: deep
Requires-Dist: torch>=2.0; extra == "deep"
Provides-Extra: phate
Requires-Dist: phate>=1.0; extra == "phate"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: ruff>=0.1; extra == "dev"
Requires-Dist: mypy>=1.5; extra == "dev"
Requires-Dist: pre-commit>=3.5; extra == "dev"
Provides-Extra: all
Requires-Dist: dimkit[deep,phate,plot,umap]; extra == "all"
Dynamic: license-file

# dimkit

A unified Python library for dimensionality reduction with a consistent, sklearn-compatible API.

dimkit wraps 10+ reduction methods behind a single interface so you can swap algorithms without rewriting code, compare them side-by-side, and get practical guidance on when to use each one.

---

## Features

- **Unified API** — every reducer implements `fit`, `transform`, `fit_transform`, `get_params`, `set_params`, `summary`
- **10 reduction methods** — PCA, TruncatedSVD, FactorAnalysis, FastICA, t-SNE, UMAP*, Isomap, LLE, MDS, SpectralEmbedding, Autoencoder*
- **Honest about limitations** — methods like t-SNE that don't support `transform()` raise `TransformNotSupportedError` with clear guidance
- **Compare reducers** — `compare_reducers()` runs multiple methods and collects runtime, trustworthiness, EVR, and reconstruction error
- **Recommender** — `recommend_reducer()` gives rule-based advice for your use case
- **Visualization utilities** — scatter plots, scree plots, cumulative variance, side-by-side comparisons
- **Evaluation metrics** — trustworthiness, continuity, pairwise distance correlation, reconstruction MSE
- **Lightweight core** — UMAP and the autoencoder are optional extras

\* Requires optional dependencies.

---

## Installation

```bash
# Core (PCA, SVD, FA, ICA, t-SNE, Isomap, LLE, MDS, SpectralEmbedding)
pip install dimkit

# With UMAP support
pip install dimkit[umap]

# With Plotly (interactive plots)
pip install dimkit[plot]

# With PyTorch autoencoder
pip install dimkit[deep]

# Everything
pip install dimkit[all]

# Development
pip install dimkit[dev]
```

### Install from source (editable)

```bash
git clone https://github.com/example/dimkit.git
cd dimkit
pip install -e ".[dev]"
```

---

## Quick Start

```python
from sklearn.preprocessing import StandardScaler
from dimkit import PCAReducer
from dimkit.datasets import load_iris

X, y, _ = load_iris()
X_scaled = StandardScaler().fit_transform(X)

reducer = PCAReducer(n_components=2)
X_2d = reducer.fit_transform(X_scaled)

print(reducer.explained_variance_ratio_)
# [0.729 0.228]
```

---

## Supported Methods

| Class | Method | Family | Deterministic | Supports `transform()` |
|---|---|---|---|---|
| `PCAReducer` | PCA | Linear | Yes | Yes |
| `TruncatedSVDReducer` | TruncatedSVD / LSA | Linear | Yes | Yes |
| `FactorAnalysisReducer` | Factor Analysis | Linear | Yes | Yes |
| `FastICAReducer` | FastICA | Linear | No | Yes |
| `TSNEReducer` | t-SNE | Manifold | No | **No** |
| `UMAPReducer` | UMAP | Manifold | No | Yes |
| `IsomapReducer` | Isomap | Manifold | Yes | Yes |
| `LLEReducer` | LLE | Manifold | Yes | Yes |
| `MDSReducer` | MDS | Manifold | No | **No** |
| `SpectralReducer` | SpectralEmbedding | Manifold | No | **No** |
| `AutoencoderReducer` | Autoencoder | Deep | No | Yes |

---

## Comparing Multiple Methods

```python
from dimkit import PCAReducer, TSNEReducer, IsomapReducer
from dimkit.compare import compare_reducers
from sklearn.preprocessing import StandardScaler
from dimkit.datasets import load_digits

X, y, _ = load_digits(n_class=5)
X_scaled = StandardScaler().fit_transform(X)

report = compare_reducers(
    X_scaled,
    reducers=[
        PCAReducer(n_components=2),
        TSNEReducer(n_components=2, perplexity=30, random_state=42),
        IsomapReducer(n_components=2, n_neighbors=10),
    ],
    compute_trustworthiness=True,
)

print(report.summary())
print(f"Fastest:              {report.fastest().reducer_name}")
print(f"Best trustworthiness: {report.best_trustworthiness().reducer_name}")
```

---

## Visualizing Embeddings

```python
from dimkit.visualization import plot_embedding, plot_scree, plot_comparison

# Single embedding
ax = plot_embedding(X_2d, labels=y, title="PCA — Digits")

# Scree plot
ax = plot_scree(reducer)

# Side-by-side comparison
fig = plot_comparison(report.results, labels=y)
```

---

## Getting a Recommendation

```python
from dimkit.compare import recommend_reducer

rec = recommend_reducer(
    goal="visualization",       # or "preprocessing", "clustering", "manifold"
    data_shape=(5000, 100),
    sparse=False,
    interpretability_needed=False,
)
print(rec)
# Recommendation: UMAP
# Alternatives:   t-SNE, PCA
# Reasoning: UMAP produces visually strong 2D/3D embeddings...
```

---

## Handling Unsupported Operations

dimkit raises clear, informative errors rather than silently failing:

```python
from dimkit import TSNEReducer
from dimkit.exceptions import TransformNotSupportedError

reducer = TSNEReducer(n_components=2, random_state=42)
reducer.fit_transform(X_scaled)

try:
    reducer.transform(X_scaled[:10])  # t-SNE does not support this
except TransformNotSupportedError as e:
    print(e)
    # 't-SNE' does not support transform() on new data. Use fit_transform()
    # on the full dataset instead. For out-of-sample projection, use UMAPReducer.
```

---

## Preprocessing Helpers

```python
from dimkit.utils.preprocessing import (
    standardize_features,
    scale_features,
    check_for_missing,
    impute_missing,
)

# Check for NaN before processing
info = check_for_missing(X)
if info["has_missing"]:
    X = impute_missing(X, strategy="mean")

# Scale (returns array + fitted scaler)
X_scaled, scaler = scale_features(X, method="standard")
```

---

## Evaluation Metrics

```python
from dimkit.metrics import (
    trustworthiness,
    continuity,
    reconstruction_mse,
    pairwise_distance_correlation,
)

trust = trustworthiness(X_original, X_embedded, n_neighbors=5)
cont  = continuity(X_original, X_embedded, n_neighbors=5)
corr  = pairwise_distance_correlation(X_original, X_embedded)
mse   = reconstruction_mse(X_original, X_reconstructed)
```

---

## Running Tests

```bash
cd dimkit
pip install -e ".[dev]"
pytest
```

---

## Limitations and Cautions

### t-SNE
- **Cluster sizes and inter-cluster distances in t-SNE plots are not interpretable**
- Results vary significantly between runs — always set `random_state`
- Perplexity strongly affects visual output; try several values (5, 30, 50)
- Not suitable for large datasets (>50k samples)

### UMAP
- Global distances are more meaningful than t-SNE but still approximate
- `n_neighbors` controls local/global balance — tune it
- Always set `random_state`

### General
- Dimensionality reduction for visualization does not guarantee useful features for modeling
- Preprocessing matters: scale your data before applying most reducers
- Nonlinear embeddings can distort both local and global relationships
- Visual cluster separation does not imply that clusters are separable in feature space

---

## Project Structure

```
dimkit/
  pyproject.toml
  src/
    dimkit/
      base.py           # Abstract BaseReducer
      exceptions.py     # Custom exceptions
      reducers/         # One file per method
      compare/          # compare_reducers, recommend_reducer
      metrics/          # trustworthiness, reconstruction MSE, runtime
      visualization/    # scatter, scree, comparison plots
      datasets/         # sample datasets (Iris, Digits, blobs, Swiss roll)
      utils/            # validation, preprocessing helpers
  tests/                # pytest suite
  examples/             # runnable example scripts
  docs/                 # method_guide.md, examples.md
```

---

## Future Enhancements

- Kernel PCA (`KernelPCAReducer`)
- Non-negative Matrix Factorisation (NMF)
- PHATE (Potential of Heat-diffusion for Affinity-based Trajectory Embedding)
- Interactive Plotly-based comparison dashboard
- Variational Autoencoder (VAE) reducer
- Pipeline integration with sklearn `Pipeline`
- Hyperparameter tuning helpers (grid search over perplexity / n_neighbors)
- CSV/DataFrame loading convenience wrapper

---

## License

MIT — see [LICENSE](LICENSE).
