Metadata-Version: 2.4
Name: ml-experiment-framework
Version: 0.1.0
Summary: Dynamic, leakage-safe classic Machine Learning experimentation framework
Author-email: Taha Hussein <taha.hussein.two@example.com>
License: MIT
Project-URL: Homepage, https://github.com/اسمك/ml-experiment-framework
Project-URL: Repository, https://github.com/اسمك/ml-experiment-framework
Keywords: machine-learning,automl,tabular,scikit-learn,data-science
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scikit-learn>=1.3.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: joblib>=1.3.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: matplotlib>=3.7.0
Requires-Dist: seaborn>=0.12.0
Requires-Dist: scipy>=1.10.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Provides-Extra: bayesian
Requires-Dist: optuna>=3.0; extra == "bayesian"
Provides-Extra: shap
Requires-Dist: shap>=0.42; extra == "shap"
Dynamic: license-file

# Classic Machine Learning Framework

A **dynamic, leakage-safe, production-oriented** classic ML experimentation framework built primarily on scikit-learn.

It behaves like a lightweight AutoML / ML experiment runner for **tabular** data:

- Automatically inspects data
- Detects problem type (binary / multiclass classification, regression)
- Detects feature types (numeric, categorical, boolean, datetime, ID-like, text-like, constant, high-missingness)
- Builds reasoned preprocessing pipelines
- Runs staged model selection (baseline → candidates → shortlist → tune)
- Evaluates on a held-out test set **once**
- Produces error analysis, feature importance, and a full experiment report

**Design priorities:** correctness, no data leakage, reproducibility, strong baselines, explainable decisions, maintainability, extensibility.

---

## Architecture

```
ml_framework/
├── main.py                 # CLI orchestration
├── configs/default.yaml
├── src/
│   ├── data/               # load, validate, profile, split
│   ├── detection/          # problem type, feature types, target, decision engine
│   ├── preprocessing/      # ColumnTransformer pipelines
│   ├── features/           # engineering / selection hooks
│   ├── models/             # extensible registry
│   ├── training/           # baseline, CV, tuning, final train
│   ├── evaluation/         # metrics, test eval, error analysis
│   ├── explainability/     # permutation (+ optional SHAP)
│   ├── experiments/        # runner + reporter
│   ├── persistence/        # joblib full-pipeline save/load
│   └── utils/              # logging, config, seeds
├── artifacts/              # models, reports, plots
└── tests/
```

Every major automatic choice is logged as:

| Field | Meaning |
|-------|---------|
| Decision | What was chosen |
| Reason | Why |
| Action | Concrete effect |
| Confidence | high / medium / low |

---

## Installation

```bash
cd ml_framework
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
```

Python 3.11+ recommended.

---

## Quick start

```bash
# Classification or regression — framework detects automatically
python main.py --data path/to/train.csv --target my_target

# Explicit problem type
python main.py --data train.csv --target SalePrice --problem-type regression

# External test set (Kaggle-style)
python main.py --train train.csv --test test.csv --target SalePrice

# Config file
python main.py --config configs/default.yaml --data train.csv --target y

# Disable tuning for a fast run
python main.py --data train.csv --target y --no-tuning
```

### Predict

```bash
python main.py predict \
  --model artifacts/models/best_model.joblib \
  --data new_data.csv \
  --output predictions.csv
```

### Profile only

```bash
python main.py profile --data train.csv --target y
```

---

## Configuration

See `configs/default.yaml`. Important knobs:

- `problem.type`: `auto` | `binary_classification` | `multiclass_classification` | `regression`
- `split.test_size`, `random_state`
- `preprocessing.high_cardinality_threshold`, `missing_threshold`
- `models.include` / `exclude`
- `tuning.enabled`, `method` (`randomized_search` | `grid_search`), `n_iter`, `shortlist_size`
- `evaluation.primary_metric`: `auto` or sklearn scorer name / friendly alias (`rmse`, `f1`, `roc_auc`, …)
- `explainability.enabled`

CLI flags override YAML.

---

## Supported problems & models

**Problems:** binary classification, multiclass classification, regression.

**Models (registry):** LogisticRegression, RidgeClassifier, DecisionTree, RandomForest, ExtraTrees, HistGradientBoosting, GradientBoosting, SVC, KNeighbors (classification); LinearRegression, Ridge, Lasso, ElasticNet, DecisionTree, RandomForest, ExtraTrees, HistGradientBoosting, GradientBoosting, SVR, KNeighbors (regression).

The **Decision Engine** selects a small candidate set based on dataset size and feature mix — it does **not** brute-force every model.

---

## How leakage is prevented

1. Train/test **split happens before any fit**.
2. All imputation, scaling, encoding live inside an sklearn `Pipeline` + `ColumnTransformer`.
3. CV and tuning operate on the full pipeline (preprocess + model).
4. Final metrics are computed **once** on the untouched test set.
5. Model selection uses CV scores, never test scores.
6. External test sets are never used during training or tuning.

---

## Extensibility

### Add a model

```python
from src.models.registry import register_model
from sklearn.ensemble import AdaBoostClassifier

register_model(
    "AdaBoostClassifier",
    AdaBoostClassifier,
    problem_types=["binary_classification", "multiclass_classification"],
    default_params={"random_state": 42},
    search_space={"n_estimators": [50, 100, 200]},
)
```

### Add a data loader

```python
from src.data.loader import register_loader

@register_loader("feather")
def load_feather(path):
    import pandas as pd
    return pd.read_feather(path)
```

### Custom metric

Pass `--metric my_scorer` if registered with sklearn, or set `evaluation.primary_metric` in YAML.

---

## Output structure

```
artifacts/
├── models/best_model.joblib    # full pipeline
├── reports/
│   ├── final_report.json
│   └── final_report.html
└── plots/
    ├── target_distribution.png
    ├── missing_heatmap.png
    ├── correlation_heatmap.png
    └── residuals.png           # regression
```

---

## Example summary output

```
============================================================
ML EXPERIMENT COMPLETE
============================================================

Problem: Regression
Dataset: 1460 rows × 81 columns
Train: 1168 rows
Test: 292 rows
Primary Metric: RMSE
Baseline: -0.42
Best Model: HistGradientBoostingRegressor
CV: -0.251 ± 0.009
Test rmse: 0.237
Test mae: 0.164
Test r2: 0.891
Model saved: artifacts/models/best_model.joblib
Report: artifacts/reports/final_report.html
============================================================
```

---

## Limitations

- Classic tabular ML only (no deep learning, no raw text/image models).
- Text-like columns are detected and dropped with a clear message.
- Very high-cardinality categoricals use OrdinalEncoder (not target encoding) to avoid leakage.
- Bayesian optimization / SHAP are optional extras.
- Not a guarantee of the globally optimal model — it aims for strong, reproducible baselines with transparent decisions.

---

## Tests

```bash
cd ml_framework
pytest tests/ -q
```

---

## License

MIT-style — use freely in research and production prototypes.
