Metadata-Version: 2.4
Name: pythios
Version: 0.1.0
Summary: The oracle for your data — low-code ML for everyone.
Author-email: Advaith <advaith@pythios.xyz>
License-Expression: MIT
Project-URL: Homepage, https://pythios.xyz
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENCE
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Requires-Dist: matplotlib>=3.7
Requires-Dist: seaborn>=0.12
Requires-Dist: requests>=2.28
Requires-Dist: xgboost>=2.0
Requires-Dist: lightgbm>=4.0
Requires-Dist: optuna>=3.0
Provides-Extra: full
Requires-Dist: torch>=2.0; extra == "full"
Requires-Dist: tensorflow>=2.13; extra == "full"
Requires-Dist: shap>=0.43; extra == "full"
Requires-Dist: umap-learn>=0.5; extra == "full"
Requires-Dist: plotly>=5.0; extra == "full"
Requires-Dist: sentence-transformers>=2.2; extra == "full"
Requires-Dist: faiss-cpu>=1.7; extra == "full"
Requires-Dist: Pillow>=9.0; extra == "full"
Requires-Dist: imbalanced-learn>=0.11; extra == "full"
Dynamic: license-file

# Pythios — The Oracle for Your Data

> *Low-code machine learning for everyone — from beginners to professionals.*

**Domain:** pythios.xyz
**Package:** `pip install pythios`
**Pronunciation:** PITH-ee-os
**Tagline:** Named after Apollo Pythios — the oracular epithet of Apollo at Delphi.
The Pythia received a question and returned a prophecy. That is exactly what
a machine learning model does.

---

## IMPORTANT — READ THIS FIRST

This README is the single source of truth for building Pythios from scratch.
Read it entirely before writing a single line of code. Every design decision,
every method signature, every edge case is documented here. The goal is that
this project builds correctly on the first attempt with no ambiguity.

---

## Overview

Pythios is a low-code Python ML library that wraps scikit-learn, PyTorch,
TensorFlow, XGBoost, LightGBM, Optuna, SHAP, UMAP, sentence-transformers,
and Faiss behind a unified, beginner-friendly interface — while remaining
powerful enough for professionals.

The philosophy:
- One object (`Data`) flows through the entire pipeline
- Every class works with zero arguments (sensible defaults everywhere)
- Heavy dependencies are lazy-imported (only loaded when actually needed)
- Task type (binary / multiclass / regression) is auto-detected and propagated
- Output is always human-readable — no raw numpy arrays printed to console
- Never mutate data in place — transformations return new `Data` objects

---

## Complete Project Structure

Build EXACTLY this structure. Do not deviate.

pythios/                          ← outer project root
├── pythios/                      ← the actual Python package
│   ├── __init__.py
│   ├── _utils.py
│   ├── data.py
│   ├── preprocess.py
│   ├── visualize.py
│   ├── models.py
│   ├── evaluate.py
│   ├── unsupervised.py
│   ├── embeddings.py
│   └── tools.py
├── tests/
│   ├── __init__.py               ← empty file
│   ├── test_data.py
│   ├── test_preprocess.py
│   ├── test_visualize.py
│   ├── test_models.py
│   ├── test_evaluate.py
│   ├── test_unsupervised.py
│   ├── test_embeddings.py
│   └── test_tools.py
├── demo/
│   └── demo_all.py               ← full runnable demo (see section below)
├── pyproject.toml
└── README.md
```

---

## pyproject.toml — Copy This Exactly

```toml
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "pythios"
version = "0.1.0"
description = "The oracle for your data — low-code ML for everyone."
readme = "README.md"
requires-python = ">=3.9"
license = { text = "MIT" }
authors = [{ name = "Advaith", email = "advaith@pythios.xyz" }]
dependencies = [
    "pandas>=2.0",
    "numpy>=1.24",
    "scikit-learn>=1.3",
    "matplotlib>=3.7",
    "seaborn>=0.12",
    "requests>=2.28",
]

[project.optional-dependencies]
full = [
    "xgboost>=2.0",
    "lightgbm>=4.0",
    "torch>=2.0",
    "torchvision>=0.15",
    "tensorflow>=2.13",
    "optuna>=3.0",
    "shap>=0.43",
    "umap-learn>=0.5",
    "plotly>=5.0",
    "sentence-transformers>=2.2",
    "faiss-cpu>=1.7",
    "Pillow>=9.0",
    "imbalanced-learn>=0.11",
]

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]

[tool.setuptools]
packages = ["pythios"]
```

---

## Installation

```bash
# Clone or create the project folder
cd pythios

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate       # Mac/Linux

# Install in editable mode with base dependencies
pip install -e .

# Install ALL optional dependencies (recommended for development)
pip install -e ".[full]"

# Install pytest
pip install pytest
```

---

## `__init__.py` — The Package Entry Point

```python
"""
Pythios — The oracle for your data.

Low-code machine learning for everyone.
pip install pythios | pythios.xyz

Classes:
    Data         — load, inspect, split datasets
    Preprocess   — scale, encode, impute, select features
    Visualize    — distributions, correlations, missing values
    Models       — compare, train, tune, save ML models
    Evaluate     — metrics, SHAP, ROC curves, cross-validation
    Unsupervised — clustering, dimensionality reduction
    Embeddings   — text and image vectors, similarity search
    Tools        — utilities, LLM coding assistant, profiling
"""

from .data import Data
from .tools import Tools

try:
    from .preprocess import Preprocess
except Exception:
    pass

try:
    from .visualize import Visualize
except Exception:
    pass

try:
    from .models import Models
except Exception:
    pass

try:
    from .evaluate import Evaluate
except Exception:
    pass

try:
    from .unsupervised import Unsupervised
except Exception:
    pass

try:
    from .embeddings import Embeddings
except Exception:
    pass

__version__ = "0.1.0"
__author__ = "Advaith"
__all__ = [
    "Data",
    "Preprocess",
    "Visualize",
    "Models",
    "Evaluate",
    "Unsupervised",
    "Embeddings",
    "Tools",
]
```

---

## Build Order

Build files in this exact order. Each one depends on the previous.

1. `_utils.py` — shared helpers (lazy import, task detection, formatting)
2. `data.py` — Data class (everything else depends on this)
3. `tools.py` — Tools class (standalone utilities + LLM)
4. `preprocess.py` — depends on Data
5. `models.py` — depends on Data
6. `evaluate.py` — depends on Models and Data
7. `visualize.py` — depends on Data
8. `unsupervised.py` — depends on Data
9. `embeddings.py` — standalone but can accept Data

---

## FILE 1: `_utils.py`

This file contains three things: the lazy import helper, the task type
detector, and the table formatter. These are used by every other file.

### `lazy_import(module_name, install_name=None)`

Imports a module only when called. If the module is not installed, raises
an ImportError with a clear pip install instruction. This is the mechanism
that keeps the base Pythios install lightweight — torch, tensorflow, shap
etc. are never imported at the top of any file.

```python
def lazy_import(module_name: str, install_name: str = None):
    try:
        return importlib.import_module(module_name)
    except ImportError:
        pkg = install_name or module_name
        raise ImportError(
            f"'{module_name}' is required for this feature.\n"
            f"Install it with: pip install {pkg}\n"
            f"Or install everything: pip install pythios[full]"
        ) from None
```

### `detect_task_type(target_series) -> str`

Auto-detects task type from the target column. This is the most important
utility function — its output propagates through the entire pipeline.

Rules (apply in this order):
1. If dtype is object or category → classification
   - 2 unique values → "binary"
   - 3+ unique values → "multiclass"
2. If dtype is float → "regression" (always)
3. If dtype is integer:
   - Find unique values, sort them
   - Check if they form a CONTIGUOUS range (e.g. [0,1] or [0,1,2,3])
   - AND max value is <= 100 AND n_unique <= 20
   - If contiguous + small → classification
     - n_unique == 2 → "binary"
     - n_unique > 2 → "multiclass"
   - Otherwise → "regression"

The contiguous range check is the key insight. Class labels like [0,1,2,3]
are contiguous. Salary values like [50000,60000,70000] are not contiguous
even though they have few unique values in a small dataset.

Implementation:
```python
def detect_task_type(target_series) -> str:
    if target_series is None or len(target_series) == 0:
        return None
    n_unique = target_series.nunique()
    is_categorical = (
        target_series.dtype == "object"
        or target_series.dtype.name == "category"
    )
    is_float = target_series.dtype.kind == "f"

    if is_categorical:
        return "binary" if n_unique == 2 else "multiclass"
    elif is_float:
        return "regression"
    else:
        unique_vals = sorted(target_series.dropna().unique())
        min_val = unique_vals[0]
        max_val = unique_vals[-1]
        is_contiguous = (max_val == min_val + n_unique - 1)
        is_small = (n_unique <= 20 and max_val <= 100)
        if is_contiguous and is_small:
            return "binary" if n_unique == 2 else "multiclass"
        else:
            return "regression"
```

### `format_table(headers, rows, title=None)`

Pretty-prints a formatted table. Calculates column widths automatically.
Used throughout Pythios for readable console output.

---

## FILE 2: `data.py`

The `Data` class is the universal container. Everything else takes a Data
object as input.

### Internal State

```python
self._df        # pd.DataFrame — full original data, never mutated
self.target     # str — name of target column (None if unsupervised)
self.task_type  # str — "binary", "multiclass", "regression", or None
self.X_train    # pd.DataFrame — features, training set (set by .split())
self.X_test     # pd.DataFrame — features, test set
self.y_train    # pd.Series — target, training set
self.y_test     # pd.Series — target, test set
self._is_split  # bool — whether .split() has been called
```

### Constructors (all class methods)

#### `Data.from_csv(path, target, sep=",", encoding="utf-8")`
Loads a CSV file using pd.read_csv. Returns a Data object.

#### `Data.from_dataframe(df, target)`
Wraps an existing pandas DataFrame. Does df.copy() internally.

#### `Data.from_url(url, target, sep=",")`
Downloads a CSV from a URL. Uses lazy_import("requests").
Uses requests.get(), then pd.read_csv(StringIO(response.text)).

#### `Data.from_excel(path, target, sheet=0)`
Loads an Excel file using pd.read_excel(path, sheet_name=sheet).

#### `Data.from_sklearn(name, target=None)`

Loads a built-in sklearn dataset by name. No file download needed.
This is for quick prototyping and testing.

Supported datasets and their loaders:
```python
{
    "iris":               datasets.load_iris,
    "wine":               datasets.load_wine,
    "breast_cancer":      datasets.load_breast_cancer,
    "digits":             datasets.load_digits,
    "diabetes":           datasets.load_diabetes,
    "california_housing": datasets.fetch_california_housing,
    "linnerud":           datasets.load_linnerud,
}
```

Implementation details:
- Import sklearn.datasets at the top of this method (not lazy — sklearn
  is a required dependency)
- Build DataFrame from dataset.data using dataset.feature_names if available,
  otherwise name columns "feature_0", "feature_1", etc.
- Target column name defaults to "target" if not provided
- For classification datasets (those with target_names attribute),
  convert integer targets to string class names using:
  `pd.Categorical.from_codes(dataset.target, dataset.target_names)`
  BUT only if len(unique targets) <= 20 (to avoid treating regression as classification)
- Print confirmation: `✓ Loaded sklearn dataset 'iris' — 150 rows, 5 cols`
- Raise ValueError with list of available names if unknown name provided

#### `Data.from_tab(path, target)`

Loads an Orange .tab file. These have a specific 3-row header format:
- Row 1: column names (tab-separated)
- Row 2: column types (continuous, discrete, string)
- Row 3: column roles (class, meta, ignore, or blank)

Implementation:
1. Open file, read first 3 lines to get col_names, col_types, col_roles
2. Use pd.read_csv(path, sep="\t", skiprows=3, header=None, names=col_names)
3. Drop columns where role is "meta" or "ignore" (unless it's the target)
4. Cast columns where type is "continuous" to float using pd.to_numeric(errors="coerce")
5. Print dropped columns if any
6. Print confirmation

#### `Data.from_sklearn` — Available Datasets Table

| Name | Task | Rows | Features |
|---|---|---|---|
| iris | multiclass | 150 | 4 |
| wine | multiclass | 178 | 13 |
| breast_cancer | binary | 569 | 30 |
| digits | multiclass | 1797 | 64 |
| diabetes | regression | 442 | 10 |
| california_housing | regression | 20640 | 8 |
| linnerud | regression | 20 | 3 |

### Inspection Methods

#### `data.describe()`

Prints an Orange-style dataset summary. No return value (returns None).

Format:
```
Pythios · Data Summary
────────────────────────────────────────────────────────────────────────────────
 Rows: 891   Columns: 12
 Target: Survived (binary)
 Missing values: 3 column(s) affected

 Column          Type        Missing    Unique    Notes
────────────────────────────────────────────────────────────────────────────────
 Age             float64     19.9%      88
 Cabin           object      77.1%      147       high cardinality
 Embarked        object      0.2%       3
 Fare            float64     0.0%       248
 Survived        int64       0.0%       2         ← TARGET
```

Notes logic (check in this order):
- If col == self.target → "← TARGET"
- Elif pct_missing > 50 → "mostly missing"
- Elif pct_missing > 20 → "high missing"
- Elif n_unique == 1 → "constant value"
- Elif n_unique > 100 and dtype == "object" → "high cardinality"
- Else → ""

#### `data.missing_report() -> pd.DataFrame`

Returns a DataFrame with columns: Column, Missing_Count, Missing_Percent.
Only includes columns that actually have missing values.
Sorted by Missing_Count descending.

### Split Method

#### `data.split(test=0.2, seed=42, stratify=True)`

Splits data into train/test sets using sklearn's train_test_split.

Critical rules:
- If already split → raise ValueError (not a warning, a hard error)
- stratify only applies when task_type in ["binary", "multiclass"] AND y is not None
- Reset index on all four output DataFrames/Series (drop=True)
- Print: `✓ Split 891 rows → 712 train (79.9%), 179 test (20.1%)`
- Store results in self.X_train, self.X_test, self.y_train, self.y_test
- Set self._is_split = True

### Properties

```python
@property shape -> tuple          # self._df.shape
@property dtypes -> pd.Series     # self._df.dtypes
@property columns -> list         # self._df.columns.tolist()
```

### Utility Methods

All of these return NEW Data objects — never mutate self.

#### `data.head(n=5) -> pd.DataFrame`
Returns self._df.head(n). Note: returns DataFrame not Data.

#### `data.tail(n=5) -> pd.DataFrame`
Returns self._df.tail(n).

#### `data.copy() -> Data`
Deep copies _df, target, task_type. If split exists, copies all four
split DataFrames too. Returns new Data object.

#### `data.drop_columns(cols: list) -> Data`
Drops specified columns. Uses errors="ignore" so missing columns don't crash.
New target = self.target (unchanged — target is never dropped by this method).

#### `data.select_columns(cols: list) -> Data`
Keeps only specified columns.
If self.target not in cols → new target = None.

#### `data.sample(n=1000, seed=42) -> Data`
Random sample of n rows (capped at len). Returns new Data object.

### `__repr__` and `__str__`

```python
def __repr__(self):
    split_str = " [split]" if self._is_split else ""
    return f"Data(shape={self.shape}, target='{self.target}', task={self.task_type}){split_str}"

def __str__(self):
    return f"Pythios Data — {self.shape[0]:,} rows × {self.shape[1]} cols"
```

---

## FILE 3: `tools.py`

Tools contains two categories of functionality:
1. Data utilities (profile, suggest, detect_outliers, balance, validate, export)
2. LLM-powered coding assistant (llm, coder, explain, review, fix, ask_about_data)

ALL methods are static or classmethods. Tools is NEVER instantiated.

### The `CodeResult` class (defined ABOVE the Tools class in tools.py)

This is returned by `Tools.coder()` and `Tools.fix()`.

```python
class CodeResult:
    def __init__(self, code: str, description: str):
        self.code = code
        self.description = description
        self.namespace = {}
        self._ran = False

    def show(self) -> "CodeResult":
        # Print code with line numbers
        # Return self for chaining

    def run(self, verbose=True) -> "CodeResult":
        # Execute code with exec() in self.namespace
        # Inject variables into caller's frame using:
        #   frame = inspect.currentframe().f_back
        #   frame.f_locals.update({k: v for k, v in self.namespace.items() if not k.startswith("__")})
        #   ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), ctypes.c_int(0))
        # Print confirmation with list of created variables
        # Return self for chaining
```

The frame injection trick is what makes `coder.run(); print(n)` work —
variables created inside the exec'd code are injected into the caller's
actual Python frame so they appear as local variables.

Required imports at top of tools.py:
```python
import inspect
import ctypes
import re
import os
import time
import pandas as pd
import numpy as np
from ._utils import lazy_import, format_table
```

### LLM Configuration (class variables on Tools)

```python
_llm_model    = "llama3"
_llm_base_url = "http://localhost:11434"
```

These are defaults. Users change them with `Tools.configure_llm()`.

### `Tools._call_llm(prompt) -> str` (classmethod, private)

Calls the Ollama REST API at `{_llm_base_url}/api/generate`.
Uses requests.post with json={"model": cls._llm_model, "prompt": prompt, "stream": False}.
Timeout: 120 seconds.
On any exception, raise ConnectionError with instructions to install Ollama
and run `ollama pull {model_name}`.

### Data Utility Methods

#### `Tools.profile(data) -> pd.DataFrame`
Computes per-column stats: dtype, missing%, mean, std, min, 25%, 50%,
75%, max, skewness. Numeric columns get all stats. Object columns get "—"
for numeric stats. Prints full summary including memory usage. Returns DataFrame.

#### `Tools.suggest(data)`
Analyses data and prints actionable suggestions. Checks for:
1. Columns with >20% missing → suggest imputation
2. Class imbalance (ratio > 3:1) → suggest Tools.balance()
3. Constant columns (nunique <= 1) → suggest dropping
4. High cardinality categoricals (nunique > 50, not target) → suggest encoding
5. Duplicate rows → suggest deduplication
6. Skewed features (|skew| > 2) → suggest robust scaling

Format each issue as:
```
 ⚠  {issue description}
     → {suggested action}
```

If no issues: print `✓ No obvious issues detected. Looks like a clean dataset!`

#### `Tools.detect_outliers(data, method="iqr", threshold=1.5) -> pd.DataFrame`
Checks all numeric columns.
- "iqr": flag values outside [Q1 - threshold*IQR, Q3 + threshold*IQR]
- "zscore": flag values where |z-score| > threshold
Returns DataFrame with columns: Column, Outlier_Count, Outlier_%, Bounds.
Sorted by Outlier_Count descending.

#### `Tools.balance(data, method="oversample", seed=42) -> Data`
Fixes class imbalance. Requires target to be set and task_type to be
binary or multiclass. Raises ValueError otherwise.

- "oversample": duplicate minority class rows with replacement
- "undersample": randomly remove majority class rows
- "smote": use imbalanced-learn SMOTE (lazy import "imblearn.over_sampling")

Print before/after class distribution. Return new Data object.

#### `Tools.validate(data) -> bool`
Runs sanity checks. Returns True if all pass, False if any fail.
Checks: empty dataset, missing target, all-missing columns,
infinite values, single-class target.
Print ✓ for passes, ✗ for failures.

#### `Tools.export(data, path, split="full")`
Save to file. Support .csv, .xlsx, .json, .parquet based on extension.
split can be "full", "train", or "test" (train/test require .split() called).
Raise ValueError for unsupported extensions.
Print: `✓ Exported 891 rows to 'output.csv'`

#### `Tools.memory_usage(data) -> str`
Return human-readable memory string: "4.2 MB", "800 KB", "1.2 GB".
Use data._df.memory_usage(deep=True).sum() for byte count.

#### `Tools.timer` (context manager class nested inside Tools)
```python
class timer:
    def __init__(self, label="operation"):
        self.label = label
    def __enter__(self):
        self._start = time.time()
        return self
    def __exit__(self, *args):
        elapsed = time.time() - self._start
        print(f"⏱  {self.label} took {elapsed:.2f}s")
```

#### `Tools.set_seed(seed=42)`
Sets seed for: random, numpy, and torch (lazy import — only if installed).
Print: `✓ Random seed set to 42`

#### `Tools.compare_datasets(data1, data2) -> pd.DataFrame`
Side-by-side comparison of two Data objects.
Compare: rows, columns, missing values, memory, numeric cols, categorical cols.
Useful for checking before vs after preprocessing.

### LLM Methods

#### `Tools.configure_llm(model=None, base_url=None)`
Updates cls._llm_model and/or cls._llm_base_url.
Print confirmation.

#### `Tools.llm(prompt, print_response=True) -> str`
Direct LLM prompt. Prints header, calls _call_llm, prints response.
Returns response string.

#### `Tools.coder(description, show=True) -> CodeResult`
Generates Python code from natural language.

System prompt must:
1. Ask for ONLY raw Python code, no markdown, no backticks, no explanation
2. Ask for no import statements unless absolutely necessary
3. Ask to store results in clearly named variables

After getting response, strip markdown fences with regex:
```python
code = re.sub(r"```(?:python)?", "", raw)
code = re.sub(r"```", "", code).strip()
```

Return CodeResult(code=code, description=description).
If show=True, call result.show() before returning.

#### `Tools.explain(code) -> str`
Sends code to LLM asking for plain English explanation.
Prints explanation. Returns string.

#### `Tools.review(code) -> str`
Sends code to LLM asking for bug review, performance issues, PEP8 style.
Asks LLM to respond as a bullet list. Prints review. Returns string.

#### `Tools.fix(code, error) -> CodeResult`
Sends broken code + error message to LLM, gets fixed code back.
Strip markdown fences from response.
Call result.show() before returning.

#### `Tools.ask_about_data(data, question) -> str`
Builds a dataset summary (NOT raw rows — only statistics) and sends
it to the LLM with the user's question. The summary includes:
- Row and column counts
- Target and task type
- Column dtypes
- Missing percentages
- 5 sample values per column (from .unique())

IMPORTANT: Raw data rows are never sent to the LLM. Only statistics.
This preserves privacy and keeps prompts short.

---

## FILE 4: `preprocess.py`

### Class Behaviour

Preprocess wraps sklearn's ColumnTransformer and Pipeline. It accepts a
Data object and builds a transformation pipeline that is fitted on X_train
and applied to both X_train and X_test (preventing data leakage).

### Constructor

```python
def __init__(self, data: Data):
    # Store data reference
    # Detect numeric and categorical columns (excluding target)
    # Initialize _transformer to None
    # Initialize _steps list (records what the user configured)
```

Column detection:
```python
numeric_cols     = df.select_dtypes(include="number").columns.tolist()
categorical_cols = df.select_dtypes(include="object").columns.tolist()
# Remove target from both lists
```

### Configuration Methods (all return self for chaining)

#### `p.auto()`
Automatically configure the full pipeline. This is the recommended path.

For numeric columns: median imputation + StandardScaler
For categorical columns: most_frequent imputation + OneHotEncoder(handle_unknown="ignore", sparse_output=False)
For high-cardinality categorical columns (nunique > 15): OrdinalEncoder instead of OneHotEncoder

#### `p.scale(method="standard")`
Options: "standard" (StandardScaler), "minmax" (MinMaxScaler),
"robust" (RobustScaler), "none" (no scaling).
Stores setting in self._scale_method.

#### `p.encode(method="onehot")`
Options: "onehot" (OneHotEncoder), "label" (LabelEncoder),
"ordinal" (OrdinalEncoder), "target" (TargetEncoder).
Stores setting in self._encode_method.

#### `p.impute(method="median", k=5)`
Options: "median", "mean", "mode", "knn" (KNNImputer with k neighbors),
"drop" (dropna).
Stores setting in self._impute_method.

#### `p.select_features(k=10, method="mutual_info")`
Options: "mutual_info" (SelectKBest with mutual_info), "variance",
"rfe" (with random forest estimator).
Stores setting in self._feature_selection.

#### `p.pca(n_components=0.95)`
n_components can be int (exact) or float < 1 (variance explained).
Stores setting in self._pca_components.

### `p.apply() -> Data`

This is the most important method. It:
1. Builds the sklearn ColumnTransformer from stored settings
2. Fits on X_train ONLY
3. Transforms both X_train and X_test
4. Wraps results back into a new Data object (preserving y_train, y_test, task_type, target)
5. Restores column names (use get_feature_names_out() from ColumnTransformer)
6. Prints summary: `✓ Preprocessing applied — 10 → 47 features (after one-hot encoding)`
7. Returns new Data object

CRITICAL: Must fit only on training data. Never fit on test data.
The returned Data object must have _is_split = True with the transformed splits.

### Properties

```python
@property
def pipeline(self):
    # Returns the underlying sklearn Pipeline/ColumnTransformer
    # Raises RuntimeError if apply() hasn't been called yet

def get_feature_names(self) -> list:
    # Returns output column names after encoding
    # Raises RuntimeError if apply() hasn't been called yet
```

---

## FILE 5: `models.py`

### Class Behaviour

Models takes a Data object (must be split) and provides a unified interface
to train scikit-learn, XGBoost, LightGBM, and PyTorch neural networks.

### Constructor

```python
def __init__(self, data: Data):
    if not data._is_split:
        raise ValueError(
            "Data must be split before training. Call data.split() first."
        )
    self._data = data
    self.current_model = None
    self.model_name = None
    self.params = {}
    self.compare_results = None
```

### Model Registry

Maintain a dict mapping string names to model constructors.
Task type determines which variant to use.

```python
_REGISTRY = {
    "random_forest":      {"binary": RandomForestClassifier,
                           "multiclass": RandomForestClassifier,
                           "regression": RandomForestRegressor},
    "logistic_regression":{"binary": LogisticRegression,
                           "multiclass": LogisticRegression,
                           "regression": None},  # not applicable
    "linear_regression":  {"regression": LinearRegression,
                           "binary": None,
                           "multiclass": None},
    "svm":                {"binary": SVC,
                           "multiclass": SVC,
                           "regression": SVR},
    "knn":                {"binary": KNeighborsClassifier,
                           "multiclass": KNeighborsClassifier,
                           "regression": KNeighborsRegressor},
    "decision_tree":      {"binary": DecisionTreeClassifier,
                           "multiclass": DecisionTreeClassifier,
                           "regression": DecisionTreeRegressor},
    "gradient_boosting":  {"binary": GradientBoostingClassifier,
                           "multiclass": GradientBoostingClassifier,
                           "regression": GradientBoostingRegressor},
    "naive_bayes":        {"binary": GaussianNB,
                           "multiclass": GaussianNB,
                           "regression": None},
    "ridge":              {"regression": Ridge,
                           "binary": None,
                           "multiclass": None},
    "lasso":              {"regression": Lasso,
                           "binary": None,
                           "multiclass": None},
    "xgboost":            {"binary": "xgboost.XGBClassifier",
                           "multiclass": "xgboost.XGBClassifier",
                           "regression": "xgboost.XGBRegressor"},
    "lightgbm":           {"binary": "lightgbm.LGBMClassifier",
                           "multiclass": "lightgbm.LGBMClassifier",
                           "regression": "lightgbm.LGBMRegressor"},
}
```

For string values (lazy imports), parse the module and class name
and use lazy_import to load them on demand.

### `m.compare(include=None, time_limit=None) -> pd.DataFrame`

Train every model in the registry on X_train, score on X_test.
Skip models where the variant for the current task is None.
If include is provided, only train those models.
If time_limit is provided (seconds), stop after that time.

For each model, catch all exceptions and record as "ERROR" in results —
compare() should never crash even if one model fails.

Primary metric:
- binary → AUC-ROC
- multiclass → macro F1
- regression → R²

Print ranked table:
```
Pythios · Model Comparison  (task: binary, metric: AUC-ROC)
────────────────────────────────────────────────────────────────────
 Rank  Model                Score     Time(s)
────────────────────────────────────────────────────────────────────
 1     xgboost              0.891     1.2
 2     random_forest        0.883     0.8
 3     gradient_boosting    0.876     2.1
 ...
────────────────────────────────────────────────────────────────────
 Best: xgboost · run m.train("xgboost") to use it
```

Store results in self.compare_results (DataFrame).
Return the DataFrame.

### `m.train(model_name, **kwargs) -> self`

Look up model in registry for current task type.
Raise ValueError if model is None for this task (e.g. "lasso" for binary).
Raise ValueError if model_name not in registry.

For lazy-imported models (xgboost, lightgbm), parse the module path string
and load with lazy_import.

For "neural_net": see Neural Network Builder section below.

Fit on X_train, y_train.
Store in self.current_model, self.model_name, self.params = kwargs.
Print: `✓ Trained random_forest on 712 samples`

### `m.tune(model_name, trials=50) -> self`

Use Optuna (lazy import) to tune hyperparameters.
Define an objective function that:
1. Samples params using trial.suggest_* methods
2. Builds and fits model on X_train
3. Returns score on X_test

After optimization, retrain with best params using self.train().
Print best params and best score.

Default search spaces per model:
- random_forest: n_estimators (50-500), max_depth (3-20), min_samples_split (2-20)
- xgboost: n_estimators (50-500), max_depth (3-10), learning_rate (0.01-0.3), subsample (0.6-1.0)
- lightgbm: n_estimators (50-500), num_leaves (20-300), learning_rate (0.01-0.3)

Suppress Optuna logging (optuna.logging.set_verbosity(optuna.logging.WARNING)).

### Neural Network Builder

When model_name == "neural_net":

```python
def _build_and_train_nn(self, layers=[128, 64], epochs=50, lr=0.001, backend="pytorch"):
```

For PyTorch backend:
1. lazy_import("torch")
2. Convert X_train, X_test, y_train, y_test to tensors
3. Build MLP: Linear layers with ReLU activations between them
4. Final activation: Sigmoid (binary), Softmax (multiclass), none (regression)
5. Loss: BCELoss (binary), CrossEntropyLoss (multiclass), MSELoss (regression)
6. Optimizer: Adam with lr
7. Training loop: epochs iterations over batched data
8. Print progress every 10 epochs: `Epoch 10/50 — loss: 0.342`
9. Wrap trained model in a sklearn-compatible wrapper with .predict() and .predict_proba()

For TensorFlow backend:
1. lazy_import("tensorflow")
2. Build Sequential model with Dense layers
3. Compile with appropriate optimizer and loss
4. Fit with validation_split=0.1

### `m.save(path) -> None`

Use joblib.dump to save:
```python
{
    "model": self.current_model,
    "model_name": self.model_name,
    "params": self.params,
    "task_type": self._data.task_type,
    "feature_names": self._data.X_train.columns.tolist(),
    "target": self._data.target,
    "saved_at": str(datetime.now()),
}
```

### `m.load(path) -> self`

Use joblib.load. Restore model and metadata.

### `m.predict(new_data) -> np.ndarray`

Accepts Data object or pd.DataFrame.
If Data object, use ._df (drop target column if present).
Return predictions from self.current_model.predict().

### `m.predict_proba(new_data) -> np.ndarray`

Same as predict but uses .predict_proba(). Raise ValueError if model
doesn't support probability prediction.

---

## FILE 6: `evaluate.py`

### Constructor

```python
def __init__(self, model: Models):
    self._model = model
    self._data = model._data
```

### `e.metrics(on="test") -> dict`

Get predictions then compute task-appropriate metrics.

Binary metrics:
```python
from sklearn.metrics import (
    accuracy_score, f1_score, precision_score, recall_score,
    roc_auc_score, matthews_corrcoef
)
return {
    "accuracy": accuracy_score(y_true, y_pred),
    "f1": f1_score(y_true, y_pred, average="binary"),
    "precision": precision_score(y_true, y_pred),
    "recall": recall_score(y_true, y_pred),
    "auc_roc": roc_auc_score(y_true, y_pred),
    "mcc": matthews_corrcoef(y_true, y_pred),
}
```

Multiclass metrics:
```python
return {
    "accuracy": accuracy_score(y_true, y_pred),
    "macro_f1": f1_score(y_true, y_pred, average="macro"),
    "weighted_f1": f1_score(y_true, y_pred, average="weighted"),
}
```

Regression metrics:
```python
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
return {
    "mae": mean_absolute_error(y_true, y_pred),
    "rmse": np.sqrt(mean_squared_error(y_true, y_pred)),
    "r2": r2_score(y_true, y_pred),
}
```

Print formatted table of metrics. Return dict.

### `e.overfit_check()`
Call metrics(on="train") and metrics(on="test").
Print both side by side.
For each metric, if gap > 0.1 print a warning flag.

### `e.confusion_matrix(normalise=True)`
Use sklearn.metrics.confusion_matrix.
Plot with matplotlib using imshow. Add text annotations (actual counts).
Title: "Confusion Matrix — {model_name}".

### `e.roc_curve()`
Binary: plot ROC curve with AUC in legend.
Multiclass: one-vs-rest ROC curve per class.
Raise ValueError for regression tasks.

### `e.precision_recall_curve()`
Binary only. Plot precision-recall curve.

### `e.residuals()`
Regression only. Two subplots:
1. Predicted vs actual scatter (with y=x line)
2. Residual distribution histogram

### `e.shap(sample=None)`
Lazy import shap.
If sample is not None, use shap.sample(X, sample).
Auto-select explainer (see below).
Plot shap.summary_plot (beeswarm).

Explainer selection:
```python
tree_types = {RandomForestClassifier, RandomForestRegressor,
              GradientBoostingClassifier, GradientBoostingRegressor,
              DecisionTreeClassifier, DecisionTreeRegressor}
# Also check for XGBClassifier/XGBRegressor by name

if type(model) in tree_types or "XGB" in type(model).__name__ or "LGBM" in type(model).__name__:
    explainer = shap.TreeExplainer(model)
elif type(model) in {LogisticRegression, Ridge, Lasso}:
    explainer = shap.LinearExplainer(model, X)
else:
    explainer = shap.KernelExplainer(model.predict, shap.sample(X, 100))
```

### `e.shap_waterfall(index=0)`
Single prediction explanation. Use shap.plots.waterfall.

### `e.shap_dependence(feature)`
Dependence plot for one feature. Use shap.dependence_plot.

### `e.feature_importance() -> pd.DataFrame`
Try model.feature_importances_ first (RF, XGB, LGBM, tree models).
Fall back to permutation_importance(model, X_test, y_test, n_repeats=10).
Plot horizontal bar chart (top 20 features max).
Return DataFrame with Feature and Importance columns.

### `e.cross_validate(k=5, stratify=True) -> dict`
Use sklearn cross_val_score on FULL dataset (not just train split).
Use StratifiedKFold for classification, KFold for regression.
Print: mean ± std per metric.
Return dict of scores.

### `e.report(save=None)`
Print complete plain-text summary (no matplotlib):
- Model name and task type
- Test set size
- All metrics
- Top 5 features by importance (if available)
- Any warnings

If save is provided, write to that file path.

---

## FILE 7: `visualize.py`

### Constructor

```python
def __init__(self, data: Data):
    self._data = data
    import matplotlib.pyplot as plt
    import seaborn as sns
    plt.style.use("seaborn-v0_8-whitegrid")
```

All methods accept `save=None` (path to save instead of showing)
and `interactive=False` (use plotly instead of matplotlib).

### Methods

#### `v.distribution(col=None, save=None, interactive=False)`
If col is None: histograms for all numeric columns in a grid layout.
If col is specified: single histogram with KDE overlay.
Auto-size figure based on number of columns.

#### `v.correlation(save=None)`
Pearson correlation heatmap for all numeric columns.
Use seaborn.heatmap with annot=True, fmt=".2f", cmap="coolwarm".

#### `v.missing(save=None)`
Missingness matrix. White = present, black/red = missing.
Plot using matplotlib imshow on df.isnull() boolean mask.
Show percentage missing in ylabel.

#### `v.pairplot(save=None)`
seaborn pairplot. If target is set and task is classification,
colour by target using hue=self._data.target.

#### `v.class_balance(save=None)`
Bar chart of target value counts. Only for classification tasks.
Raise ValueError for regression.

#### `v.target_vs(feature, save=None)`
If regression: scatter plot of feature (x) vs target (y).
If classification: violin plot, x=target classes, y=feature values.

#### `v.target_vs_all(save=None)`
Grid of target_vs for every feature. Auto-size figure.

#### `v.feature_importance(model, save=None)`
Horizontal bar chart. Accepts a trained Models object.
Uses model.current_model.feature_importances_ if available.

#### `v.learning_curve(model, save=None)`
Train vs validation score over increasing training set sizes.
Use sklearn.model_selection.learning_curve.

#### `v.residuals(model, save=None)`
Regression only. Delegates to Evaluate(model).residuals().

#### `v.pca_explained(save=None)`
Scree plot — cumulative variance explained by PCA components.
Fit PCA on X_train (or full _df if not split).
Plot with a horizontal line at 95%.

#### `v.cluster_plot(labels, save=None)`
Scatter of first 2 PCA components, coloured by label array.
Used after Unsupervised clustering.

#### `v.summary(save=None)`
2x2 grid: distribution (numeric only) + correlation + missing + class balance.
All in one figure.

---

## FILE 8: `unsupervised.py`

### Constructor

```python
def __init__(self, data: Data):
    self._data = data
    self.labels = None
    self.embeddings = None
    # Extract feature matrix (drop target if present)
    if data.target and data.target in data._df.columns:
        self._X = data._df.drop(columns=[data.target]).select_dtypes(include="number").values
    else:
        self._X = data._df.select_dtypes(include="number").values
```

### `u.cluster(method="kmeans", **kwargs) -> self`

Methods and their kwargs:
- "kmeans": k (default 5), random_state=42, n_init=10
- "dbscan": eps (default 0.5), min_samples (default 5)
- "agglomerative": k (default 5), linkage (default "ward")
- "gaussian_mixture": k (default 5), random_state=42

Fit the chosen model on self._X.
Store predicted labels in self.labels.
Print: `✓ KMeans clustering complete — 5 clusters found`
Return self.

### `u.find_k(max_k=15) -> int`

Run KMeans for k in range(2, max_k+1).
Compute inertia and silhouette_score for each k.
Plot two subplots side by side: elbow (inertia) and silhouette.
Find best k as argmax of silhouette scores.
Print: `Suggested k = {best_k} (highest silhouette score)`
Return best_k as int.

### `u.reduce(method="pca", n=2, **kwargs) -> self`

Methods:
- "pca": sklearn.decomposition.PCA(n_components=n)
- "tsne": sklearn.manifold.TSNE(n_components=n, perplexity=kwargs.get("perplexity", 30))
- "umap": lazy_import("umap", "umap-learn"), UMAP(n_components=n, n_neighbors=kwargs.get("n_neighbors", 15))

Fit and transform self._X.
Store result in self.embeddings.
Print: `✓ Reduced to {n}D using {method} — shape: ({len(X)}, {n})`
Return self.

### `u.plot_clusters(labels=None, save=None)`

2D scatter plot. If labels is None, use self.labels.
If self.embeddings is not None and has 2 columns, use it for x/y.
Otherwise reduce to 2D with PCA first.
Colour points by label. Add legend.

### `u.silhouette_score() -> float`

Requires self.labels to be set.
Use sklearn.metrics.silhouette_score(self._X, self.labels).
Print: `Silhouette score: {score:.4f} (range: -1 to 1, higher is better)`
Return float.

### `u.davies_bouldin_score() -> float`

Use sklearn.metrics.davies_bouldin_score(self._X, self.labels).
Print: `Davies-Bouldin score: {score:.4f} (lower is better)`
Return float.

---

## FILE 9: `embeddings.py`

The most standalone class — does not require Data object (but can accept one).

### Constructor

```python
def __init__(self):
    self._text_model = None
    self._text_model_name = None
    self._image_model = None
    self._image_model_name = None
    self._index = None
    self.dim = None
```

### `emb.text(model_name="all-MiniLM-L6-v2") -> self`
Store model_name. Don't load yet (load lazily at encode time).
Return self.

### `emb.encode(texts: list) -> np.ndarray`
Lazy load sentence_transformers.SentenceTransformer if not loaded.
Call .encode(texts, convert_to_numpy=True, show_progress_bar=len(texts)>100).
Store self.dim = result.shape[1].
Return np.ndarray of shape (N, dim).

### `emb.image(model_name="resnet50") -> self`
Store model_name. Return self.

### `emb.encode_images(paths: list) -> np.ndarray`
Lazy import torch, torchvision, PIL.Image.
Standard ImageNet preprocessing:
```python
transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
```
Load model from torchvision.models, remove final classification layer.
Run inference in torch.no_grad() context.
Return np.ndarray of shape (N, dim).

### `emb.similarity(vec_a, vec_b) -> float`
Cosine similarity between two vectors.
```python
a = vec_a / (np.linalg.norm(vec_a) + 1e-9)
b = vec_b / (np.linalg.norm(vec_b) + 1e-9)
return float(np.dot(a, b))
```

### `emb.similarity_matrix(vectors) -> np.ndarray`
Pairwise cosine similarity matrix. Shape (N, N).
Normalize all vectors first, then return vectors @ vectors.T.

### `emb.search(query, corpus_vectors, top_k=5) -> list`
If query is a string, encode it first.
Compute cosine similarity with corpus_vectors.
Return list of (index, score) tuples sorted by score descending.

### `emb.build_index(vectors) -> self`
Lazy import faiss (install_name="faiss-cpu").
```python
dim = vectors.shape[1]
self._index = faiss.IndexFlatL2(dim)
self._index.add(vectors.astype("float32"))
```
Print: `✓ Faiss index built — {len(vectors)} vectors, dim={dim}`
Return self.

### `emb.search_index(query_vec, top_k=5) -> list`
Requires self._index. Raise RuntimeError if not built.
Return list of (index, distance) tuples.

### `emb.save_index(path) -> None`
Lazy import faiss.
faiss.write_index(self._index, path).

### `emb.load_index(path) -> self`
faiss.read_index(path). Return self.

---

## Tests

Write tests in the `tests/` folder. Each test file tests one class.

### `tests/test_data.py`

The following tests must all pass:

```python
import pandas as pd
import pytest
from pythios import Data


def make_sample_df():
    return pd.DataFrame({
        "age":    [25, 30, 35, 40, 45],
        "salary": [50000, 60000, 70000, 80000, 90000],
        "hired":  [1, 0, 1, 0, 1],
    })


def test_from_dataframe():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert d.shape == (5, 3)
    assert d.target == "hired"


def test_task_type_binary():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert d.task_type == "binary"


def test_task_type_regression():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="salary")
    assert d.task_type == "regression"


def test_task_type_multiclass():
    df = pd.DataFrame({
        "x": range(6),
        "y": [0, 1, 2, 0, 1, 2]
    })
    d = Data.from_dataframe(df, target="y")
    assert d.task_type == "multiclass"


def test_task_type_string_binary():
    df = pd.DataFrame({"x": [1, 2, 3], "y": ["cat", "dog", "cat"]})
    d = Data.from_dataframe(df, target="y")
    assert d.task_type == "binary"


def test_split():
    df = pd.DataFrame({"x": range(100), "y": [i % 2 for i in range(100)]})
    d = Data.from_dataframe(df, target="y")
    d.split(test=0.2, seed=42)
    assert len(d.X_train) == 80
    assert len(d.X_test) == 20
    assert d._is_split is True


def test_split_twice_raises_error():
    df = pd.DataFrame({"x": range(50), "y": [i % 2 for i in range(50)]})
    d = Data.from_dataframe(df, target="y")
    d.split(test=0.2)
    with pytest.raises(ValueError):
        d.split(test=0.2)


def test_drop_columns():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    d2 = d.drop_columns(["age"])
    assert "age" not in d2.columns
    assert "salary" in d2.columns


def test_select_columns():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    d2 = d.select_columns(["age", "hired"])
    assert d2.columns == ["age", "hired"]


def test_sample():
    df = pd.DataFrame({"x": range(100), "y": range(100)})
    d = Data.from_dataframe(df, target="y")
    d2 = d.sample(n=10)
    assert len(d2._df) == 10


def test_copy():
    df = make_sample_df()
    d1 = Data.from_dataframe(df, target="hired")
    d2 = d1.copy()
    d2._df.loc[0, "age"] = 999
    assert d1._df.loc[0, "age"] == 25


def test_shape_property():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert d.shape == (5, 3)


def test_columns_property():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert "age" in d.columns
    assert "salary" in d.columns
    assert "hired" in d.columns


def test_invalid_target_raises_error():
    df = make_sample_df()
    with pytest.raises(ValueError):
        Data.from_dataframe(df, target="nonexistent_column")


def test_from_sklearn():
    d = Data.from_sklearn("iris")
    assert d.shape[0] == 150
    assert d.target == "target"


def test_from_sklearn_invalid():
    with pytest.raises(ValueError):
        Data.from_sklearn("does_not_exist")


def test_repr():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert "Data" in repr(d)


def test_missing_report():
    df = pd.DataFrame({
        "a": [1, None, 3],
        "b": [1, 2, 3],
        "y": [0, 1, 0]
    })
    d = Data.from_dataframe(df, target="y")
    report = d.missing_report()
    assert "a" in report["Column"].values
    assert "b" not in report["Column"].values
```

### `tests/test_tools.py`

```python
import pandas as pd
import pytest
from pythios import Data, Tools


def make_data():
    df = pd.DataFrame({
        "age":    [25, 30, 35, 40, 45, 50, 55, 60],
        "salary": [50000, 60000, 70000, 80000, 90000, 100000, 110000, 120000],
        "score":  [1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8],
        "hired":  [0, 1, 0, 1, 0, 1, 0, 1],
    })
    return Data.from_dataframe(df, target="hired")


def test_profile_returns_dataframe():
    d = make_data()
    result = Tools.profile(d)
    assert isinstance(result, pd.DataFrame)
    assert "Column" in result.columns


def test_memory_usage_returns_string():
    d = make_data()
    result = Tools.memory_usage(d)
    assert isinstance(result, str)
    assert any(unit in result for unit in ["B", "KB", "MB", "GB"])


def test_detect_outliers_iqr():
    d = make_data()
    result = Tools.detect_outliers(d, method="iqr")
    assert isinstance(result, pd.DataFrame)
    assert "Outlier_Count" in result.columns


def test_detect_outliers_zscore():
    d = make_data()
    result = Tools.detect_outliers(d, method="zscore", threshold=3.0)
    assert isinstance(result, pd.DataFrame)


def test_detect_outliers_invalid_method():
    d = make_data()
    with pytest.raises(ValueError):
        Tools.detect_outliers(d, method="invalid")


def test_validate_clean_data():
    d = make_data()
    result = Tools.validate(d)
    assert result is True


def test_validate_empty_data():
    df = pd.DataFrame({"x": [], "y": []})
    d = Data(df, target="y")
    result = Tools.validate(d)
    assert result is False


def test_export_csv(tmp_path):
    d = make_data()
    path = str(tmp_path / "test.csv")
    Tools.export(d, path)
    import pandas as pd
    loaded = pd.read_csv(path)
    assert len(loaded) == 8


def test_export_json(tmp_path):
    d = make_data()
    path = str(tmp_path / "test.json")
    Tools.export(d, path)
    import json
    with open(path) as f:
        data = json.load(f)
    assert len(data) == 8


def test_export_invalid_extension(tmp_path):
    d = make_data()
    with pytest.raises(ValueError):
        Tools.export(d, str(tmp_path / "test.xyz"))


def test_export_train_split(tmp_path):
    d = make_data()
    d.split(test=0.25, seed=42)
    path = str(tmp_path / "train.csv")
    Tools.export(d, path, split="train")
    import pandas as pd
    loaded = pd.read_csv(path)
    assert len(loaded) == 6


def test_export_no_split_raises(tmp_path):
    d = make_data()
    with pytest.raises(ValueError):
        Tools.export(d, str(tmp_path / "train.csv"), split="train")


def test_timer():
    import time
    with Tools.timer("test operation"):
        time.sleep(0.01)


def test_compare_datasets():
    d1 = make_data()
    d2 = make_data()
    result = Tools.compare_datasets(d1, d2)
    assert isinstance(result, pd.DataFrame)


def test_set_seed():
    Tools.set_seed(42)
    import numpy as np
    val1 = np.random.rand()
    Tools.set_seed(42)
    val2 = np.random.rand()
    assert val1 == val2


def test_balance_oversample():
    df = pd.DataFrame({
        "x": range(15),
        "y": [0]*10 + [1]*5
    })
    d = Data.from_dataframe(df, target="y")
    balanced = Tools.balance(d, method="oversample")
    counts = balanced._df["y"].value_counts()
    assert counts[0] == counts[1]


def test_balance_undersample():
    df = pd.DataFrame({
        "x": range(15),
        "y": [0]*10 + [1]*5
    })
    d = Data.from_dataframe(df, target="y")
    balanced = Tools.balance(d, method="undersample")
    counts = balanced._df["y"].value_counts()
    assert counts[0] == counts[1]


def test_balance_no_target():
    df = pd.DataFrame({"x": range(10)})
    d = Data(df)
    with pytest.raises(ValueError):
        Tools.balance(d)


def test_suggest_runs_without_error():
    d = make_data()
    Tools.suggest(d)


def test_coder_returns_code_result():
    # This test does NOT require Ollama — it tests the CodeResult class directly
    from pythios.tools import CodeResult
    result = CodeResult(code="n = 42", description="test")
    result.run()
    assert result._ran is True
    assert "n" in result.namespace
    assert result.namespace["n"] == 42


def test_code_result_show():
    from pythios.tools import CodeResult
    result = CodeResult(code="x = 1 + 1", description="test")
    result.show()  # should not raise


def test_configure_llm():
    Tools.configure_llm(model="mistral")
    assert Tools._llm_model == "mistral"
    Tools.configure_llm(model="llama3")  # reset
```

---

## `demo/demo_all.py` — Complete Runnable Demo

This file demonstrates every feature of Pythios. It can be run top to
bottom after building the library. It uses only sklearn built-in datasets
(no external files needed) so it works immediately.

```python
"""
Pythios — Complete Demo
Run this file to test every feature of the library.
No external data files needed — uses sklearn built-in datasets.

Usage:
    python demo/demo_all.py
"""

import pandas as pd
import numpy as np
from pythios import Data, Tools


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 1: DATA LOADING
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 1: DATA LOADING")
print("="*70)

# Load from sklearn built-in datasets — no file needed
print("\n--- from_sklearn ---")
d_iris        = Data.from_sklearn("iris")
d_wine        = Data.from_sklearn("wine")
d_cancer      = Data.from_sklearn("breast_cancer")
d_digits      = Data.from_sklearn("digits")
d_diabetes    = Data.from_sklearn("diabetes")
d_california  = Data.from_sklearn("california_housing")

# Load from a pandas DataFrame
print("\n--- from_dataframe ---")
df = pd.DataFrame({
    "age":    [25, 30, 35, 40, 45, 50, 55, 60, 65, 70],
    "income": [40000, 50000, 60000, 75000, 85000, 95000, 105000, 115000, 120000, 125000],
    "score":  [1.2, 2.3, 3.4, 4.5, 5.6, 6.7, 7.8, 8.9, 9.1, 9.5],
    "hired":  [0, 0, 1, 1, 0, 1, 1, 1, 0, 1],
})
d_custom = Data.from_dataframe(df, target="hired")
print(d_custom)

# Load from URL (tests requests lazy import)
print("\n--- from_url ---")
d_url = Data.from_url(
    "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv",
    target="Survived"
)
print(d_url)


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 2: DATA INSPECTION
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 2: DATA INSPECTION")
print("="*70)

print("\n--- describe() ---")
d_url.describe()

print("\n--- missing_report() ---")
print(d_url.missing_report())

print("\n--- task types ---")
print(f"iris task type:         {d_iris.task_type}")         # multiclass
print(f"breast_cancer task:     {d_cancer.task_type}")       # binary
print(f"diabetes task:          {d_diabetes.task_type}")     # regression
print(f"california task:        {d_california.task_type}")   # regression
print(f"custom hired task:      {d_custom.task_type}")       # binary

print("\n--- shape, columns, dtypes ---")
print(f"Shape: {d_iris.shape}")
print(f"Columns: {d_iris.columns}")
print(f"Dtypes:\n{d_iris.dtypes}")

print("\n--- head ---")
print(d_iris.head(3))


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 3: DATA MANIPULATION
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 3: DATA MANIPULATION")
print("="*70)

print("\n--- split ---")
d_cancer_work = d_cancer.copy()
d_cancer_work.split(test=0.2, seed=42)
print(f"X_train shape: {d_cancer_work.X_train.shape}")
print(f"X_test shape:  {d_cancer_work.X_test.shape}")

print("\n--- drop_columns ---")
d2 = d_url.drop_columns(["Name", "Ticket", "Cabin"])
print(f"Columns after drop: {d2.columns}")

print("\n--- select_columns ---")
d3 = d_url.select_columns(["Age", "Fare", "Pclass", "Survived"])
print(f"Columns after select: {d3.columns}")

print("\n--- sample ---")
d4 = d_url.sample(n=100)
print(f"Sampled: {d4.shape[0]} rows")

print("\n--- copy ---")
d_original = d_custom.copy()
d_modified = d_original.copy()
d_modified._df.loc[0, "age"] = 999
print(f"Original age[0]: {d_original._df.loc[0, 'age']}")   # still 25
print(f"Modified age[0]: {d_modified._df.loc[0, 'age']}")   # 999


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 4: TOOLS UTILITIES
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 4: TOOLS")
print("="*70)

print("\n--- profile ---")
Tools.profile(d_custom)

print("\n--- suggest ---")
Tools.suggest(d_url)

print("\n--- detect_outliers (IQR) ---")
report = Tools.detect_outliers(d_url, method="iqr")
print(report.head())

print("\n--- detect_outliers (z-score) ---")
report2 = Tools.detect_outliers(d_url, method="zscore", threshold=3.0)
print(report2.head())

print("\n--- validate ---")
is_valid = Tools.validate(d_url)
print(f"Dataset valid: {is_valid}")

print("\n--- memory_usage ---")
print(f"Memory: {Tools.memory_usage(d_url)}")

print("\n--- compare_datasets ---")
d_small = d_url.sample(100)
Tools.compare_datasets(d_url, d_small)

print("\n--- set_seed ---")
Tools.set_seed(42)

print("\n--- timer ---")
import time
with Tools.timer("demo sleep"):
    time.sleep(0.05)

print("\n--- balance (oversample) ---")
d_url_split = d_url.copy()
balanced = Tools.balance(d_url_split, method="oversample", seed=42)
print(f"Balanced target counts:\n{balanced._df['Survived'].value_counts()}")

print("\n--- balance (undersample) ---")
undersampled = Tools.balance(d_url_split, method="undersample", seed=42)
print(f"Undersampled target counts:\n{undersampled._df['Survived'].value_counts()}")

print("\n--- export (CSV) ---")
Tools.export(d_custom, "/tmp/pythios_demo_export.csv")

print("\n--- export (JSON) ---")
Tools.export(d_custom, "/tmp/pythios_demo_export.json")

print("\n--- export train split ---")
d_cancer_work_export = d_cancer.copy()
d_cancer_work_export.split(test=0.2)
Tools.export(d_cancer_work_export, "/tmp/pythios_train.csv", split="train")
Tools.export(d_cancer_work_export, "/tmp/pythios_test.csv", split="test")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 5: PREPROCESSING
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 5: PREPROCESS")
print("="*70)

try:
    from pythios import Preprocess

    d_titanic = d_url.drop_columns(["Name", "Ticket", "Cabin", "PassengerId"])
    d_titanic.split(test=0.2, seed=42)

    print("\n--- auto() ---")
    p = Preprocess(d_titanic)
    p.auto()
    clean = p.apply()
    print(f"Before: {d_titanic.X_train.shape}")
    print(f"After:  {clean.X_train.shape}")

    print("\n--- manual pipeline ---")
    d_titanic2 = d_url.drop_columns(["Name", "Ticket", "Cabin", "PassengerId"])
    d_titanic2.split(test=0.2, seed=42)
    p2 = Preprocess(d_titanic2)
    p2.impute("median")
    p2.scale("robust")
    p2.encode("onehot")
    clean2 = p2.apply()
    print(f"Manual pipeline result shape: {clean2.X_train.shape}")

    Tools.compare_datasets(d_titanic, clean)

except ImportError:
    print("Preprocess not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 6: MODELS
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 6: MODELS")
print("="*70)

try:
    from pythios import Models

    d_cancer_m = d_cancer.copy()
    d_cancer_m.split(test=0.2, seed=42)
    m = Models(d_cancer_m)

    print("\n--- compare() ---")
    results = m.compare()
    print(results.head())

    print("\n--- train random_forest ---")
    m.train("random_forest", n_estimators=100)

    print("\n--- train logistic_regression ---")
    m2 = Models(d_cancer_m)
    m2.train("logistic_regression")

    print("\n--- train xgboost ---")
    try:
        m3 = Models(d_cancer_m)
        m3.train("xgboost")
    except ImportError as e:
        print(f"XGBoost not installed: {e}")

    print("\n--- predict ---")
    preds = m.predict(d_cancer_m)
    print(f"Predictions shape: {preds.shape}")
    print(f"First 5: {preds[:5]}")

    print("\n--- save and load ---")
    m.save("/tmp/pythios_rf_model.pkl")
    m_loaded = Models(d_cancer_m)
    m_loaded.load("/tmp/pythios_rf_model.pkl")
    print(f"Loaded model name: {m_loaded.model_name}")

    print("\n--- tune (quick, 10 trials) ---")
    m_tune = Models(d_cancer_m)
    m_tune.tune("random_forest", trials=10)

    # Regression task
    print("\n--- regression model ---")
    d_diab = d_diabetes.copy()
    d_diab.split(test=0.2, seed=42)
    m_reg = Models(d_diab)
    m_reg.train("random_forest")
    preds_reg = m_reg.predict(d_diab)
    print(f"Regression predictions sample: {preds_reg[:3]}")

except ImportError:
    print("Models not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 7: EVALUATE
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 7: EVALUATE")
print("="*70)

try:
    from pythios import Evaluate, Models

    d_cancer_e = d_cancer.copy()
    d_cancer_e.split(test=0.2, seed=42)
    m_eval = Models(d_cancer_e)
    m_eval.train("random_forest")
    e = Evaluate(m_eval)

    print("\n--- metrics (test) ---")
    metrics = e.metrics(on="test")
    print(metrics)

    print("\n--- metrics (train) ---")
    metrics_train = e.metrics(on="train")
    print(metrics_train)

    print("\n--- overfit_check ---")
    e.overfit_check()

    print("\n--- cross_validate ---")
    cv_results = e.cross_validate(k=5)
    print(cv_results)

    print("\n--- feature_importance ---")
    fi = e.feature_importance()
    print(fi.head())

    print("\n--- report ---")
    e.report()

    print("\n--- report (save) ---")
    e.report(save="/tmp/pythios_eval_report.txt")
    print("Report saved to /tmp/pythios_eval_report.txt")

    print("\n--- confusion_matrix ---")
    e.confusion_matrix()

    print("\n--- roc_curve ---")
    e.roc_curve()

    print("\n--- shap ---")
    try:
        e.shap(sample=50)
    except ImportError:
        print("shap not installed — skipping")

    # Regression evaluate
    print("\n--- regression evaluate ---")
    d_diab_e = d_diabetes.copy()
    d_diab_e.split(test=0.2, seed=42)
    m_diab = Models(d_diab_e)
    m_diab.train("random_forest")
    e_reg = Evaluate(m_diab)
    e_reg.metrics()
    e_reg.residuals()

except ImportError:
    print("Evaluate not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 8: VISUALIZE
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 8: VISUALIZE")
print("="*70)

try:
    from pythios import Visualize

    d_vis = d_url.copy()

    print("\n--- distribution (all columns) ---")
    v = Visualize(d_vis)
    v.distribution(save="/tmp/pythios_distribution.png")

    print("\n--- distribution (single column) ---")
    v.distribution("Age", save="/tmp/pythios_dist_age.png")

    print("\n--- correlation ---")
    v.correlation(save="/tmp/pythios_correlation.png")

    print("\n--- missing ---")
    v.missing(save="/tmp/pythios_missing.png")

    print("\n--- class_balance ---")
    v.class_balance(save="/tmp/pythios_balance.png")

    print("\n--- target_vs ---")
    v.target_vs("Age", save="/tmp/pythios_target_age.png")
    v.target_vs("Fare", save="/tmp/pythios_target_fare.png")

    print("\n--- pca_explained ---")
    d_vis_num = d_url.drop_columns(["Name", "Ticket", "Cabin"])
    d_vis_num.split(test=0.2)
    v2 = Visualize(d_vis_num)
    v2.pca_explained(save="/tmp/pythios_pca.png")

    print("\n--- summary ---")
    v.summary(save="/tmp/pythios_summary.png")

    print("All plots saved to /tmp/")

except ImportError:
    print("Visualize not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 9: UNSUPERVISED
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 9: UNSUPERVISED")
print("="*70)

try:
    from pythios import Unsupervised

    d_clust = d_iris.copy()

    print("\n--- kmeans clustering ---")
    u = Unsupervised(d_clust)
    u.cluster("kmeans", k=3)
    print(f"Labels: {u.labels[:10]}")

    print("\n--- find_k ---")
    best_k = u.find_k(max_k=10)
    print(f"Best k: {best_k}")

    print("\n--- dbscan ---")
    u2 = Unsupervised(d_clust)
    u2.cluster("dbscan", eps=1.0, min_samples=5)
    print(f"DBSCAN labels (unique): {np.unique(u2.labels)}")

    print("\n--- PCA reduction ---")
    u3 = Unsupervised(d_clust)
    u3.reduce("pca", n=2)
    print(f"Embeddings shape: {u3.embeddings.shape}")

    print("\n--- t-SNE reduction ---")
    u4 = Unsupervised(d_clust)
    u4.reduce("tsne", n=2)
    print(f"t-SNE embeddings shape: {u4.embeddings.shape}")

    print("\n--- UMAP reduction ---")
    try:
        u5 = Unsupervised(d_clust)
        u5.reduce("umap", n=2)
        print(f"UMAP embeddings shape: {u5.embeddings.shape}")
    except ImportError:
        print("umap-learn not installed — skipping UMAP")

    print("\n--- cluster_plot ---")
    u6 = Unsupervised(d_clust)
    u6.reduce("pca", n=2)
    u6.cluster("kmeans", k=3)
    u6.plot_clusters(save="/tmp/pythios_clusters.png")

    print("\n--- silhouette_score ---")
    u6.silhouette_score()

    print("\n--- davies_bouldin_score ---")
    u6.davies_bouldin_score()

except ImportError:
    print("Unsupervised not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 10: EMBEDDINGS
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 10: EMBEDDINGS")
print("="*70)

try:
    from pythios import Embeddings

    emb = Embeddings()

    print("\n--- text embeddings ---")
    emb.text("all-MiniLM-L6-v2")
    texts = [
        "Machine learning is fascinating",
        "Deep learning uses neural networks",
        "Python is great for data science",
        "I love pizza and pasta",
        "The weather is sunny today",
    ]
    vectors = emb.encode(texts)
    print(f"Text vectors shape: {vectors.shape}")
    print(f"Embedding dim: {emb.dim}")

    print("\n--- similarity ---")
    score = emb.similarity(vectors[0], vectors[1])
    print(f"Similarity (ML vs DL): {score:.4f}")

    score2 = emb.similarity(vectors[0], vectors[3])
    print(f"Similarity (ML vs pizza): {score2:.4f}")

    print("\n--- similarity_matrix ---")
    matrix = emb.similarity_matrix(vectors)
    print(f"Similarity matrix shape: {matrix.shape}")

    print("\n--- search ---")
    query = "neural networks and AI"
    results = emb.search(query, vectors, top_k=3)
    print(f"Search results for '{query}':")
    for idx, score in results:
        print(f"  [{idx}] score={score:.4f} — {texts[idx]}")

    print("\n--- Faiss index ---")
    try:
        emb.build_index(vectors)
        query_vec = emb.encode(["artificial intelligence"])[0]
        index_results = emb.search_index(query_vec, top_k=3)
        print(f"Faiss search results: {index_results}")

        emb.save_index("/tmp/pythios_demo.faiss")
        emb2 = Embeddings()
        emb2.load_index("/tmp/pythios_demo.faiss")
        print("Faiss index saved and loaded successfully")

    except ImportError:
        print("faiss-cpu not installed — skipping Faiss index demo")

    print("\n--- image embeddings ---")
    print("(Skipping image embeddings — no test images available)")
    print("Usage:")
    print("  emb.image('resnet50')")
    print("  vecs = emb.encode_images(['cat.jpg', 'dog.jpg'])")

except ImportError:
    print("Embeddings not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 11: TOOLS LLM (requires Ollama running locally)
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 11: TOOLS LLM")
print("="*70)
print("(LLM features require Ollama running at http://localhost:11434)")
print("Install: https://ollama.com | Then: ollama pull llama3")
print()

# Test CodeResult directly (does NOT require Ollama)
print("--- CodeResult (no LLM needed) ---")
from pythios.tools import CodeResult

result = CodeResult(code="total = sum(range(1, 101))", description="sum 1 to 100")
result.show()
result.run()
print(f"total = {result.namespace['total']}")   # 5050

result2 = CodeResult(
    code="fibs = [0, 1]\nfor i in range(8):\n    fibs.append(fibs[-1] + fibs[-2])",
    description="fibonacci"
)
result2.show()
result2.run()
print(f"fibs = {result2.namespace['fibs']}")

print("\n--- LLM features (skip if Ollama not running) ---")
SKIP_LLM = True   # Set to False if Ollama is running

if not SKIP_LLM:
    print("\n--- Tools.llm ---")
    Tools.llm("In one sentence, what is a random forest?")

    print("\n--- Tools.coder ---")
    coder = Tools.coder("compute the factorial of 10 and store it in result")
    coder.run()
    print(f"factorial result = {coder.namespace.get('result')}")

    print("\n--- Tools.explain ---")
    Tools.explain("result = [x**2 for x in range(10) if x % 2 == 0]")

    print("\n--- Tools.review ---")
    Tools.review("def divide(a, b):\n    return a / b")

    print("\n--- Tools.fix ---")
    fixed = Tools.fix("print(10 / 0)", error="ZeroDivisionError: division by zero")
    fixed.run()

    print("\n--- Tools.ask_about_data ---")
    d_ask = Data.from_sklearn("breast_cancer")
    Tools.ask_about_data(d_ask, "Which features are most likely to be predictive?")

    print("\n--- configure_llm ---")
    Tools.configure_llm(model="codellama")
    Tools.configure_llm(model="llama3")   # reset to default


# ─────────────────────────────────────────────────────────────────────────────
# DONE
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" DEMO COMPLETE")
print("="*70)
print("""
All sections that are not 'not built yet' ran successfully.

Next steps:
  1. Run the test suite:  pytest tests/ -v
  2. Try the LLM tools:   set SKIP_LLM = False and start Ollama
  3. pip install pythios[full] to unlock all optional features
""")
```

---

## Key Design Rules — Never Violate These

1. **Never import torch, tensorflow, shap, umap, faiss, xgboost, lightgbm,
   sentence_transformers, or Pillow at the top of any file.**
   Always use `lazy_import()` inside the method that needs them.

2. **Never mutate a Data object in place.**
   Methods that transform data return a new Data object.

3. **`apply()` in Preprocess always fits on X_train only.**
   Never fit on X_test or the full dataset. This is a data leakage violation.

4. **Data.split() raises ValueError if called twice.**
   Not a warning — a hard error. Prevents accidental re-splitting.

5. **Task type auto-detection is immutable.**
   Once set in `__init__`, task_type never changes. It propagates downstream.

6. **compare() in Models never crashes.**
   Wrap each model in try/except and record errors. Users should always
   get a result table even if some models fail.

7. **Tools is never instantiated.**
   All methods are @staticmethod or @classmethod.

8. **report() in Evaluate is plain text only.**
   No matplotlib. Visual methods are separate.

9. **The target column is never transformed by Preprocess.**
   It is separated before transformation and reattached to the output Data object.

10. **Column names are always preserved.**
    After OneHotEncoding, use get_feature_names_out() and restore as DataFrame.
    Never work with raw numpy arrays if column names can be preserved.

---

## Running the Full Test Suite

```bash
# Run all tests
pytest tests/ -v

# Run specific class tests
pytest tests/test_data.py -v
pytest tests/test_tools.py -v
pytest tests/test_models.py -v

# Run demo
python demo/demo_all.py

# Run with coverage
pip install pytest-cov
pytest tests/ --cov=pythios --cov-report=term-missing
```

Expected results:
- All tests in test_data.py pass (14 tests)
- All tests in test_tools.py pass (18 tests)
- demo/demo_all.py runs without errors for all built sections
- Sections for unbuilt classes print "not built yet — skipping" gracefully

---

## LLM Setup (for Tools.coder and Tools.llm)

Pythios uses Ollama for local LLM inference. No API key needed.
All inference is local — no data leaves your machine.

```bash
# Install Ollama
# Visit https://ollama.com and download for your platform

# Pull a model (pick one)
ollama pull llama3          # general purpose, recommended default
ollama pull codellama       # better for code generation
ollama pull mistral         # fast and capable
ollama pull phi3            # lightweight, runs on low-RAM machines

# Start the server
ollama serve

# In Python
from pythios import Tools
Tools.configure_llm(model="codellama")   # switch models anytime
Tools.llm("Explain the bias-variance tradeoff")
```

---

## Version History

- 0.1.0 — Initial release. Data, Tools, Preprocess, Models, Evaluate,
           Visualize, Unsupervised, Embeddings all implemented.
