Metadata-Version: 2.4
Name: varframe
Version: 1.3.0
Summary: Declarative DataFrame variable management with automatic DAG dependency resolution and ML model integration
Author-email: Santiago Romagosa <romagosasantiago@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Santiago Romagosa
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/Santi-49/varframe
Project-URL: Repository, https://github.com/Santi-49/varframe
Keywords: pandas,dataframe,variables,dag,machine-learning,feature-engineering,data-pipeline,workflow,data-science,etl
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=1.3.0
Provides-Extra: ml
Requires-Dist: scikit-learn>=1.0.0; extra == "ml"
Requires-Dist: joblib>=1.0.0; extra == "ml"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5.0; extra == "docs"
Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
Provides-Extra: all
Requires-Dist: varframe[dev,docs,ml]; extra == "all"
Dynamic: license-file

# VarFrame

[![PyPI version](https://badge.fury.io/py/varframe.svg)](https://badge.fury.io/py/varframe)
[![Python Versions](https://img.shields.io/pypi/pyversions/varframe.svg)](https://pypi.org/project/varframe/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Declarative DataFrame variable management with automatic DAG dependency resolution and ML model integration.**

---

## What is VarFrame?

VarFrame is a library that allows you to define DataFrame columns as **Python classes** rather than imperative scripts. It manages dependencies, types, and execution order automatically using a generic DAG (Directed Acyclic Graph) solver.

It is designed for **complex, production-grade data pipelines** where traceability, correctness, and structure are more important than raw implementation speed.

```mermaid
graph TD
    Raw[Raw DataFrame] -->|extracts| Base[BaseVariable]
    Base -->|inputs| Derived[DerivedVariable]
    Base -->|features| Model[ML Model]
    Derived -->|features| Model
    Model -->|predicts| Pred[ModelVariable]
    Pred -->|inputs| Ensemble[Ensemble Model]
    Ensemble -->|predicts| Final[Final Prediction]

    style Raw fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    style Base fill:#fff9c4,stroke:#fbc02d,stroke-width:2px
    style Derived fill:#e0f2f1,stroke:#00695c,stroke-width:2px
    style Model fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style Pred fill:#fce4ec,stroke:#c2185b,stroke-width:2px
    style Ensemble fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
```

## Why VarFrame?

### The Problem with Traditional Scripts
In traditional pandas scripts (`df['b'] = df['a'] + 1`), logic is often:
- **Fragile**: Reordering cells or lines breaks dependencies silently.
- **Opaque**: It's hard to tell *exactly* which columns are needed effectively.
- **Hard to Test**: You have to test "intermediate states" of a large dataframe.

### The VarFrame Solution
VarFrame treats variables as **definitions** (Classes) rather than **steps**.

| Feature | VarFrame | Traditional Script |
| :--- | :--- | :--- |
| **Dependency Resolution** | **Automatic (DAG)**. Order doesn't matter; the framework solves it. | **Manual**. You must order operations correctly yourself. |
| **Logic Encapsulation** | Logic, metadata, and types live in one Class. Self-documenting. | Distributed across scripts. logic often mixed with execution. |
| **ML Integration** | Models are just "Computed Variables". Predictions are treated like any other column. | Often separate "training" and "inference" pipelines. |
| **Testing** | Unit test single `calculate(df)` methods in isolation. | Integration testing entire scripts is required. |

### Best For
- **Feature Stores**: Reuse definitions across training and serving.
- **Complex DAGs**: When variable F depends on E, which depends on D, C, and B...
- **Ensemble/Stacking**: Where model predictions feed into other models (see `examples/ensemble_demo.py`).

## Installation

```bash
pip install varframe           # Core only (pandas)
pip install varframe[ml]       # + scikit-learn, joblib
pip install varframe[all]      # Everything
```

## Quick Start

### 1. Define Variables

```python
from varframe import BaseVariable, DerivedVariable, VarFrame

# Map a raw column with type enforcement
class Lap(BaseVariable):
    """Current lap number."""
    name = "lap"
    raw_column = "lap_num"
    dtype = "int"

class Gap(BaseVariable):
    """Gap to leader in seconds."""
    name = "gap"
    raw_column = "gap_to_leader"
    dtype = "float"

# Create a computed column with dependencies
class GapDelta(DerivedVariable):
    """Change in gap from previous row."""
    name = "gap_delta"
    dependencies = [Gap]
    
    @classmethod
    def calculate(cls, df):
        return df["gap"] - df["gap"].shift(1)
```

### 2. Create a VarFrame

```python
import pandas as pd

# Raw data with original column names
df_raw = pd.DataFrame({
    "lap_num": [1, 2, 3],
    "gap_to_leader": [0.0, 1.2, 0.8]
})

# Create VarFrame - columns are computed automatically
# Dependencies are resolvd automatically!
vf = VarFrame(df_raw, [Lap, Gap, GapDelta])

print(vf)
#    lap  gap  gap_delta
# 0    1  0.0        NaN
# 1    2  1.2        1.2
# 2    3  0.8       -0.4
```

### 3. Access Variables

```python
# By name
vf["gap"]

# By class
vf[Gap]

# Multiple variables
vf[[Lap, Gap]]

# Filter by type
vf.filter_by_type(DerivedVariable)  # Only computed columns
```

## ML Model Integration

Define models declaratively and use predictions as variables:

```python
from varframe import BaseModel, ModelVariable
from sklearn.ensemble import RandomForestRegressor

class GapPredictor(BaseModel):
    """Predicts future gap based on features."""
    name = "gap_predictor"
    inputs = [Lap, Gap]
    target = GapDelta
    model_class = RandomForestRegressor
    hyperparameters = {"n_estimators": 100, "max_depth": 5}

# Train the model
GapPredictor.train(training_vf)

# Use predictions as a variable
class PredictedGapDelta(ModelVariable):
    name = "predicted_gap_delta"
    model_class = GapPredictor

vf.add_variables(PredictedGapDelta)
```

## Optimization & Export

### Lazy Loading
Optimize memory by marking variables as `lazy = True`. They are computed on-demand and not stored in the DataFrame.

```python
class HugeFeature(DerivedVariable):
    lazy = True
    dependencies = [RawData]

    @classmethod
    def calculate(cls, df):
        return df["raw"] * 1000
```

### Flexible Views
Export specific subsets of data using `vf.view()`:

```python
# Export only base variables
df_base = vf.view(include=["base"])

# Export specific variables (computes lazy vars on demand)
df_custom = vf.view(variables=[HugeFeature])
```

### Persistence (Import/Export)
VarFrame provides smart I/O methods that handle variable metadata automatically.

#### Export
Enhanced `to_csv` and `to_parquet` methods:
- **Safety**: Warns about uncomputed lazy variables.
- **On-the-fly**: Use `include` or `variables` to compute during export.
- **Defaults**: Autosaves to `{vf.name}.csv` if no path provided.
- **Metadata**: `to_parquet` embeds variable names in file metadata.

```python
# Export everything (computing lazy vars) to "my_data.csv"
vf.to_csv("my_data.csv", include=['all'])
```

#### Import (Auto-Discovery)
Load data without ensuring variables are manually passed. `VarFrame` scans your environment for matching `BaseVariable` and `DerivedVariable` definitions.

```python
# Matches columns to your Python classes automatically!
vf_loaded = VarFrame.load_csv("my_data.csv")

# Parquet is even safer (uses file metadata if available)
vf_pq = VarFrame.load_parquet("my_data.parquet")
```

## API Reference

### Variable Classes

| Class | Purpose |
|-------|---------|
| `BaseVariable` | Maps a raw column (with optional dtype conversion) |
| `DerivedVariable` | Computed from other variables. Set `lazy=True` for on-demand computation. |
| `ModelVariable` | Predictions from an ML model |

### VarFrame Methods

| Method | Description |
|--------|-------------|
| `add_variables(*vars, compute=True)` | Compute and add new variables (or register if compute=False) |
| `add_variable(*vars)` | Alias for `add_variables(*vars)` |
| `filter_by_type(type)` | Filter to `BaseVariable` or `DerivedVariable` only |
| `get_variable(name)` | Get variable class by name |
| `view(include=..., variables=...)` | Export DataFrame with specific variables (handles lazy computation) |
| `list_variables()` | List all variable names |
| `describe_variables()` | Summary DataFrame of all variables |
| `to_csv(...)` / `to_parquet(...)` | Enhanced export with lazy computation & metadata |
| `load_csv(path)` / `load_parquet(path)` | Class methods to load VarFrame with auto-discovery |
| `to_pandas()` / `to_ml()` | Convert to plain DataFrame for ML pipelines |

### BaseModel Methods

| Method | Description |
|--------|-------------|
| `train(vf)` | Train on a VarFrame |
| `predict(vf)` | Generate predictions |
| `evaluate(vf)` | Compute metrics |
| `save(path)` / `load(path)` | Persist and restore model |

## License

MIT
