Metadata-Version: 2.4
Name: hybrid-frame
Version: 0.3.1
Summary: High-performance DataFrame unifying DuckDB (out-of-core) and Pandas (ML feature engineering)
Author: mukesh
License: MIT
Project-URL: Homepage, https://github.com/shell-bay/hybrid-frame
Project-URL: Repository, https://github.com/shell-bay/hybrid-frame
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Database :: Database Engines/Servers
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: duckdb>=0.8.0
Requires-Dist: pandas>=1.5.0
Provides-Extra: arrow
Requires-Dist: pyarrow>=6.0.0; extra == "arrow"
Provides-Extra: memory-guard
Requires-Dist: psutil>=5.9.0; extra == "memory-guard"
Provides-Extra: ml
Requires-Dist: scikit-learn>=1.2.0; extra == "ml"
Provides-Extra: all
Requires-Dist: pyarrow>=6.0.0; extra == "all"
Requires-Dist: psutil>=5.9.0; extra == "all"
Requires-Dist: scikit-learn>=1.2.0; extra == "all"

# HybridFrame

A high-performance dual-engine DataFrame that transparently shifts between DuckDB (lazy/out-of-core) and Pandas (materialised/in-memory).

## Installation

```bash
pip install hybrid-frame
```

With optional dependencies:

```bash
pip install hybrid-frame[arrow]       # zero-copy Arrow materialisation
pip install hybrid-frame[memory-guard] # psutil-based OOM protection
pip install hybrid-frame[ml]           # scikit-learn for ML export
pip install hybrid-frame[all]          # everything
```

## Quick Start

```python
from hybrid_frame import HybridFrame
import pandas as pd

# Wrap an existing DataFrame
hf = HybridFrame.from_pandas(pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]}))

# Lazy DuckDB operations — zero Pandas copies
hf.filter("x > 1").sort_values("y", ascending=False)

# Chain into Pandas for ML feature engineering
result = hf.assign(x2=lambda df: df["x"] * 2).to_pandas()

# Stream a CSV lazily from disk
hf = HybridFrame.from_csv("large_file.csv")
print(hf.filter("age > 21").shape)
```

## Features

- **Zero-copy engine transitions** — stays in Pandas mode for simple operations (head/tail/select/rename/drop), auto-transitions to DuckDB for complex SQL operations (filter/sort/groupby/join)
- **Lazy DuckDB connections** — no connection overhead until DuckDB is actually needed
- **Out-of-core streaming** — `fetch_chunked()`, `to_pandas_iter()`, `to_arrow_reader()` for memory-efficient processing
- **ML-ready export** — `to_ml_ready(target_column)` returns `(X, y)` tuple
- **File I/O** — `from_csv()`, `from_parquet()`, `write_csv()`, `write_parquet()`
- **Set operations** — `union`, `intersect`, `except_`
- **Cumulative operations** — `diff`, `cumsum`, `cumprod`, `cummin`, `cummax`
- **Value imputation** — `fillna`, `dropna`, `replace`, `clip`, `time_series_impute`
- **DuckDB SQL passthrough** — `sql("SELECT ... FROM self WHERE ...")`
- **Memory guard** — automatic OOM protection for large materialisations

## Documentation

Full API reference is available in the class docstrings. Run `help(HybridFrame)` or visit [GitHub](https://github.com/shell-bay/hybrid-frame).
