Metadata-Version: 2.4
Name: tensorframe
Version: 0.1.0
Summary: Arrow-backed DataFrames with native tensor column support for PyTorch/JAX
Author-email: "Md. Shahinur Hasan" <shahinhasanronys@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Shahinurhasans/TensorFrame
Project-URL: Documentation, https://github.com/Shahinurhasans/TensorFrame#readme
Project-URL: Repository, https://github.com/Shahinurhasans/TensorFrame
Project-URL: Issues, https://github.com/Shahinurhasans/TensorFrame/issues
Keywords: dataframe,tensor,pytorch,arrow,machine-learning,deep-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: pyarrow>=12.0
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == "torch"
Provides-Extra: jax
Requires-Dist: jax>=0.4; extra == "jax"
Requires-Dist: jaxlib>=0.4; extra == "jax"
Provides-Extra: io
Requires-Dist: pyarrow>=12.0; extra == "io"
Requires-Dist: nibabel>=4.0; extra == "io"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: torch>=2.0; extra == "dev"
Provides-Extra: all
Requires-Dist: tensorframe[dev,io,jax,torch]; extra == "all"

# TensorFrame

**Arrow-backed DataFrames with native tensor column support for PyTorch/JAX.**

TensorFrame bridges the gap between tabular data processing (Pandas/Polars) and deep learning tensor execution (PyTorch/JAX) — without falling back to slow Python object arrays.

## Install

```bash
pip install tensorframe
```

## Quick Start

```python
import tensorframe as tf
import numpy as np
import pandas as pd

# Create a DataFrame with a 1024-D embedding column
embeddings = np.random.randn(1000, 1024).astype("float32")
df = pd.DataFrame({
    "patient_id": range(1000),
    "age":        np.random.randint(20, 80, 1000).astype(float),
    "features":   pd.Series(tf.TensorArray(embeddings)),
})

# L2-normalise directly on the column
df["norm"] = df["features"].tensor.l2_normalize()

# Filter rows by cosine similarity to a query vector
target = np.random.randn(1024).astype("float32")
scores  = df["norm"].tensor.dot(target)
filtered = df[scores > 0.5]

# Zero-friction PyTorch DataLoader
loader = tf.to_pytorch_dataloader(
    filtered,
    features=["norm", "age"],
    targets="patient_id",
    batch_size=64,
    shuffle=True,
)
for batch in loader:
    x = batch["norm"]      # torch.Tensor (64, 1024)
    y = batch["target"]    # torch.Tensor (64,)
```

## Read Parquet (auto-detects tensor columns)

```python
df = tf.read_parquet("multimodal_biomarkers.parquet")
# Any FixedSizeList columns are automatically wrapped as TensorArray
```

## Key Features

| Feature | Details |
|---|---|
| Arrow-backed storage | `FixedSizeList` — one contiguous C memory block per column |
| Zero-copy to numpy | `np.frombuffer` wraps the Arrow buffer — no allocation |
| Efficient PyTorch bridge | Single `memcpy` from Arrow → writeable numpy → `torch.from_numpy` |
| `.tensor` accessor | `l2_normalize`, `dot`, `cosine_similarity`, `mean`, `norm`, `reshape` |
| Parquet round-trip | Tensor shape metadata preserved through write/read cycles |
| NIfTI support | `tf.read_nifti_batch(paths)` for volumetric medical imaging |
| Out-of-core iteration | `tf.iter_chunks`, `ChunkedDataLoader` for large datasets |

## Requirements

- Python ≥ 3.9
- `numpy`, `pandas`, `pyarrow`
- Optional: `torch` (for GPU ops and DataLoader), `nibabel` (NIfTI I/O)

## Author

**Md. Shahinur Hasan** — [shahinhasanronys@gmail.com](mailto:shahinhasanronys@gmail.com)

GitHub: [https://github.com/Shahinurhasans/TensorFrame](https://github.com/Shahinurhasans/TensorFrame)

## License

Apache 2.0
