Metadata-Version: 2.4
Name: mxframe
Version: 0.2.1
Summary: GPU-accelerated DataFrame engine: Python ergonomics + Mojo AOT kernels, cross-vendor (NVIDIA/AMD/Apple).
Author-email: Abhishek Sreesaila <abhisheksreesaila@users.noreply.github.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/abhisheksreesaila/mxframe
Project-URL: Repository, https://github.com/abhisheksreesaila/mxframe
Keywords: dataframe,gpu,mojo,arrow,analytics,tpch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Database
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: pyarrow>=14.0
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Provides-Extra: runtime
Requires-Dist: modular>=25.5; extra == "runtime"
Provides-Extra: polars
Requires-Dist: polars>=0.20; extra == "polars"
Provides-Extra: sql
Requires-Dist: sqlglot>=25.0; extra == "sql"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: polars>=0.20; extra == "dev"
Requires-Dist: sqlglot>=25.0; extra == "dev"
Requires-Dist: duckdb>=0.9; extra == "dev"

# 🚀 MXFrame

> **GPU-accelerated DataFrames — Python ergonomics, Mojo speed, every GPU.**

MXFrame is a DataFrame query engine that pairs a Polars-style Python API with
pre-compiled Mojo AOT kernels. The same code runs on **NVIDIA, AMD, and Apple Silicon** —
no CUDA required, no JIT compilation at query time.

[![TPC-H](https://img.shields.io/badge/TPC--H-22%2F22%20queries-brightgreen)](docs/benchmarks.md)
[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](pyproject.toml)
[![Platform](https://img.shields.io/badge/platform-linux--x86__64-lightgrey)](pyproject.toml)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
[![Visualizer](https://img.shields.io/badge/interactive-query%20visualizer-blueviolet)](visualize/mxframe_pipeline.html)

---

## ✨ Why MXFrame?

| | pandas | Polars | cuDF (Rapids) | **MXFrame** |
|---|---|---|---|---|
| GPU support | ❌ | ❌ | ✅ NVIDIA only | ✅ **Any GPU** |
| Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
| Install complexity | pip | pip | CUDA + Rapids stack | **pixi install** |
| TPC-H competitive | ❌ | ✅ | ✅ | ✅ |
| Cross-vendor | ❌ | ❌ | ❌ | ✅ NVIDIA/AMD/Apple |

MXFrame is the **cuDF architecture without the CUDA lock-in**.  
The kernels are compiled **once** to a `.so` at build time — loaded in ~1 ms at process start,
then pure dispatch on every query. No per-query JIT tax.
> 🎥 **See how it works:** [`visualize/mxframe_pipeline.html`](visualize/mxframe_pipeline.html) —
> interactive step-through of how a Python query becomes a plan tree and dispatches to GPU kernels.
> Open in any browser, no server needed.
## ⚡ Quick Start
---


### Install (Development)### Install (Development)

## ⚡ Quick Start

```sh
# 1. Install pixi (Modular's package manager)
curl -fsSL https://pixi.sh/install.sh | bash

# 2. Clone and set up
git clone https://github.com/abhisheksreesaila/mxframe
cd mxframe
pixi install

# 3. Verify GPU is working
pixi run python3 scripts/_check_gpu.py
```

### Your First Query

```python
import pyarrow as pa
from mxframe import LazyFrame, Scan, col, lit

# Create an Arrow table (or load from Parquet/CSV)
data = pa.table({
    "dept":   pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
    "salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
    "age":    pa.array([32, 28, 35, 29, 40], pa.int32()),
})

# Build a lazy query plan — nothing executes yet
result = (
    LazyFrame(Scan(data))
    .filter(col("age") > lit(28))
    .groupby("dept")
    .agg(
        col("salary").sum().alias("total_salary"),
        col("salary").mean().alias("avg_salary"),
        col("age").count().alias("headcount"),
    )
    .sort(col("total_salary"), descending=True)
    .compute(device="gpu")   # or "cpu"
)

print(result.to_pandas())
```

Output:
```
  dept  total_salary  avg_salary  headcount
0  eng         345.0       115.0          3
1  mkt         110.0       110.0          1
```

---

## 📊 TPC-H Benchmark — all 22 queries

> **Hardware:** NVIDIA RTX 3090 (sm_86) · AMD 12-core CPU · Mojo 0.26.2 AOT kernels
> **Baselines:** Polars 1.29+ · Pandas 3.0 · MXFrame CPU path · MXFrame GPU path
> **Data:** TPC-H schema synthetic data (numpy RNG, fixed seed)
> **Methodology:** 1 warmup run + **median of 3 timed runs**, per engine. Warmup primes every cache an app would have primed on query #2 in production — so these numbers reflect steady-state dispatch, not first-call JIT cost.
> **Source of truth:** [`scripts/bench_results_1M.csv`](scripts/bench_results_1M.csv) and
> [`scripts/bench_results_10M.csv`](scripts/bench_results_10M.csv) — committed in repo, reproducible via
> [`scripts/benchmark_all_22.py`](scripts/benchmark_all_22.py)

### How the kernels dispatch

| Device | Path | Coverage |
|---|---|---|
| **CPU** | 100% ctypes into pre-compiled `libmxkernels_aot.so` | All 22 queries — group aggs, masked aggs, inner + left joins, gather, filter, sort, unique |
| **GPU** | ctypes into pre-compiled `libmxkernels_aot_gpu.so` | All grouped aggs (sum/min/max/count) + masked global aggs |
| **GPU** | MAX Graph, shape-cached model | Hash joins only — compiled once per `(n_left, n_right)` shape, cached for the session |

No per-query JIT on CPU. GPU aggregations skip MAX Graph entirely. GPU joins compile a model once per shape and reuse it — the warmup run primes that cache.

### 🔥 Mojo Kernel Catalogue

Every operator that matters has a hand-written Mojo kernel — there is no NumPy or PyArrow fallback in the hot path.
Source lives in [`kernels/`](kernels/), compiled once to `kernels.mojopkg` (MAX Graph ops) and `libmxkernels_aot.so` / `libmxkernels_aot_gpu.so` (AOT ctypes, zero session overhead).

> **Interactive explainer:** [`visualize/mxframe_pipeline.html`](visualize/mxframe_pipeline.html) —
> open in any browser to step through how a Python query becomes a plan tree and dispatches to these kernels.

#### Grouped aggregation kernels (`kernels/`)

| Kernel | What it does |
|---|---|
| `group_sum.mojo` | Per-group sum — shared-memory privatisation up to 8 192 groups, global-atomic fallback beyond |
| `group_min.mojo` | Per-group minimum — same shared-mem / atomic-fallback design |
| `group_max.mojo` | Per-group maximum |
| `group_count.mojo` | Per-group row count (output is `float32` for consistent tensor shapes) |
| `group_composite.mojo` | Fused multi-key composite ID: `out[i] = k0[i]·s0 + k1[i]·s1 + …` in one memory pass |

#### Global (ungrouped) aggregation kernels

| Kernel | What it does |
|---|---|
| `masked_global_agg.mojo` | Fused masked reductions: `sum`, `sum(a×b)`, `min`, `max` — a single GPU pass computes the scalar result using only rows where `mask[i]=1`. No intermediate filtered-table allocation. |

#### Join kernels

| Kernel | What it does |
|---|---|
| `join_count.mojo` | Phase 1 of inner hash join — for each left key, count matching right rows |
| `join_scatter.mojo` | Phase 2 of inner hash join — emit `(left_idx, right_idx)` index pairs |
| `join_count_left.mojo` | Phase 1 of left-outer hash join — count matches, record 1 for unmatched left rows |
| `join_scatter_left.mojo` | Phase 2 of left-outer hash join — emit pairs; unmatched left rows get `right_idx = -1` |

#### Sort, gather, and utility kernels

| Kernel | What it does |
|---|---|
| `sort_indices.mojo` | Bitonic sort — returns a permutation array; used as post-op for `ORDER BY` on GPU |
| `unique_mask.mojo` | Mark `out[i]=1` at the first row of each run in a sorted array; used by `DISTINCT` |
| `filter_gather.mojo` | Prefix-sum + scatter — compact a column by a boolean mask in two passes |

#### AOT-only kernels (`kernels_aot/`)

These live in the AOT shared libraries (`libmxkernels_aot.so` / `libmxkernels_aot_gpu.so`) and are called via ctypes — no MAX Graph session needed, cold start ≈ 0 ms.

| Function | What it does |
|---|---|
| `group_encode_i32_gpu` | GPU open-addressing hash table: assigns dense `int32` group IDs to an integer key column, replacing PyArrow `dictionary_encode` |
| `masked_global_{min,max}_f32_gpu` | GPU block-reduction min/max for masked global aggregations (added Phase 1) |
| `gather_{f32,i32,i64}_gpu` | Coalesced GPU gather — used after join and sort to reorder columns on-device |

---

### 1 M rows — warm median of 3 runs *(as of July 7 2026)*

All times in **milliseconds · lower is better**. 🟢 = faster than Polars baseline; **bold** = MXFrame wins.
Q1–Q7 and Q10 measured July 7 2026; remaining rows measured April 20 2026 — same hardware, same methodology.

| Query | Description | MX CPU | MX GPU | Polars | Pandas | CPU vs Polars | GPU vs Polars |
|---|---|---:|---:|---:|---:|---:|---:|
| Q1  | Filter + 8 aggregations       | 🟢 **11.2** | 87.2     | 36.0  | 109.1 | **3.2×**   | 0.4× |
| Q2  | Min-cost supplier             | 🟢 **6.6**  | 🟢 **15.6** | 16.3 | 9.5  | **2.5×**   | **1.0×** |
| Q3  | 3-table join + agg            | 🟢 **5.6**  | 🟢 **23.3** | 26.4 | 20.4 | **4.7×**   | **1.1×** |
| Q4  | Order priority                | 15.0     | 192.5       | 15.0 | 29.9 | 1.0×       | 0.1× |
| Q5  | Multi-join + groupby          | 🟢 **0.6**  | 🟢 **4.6**  | 26.8 | 21.0 | **44.7×**  | **5.8×** |
| Q6  | Masked global agg             | 🟢 **6.6**  | 11.4        | 10.6 | 7.6  | **1.6×**   | 0.9× |
| Q7  | Shipping volume               | 🟢 **0.7**  | 🟢 **10.1** | 40.1 | 21.5 | **57.3×**  | **4.0×** |
| Q8  | Market share                  | 🟢 **0.9**  | 🟢 **4.7**  | 20.2 | 10.3 | **22.4×**  | **4.3×** |
| Q9  | Product profit (6-table join) | 🟢 **0.6**  | 🟢 **6.6**  | 39.9 | 17.8 | **66.5×**  | **6.0×** |
| Q10 | Customer revenue              | 🟢 **3.6**  | 🟢 **16.5** | 37.5 | 22.4 | **10.4×**  | **2.3×** |
| Q11 | Important stock               | 🟢 **0.5**  | 🟢 **2.7**  | 7.4  | 3.0  | **14.8×**  | **2.7×** |
| Q12 | 2-table join + agg            | 🟢 **1.0**  | 🟢 **4.4**  | 25.0 | 788.6 | **25.0×** | **5.7×** |
| Q13 | Customer distribution         | 🟢 **16.2** | 30.2        | 27.5 | 33.5 | **1.7×**   | 0.9× |
| Q14 | Promo revenue                 | 🟢 **3.7**  | 🟢 **6.3**  | 8.6  | 288.6 | **2.3×**  | **1.4×** |
| Q15 | Top-supplier revenue          | 🟢 **1.3**  | 11.8        | 9.9  | 6.4  | **7.6×**   | 0.8× |
| Q16 | Part/supplier relationships   | 🟢 **2.1**  | 🟢 **6.3**  | 16.9 | 6.8  | **8.0×**   | **2.7×** |
| Q17 | Small-qty order               | 🟢 **0.3**  | 🟢 **2.7**  | 7.9  | 4.7  | **26.3×**  | **2.9×** |
| Q18 | Large-volume customers        | 🟢 **4.2**  | 🟢 **22.3** | 33.4 | 16.8 | **7.9×**   | **1.5×** |
| Q19 | Discounted revenue            | 🟢 **10.9** | 🟢 **11.2** | 19.6 | 22.4 | **1.8×**   | **1.8×** |
| Q20 | Potential part promo          | 🟢 **4.3**  | 🟢 **5.6**  | 30.0 | 9.6  | **7.0×**   | **5.4×** |
| Q21 | Suppliers who kept (EXISTS)   | 🟢 **26.0** | 64.1        | 31.3 | 28.6 | **1.2×**   | 0.5× |
| Q22 | Global sales opportunity      | 🟢 **7.6**  | 🟢 **16.5** | 25.4 | 56.7 | **3.3×**   | **1.5×** |

**1 M summary:** MX CPU beats Polars on **21/22** queries (Q4 tied); MX GPU beats Polars on **16/22** queries. Headline CPU wins: **Q9 66×**, **Q7 57×**, **Q5 45×**, **Q12 25×**, **Q17 26×**. Headline GPU wins: **Q9 6.0×**, **Q5 5.8×**, **Q12 5.7×**, **Q20 5.4×**, **Q8 4.3×**.



### 10 M rows — warm median of 3 runs *(as of April 20 2026)*

All times in **milliseconds · lower is better**. 🟢 = faster than Polars baseline; **bold** = MXFrame wins.
At 10M rows the join-heavy queries show the biggest wins — the GPU kernel advantage compounds with data size.

| Query | Description | MX CPU | MX GPU | Polars | Pandas | CPU vs Polars | GPU vs Polars |
|---|---|---:|---:|---:|---:|---:|---:|
| Q1  | Filter + 8 aggregations       | 🟢 **361.0**  | 1190.7      | 946.5  | 1771.7  | **2.6×**   | 0.8× |
| Q2  | Min-cost supplier             | 🟢 **5.7**    | 🟢 **11.9** | 15.4   | 7.6     | **2.7×**   | **1.3×** |
| Q3  | 3-table join + agg            | 🟢 **57.7**   | 🟢 **67.2** | 72.2   | 581.8   | **1.3×**   | **1.1×** |
| Q4  | Order priority                | 301.5         | 492.2       | 113.8  | 807.6   | 0.4×       | 0.2× |
| Q5  | Multi-join + groupby          | 🟢 **2.8**    | 🟢 **13.8** | 60.7   | 332.9   | **21.7×**  | **4.4×** |
| Q6  | Masked global agg             | 399.6         | 523.2       | 92.0   | 246.4   | 0.2×       | 0.2× |
| Q7  | Shipping volume               | 🟢 **1.8**    | 🟢 **16.4** | 76.4   | 392.5   | **42.4×**  | **4.7×** |
| Q8  | Market share                  | 🟢 **1.3**    | 🟢 **3.8**  | 40.9   | 55.1    | **31.5×**  | **10.8×** |
| Q9  | Product profit                | 🟢 **0.7**    | 🟢 **7.4**  | 89.7   | 431.3   | **128.1×** | **12.1×** |
| Q10 | Customer revenue              | 🟢 **39.2**   | 🟢 **45.0** | 131.1  | 216.2   | **3.3×**   | **2.9×** |
| Q11 | Important stock               | 🟢 **0.4**    | 🟢 **3.1**  | 6.6    | 2.5     | **16.5×**  | **2.1×** |
| Q12 | 2-table join + agg            | 🟢 **1.3**    | 🟢 **4.4**  | 116.4  | 6853.6  | **89.5×**  | **26.5×** |
| Q13 | Customer distribution         | 385.1         | 396.1       | 285.8  | 463.9   | 0.7×       | 0.7× |
| Q14 | Promo revenue                 | 🟢 **14.4**   | 🟢 **16.9** | 29.7   | 2719.2  | **2.1×**   | **1.8×** |
| Q15 | Top-supplier revenue          | 🟢 **2.7**    | 57.4        | 16.1   | 30.0    | **6.0×**   | 0.3× |
| Q16 | Part/supplier relationships   | 🟢 **2.7**    | 🟢 **6.5**  | 16.3   | 6.8     | **6.0×**   | **2.5×** |
| Q17 | Small-qty order               | 🟢 **0.6**    | 🟢 **1.5**  | 14.6   | 17.0    | **24.3×**  | **9.7×** |
| Q18 | Large-volume customers        | 🟢 **46.8**   | 69.4        | 63.4   | 242.8   | **1.4×**   | 0.9× |
| Q19 | Discounted revenue            | 🟢 **100.4**  | 🟢 **97.9** | 112.9  | 234.8   | **1.1×**   | **1.2×** |
| Q20 | Potential part promo          | 🟢 **32.3**   | 🟢 **37.1** | 39.9   | 52.9    | **1.2×**   | **1.1×** |
| Q21 | Suppliers who kept            | 756.0         | 705.3       | 85.9   | 216.6   | 0.1×       | 0.1× |
| Q22 | Global sales opportunity      | 🟢 **54.1**   | 🟢 **59.2** | 132.8  | 1292.1  | **2.5×**   | **2.2×** |

**10 M summary:** MX CPU beats Polars on **18/22** queries; MX GPU beats Polars on **15/22**. 🟢 Join-heavy queries scale dramatically: **Q9 128×**, **Q12 89×**, **Q7 42×**, **Q8 31×**, **Q17 24×**, **Q5 22×** CPU vs Polars. GPU join wins: **Q12 26.5×**, **Q9 12×**, **Q8 10.8×**, **Q17 9.7×**. At 10M, Q4/Q6/Q13/Q21 show Polars' strength on pure CPU-vectorised operations — these are Phase 6 targets (GPU string encoding + device-resident joins).
### Where MXFrame loses (same at both scales)

- **Q4, Q6, Q13, Q21** — operations where our kernel path falls back to PyArrow compute or does extra passes. These are the focus of the next milestone (see [`roadmap.md`](roadmap.md)).

### What the numbers mean

- **Correctness ✅** — all 22 queries return results that round-trip through Pandas and match Polars output.
- **Coverage ✅** — every TPC-H query has a CPU AOT path; all group aggs/masked aggs have GPU AOT paths; GPU joins use shape-cached MAX Graph models.
- **No JIT tax in steady state** — after the first query of each shape warms the GPU join model cache, every subsequent call is pure dispatch. The CPU path has no JIT at all.
- **Why GPU doesn't always win** — GPU wins scale with workload size and kernel coverage. At 10 M, GPU crushes Polars on the join-heavy queries (Q8/Q9/Q12) where Mojo's shape-cached kernels pay off. Where GPU loses, it's either PCIe overhead on tiny outputs (Q1, Q6) or ops that still route through PyArrow fallback (Q4, Q13, Q21).



---

## ⚠️ Current Limitations: Where PyArrow and NumPy Still Run

MXFrame is **not 100% Mojo end-to-end** — and that is a deliberate, documented trade-off, not a hack.
The computational hot path (aggregation, joins, sort, gather) is fully Mojo-compiled.
Certain orchestration steps still use PyArrow or NumPy for specific technical reasons explained below.

### What still runs in PyArrow / NumPy

| Operation | Library used | Technical reason |
|---|---|---|
| **String predicate masks** — `isin`, `startswith`, `contains`, `==` on string cols | PyArrow compute (SIMD C++) | GPU string kernels require UTF-8 variable-length storage. The mask itself is cheap (microseconds at 1M rows); the expensive aggregation or join that follows still runs on GPU. |
| **String group-key encoding** — e.g. `l_returnflag`, `n_name` | PyArrow `dictionary_encode` | Phase 3 added GPU hash-table encoding for *integer* keys. String hashing on GPU requires a variable-length string type not yet in the kernel set. Result is cached after the first call — zero cost on hot runs. |
| **Table assembly after joins** — string and date columns | `pa.Table.take()` | Numeric columns (`float32`, `int32`) are already gathered on GPU via `gather_f32/i32_gpu`. String and date columns have no GPU column representation yet, so they are gathered on CPU via Arrow's optimised `.take()`. |
| **Intermediate table materialisation** between join and aggregation | `pa.Table.from_arrays` | After a join, the result is stored as a CPU Arrow table before the next plan step. Phase 4 removes this round-trip — see Roadmap below. |
| **Window functions** — `rank`, `dense_rank`, `lag`, `lead`, `cum_sum` | NumPy | These are not required by any of the 22 TPC-H queries. They exist in the API for completeness; GPU ports are Phase 7+. |
| **DISTINCT** | `pa.Table.group_by().aggregate([])` | Always applied as a post-op on an already-aggregated result (typically < 1 000 rows). The cost is negligible; porting it is not a priority. |
| **Plan-level orchestration** — predicate push-down, join reorder, filter merge | Pure Python | This is *metadata*, not data movement. No rows are touched here. |

### Why the benchmarks are still honest

The benchmark methodology uses **1 warmup run + median of 3 timed runs**.
The warmup run primes three caches:

- **`_GROUP_ENCODE_CACHE`** — group-key dictionary encoding result (PyArrow SIMD, runs once per unique `(table, keys)` pair)
- **`_INPUT_PREP_CACHE`** — column array preparation and filter masking
- **`_JOIN_RESULT_CACHE`** — join output `pa.Table` (same input tables → same result)

After warmup, the 3 timed runs do **zero PyArrow encoding** and **zero re-upload** of stable column data.
They measure pure kernel dispatch + GPU execution — which is the steady-state production cost.

Cold-run numbers (first query after `clear_cache()`) are higher because encoding and upload happen once.
That is a real cost and Phase 4 reduces it for join-heavy queries.

---

## 🗺️ Roadmap: Remaining Mojo Work

The table below lists what remains to make MXFrame fully GPU-native.
Items are ordered by performance impact, not difficulty.

| Phase | What | Queries affected | Status |
|---|---|---|---|
| **Phase 4** | **Device-resident join pipeline** — after `gather_f32/i32_gpu` produces gathered columns keep them as device buffers; feed directly into `group_sum_f32_gpu` without `pa.Table.from_arrays` round-trip | Q3, Q5, Q7, Q8, Q9, Q10, Q18, Q21 | 🔲 Planned |
| **Phase 5** | **GPU top-k kernel** — `ORDER BY … LIMIT n` via per-block heap; avoids full bitonic sort of large intermediates before the final slice | Q3, Q5, Q8, Q9, Q20, Q21 | 🔲 Planned |
| **Phase 6** | **GPU string/category encoding** — open-addressing hash table for variable-length keys, replaces PyArrow `dictionary_encode` for string group-by columns | Q1, Q4, Q5, Q7, Q8, Q9, Q12 | 🔭 Future |
| **Phase 7** | **GPU string predicate evaluation** — `isin`, `startswith`, `contains` on GPU; eliminates PyArrow boolean mask for string filters | Q4, Q12, Q16, Q19 | 🔭 Future |
| **Phase 8** | **GPU window functions** — `rank`, `dense_rank`, `lag`, `lead`, `cum_sum` | Non-TPC-H workloads | 🔭 Future |

### Why Phases 4–5 are the highest ROI

Phases 4 and 5 target the **cold-run** round-trip: the CPU `pa.Table` that currently sits between a join and its downstream aggregation.
On hot runs this round-trip is already eliminated by the join-result cache — so steady-state benchmark numbers are unaffected.
On cold runs (fresh process, first query of a new shape), removing `pa.Table.from_arrays` + re-upload can save 20–80 ms for 1M-row join intermediates.

Phases 6 and 7 target operations that are **already SIMD-fast in PyArrow C++** and **already cached** after the first call.
They matter most for workloads that constantly see new table instances (streaming, per-request analytics) where the cache cannot absorb the encoding cost.

> **Current GPU coverage:** `python scripts/audit_gpu_paths.py --device gpu --min-gpu-coverage 0.8`
> exits 0 with **22/22 queries GPU-CLEAN (100%)** — every computationally heavy operator
> reaches a Mojo GPU kernel. The remaining PyArrow code is orchestration and string handling,
> not the reduction or join bottleneck.

---

## 🔁 Reproducing the Benchmark

To run the benchmark with **official TPC-H data** (generated by DuckDB's
faithful port of the TPC-H `dbgen` tool):

```sh
# Step 1 — generate TPC-H data (requires: pip install duckdb)
#   SF=1  →  ~6M lineitem rows,  ~200 MB Parquet
#   SF=0.1 → ~600K rows, quick sanity check
pixi run python3 scripts/gen_tpch_parquet.py --sf 1

# Step 2 — run the 22-query benchmark against real data
pixi run python3 scripts/bench_real_tpch.py --data-dir data/tpch_sf1 --runs 3
```

The generated `data/tpch_sf1/` directory contains 8 Parquet files (one per
TPC-H table) that you can inspect, share, or version-control.

### Scale factor guide

| `--sf` | lineitem rows | approx size | use case |
|---|---|---|---|
| 0.01 | ~60K | 2 MB | smoke test / CI |
| 0.1 | ~600K | 20 MB | local dev |
| 1 | ~6M | 200 MB | standard published benchmark |
| 10 | ~60M | 2 GB | stress test |

### Data lineage & legal note

- Data is generated by **DuckDB's TPC-H extension** — a faithful port of the
  official TPC-H `dbgen` v3.0.1 with the same value distributions (uniform,
  Zipfian, pseudo-random vocab).
- **TPC-H® is a trademark of the Transaction Processing Performance Council.**
  These results are an _independent, non-certified_ benchmark. They may _not_
  be reported as "TPC-H results" without formal TPC certification.
- Reference: <https://www.tpc.org/tpch/>

---

## 🔤 API Reference

### LazyFrame

```python
from mxframe import LazyFrame, Scan, col, lit, when

lf = LazyFrame(Scan(arrow_table))
```

| Method | Description | Example |
|---|---|---|
| `.filter(expr)` | Row filter | `.filter(col("x") > lit(10))` |
| `.select(*cols)` | Column projection | `.select("a", "b", col("c").alias("d"))` |
| `.with_columns(*exprs)` | Add/replace columns | `.with_columns((col("a") * lit(2)).alias("a2"))` |
| `.groupby(*keys)` | Start grouped agg | `.groupby("dept", "region")` |
| `.join(other, left_on, right_on, how)` | Hash join | `.join(lf2, "id", "fk_id", how="inner")` |
| `.sort(expr, descending)` | Sort rows | `.sort(col("revenue"), descending=True)` |
| `.limit(n)` | Take first N rows | `.limit(100)` |
| `.distinct()` | Deduplicate rows | `.distinct()` |
| `.compute(device)` | Execute the plan | `.compute(device="gpu")` |

### Expressions (`col`, `lit`, `when`)

```python
# Arithmetic
col("price") * (lit(1.0) - col("discount"))

# Comparison
col("date") >= lit(19940101)

# Boolean combine
(col("x") > lit(0)) & (col("y") < lit(100))

# Conditional
when(col("nation") == lit("BRAZIL"), col("revenue"), lit(0.0))

# String
col("phone").startswith("13")

# Date parts
col("orderdate").year()    # extract year as int32

# Aggregations (inside .agg())
col("salary").sum()
col("salary").mean()
col("salary").min()
col("salary").max()
col("id").count()
```

### SQL Frontend

```python
from mxframe.sql_frontend import sql

result = sql("""
    SELECT dept, SUM(salary) AS total, COUNT(*) AS n
    FROM employees
    WHERE age > 30
    GROUP BY dept
    ORDER BY total DESC
""", employees=arrow_table)
```

---

## 🔧 Supported Operations

| Category | Operations |
|---|---|
| **Filter** | `>`, `>=`, `<`, `<=`, `==`, `!=`, `&`, `\|`, `~`, `isin`, `startswith`, `contains` |
| **Aggregation** | `sum`, `mean`, `min`, `max`, `count` |
| **Groupby** | Single key, multi-key, composite key |
| **Join** | Inner, Left outer |
| **Sort** | Single/multi column, ascending/descending |
| **Window** | `year()` date part extraction |
| **Projection** | `select`, `with_columns`, `alias`, arithmetic expressions |
| **Semi-join** | Via unique-key inner join |
| **Anti-join** | Via `pc.is_in` + `pc.invert` |
| **Distinct** | Full row deduplication |
| **SQL** | `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`, `JOIN` |

---

## 📁 Project Structure

```
mxframe/
├── __init__.py            ← Public API (LazyFrame, Scan, col, lit, when, sql)
├── lazy_frame.py          ← LazyFrame, LazyGroupBy, Scan
├── lazy_expr.py           ← Expr, col(), lit(), when()
├── compiler.py            ← LogicalPlan → MAX Graph compiler
├── custom_ops.py          ← Dispatch: AOT kernels / MAX Graph / PyArrow fallback
├── optimizer.py           ← Plan rewrites (filter pushdown, join reordering)
├── plan_validation.py     ← Pre-execution plan checks
├── sql_frontend.py        ← SQL → LogicalPlan via sqlglot
│
├── kernels_aot/           ← Pre-compiled AOT shared libraries
│   ├── libmxkernels_aot.so      ← CPU kernels (ctypes-callable)
│   └── libmxkernels_aot_gpu.so  ← GPU kernels (CUDA/ROCm/Metal)
│
├── kernels/               ← Mojo kernel source (build time only)
│   ├── group_sum.mojo, group_min.mojo, group_max.mojo ...
│   ├── join_scatter.mojo, join_count.mojo
│   ├── join_scatter_left.mojo, join_count_left.mojo
│   └── filter_gather.mojo, gather_rows.mojo, unique_mask.mojo ...
│
├── kernels_aot/
│   ├── kernels_aot.mojo         ← CPU AOT entry points
│   └── kernels_aot_gpu.mojo     ← GPU AOT entry points
│
├── scripts/
│   ├── bench_simple.py          ← Clean 4-col benchmark (Pandas|Polars|MX CPU|MX GPU)
│   ├── benchmark_tpch.py        ← All 22 TPC-H query implementations
│   ├── _test_aot_smoke.py       ← AOT kernel smoke tests
│   └── quickstart.py            ← Minimal hello-world example
│
└── docs/
    ├── vision-and-architecture.md
    ├── CONTRIBUTING.md          ← Developer guide
    └── PUBLISHING.md            ← pip release steps
```

---

## 🖥️ Device Selection

```python
# CPU (default, works everywhere)
result = lf.compute(device="cpu")

# GPU (requires NVIDIA/AMD/Apple Silicon with MAX runtime)
result = lf.compute(device="gpu")
```

The GPU path uses Mojo's `DeviceContext` — the same source compiles to:
- **PTX** on NVIDIA (CUDA-compatible)
- **HSA/ROCm** on AMD
- **Metal** on Apple Silicon

---

## 🧪 Running Tests

```sh
# Smoke tests — AOT kernels
pixi run python3 scripts/_test_aot_smoke.py

# All TPC-H correctness checks
pixi run python3 scripts/_test_phase6_tpch_tier2.py

# GPU check
pixi run python3 scripts/_check_gpu.py

# Full 22-query benchmark
pixi run python3 scripts/bench_simple.py --rows 1000000 --runs 3
```

---

## 📦 Dependencies

| Package | Required | Purpose |
|---|---|---|
| `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
| `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
| `pandas >= 2.0` | ✅ | Reference implementations, Pandas bridge |
| `modular >= 26.2` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
| `polars >= 0.20` | optional | Polars bridge + benchmark comparison |
| `sqlglot >= 25` | optional | SQL frontend parsing |

---

## 🤝 Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for the developer guide, kernel writing guidelines,
and how to add new TPC-H queries.

---

## 📄 License

Apache 2.0 — see [LICENSE](LICENSE).
