Metadata-Version: 2.5
Name: narwhals-datafusion
Version: 0.1.2
Summary: Apache DataFusion backend for Narwhals, via the Narwhals plugin system
Project-URL: Repository, https://github.com/s5dsn-eqee/narwhals-datafusion
License-Expression: MIT
License-File: LICENSE.md
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: datafusion>=54
Requires-Dist: narwhals>=2.25
Requires-Dist: pyarrow>=14.0.0
Provides-Extra: extra-functions
Requires-Dist: datafusion-extra-functions-ffi>=0.1; extra == 'extra-functions'
Description-Content-Type: text/markdown

# narwhals-datafusion

[![CI](https://github.com/s5dsn-eqee/narwhals-datafusion/actions/workflows/ci.yml/badge.svg)](https://github.com/s5dsn-eqee/narwhals-datafusion/actions/workflows/ci.yml)
[![PyPI version](https://badge.fury.io/py/narwhals-datafusion.svg)](https://badge.fury.io/py/narwhals-datafusion)
[![PEP 740](https://img.shields.io/badge/PEP%20740-attested-3775A9?logo=pypi&logoColor=white)](https://pypi.org/project/narwhals-datafusion/#files)
[![Downloads](https://static.pepy.tech/badge/narwhals-datafusion/month)](https://pepy.tech/project/narwhals-datafusion)

[Apache DataFusion](https://datafusion.apache.org/python/) backend for
[Narwhals](https://github.com/narwhals-dev/narwhals), implemented as an
out-of-tree plugin via the `narwhals.plugins` entry-point system.

## Usage

```python
import narwhals as nw
import pyarrow as pa
from datafusion import SessionContext

ctx = SessionContext()
df = ctx.from_arrow(pa.table({"a": [1, 2, 3], "b": ["x", "y", "x"]}))

lf = nw.from_native(df)  # -> nw.LazyFrame, dispatched to this plugin
result = (
    lf.group_by("b")
    .agg(nw.col("a").sum())
    .sort("b")
    .collect(backend="pyarrow")
)
```

Everything stays lazy until `.collect()`: narwhals expressions are translated to
`datafusion.Expr`, frame verbs to `datafusion.DataFrame` methods, and the plan
executes in DataFusion's Rust engine. Dtypes are pyarrow end-to-end.

## Architecture

The backend sits on narwhals' shared SQL layer (`narwhals._sql`), the same
abstraction DuckDB, Ibis, and Spark use. It subclasses narwhals' private
internals (`narwhals._sql`, `narwhals._compliant`), so it is tested against one
exact release, vendored as the `narwhals/` git submodule (currently 2.25), and
the dependency pins are floors only. The plugin provides:

| Module | Class |
|---|---|
| `dataframe.py` | `DataFusionLazyFrame` — frame verbs over `datafusion.DataFrame` |
| `expr.py` | `DataFusionExpr` — the six `SQLExpr` hooks + backend specifics |
| `namespace.py` | `DataFusionNamespace` — the four `SQLNamespace` primitives, IO, horizontal fns |
| `group_by.py`, `selectors.py`, `expr_str/dt/list/struct.py` | supporting surface |
| `utils.py` | function-name remapping, window/sort builders, dtype bridge (delegates to `narwhals._arrow`) |

## `mode`, `skew`, `kurtosis` via `datafusion-extra-functions`

DataFusion core deliberately keeps its function library lean, so aggregates
like `mode`/`skewness`/`kurtosis` live in the contrib
[`datafusion-extra-functions`](https://github.com/datafusion-contrib/datafusion-extra-functions)
crate (Rust-only, no wheel on PyPI). The
[`datafusion-extra-functions-ffi`](https://github.com/s5dsn-eqee/datafusion-extra-functions-ffi)
package — a prebuilt wheel, installed through the `extra-functions` extra —
exposes its aggregate UDFs to datafusion-python via the
`__datafusion_aggregate_udf__` PyCapsule protocol; they back `Expr.mode`,
`Expr.skew`, and `Expr.kurtosis`:

```sh
pip install "narwhals-datafusion[extra-functions]"
```

Without the extra those three methods raise `NotImplementedError` and
everything else works. The shim's FFI ABI is tied to the `datafusion` major it
was compiled against (currently 54), which the wheel's own dependency pin
enforces.

## API coverage

Status of the narwhals public API on this backend (`narwhals==2.25`,
`datafusion==54`). ⚠️ entries work with the caveat in parentheses; details in
[Known limitations](#known-limitations-as-of-datafusion-54).

| Namespace | ✅ Supported | ⚠️ Partial | ❌ Not supported |
|---|---|---|---|
| `Expr` | `abs` `alias` `all` `any` `any_value` `ceil` `clip` `cos` `count` `cum_count` `cum_max` `cum_min` `cum_sum` `diff` `exp` `fill_nan` `first` `floor` `is_between` `is_close` `is_duplicated` `is_finite` `is_first_distinct` `is_in` `is_last_distinct` `is_nan` `is_null` `is_unique` `last` `len` `log` `max` `mean` `median` `min` `null_count` `over` `pipe` `rank` `rolling_mean` `rolling_std` `rolling_sum` `rolling_var` `round` `shift` `sin` `sqrt` `std` `sum` `var` | `cast` (no `Enum`) · `fill_null` (no `strategy` + `limit`) · `kurtosis` (`[extra-functions]` extra) · `mode` (`[extra-functions]` extra, `keep="any"` only) · `n_unique` (not over windows) · `replace_strict` (explicit `default` required) · `skew` (`[extra-functions]` extra) | `cum_prod` `quantile` |
| `Expr.str` | `contains` `ends_with` `head` `len_chars` `pad_end` `pad_start` `replace_all` `slice` `split` `starts_with` `strip_chars` `strip_chars_end` `strip_chars_start` `tail` `to_lowercase` `to_time` `to_uppercase` `zfill` | `to_date`/`to_datetime` (explicit `format` required) · `to_titlecase` (no word breaks on digits) | `replace` (use `replace_all`) |
| `Expr.dt` | `convert_time_zone` `date` `day` `hour` `microsecond` `millisecond` `minute` `month` `nanosecond` `ordinal_day` `second` `to_string` `truncate` `weekday` `year` | `replace_time_zone` (`None`/`"UTC"` only) | `offset_by` `timestamp` `total_microseconds` `total_milliseconds` `total_minutes` `total_nanoseconds` `total_seconds` |
| `Expr.list` | `contains` `get` `len` `max` `min` `sort` | `unique` (`maintain_order=False` only) | `mean` `median` `sum` |
| `Expr.struct` | `field` | | |
| `LazyFrame` | `collect` `collect_schema` `drop` `drop_nulls` `filter` `group_by` `head` `join` `rename` `select` `sort` `top_k` `unique` `unpivot` `with_columns` `with_row_index` | `explode` (single column) · `sink_parquet` (file path only) | `join_asof` |

Not listed: methods narwhals itself doesn't support on *any* lazy/SQL backend
(`Expr.filter`, `Expr.drop_nulls`, `Expr.unique`, `Expr.map_batches`,
`Expr.ewm_mean`, `LazyFrame.tail`, `LazyFrame.gather_every`).

## Known limitations (as of datafusion 54)

- `join_asof`, exact `quantile`,
  `cum_prod`, `list.sum/mean/median`, `dt.total_*`, `dt.offset_by`,
  `dt.timestamp`, `str.replace`, `Enum` casts — no engine support;
  raise `NotImplementedError`.
- `mode`, `skew`, `kurtosis` need the `extra-functions` extra (see above);
  without it they raise `NotImplementedError` naming it.
- `n_unique().over(...)` raises: DataFusion silently ignores `DISTINCT` inside
  window aggregates, which would return wrong results. (The same engine quirk
  drops `ORDER BY`/`IGNORE NULLS` declared inside window aggregates — this
  backend moves those modifiers onto the window itself.)
- `fill_null(strategy=..., limit=n)` raises: bounded window frames with
  `first_value`/`last_value` need `retract_batch`, unimplemented engine-side.
- `replace_time_zone` supports `None` (strip) and `"UTC"` only; use
  `convert_time_zone` for instant-preserving conversions.
- `str.to_datetime`/`to_date` require an explicit `format`.
- `replace_strict` requires an explicit `default`.
- `nw.scan_csv`/`nw.scan_parquet` cannot dispatch to a plugin backend yet
  (narwhals gap); read with a DataFusion `SessionContext` and pass the frame
  to `nw.from_native`.
- `str.to_titlecase` uses `initcap`, which doesn't break words on digits.
- No row-order guarantees except after `sort` (standard for SQL engines):
  `concat` may interleave its inputs and backward `fill_null` may reorder
  rows. Sort explicitly when order matters.

## Development

```sh
git submodule update --init      # narwhals, pinned to the targeted release
uv sync --group tests --extra extra-functions
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow.

## Running narwhals' own test suite against this backend

The narwhals repo is vendored as a git submodule pinned to the targeted
release:

```sh
git submodule update --init
uv run --group tests python run_tests.py    # known failures deselected
```

For the full, unfiltered run:

```sh
uv run --group tests pytest narwhals/tests -c narwhals/pyproject.toml \
    -p narwhals_datafusion.testing -p env --use-external-constructor
```

Regenerate the deselect list after fixing tests or bumping the submodule with
`uv run --group tests python update_run_tests.py`.
