Metadata-Version: 2.4
Name: lance_nd
Version: 0.1.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Dist: nd-arrow-array[arrow,xarray]>=0.5,<0.6
Requires-Dist: numpy>=1.24
Requires-Dist: pyarrow>=14
Requires-Dist: xarray>=2023.1
Requires-Dist: dask ; extra == 'dev'
Requires-Dist: mypy ; extra == 'dev'
Requires-Dist: pandas-stubs ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Provides-Extra: dev
Summary: Store xarray datasets as N-dimensional Arrow arrays in Lance
Keywords: arrow,lance,xarray,netcdf,n-dimensional
License: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/robinskil/lance-nd
Project-URL: Issues, https://github.com/robinskil/lance-nd/issues
Project-URL: Repository, https://github.com/robinskil/lance-nd

# lance_nd for Python

Store xarray datasets as N-dimensional Arrow arrays in a
[lance-nd](https://github.com/robinskil/lance-nd) table.

The package does three things:

1. It converts each `xarray.Dataset` to an `NdRecordBatch` of the
   [nd-arrow-array](https://pypi.org/project/nd-arrow-array/) package.
2. Rust writes the datasets to a lance-nd table, and commits all of them in
   one version when you call `finish()`.
3. It reads the table back as a stream of `NdRecordBatch` objects. Each
   object gives its dataset with `to_xarray()`.

## Install

Build the package from this directory with [maturin](https://www.maturin.rs/):

```bash
python -m venv .venv
.venv/bin/python -m pip install maturin
.venv/bin/python -m maturin develop --release --extras dev
```

On Windows, use `.venv/Scripts/python`. The `dev` extra installs the test
and type check tools.

Without `--release`, maturin builds the Rust code without optimizations,
and writes are several times slower.

## Usage

```python
import xarray as xr
import lance_nd

# Create a table. Readers see nothing before finish().
with lance_nd.create("data.lance", stats=True) as writer:
    writer.write(ds_a)                   # an xarray.Dataset
    writer.write([ds_b, ds_c])           # a list of datasets

# Append dask chunks, one row per chunk.
with lance_nd.append("data.lance") as writer:
    writer.write(ds_d, chunks=["time"])

table = lance_nd.open("data.lance")
for batch in table.scan(filter="__global_attrs__.title = 'a'", prune="sst.values > 300"):
    ds = batch.to_xarray()               # an nd_arrow_array.NdRecordBatch
```

The [Python wiki page](https://github.com/robinskil/lance-nd/wiki/Python)
explains the writer rules, chunks, encoding options, stats, scans, casting,
object stores and the encoding rules.

## Tests

```bash
.venv/bin/python -m pytest tests
```

`tests/test_s3.py` runs when `LANCE_ND_S3_ENDPOINT` points to an S3 server
with the bucket `lance-nd-test`, see the [Object stores](https://github.com/robinskil/lance-nd/wiki/Object-Stores) wiki page.

