Metadata-Version: 2.5
Name: datashader-browser
Version: 0.2.0a0
Summary: Bounded-memory Datashader pipelines for Python running in the browser
Project-URL: Homepage, https://github.com/SuMayaBee/datashader-browser
Project-URL: Issues, https://github.com/SuMayaBee/datashader-browser/issues
Project-URL: Repository, https://github.com/SuMayaBee/datashader-browser
Author: Sumaiya Islam
License-Expression: BSD-3-Clause
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Python: >=3.10
Requires-Dist: datashader<0.20,>=0.19.1
Requires-Dist: pandas>=2.2
Provides-Extra: data
Requires-Dist: pillow>=11; extra == 'data'
Requires-Dist: pyarrow>=21; extra == 'data'
Provides-Extra: holoviews
Requires-Dist: holoviews<2,>=1.21; extra == 'holoviews'
Provides-Extra: test
Requires-Dist: holoviews<2,>=1.21; extra == 'test'
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# datashader-browser

Run Datashader-powered visualizations entirely in a browser with Python and
WebAssembly—without assuming that the complete dataset fits in browser memory.

This is an early, independent project. It is not yet an official HoloViz
project and is not affiliated with or endorsed by the Datashader maintainers.

[Try the live browser demo](https://sumayabee.github.io/datashader-browser/).
Open `streaming-points.ipynb` and run all cells.

## What works now

The first vertical slice is a bounded-memory point aggregator:

```python
import datashader as ds
import datashader.transfer_functions as tf

from datashader_browser import StreamingCanvas, iter_csv_chunks, iter_parquet_batches

source = iter_csv_chunks(
    "points.csv",
    chunksize=250_000,
    columns=["longitude", "latitude", "value"],
)

canvas = StreamingCanvas(
    plot_width=1000,
    plot_height=600,
    x_range=(-180, 180),
    y_range=(-90, 90),
)
aggregate = canvas.points(source, "longitude", "latitude", agg=ds.mean("value"))
image = tf.shade(aggregate)

# A local Parquet file can yield projected Arrow record batches directly.
parquet_source = iter_parquet_batches(
    "points.parquet", columns=["longitude", "latitude"], batch_size=250_000
)
aggregate = canvas.points(parquet_source, "longitude", "latitude", agg=ds.count())
```

Only one input chunk and Datashader's fixed-size aggregate buffers are live at
once. The implementation calls Datashader's own compiler, so point binning and
reductions match native Datashader instead of being reimplemented.

Currently verified reductions are `count`, `sum`, `mean`, `min`, and `max`.
Explicit `x_range` and `y_range` values are required to preserve a true
single-pass pipeline.

## Interactive HoloViews integration

The optional HoloViews adapter connects Bokeh viewport events to repeatable,
bounded Parquet scans:

```bash
pip install "datashader-browser[data,holoviews]"
```

```python
from datashader_browser import ParquetSource
from datashader_browser.holoviews import datashade

source = ParquetSource("points.parquet", batch_size=250_000)
plot = datashade(
    source,
    "longitude",
    "latitude",
    x_range=(-180, 180),
    y_range=(-90, 90),
)
plot
```

HoloViews and Bokeh supply axes, zoom and pan tools, and dynamic plot updates.
On each viewport change, the adapter calls `StreamingCanvas` and returns a new
`RGB` element. The current PyArrow backend applies viewport predicates within
each bounded batch but still scans the file sequentially. Spatially pruned
reads remain a data-plane optimization rather than a current performance claim.

## NYC Taxi real-data verification

The streaming accumulator successfully rendered all **11,842,094 pickup
locations** in HoloViz's transformed
[January 2015 NYC Taxi dataset](https://examples.holoviz.org/gallery/nyc_taxi/nyc_taxi.html).
The 385 MB Parquet file was decoded as 48 Arrow/Pandas batches of at most
250,000 rows; the complete table was never materialized as one DataFrame.

![Datashader rendering of 11.8 million January 2015 NYC taxi pickups](docs/assets/nyc-taxi-pickups.png)

One warm-cache Linux run produced the 1200-by-700 image in 3.24 seconds with
514 MiB peak process RSS. Runtime and memory will vary by machine. Reproduce it
with:

```bash
uv sync --extra data
uv run python examples/nyc_taxi.py --download
```

This benchmark validates `StreamingCanvas` against a real large dataset on
native Python. The browser build now includes an experimental PyArrow Parquet
reader: open `nyc-taxi-browser.ipynb`, place the source file in the browser
filesystem, and it streams projected Arrow record batches through the same
accumulator. Full 385 MB end-to-end browser timing and memory results are still
an explicit validation task, not a claim made by this repository.

## Browser runtime

The initial deployment target is JupyterLite with the Xeus-Python WebAssembly
kernel. Its environment is described in [`environment.yml`](environment.yml)
and uses the experimental Numba and llvmlite packages from emscripten-forge.
The kernel executes in a Web Worker, keeping aggregation off the browser UI
thread. The Parquet-capable build is currently about 256 MB because PyArrow
brings compiled Arrow and compression dependencies; this is an engineering
baseline, not the final delivery architecture. The first visit therefore needs
a substantial download, while subsequent use benefits from the browser cache.

Browser verification: `streaming-parquet.ipynb` has been run end-to-end in the
Xeus-Python WebAssembly kernel. It created a 57.2 MB, two-million-row Parquet
file in the browser filesystem, streamed it through 20 Arrow batches, checked
the final row count, and rendered the shaded aggregate with no browser console
errors. The full 385 MB NYC Taxi browser benchmark remains a separate task.
`interactive-parquet.ipynb` is the HoloViews integration example; browser
interaction validation is tracked separately from the existing static render.

```bash
uv sync --extra test
uv run pytest

# Build the browser site (requires the build dependencies below).
uv pip install -r requirements-build.txt
uv run jupyter lite build \
  --XeusAddon.environment_file=environment.yml \
  --contents content \
  --output-dir dist
```

## Large-data strategy

“In the browser” does not mean that an unlimited dataset can be copied into a
Pandas DataFrame. The project uses three execution levels:

1. **Stream:** scan CSV or local Parquet/Arrow batches through a bounded-memory Datashader
   accumulator. Total row count affects time, not peak aggregate memory.
2. **Prune:** use Parquet row groups, spatial partitions, and DuckDB-Wasm to
   read only the viewport and required columns.
3. **Tile:** for billion-row interactive datasets, fetch spatial partitions or
   precomputed multiresolution aggregates instead of rescanning the source on
   every zoom.

See [Architecture](docs/architecture.md), [Research](docs/research.md), and the
[roadmap](ROADMAP.md) for the engineering decisions and current limitations.

## Status

This repository is an alpha-stage research implementation. The WebAssembly
Numba/llvmlite stack works in current demonstrations but its patches have not
yet been fully upstreamed. Production use should wait for reproducible browser
builds, compatibility CI, and measured browser memory limits.

## License

BSD-3-Clause. Datashader is a separate BSD-3-Clause project owned by its
respective copyright holders and contributors.
