Metadata-Version: 2.5
Name: lattice-grid-jupyter
Version: 1.110.3.2
Summary: Edit a pandas DataFrame as a Lattice Grid in Jupyter, with a live round-trip back to Python.
Project-URL: Homepage, https://latticegrid.dev
License: Proprietary
Keywords: anywidget,dataframe,datagrid,jupyter,lattice-grid,pandas,spreadsheet
Requires-Python: >=3.9
Requires-Dist: anywidget>=0.9
Requires-Dist: lattice-grid-pandas==1.110.3.2
Requires-Dist: pandas>=1.5
Provides-Extra: dev
Requires-Dist: duckdb>=1.0; extra == 'dev'
Requires-Dist: ipykernel; extra == 'dev'
Requires-Dist: jupyterlab>=4; extra == 'dev'
Requires-Dist: nbconvert; extra == 'dev'
Requires-Dist: playwright>=1.40; extra == 'dev'
Requires-Dist: polars>=1.0; extra == 'dev'
Requires-Dist: pyarrow>=14; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Provides-Extra: large
Requires-Dist: duckdb>=1.0; extra == 'large'
Requires-Dist: pyarrow>=14; extra == 'large'
Description-Content-Type: text/markdown

# lattice-grid-jupyter

Edit a **pandas DataFrame** as an interactive [Lattice Grid](https://latticegrid.dev)
inside Jupyter, with a live two-way round-trip back to Python.

```python
import pandas as pd
from lattice_grid_jupyter import LatticeGridWidget

df = pd.DataFrame({"name": ["Ada", "Grace"], "score": [91, 88], "active": [True, False]})

w = LatticeGridWidget(df)   # renders an editable grid
w                           # display it in a notebook cell
# ...edit cells in the grid...
w.df                        # the DataFrame, reflecting your edits (dtypes preserved)
```

Built on [anywidget](https://anywidget.dev), so it works in JupyterLab, classic
Notebook, VS Code, Colab, and anywhere ipywidgets render.

---

## Install

```bash
pip install lattice-grid-jupyter
```

Depends on `anywidget` and `pandas`. The wheel **vendors** the published Lattice
Grid bundle (`@toclocoinc/lattice-grid@1.110.3`) so offline notebooks work with no
network at render time.

### Which grid am I getting?

The one the version number says. A wheel is numbered **`<grid version>.<revision>`**
— `1.69.0.0` carries Lattice Grid `1.69.0` — so the answer is on the tin, and in
Python:

```python
import lattice_grid_pandas
lattice_grid_pandas.GRID_VERSION   # '1.69.0' -- read from the vendored bundle
```

All three packages (`lattice-grid-pandas`, `lattice-grid-jupyter`,
`lattice-grid-dash`) share the number and are released together; the two host
wrappers pin the shared one exactly, so a mixed install is refused rather than
quietly assembled. A revision above `0` is a packaging fix for the same grid.

Every grid release rebuilds these wheels automatically, so `pip install
--upgrade lattice-grid-jupyter` gets you the current grid.

## The grid bundle: CDN vs vendored

Two ways the browser gets the grid JS/CSS, chosen per widget:

| Mode | How | When |
|------|-----|------|
| **CDN** (default) | dynamic `import()` from jsDelivr at render time | online notebooks; smallest saved-notebook size |
| **Vendored** (`offline=True`) | the bundle text ships in the widget model | air-gapped / offline notebooks |

```python
w = LatticeGridWidget(df)                # CDN (default)
w = LatticeGridWidget(df, offline=True)  # vendored, no network needed
```

The default is CDN. Vendored mode carries ~3 MB of grid JS in the widget model,
which inflates a saved `.ipynb`; prefer CDN when you have network. (An alternative
not implemented here would serve the vendored asset over Jupyter's own static
handler instead of the comm; see *Notes* below.)

## What the DataFrame maps to

**dtypes -> grid column types**

| pandas dtype | grid column type |
|--------------|------------------|
| `int*`, `Int*` (nullable), `float*` | `number` |
| `bool`, `boolean` (nullable) | `boolean` |
| `datetime64` (naive or tz-aware) | `timestamp` (ISO 8601) |
| `category`, `object`, `string`, `timedelta` | `text` |

**Index**

* A default `RangeIndex(0..n-1)` is not shown as a column (it adds nothing).
* A named / non-default index becomes one **read-only** leading column.
* A `MultiIndex` becomes one read-only column **per level**.
* Row identity is a **stable, positional id** (`__row_id__`), never the index
  label — so **duplicate index labels** and every other index shape work
  correctly and edits always land on the right row.

**Missing values** — `NaN`, `NaT` and `pd.NA` all serialize to JSON `null`.

## Editing round-trip

Every committed cell edit in the grid flows back to `w.df`, cast to the column's
dtype (an int column stays int, etc.). Index columns are read-only.

## Live updates: Python -> grid

```python
w.set_data(new_df)                 # replace the whole frame, repaint
keys = w.append_rows([{...}, ...]) # append rows (DataFrame or list-of-dicts)
w.append_rows(other_df)
w.delete_rows(keys)                # delete by the keys append_rows returned
```

`append_rows` / `delete_rows` push **incremental** changes to the grid
(`grid.rows.apply({add|remove})`); `set_data` does a full `grid.rows.load()`.
Row keys are stable, so keys returned by `append_rows` stay valid for
`delete_rows` and for later edits.

## Selection, the filtered view and state: grid -> Python

The grid is not a one-way display. What the user does in it comes back as
DataFrames and a plain dict (only row **keys** cross the comm; Python slices the
frame it already holds, so dtypes, the original index and missing values are the
frame's own):

```python
w.selected    # the selected rows: a DataFrame with the original index and dtypes
              # (an EMPTY frame, never None, when nothing is selected)
w.view        # the rows passing the grid's filters (quick search included),
              # in the grid's current sort order
w.state       # filter / sort / column / group / pivot state, a plain dict
w.state = {...}   # restore a state (e.g. one you saved earlier)
```

```python
def on_select(change):
    print(len(change["new"]), "rows selected")        # change["new"] is a DataFrame

w.observe(on_select, names="selected")
w.observe(lambda c: print(len(c["new"]), "rows in view"), names="view")
```

`selected` and `view` notify **once per user change**: the front end debounces
(120 ms), so typing in the quick search does not send a message per keystroke,
and an unchanged result never notifies. Before the grid has reported, `view` is
the whole frame. Row selection is on by default (`selection="multiple"`; override
with `options={"selection": "single"}`).

**Tutorial step: select rows, get a DataFrame.**

1. `w = LatticeGridWidget(df); w`: click a row, Ctrl-click another.
2. In the next cell: `w.selected` shows exactly those rows, with their original
   index labels, so `df.loc[w.selected.index]` addresses the same rows.
3. Type in the grid's search box, then `w.view` is the matching rows in the order
   you see them. `w.view.to_csv("shown.csv")` exports what is on screen.

**Views only the browser can compute (a data router's shaped rows).** Most views
are computed in Python from the frame and the grid's state, so `w.view` returns at
once, in any cell, under Run All and with no browser. A router's shaped view
(joins, rollups, spreads) is computed by the browser. Reading `w.view` for it
before the browser has reported raises `LatticeViewNeedsBrowser` straight away,
never a wait or a hang: a notebook cell cannot receive the browser's reply while
it is still running. Read it in a later cell once the grid has rendered, or have it
delivered when it arrives:

```python
from lattice_grid_jupyter import LatticeViewNeedsBrowser

try:
    rows = w.view
except LatticeViewNeedsBrowser:
    rows = None                     # read it in a later cell, or:

handle = w.on_view(lambda view: print(len(view), "shaped rows"))
# ... called each time the browser reports a new view; handle.remove() stops it
```

## Grid options from Python

```python
w = LatticeGridWidget(
    df,
    options={"row_height": 32, "stripe": True},                 # grid options
    columns=[{"field": "score", "title": "Score", "width": 120}],  # per column
    histograms=True,           # header histograms
    profile=True,              # the Statistics tool panel
    pivot="region",            # pivot on a field (or pivot={...} for the option)
)
w.options = {"row_height": 44}   # updates the LIVE grid; the data is not re-sent
```

* Options and column entries are **plain dicts**. `snake_case` keys are accepted
  and mapped to the grid's `camelCase` (`row_height` -> `rowHeight`), but **only
  when the camelCase form is a real grid property name**: ids you chose (a column
  called `sales_total`, the keys of `context`) are never rewritten.
* **Options are data.** Functions (`compute`, `row_class`, hooks, renderers)
  cannot reach the browser: passing one raises `TypeError` rather than being
  dropped. Use the declarative forms (format/type strings, formulas, presets).
* `rows`, `rowKey` and the column set are built from the DataFrame; the keys are
  ignored with a warning. Per-column options go in `columns=[...]`, matched on
  `field`; they survive `set_data`.
* **Unknown or mistyped options are not silent.** The grid validates in the
  browser and its own warning ids come back as Python warnings (class
  `LatticeGridWarning`), e.g. `[lattice] config.unknown:row_hieght: 'row_hieght'
  is not a configuration key this grid recognises` or `config.value:rowHeight`
  for a wrong type. Per-column typos give `column.unknown:<key>`.
* `widget.options = {...}` replaces the options on the live grid by sending the
  options alone (keys you drop are reset). Per-column options and the flags
  apply at construction.

**Recipe: profile a DataFrame.**

```python
w = LatticeGridWidget(df, profile=True, histograms=True)
w
```

Header histograms show each column's distribution and clicking a bar filters on
it (then `w.view` is the filtered frame). Open the **Statistics** tab on the right
edge, click a cell, and the panel shows the present/missing/distinct counts, the
twelve numeric figures and a histogram (or, for text, the top values), all
following the filters.

## Charts

```python
from ipywidgets import HBox
from lattice_grid_jupyter import LatticeGridWidget, LatticeChart

grid = LatticeGridWidget(df)
roc = LatticeChart(grid, type="roc", label="churned", score="score")
HBox([grid, roc])        # filter the grid: the chart redraws in the browser
```

`LatticeChart(grid_or_df, type="bar", **spec)` is a chart **viewer of the grid's
dataset**, bound through the grid's own binding: it follows the grid's filters, quick
search and selection live in the browser, with no round trip to Python. A DataFrame
can be charted directly (`LatticeChart(df, type="scatter", x="a", y="b")`).

| | |
|---|---|
| `type` | any grid chart type, plus the opt-in `roc`, `fan`, `decomposition`, `splom`, `hexbin`, `ridgeline` (proven here: `bar`, `line`, `pie`, `scatter` and those six) |
| `**spec` | the chart spec as plain data: `x`, `y`, `series`, `title`, `scheme`, `measures`, ...; `snake_case` is mapped; an unknown key warns by name (`config.unknown:<key>`) and is dropped; functions raise `TypeError` |
| `height`, `offline`, `licence` | size; vendored (offline) or CDN modules; inherited from the grid when omitted |
| `chart.update(type=..., **spec)` | change the type and replace the spec live (one redraw) |
| `chart.set_data(df)` | new data for a DataFrame-backed chart |
| `chart.draws` | how many times the browser has drawn it |

Chart code loads per type (the base chart module plus the one type module), never as
one bundle with every chart. On a **windowed (large) grid** the chart is drawn from
the grouped aggregates Python already computes (`bar`, `line`, `pie`, ... with `x` and
`y`) or refused with a named warning (`chart.windowed.refused:<type>`); it never reads
the rows. One example per type, the large-grid rules and the layout notes are in
[`docs/charts.md`](docs/charts.md).

## The data router

```python
from lattice_grid_jupyter import LatticeGridWidget, LatticeRouter

router = LatticeRouter(
    sources={"customer": {"data": customers, "key": "customer_id",
                          "join": [{"from": "txn", "many": True, "foreign_key": "customer_id",
                                    "aggregate": {"spend": {"fn": "sum", "field": "amount"}}}]},
             "txn": {"data": txns, "key": "txn_id"}},
    routes={"customer": "customer"},
)
w = LatticeGridWidget(router, route="customer")   # the shaped rows
router.update("txn", new_txns)                    # a keyed diff: only the affected parents re-emit
w.view                                            # the shaped rows as a DataFrame
```

`LatticeRouter(sources=, routes=)` declares DataFrames as sources of the grid's own
**data router** with its own join (lookup, collect, rollup onto the parent), spread,
unnest and field-coercion specs as plain dicts (`snake_case` mapped, an unknown key warned
by name, functions refused). The router **runs in the browser on the grid's module**; Python
does not reimplement join semantics. `router.update(source, df)` sends a keyed diff and
`widget.router_stats` counts the rows the route re-emitted. `widget.view` is the shaped
rows (after the grid's filters) as a DataFrame, so a feature table goes straight into a
model. The spec reference and a feature-table recipe on a seeded customers / transactions /
labels fixture are in [`docs/router.md`](docs/router.md).

## Large data

Below **100,000 rows** the widget sends the whole frame to the browser
(column-major, vectorized) and the grid filters, sorts and counts it there.
**At 100,000 rows and above** it switches to a **windowed source**: no rows are
sent up front, and

* the browser asks Python for each **window of rows as the person scrolls**
  (100 rows per request; the grid's loading rows and loading banner show while
  a window is in flight);
* **sort, filter, quick search, group-by subtotals, the distinct values a
  filter menu lists, header histograms and column profiles** are computed in
  Python over the **whole frame**, and only the results cross the comm: a
  window of rows, a list of groups, a set of bucket counts, a profile;
* `widget.selected` still returns the selected rows as a DataFrame, and
  `widget.view` returns a **lazy handle** instead of materialising millions of
  rows (see below).

```python
w = LatticeGridWidget(big_df)                          # 10M rows: windowed automatically
w = LatticeGridWidget(df, large_threshold=1_000_000)   # move the threshold
w = LatticeGridWidget(df, windowed=True)               # always windowed (False: never)
w.windowed                                             # which mode this widget is in

w = LatticeGridWidget("events.parquet")                # a Parquet file, read by DuckDB
w = LatticeGridWidget(polars_df)                       # a Polars frame (no copy into pandas)
```

**`view` above the threshold** is a `LazyView`: nothing is computed until you
ask, and then only what you ask for.

```python
v = w.view          # <LazyView 1,204,331 of 10,000,000 rows (lazy) ...>
len(v)              # how many rows pass the grid's filters
v.head(20)          # the first 20, in the grid's sort order, as a DataFrame
v.slice(1000, 1100) # any range of the view
v.to_pandas()       # the whole view, explicitly (may be large)
v.positions()       # row positions in the source frame
```

`w.observe(fn, names="view")` fires when the grid's filter, quick search or
sort changes, with a fresh `LazyView`.

**Measured for 10,000,000 rows x 20 columns** (JupyterLab 4, headless Chrome,
16-core development machine): first paint about 350 ms; a scroll jump to a
far, never-fetched window about 70 to 80 ms (median); sorting all 10M rows on a
number about 2 s, after which scrolling the sorted view is as fast as unsorted.
Before this, the widget crossed 5 s to first paint between 50,000 and 100,000
rows. Details and every figure: [`docs/large-data.md`](docs/large-data.md).

**What to know:**

* **Inputs.** A pandas DataFrame (editable: a cell edit reaches `w.df`); a
  Parquet path or a Polars DataFrame (read-only), held as one Arrow table
  (a Parquet file is read by DuckDB's parallel reader) and converted to
  pandas one column at a time, only when a query touches that column. These
  need `pip install "lattice-grid-jupyter[large]"` (duckdb, pyarrow).
* **One set of semantics.** The Python engine follows the grid's own kernels:
  text comparisons ignore case unless the filter says `caseSensitive`, missing
  values sort last in both directions, ties keep frame order, a `ne` /
  `notContains` matches a missing value. Tests hold the windowed answers equal
  to the grid's client-side answers on the same frame.
* **Differences from the client path.** Quick search matches each field's
  value text (`1234.5`, an ISO timestamp), not the formatted cell text, and a
  term cannot span two columns. Text sorts by case-folded order, an
  approximation of the browser's collator for non-ASCII text. Date histograms
  bucket in UTC. Pivoting is not pushed: a pivot over a windowed grid is
  refused by the grid by name (`pushdown:pivot-unsupported`).
* **Cost.** The first query that touches a column over 10M rows converts or
  scans it (a sort on a text column is the slowest: seconds); the order is then
  cached, so scrolling through a sorted view only slices it. Each new filter is
  one pass over the frame.
* `append_rows` / `delete_rows` / `set_data` work above the threshold (the grid
  re-sources itself); `set_data` across the threshold switches the mode.

### DuckDB as the source: every request is SQL

Give the widget a **DuckDB relation**, or a **connection plus a table or a query**, and the
data never leaves DuckDB. Each request the grid makes (a row window, sort, every filter
operator, quick search, grouping with aggregates, header histograms, the Statistics tab, the row
count) becomes one SQL statement on your connection, and only the answer reaches Python: the
requested window of rows, a list of groups, a few bucket counts.

```python
import duckdb
from lattice_grid_jupyter import LatticeGridWidget

con = duckdb.connect()
con.execute("""CREATE TABLE events AS
               SELECT i, 'user ' || (i % 5000) AS who, i % 7 AS day, random() * 100 AS amount
               FROM range(5000000) t(i)""")

w = LatticeGridWidget(con.table("events"))                # a relation (con.sql(...), duckdb.sql(...))
w = LatticeGridWidget(con=con, table="events")            # a connection + a table ("schema.table" works)
w = LatticeGridWidget(con=con, sql="SELECT * FROM events WHERE day < 5 ORDER BY i")   # or a query
con.execute("COPY events TO 'events.parquet' (FORMAT parquet)")
w = LatticeGridWidget("events.parquet", pushdown=True)    # a Parquet path, queried in place
w = LatticeGridWidget(con.table("events"))                # (back to the relation for the rest)

w.state = {"version": 2, "filters": {"col": "who", "op": "contains", "value": "O'Brien"},
           "sort": [{"col": "amount", "dir": "desc"}]}
v = w.view        # a lazy handle: len(v) is a count(*), v.head(20) fetches 20 rows
len(v), v.head(5)
```

* **Values are bound.** Every user value (a filter value, a quick-search term, a range bound, a
  window offset) is a prepared-statement parameter, never part of the SQL text, so a value
  containing a quote is just a value. The only text taken from a request is a column name, and
  only when it is one of the source's columns (then quoted).
* **Same answers as the Arrow path.** The filter operators, case rules, missing-value ordering,
  histogram buckets and profile figures are the ones the Python engine already uses; tests hold
  them equal on a seeded sample and run a 5,000,000-row table through every request kind, reading
  the statements back from `w._engine.sql_log` (each entry is `(sql, params, rows_returned)`).
* **Memory.** Only the window being shown is held in Python. The unfiltered, unsorted window is
  a plain `LIMIT/OFFSET`.
* **Row identity** is the row's position in the source, counted by DuckDB in scan order. That is
  stable for a table or a file; give a `sql=` query an `ORDER BY` if its order must be repeatable.
* **Read-only**, like the Parquet and Polars inputs; `w.df` is not available (use `w.view`).
  Below `large_threshold` rows a DuckDB source is read into pandas like any frame, unless you pass
  `windowed=True`.
* **Cost.** A new filter is one DuckDB pass (about a second on 5M rows). A sort is a top-N
  query; jumping far into a sorted view of a text column sorts all of it (seconds).
* A `sql=` string is SQL you wrote: it is run as given. A `pushdown=True` Parquet path needs
  `duckdb`, as the `[large]` extra installs it.

## Licensing

The grid renders **fully and unwatermarked on `localhost`** (the normal
data-science case) with **no key** — that is the free tier. A widget served from
a non-localhost origin (a deployed notebook server, Voila, Binder on a public
host) should pass a key:

```python
w = LatticeGridWidget(df, licence="LG-...")
```

The key is threaded straight into `createGrid({ licence })`; `grid.licence.state()`
resolves to `localhost`, `licensed`, or `trial`.

## Hosts

JupyterLab 4 and Notebook 7 are proven by automated tests; VS Code and Colab need
a manual check. See [`docs/hosts.md`](docs/hosts.md). `tests/test_grid_version.py`
checks that the widget, the vendored bundle, the chart and router modules, the
name tables and the Dash bundle all carry the same grid.

## Development / tests

```bash
pip install -e ".[dev]"
pytest tests/test_serialize.py tests/test_roundtrip.py   # Python round-trip (no browser)
pytest tests/test_smoke_browser.py                        # real-browser smoke test
pytest                                                    # everything, incl. the Lab/Notebook host proofs and the Dash package
```

The smoke test drives a **real Chromium** via Playwright: it renders the widget's
front end, performs a real edit in the grid, and asserts the edit payload reached
the (mocked) model — the one gap the headless Python tests cannot cover. It uses
the vendored bundle, so it needs no network. If no browser is available it is
skipped (never faked).

## Notes / not in this release

* Vendored mode transfers the bundle through the widget comm; serving it over
  Jupyter's static handler would keep saved notebooks small — a future option.
* `set_data` assumes the same schema (columns/index shape). A different schema
  should use a fresh widget.
* Windows travel as columnar JSON. Arrow IPC was measured (card 1618) and is
  faster only for bulk transfers the windowed design never makes; see
  `docs/large-data.md`.

---
Built with [Lattice Grid](https://www.latticegrid.dev), a JavaScript data grid with a Data Router: one live feed keeps grids, charts, boards, Gantt and KPI tiles in step. [Documentation](https://www.latticegrid.dev/docs/) · [Demos](https://www.latticegrid.dev/demos/) · [Licence](https://www.latticegrid.dev/licence/)
