Metadata-Version: 2.5
Name: jupyter-storage-explorer
Version: 0.1.0
Summary: A storage explorer for Jupyter: browse cloud storage and data lakes, preview Parquet, Delta Lake and more
Requires-Python: >=3.11
Requires-Dist: anywidget>=0.9
Requires-Dist: fsspec>=2023.1.0
Requires-Dist: pygments>=2.12
Requires-Dist: traitlets>=5.4
Description-Content-Type: text/markdown

# Jupyter Storage Explorer

A file explorer for cloud storage and data lakes, right inside your Jupyter notebook, in the spirit of Azure Storage
Explorer.

Browse Amazon S3, Azure Data Lake / Blob Storage, Google Cloud Storage, local disks and more; preview Parquet files,
Delta Lake tables, CSV, Excel, JSON and other formats; page through, sort and filter tables; get ready-to-run code that
loads the data into your notebook; and upload, download, move and edit files without leaving JupyterLab.

Two widgets:

- **`StorageExplorer`**: the file explorer, with previews and the data browser.
- **`DataExplorer`**: the data browser alone, for one table.

![StorageExplorer with a JSON Lines file open in the data browser](figs/storage-explorer.png)

## Installation

```bash
pip install jupyter-storage-explorer
```

Install the package for your storage too:

| Storage                        | Package    | Address                                              |
|--------------------------------|------------|------------------------------------------------------|
| Amazon S3                      | `s3fs`     | `s3://bucket/path`                                   |
| Azure Data Lake / Blob Storage | `adlfs`    | `abfs://container@account.dfs.core.windows.net/path` |
| Google Cloud Storage           | `gcsfs`    | `gs://bucket/path`                                   |
| Local disk, in-memory          | (included) | `/path`, `memory://`                                 |

Previewing and querying data files needs a data library, e.g. `pip install polars deltalake`. If one is missing, the
preview tells you what to install.

## Quick start

```python
from jupyter_storage_explorer import StorageExplorer

StorageExplorer("s3://my-bucket/data")
```

The explorer appears as the cell's output. Other examples:

```python
StorageExplorer()  # the current folder
StorageExplorer("abfs://lake@myaccount.dfs.core.windows.net/raw")  # Azure Data Lake
StorageExplorer("s3://my-bucket/data", profile="analytics")  # Amazon S3, with options
StorageExplorer("gs://my-bucket", token="google_default")  # Google Cloud
```

Options after the address (like `profile` above) are passed to the storage package, for example to choose credentials.

To open one table directly, see [DataExplorer](#dataexplorer-one-table). The `notebooks` folder has a demo of each
widget (`storage_explorer.ipynb`, `data_explorer.ipynb`) on sample data made by `make_samples.py`.

### From the JupyterLab launcher

The separate package **[`jupyter-storage-explorer-launcher`](launcher/README.md)** adds **Storage Explorer** to
JupyterLab's launcher and command palette, opening the explorer in its own tab on the launcher's folder (the one the
file browser showed):

```bash
pip install jupyter-storage-explorer-launcher
```

![The Storage Explorer item in JupyterLab's launcher](launcher/figs/launcher.png)

What the tab shows can be configured with a `launcher.py` file; see the [launcher's README](launcher/README.md).

### Several storages in one explorer

Give each storage a name; a drop-down on the left of the address bar switches between them, and each one remembers the
folder you were in.

```python
StorageExplorer(
    filesystems={
        "Data lake": "abfs://lake@myaccount.dfs.core.windows.net",
        "Raw files (S3)": "s3://my-bucket/raw",
        "Local": "/home/me/data",
    },
)
```

## Using the explorer

- **Browse**: click folders in the tree, double-click them in the list, or click the address bar; ↑ goes up, ⟳
  refreshes. Ctrl+click (⌘+click) selects several items.
- **Preview**: double-click a file, or use **Preview** (with nothing selected, it previews the current folder, e.g. a
  Delta Lake table). Tables open in the data browser instead (see [Query data](#query-data)). Esc closes the panel.
- **Right-click**: Preview as, Open, Edit (text files), Python code to load the data (polars, pandas, PySpark, …),
  Download, Copy URL.
- **Drag and drop**: drop files from JupyterLab's file browser or your computer to upload them; drag files onto a folder
  to move them. Both ask for confirmation.

## Query data

With [polars](https://pola.rs) installed (plus `deltalake` for Delta Lake, `fastexcel` for Excel), tables open in a data
browser that reads 1,000 rows at a time:

- ◀ ▶ to page, click a column name to sort, its funnel to filter (*Null / empty*, *Range*, *Contains*, *Regex*).
- **Select columns** to hide columns, drag a header's edge to resize, **Reset** to clear filters, **⟳** to reload.

Filters, sorting, hidden columns and widths are remembered per table until the kernel restarts, or in a file with
`query_settings=QuerySettingsStore("query-settings.json")` (from `jupyter_storage_explorer.query`). polars reads the
data with the credentials in the environment, not the explorer's options.

### DataExplorer: one table

`DataExplorer` opens the same data browser for a single table, without the file explorer around it:

```python
from jupyter_storage_explorer import DataExplorer

DataExplorer(
    "s3://my-bucket/lake/sales"
)  # a Delta Lake table, Parquet, CSV, JSON Lines or Excel
DataExplorer("data/orders.tsv", format="csv")  # choose the format by name
DataExplorer(
    "abfs://lake@myaccount.dfs.core.windows.net/raw/orders.parquet", height="600px"
)
```

![DataExplorer showing a Delta Lake table in a notebook](figs/data-explorer.png)

It takes the same storage options, `format_options=` (see [Format options](#format-options)) and `query_settings=` as
`StorageExplorer`, and shares its remembered settings. If
polars (or e.g. `deltalake`) is missing, it raises an error with the `%pip install` command.

## Supported formats

| Format                                                       | Name         | Preview                       | Query data | Load with                                           |
|--------------------------------------------------------------|--------------|-------------------------------|------------|-----------------------------------------------------|
| Delta Lake tables                                            | `delta`      | First rows                    | ✓         | polars (with deltalake), PySpark (with delta-spark) |
| Parquet files, and folders of them (e.g. Spark, Hive output) | `parquet`    | First rows                    | ✓         | polars, pandas, PySpark                             |
| CSV, TSV                                                     | `csv`        | First rows                    | ✓         | polars, pandas, PySpark                             |
| JSON Lines                                                   | `json-lines` | First records                 | ✓         | polars, pandas, PySpark                             |
| JSON                                                         | `json`       | Collapsible tree              |            | Python, polars, pandas                              |
| Excel (`.xlsx`)                                              | `excel`      | First rows of the first sheet | ✓         | polars, pandas                                      |
| Markdown                                                     | `markdown`   | Rendered                      |            | Python                                              |
| Text, code, configuration (YAML, SQL, Python, …)             | `text`       | With syntax highlighting      |            | Python                                              |
| Images (PNG, JPEG, GIF, SVG, …)                              | `image`      | Shown                         |            | Pillow                                              |

The name identifies the format in `DataExplorer(..., format=...)` and in CSS (see
[Customizing icons](DEVELOPMENT.md#customizing-icons)).

### Format options

`format_options` is one dictionary for all formats; each format uses the options it takes. Options are named after the
library they are for, and the ones for a single format also after the format:

```python
StorageExplorer(
    "s3://my-bucket/lake",
    format_options={
        "polars_storage_options": {"aws_region": "eu-west-1"},
        "pandas_storage_options": {"anon": False},
        "polars_delta_use_pyarrow": True,
    },
)
```

| Option                         | Formats                                         | Meaning                                                                                                               |
|--------------------------------|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| `polars_storage_options`       | `delta`, `parquet`, `csv`, `json-lines`         | polars' `storage_options` (object_store's names, not fsspec's), in the preview, the data browser and the polars code. |
| `polars_credential_provider`   | `delta`, `parquet`, `csv`, `json-lines`         | polars' `credential_provider`, in the preview and the data browser (not in the code).                                 |
| `pandas_storage_options`       | `parquet`, `csv`, `json-lines`, `json`, `excel` | pandas' `storage_options` (fsspec's), in the pandas code.                                                             |
| `fsspec_storage_options`       | `json`, `excel`, `markdown`, `text`, `image`    | `fsspec.open`'s keyword arguments, in the code that reads with fsspec.                                                |
| `polars_delta_table_options`   | `delta`                                         | `scan_delta`'s `delta_table_options`.                                                                                 |
| `polars_delta_use_pyarrow`     | `delta`                                         | `scan_delta`'s `use_pyarrow`.                                                                                         |
| `polars_delta_pyarrow_options` | `delta`                                         | `scan_delta`'s `pyarrow_options`.                                                                                     |

Storage options that look like credentials are left out of the code. Without storage options, the code relies on the
environment's configuration and credentials.

## Working with the explorer from Python

```python
explorer = StorageExplorer("s3://my-bucket/data")
explorer
```

```python
explorer.folder_url  # the URL of the folder shown, e.g. to pass to polars or pandas
explorer.selected_urls  # the URLs of the selected items
```

## Options

| Option                              | Meaning                                                                                                                               |
|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
| `height="400px"`                    | Height of the tree and list; the preview panel adds its own space below. `"fill"`: as tall as the container, e.g. the launcher's tab. |
| `max_upload_bytes=10 * 1024 * 1024` | Largest file you can upload from your computer (default 10 MB; `None` for no limit).                                                  |
| `filesystems={...}`                 | Several named storages, as shown above.                                                                                               |
| `query_settings=...`                | Where the data browser keeps its settings; see [Query data](#query-data).                                                             |
| `format_options={...}`              | Options for the formats (see [Format options](#format-options)). Also for `DataExplorer`.                                             |

## Requirements

- Python 3.11 or later
- JupyterLab 4. The widgets also show in other Jupyter frontends, but dragging files from JupyterLab's file browser
  works only in JupyterLab.
- For Query data and `DataExplorer`: a recent polars (tested with polars 2.0).

## Development

Building from source, the JupyterLab extension, releasing, and customizing the explorer's icons with CSS:
see [DEVELOPMENT.md](DEVELOPMENT.md).
