Metadata-Version: 2.5
Name: jupyter-storage-explorer
Version: 0.12.0
Summary: Explore your data lake without leaving Jupyter: built for data professionals
Project-URL: Homepage, https://github.com/sugibuchi/jupyter-storage-explorer
Project-URL: Source, https://github.com/sugibuchi/jupyter-storage-explorer
Project-URL: Issues, https://github.com/sugibuchi/jupyter-storage-explorer/issues
Project-URL: Changelog, https://github.com/sugibuchi/jupyter-storage-explorer/releases
Author: Tsuyoshi SUGIBUCHI
License-Expression: MIT
License-File: LICENSE
Keywords: anywidget,azure,data lake,delta lake,file browser,fsspec,gcs,jupyter,jupyterlab,parquet,s3
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Jupyter
Classifier: Framework :: Jupyter :: JupyterLab
Classifier: Framework :: Jupyter :: JupyterLab :: 4
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Filesystems
Requires-Python: >=3.11
Requires-Dist: anywidget>=0.9
Requires-Dist: fsspec>=2023.1.0
Requires-Dist: narwhals>=2.20
Requires-Dist: pygments>=2.12
Requires-Dist: traitlets>=5.4
Provides-Extra: abfs
Requires-Dist: adlfs>=2022.2.0; extra == 'abfs'
Requires-Dist: azure-storage-file-datalake>=12.0.0; extra == 'abfs'
Description-Content-Type: text/markdown

# Jupyter Storage Explorer

*Explore your data lake without leaving Jupyter: built for data professionals*

[![PyPI - Version](https://img.shields.io/pypi/v/jupyter-storage-explorer?color=blue)](https://pypi.org/project/jupyter-storage-explorer/)
[![codecov](https://codecov.io/gh/sugibuchi/jupyter-storage-explorer/graph/badge.svg?token=TFA12UL1IQ)](https://codecov.io/gh/sugibuchi/jupyter-storage-explorer)
[![Try it on Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/sugibuchi/jupyter-storage-explorer/main?urlpath=lab/tree/binder/demo.ipynb)

**`StorageExplorer`** is a file explorer for cloud storage and data lakes, right inside your Jupyter notebook, in the
spirit of Azure Storage Explorer.

Browse Amazon S3, Azure Data Lake / Blob Storage, Google Cloud Storage, local disks and more; preview Parquet files,
Delta Lake tables, CSV, Excel, JSON and other formats; page through, sort and filter tables; get ready-to-run code that
loads the data into your notebook; and upload, download, move and edit files without leaving JupyterLab. See
[Using the explorer](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/docs/storage-explorer.md#using-the-explorer) for all you can do with it.

![Storage Explorer in JupyterLab: a Delta Lake table queried, files filtered, and a Markdown file edited in JupyterLab's editor and previewed](https://raw.githubusercontent.com/sugibuchi/jupyter-storage-explorer/main/docs/figs/storage-explorer.gif)

This package also provides the **`DataExplorer`** widget, which lets you browse tabular data in your notebook: any
dataframe of pandas, polars, Spark, DuckDB and more. Page through, sort and filter it in the widget, then get the
result back in code. See [Using the data browser](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/docs/data-explorer.md#using-the-data-browser) for all you can do with it.

![DataExplorer in a notebook: a Parquet file paged through, columns hidden, filtered and sorted](https://raw.githubusercontent.com/sugibuchi/jupyter-storage-explorer/main/docs/figs/data-explorer.gif)

## Highlights

- **`StorageExplorer`**: any fsspec storage, several in one explorer; copy and paste between storages and notebooks.
- **`DataExplorer`**: page, sort and filter any dataframe; get the result back in code.
- **Previews** of Delta Lake, Parquet, CSV, JSON, Excel, text and images, with code to load them.
- **Background file operations** with progress and Cancel; folder operations only in Unsafe mode.
- **Work in JupyterLab**: open storage files in JupyterLab's own editors and viewers, and save them back; query tables
  in tabs of their own.
- **Bookmarks**, remembered queries, transfer rules, and a JupyterLab launcher item.

## Installation

```bash
pip install jupyter-storage-explorer
```

Install the package for your storage too:

| Storage                        | Package    | Address                                              |
|--------------------------------|------------|------------------------------------------------------|
| Amazon S3                      | `s3fs`     | `s3://bucket/path`                                   |
| Azure Data Lake / Blob Storage | `adlfs`    | `abfs://container@account.dfs.core.windows.net/path` |
| Google Cloud Storage           | `gcsfs`    | `gs://bucket/path`                                   |
| Local disk, in-memory          | (included) | `/path`, `memory://`                                 |

Previewing and querying data files needs a data library, e.g. `pip install polars deltalake`; the preview tells you
which.

## Quick start

**Try it without installing anything**: [open the demo on Binder](https://mybinder.org/v2/gh/sugibuchi/jupyter-storage-explorer/main?urlpath=lab/tree/binder/demo.ipynb), a made-up data lake in a notebook and in
the launcher's tab (it may take a minute to start).


### From the JupyterLab launcher

**[`jupyter-storage-explorer-launcher`](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/launcher/README.md)**
adds **Storage Explorer** to JupyterLab's launcher, opening the explorer in its own tab:

```bash
pip install jupyter-storage-explorer-launcher
```

![The Storage Explorer item in JupyterLab's launcher](https://raw.githubusercontent.com/sugibuchi/jupyter-storage-explorer/main/launcher/figs/launcher.png)

### From Python code

```python
from jupyter_storage_explorer import StorageExplorer

StorageExplorer("s3://my-bucket/data")
```

Options after the address go to the storage package, e.g. `profile="analytics"`. Double-click a file to preview it;
right-click for everything else.

Browse a dataframe and get the rows you filtered back:

```python
import polars as pl
from jupyter_storage_explorer import DataExplorer

explorer = DataExplorer(pl.scan_parquet("s3://my-bucket/lake/sales/*.parquet"))
explorer
```

```python
explorer.result().collect()  # after filtering and sorting in the widget
```

The `notebooks` folder has a demo of each widget on sample data.

## Documentation

- **[StorageExplorer](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/docs/storage-explorer.md)**:
  storages, using the explorer, transfer policy, formats, working with it from Python and options.
- **[DataExplorer](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/docs/data-explorer.md)**: the data
  browser, `result()`, `on_change()`, `QuerySettings` and options.
- **[The launcher](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/launcher/README.md)**: configuring
  its tab with `launcher.py` files.
- **[Development](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/DEVELOPMENT.md)**: building,
  releasing, and custom icons.

## Requirements

- Python 3.11 or later
- JupyterLab 4.0 or later (other Jupyter frontends show the widgets, without drag and drop from JupyterLab's file browser)
- For querying data: polars (tested with 2.0)

## Why Storage Explorer?

Storage Explorer aims to be a **data-aware file explorer for large data lakes**, made for the daily work of data
scientists and engineers. Rather than adding data lake access to JupyterLab's file browser, a narrow sidebar, it is a
widget in JupyterLab's main area, in a notebook or a tab of its own, with room for what working with data needs:

- **Built for large folders**: data lakes often hold thousands of items in one folder (partitions, part files, daily
  drops). The list is paged, sorted and filtered as you type, regular expressions included; folders of tens of
  thousands of items are listed by the kernel, a page at a time. Moving, copying, deleting and uploading run in the
  background, with their progress and Cancel, so the explorer stays usable while thousands of files move.
- **Preview as you browse**: tree, list and preview side by side. Click through files and see each one at once,
  without opening and closing tabs.
- **Tables, not just files**: Delta Lake tables and partitioned Parquet datasets are recognised as tables, not
  folders. Open them in the data browser to choose among hundreds of columns and filter and sort the data where it
  lives.
- **From browsing to code**: copy ready-to-run code to load a file or table with polars, pandas, PySpark and more, or
  the URLs of the files you selected, ready for any library. In Python, `folder_url` and `selected_urls` give the
  folder shown and the files selected.
- **Part of your notebook**: explorers are widgets, placed in your notebook next to your analysis. Your code can
  follow what you browse (`on_change()`, `on_query_change()`): a chart of the file you click, redrawn as you filter it
  in the data browser (`query_result()`). `DataExplorer` brings the same data browser to any dataframe, and gives the
  rows you filtered back to your code (`result()`). See
  [Working with the explorer from Python](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/docs/storage-explorer.md#working-with-the-explorer-from-python).
- **Safe by default**: in a data lake, a folder is often a whole table. Folders can't be moved, copied, renamed or
  deleted by a stray drag: that takes
  [Unsafe mode](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/docs/storage-explorer.md#unsafe-operations),
  which turns itself off after one operation. Teams can also restrict where files may go with a
  [transfer policy](https://github.com/sugibuchi/jupyter-storage-explorer/blob/main/docs/storage-explorer.md#transfer-policy).
- **Runs in your kernel**: with your credentials and your libraries, and no server setup.

It complements JupyterLab's file browser rather than replacing it: open files in JupyterLab's own editors, drag them
onto its file browser, or query a table in a tab of its own.
