Metadata-Version: 2.4
Name: s3view
Version: 0.1.0
Summary: Fast, lightweight S3 browser with true video streaming and remote FITS/ASDF previews
Author-email: Luke Bouma <bouma.luke@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/lgbouma/s3view
Project-URL: Repository, https://github.com/lgbouma/s3view
Project-URL: Issues, https://github.com/lgbouma/s3view/issues
Keywords: s3,astronomy,fits,asdf,quicklook,streaming,browser
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: System :: Filesystems
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: botocore>=1.29
Provides-Extra: images
Requires-Dist: pillow; extra == "images"
Provides-Extra: fits
Requires-Dist: astropy; extra == "fits"
Requires-Dist: numpy; extra == "fits"
Provides-Extra: asdf
Requires-Dist: pyyaml; extra == "asdf"
Requires-Dist: numpy; extra == "asdf"
Provides-Extra: all
Requires-Dist: pillow; extra == "all"
Requires-Dist: astropy; extra == "all"
Requires-Dist: numpy; extra == "all"
Requires-Dist: pyyaml; extra == "all"
Requires-Dist: matplotlib; extra == "all"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pillow; extra == "test"
Requires-Dist: astropy; extra == "test"
Requires-Dist: numpy; extra == "test"
Requires-Dist: pyyaml; extra == "test"
Dynamic: license-file

# s3view

[![tests](https://github.com/lgbouma/s3view/actions/workflows/ci.yml/badge.svg)](https://github.com/lgbouma/s3view/actions/workflows/ci.yml)
[![python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)

A fast, lightweight S3 browser for astronomy. It behaves like a file
manager — click through prefixes, preview things, hit space bar — but it
**streams instead of downloading**.

Compatible with: FITS, ASDF, movies, standard image formats, PDFs, standard
text formats.

Run from command line:
```
s3view                          # open your default location
s3view s3://bucket/prefix/      # open somewhere specific
s3view --set-start s3://bucket/prefix/    # remember that as the default
```

It starts a local server, opens your browser and prints a URL. `Ctrl-C` quits.
On first run, with nothing configured, it lists your buckets and lets you pick.

![s3view browsing a night of pipeline products: a 191 MB FITS frame previewed from ~8 MB of ranged reads, an ASDF array, gallery thumbnails, and a 900-object prefix](https://raw.githubusercontent.com/lgbouma/s3view/main/docs/demo.gif)

## Installation

```bash
uv tool install 's3view[all]'
```

One command, no clone, and nothing added to the environments you work in.
`[all]` brings the preview stack with it — astropy, numpy, Pillow, matplotlib,
PyYAML — so every feature is on from the first run. Python 3.10+; no boto3, no
web framework, no build step, no `npm install`.

To try it without installing anything at all:

```bash
uvx 's3view[all]' s3://bucket/prefix/
```

Or use whichever packaging tool you already have:

```bash
pipx install 's3view[all]'
pip install 's3view[all]'
```

Leave the `[all]` off for a minimal install. `botocore` is the only hard
dependency; everything the extras would have added degrades gracefully, and
s3view tells you at launch what is live (`thumbnails:on  fits:on`). The one
combination to avoid is an isolated install *without* the extras: a `pipx` or
`uv tool` environment cannot see the astropy and Pillow in your normal
environment, so FITS, ASDF and thumbnails quietly switch off.

### Running from a checkout

For development, or if you would rather s3view used the scientific stack you
have already built than carry its own copy of it:

```bash
git clone git@github.com:lgbouma/s3view.git /my/preferred/dir/s3view
echo 'export PATH="/my/preferred/dir/s3view/bin:$PATH"' >> ~/.bashrc   # or ~/.zshrc
exec $SHELL
```

`bin/s3view` runs the checkout under whichever `python3` is first on your PATH.
That is the point — it inherits your stack — but it also means that environment
needs `botocore`, and a per-project virtualenv frequently does not even when
your usual environment does. If the wrong interpreter wins, name the right one:

```bash
export S3VIEW_PYTHON="$HOME/envs/astro/bin/python"
```

`pip install -e /my/preferred/dir/s3view` is the same idea with a console
script instead of a PATH entry, and edits still take effect immediately.

### Optional dependencies

Everything below degrades gracefully, and the launch banner says which of them
are live.

| package | enables |
|---|---|
| `pillow` | image thumbnails, and previews of formats browsers cannot decode |
| `numpy` | any array preview at all |
| `astropy` | FITS previews (zscale in particular) |
| `pyyaml` | ASDF previews |
| `matplotlib` | colormaps beyond grayscale |

### Credentials and endpoints

Standard botocore resolution: environment variables, `~/.aws/credentials`,
`AWS_PROFILE`, SSO, instance roles. Override per run with `--profile` and
`--region`. Non-AWS S3-compatible stores work via `--endpoint-url` (MinIO,
Ceph, Cloudflare R2, Wasabi).

**Buckets you can read but do not own.** S3's `ListBuckets` returns only the
buckets owned by the calling account. A bucket shared with you cross-account
through a bucket policy is fully readable yet never appears in that list — a
common arrangement for shared project data. s3view therefore adds any bucket it
knows you can reach (bookmarked, or the one you are in) to the sidebar and marks
it with a `·`. To reach one for the first time, either pass it on the command
line or press `⌘L` and type the `s3://` path, then bookmark it with ☆.

## What it previews

| type | how |
|---|---|
| **FITS** (`.fits`, `.fit`, `.fts`, `.fz`) | strided ranged reads; HDU picker, stretch, colormap, resolution, full header text |
| **ASDF** (`.asdf`) | same, driven by the YAML tree and block index; pick any named array (`roman.data`, `roman.err`, `roman.dq`, …) and read the tree |
| mp4 / mov / webm / m4v | streamed from S3 by range request, with a running "~X MB transferred" readout |
| png / jpg / gif / webp / tif | presigned direct load; server-side thumbnail for formats the browser cannot decode |
| txt / json / yaml / cfg / param / log / py / csv | first 256 KB via one ranged read |
| pdf | presigned, in an iframe |
| anything else | metadata plus a download link |

Array previews share one interface: choose the HDU or array, a stretch
(`zscale`, `asinh`, `log`, `99.5%`, `minmax`), a colormap, and a resolution from
256 to 1024 px. The footer always reports what it actually read — e.g.
`2.1 s · read ~8.4 MB of 191 MB (4.2%)` — so the cost is never hidden from you.

Containers the browser cannot decode (mkv, avi) offer **Open in player**, which
hands the presigned URL to IINA or VLC — still streaming, never downloading.

## Configuration

`~/.config/s3view/config.json`, written on first run. Set `start` to open
somewhere by default:

```json
{
  "start": "s3://your-bucket/your/prefix/",
  "bookmarks": [
    {"name": "your-prefix", "uri": "s3://your-bucket/your/prefix/"}
  ],
  "profile": null,
  "region": null,
  "endpoint_url": null,
  "page_size": 1000,
  "presign_expires": 3600,
  "external_player": "IINA"
}
```

`start` accepts any `s3://bucket/prefix/`; leave it `null` for the bucket picker.
`s3view --set-start s3://bucket/prefix/` writes it for you, and ☆ manages
bookmarks. No bucket names are baked into the source.

## Security

The server binds `127.0.0.1` only, and every API call requires a token generated
fresh at startup and carried in the URL it opens. Without this, any web page you
happened to have open could quietly read your buckets through localhost.
Requests arriving with a foreign `Origin` header are rejected.

Presigned URLs default to one-hour expiry and are minted only for objects you
actually open. **Copy URL** puts one on your clipboard deliberately — treat it
as a password for that object until it expires.

## Development

```bash
pip install -e ".[test]"
pytest
```

The suite needs **no AWS credentials and no network**. It builds synthetic FITS
and ASDF files in memory and serves them through a fake S3 that records every
ranged read, so the tests can assert on *how much* was fetched — the property
the whole program exists to protect. Both the contiguous and strided read paths
are exercised and checked against each other for identical pixels.

CI runs on Python 3.10–3.13 on Linux plus macOS, and a separate job installs
*only* botocore to prove the optional dependencies really do degrade gracefully
rather than crashing.

### Releasing

Version lives in one place, `s3view/__init__.py`; `pyproject.toml` reads it
from there. To cut a release, bump it, commit, then:

```bash
git tag v0.1.1 && git push origin v0.1.1
```

The `release` workflow builds the sdist and wheel, refuses the tag if it
disagrees with `s3view.__version__`, installs the wheel into a clean
environment and starts the CLI from it, then uploads to PyPI through [trusted
publishing](https://docs.pypi.org/trusted-publishers/) — there is no API token
anywhere in the repository or its secrets. Running the workflow by hand
(`workflow_dispatch`) does everything except the upload, which is the way to
rehearse a release: PyPI never allows a version number to be reused, even after
the file is deleted.

To re-record the README animation (needs `playwright` and `ffmpeg`, neither of
them a runtime dependency):

```bash
pip install playwright && playwright install chromium
python tools/record_demo.py            # -> docs/demo.gif
```

## Why it is fast

**Video and audio never pass through this program.** The page is handed a
short-lived presigned S3 URL and the browser's own media stack range-requests it
directly, so seeking is cheap no matter how large the file. Measured from inside
the browser against a 731 MB mp4, on a ~2.4 MB/s link:

| request | result |
|---|---|
| first 64 KB | `206`, 717 ms |
| 64 KB from the **middle** | `206`, 321 ms |
| 64 KB from the **end** | `206`, 251 ms |

**Array images are read by byte range, not downloaded.** Both FITS and ASDF are
self-describing: a small ranged read of the metadata is enough to compute the
exact byte offset of every row of every array. s3view then fetches only the rows
the preview needs, in parallel, and bins the columns it already has in memory.
For a 200 MB, 4088×4088 float32 detector frame:

| preview | bytes read | time |
|---|---|---|
| 256 px | 4.2 MB (2.2%) | ~1.3 s |
| 512 px | 8.4 MB (4.2%) | ~2.1 s |
| 1024 px | 16.7 MB (8.7%) | ~3.8 s |
| downloading it instead | 200 MB | ~90 s |

Ranged reads bypass botocore's request machinery: the object is presigned once
and every range is pulled over a pooled HTTPS connection, because signing 512
separate requests costs more CPU than the transfer costs bandwidth.

**Listings are paginated and virtualized.** A prefix of 1000 objects renders
about 60 DOM nodes; the next page is prefetched in the background while you read
the current one. A prefix holding 187 GB across 1000 files opens as fast as an
empty one.

**Thumbnails are lazy and rate-limited.** Only tiles actually on screen are
requested, three at a time, cached on disk under `~/.cache/s3view`. Array
thumbnails in gallery view cost megabytes each, so they sit behind a toggle.

## Known limitations

- Array previews sample rows rather than averaging them vertically, so a
  decimated view of a crowded field aliases. Columns *are* averaged. Raise the
  resolution to sample more rows.
- ASDF blocks compressed with `lz4` or `blosc` cannot be read by range and are
  not supported; uncompressed, `zlib` and `bzip2` blocks are.
- `GetBucketLocation` is frequently denied on cross-account buckets, so the
  region is taken from the `x-amz-bucket-region` header on HeadBucket instead.
  If both are denied, s3view falls back to your configured default region.
- Chrome will not load video in a hidden or background tab; if a movie sits on a
  spinner, bring the window to the front.
- Read-only. There is no upload, rename, or delete.
