Metadata-Version: 2.5
Name: pixel-clarity
Version: 0.4.0
Summary: Pixel Clarity in Jupyter: a fast viewer for large astronomical images, mosaics, cubes and tables, with the data kept on the server
Project-URL: Homepage, https://clarityorbital.com
Author: Clarity Orbital
License-Expression: MIT
License-File: LICENSE
License-File: src/pixel_clarity/static/THIRD_PARTY_NOTICES.md
Keywords: asdf,astronomy,fits,image viewer,jupyter,parquet,table viewer,widget
Classifier: Framework :: Jupyter
Classifier: Framework :: Jupyter :: JupyterLab
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Python: >=3.10
Requires-Dist: anywidget>=0.9
Requires-Dist: nodejs-wheel-binaries>=20
Requires-Dist: numpy
Provides-Extra: regions
Requires-Dist: regions; extra == 'regions'
Provides-Extra: s3
Requires-Dist: boto3; extra == 's3'
Description-Content-Type: text/markdown

# Pixel Clarity for Jupyter

A fast viewer for large astronomical images, mosaics, cubes and tables, in a notebook. It is the
[Pixel Clarity](https://clarityorbital.com) VS Code viewer as a Jupyter widget: the same stretch,
colors, regions and statistics, WCS readout, mosaics placed on the sky, RGB, stacks, data quality
masks, magnitudes, full-resolution PNG export, shared pages, the Aladin Lite sky panel, and the
table viewer for catalogs.

The data stays on the server. A process next to the kernel reads the files there (local paths, or
S3 objects and URLs by range requests) and the browser receives only the tiles, rows and values it
shows, so a 4k × 4k detector, a mosaic of many detectors or a catalog of millions of rows stays
quick to look at over an ordinary internet connection.

```python
import pixel_clarity as pc

pc.show("image.asdf")                                      # a file on the server
pc.show("s3://bucket/path/image.asdf")                     # read where it is, no download
pc.show(model)                                             # a data model, HDU, NDData or array in the kernel
pc.mosaic("exposure/")                                     # a folder of detector files, placed on the sky
pc.mosaic(files, color=True)                               # two or more filters as colors
pc.rgb("red.fits", "green.fits", "blue.fits")
pc.stack(sorted(glob("frames/*.fits")))                    # frames one at a time, with a slider
pc.blink(["epoch1.fits", "epoch2.fits"])                  # blink between images, lined up on the sky
pc.show("image.fits", cmap="viridis", scale="asinh", limits="99.5", height=700)
pc.table("catalog.parquet")                                # a table: sorted, filtered and plotted on the server
pc.popup(array)                                            # in a browser tab of its own, from any Python program
```

## Install

```
%pip install pixel-clarity
```

Restart the kernel afterwards (or use a new one). It needs JupyterLab 4 or Notebook 7 with
ipywidgets 8, which most hubs have. On a JupyterHub where packages installed in the kernel's
environment don't survive a server restart, `%pip install --user pixel-clarity` puts it in your
home directory instead.

It brings its own Node.js (`nodejs-wheel-binaries`) for the process that reads the data; set
`PIXEL_CLARITY_NODE` to use another (version 20 or later).

## How it connects

The viewer reaches its process the quickest way the page allows, shown by the dot under it:

- **Green, direct:** through [jupyter-server-proxy](https://github.com/jupyterhub/jupyter-server-proxy)
  (installed on many hubs), or straight to the server when the browser runs on the same machine.
  Tiles never wait for the kernel, so the view keeps working while a cell runs. *Open in a new tab*
  shows the same view in a browser tab of its own.
- **Amber, through the kernel:** widget messages relayed by the kernel. It works anywhere widgets
  do, but the view waits while a cell runs. `pc.show(..., connection="kernel")` asks for this.

## What it reads

- Paths: FITS (including tile-compressed `.fits.fz`), ASDF (including lz4 blocks and gwcs), HDF5,
  NumPy `.npy`/`.npz`, TIFF and PNG. A folder or a glob pattern shows its image files; a
  `.pcmosaic` file (saved from a mosaic's Detectors panel) opens that mosaic again.
- `s3://` URIs: public buckets anonymously; `anon=False` signs the URL with the kernel's AWS
  credentials (`pip install 'pixel-clarity[s3]'`). An open `s3fs` file works too.
- `http(s)://` URLs to servers that answer Range requests.
- HiPS (Hierarchical Progressive Surveys), however large: a HiPS's folder on the server (the one
  holding `properties`), or its URL (one whose last part has no file extension). Only the tiles
  in view are read.
- From the kernel: NumPy arrays (masked arrays as NaN), astropy HDUs, HDU lists and NDData/CCDData
  (with their WCS), ASDF files and data models. These are written once to a temporary file
  on the server's local disk (an unchanged array is written only once), removed when the kernel
  stops. Data that is a file's already is read from that file instead: an array memory-mapped
  from a `.npy` file (`np.load(path, mmap_mode="r")`), or an HDU of a FITS file opened with
  `fits.open`, as long as neither its data nor its header has been touched.

Only the images shown are read, not whole files. From a URL, each image shown is downloaded once:
the viewer keeps a summary of it (a reduced copy and its statistics), so showing it again (in
another viewer, in a mosaic, or after the viewer slept while idle) reads only the rows under the
view at full resolution. Summaries stay with the kernel's own viewer process, readable only by
you, and go when the kernel stops; one is used only after the URL has been read successfully
again, so nothing is shown from a file you can no longer read.

## Settings

Keywords of `show` (and `Viewer`), named after the VS Code extension's settings; `pc.settings()`
lists them with their values and defaults. Set them for every viewer with
`pc.defaults["cmap"] = "viridis"`.

| Keyword | Values |
|---|---|
| `scale` | `linear`, `log`, `pow`, `sqrt`, `squared`, `asinh`, `sinh`, `histeq`, `mtf` |
| `limits` | `minmax`, `zscale`, `99.5`, `99`, `98`, `95`, `90` |
| `cmap` | `gray`, `viridis`, `magma`, `inferno`, `heat`, `cool`, … |
| `invert`, `auto` | `True`/`False` (`auto`: Auto enhance on opening) |
| `zoom` | `fit` or `1` |
| `sidebar`, `cuts` | show the side panel, the cut plots (`cuts` is off by default in a notebook) |
| `mosaic_layout`, `match_sky`, `flux_calibrate`, `feather`, `trim_edges` | mosaic placement and level matching |
| `online`, `survey` | SIMBAD and Aladin Lite (CDS) from the browser |

Other keywords: `height` (pixels; the grip under the viewer changes it), `kind` (`single`,
`mosaic`, `stack`, `rgb`), `title`, `names`, `layout`, `active`, `color`, `blink`, `groups`, `levels`,
`state` (a view saved from `viewer.state`), `out_dir` (where exports, shared pages and saved
mosaics are written; the kernel's working directory by default).

## In the viewer

Everything in the VS Code viewer's toolbar and panels works here. Files it writes (a
full-resolution export, a shared page, a saved mosaic, a HiPS built with right-click › *Build
HiPS…*) are written on the server, next to the notebook, and named in the line under the viewer.
Where VS Code asks, these use its defaults: *Build HiPS…* writes the values at the order nearest
the data's pixels (and, for an RGB set or color mosaic, the colors as shown too). A shared page
of images read from URLs carries their preview or compact data; asked to link to the data, it
links a URL only when anyone may read it for good (https, not signed, not on this machine). *Save view as shown*, region files and color
settings download to your computer. Live reload follows changes to local files.

A viewer nobody has used for five minutes (thirty if it is on screen) frees the memory its data
took; it reads the data again when you come back to it. Large local images keep their overview,
statistics and zoomed-out tiles in the viewer process's scratch folder while the kernel runs, so
opening one again (or waking its viewer) doesn't read the whole file again: at most
`PIXEL_CLARITY_CACHE_MB` (2000 by default; 0 turns it off) and a quarter of the free disk.

## Regions from and to Python

```python
v = pc.show("image.fits")
v.add_regions(regions)          # `regions` objects (pixel or sky), a DS9 .reg file's path, or DS9 text
v.regions                       # what is drawn on the view now, as `regions` pixel regions
v.regions[0].to_sky(wcs)
print(v.regions_ds9)            # the same as DS9 region text (image coordinates), no extra package needed
```

`v.regions` needs the `regions` package (`pip install 'pixel-clarity[regions]'`). It reads what any
viewer of `v` last saved, including one opened in a tab of its own, even while a cell runs.
Coordinates are the image's pixels (a stack's frames, an RGB set's red image); for a mosaic, those
of the mosaic's own plane. Regions given in sky coordinates are placed with the image's WCS; a
viewer still opening gets them once its image is shown. More than 2000 shapes show as a catalog
overlay rather than editable regions.

## Tables

```python
pc.table("catalog.parquet")            # CSV, TSV, ECSV (also .gz) or Parquet: a path, an s3:// URI or a URL
pc.table(t)                            # an astropy, pandas or polars table (or a structured array) in the kernel
pc.show("catalog.csv")                 # show() opens table files in the table viewer too
pc.table("catalog.csv", plot=True)     # with the plot showing
```

A table stays on the server as an image does: sorting, filtering (`mag < 20 and class == 'STAR'`),
computed columns, column statistics and plots are worked out there, and the page receives the
rows on screen. A Parquet file that would take more than a quarter of the server's memory asks
which columns to load. Tables from the kernel are written to a temporary file: Parquet when
pyarrow is installed (units, masks and times kept), else ECSV (astropy) or CSV (pandas).

*Export* writes CSV, TSV, ECSV or a region file next to the notebook. *Overlay on image* draws the
rows on the image viewer used last, named in the line under the table. `t.state` holds the
filters, sort, columns and plot; pass it as `state=` to open another the same way.

## Outside a notebook

`pc.popup()` shows the same things in a browser tab of its own, from any Python program: a script,
IPython, or a notebook (where it gives a link to the tab).

```python
import numpy as np
import pixel_clarity as pc

for step in range(100):
    image = simulate(step)
    pc.popup(image, title="simulation")          # the same tab, updated in place

pc.popup(catalog, window="stars")                # a second tab
```

Called again with the same `window` (default `"main"`), it shows the new data in the same tab. An
array of the same size and type is read again in place, keeping the zoom, pan and colors,
so a loop can watch it change; anything else starts the tab afresh. The window it returns has
`state`, `regions` and `add_regions()` as a notebook viewer does.

Everything stays with the program: its viewer process and files go when it ends. A script waits
at its end until its tabs are closed (or Ctrl-C), so they stay up to be looked at; `pc.wait()`
waits sooner, and `PIXEL_CLARITY_POPUP_WAIT=0` doesn't wait. The page is served on this machine
only (127.0.0.1): over SSH it prints the address to forward (`ssh -L port:127.0.0.1:port`), and
`PIXEL_CLARITY_PORT` keeps the port the same from run to run.

## Not yet

- Opening files from JupyterLab's file browser: for now, `pc.show()` them from a cell.

## Development

The viewer's sources are the VS Code extension's repository; `npm run build` bundles them, with
this package's own parts from `js/`, into `src/pixel_clarity/static/`.

```
npm install
PC_SRC=../pixel-clarity npm run build    # PC_SRC defaults to ../pixel-clarity
npm run typecheck
uv sync
uv run pytest
node scripts/e2e.mjs [--remote=<url>] [--mosaic=<folder>] [--dark] [--aladin] [--no-proxy]   # JupyterLab in headless Chromium
uv build                                       # the wheel, with the built scripts in it
```

To release, set the version in `src/pixel_clarity/_version.py`, commit, and push a tag `v<version>`:
`.github/workflows/pypi.yml` builds against the extension's sources at `UPSTREAM_REF`, runs the
tests and publishes to PyPI.

`notebooks/local_try.ipynb` walks through the viewer by hand on your own machine, with test
images it makes itself.

MIT licensed. The bundled third-party code is listed in `static/THIRD_PARTY_NOTICES.md`; Aladin
Lite (LGPL) is loaded from CDS when its panel opens, not bundled.
