Metadata-Version: 2.4
Name: scrollscout
Version: 0.3.1
Summary: Scout Vesuvius Challenge scrolls: stream and preview Herculaneum CT volumes from a laptop without downloading terabytes or tripping rate limits.
Author: Alexandre Dufour-Richard
License: MIT
Project-URL: Homepage, https://github.com/vonduffen/scrollscout
Project-URL: Repository, https://github.com/vonduffen/scrollscout
Project-URL: Issues, https://github.com/vonduffen/scrollscout/issues
Project-URL: Vesuvius Challenge, https://scrollprize.org
Keywords: vesuvius-challenge,herculaneum,ome-zarr,ct,papyrology
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: zarr<3,>=2.14
Requires-Dist: numpy>=1.23
Requires-Dist: fsspec>=2023.1
Requires-Dist: aiohttp>=3.8
Requires-Dist: requests>=2.28
Requires-Dist: Pillow>=9.0
Requires-Dist: tifffile>=2023.1
Provides-Extra: catalog
Requires-Dist: vesuvius>=0.1.9; extra == "catalog"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# scrollscout

**Scout [Vesuvius Challenge](https://scrollprize.org) scrolls from a laptop — stream and preview the Herculaneum CT volumes without downloading terabytes or tripping the data server's rate limiter.**

The scroll volumes are enormous. Scroll 1 alone is **14376 × 7888 × 8096 voxels ≈ 918 GB** at full resolution. If you're on a laptop with limited disk, you can't (and shouldn't) download them to take a look. The data *is* streamable as multiscale OME-Zarr — but two things bite newcomers immediately:

1. **You don't know which resolution level to read.** Ask for a full-res cross-section and you'll try to pull hundreds of GB.
2. **Naive reads get rate-limited.** A large `array[z, y0:y1, x0:x1]` makes zarr fire one HTTP request per chunk, all at once. The server replies `429 Too Many Requests` and your read dies:

   ```
   aiohttp.client_exceptions.ClientResponseError: 429, message='Too Many Requests',
   url='https://dl.ash2txt.org/.../54keV_7.91um_Scroll1A.zarr/0/56/19/26'
   ```

`scrollscout` handles both for you: it **auto-selects the coarsest pyramid level** that satisfies your requested pixel width, and reads it in **small, chunk-aligned tiles with exponential backoff on 429**, so a read *slows down* under pressure instead of failing.

![Scroll 1 mid cross-section](https://raw.githubusercontent.com/vonduffen/scrollscout/main/examples/scroll1_mid.png)

*Scroll 1 (PHerc. Paris 4), middle cross-section — streamed with `scrollscout thumb 1 -w 1000`. Total transfer: a few MB, not 918 GB.*

## Install

```bash
pip install scrollscout
# optional: use the live scroll catalogue from the official `vesuvius` package
pip install "scrollscout[catalog]"
```

You must accept the Vesuvius Challenge [data license](https://scrollprize.org/data) before accessing scroll data. scrollscout does not redistribute any scroll data; it streams the public server on your behalf.

## Command line

```bash
# list a scroll's multiscale pyramid — see the sizes before you read anything
scrollscout info 1

# stream a middle cross-section to a PNG (auto-picks the pyramid level)
scrollscout thumb 1 --width 1000 -o scroll1.png

# slice along a different axis: z (default, circular cross-section),
# y or x (longitudinal views down the scroll's length)
scrollscout thumb 1 --axis y --width 700 -o scroll1_length.png

# a shallower slice of scroll 2, larger
scrollscout thumb 2 --frac 0.25 --width 1200 -o scroll2_quarter.png

# scrub through slices interactively in your browser (see below)
scrollscout view
```

### Interactive viewer

`scrollscout view` launches a local web app (Python stdlib only, nothing stored
on disk) where you pick a scroll, choose the `z`/`y`/`x` axis, set a target
width, and **drag a slider to scrub through cross-sections**. Each slice is
streamed on demand through the same throttle-safe reader and cached in memory,
so scrubbing back over visited slices is instant.

```bash
scrollscout view                      # opens http://127.0.0.1:8000
scrollscout view --port 9000 --no-browser
```

Uncached slices take a few seconds to stream at a rate-limit-safe resolution;
revisited slices are instant. It's a previewer, not a real-time volume renderer.

### Ink labeler

`scrollscout label <segment_dir>` opens a browser tool for **hand-labeling ink**
on a segment's flattened surface — the ground-truth masks that ink-detection
models train on. Get a segment and label it in two commands:

```bash
# 1. stream a segment's layers into a ready-to-label folder
scrollscout fetch-segment --scroll 1 --segment 20230530164535

# 2. open the labeler on it
scrollscout label segments/20230530164535      # opens http://127.0.0.1:8010
```

`fetch-segment` streams the mask + a central band of layers (`--layers lo-hi`
or `all`). Add `--crop Y0 Y1 X0 X1` to grab just a window of a large segment —
it byte-range-reads only the needed rows, so cropping a multi-GB segment is
cheap. It's resumable (existing layers are skipped).

- **Scrub** the layer stack (slider or ←/→) to find ink at the surface depth
- **Brightness / contrast** sliders adjust the view live; layers are
  percentile-stretched server-side for visibility
- **Paint** over ink with the brush (adjustable size, erase, undo, clear)
- **Save** writes `<segment_id>_inklabels.png` — a binary mask at the layer's
  native resolution, the format training pipelines expect

Stdlib-only and fully local; nothing is uploaded. Existing labels reload on
reopen, so you can label across sessions.

## From preview to ink detection (free GPU)

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vonduffen/scrollscout/blob/main/notebooks/ink_detection_colab.ipynb)

Once you've scouted a scroll, the companion notebook
[`notebooks/ink_detection_colab.ipynb`](notebooks/ink_detection_colab.ipynb)
takes the next step: it streams a Scroll 1 *segment* and runs the open-source
Grand Prize TimeSformer **ink-detection** model on it, using a **free Colab/Kaggle
GPU** — no local GPU required. It's the on-ramp from previewing data on a laptop
to running the ML that actually reads the scrolls.

Note: the notebook runs on Scroll 1 (already-public text). Text recovered from an
unread scroll is a private prize submission under the data terms — don't publish
it without Vesuvius Challenge approval.

`scrollscout info 1` prints:

```
Scroll 1
  pyramid levels (6):
    level             shape (z,y,x)          chunks         scale  approx full size
        0       (14376, 7888, 8096)  (128, 128, 128)  (1.0, 1.0, 1.0)    918.07 GB
        1        (7188, 3944, 4048)  (128, 128, 128)  (2.0, 2.0, 2.0)    114.76 GB
        2        (3594, 1972, 2024)  (128, 128, 128)  (4.0, 4.0, 4.0)     14.34 GB
        3         (1797, 986, 1012)  (128, 128, 128)  (8.0, 8.0, 8.0)      1.79 GB
        4           (899, 493, 506)  (128, 128, 128)  (16.0,16.0,16.0)     0.22 GB
        5           (450, 247, 253)  (128, 128, 128)  (32.0,32.0,32.0)     0.03 GB
```

## Python API

```python
import scrollscout as ss

vol = ss.open_scroll(1)            # streams only ~KB of metadata
print(vol.level_for_width(1000))   # -> Level(path='3', ...)

# get a cross-section as a numpy array (throttle-safe under the hood)
arr = ss.thumbnail(1, axis="z", frac=0.5, width=1000, out="scroll1.png", verbose=True)
print(arr.shape, arr.dtype)        # (986, 1012) uint8

# a longitudinal slice down the scroll's length
arr_y = ss.thumbnail(1, axis="y", frac=0.5, width=700)

# launch the interactive viewer from Python
ss.serve(port=8000)
```

## How the throttle-safe read works

- `level_for_width(target_px)` picks the **coarsest** pyramid level whose in-plane
  width still meets your target. A 1000 px view comes from level 3 (~64 chunks),
  never level 0 (~4000 chunks). Under-fetching is the first line of defense.
- `read_slice()` walks the chosen slice in `tile`-sized blocks with a short pause
  between tiles, and retries each tile with exponential backoff (`base_delay * 2**n`)
  when the server returns 429. Non-rate-limit errors propagate immediately.

## Scope & limitations

- Reads scroll **volumes** (the `volumes_zarr*` OME-Zarr). Segment/surface
  meshes and ink predictions are out of scope for v0.1.
- Preview/QA tool, not an analysis pipeline — it gets you *looking* at the data
  fast so you can decide where to work.
- Slices along any of the `z`/`y`/`x` axes. Oblique/arbitrary planes are not
  supported.

## License

MIT (see `LICENSE`). Scroll data itself is governed by the Vesuvius Challenge /
EduceLab-Scrolls terms and is **not** included or redistributed here.

If you use scroll data in published work, cite EduceLab-Scrolls: Parsons, S.,
Parker, C. S., Chapman, C., Hayashida, M., & Seales, W. B. (2023).
*EduceLab-Scrolls: Verifiable Recovery of Text from Herculaneum Papyri using
X-ray CT.* arXiv:2304.02084.
