Metadata-Version: 2.5
Name: parquetscope
Version: 0.4.0
Summary: Visualize DuckDB’s Parquet range reads, map them to file structure, and compare query runs.
Project-URL: Homepage, https://github.com/0then0/parquetscope
Project-URL: Documentation, https://github.com/0then0/parquetscope/blob/main/docs/README.md
Project-URL: Changelog, https://github.com/0then0/parquetscope/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/0then0/parquetscope/issues
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: duckdb,http,observability,parquet,range-requests
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Debuggers
Requires-Python: >=3.10
Requires-Dist: duckdb==1.5.6
Requires-Dist: fastapi>=0.135
Requires-Dist: pyarrow>=23
Requires-Dist: pydantic>=2.12
Requires-Dist: uvicorn>=0.41
Description-Content-Type: text/markdown

<p align="center"><img src="docs/icon.svg" alt="ParquetScope" width="112" height="112"></p>

# ParquetScope

See how DuckDB reads Parquet over HTTP: requests, byte ranges, column chunks, which files of a set it touches, and the effect of a different file layout.

[![Version](https://img.shields.io/badge/version-0.4.0-16775c)](https://github.com/0then0/parquetscope/releases/tag/v0.4.0)
[![Python](https://img.shields.io/badge/Python-3.10%2B-3776ab?logo=python&logoColor=white)](pyproject.toml)
[![DuckDB](https://img.shields.io/badge/DuckDB-1.5.6-fff000?logo=duckdb&logoColor=black)](pyproject.toml)
[![CI](https://github.com/0then0/parquetscope/actions/workflows/ci.yml/badge.svg)](https://github.com/0then0/parquetscope/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)

ParquetScope captures one SELECT against one local Parquet file, or against an explicit, bounded set of files, serves them through a controlled HTTP endpoint, and maps the observed reads to each file's physical structure. Compare queries or file layouts, inspect linked requests and chunks, and share a capture that others can open without the source data.

![Range lab showing real HTTP requests and the selected customer_id chunk](docs/range-inspector.png)

## Install and run

Install from [PyPI](https://pypi.org/project/parquetscope/) with [uv](https://docs.astral.sh/uv/) (the [release page](https://github.com/0then0/parquetscope/releases/tag/v0.4.0) also carries the wheel):

```sh
uv tool install --python 3.13 parquetscope
parquetscope setup
parquetscope serve --demo
```

Open **http://127.0.0.1:8400** and choose **Run layout demo**. The demo creates two equivalent Parquet layouts and measures the same query against both.

Python 3.10+ is required. The wheel includes the browser UI; Node.js is needed only for frontend development. `setup` downloads the signed httpfs extension for DuckDB 1.5.6 once. Subsequent local captures and offline viewing need no external internet, account, or cloud credentials.

To run from source instead:

```sh
uv sync --frozen
uv run parquetscope setup
uv run parquetscope serve --demo
```

## Inspect your data

```sh
parquetscope serve /path/to/events.parquet
```

Use the `input` view in the SQL workspace. For example, on a file containing `customer_id` and `amount_cents`:

```sql
SELECT customer_id, sum(amount_cents)
FROM input
WHERE customer_id BETWEEN 1000 AND 1099
GROUP BY customer_id
```

In **Range lab**, select a GET to see its response interval, server writes, and intersecting column chunks. Select a chunk under **Structure** to find related requests. **Zoom to selected range** expands the selected interval; **Whole file** restores context. **Compare runs** shows each run's query, input identity, settings, and measured totals.

The source file is read-only. Each capture uses a fresh worker and executes one fully consumed SELECT. ParquetScope is a local diagnostic tool for trusted files and SQL, rather than a sandbox for hostile workloads.

## Compare HTTP server profiles

Use **SQL workspace → HTTP profile** or explicit CLI options to compare fewer requests
against additional bytes under controlled server conditions:

```sh
parquetscope capture events.parquet --sql 'SELECT sum(amount_cents) FROM input' \
  --response-delay-ms 40 --body-bytes-per-second 2097152 --deadline 60 \
  --output shaped.parquetscope.json
```

The default is zero delay and unrestricted body writes. Delay applies once before each
prepared HEAD/GET response's headers. The body rate is shared by all GET responses and all
files in one capture. A token bucket permits `max(1, min(16384, ceil(rate / 10)))` burst bytes.
Low rates can reach the existing execution deadline. Profile changes apply to the next run.
Comparison and offline viewers retain requested/effective profiles, burst and timing evidence;
older artifacts explicitly have no recorded profile.

This models server waits and body writes, not transport RTT or a cloud provider. Successful
writes mean acceptance into the server response stream, not confirmed DuckDB receipt. See
the [measurement contract](docs/measurements.md#http-profiles) and the reproducible
[HTTP profile lab](docs/http-profile-lab/README.md).

## Inspect a set of files

List the files in a [manifest](docs/manifest.md) and give it to `serve` or `capture`:

```sh
parquetscope serve --manifest hive.manifest.json
parquetscope capture --manifest hive.manifest.json \
  --sql 'SELECT count(*) FROM input WHERE month = 3' --output set.parquetscope.json
```

The **Files** view lists every file by its location and partition segments with its requests, bytes and observed access (GET, HEAD only, or no observed requests). **Inspect** opens that file in Range lab. Offsets always belong to one file. **Compare runs** matches files by content hash, so two layouts of identical bytes line up row by row, and lists read options, versions and the limits of the comparison. Binding the view is part of the capture. Requests are labelled by the phase in which they started.

The manifest is explicit: no directory walking, globbing or discovery, a fixed budget (256 files, 16 GiB) and one schema per set. See the [partition-pruning lab](docs/pruning-lab/README.md) for a recorded flat versus Hive comparison of identical bytes.

## Save and share

```sh
parquetscope capture events.parquet \
  --sql 'SELECT sum(amount_cents) FROM input WHERE customer_id BETWEEN 1000 AND 1099' \
  --output run.parquetscope.json

parquetscope serve --artifact run.parquetscope.json
parquetscope viewer run.parquetscope.json --output run.viewer.html
```

A capture contains the query, file identity, layout, HTTP observations, and derived metrics. It omits result preview rows and the original source path. SQL, column names, and headers remain visible, as are the relative paths and partition values of a file set, so check the contents before sharing.

The standalone HTML viewer opens directly as a file, without Python, a server, or network access. It can import additional captures and compare them offline. The UI's **Export JSON** and **Offline viewer** buttons provide the same exports.

## Explore a real dataset

The [NYC TLC case study](docs/real-data/README.md) measures the same DECIMAL aggregate over 3,475,226 Yellow Taxi records in three layouts:

- **As delivered:** 59,158,238-byte file; 14 GET + 2 HEAD; 11,387,877 server-written body bytes.
- **Repacked in original order:** 53,370,794-byte file; 118 GET + 2 HEAD; 11,446,545 body bytes.
- **Sorted by pickup location:** 59,417,539-byte file; 10 GET + 2 HEAD; 795,241 body bytes.

All three repetitions of each layout produced those totals and identical query results. A smaller file needed more requests, while the sorted file needed fewer body bytes. These are observations for one workload, not guaranteed request sequences or proportional speedups.

The case study includes nine captures, three standalone viewers, a report, checksums, and reproduction instructions. To build its static interactive lab from the repository:

```sh
# Install the development environment first; see CONTRIBUTING.md.
pnpm --dir frontend build:lab
.venv/bin/python -m http.server 8418 --bind 127.0.0.1 --directory .runtime/pages
```

Open **http://127.0.0.1:8418/parquetscope/**. The lab browses recorded captures and runs no SQL. It loads neither the source dataset nor an execution backend.

## Read the measurements correctly

**Server-written bytes** count successful writes to the response stream. They do not establish how many bytes DuckDB received, decoded, or used. **Unique coverage** is the union of confirmed response intervals; **repeated coverage** is their summed length minus that union. Overlap is not automatically waste.

SQL success and capture completeness are separate. Incomplete traces retain available observations and mark totals as an observed subset. A fresh worker resets the client, but OS and disk caches remain uncontrolled. Loopback measurements do not measure S3 latency or cloud costs.

See [Understanding measurements](docs/measurements.md) for definitions and limits.

## Documentation

- [User guide](docs/usage.md): CLI workflows, the viewer, settings, and troubleshooting.
- [Documentation index](docs/README.md): user and developer references.
- [Artifact formats](docs/artifacts.md): portable v1 and v2 captures and validation rules.
- [File-set manifest](docs/manifest.md) and the [partition-pruning lab](docs/pruning-lab/README.md).
- [NYC TLC case study](docs/real-data/README.md): results, downloads, and reproduction.
- [Contributing](CONTRIBUTING.md): development environment, architecture, and checks.
- [Testing and compatibility](docs/validation.md) and [release guide](docs/releasing.md).

## Community and maintenance

Read [Contributing](CONTRIBUTING.md) before submitting a change. Use the repository's
[issue forms](https://github.com/0then0/parquetscope/issues/new/choose) for bugs,
feature requests, and documentation issues. Participation follows the
[code of conduct](CODE_OF_CONDUCT.md). Report security concerns through the process
in [SECURITY.md](SECURITY.md), and see [CHANGELOG.md](CHANGELOG.md) for release history
and changes on `main`.

ParquetScope supports DuckDB 1.5.6 (and 2.0.0 for file-set captures, in an isolated environment prepared with `parquetscope setup --engine 2.0.0`), one file or one explicit set of identically structured files per run. It does not capture production S3 traffic or listing/globbing, merge schemas, run engines other than DuckDB, or predict query costs. The project is licensed under [Apache-2.0](LICENSE).
