Metadata-Version: 2.4
Name: iparq
Version: 0.2.0
Summary: Display version compression and bloom filter information about a parquet file
Author-email: MiguelElGallo <miguel.zurcher@gmail.com>
License-File: LICENSE
Requires-Python: >=3.9
Requires-Dist: pyarrow
Requires-Dist: pydantic
Requires-Dist: rich
Requires-Dist: typer[all]
Provides-Extra: checks
Requires-Dist: mypy>=1.14.1; extra == 'checks'
Requires-Dist: ruff>=0.9.3; extra == 'checks'
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == 'test'
Description-Content-Type: text/markdown

# iparq

[![Python package](https://github.com/MiguelElGallo/iparq/actions/workflows/python-package.yml/badge.svg)](https://github.com/MiguelElGallo/iparq/actions/workflows/python-package.yml)

[![Dependabot Updates](https://github.com/MiguelElGallo/iparq/actions/workflows/dependabot/dependabot-updates/badge.svg)](https://github.com/MiguelElGallo/iparq/actions/workflows/dependabot/dependabot-updates)

[![Upload Python Package](https://github.com/MiguelElGallo/iparq/actions/workflows/python-publish.yml/badge.svg)](https://github.com/MiguelElGallo/iparq/actions/workflows/python-publish.yml)

![alt text](media/iparq.png)
After reading [this blog](https://duckdb.org/2025/01/22/parquet-encodings.html), I began to wonder which Parquet version and compression methods the everyday tools we rely on actually use, only to find that there’s no straightforward way to determine this. That curiosity and the difficulty of quickly discovering such details motivated me to create iparq (Information Parquet). My goal with iparq is to help users easily identify the specifics of the Parquet files generated by different engines, making it clear which features—like newer encodings or certain compression algorithms—the creator of the parquet is using.

***New*** Bloom filters information: Displays if there are bloom filters.
Read more about bloom filters in this [great article](https://duckdb.org/2025/03/07/parquet-bloom-filters-in-duckdb.html).


## Installation

### Zero installation - Recommended

1) Make sure to have Astral’s UV installed by following the steps here:  

    <https://docs.astral.sh/uv/getting-started/installation/>

2) Execute the following command:

    ```sh
    uvx iparq yourparquet.parquet
    ```

### Using pip

1) Install the package using pip:

    ```sh
    pip install iparq
    ```

2) Verify the installation by running:

    ```sh
    iparq --help
    ```

### Using uv

1) Make sure to have Astral’s UV installed by following the steps here:  

    <https://docs.astral.sh/uv/getting-started/installation/>

2) Execute the following command:

    ```sh
    uv pip install iparq
    ```

3) Verify the installation by running:

    ```sh
    iparq --help
    ```

### Using Homebrew in a MAC

1) Run the following:

    ```sh
    brew tap MiguelElGallo/tap https://github.com/MiguelElGallo//homebrew-iparq.git
    brew install MiguelElGallo/tap/iparq
    iparq —help
    ```

## Usage

Run

```sh
iparq <filename>
```

Replace `<filename>` with the path to your .parquet file. The utility will read the metadata of the file and print the compression codecs used in the parquet file.

## Example ouput - Bloom Filters

```log
ParquetMetaModel(
    created_by='DuckDB version v1.2.1 (build 8e52ec4395)',
    num_columns=1,
    num_rows=100000000,
    num_row_groups=10,
    format_version='1.0',
    serialized_size=1196
)
Column Compression Info:
Row Group 0:
  Column 'r' (Index 0): SNAPPY
Row Group 1:
  Column 'r' (Index 0): SNAPPY
Row Group 2:
  Column 'r' (Index 0): SNAPPY
Row Group 3:
  Column 'r' (Index 0): SNAPPY
Row Group 4:
  Column 'r' (Index 0): SNAPPY
Row Group 5:
  Column 'r' (Index 0): SNAPPY
Row Group 6:
  Column 'r' (Index 0): SNAPPY
Row Group 7:
  Column 'r' (Index 0): SNAPPY
Row Group 8:
  Column 'r' (Index 0): SNAPPY
Row Group 9:
  Column 'r' (Index 0): SNAPPY
Bloom Filter Info:
Row Group 0:
  Column 'r' (Index 0): Has bloom filter
Row Group 1:
  Column 'r' (Index 0): Has bloom filter
Row Group 2:
  Column 'r' (Index 0): Has bloom filter
Row Group 3:
  Column 'r' (Index 0): Has bloom filter
Row Group 4:
  Column 'r' (Index 0): Has bloom filter
Row Group 5:
  Column 'r' (Index 0): Has bloom filter
Row Group 6:
  Column 'r' (Index 0): Has bloom filter
Row Group 7:
  Column 'r' (Index 0): Has bloom filter
Row Group 8:
  Column 'r' (Index 0): Has bloom filter
Row Group 9:
  Column 'r' (Index 0): Has bloom filter
Compression codecs: {'SNAPPY'}
```

## Example output 

```log
ParquetMetaModel(
    created_by='parquet-cpp-arrow version 14.0.2',
    num_columns=19,
    num_rows=2964624,
    num_row_groups=3,
    format_version='2.6',
    serialized_size=6357
)
Column Compression Info:
Row Group 0:
  Column 'VendorID' (Index 0): ZSTD
  Column 'tpep_pickup_datetime' (Index 1): ZSTD
  Column 'tpep_dropoff_datetime' (Index 2): ZSTD
  Column 'passenger_count' (Index 3): ZSTD
  Column 'trip_distance' (Index 4): ZSTD
  Column 'RatecodeID' (Index 5): ZSTD
  Column 'store_and_fwd_flag' (Index 6): ZSTD
  Column 'PULocationID' (Index 7): ZSTD
  Column 'DOLocationID' (Index 8): ZSTD
  Column 'payment_type' (Index 9): ZSTD
  Column 'fare_amount' (Index 10): ZSTD
  Column 'extra' (Index 11): ZSTD
  Column 'mta_tax' (Index 12): ZSTD
  Column 'tip_amount' (Index 13): ZSTD
  Column 'tolls_amount' (Index 14): ZSTD
  Column 'improvement_surcharge' (Index 15): ZSTD
  Column 'total_amount' (Index 16): ZSTD
  Column 'congestion_surcharge' (Index 17): ZSTD
  Column 'Airport_fee' (Index 18): ZSTD
Row Group 1:
  Column 'VendorID' (Index 0): ZSTD
  Column 'tpep_pickup_datetime' (Index 1): ZSTD
  Column 'tpep_dropoff_datetime' (Index 2): ZSTD
  Column 'passenger_count' (Index 3): ZSTD
  Column 'trip_distance' (Index 4): ZSTD
  Column 'RatecodeID' (Index 5): ZSTD
  Column 'store_and_fwd_flag' (Index 6): ZSTD
  Column 'PULocationID' (Index 7): ZSTD
  Column 'DOLocationID' (Index 8): ZSTD
  Column 'payment_type' (Index 9): ZSTD
  Column 'fare_amount' (Index 10): ZSTD
  Column 'extra' (Index 11): ZSTD
  Column 'mta_tax' (Index 12): ZSTD
  Column 'tip_amount' (Index 13): ZSTD
  Column 'tolls_amount' (Index 14): ZSTD
  Column 'improvement_surcharge' (Index 15): ZSTD
  Column 'total_amount' (Index 16): ZSTD
  Column 'congestion_surcharge' (Index 17): ZSTD
  Column 'Airport_fee' (Index 18): ZSTD
Row Group 2:
  Column 'VendorID' (Index 0): ZSTD
  Column 'tpep_pickup_datetime' (Index 1): ZSTD
  Column 'tpep_dropoff_datetime' (Index 2): ZSTD
  Column 'passenger_count' (Index 3): ZSTD
  Column 'trip_distance' (Index 4): ZSTD
  Column 'RatecodeID' (Index 5): ZSTD
  Column 'store_and_fwd_flag' (Index 6): ZSTD
  Column 'PULocationID' (Index 7): ZSTD
  Column 'DOLocationID' (Index 8): ZSTD
  Column 'payment_type' (Index 9): ZSTD
  Column 'fare_amount' (Index 10): ZSTD
  Column 'extra' (Index 11): ZSTD
  Column 'mta_tax' (Index 12): ZSTD
  Column 'tip_amount' (Index 13): ZSTD
  Column 'tolls_amount' (Index 14): ZSTD
  Column 'improvement_surcharge' (Index 15): ZSTD
  Column 'total_amount' (Index 16): ZSTD
  Column 'congestion_surcharge' (Index 17): ZSTD
  Column 'Airport_fee' (Index 18): ZSTD
Compression codecs: {'ZSTD'}
```