Metadata-Version: 2.4
Name: geoparquet-lite
Version: 0.1.0
Summary: Read, write and validate GeoParquet 1.0/1.1 with only pyarrow: WKB and native GeoArrow encodings, bbox covering, row-group spatial filtering.
Author-email: Yusuf Eminoğlu <yusufeminoglu@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/YusufEminoglu/geoparquet_lite_sdk
Project-URL: Repository, https://github.com/YusufEminoglu/geoparquet_lite_sdk
Project-URL: Issues, https://github.com/YusufEminoglu/geoparquet_lite_sdk/issues
Project-URL: Changelog, https://github.com/YusufEminoglu/geoparquet_lite_sdk/blob/main/CHANGELOG.md
Keywords: geoparquet,parquet,pyarrow,geoarrow,wkb,gis,geospatial,geojson,spatial-index,bbox,duckdb,data-engineering,serverless
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyarrow>=12.0.0
Provides-Extra: shapely
Requires-Dist: shapely>=2.0.0; extra == "shapely"
Provides-Extra: geopandas
Requires-Dist: geopandas>=0.14.0; extra == "geopandas"
Requires-Dist: shapely>=2.0.0; extra == "geopandas"
Requires-Dist: pyproj>=3.3.0; extra == "geopandas"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: license-file

# geoparquet-lite

[![PyPI](https://img.shields.io/pypi/v/geoparquet-lite.svg)](https://pypi.org/project/geoparquet-lite/)
[![Python](https://img.shields.io/pypi/pyversions/geoparquet-lite.svg)](https://pypi.org/project/geoparquet-lite/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/YusufEminoglu/geoparquet_lite_sdk/blob/main/LICENSE)

Read, write and validate **GeoParquet 1.0 and 1.1** with nothing but **pyarrow**.

No geopandas, shapely, GDAL or PROJ is required. That makes it a good fit for data
pipelines, serverless functions, DuckDB and Spark interop, and small containers where a full
geospatial stack is too heavy. geopandas and shapely remain available as optional extras for
interop.

```bash
pip install geoparquet-lite                 # pyarrow only
pip install "geoparquet-lite[geopandas]"    # + GeoDataFrame bridges
```

## Features

- **Metadata**: read, write and validate the `geo` key of GeoParquet 1.0.0 and 1.1.0. The
  validator applies every constraint of the official JSON schemas plus the rules that only
  the spec text states. It needs no `jsonschema` dependency.
- **Encodings**: WKB, and the 1.1 native GeoArrow encodings `point`, `linestring`,
  `polygon`, `multipoint`, `multilinestring` and `multipolygon` (separated struct/list
  layout). Columns can be converted in both directions.
- **Pure-Python WKB codec**: 2D, Z, M and ZM, all seven OGC types, both byte orders, and
  ISO WKB as well as PostGIS EWKB input. It converts between WKB, GeoJSON-like dicts and
  GeoArrow arrays.
- **Spatial filtering**: writes a 1.1 `covering.bbox` column. On read, row groups are
  pruned using the Parquet min/max statistics of that column, or of the `x`/`y` leaves of a
  native encoding. The remaining rows are then filtered exactly by bounding box, or by true
  intersection when shapely is installed. Query boxes that cross the antimeridian are
  supported.
- **Hilbert sort** on write, so row groups are spatially compact and pruning skips more of
  the file.
- **CRS** as PROJJSON, following the spec's defaults: no key means OGC:CRS84 and `null`
  means unknown. PROJJSON for OGC:CRS84, EPSG:4326, EPSG:3857, EPSG:4979 and EPSG:4258 is
  bundled, and pyproj is used for other codes if it is installed.
- `edges` (planar/spherical), `orientation` (counterclockwise, with ring rewinding),
  `epoch`, `geometry_types` and the column `bbox`.
- Column projection, batch iteration and fsspec/pyarrow filesystems.
- A `validate(path)` report with clear messages, and `info(path)`.
- A CLI: `geoparquet-lite info | validate | to-geojson | from-geojson | add-bbox | convert`.

## Quick start

```python
import geoparquet_lite as gpq

fc = {
    "type": "FeatureCollection",
    "features": [
        {"type": "Feature", "geometry": {"type": "Point", "coordinates": [29.0, 41.0]},
         "properties": {"name": "Istanbul"}},
        {"type": "Feature", "geometry": {"type": "Point", "coordinates": [32.85, 39.93]},
         "properties": {"name": "Ankara"}},
    ],
}

# GeoParquet 1.1, WKB, bbox covering column, zstd. Returns the written `geo` metadata.
gpq.write(fc, "cities.parquet", crs="EPSG:4326", sort_hilbert=True, row_group_size=50_000)

# Native GeoArrow encoding instead of WKB:
gpq.write(fc, "cities_native.parquet", encoding="geoarrow")

# Read only what intersects a box. Row groups outside it are never read.
table = gpq.read_table("cities.parquet", bbox=(28, 40, 30, 42), columns=["name", "geometry"])

# Stream batches, or GeoJSON features:
for batch in gpq.iter_batches("cities.parquet", batch_size=10_000):
    ...
for feature in gpq.read_features("cities.parquet", bbox=(28, 40, 30, 42)):
    print(feature["properties"]["name"], feature["geometry"])

report = gpq.validate("cities.parquet")
print(report)            # "cities.parquet: valid GeoParquet (0 error(s), 0 warning(s), ...)"
assert report.ok

print(gpq.info("cities.parquet"))          # rows, row groups, encodings, CRS, row-group bboxes
```

### Converting and upgrading files

```python
gpq.convert("in.parquet", "native.parquet", encoding="geoarrow")   # WKB -> native
gpq.convert("native.parquet", "wkb.parquet", encoding="WKB")       # native -> WKB
gpq.add_bbox_covering("v1_0.parquet", "v1_1.parquet", sort_hilbert=True)

table = gpq.read_table("in.parquet", to_encoding="geoarrow")       # convert while reading
table = gpq.convert_table(table, "WKB")
gpq.write(table, "out.parquet")       # tables keep their CRS / edges / orientation metadata
```

### WKB and GeoArrow helpers

```python
from geoparquet_lite import wkb, geoarrow

blob = wkb.dumps({"type": "Polygon", "coordinates": [[[0, 0], [1, 0], [1, 1], [0, 0]]]})
wkb.loads(blob)                         # -> GeoJSON-like dict
wkb.read_header(blob)                   # WKBHeader(Polygon XY, ISO, little)

# XYM has no GeoJSON form, so a "dims" key marks it:
wkb.dumps({"type": "Point", "coordinates": [1, 2, 7], "dims": "XYM"})

arr = geoarrow.from_geometries([...])           # narrowest native encoding that fits
geoarrow.to_geometries(arr, "polygon")
geoarrow.to_wkb(arr, "polygon"); geoarrow.from_wkb(wkb_array, "multipolygon")
```

### geopandas / shapely (optional)

```python
from geoparquet_lite import interop

gdf = interop.to_geopandas(gpq.read_table("cities.parquet", bbox=(28, 40, 30, 42)))
interop.from_geopandas(gdf, "out.parquet", encoding="geoarrow")
geoms = interop.to_shapely(table.column("geometry"), "WKB")
```

## CLI

```text
geoparquet-lite info data.parquet [--json]
geoparquet-lite validate data.parquet [--metadata-only] [--max-rows N] [--strict] [--json]
geoparquet-lite to-geojson data.parquet [-o out.geojson] [--bbox xmin,ymin,xmax,ymax] [--columns a,b] [--seq]
geoparquet-lite from-geojson in.geojson out.parquet [--encoding WKB|geoarrow|polygon|...]
        [--crs EPSG:4326|unknown|crs.json] [--edges spherical] [--orientation]
        [--geoparquet-version 1.0.0|1.1.0] [--no-bbox] [--sort] [--row-group-size N]
geoparquet-lite add-bbox in.parquet out.parquet [--name bbox] [--sort]
geoparquet-lite convert in.parquet out.parquet --encoding geoarrow
```

`validate` exits with status 1 when the file has errors, or with `--strict` when it has
warnings. Any other failure exits with status 2. `from-geojson` also reads newline-delimited
GeoJSON and RFC 8142 text sequences, and reads stdin when the input is `-`.

## Spec coverage

| GeoParquet feature | 1.0 | 1.1 | Notes |
| --- | --- | --- | --- |
| `version`, `primary_column`, `columns` | yes | yes | written as `1.1.0` by default; `version="1.0.0"` is available |
| `encoding: WKB` | yes | yes | ISO WKB, little-endian on write; EWKB is accepted on read and flagged by `validate` |
| native encodings (`point` ... `multipolygon`) | n/a | yes | separated layout, XY/XYZ/XYM/XYZM; single and multi variants merge into the multi encoding |
| `geometry_types` | yes | yes | computed on write; checked per row (WKB) or per column (native) |
| `crs` (PROJJSON, absent = OGC:CRS84, `null` = unknown) | yes | yes | checks required PROJJSON members per CRS type |
| `edges`, `orientation`, `epoch`, `bbox` | yes | yes | `orientation="counterclockwise"` rewinds rings on write and is checked by `validate` |
| `covering.bbox` | n/a | yes | struct of `xmin/ymin/xmax/ymax` doubles; used for row-group pruning and verified per row |

`validate()` also checks the following:

- the physical Arrow type of each geometry column matches its declared encoding (and
  interleaved layouts are reported);
- every geometry decodes;
- each geometry lies inside the column `bbox` and inside its covering bbox;
- the covering column has min/max statistics.

**How the validator compares with the official JSON schemas.** It gives the same verdict as
the official `schema.json` for v1.0.0 and v1.1.0 (with the PROJJSON v0.7 schema) on 106
metadata objects. Those objects come from files written by this package, geopandas 1.2,
GDAL 3.13 and DuckDB 2.0. It is stricter in two places:

- A native `encoding` must agree with `geometry_types`. The specification text requires
  this, but a JSON schema cannot express it.
- A `crs` object must be a CRS. The PROJJSON schema would also accept, for example, a bare
  ellipsoid.

The official schema files are not bundled.

## Interoperability (tested)

- **geopandas 1.2 / pyarrow 26.** `geopandas.read_parquet` reads the files we write, in
  both WKB and native encodings, including with a `bbox=` filter and with the CRS kept. We
  read files from `GeoDataFrame.to_parquet` in both WKB and `geometry_encoding="geoarrow"`,
  with or without `write_covering_bbox`, and they pass `validate()`.
- **GDAL 3.13 Parquet driver.** `ogrinfo` reads our WKB and native files: feature count,
  geometry type, extent, CRS and `-spat` filtering all work. Files from GDAL's
  `ogr2ogr -f Parquet`, in WKB and in `GEOMETRY_ENCODING=GEOARROW`, read and validate here.
- **DuckDB 2.0 spatial.** DuckDB reads our WKB files as `GEOMETRY` with the CRS, and
  `ST_Intersects` and the `bbox` covering struct work in SQL. GeoParquet written by DuckDB
  `COPY ... TO` reads and validates here. DuckDB 2.0 exposes columns with a native encoding
  as their raw `STRUCT`/`LIST` type rather than as `GEOMETRY`. Use WKB when DuckDB is the
  consumer.

## Limitations

- Geometry work is plain Python, roughly 30–70k simple features per second for
  writing and validation. Reading and bbox filtering use pyarrow compute and are fast. This
  package is not a replacement for GDAL in bulk ETL.
- `predicate="bbox"` (the default) keeps rows whose geometry *bounding box* intersects the
  query. For exact geometric intersection, pass `predicate="intersects"`, which requires
  shapely.
- Bounding boxes are computed on the raw coordinates. For `edges="spherical"`, the box of an
  edge that bends poleward of its vertices is not widened.
- Covering columns are written in 2D only (no `zmin`/`zmax`).
- Parquet's native `GEOMETRY`/`GEOGRAPHY` logical types (GeoParquet 2.0 / Parquet 2.11) are
  not handled.

## Development

```bash
pip install -e ".[dev,geopandas]"
pytest --cov=geoparquet_lite
ruff check . && ruff format --check src tests && mypy src
```

The GDAL interop tests run when `ogrinfo` is on `PATH`, or when `GEOPARQUET_LITE_OGRINFO`
points to it. The DuckDB tests run when `duckdb` is installed.

## Data credits

The bundled PROJJSON definitions in `geoparquet_lite/data/projjson.json` were exported with
PROJ from the EPSG Geodetic Parameter Dataset (https://epsg.org), © IOGP, and from the OGC
CRS84 definition.

## License

MIT © Yusuf Eminoğlu
