Metadata-Version: 2.4
Name: vecorel-cli
Version: 0.4.0
Summary: CLI tools such as validation and file format conversion for vecorel.
Author: Matthias Mohr
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/vecorel/cli
Project-URL: Bug Reports, https://github.com/vecorel/cli/issues
Project-URL: Source, https://github.com/vecorel/cli
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: click<9.0,>=8.1
Requires-Dist: geopandas<2.0,>=1.1.0
Requires-Dist: requests<3.0,>=2.30
Requires-Dist: shapely<3.0,>=2.1
Requires-Dist: numpy<3.0,>=2.0
Requires-Dist: pyarrow<24.0,>=21.0
Requires-Dist: py7zr<2.0,>=1.0
Requires-Dist: inflate64<2.0,>=1.0
Requires-Dist: multivolumefile<1.0,>=0.2.3
Requires-Dist: fsspec>=2025.7.0
Requires-Dist: jsonschema[format]<5.0,>=4.20
Requires-Dist: aiohttp<4.0,>=3.13.5
Requires-Dist: yarl<2.0,>=1.20
Requires-Dist: rarfile<5.0,>=4.0
Requires-Dist: semantic-version<3.0,>=2.10.0
Requires-Dist: json-stream<3.0,>=2.3.0
Requires-Dist: loguru==0.7.3
Requires-Dist: duckdb<2.0,>=1.4
Provides-Extra: s3
Requires-Dist: s3fs>=2025.7.0; extra == "s3"
Provides-Extra: gcs
Requires-Dist: gcsfs>=2025.7.0; extra == "gcs"
Provides-Extra: cloud
Requires-Dist: s3fs>=2025.7.0; extra == "cloud"
Requires-Dist: gcsfs>=2025.7.0; extra == "cloud"
Dynamic: license-file

# Vecorel CLI

A command-line interface (CLI) for working with Vecorel files.

- [Getting Started](#getting-started)
- [Documentation / Commands](#commands)
- [Development](#development)

## Getting Started

In order to make working with Vecorel easier we have developed command-line interface (CLI) tools such as
inspection, validation and file format conversions.

### Installation

#### Using Pixi (Recommended)

This project uses [Pixi](https://pixi.sh/) for dependency management. Install Pixi first, then:

```bash
# Clone the repository and navigate to it
git clone https://github.com/vecorel/cli.git
cd cli

# Install all dependencies
pixi install

# Run the CLI
pixi run vec
```

#### Using pip or uv

Alternatively, you can install from PyPI with **Python 3.11** or any later version:

```bash
pip install vecorel-cli
# or
uv pip install vecorel-cli
```

### Execute a command

After the installation you should be able to run the following command: `vec` (or `pixi run vec` if using Pixi)

You should see usage instructions and [available commands](#commands) for the CLI.

Vecorel CLI supports various commands to work with the files:

- [Vecorel CLI](#vecorel-cli)
  - [Getting Started](#getting-started)
    - [Installation](#installation)
      - [Using Pixi (Recommended)](#using-pixi-recommended)
      - [Using pip](#using-pip)
    - [Execute a command](#execute-a-command)
  - [Commands](#commands)
    - [Validation](#validation)
    - [Create Vecorel GeoParquet from GeoJSON](#create-vecorel-geoparquet-from-geojson)
    - [Create Vecorel GeoJSON from GeoParquet](#create-vecorel-geojson-from-geoparquet)
    - [Inspect Vecorel GeoParquet file](#inspect-vecorel-geoparquet-file)
    - [Merge Vecorel GeoParquet files](#merge-vecorel-geoparquet-files)
    - [Create JSON Schema from Vecorel Schema](#create-json-schema-from-vecorel-schema)
    - [Validate a Vecorel Schema](#validate-a-vecorel-schema)
    - [Improve a Vecorel Parquet file](#improve-a-vecorel-parquet-file)
    - [Update an extension template with new names](#update-an-extension-template-with-new-names)
    - [Converter for existing datasets](#converter-for-existing-datasets)
  - [Development](#development)
    - [Implement a converter](#implement-a-converter)

## Commands

### Validation

To validate a Vecorel GeoParquet or GeoJSON file, you can for example run:

- GeoJSON: `vec validate example.json --collection collection.json`
- GeoParquet: `vec validate example.parquet --data`

Check `vec validate --help` for more details.

The validator also supports remote files.

- `http://` or `https://`: no further configuration is needed.
- `s3://`: With Pixi, run `pixi install -e s3` or with pip/uv, run `pip install vecorel-cli[s3]` and you may need to set environment variables.
  Refer to the [s3fs credentials documentation](https://s3fs.readthedocs.io/en/latest/#credentials) for how to define credentials.
- `gs://`: With Pixi, run `pixi install -e gcs` or with pip/uv, run `pip install vecorel-cli[gcs]`.
  By default, `gcsfs` will attempt to use your default gcloud credentials or, attempt to get credentials from the google metadata service, or fall back to anonymous access.

### Create Vecorel GeoParquet from GeoJSON

To create a Vecorel-compliant GeoParquet for a Vecorel-compliant set of GeoJSON files containing Features or FeatureCollections,
you can for example run:

- `vec create-geoparquet geojson/example.json -o example.parquet -c geojson/collection.json`

Check `vec create-geoparquet --help` for more details.

### Create Vecorel GeoJSON from GeoParquet

To create one or multiple Vecorel-compliant GeoJSON file(s) for a Vecorel-compliant GeoParquet file,
you can for example run:

- GeoJSON FeatureCollection:
  `vec create-geojson example.parquet -o dest-folder`
- GeoJSON Features (with indentation and max. 100 features):
  `vec create-geojson example.parquet -o dest-folder -n 100 -i 2 -f`

  `vec create-geojson example.parquet -o dest-folder -n 100 -i 2 -f`

Check `vec create-geojson --help` for more details.

### Inspect Vecorel GeoParquet file

To look into a Vecorel GeoParquet file to get a rough understanding of the content, the following can be executed:

- `vec describe example.parquet`

Check `vec describe --help` for more details.

### Merge Vecorel GeoParquet files

Merges multiple Vecorel datasets to a combined Vecorel dataset:

- `vec merge ec_ee.parquet ec_lv.parquet -o merged.parquet`
- Only the core properties and some additional properties: `vec merge ec_ee.parquet ec_lv.parquet -o merged.parquet -i ec:hcat_name -i ec:hcat_code`
- All properties except for some: `vec merge ec_ee.parquet ec_lv.parquet -o merged.parquet -e ec:translated_name`

The merged dataset is in EPSG:4326 by default. Use `--crs` to choose another CRS,
or `--crs first` to keep the CRS of the first dataset.

Local GeoParquet files that are all in the target CRS are merged with DuckDB, so they don't need to fit into memory.
All other datasets (e.g. GeoJSON or datasets that need to be reprojected) are merged in memory.
Use `--engine` to choose the engine explicitly.

`-i` and `-e` apply to all properties, including collection-level metadata.
The geometry is required, so it can't be excluded.
Constants that differ between the datasets are moved from the collection metadata to the features.
The collection of the features is stored in a column if the datasets have multiple collections
or if a dataset has a collection column already, otherwise only in the collection metadata.

Geometries are merged as they are, except for the reprojection to the target CRS:
they are neither made valid nor converted to other geometry types (use `vec improve -g` for that),
and features with an empty or missing geometry are not dropped (see below).
The bounding boxes (`bbox`) are computed again for the merged GeoParquet file.

#### Strict and non-strict mode

By default, `vec merge` is strict: problems that make the merged dataset invalid are errors
and no dataset is written.
With `--no-strict`, `vec merge` is fail-safe: these problems are reported as warnings and
the dataset is written anyway. Check it with `vec validate` afterwards.

| Problem | Strict (default) | `--no-strict` |
| ------- | ---------------- | ------------- |
| A required property has no value for some features | Error | Warning, the property is written as nullable |
| An id repeats within a collection | Error | Warning |
| The collection of a feature can't be determined | Error | Warning, the collection is left empty |
| `-i` or `-e` removes a required property | Error | Warning |
| A required collection-only property differs between the datasets | Error | Warning, the property is removed |
| An optional collection-only property differs between the datasets | Warning, the property is removed | Warning, the property is removed |
| A feature has an empty or missing geometry | Error | Warning, the feature is kept |
| A collection-level value doesn't fit the data type of its schema | Error | Warning, the value is left empty |
| The datasets use different versions of the Vecorel specification or of an extension | Error, before any data is read | Error, before any data is read |
| The schemas of the datasets conflict | Error | Error |

Datasets with different versions of the Vecorel specification or of an extension can't be
merged, as the CLI can't upgrade them automatically yet.

The strict mode only checks what a merge can break or check with little effort.
It doesn't validate the values against the schemas, e.g. patterns or value ranges,
so a merged dataset is only valid if the values of the source datasets are valid.

Check `vec merge --help` for more details.

### Create JSON Schema from Vecorel Schema

To create a JSON Schema for a Vecorel Schema YAML file, you can for example run:

- `vec jsonschema example.json --id=https://vecorel.org/specification/v0.1.0/geojson/schema.json -o schema.json`

Check `vec jsonschema --help` for more details.

### Validate a Vecorel Schema

To validate a Vecorel Schema YAML file, you can for example run:

- `vec validate-schema schema/schema.yaml`

Check `vec validate-schema --help` for more details.

### Improve a Vecorel Parquet file

Various "improvements" can be applied to a Vecorel GeoParquet file.
The commands allows to

- change the CRS (`--crs`)
- change the GeoParquet version (`-gp1`) and compression (`-pc`)
- add/fill missing perimeter/area values (`-sz`)
- fix invalid geometries (`-g`)
- rename columns (`-r`)

Example:

- `vec improve file.parquet -o file2.parquet -g -sz -r old=new -pc zstd`

Check `vec improve --help` for more details.

### Update an extension template with new names

Once you've created and git cloned a new extension, you can use the CLI
to update all template placeholders with proper names.

For example, if your extension is meant to have

- the title "Administrative Division Extension",
- the prefix `admin` (e.g. field `admin:country_code` or `admin:subdivision_code`),
- is hosted at `https://github.io/vecorel/administrative-division-extension`
  (organization: `vecorel`, repository `/administrative-division-extension`),
- and you run Vecorel in the folder of the extension.

Then the following command could be used:

- `vec rename-extension . -t "Administrative Division" -p admin -s administrative-division-extension -o vecorel`

Check `vec rename-extension --help` for more details.

### Converter for existing datasets

The CLI ships various converters for existing datasets.

To get a list of available converters/datasets with title, license, etc. run:

- `vec converters`

Use any of the IDs from the list to convert an existing dataset to Vecorel:

- `vec convert de_nrw`

See [Implement a converter](#implement-a-converter) for details about how to

## Development

This project uses [Pixi](https://pixi.sh/) for dependency management and development workflows.

```bash
# Install all dependencies including development tools
pixi install -e dev

# Install the package in editable mode
pixi run install-dev

# Run tests
pixi run -e dev test

# Format and lint code
pixi run format
pixi run lint

# Run all checks (lint, format, test)
pixi run -e dev check

# Install and run pre-commit
pixi run pre-commit-install
pixi run pre-commit-run
```

### Implement a converter

The following high-level description gives an idea how to implement a converter in Vecorel CLI:

1. Create a new file in `vecorel_cli/datasets` based on the `template.py`
2. Fill in the required variables / test it / run it
3. Add missing dependencies into `pyproject.toml` (and the appropriate pixi feature group if needed)
4. Add the converter to the list above
5. Create a PR to submit your converter for review
