Metadata-Version: 2.4
Name: zarr-copy
Version: 0.1.0
Summary: Copy a zarr dataset while changing chunking - variable by variable
Requires-Python: >=3.9
Requires-Dist: fsspec>=2025.10.0
Requires-Dist: numcodecs>=0.12
Requires-Dist: s3fs>=2025.10.0
Requires-Dist: zarr>=2.18.2
Description-Content-Type: text/markdown

# zarr-copy

Copy a zarr dataset while optionally rechunking variables — variable by variable.

## Installation

```bash
pip install -e .
```

Or with [uv](https://github.com/astral-sh/uv):

```bash
uv sync
```

## Usage

```
zarr-copy [-h] [-v VAR [VAR ...]] [--rechunk DIM=SIZE [DIM=SIZE ...]]
          [--double-remap DIM] [--tmp-path DIR]
          zarr_in zarr_out
```

### Positional arguments

| Argument   | Description                                               |
|------------|-----------------------------------------------------------|
| `zarr_in`  | Path or S3 URI of the source zarr group                   |
| `zarr_out` | Path or S3 URI of the destination zarr group              |

### Optional arguments

| Flag                   | Description                                                                              |
|------------------------|------------------------------------------------------------------------------------------|
| `-v VAR …`             | Variable names to copy. Copies all variables when omitted.                               |
| `--rechunk DIM=SIZE …` | Rechunk one or more dimensions, e.g. `--rechunk time=1 lat=256`.                         |
| `--double-remap DIM`   | Use the two-pass copy strategy for the given dimension to reduce peak memory usage.      |
| `--tmp-path DIR`       | Directory for temporary files used by `--double-remap`. Defaults to the system temp dir. |

### Examples

Copy all variables without rechunking:

```bash
zarr-copy /data/input.zarr /data/output.zarr
```

Copy selected variables and rechunk the `time` dimension:

```bash
zarr-copy /data/input.zarr /data/output.zarr -v temperature salinity --rechunk time=1
```

Copy from S3, rechunk multiple dimensions, and use the double-remap strategy:

```bash
zarr-copy s3://my-bucket/input.zarr /data/output.zarr \
  --rechunk time=1 lat=256 lon=256 \
  --double-remap time \
  --tmp-path /scratch
```

Write to S3 from a local source:

```bash
zarr-copy /data/input.zarr s3://my-bucket/output.zarr \
  --rechunk lat=256 lon=256
```

Increase spatial chunk sizes for better read performance along lat/lon:

```bash
zarr-copy /data/input.zarr /data/output.zarr --rechunk lat=256 lon=256
```

Optimise for time-series access by making the time dimension unchunked (one chunk per step):

```bash
zarr-copy /data/input.zarr /data/output.zarr --rechunk time=1
```

Rechunk a large dataset where the time axis is being heavily reordered, using
`/scratch` for intermediate storage to cap memory use:

```bash
zarr-copy /data/input.zarr /data/output.zarr \
  -v u v w temperature \
  --rechunk time=1 level=1 lat=180 lon=360 \
  --double-remap time \
  --tmp-path /scratch
```

## Notes

- Rechunking requires `_ARRAY_DIMENSIONS` metadata on each array (standard for
  xarray/CF-convention zarr stores).
- S3 access is handled transparently via [s3fs](https://s3fs.readthedocs.io/).
- The `--double-remap` strategy writes data to a temporary intermediate zarr
  (lz4-compressed) to avoid holding large in-memory buffers during rechunking.
