Metadata-Version: 2.5
Name: iceberg-zamboni
Version: 0.4.0
Summary: Iceberg table maintenance -- compaction, Z-order, partition evolution, snapshot expiry and orphan-file removal -- without needing Trino or Spark
Project-URL: Homepage, https://github.com/paulcaron16k/Zamboni
Project-URL: Repository, https://github.com/paulcaron16k/Zamboni
Project-URL: Documentation, https://github.com/paulcaron16k/Zamboni#readme
Project-URL: Changelog, https://github.com/paulcaron16k/Zamboni/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/paulcaron16k/Zamboni/issues
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: compaction,duckdb,iceberg,lakehouse,maintenance,pyiceberg
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.11
Requires-Dist: duckdb>=1.5.0
Requires-Dist: pyarrow>=18
Requires-Dist: pyiceberg-core<0.11,>=0.10.1
Requires-Dist: pyiceberg[pyarrow]<0.13,>=0.12
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: azure
Requires-Dist: adlfs>=2024.7.0; extra == 'azure'
Provides-Extra: cloud
Requires-Dist: adlfs>=2024.7.0; extra == 'cloud'
Requires-Dist: gcsfs>=2024.6.0; extra == 'cloud'
Requires-Dist: s3fs>=2024.6.0; extra == 'cloud'
Provides-Extra: gcs
Requires-Dist: gcsfs>=2024.6.0; extra == 'gcs'
Provides-Extra: s3
Requires-Dist: s3fs>=2024.6.0; extra == 's3'
Provides-Extra: spark
Requires-Dist: pyspark-client<5,>=4.0; extra == 'spark'
Provides-Extra: spark-lib
Requires-Dist: pyspark<5,>=3.5; extra == 'spark-lib'
Provides-Extra: sql
Requires-Dist: pyiceberg[sql-sqlite]<0.13,>=0.12; extra == 'sql'
Provides-Extra: trino
Requires-Dist: trino>=0.330.0; extra == 'trino'
Description-Content-Type: text/markdown

# Zamboni

[![CI](https://github.com/paulcaron16k/Zamboni/actions/workflows/ci.yml/badge.svg)](https://github.com/paulcaron16k/Zamboni/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

Iceberg table maintenance -- compaction, Z-order clustering, partition evolution,
dangling-delete removal, manifest rewriting, snapshot expiry and orphan-file removal --
**without needing Trino or Spark**. It can drive either Trino or Spark if you have one.

```bash
pipx install iceberg-zamboni       # the distribution name; you still `import zamboni`
# Copy zamboni.yml.sample to $HOME/.zamboni.yml and edit
cat ~/.zamboni.yml
zamboni doctor                      # is this PyIceberg build usable?
zamboni --db acme maintenance       # previews; --yes commits
```

**New here?** [docs/user_guide.md](docs/user_guide.md) is the place to start: four ways to
run it, an engine-capability table to choose with, secrets handling, and the memory
numbers. This README is the *why* -- the evidence behind the design decisions.


## Installation Use-Cases

There are various ways to create and use an Iceberg DB. While Zamboni is for maintenance
only, depending on how you _CREATE_ and _USE_ your Iceberg DB the installation varies.

**Everything about *using* the tables is reference only.** Reading them with DuckDB,
pandas, polars, Trino or a BI tool is outside what Zamboni does -- it compacts, sorts,
expires and reclaims, and then gets out of the way. The reading tools are named below so
you can see where a maintenance install sits in a working stack, not because this package
provides, configures or supports them.

### Dev -- `iceberg-zamboni[sql]`

A laptop, no services, nothing to start. A local SQLite catalog and a warehouse directory,
which is enough to write tables programmatically through the PyIceberg catalog and read
them back with pandas or DuckDB. `zamboni-demo` runs on exactly this.

```bash
pipx install "iceberg-zamboni[sql]"
zamboni-demo next-day        # five days of simulated ingestion, then maintain it
```

### Home Lab -- `iceberg-zamboni[s3]`

A real catalog and a real object store, still on one machine: **Lakekeeper** (which brings
Postgres) and **MinIO**, both from the dev stack's docker services. This is the smallest
setup that behaves like production -- REST catalog, S3 paths, vended credentials -- and the
best base for programmatic data science against the tables: pandas or polars, embedded
DuckDB, SQLAlchemy.

```bash
cd dev-stack && docker compose up -d --wait      # Lakekeeper + Postgres + MinIO
pipx install "iceberg-zamboni[s3]"
zamboni maintenance --warehouse demo --yes
```

### Home Lab Professional -- `iceberg-zamboni[s3]` + Trino for *query*

Home Lab, plus the dev stack's Trino profile:

```bash
cd dev-stack && docker compose --profile trino up -d trino
```

Trino is what lets **Apache Superset** and other BI tools query the Iceberg tables -- they
speak SQL to an engine, and DuckDB will not serve them. **You do _NOT_ use Trino for
maintenance here.** Compaction stays with the local engine: Trino cannot Z-order, cannot
remove dangling deletes, and enforces retention floors that reject Zamboni's defaults.
Adding Trino for BI does not mean adding `--engine trino`.

The `s3` extra is unchanged -- Superset talks to Trino, not to Zamboni, so nothing here
needs the `trino` extra either. That extra is only for `--engine trino`.

### Enterprise -- `iceberg-zamboni[s3,spark]`

Your Spark cluster does the heavy lifting: large-scale compaction and sorting, on hardware
sized for it rather than on one machine's disk and CPU. IT provides the object store, the
REST catalog (Lakekeeper or Polaris), and the Spark master -- you point Zamboni at those
services when you run it. Which object stores have actually been run against, and which
have not, is in [Verified against](#verified-against) below.

```bash
pip install "iceberg-zamboni[s3,spark]"
zamboni maintenance --warehouse acme --engine spark \
        --spark-remote sc://spark.internal:15002 --yes
```

`spark` here is the Spark **Connect client**, ~13MB and no JVM on your side. For BI, IT
provides a Trino cluster -- again for query, not for maintenance.

## Install

### Optional Package Extensions

Depending on the installation use-case you are using, the installation requires different
extensions or extras.

| Extra | Purpose, and when you need it | Python dependencies |
|---|---|---|
| `s3` | The **fsspec** filesystem for S3-compatible storage. Not needed merely because data lives in a bucket: `s3` prefers PyIceberg's `PyArrowFileIO`, whose S3 filesystem is built into pyarrow. What forces fsspec is a catalog that vends **remote signing**, or one that sets `py-io-impl` -- and that is also the posture in which orphan removal is refused, because the signer will not sign `ListObjectsV2` | `s3fs` |
| `azure` | **Required for Azure.** `abfs`/`abfss` *prefer* `FsspecFileIO`, and the fallback to `PyArrowFileIO` does not save you: PyIceberg falls through only on a `ModuleNotFoundError` at construction, and `FsspecFileIO` constructs fine without `adlfs`. The failure is deferred to first use | `adlfs` |
| `gcs` | Not needed for ordinary GCS -- `gs` maps to `PyArrowFileIO` **only**, so fsspec is never chosen for it and pyarrow's `GcsFileSystem` does the work. This covers the narrow case where a catalog overrides `py-io-impl` to fsspec and takes that choice away | `gcsfs` |
| `cloud` | All three of the above, for an image built before the tenant's cloud is known. Not the default: one deployment uses one cloud | `s3fs`, `adlfs`, `gcsfs` |
| `sql` | A local SQLite catalog, with no catalog server to run. Enough to create and open a warehouse on a laptop; used by the test suite, the demo, and dry runs. Not needed against a REST catalog such as Lakekeeper or Polaris | `pyiceberg[sql-sqlite]` (`sqlalchemy`) |
| `trino` | The client for `--engine trino`, which runs five of the six operations as `ALTER TABLE … EXECUTE`. **Requires a local Trino stand-alone instance via `dev-stack --profile trino`, or an Enterprise Trino cluster** -- this extra is a client and starts nothing. Not needed to *query* through Trino from a BI tool | `trino` |
| `spark` | The Spark **Connect** client for `--engine spark --spark-remote sc://…`, which runs all six operations through the Iceberg Spark procedures, Z-order included. **Requires a local Spark stand-alone instance via `dev-stack --profile spark`, or an Enterprise Spark master/cluster.** ~13MB on disk, pure Python, and **no JVM on your machine** -- the driver runs on the server. Needs Spark 4 | `pyspark-client` |
| `spark-lib` | The **embedded** Spark library: a driver JVM inside this process, so no Spark server or cluster is required anywhere. The cost is why it is not the default -- **~472MB on disk** and **~470MB resident** for even a trivial session (measured: 440MB JVM + 34MB Python), rising toward `spark.driver.memory` (1GB by default) under real work, plus a JDK 17 or 21 you install yourself. Mutually exclusive with `spark`: both provide the `pyspark` module | `pyspark` |

`spark` and `spark-lib` are the two halves of the same engine and the names say which side
the JVM is on. Pick `spark` if a cluster exists, `spark-lib` only if none does and you
would rather carry half a gigabyte than run a server.

### Verified against

Zamboni does not implement any object-store protocol: it calls PyIceberg's `FileIO`, which
calls pyarrow or fsspec. So the axis that actually changes its behaviour is **not the
vendor** but the storage *posture* the catalog hands it — credential vending gives
`PyArrowFileIO` and every operation works, while remote signing gives `FsspecFileIO` and a
signer that refuses `ListObjectsV2`, `HeadObject` and multi-object `DELETE`, so orphan
removal cannot run at all. Both postures are exercised on every push.

That is why one store in CI is a deliberate choice rather than a gap, and why this table
names where the others are verified instead of implying they are not:

| Object store | Status |
|---|---|
| **Silo** (maintained MinIO fork) | **Verified, every push.** The `dev-stack` and `spark` CI jobs run a real Lakekeeper against it, in both postures — a credential-vending warehouse and a remote-signing one |
| **MinIO** | Previously verified, on the same jobs, until MinIO ended community distribution and the images stopped resolving. Silo is the same lineage under maintenance and keeps the S3 API, so the coverage carried over rather than lapsed |
| **Garage** | Verified **elsewhere**: ExperienceFlow's end-to-end ELT testing runs IWS + Zamboni against it. Deliberately not duplicated here — it would re-test the same two postures against a second implementation of the same API |
| **AWS S3** | Not run against. Nothing is known to be wrong; nothing has been measured either |
| **GCS** | Not run against. `gs` maps to `PyArrowFileIO` only, so pyarrow's `GcsFileSystem` does the work and **no extra is required** |
| **Azure Blob** | Not run against, and it is the one that needs a package: see the `azure` extra above |
| **Ceph RGW** | Not run against |

"Not run against" is the honest state for most S3-compatible stores, and the reason to
publish it is that the differences are real when they bite. One example from this
codebase's own ecosystem: botocore's flexible-checksum default breaks ranged `GET`s
against Garage, and Zamboni issues ranged reads for every Parquet footer.

### What "Spark works" has actually been run against

The extras admit more combinations than CI exercises, and the honest thing is to say which
is which rather than let a version floor imply coverage it does not have:

| Path | Status |
|---|---|
| `--spark-remote sc://…` (Connect) | **Verified.** Spark 4.0.4 with Iceberg 1.11, on every push — the `spark` CI job builds the server and runs the live tests against it |
| `--spark-master local[*]` (embedded) | Best effort. Nothing in the maintainer is Spark-4 only, and it is not exercised by CI |
| `--spark-master spark://…` (classic cluster) | Best effort. Driver and cluster versions must match, which is why the `spark-lib` floor is 3.5 rather than 4.0 |

**Spark 3.5 is admitted deliberately.** Connect cannot reach a 3.5 cluster at all —
`pyspark-client` did not exist before 4.0 — so `spark-lib` is the only route to one. And
3.5 is not dead: Iceberg publishes `iceberg-spark-runtime-3.5` at its current 1.11.0, and
Spark 3.5.x has an **extended LTS running to November 2027** (security fixes only). Usable
and unverified is a real state; the run logs which Spark it reached, so a bug report
carries the combination rather than requiring someone to reconstruct it.

It will stay unverified, deliberately. A CI leg for 3.5 built today would be maintained
for about a year and then retired with the LTS, for a combination nobody has reported
using — so the gap is a choice rather than an omission, and it reopens the moment someone
turns up on 3.5.

Whichever you use, **the Iceberg runtime jar is yours to supply** and must match the Spark
version — `iceberg-spark-runtime-3.5_2.13` or `-4.0_2.13`. Zamboni sets
`spark.sql.extensions` and deliberately nothing else: a maintenance tool that invented
`spark.sql.catalog.*` would be changing where your data is.

### Installation Best-Practices

**Never into your system Python.** Pick by what you want:

**The distribution is `iceberg-zamboni`; the import is `zamboni`.** They differ because
`zamboni` on PyPI is a dormant registration by an unrelated project. Only the distribution
name has to be globally unique, and the mismatch is ordinary — `beautifulsoup4` imports as
`bs4`.

**The CLI, isolated — recommended.** `pipx` gives it its own virtual environment and puts
`zamboni` on your PATH. We include `sql` to run zamboni-demo:

```bash
pipx install "iceberg-zamboni[s3,sql]"
uv tool install "iceberg-zamboni[s3,sql]"     # the uv equivalent
```

**The library, in your project's environment.** Activate a virtual environment first:

```bash
python -m venv .venv && . .venv/bin/activate
pip install "iceberg-zamboni[s3,sql]"

uv add "iceberg-zamboni[s3,sql]"              # the uv equivalent; no activation needed
```

**From the repository**, for the unreleased tip:

```bash
pipx install "git+https://github.com/paulcaron16k/Zamboni"
```

## Status: what depending on this commits you to

Read this before depending on it.

- **`0.x`.** The public surface is believed reasonably firm, but a `0.x` release may still
  break a CLI script or an API call; every effort will be made to avoid it. From `1.0.0`
  onward the usual semver promise applies -- minor and patch releases will not break you.
  **The unusual part, and the reason to read
  [docs/releasing.md](docs/releasing.md):** for this tool a changed *default* is a breaking
  change even when no signature moves. Lowering `older_than_days` deletes files on the next
  nightly run with nothing in the API having moved at all, so the destructive defaults are
  treated as public surface.
- **Small team (of one).** Security response is best-effort rather than contractual
  ([SECURITY.md](SECURITY.md)), and there is no backport branch: fixes land on the latest
  `0.x`, or `1.x.x` once released, with a minor version, or possibly only patch
  version, increment. Contributors welcome.
- **CI runs green.** Five jobs on every push and pull request, with `test` running once per
  supported Python -- including `dev-stack`, which brings up a real
  Lakekeeper, Postgres and MinIO and runs the demo end to end, and `spark`, which builds a
  Spark Connect server with the Iceberg runtime. The badge above reports the current state
  rather than a claim about it.
- **Verified against real infrastructure.** Every operation has been run against a
  live Lakekeeper + MinIO, and against real Trino 483 and Spark 4.0.4 servers.
- **PyIceberg `>=0.12,<0.13`.** 0.11.x is no longer supported: the floor moved when 0.12
  shipped the fix for the defect the old cap was holding against -- see
  [below](#why-pyiceberg-is-pinned-to-012x).
- **On PyPI as `iceberg-zamboni`**, imported as `zamboni`. See
  [Install](#install); the names differ because `zamboni` on PyPI is a dormant
  registration by an unrelated project.

## What it is, and what it is not

**It is** a maintenance tool for Iceberg tables that runs as a single process: PyIceberg for
metadata, DuckDB for sorting. One command a cron line can call, one declarative file
describing what each table's layout should be and what maintenance may delete. It may optionally
use Trino or Spark standalone or clusters.

**It is not:**

- **A query engine.** It rewrites files and commits snapshots; it does not serve queries.
- **A catalog.** It talks to yours.
- **A scheduler.** It exits with a code; cron, Airflow or systemd decides when.
- **An ingestion tool.** Something else writes the data.
- **A cluster.** Throughput is one machine's disk and CPU. For a partition larger than that
  machine can chew through in the window you have, use `--engine spark`.
- **A replacement for Trino or Spark** where you already run them. It is what you use when
  you do not, and it drives them through the same config when you do.

---

Nothing in the MinIO + Lakekeeper stack compacts tables today:

- **Lakekeeper OSS** ships queues for its own bookkeeping only. Verified against running
  servers: `v0.13.1` reports `["tabular_expiration", "tabular_purge", "task_log_cleanup"]`
  and `latest-main` reports `["soft_deletion", "tabular_purge", "task_log_cleanup"]`.
  `tabular_expiration` is easy to misread — it expires soft-deleted *tables* after a delay
  (7 days by default), not snapshots. None of these compacts data files, expires snapshots
  or removes orphan files. `tests/test_dev_stack.py` checks the queue set against an
  allow-list, so a genuinely new queue fails the build rather than slipping past a
  keyword match.
- **PyIceberg** (0.12.0, the current release) exposes `table.maintenance.expire_snapshots()`
  and nothing else. 0.12 made it a builder — `older_than`, `by_id`, `by_ids`, `commit` —
  which is more than 0.11.x offered, but it is still metadata-only: it emits a
  `RemoveSnapshotsUpdate` and never deletes a file.
- **DuckDB's released Iceberg extension** has no compaction. `iceberg_rewrite_data_files`
  exists on `duckdb-iceberg` main but is not in the shipped extension; loading `iceberg`
  in DuckDB 1.5.4 exposes 16 `iceberg_*` functions and that is not one of them.

## Try it: the Health Information Management System (HIMS) In-Patient discharge demo

Five days of simulated hospital discharge ingestion, so you can watch small files
accumulate and see what maintenance does about them. Copy-on-Write (CoW) and
Merge-on-Read (MoR) are selectable, the default is `mode cow` - which is not optimal for
ingestion via frequent updates or transactional data and why MoR mode was invented in Apache
Iceberg V2 specification and improved in the V3 specification. The Iceberg warehouse created is
in data/healthims/iceberg_warehouse/ in namespace (a.k.a. DB schema) healthims/ folder.s
The `iceberg_warehouse` folder is created _after_ the first `next-day` action.

**From an install**, with no clone — the five days of input CSV data ship in the wheel
(212 KB), and the demo writes to `./zamboni-demo/` in whatever directory you run it from:

```bash
pipx install "iceberg-zamboni[sql]"
zamboni-demo next-day        # x5
zamboni-demo query           # note "files scanned"
zamboni-demo maintenance
zamboni-demo query           # identical rows, far fewer files
```

**From a clone**, `./bin/zamboni-demo` is the same program and keeps its state in
`data/healthims/`:

```bash
./bin/zamboni-demo clear
./bin/zamboni-demo mode cow        # Copy-on-Write is the default. Clear and repeat with "mode mor"
                                   # and time the next-day ingestion to see performance difference.
./bin/zamboni-demo next-day        # Repeat 5 times -- each prints status; file counts climb
./bin/zamboni-demo query           # note "files scanned"
./bin/zamboni-demo maintenance     # compact + drop dangling deletes + expire + remove orphans
./bin/zamboni-demo query           # identical rows, far fewer files
./bin/zamboni-demo next-day        # "No More Data"
```

The employees table is deliberately included as a control: a full daily replace leaves
exactly one live file, so maintenance correctly reports nothing to do there.

After five days in the default copy-on-write mode, `hims_events` holds 625 rows in
**58 data files averaging 3.8 KiB**, with **340 KiB of metadata over 218 KiB of data** --
metadata costs more than the data it describes. One `maintenance` run takes that to a
single file and 89 KiB of metadata, with every row unchanged.

The queries feel it. Files scanned and elapsed time, before and after the same
`maintenance` run:

| Query | files before | files after | ms before | ms after |
|---|---|---|---|---|
| Discharges by replication key | 18 | 2 | 142 | 26 |
| Cancelled and restarted | 75 | 2 | 357 | 31 |
| Event trace for one discharge | 59 | 2 | 262 | 53 |
| Daily EVS turnaround metric | 58 | 1 | 253 | 38 |

Timings include the Iceberg read, which is the part compaction changes; they are still
labelled indicative in the tool's own output because at 625 rows fixed overhead is a large
share of them. Files scanned is exact.

### Why storage does not fall

`maintenance` reports two counts per table, and they diverge:

```
data files    1      total    7.0KiB   avg    7.0KiB
on disk       120    total  538.6KiB   119 superseded, 0 unreferenced
```

Compaction supersedes files; it does not delete them. Every one of those 119 is still
referenced by an older snapshot and still readable by time travel. Only snapshot expiry
drops those references, and only then can the files be deleted.

The default run does both -- and still deletes nothing, because the demo's files were
written minutes ago and both defaults are age-based (5 days for snapshots, 3 for orphans).
It says so rather than hiding it:

```
healthims.hims_discharge: scanned 426 file(s), 421 referenced, 5 unreferenced
  deleted 0 file(s) (0B)
  5 file(s) (7.0KiB) left in place: younger than the 3-day age guard
```

Those are `metadata.json` files nothing references, and the cause is worth knowing:
`SqlCatalog.create_table` writes the metadata file to storage *before* it attempts the
catalog insert, so `create_table_if_not_exists` on an existing table leaves the file behind.
The demo calls it on every state-changing command, so each day after the first strands one
per table. That is the canonical orphan -- a write that was made and then abandoned -- and
it is why orphan removal cannot ask the catalog what exists.

The `on disk` line above counts parquet only, to stay comparable with `data files`, which is
why it reports zero unreferenced. Superseded and unreferenced are different problems with
different owners: expiry and orphan removal respectively.

`./bin/zamboni-demo maintenance --reclaim-now` lifts both. Storage then falls to exactly what is
live -- 120 parquet files to 1, with all 44 rows intact:

```
data files    1      total    7.0KiB   avg    7.0KiB
on disk       1      total    7.0KiB   0 superseded, 0 unreferenced
```

That flag is a demo affordance, not a recommendation: the age guard is what stops orphan
removal deleting a file another writer has written but not yet committed.

`./bin/zamboni-demo mode mor` replays the same five days as a merge-on-read table, where updates
land as position deletes instead of rewrites. Those delete files are **written directly to
simulate what Spark or Flink would emit** -- PyIceberg's `delete()` and `upsert()` are both
copy-on-write, so it cannot produce them. The demo says so in its own output.

Simulating that faithfully needs one thing PyIceberg does not provide. Appending a delete
file through its own producer puts it in a manifest labelled `content: data`, because
`ManifestWriterV2` hardcodes the content and there is no delete-manifest writer. PyIceberg
and DuckDB both dispatch on the *entry's* content and read such a table correctly, so
nothing would have failed -- but an engine that prunes on manifest content would apply no
deletes at all. `zamboni/testing.py` supplies the missing writer, and a test asserts every
manifest's content agrees with the files inside it.

### Query it from the DuckDB CLI

The demo writes real Iceberg tables, so anything that reads Iceberg reads them -- including
`duckdb` from a shell, with no Zamboni in the picture at all. That is worth seeing: it is
what makes maintenance *worth* doing, and it is how you check that a compaction changed
file counts and not answers.

**Ask the catalog which metadata is current.** This is the step to do properly, because it
is the entire reason a catalog exists: it holds the pointer to the table's current
metadata, and every writer updates it atomically. The demo's catalog is SQLite, so a
one-liner reads it:

```console
$ uv run python -c "
from pyiceberg.catalog.sql import SqlCatalog
c = SqlCatalog('healthims', uri='sqlite:///data/healthims/iceberg_catalog.db',
               warehouse='file://data/healthims/iceberg_warehouse')
print(c.load_table('healthims.hims_events').metadata_location)"
file:///.../data/healthims/iceberg_warehouse/healthims/hims_events/metadata/00065-e4147334-....metadata.json
```

**Then hand that to DuckDB.** `iceberg_scan()` takes the `file://` URI as it comes:

```console
$ duckdb
INSTALL iceberg; LOAD iceberg;

CREATE VIEW hims_events AS SELECT * FROM iceberg_scan(
  'file:///.../hims_events/metadata/00065-e4147334-....metadata.json');
```

Getting the DuckDB CLI: `curl https://install.duckdb.org | sh`, or `brew install duckdb`.
Note that `pip install duckdb` and `pipx install duckdb` give you the Python *library* --
the wheel ships no console script, so neither puts a `duckdb` command on your PATH.

**Daily EVS turnaround** -- how long a room sits between the patient leaving and the bed
being ready. Environmental Services cleans the room, and the metric is the gap between two
events sharing a `process_id`, the second of which arrives *after* the discharge process
has already reached its terminal state:

```sql
WITH out AS (
    SELECT process_id, occurred_at AS left_at FROM hims_events
    WHERE event_name = 'patient_displaced_discharged'
), clean AS (
    SELECT process_id, occurred_at AS ready_at FROM hims_events
    WHERE event_name = 'bed_cleaned'
)
SELECT CAST(o.left_at AS DATE)                                    AS day,
       COUNT(*)                                                   AS rooms,
       ROUND(AVG(date_diff('minute', o.left_at, c.ready_at)), 1)  AS avg_turnaround_min,
       MAX(date_diff('minute', o.left_at, c.ready_at))            AS worst_min
FROM out o JOIN clean c USING (process_id)
GROUP BY 1 ORDER BY 1;
```

```
┌────────────┬───────┬────────────────────┬───────────┐
│    day     │ rooms │ avg_turnaround_min │ worst_min │
├────────────┼───────┼────────────────────┼───────────┤
│ 2026-01-05 │     7 │               77.7 │       104 │
│ 2026-01-06 │     9 │              104.3 │       140 │
│ 2026-01-07 │     9 │               85.9 │       115 │
│ 2026-01-08 │    10 │               92.6 │       113 │
│ 2026-01-09 │     9 │               75.2 │       100 │
└────────────┴───────┴────────────────────┴───────────┘
```

The SQL is the demo's own -- `src/zamboni/demo/queries.py` runs this exact statement -- so the
README and the demo cannot drift into computing different things under one name.

**Run it before and after `./bin/zamboni-demo maintenance`.** The metrics do not move, but the
query latency does. Measured on the five-day demo: `hims_events` went from **60 live data files
to 5**, and the output above was byte-identical either side, once through the pre-maintenance
metadata pointer and once through the new one. That is the whole claim in one comparison --
compaction, expiry and orphan removal change how the data is stored and never what it says.
`./bin/zamboni-demo query` runs this and three others for exactly that reason, and reports
"files scanned" alongside, which *does* move.

**Skipping the pointer lookup.** If you would rather not ask the catalog, DuckDB can find
the newest metadata itself, but it makes you say so:

```sql
SET unsafe_enable_version_guessing = true;
CREATE VIEW hims_events AS SELECT * FROM
  iceberg_scan('data/healthims/iceberg_warehouse/healthims/hims_events');
```

The refusal is a good one and the setting is named honestly: globbing the metadata
directory can pick up a file a writer has not committed yet. It is fine against this demo
warehouse, which nothing else is writing to, and it is not how you should read a live
table.

**Against a REST catalog** -- `./bin/zamboni-demo --catalog lakekeeper` -- none of this applies:
DuckDB attaches the catalog with `ATTACH ... (TYPE ICEBERG)` and resolves tables by name,
which is what a real deployment does.

Requirements and domain model: [data/healthims/Demo_Requirements.md](data/healthims/Demo_Requirements.md).
Event catalogue: [data/healthims/HIMS_Discharge_Process_Events.md](data/healthims/HIMS_Discharge_Process_Events.md).

## CI

[.github/workflows/ci.yml](.github/workflows/ci.yml) defines five jobs on push and pull
request:

| Job | What it guards |
|---|---|
| `lint` | ruff check and format; mypy over `src` and `scripts`; `uv sync --frozen` fails on a stale lockfile; pre-commit and CI must pin the same ruff; every source file carries its SPDX tag |
| `test` | The suite on Python **3.11, 3.12 and 3.13** — every version `pyproject.toml` carries a classifier for. 3.11 is the floor `requires-python` declares, 3.13 is the pinned development version, and 3.12 is tested because it is claimed |
| `executables` | `bin/` regenerates to a no-op, and both PEP 723 scripts run **from outside the project directory** |
| `spark` | Builds a Spark Connect server with the Iceberg runtime and S3A, then runs the live Spark tests against it |
| `dev-stack` | The real thing: brings up Lakekeeper + Postgres + MinIO from `.env.sample`, bootstraps it, runs the dev-stack tests, then the demo end to end |

Two jobs guard against a green tick that means nothing. `dev-stack` sets
`ZAMBONI_REQUIRE_DEV_STACK=1`, which turns "cannot reach the stack" from a skip into a
failure; `spark` selects its tests by marker and fails if any of them *skipped*, because
those fixtures skip on a closed port by design. Without both, a stack that never started
yields a suite of skips and a tick that means nothing was tested.

The `executables` job earned its place before it ever ran: `bin/` was found stale against
three separate changes, each of which this job would have caught and nothing else did.

The other two workflows do not gate anything.
[version-watch.yml](.github/workflows/version-watch.yml) runs on the 1st of the month and
asks PyPI whether anything has been released above a version cap `pyproject.toml` declares —
`pyiceberg<0.12` and the dev group's `pyspark-client<4.1` are both waiting on a release that
has to be tested before the bound moves. It keeps one issue current and starts no containers.
There is deliberately **no nightly re-run of the suite**: every input to these tests is
pinned — `uv.lock`, exact image tags in `dev-stack/.env.sample`, pinned Maven jars in the
Spark image, SHA-pinned actions — so against an unchanged commit it would re-prove the tick
that commit already has. [release.yml](.github/workflows/release.yml) publishes to PyPI on a
`v*.*.*` tag, running the suite again first.

Locally, [.pre-commit-config.yaml](.pre-commit-config.yaml) runs the fast checks on every
commit:

```bash
uv run pre-commit install     # `make venv` is what installs pre-commit itself
uv run pre-commit run --all-files
```

The full suite stays out of the hook deliberately — a four-minute hook gets bypassed, and a
bypassed hook is worse than none. Run it with `make test`, below.

## Running Tests

**`make` on its own prints every target.** Each one names the CI job it corresponds to, so
"did I break CI" is answerable before pushing rather than after:

```bash
make                      # the target list, and which stack is currently up
make doctor               # can this machine run everything? checks the toolchain, not the code
make venv                 # .venv from uv.lock — the package, the dev dependencies, and pip
make ci                   # every CI check that needs no containers
```

New here, or setting up a machine: **[ONBOARDING.md](ONBOARDING.md)** is the worked
version of this section, with a checkpoint after each step.

Every target that runs Python goes through `uv`, so it uses `.venv` built from `uv.lock` and
never whatever is installed globally. Each calls a `require_venv` guard first: if `.venv` is
missing it **warns and builds it** rather than telling you to, because `uv sync` is
deterministic here and there is only one right answer. If a *different* virtualenv is
activated it warns about that too — `uv` ignores it, which is confusing precisely once.

**pip is installed inside `.venv` deliberately**, with `uv venv --seed`. `uv sync` does not
put it there, and the failure that causes is silent and lands somewhere else: with `.venv`
activated but no pip in it, `pip install x` runs whichever pip is next on `PATH` and installs
into *that* interpreter — on one machine here, a Python 3.10 pip against a 3.13 project. Any
target repairs it. Adding a real dependency is still `uv add`, because that updates `uv.lock`;
the seeded pip is there so an accident stays inside the virtualenv.

### The CI jobs, one at a time

| CI job | Run it locally | Needs |
|---|---|---|
| `lint` | `make lint` | nothing |
| `test` | `make test` | nothing |
| `test`, all three Pythons | `make test-matrix` | nothing; restores `.venv` afterwards |
| `executables` | `make test-executables` | nothing |
| `dev-stack` | `make test-local`, then `make test-demo` | the local stack |
| `dev-stack`, Trino leg | `make test-trino` | Trino in the stack |
| `spark` | `make test-spark` | Spark in the stack |

`make lint` is the whole job; its pieces are separate targets for a tighter loop while
editing — `make format` writes the formatting fix, and there are `make ruff`,
`make format-check`, `make typecheck`, `make precommit` and `make ruff-pin` (the check that
`.pre-commit-config.yaml` and `uv.lock` name the same ruff, without which every commit
churns). `make test-docs` runs the documentation invariants alone in under a second, which
is the one to keep running while editing prose.

### The dev stack, per engine

The base stack is Lakekeeper, Postgres and MinIO. Each engine is a compose profile on top,
and adding one does not disturb what is already running:

```bash
make local-stack-start      # Lakekeeper + Postgres + MinIO, no engine
make trino-stack-start      # ... plus Trino          (--profile trino)
make spark-stack-start      # ... plus Spark Connect  (--profile spark)

make stack-status           # local, local+trino, local+spark, local+trino+spark, or none
make stack-stop             # stop everything, keep the warehouse
make stack-clean            # stop everything and delete the volumes
```

Which states exist, which target moves you along each edge, and what each state lets you
run:

```mermaid
flowchart LR
  N["<b>none</b><br/>no containers<br/><br/><i>every target above<br/>that needs nothing</i>"]
  L["<b>local</b><br/>Lakekeeper · Postgres · MinIO<br/><br/>make test-local<br/>make test-demo"]
  LT["<b>local+trino</b><br/><br/>make test-trino"]
  LS["<b>local+spark</b><br/><br/>make test-spark"]
  LTS["<b>local+trino+spark</b><br/><br/>make test-trino<br/>make test-spark"]

  N -->|local-stack-start| L
  N -->|trino-stack-start| LT
  N -->|spark-stack-start| LS
  L -->|trino-stack-start| LT
  L -->|spark-stack-start| LS
  LT -->|spark-stack-start| LTS
  LS -->|trino-stack-start| LTS

  L -.->|stack-stop| N
  LT -.->|stack-stop| N
  LS -.->|stack-stop| N
  LTS -.->|stack-stop| N
```

The edge the table cannot express is the one that goes *backwards*: **starting an engine takes
`test-local` and `test-demo` away.** Those two require no engine at all, so `local+trino` is
not a superset of `local` — it is a different state, and `make stack-stop` is how you get the
first two back. Every other transition only adds.

`make stack-status` prints which of these you are in. The start targets write
`dev-stack/.env` from the sample if it is missing, check the pinned
subnet is free, and bootstrap the warehouse. `local-stack-stop`, `trino-stack-stop` and
`spark-stack-stop` are the same teardown, which names both profiles — a plain
`docker compose down` leaves the engine container standing.

**A test target that needs a stack refuses the wrong one rather than skipping.** The
fixtures in `tests/test_dev_stack.py` skip when a port is closed, by design, so a green run
against a stack that never started would mean nothing was tested. `make test-spark` checks
that Spark is in the running stack, sets `ZAMBONI_REQUIRE_DEV_STACK=1`, selects by marker,
and **fails if anything skipped** — the same three guards CI uses. `make test-local` also
requires that *no* engine is running, since one left over from an earlier session changes
what the dev-stack tests exercise.

## Contributing, security, licence

- **[CONTRIBUTING.md](CONTRIBUTING.md)** — the five conventions that make this codebase's
  claims trustworthy, each with the evidence that earned it. Read rule 1 before opening a
  pull request: verify a claim before making it.
- **[SECURITY.md](SECURITY.md)** — this tool deletes files, so the failure mode of a defect
  is somebody's data. Report anything that could delete a still-referenced file privately,
  and note that it does **not** need to be attacker-triggerable to count.
- **Licence: Apache-2.0** — the same licence as Iceberg and PyIceberg, so contributions
  flow both ways without friction. Every source file carries an SPDX tag; see
  [LICENSE](LICENSE).

## Documentation

**Start here: [docs/user_guide.md](docs/user_guide.md)** — the four ways to run
Zamboni (Python API, cron + CLI, Trino, Spark), a capability table to choose an
engine with, secrets handling, and the local engine's measured memory ceiling.

- **[docs/design.md](docs/design.md)** — high-level design: how Iceberg stores a table, how
  that grows, why each maintenance operation exists, architecture, sequence diagrams,
  constraints, and who owns what.
- **[docs/plan.md](docs/plan.md)** — delivery plan: scope, phasing, requirements
  traceability, verification approach, residual risk.
- **[docs/runbook.md](docs/runbook.md)** — when a maintenance cycle fails: exit codes,
  getting a stack trace out of cron, table status, a health check, and recovery.
- **[docs/runbook-dev.md](docs/runbook-dev.md)** — running each step by hand: the order
  and why each position matters, cadence arithmetic, sizing the orphan guard, the dev stack.
- **[docs/devops.md](docs/devops.md)** — running it in production: the cron line, `zamboni.yml`
  and `.env`, why there is no shell wrapper, and the multi-tenant layout for one
  warehouse per customer.
- **[docs/tasks_historical.md](docs/tasks_historical.md)** — the ZMBNI backlog as it stood
  when tracking moved to GitHub: 125 delivered stories with the reasoning behind each, and
  a map from every migrated id to its issue. Frozen. Current work is on
  [board #23](https://github.com/users/paulcaron16k/projects/23).
- **[docs/roadmap.md](docs/roadmap.md)** — the six features planned beyond `v0.1.0`, five of
  which shipped. Mostly a record now: what each was for, the evidence it was chosen on, and
  the one place the sequencing was wrong. The remaining item is PyIceberg 0.12, blocked
  upstream.
- **[docs/engine-comparison.md](docs/engine-comparison.md)** — what Zamboni, Trino and Spark
  each can do, and the twelve places where the same-sounding operation differs.
- **[docs/ice-keeper-comparison.md](docs/ice-keeper-comparison.md)** — against a deployed
  maintenance service: what to adopt, what to decline, and the defect the comparison found.
- **[docs/table-config.md](docs/table-config.md)** — the `table-config.json` specification.
- **[docs/live-verification.md](docs/live-verification.md)** — what runs against a real
  Lakekeeper + MinIO, what a remote-signing warehouse blocks, and the two bugs it found.
- **[docs/releasing.md](docs/releasing.md)** — what a version number promises, and the release
  checklist. Mostly the first half: for a tool that deletes files, a changed default is a
  breaking change with no signature moved, so the destructive defaults are public surface.
- **[CHANGELOG.md](CHANGELOG.md)** — notable changes, with `BREAKING` and `SAFETY` called out.
- **[examples/table-config.json](examples/table-config.json)** — worked example.

## Declarative table layout

Partitioning, partition ageing, sorting and z-order are declared per table in
`table-config.json` — see **[docs/table-config.md](docs/table-config.md)** for the full
specification and [examples/table-config.json](examples/table-config.json) for a worked
example. Analysts can author it inside the Meltano/Singer catalog under an `x-iceberg`
key and generate the file:

```bash
zamboni from-catalog .meltano/catalog.json --namespace analytics -o table-config.json
zamboni validate-config table-config.json
zamboni compact analytics.events --table-config table-config.json --yes
```

Days-to-months partition evolution is **on by default** (90 days); disable it fleet-wide in
`defaults.partition_evolution.enabled`, per table, or at generation time with
`from-catalog --no-evolution`.

> The catalog is a good authoring surface but a bad transport: the Singer SDK's typed
> `Metadata`/`Schema` dataclasses silently drop unknown keys on round-trip, verified
> against the SDK checkout. Meltano core keeps them (it edits raw dicts). Hence generate an
> artifact rather than reading the catalog at compaction time.

## What this does

`TableCompactor` reads a table's small files through PyIceberg's scan, rewrites them into
target-sized files with PyIceberg's own writer, and commits the swap as a single Iceberg
`replace` snapshot per partition.

```python
import os

from zamboni import CatalogSession, CompactionConfig, S3Settings, TableCompactor

session = CatalogSession.for_lakekeeper(
    uri="http://localhost:8181/catalog",
    warehouse="demo",
    # From the environment, or your secret manager -- never a literal, and never
    # a command-line flag. See docs/user_guide.md#secrets.
    credential=os.environ["ZAMBONI_CREDENTIAL"],
    oauth2_server_uri="http://localhost:30080/realms/iceberg/protocol/openid-connect/token",
    scope="lakekeeper",
    # Omit entirely when the catalog vends credentials, which is the arrangement
    # worth having: one revocable secret instead of long-lived object-store keys.
    s3=S3Settings(
        endpoint="http://localhost:9000",
        access_key_id=os.environ["ZAMBONI_S3_ACCESS_KEY_ID"],
        secret_access_key=os.environ["ZAMBONI_S3_SECRET_ACCESS_KEY"],
    ),
)

compactor = TableCompactor(session, "default.events", CompactionConfig())

print(compactor.describe().summary())   # read-only: layout, blockers, warnings
print(compactor.plan().describe())      # what would be rewritten, and what is skipped
print(compactor.execute().describe())   # do it
```

`execute(dry_run=True)` plans and logs without touching the table.

## Dev stack

A Lakekeeper + Postgres + MinIO stack, configured so reclamation works, lives in
[dev-stack/](dev-stack/):

```bash
make local-stack-start                         # .env, compose up, bootstrap; see Running Tests
make test-local                                # the stack tests, refusing to skip

export ZAMBONI_URI=http://localhost:8182/catalog
export ZAMBONI_WAREHOUSE=zamboni
./bin/zamboni-demo --catalog lakekeeper next-day        # the demo, on Lakekeeper and MinIO
```

The equivalent by hand, if you would rather see what those targets do:

```bash
cp dev-stack/.env.sample dev-stack/.env
cd dev-stack && docker compose up -d --wait && uv run bootstrap.py
uv run pytest tests/test_dev_stack.py          # skipped when the stack is down
```

The same 13 checks can be pointed at a deployment this repo did not create:

```bash
uv run scripts/verify-live.py --port 8181 --warehouse acme_db \
    --s3-host 172.19.0.2 --s3-port 9000
```

A failure is the diagnosis. A warehouse that cannot reclaim storage fails
`test_the_warehouse_vends_credentials_rather_than_signing` by name.

Ports are shifted off the defaults so it coexists with anything else you are running.
The two non-obvious settings — why the warehouse needs `sts-enabled` and why its S3 endpoint
is the compose gateway rather than `minio` — are explained in
[dev-stack/README.md](dev-stack/README.md).

## Environment

Everything runs from a locked virtualenv, and nothing resolves against global site-packages.
That is not fastidiousness: PyIceberg's SQL-catalog extra pins SQLAlchemy, and a shared
site-packages is under no obligation to agree. A venv makes that a non-question rather than a
resolution problem.

```bash
uv sync            # builds .venv from uv.lock, Python pinned by .python-version
uv run pytest -q   # no Docker needed; the dev-stack tests skip when it is down
uv run ruff check src tests scripts
```

To move to newer dependencies deliberately:

```bash
uv sync --upgrade
uv run pytest -q
uv run scripts/build-executable.py   # keep the executable in step with the lock
```

### The `zamboni` executable

`bin/zamboni` is a single-file PEP 723 script with every dependency pinned from
`uv.lock` and a `#!/usr/bin/env -S uv run --script` shebang. uv builds and caches an
isolated environment for it on first run, so it works from any directory without an
activated venv and never picks up global packages:

```console
$ /path/to/Zamboni/bin/zamboni doctor
  pyiceberg                    0.12.0
  operation injectable         True
  REPLACE summary native       False
  streaming writes             True
  ...
usable: True
```

Regenerate it with `uv run scripts/build-executable.py` after any lock change. It pins
`requires-python` from `.python-version`, so the executable runs on the same interpreter
the tests do — without that pin uv resolves `>=3.11` to the newest interpreter present and
you ship on a Python you never tested.

```console
$ zamboni describe default.events --uri http://localhost:8181/catalog --warehouse demo
$ zamboni plan     default.events ...
$ zamboni compact  default.events ... --yes          # without --yes, every verb previews
$ zamboni expire   default.events ... --yes          # apply retention, delete what it orphans
$ zamboni remove-orphans default.events ... --yes    # sweep unreferenced files
$ zamboni remove-dangling-deletes default.events ... --yes   # drop deletes that apply to nothing
$ zamboni rewrite-manifests default.events ... --yes  # regroup manifests by partition
$ zamboni apply-properties default.events ... --yes   # metadata-retention table properties
$ zamboni engines                                     # what each engine supports, and refuses
```

For daily operation none of that is the interface. One command runs the six in the right
order over every configured table:

```console
$ zamboni maintenance --warehouse acme --status --yes
```

With `./zamboni.yml` and `./.env` present that is the whole cron line — see
**[docs/devops.md](docs/devops.md)**, which also covers why there is deliberately no shell
wrapper and how a multi-tenant fleet is scheduled.

Each mutating verb takes `--engine` (default `local`, the PyIceberg one).

**Spark works** — all six operations, over the Iceberg Spark procedures, verified against
Spark 4.0.4:

```bash
pip install "iceberg-zamboni[spark]"             # the Connect client: ~13MB, no JVM
zamboni compact default.events --engine spark --spark-remote sc://localhost:15002 --yes
```

**Trino works** for five of the six — it cannot remove dangling deletes:

```bash
pip install "iceberg-zamboni[trino]"
zamboni compact default.events --engine trino --trino-host localhost --yes
```

Spark comes first because it is the more complete of the two: it Z-orders and Trino does
not, so on Trino only your leading `sorted_by` column gets file skipping. `zamboni engines`
reports exactly what each one does and refuses, which is worth reading before planning a
migration. The `--yes` rule holds on every engine: where one cannot preview an operation, a
run without `--yes` is *refused* rather than executed or dressed up as a dry run it did not
perform.

`expire` and `remove-orphans` are dry-run without `--yes`, like `compact`. Both take
`--table-config` for the [`retention`](docs/table-config.md#retention) block, and both
accept overrides (`--max-snapshot-age-days`, `--min-snapshots-to-keep`,
`--older-than-days`) for one-off runs.

Catalog and S3 settings also read from `ZAMBONI_*` environment variables, so the same
invocation works from a shell, a cron entry, or a container.

## Memory

`MemoryMode.CHUNKED` bounds peak memory to roughly one output file. On a build without
streaming writes the bin-packing happens here before calling PyIceberg's writer; on a build
with them (`streaming_write_supported`) the record-batch reader is handed straight to
PyIceberg for unpartitioned tables, which bin-packs it itself. Partitioned tables always
bin-pack locally — partitioned streaming is apache/iceberg-python#2152.

When `sort_expression` is set the stream is routed through DuckDB, whose `ORDER BY` spills
to `temp_directory` on disk rather than holding the group in memory.

`MemoryMode.AUTO` (the default) picks `IN_MEMORY` for groups under `memory_budget_bytes`
(256MiB) and `CHUNKED` above it. CHUNKED reads **one data file at a time**, which is what
bounds it: peak memory is set by the largest file rather than by the group, so a partition
larger than RAM still compacts. It costs about 1.5x on read, which is why small groups stay
on the materialising path. See [docs/user_guide.md](docs/user_guide.md) for the numbers.

## Limitations

### Format versions

| | V1 | V2 | V3 |
|---|---|---|---|
| This package | **blocked** | yes | yes |
| PyIceberg metadata | yes | yes | yes (deletion vectors, puffin) |
| DuckDB Iceberg writes | rejected | yes | yes |
| `iceberg_rewrite_data_files` (unreleased) | n/a | yes | **rejected** |

V1 has no sequence numbers or row-level deletes and DuckDB cannot write it at all; upgrade
to V2 first. Note the inversion: the native DuckDB compactor refuses V3, so this routine
covers strictly more.

### Merge-on-read

| Delete kind | Status |
|---|---|
| Copy-on-write | supported |
| Position deletes | supported and tested — materialised on read, new files get a higher sequence number so the old deletes no longer apply |
| Deletion vectors (V3) | supported via the same position-delete path |
| Equality deletes | **blocked** while `equality_deletes_readable` is false — `DataScan._plan_files_local` raises `ValueError("PyIceberg does not yet support equality deletes")`. The blocker lifts automatically when a build supports them. |

A merge-on-read table's manifest `record_count` is the file's *physical* row count and so
overstates the live count by exactly the deleted rows. The row-preservation check therefore
compares against a delete-aware count (`RewriteOutput.source_live_rows`), computed the way
`DataScan.count()` does: `record_count` for delete-free tasks, an actual read for the rest.
Comparing against `record_count` would reject every correct MoR rewrite.

Delete files are left referenced after compaction. `_OverwriteFiles._deleted_entries` only
marks `DataFileContent.DATA` entries deleted. This is correct but adds metadata that only
snapshot expiry will clear; the count is reported as `CompactionResult.dangling_delete_files`.

### Partitioning

- **Partition-spec evolution** is handled by splitting: each `(spec_id, partition)` becomes
  its own group and its own commit. Files are never mixed across specs. (The native DuckDB
  compactor refuses such tables outright.)
- **All transforms work**, including `bucket`. Rewrites go through
  `_dataframe_to_data_files`, which derives the partition key from the data. The
  `add_files` route cannot do this — it infers partition values from column statistics and
  raises for any transform where `preserves_order` is false.
- **Directory layout** follows whatever PyIceberg's location provider produces for the
  written partition key, so `key=value/` layout is preserved. (The native compactor writes
  compacted files flat into the data path.)

### Sort, clustering, z-order

Declared as `ordering.mode` in table-config (`sort` | `zorder` | `none`), or with the CLI
flags below. Z-order is implemented here — dense-rank each key, scale to `precision_bits`,
bit-interleave — because nothing in Iceberg, PyIceberg or duckdb-iceberg has any notion of
it. The SQL is checked against an independent Morton-encoding reference, and against the
query cost it exists to reduce: blocks touched by a filter on the *second* key.

Two mutually exclusive sort options, differing in whether the result can honestly be
labelled:

- `sort_by_table_order=True` orders by the table's **declared** sort order and stamps that
  order's id onto the output files. Truthful by construction. Only identity-transform sort
  fields can be rendered to SQL; a bucket or truncate sort field is refused rather than
  approximated, because ordering by the raw column would not satisfy the order whose id we
  are about to write.
- `sort_expression="..."` applies an arbitrary DuckDB `ORDER BY` and leaves
  `sort_order_id = None`. The rows are ordered, but not by any order the table declares.

Stamping the table's id for an arbitrary expression would be the same defect the native
duckdb-iceberg compactor has: it stamps `sort_order_id` unconditionally while emitting no
`ORDER BY` at all, so its metadata asserts an ordering the data does not have. PyIceberg
errs the other way and hardcodes `sort_order_id=None` on every file it writes.

Iceberg has no clustering concept distinct from sort order, and neither PyIceberg nor
duckdb-iceberg has z-order or Hilbert curves anywhere. Express a z-order as a
bit-interleaving expression in `sort_expression`.

## Why PyIceberg is pinned to `0.12.x`

`pyproject.toml` declares `pyiceberg[pyarrow]>=0.12,<0.13`, and `pyiceberg-core>=0.10.1,<0.11`
alongside it — 0.12 moved the Rust core out of the `[pyarrow]` extra while
`transforms.pyarrow_transform` still needs it for six transforms, so relying on that extra
now installs a build that cannot write a partitioned table.

**0.11.x is no longer supported.** The floor moved because the reason for the old `<0.12`
cap went away. That cap was held against
[iceberg-python#3758](https://github.com/apache/iceberg-python/issues/3758) — `upsert` on a
table partitioned by a type-changing transform (`day`, `month`, `year`, `hour`, `bucket`)
kept the replaced row beside its replacement and duplicated an untouched one, silently, or
raised for `bucket`. [#3780](https://github.com/apache/iceberg-python/pull/3780) fixed it,
merged 2026-08-19, and the fix is in released 0.12.

That is still not taken on trust: `test_upsert_on_a_transformed_partition_replaces_rather_than_duplicates`
is our own reproduction and it passes on the installed build. It fails on a build that
regressed, which is the only form of the claim worth keeping — the prose document that used
to sit here was replaced by the test (ZMBNI-19).

**The `<0.13` ceiling is policy, not staleness.** An open-ended bound means the day a new
minor publishes, any `uv lock --upgrade` pulls it in with nobody touching this code. The
ceiling is deliberate and it has a stated lift condition: run the suite against the new
minor and raise the bound. `scripts/version_watch.py` reads every cap out of
`pyproject.toml` and keeps an issue current when PyPI publishes above one, so a ceiling
cannot quietly become a permanent habit.

**One install does not resolve from PyPI.** Partition evolution needs a library that writes
an added data file under *its own* partition spec rather than the table default, which
stock PyIceberg does not do. That behaviour lives in our maintenance fork, and
`[tool.uv.sources]` redirects `pyiceberg` there for anyone building from this repository.
It is a uv workspace directive and **does not reach wheel metadata**: `pip install
iceberg-zamboni` gets stock PyIceberg from PyPI, within the range above. On such a build
the `added_files_honour_spec` probe answers False and `MultiSpecReplaceFiles` supplies the
behaviour from the committer instead, so **partition evolution works either way**. The
override runs *only* when the probe says the library will not do it — which is what keeps two
implementations from becoming two behaviours, and is why it cannot mask a library that has
been fixed. See [docs/pyiceberg-private-api.md](docs/pyiceberg-private-api.md).

Note the capability probes do not catch a defect like #3758 by design: they answer "can this
build do X", and such a build *can* upsert -- it simply does it wrongly. Proving
correctness means writing data and reading it back, which is a test, not a probe. Where a
probe genuinely must ask about behaviour rather than structure, it runs the operation --
see [Capability detection](#capability-detection-not-version-checks).

## Capability detection, not version checks

Every version-dependent decision routes through `capabilities.detect()`, which probes the
installed PyIceberg structurally — does this function exist, what does this signature
accept, what does this source say. `zamboni doctor` prints the result.

This is not defensive over-engineering; it is the shape of the problem. Eight probes, and
`zamboni doctor` on a checkout of this repository reports:

| Probe | Answer |
|---|---|
| `operation` injectable into `_SnapshotProducer` | yes |
| `update_snapshot_summaries` accepts `REPLACE` | no |
| `_dataframe_to_data_files` takes a `RecordBatchReader` | yes |
| `_existing_manifests` prunes by predicate | yes |
| producer derives the delete predicate | yes |
| added files honour their own partition spec | yes |
| equality deletes readable | no |
| delete manifests writable | no |

Run it yourself rather than trusting the table — that is the point of the command.

**Two installs of the same declared range differ here.** On stock PyIceberg 0.12 from PyPI,
`added files honour their own partition spec` is **no**; it is yes only on the maintenance
fork. No version number distinguishes those two, which is the argument for probes in one
line. The last two rows are why `remove-dangling-deletes` and equality-delete tables stay
blocked; they lift on their own the day upstream supports them, with nothing changed here.

The probes also gate safety. `manifest_pruning_is_safe` requires the delete-predicate
derivation whenever predicate pruning is on; a build with pruning and no derivation would
keep manifests holding removed files verbatim and count their rows twice, so the tool
refuses to run rather than corrupt a table.

## Private PyIceberg APIs

Two are load-bearing, and `committer.assert_supported_pyiceberg()` fails loudly if either
moves:

1. **`Operation.REPLACE`.** `replace` is the spec's operation for compaction and it is not
   cosmetic — incremental and CDC readers use it to decide whether a snapshot changed any
   rows. PyIceberg cannot emit one: `UpdateSnapshot.overwrite()` hardcodes `OVERWRITE`, and
   `update_snapshot_summaries` rejects anything outside `{APPEND, OVERWRITE, DELETE}` —
   still true on 0.12.0, checked. `_ReplaceFiles` lets PyIceberg compute the summary totals
   as an overwrite and relabels the finished summary. Set
   `CompactionConfig(snapshot_operation="overwrite")` to avoid the subclass entirely.
2. **`_OverwriteFiles._existing_manifests`.** It once scanned every manifest and rewrote any
   containing a removed file; 0.12 added a partition-predicate manifest evaluator plus a
   `_build_delete_files_partition_predicate` step that derives the predicate from the removed
   files, and both are present on the installed build. Both designs are correct, but a build
   with the evaluator and *without* the derivation would keep those manifests verbatim and
   double-count their rows. The guard refuses to run on such a build — and the probe for it is
   behavioural, because `_build_delete_files_partition_predicate` exists by name on a build
   that corrupts data.

Removed files are passed as the `DataFile` objects read from the manifests, never
reconstructed, because that method matches them with `entry.data_file in
self._deleted_data_files`.

## Why not Daft, Ray, or Dask

- **Dask** has no usable Iceberg integration — `dask-iceberg` and `daskberg` are alpha and
  read-only.
- **Ray**: `pyiceberg.Table.to_ray()` is `ray.data.from_arrow(self.to_arrow())`, which
  materialises the whole table before Ray sees it. `ray.data.read_iceberg` /
  `write_iceberg` avoid that but the write API is documented as alpha, and a cluster
  runtime buys nothing on a single-node MinIO deployment.
- **Daft** was kept as a candidate backend for one reason: its writes support every
  partition transform including `bucket`. That advantage turned out not to exist —
  `_dataframe_to_data_files` plus `pyiceberg-core` handles bucket partitioning here, with a
  test proving it. Daft also cannot read equality deletes and has no `replace`-snapshot
  primitive, so no Daft backend is shipped.

### Partition evolution

`partition_evolution` condenses aged fine-grained partitions into coarser ones (the
days-to-months case). New data keeps landing in the fine-grained spec: the coarse spec is
added with `AddPartitionSpecUpdate` **without** `SetDefaultSpecUpdate`.

The trap: PyIceberg's snapshot producer hardcodes the added manifest to the table's
*default* spec while grouping deleted entries by each file's own spec, so month files land
in a day-spec manifest — metadata that reads correctly until a predicate prunes on it.
`MultiSpecReplaceFiles` corrects the asymmetry, and `test_evolution.py` asserts
manifest/file spec agreement rather than only row counts.

Compound specs evolve when exactly one field matches the rule's granularity: that field coarsens, the others carry through, and grouping keys on the whole output partition. Two fields of the same granularity are skipped -- which one dates the partition is ambiguous.

## Reclaiming storage

Compaction frees nothing on its own, so two sibling operations do the reclaiming.

**`expire`** implements the Iceberg spec's retention algorithm -- branch and tag heads,
per-branch ancestry, `min-snapshots-to-keep` winning over age, `main` never expiring --
because PyIceberg implements almost none of it and deletes no files either way. It then
deletes exactly the files reachable before the commit and not after. Computing that
difference, rather than listing storage, is what makes it safe: a file no snapshot ever
referenced cannot appear in the diff, so expiry structurally cannot touch a concurrent
writer's in-flight output.

**`remove-orphans`** does look at raw storage, which is why it is the more dangerous of
the two and why it is fenced:

- Storage is listed **before** the reachable set is computed. A file committed between the
  two steps lands in both and survives; the reverse order would delete it.
- A default **3-day age guard** on file mtime, sized to the longest write the warehouse
  performs -- here that is compaction, not ingest.
- Scope is the table's own location plus any `write.data.path` / `write.metadata.path`,
  never a warehouse-wide sweep.
- The run **aborts** if any referenced file is missing from the listing (the listing is
  partial, so its complement means nothing), or if any of the data / manifest /
  manifest-list / metadata reference categories comes back empty on a table that has
  snapshots. `tests/test_orphans.py` monkeypatches each category away in turn and asserts
  nothing is deleted.

**`remove-dangling-deletes`** drops delete files that no longer apply to any live data
file. Compaction creates them by construction: it applies the deletes as it rewrites, and
the new data files carry a higher sequence number, so the old delete files stop applying —
but `_OverwriteFiles._deleted_entries` filters entries to `DataFileContent.DATA`, so
PyIceberg cannot remove them. It is metadata-only and cannot change a query result, because
a dangling delete is one no scan would have applied; the applicability question is answered
by PyIceberg's own `DeleteFileIndex` rather than by a second implementation of the rule.

Removal is limited to **whole delete manifests**, because `ManifestWriterV2` hardcodes
`content()` to `DATA` and there is no delete-manifest writer in PyIceberg. A partially
dangling manifest is reported and left alone; rewriting it would label position deletes as
data. On the demo's merge-on-read run this clears all 32 dangling deletes and takes the
table from 57 manifests to 3.

Both reclaim verbs read one referenced-file set (`reachable.py`) covering all six places Iceberg
metadata points at files -- data and delete files, manifests, manifest lists, metadata
JSON, statistics, and partition statistics. `max-ref-age-ms` is detected and reported but
applied when configured: the ref is dropped so its snapshots can expire. Off by default,
because removing a named tag or branch destroys metadata someone chose to create.

## Manifest rewriting

A query prunes manifests first, using the partition bounds each one records, then data
files. Streaming ingest defeats the first stage: every commit writes one manifest holding
whatever that batch touched, so after a few hundred appends every manifest spans every
partition and the planner opens all of them. Compaction does not fix it — it rewrites data
files and leaves the manifest set alone.

`rewrite-manifests` regroups live entries by partition, in partition order, bin-packed to
`write.manifest.target-size-bytes`. Contiguity is the point: a manifest holding a
consecutive *range* of partitions still prunes, while one holding a scattering of them has
bounds spanning the table. Sequence numbers and `snapshot_id` are preserved exactly —
reassigning a sequence number would silently change which delete files apply, which reads
correctly today and goes wrong later.

PyIceberg has no `rewrite_manifests`. `_ManifestMergeManager` merges by size at append time
and groups only by spec, so it makes manifests bigger without making them prunable.

## metadata.json retention

Two table properties, and the surprising one is measured in
[docs/table-config.md](docs/table-config.md#retention): `write.metadata.previous-versions-max`
trims the metadata *log*, not the files behind it, so setting it alone manufactures orphans
rather than reclaiming bytes. `apply-properties` sets what `table-config.json` declares and
says how many metadata files are already stranded.

## Format version 3

**Blocked for compaction.** V3 carries row lineage: a rewritten row must keep its original
`_row_id` and `_last_updated_sequence_number`. Compaction reads rows through the scan and
writes them with PyIceberg's writer, neither of which carries those fields, so it would
silently reassign them.

PyIceberg still cannot serialise V3 metadata: on 0.12.0,
`TableMetadataV3.model_dump_json()` raises `NotImplementedError: Writing V3 is not yet
supported`, citing [iceberg-python#1551](https://github.com/apache/iceberg-python/issues/1551).
That makes this fail loudly on a SQL catalog — but a REST catalog builds metadata
server-side and never calls it, so nothing upstream would stop it there. Hence an explicit
blocker rather than reliance on the upstream one.

Metadata-only maintenance rewrites no rows and is unaffected: it copies manifest entries
verbatim, `first_row_id` included.

## Not in scope

Rewriting a *partially* dangling delete manifest, and splitting a single partition across
manifests. Both are limited by PyIceberg having no delete-manifest writer; both lift
automatically if one appears, and `zamboni doctor` probes for it rather than assuming.
