Metadata-Version: 2.4
Name: hyperunique-druid
Version: 0.1.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: Implementation :: CPython
Summary: Decode, merge and estimate Apache Druid hyperUnique sketches (Rust, via PyO3).
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# hyperunique-druid

Decode, merge and estimate Apache Druid **hyperUnique** sketches
(`org.apache.druid.hll`, version 1 wire format -- *not* DataSketches HLL).
A from-scratch Rust port of Druid 30.0.0's `HyperLogLogCollector`, with
optional Python bindings (PyO3) and a ready-made ClickHouse executable-UDF
loop.

Distributed as a PyPI package (`hyperunique-druid`): abi3 wheels for CPython
>= 3.9, Rust inside. Not published to crates.io -- the Rust API is internal to
the wheel.

## Rust (inside the wheel)

```rust
use hyperunique_druid::hyperunique::merge_estimate;

let estimate: f64 = merge_estimate(&[sketch_a.as_slice(), sketch_b.as_slice()])?;
```

`hyperunique::{decode, fold, estimate}` are there for finer control.

## Python

```python
import hyperunique_druid

hyperunique_druid.merge_estimate([sketch_a, sketch_b])  # float; 0.0 for []
hyperunique_druid.serve()  # ClickHouse RowBinary UDF loop over stdin/stdout
```

`serve()` runs the whole read/merge/write loop natively with the GIL
released, so a ClickHouse Cloud UDF is just:

```python
# main.py
import hyperunique_druid

hyperunique_druid.serve()
```

```
# requirements.txt
hyperunique-druid==0.1.0
```

## Layout

```
Cargo.toml / pyproject.toml   one crate, built by cargo (Rust) or maturin (Python)
src/hyperunique.rs            the sketch logic
src/rowbinary.rs, src/udf.rs  ClickHouse RowBinary framing + UDF loop
src/python.rs                 PyO3 bindings (feature `python`)
python/hyperunique_druid/     Python package wrapping the `_native` extension
package.sh                    builds everything publishable into dist/
verify-wheels.sh              smoke-tests every Linux wheel in containers
```

## Develop

```sh
cargo test                       # Rust tests (no Python needed)
uv venv && uvx maturin develop   # build + install into .venv
```

## Release

1. Bump `version` in `Cargo.toml` (the Python version is read from it too).
   Fill in `license` and `repository` there on the first release.
2. `./package.sh` -- runs the tests and writes to `dist/`: manylinux2014
   (glibc) and musllinux_1_2 (musl) wheels for linux x86_64 and aarch64, plus
   a native wheel for local use. Requires `rustup` and `uv`.
3. `./verify-wheels.sh` -- installs each Linux wheel into real containers
   (glibc + musl, amd64 + arm64, Python 3.9 + 3.13) and runs it. Requires
   `podman` or `docker`.
4. Upload (versions are permanent; set `MATURIN_PYPI_TOKEN` first):

   ```sh
   uvx maturin upload dist/*linux*.whl
   ```
5. Pin the new version in `udfs/hyperunique-merge-pyo3/requirements.txt`.

pip picks the matching wheel, so Cloud's platform doesn't need to be known.
Other platforms (armv7, ppc64le, ...) would need a target added to
`package.sh`; there is deliberately no sdist, so those get pip's clear "no
matching distribution" error instead of a failed Rust build.

