Metadata-Version: 2.1
Name: wormhole-semantic
Version: 0.1.0
Summary: Wormhole: a semantic layer runtime in Zig for DuckDB, PySpark and pandas
Requires-Python: >=3.9
Requires-Dist: pyyaml
Requires-Dist: pyarrow
Provides-Extra: pandas
Requires-Dist: pandas; extra == 'pandas'
Description-Content-Type: text/markdown

# Wormhole

A semantic layer runtime written in Zig. Define dimensions, measures and joins once,
then query them from Python on DuckDB, PySpark or pandas through one native library.

- **SQL engines** (DuckDB, Spark): Wormhole compiles the query to engine SQL.
- **pandas**: Wormhole executes the plan itself on Arrow buffers in Zig (`Wormhole.query`), checked against DuckDB with a differential test.

Models are boring-semantic-layer style YAML and queries are its declarative query dicts
(`dimensions`, `measures`, `filters`, `order_by`, `limit`, `time_grain`, `time_range`).

## Status

| Piece | State |
|---|---|
| Expression parser, SQL emitter (DuckDB + Spark dialects), C ABI, Python wrapper | working |
| DuckDB conformance (`spec/`, 32 queries) | 32/32 |
| PySpark conformance | 32/32 |
| pandas conformance (native Arrow executor) | 32/32 |
| Differential test, executor vs DuckDB SQL (random data, nulls, non-ASCII) | 0 mismatches |
| Wheels (`tools/build_wheels.py`: Linux x86_64/aarch64, macOS x86_64/arm64, Windows) | built in CI, Linux wheel install-tested |
| Benchmarks (`tests/bench.py`) | below |

## Try it

```bash
pip install ziglang==0.16.0 duckdb pyyaml pyarrow
python -m ziglang build -Doptimize=ReleaseSafe
python tests/conformance_duckdb.py
```

```python
import sys; sys.path.insert(0, "python")
from wormhole import Wormhole

wh = Wormhole("spec/models/flights.yml")
print(wh.emit_sql({
    "model": "flights_joined",
    "dimensions": ["carriers.name"],
    "measures": ["flight_count", "delay_per_flight"],
    "filters": [{"field": "flights_joined.origin", "operator": "in", "values": ["JFK", "LGA"]}],
    "order_by": [["flight_count", "desc"]],
    "limit": 5,
}))  # SQL for DuckDB; pass dialect="spark" for Spark SQL
```

## Layout

```
src/expr.zig      scalar expression parser and SQL writer
src/sql.zig       manifest + query -> SQL (DuckDB, Spark)
src/c_api.zig     C ABI (include/wormhole.h)
python/wormhole   ctypes wrapper and YAML -> manifest translation
spec/             shared conformance suite; expected results recorded from boring-semantic-layer on DuckDB
tests/            conformance runners
docs/design.md    design note
```

## Known v1 limits

- Joins: `type: one` (many-to-one, LEFT) only. No `join_many`, self-joins, or nested joins.
- Measures: `count`, `sum`, `mean`, `min`, `max`, `nunique`, and arithmetic over other measures.
  Mixed forms such as `_.x.sum() / _.count()` are rejected when the YAML is loaded.
- A filter may not mix dimension and measure fields; `having=` is not supported yet.
- `order_by` always sorts NULLs last.
- A filter on a name that is neither a dimension nor a measure is treated as a raw column of the model's table, so a typo there surfaces as an engine error.

References: [sm8](https://github.com/EchoEnv/sm8) (Scala) and
[boring-semantic-layer](https://github.com/boringdata/boring-semantic-layer) (Python).

## Packaging

`python tools/build_wheels.py` cross-compiles `libwormhole` with `zig build` and writes one `py3-none-<platform>` wheel per target into `dist/` (Linux wheels target glibc 2.17). Wheels contain no compiled Python extension, so one wheel serves every Python 3 version. The distribution name is `wormhole-semantic` because `wormhole` on PyPI belongs to another project; you still `import wormhole`. Nothing is published to PyPI yet.

## Benchmarks

`python tests/bench.py` (2,000,000 flight rows, median of 3 runs, ms, one machine, single-threaded executor):

| query | wormhole native (pandas) | wormhole SQL on DuckDB, from DataFrames | wormhole SQL on DuckDB tables | boring-semantic-layer on DuckDB |
|---|---|---|---|---|
| count by carrier | 45 | 337 | 20 | 26 |
| aggregates by origin | 90 | 388 | 28 | 37 |
| two dimensions | 126 | 377 | 29 | 36 |
| filter in | 79 | 346 | 25 | 36 |

Reading it: on pandas data the native executor is 3 to 8 times faster than handing the same DataFrames to DuckDB on each call, because DuckDB converts them every time. Once the data sits in DuckDB tables, DuckDB is 3 to 4 times faster than the native executor (which does not use multiple threads yet), and Wormhole's SQL runs as fast as boring-semantic-layer's.
