Metadata-Version: 2.4
Name: semql-engine
Version: 0.7.0
Summary: In-process executor for semql FederatedPlans — runs per-backend fragments via caller-supplied adapters and merges results in DuckDB.
Author: Nikhil Pallamreddy
Author-email: Nikhil Pallamreddy <nikhil.pallamreddy+git@gmail.com>
License-Expression: BSD-3-Clause
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Dist: duckdb>=1.5.4
Requires-Dist: semql>=0.7.0,<0.8
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/semql-io/semql
Project-URL: Repository, https://github.com/semql-io/semql
Project-URL: Issues, https://github.com/semql-io/semql/issues
Description-Content-Type: text/markdown

# semql-engine

In-process executor for [`semql`](https://github.com/semql-io/semql)
`FederatedPlan` results. Runs each per-backend fragment via a
caller-supplied `Adapter`, materialises the rows into in-memory DuckDB,
then runs the plan's merge SQL against the assembled tables.

`semql` core stays sans-io. `semql-engine` is the opt-in package that
turns a `FederatedPlan` into result rows when you want the cross-source
execution done for you.

## Install

```sh
pip install semql-engine
```

## Quickstart

```python
import duckdb
from semql import Catalog, Dialect, compile_federated_query
from semql_engine import DuckDBAdapter, Engine

catalog = Catalog([...])  # cubes spanning multiple backends
plan = compile_federated_query(query, catalog.as_dict())

engine = Engine()
engine.register(Dialect.POSTGRES, my_pg_adapter)
engine.register(Dialect.BIGQUERY, my_bq_adapter)
result = engine.run(plan)
result.columns  # ['region', 'revenue', ...]
result.rows     # [(...), ...]   — or stream with engine.iter_rows(plan)
```

## What it does

For every fragment in the plan, the engine calls the adapter registered
for that backend with `(sql, params)`. It loads the resulting rows into
a DuckDB table named `frag_<i>` (matching `FederatedPlan.fragments`
indices) and finally applies `plan.merge_spec` — rendered to DuckDB SQL
by `semql_engine.merge` — to produce the merged shape.

Single-fragment plans (single-backend queries that went through
`compile_federated_query` anyway) work transparently — the merge is a
pass-through.

## Adapters

An `Adapter` is anything with `execute(sql, params) -> AdapterResult`
where `AdapterResult` carries `columns: list[str]` and an iterable of
row dicts. Built-ins:

- `DuckDBAdapter(con)` — runs the SQL inside an existing DuckDB
  connection. Useful for local CSV / Parquet enrichment cubes.
- `DBAPIAdapter(con)` — wraps any PEP-249 connection (psycopg, mysql,
  sqlite, etc).

Bring your own for warehouses that need a vendor SDK.

## Semi-joins

A cross-backend semi-join (restrict an outer dimension to the value set
of an inner query, shipped as a value list rather than a join) compiles
to a `SemiJoinPlan` via `semql.compile_semi_join_query`. Run it with
`run_semi_join(plan, engine)` — it executes each inner plan, projects the
key column to a value list, and runs the outer query with that list bound
as an `IN` / `NOT IN` filter:

```python
from semql import compile_semi_join_query
from semql_engine import Engine, run_semi_join

plan = compile_semi_join_query(query, catalog.as_dict())
result = run_semi_join(plan, engine)  # same ExecutionResult shape as run()
```

## Scope

v1 mirrors `compile_federated_query` v1:

- Sum / count / avg supported (avg is decomposed at compile and
  recomposed in the merge SQL); other aggregations are refused by the
  compiler before the engine ever sees them.
- Equality bridge joins only.
- No `compare` mode, no boolean `where` tree across backends.

The engine itself is small; most of the federation logic lives in
`semql.federate`.

## Status

Early development. The `Adapter` contract is stable; the federation
shape mirrors `compile_federated_query` v1 (sum / count / avg and
equality bridge joins only).
