# Opteryx Core

Opteryx Core is the SQL execution engine behind [opteryx.app](https://opteryx.app). It is a fork of [Opteryx](https://github.com/mabel-dev/opteryx) with a smaller, more opinionated API and configuration surface, shaped around the workloads used by the hosted service.

This library is designed for fast, read-heavy analytical queries over columnar data. It handles SQL parsing, planning, predicate pushdown, projection pruning, and execution so you can query datasets from Python without standing up a separate warehouse.

Query planning is Python; query execution is native. Once the planner has produced a physical plan, the engine runs it in compiled code end to end — scan, operators, scheduling, and dispatch — and neither PyArrow nor NumPy is present anywhere in the engine.

This project is opinionated toward the needs of `opteryx.app`. It is still useful as a standalone library if you want to query local Parquet, NDJSON, CSV, and `.skene` datasets, embed SQL into a Python service or notebook, or experiment with engine internals directly.

## Requirements

- Python 3.11 or later
- A C/C++ toolchain for local source builds
- Rust/Cargo for the Rust extension in `src/`

## Install

```bash
pip install opteryx-core
```

Import it as:

```python
import opteryx
```

## Quick Start: Query Local Files

If your current working directory contains local Parquet data, the simplest way to use Opteryx Core is to register a local workspace and query it with dot-separated names.

```python
import opteryx
from opteryx.connectors import DiskConnector

opteryx.register_workspace("data", DiskConnector)

session = opteryx.session()

for morsel in session.execute_to_morsels("SELECT id, name FROM data.planets WHERE id < 5"):
    print(morsel)
```

Results arrive as Draken morsels — batches of columns — streamed as the engine produces them, so a large result never has to fit in memory at once. A morsel prints as a table, and carries `num_rows`, `column_names` and `column(name).to_pylist()` for getting at the values. Once the stream has been read to the end, `session.rowcount` is the number of rows it delivered.

In this model, dataset names are resolved relative to the current working directory. For example, `data.planets` resolves to `./data/planets`, and Opteryx Core reads the files it finds there, detecting the format from their extension. See [File Formats](#file-formats) for what it can read.

There is also a command line, for querying without writing Python:

```bash
python -m opteryx "SELECT id, name FROM data.planets WHERE id < 5"
```

## What It Is For

- Powering the execution layer used by `opteryx.app`
- Running analytical SQL against local Parquet, CSV, JSONL, and `.skene` datasets
- Embedding a query engine inside Python applications, scripts, notebooks, and services
- Working on engine internals such as planning, native execution, and file-format performance
- Using the file engine or the `.skene` format on their own, via the `rugo` and `libskene` wheels

## Local Development

The supported local build path is the repository Makefile:

```bash
make dev-install
make compile
make q
```

Useful targets:

| Target | Purpose |
|--------|---------|
| `make compile` | Clean in-place build of Cython, C++, and Rust extensions |
| `make c` | Incremental extension build |
| `make q` | Fast SQL shape smoke test |
| `make test` | Full pytest suite after compiling |
| `make dt` | Draken native unit tests |
| `make check` | Ruff and import-order checks without modifying files |

Do not use `pip install .` as the primary development build path; `make compile` matches the layout expected by this repository.

## Repository Layout

| Path | Purpose |
|------|---------|
| `opteryx/` | SQL engine — parser bindings, binder, optimizer, physical planner, connectors, and the native execution engine |
| `draken/` | Native columnar vector substrate (`DrakenVector`) and morsels; zero external dependencies |
| `rugo/` | File engine — Parquet, CSV, and JSONL read and write. Also published standalone; the source is opteryx-free |
| `skene/` | The `.skene` columnar file format — C++ reader, writer, and normative specification. Also published standalone |
| `src/` | Opteryx compute extension sources: Rust (`opteryx_dialect.rs`) and C++ (`src/cpp/`) |
| `reference/` | Generated catalog snapshots (functions, operators, types, joins, clauses). Source of truth for code generation — regenerated by `make reference`, never hand-edited |
| `tests/` | Unit, integration, fuzzing, sqllogictest, and benchmark harnesses |
| `testdata/` | Local datasets and benchmark fixtures |
| `docs/` | Design documents and engineering notes (user documentation lives at [docs.opteryx.app](https://docs.opteryx.app/)) |
| `dev/` | Development, release, vendoring, and analysis scripts; never imported by production code |
| `scripts/` | CI helper scripts |
| `scratch/` | Experimental prototypes and one-off investigations; not packaged |
| `third_party/` | Vendored native dependencies |
| `build_common.py` | Shared build machinery and the single-source extension definitions for draken, rugo, and skene |

## Distributions

This is a single repository that produces three wheels from one source tree. They are packagings of the same sources, not separate forks, so they cannot drift: the extension definitions are single-sourced in `build_common.py`.

| Wheel | Import as | Contains | For |
|-------|-----------|----------|-----|
| `opteryx-core` | `opteryx` | The full SQL engine, bundling draken, rugo, and skene | Querying data with SQL — the primary distribution |
| `rugo` | `rugo` | The file engine (Parquet, CSV, JSONL) plus draken | Reading and writing files without the SQL engine |
| `libskene` | `skene` | The `.skene` format reader and writer plus draken | Lossless draken-vector serialization on its own |

`draken` is not published separately; it ships inside each of the three. `rugo` and `skene` are parallel — neither depends on the other, and the `rugo` wheel does not carry skene. Opteryx never depends on the published `rugo` or `libskene` wheels; those components are intrinsic to it, and the standalone wheels are separate packagings of the same code.

Wheels are built in CI, never locally. For local development use `make compile`, as above.

## File Formats

Datasets are read by extension, and a dataset is one format throughout — a directory mixing formats is an error rather than a best-effort read.

- **Parquet** — the default for stored data and for interchange, read through rugo
- **CSV** and **JSONL/NDJSON** — read through rugo
- **`.skene`** — the draken-native format. It stores one or more row groups of draken vectors losslessly, including the things Parquet drops: an IPv4 column round-trips as a `UINT32` refined by an `IPV4` logical descriptor rather than losing the refinement, and dictionary encoding and layout hints are restored rather than re-derived. It is deliberately not portable and no foreign reader is promised, so Parquet remains the right choice for interchange; `.skene` is for cases where the draken-native round trip is what matters. See [`skene/FORMAT.md`](skene/FORMAT.md) for the specification.

Parquet, CSV, and JSONL files can also be named directly with the `read_parquet()`, `read_csv()`, and `read_jsonl()` table functions. There is no `read_skene()` — skene datasets are read through a registered workspace like any other dataset.

## Best With Opteryx Catalog

Opteryx Core works best when paired with the `opteryx_catalog` library. That is the intended model for named datasets, catalog-backed tables, and the general experience used in `opteryx.app`.

Typical setup:

```python
import os

import opteryx

from opteryx import set_default_connector
from opteryx.connectors import OpteryxConnector
from opteryx_catalog import OpteryxCatalog

set_default_connector(
    OpteryxConnector,
    catalog=OpteryxCatalog,
    firestore_project=os.environ["GCP_PROJECT_ID"],
    firestore_database=os.environ["FIRESTORE_DATABASE"],
    gcs_bucket=os.environ["GCS_BUCKET"],
)
```

Once configured, you can query catalog-backed datasets using dot-separated names such as `public.space.planets` or `opteryx.ops.billing`.

For local data, Opteryx Core is typically used through registered workspaces such as `testdata`, `scratch`, or `data`. Queries refer to datasets by dot-separated names relative to the workspace root, for example `testdata.planets`, `testdata.satellites`, or `scratch.signals`.

## Where It Fits

Opteryx Core is best thought of as an embedded analytical engine rather than a full end-user platform. If you want a hosted experience, multi-tenant service features, and the broader product workflow, use [opteryx.app](https://opteryx.app). If you want the core engine in your own environment, this package gives you that engine directly. If you want the intended table-resolution model, pair it with `opteryx_catalog`.

## Contributing

If you use Opteryx-Core yourself, we want to hear from you.

- Use it on your own datasets
- Raise bugs when queries, schemas, or performance do not behave as expected
- Open pull requests for fixes, tests, docs, or performance improvements
- Share repro cases, failing queries, and edge-case Parquet files

This project is being actively built, and outside usage helps make it better.

Docs: https://docs.opteryx.app/
Source: https://github.com/mabel-dev/opteryx-core
License: Apache-2.0
