Metadata-Version: 2.5
Name: ray-clickhouse
Version: 1.0
Summary: Ray Data read and append connector for ClickHouse physical tables
Project-URL: Homepage, https://github.com/jiangxt2/ray-clickhouse
Project-URL: Source, https://github.com/jiangxt2/ray-clickhouse
Project-URL: Documentation, https://github.com/jiangxt2/ray-clickhouse/tree/master/doc
Project-URL: Changelog, https://github.com/jiangxt2/ray-clickhouse/tree/master/release-notes
Author: jiangxt2
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.10
Requires-Dist: clickhouse-connect[arrow]<1.6,>=1.5
Requires-Dist: pyarrow<20,>=19
Requires-Dist: ray[data]<2.59,>=2.55
Requires-Dist: typing-extensions<5,>=4.12
Provides-Extra: dev
Requires-Dist: build<2,>=1.2.2; extra == 'dev'
Requires-Dist: mypy<2,>=1.17; extra == 'dev'
Requires-Dist: pyarrow-stubs<20,>=19; extra == 'dev'
Requires-Dist: pytest-cov<8,>=6; extra == 'dev'
Requires-Dist: pytest-timeout<3,>=2.3; extra == 'dev'
Requires-Dist: pytest<10,>=8.3; extra == 'dev'
Requires-Dist: ruff==0.16.4; extra == 'dev'
Requires-Dist: twine<8,>=7; extra == 'dev'
Description-Content-Type: text/markdown

# ray-clickhouse

Ray Data read and append connector for ClickHouse physical tables.

> [!IMPORTANT]
> `ray-clickhouse` is community-maintained and is not an official Ray or ClickHouse project.
> Version `1.0` follows the two-component `major.minor` release route.

This release provides:

- structured physical-table reads returning `ray.data.Dataset`;
- single-query reads from direct tables, Views, and Distributed tables;
- Arrow streaming with bounded rows and bytes per emitted block;
- optional partition and integer range planning for direct MergeTree-family tables;
- append writes plus explicit safe `create`/`overwrite` table-management modes;
- structured ordering and Ray task resource passthrough;
- fail-closed schema, type, permission, timeout, and ambiguous-write errors.

The connector does not provide arbitrary SQL execution, arbitrary DDL, delete, update, upsert,
connector-side Distributed shard routing, cross-task transactions, or exactly-once guarantees.
Distributed reads use one ClickHouse query; ClickHouse owns shard routing.

Automatic range planning is opt-in. For a direct MergeTree-family table with a
simple sorting key starting with an integer column, use `split="auto"` to select
that column without supplying `range_column`:

```python
from ray_clickhouse import read_clickhouse

dataset = read_clickhouse(
    host="clickhouse.example",
    database="analytics",
    table="events",
    split="auto",
    target_tasks=4,
)
```

The default remains `split="single"`. Automatic planning reuses the existing
filtered range aggregate and disjoint range planner, not ordered pagination.
Unsuitable sorting keys fall back to one query; use `discovery_policy="error"`
to require an eligible range plan. Authentication, permission, and transport
failures still raise. See [Architecture](doc/source/architecture.md) for supported
keys and consistency limits.

The read API follows the Ray Data `Datasource`/`ReadTask` contract. The write API uses
`Dataset.write_datasink()` and does not monkey-patch Ray Dataset methods.
The package-root public API consists of `read_clickhouse()`, `write_clickhouse()`,
`WriteReceipt`, and the documented error hierarchy. Datasource and Datasink implementation
classes are internal and are not independent compatibility promises.

## Documentation

- [Documentation index](doc/source/index.md)
- [Compatibility](doc/source/compatibility.md)
- [Architecture](doc/source/architecture.md)
- [Security model](doc/source/security.md)
- [Troubleshooting](doc/source/troubleshooting.md)
- [Contributing](CONTRIBUTING.md)
- [Security policy](SECURITY.md)
- [Release notes](release-notes/v1.0.md)

## Query result reads

The read API also accepts a trusted query source. Query reads require
exactly one of `table` or `query`, use one data-read task, and infer the
result schema with DESCRIBE and a zero-row Arrow probe.

```python
dataset = read_clickhouse(
    host="clickhouse.example",
    database="analytics",
    query=(
        "SELECT tenant_id, count() AS event_count FROM events "
        "WHERE tenant_id = %(tenant)s GROUP BY tenant_id"
    ),
    query_parameters={"tenant": 42},
)
```

Query mode supports the documented SELECT/WITH subset, client-side named value
bindings, and unique simple result aliases. Table projection, filter, ordering,
range columns, and split modes are supplied in the SQL rather than connector
options. The connector enforces `readonly=1` and rejects multiple statements,
DDL/DML, output formats, output files, and inline settings. Metadata probes and
the data read are independent requests and do not share a database snapshot.
A zero-row probe can still incur server work. `override_num_blocks` changes
Ray output blocks and does not multiply the query.

## Usage

```python
from ray_clickhouse import read_clickhouse, write_clickhouse

dataset = read_clickhouse(
    host="clickhouse.example",
    database="analytics",
    table="events",
    columns=("event_id", "tenant_id", "payload"),
    filter="tenant_id = %(tenant)s",
    query_parameters={"tenant": 42},
)

write_clickhouse(
    dataset,
    host="clickhouse.example",
    database="analytics",
    table="events_copy",
    insert_mode="sync",
)

# Explicit table management is opt-in and restricted to safe generated DDL.
write_clickhouse(
    dataset,
    host="clickhouse.example",
    database="analytics",
    table="events_new",
    write_mode="create",
    order_by=("event_id",),
)
```

When diagnosing an Arrow or transport failure, set `diagnostic_flush_logs=True` on
`read_clickhouse()` if the ClickHouse account may run `SYSTEM FLUSH LOGS`:

```python
dataset = read_clickhouse(
    host="clickhouse.example",
    database="analytics",
    table="events",
    diagnostic_flush_logs=True,
)
```

Read failures carry a generated query ID and a bounded `client_diagnostic` with
the underlying exception type, operation, and safe message context. The
connector performs a bounded, best-effort lookup in the discovered ClickHouse
query-log table and appends the server exception code and message when available.
It never replays the failed SQL; if query logging is unavailable, the client
diagnostic and query ID remain available. Raw exception chains remain suppressed
to protect credentials. The diagnostic text is
server-provided and may include query context, so do not put credentials in query
predicates or parameters.

`filter` is a trusted predicate fragment, not an arbitrary SQL query.
Use `tenant_id = {tenant:Nullable(UInt32)}` with `query_parameters={"tenant": 42}`
for native server-side binding. Existing `%(tenant)s` client binding remains
supported. Mixing binding styles or using identifier parameters is rejected.
Typed parameter-looking text inside quotes/comments is rejected when bindings
are active. Supported hints are String, FixedString, Bool, Int8/16/32/64,
UInt8/16/32/64, Float32/64, Date/Date32, DateTime/DateTime64, Decimal/32/64/128,
and one Nullable wrapper. DateTime64 precision must be 0 through 9; timezone
hints use a quoted timezone name. Collection and identifier hints are unsupported. Values must be passed through
`query_parameters`; the connector rejects unrestricted statement execution and does not provide delete, update,
upsert, connector-side Distributed shard routing, or exactly-once semantics. `write_mode="overwrite"`
is destructive and must be selected explicitly. For create/overwrite, nullable columns must be
listed explicitly through `nullable_columns`; timestamp precision is preserved and unsupported
binary types are rejected.

Writing an empty Dataset is a confirmed no-op: it returns a zero-valued `WriteReceipt` without
performing ClickHouse discovery, table management, or INSERT. If Ray returns without invoking the
normal sink completion callback, `write_clickhouse()` raises `WriteError` instead of reporting a
zero-valued success. MaterializedView reads are not part of the supported capability profile.

## Compatibility

`ray-clickhouse` supports final Ray releases `>=2.55,<2.59` on Python 3.10–3.13.
Prerelease, development, and post-release Ray builds are rejected. PEP 440 local build
suffixes are allowed. Pandas is resolved through `ray[data]`; it is not an independent
`ray-clickhouse` compatibility promise.

| Python | Ray | Verification |
| --- | --- | --- |
| 3.12 | 2.55.0 | Unit and public-contract tests |
| 3.12 | 2.56.1 | Unit and public-contract tests |
| 3.12 | 2.57.0 | Unit and public-contract tests |
| 3.12 | 2.58.0 | Unit, public-contract, and package tests |
| 3.10 | 2.58.0 | Unit, public-contract, and wheel tests |
| 3.11 | 2.58.0 | Unit, public-contract, and wheel tests |
| 3.13 | 2.58.0 | Unit, public-contract, and wheel tests |

## Development

```bash
uv sync --extra dev
uv sync --extra dev --group docs
uv run ruff format --check .
uv run ruff check .
uv run mypy
uv run python tools/check_compatibility.py
uv run python tools/check_release.py check
uv run pytest tests/unit tests/contract
uv build
uv run twine check dist/*.whl dist/*.tar.gz
uv run python tools/check_docs.py
make -C doc html
make -C doc spelling
make -C doc linkcheck
./scripts/run_clickhouse_it.sh
./scripts/run_ray_cluster_it.sh
```

ClickHouse and multi-node Ray integration commands remain available in the repository,
but they are required only when the affected production or infrastructure behavior changes.
The pinned infrastructure baselines are ClickHouse 26.8.1.2041 and a Ray 2.58.0/Python 3.12
three-node runtime.
