Metadata-Version: 2.4
Name: data-connect-hub
Version: 0.1.0
Summary: Python SDK for the Data Connect Hub service
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/opendatahub-io/data-connect-hub
Project-URL: Repository, https://github.com/opendatahub-io/data-connect-hub
Project-URL: Documentation, https://opendatahub-io.github.io/data-connect-hub/
Project-URL: Issues, https://github.com/opendatahub-io/data-connect-hub/issues
Keywords: data,data-connect-hub,data-connectivity,flight-sql,opendatahub
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx<1,>=0.27
Requires-Dist: pydantic<3,>=2.0
Provides-Extra: flight
Requires-Dist: adbc-driver-flightsql>=1.0.0; extra == "flight"
Requires-Dist: pyarrow>=14.0; extra == "flight"
Requires-Dist: pandas>=2.0; extra == "flight"
Requires-Dist: protobuf>=6.33.5; extra == "flight"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: numpy<2.5; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Dynamic: license-file

# Data Connect Hub Python SDK

Python client library for the [Data Connect Hub](https://github.com/opendatahub-io/data-connect-hub) service.

## Installation

```bash
# REST only (default)
pip install data-connect-hub

# REST + Flight SQL
pip install "data-connect-hub[flight]"
```

To install from a source checkout, use `pip install sdk/python` or
`pip install "sdk/python[flight]"`.

TestPyPI builds are PEP 440 development releases, so `--pre` is required. Install
the SDK without dependencies from TestPyPI, then install its dependencies from
PyPI only:

```bash
pip install --pre --no-deps \
  --index-url https://test.pypi.org/simple/ \
  data-connect-hub
pip install --index-url https://pypi.org/simple/ \
  "httpx>=0.27,<1" "pydantic>=2,<3"
```

Installing from a GitHub source archive (`.../archive/main.tar.gz`) fails: archives
carry no `.git` directory, so `setuptools-scm` cannot derive a version. This is
deliberate — a silent fallback version would sort unpredictably against published
releases. Install from Git instead:

```bash
pip install "git+https://github.com/opendatahub-io/data-connect-hub.git#subdirectory=sdk/python"
```

## Quick Start

The client takes a single gateway `endpoint` — a host or `host:port`, no scheme required — and derives both the REST (`https://`) and Flight SQL (`grpc+tls://`) URLs from it. Only TLS endpoints are supported; use `insecure=True` or `ca_cert=` to control certificate verification.

```python
from data_connect_hub import CredentialsRef, DataConnectClient

client = DataConnectClient(
    endpoint="dch.example.com:8443",
    token="<your-token>",  # or use token_provider= for auto-refresh
    tenant_id="my-tenant",
)

# Or use a token provider for automatic refresh on 401:
client = DataConnectClient(
    endpoint="dch.example.com:8443",
    token_provider=lambda: get_fresh_token(),  # your function; called once, cached, refreshed on 401
    tenant_id="my-tenant",
)

# List connections (REST)
connections = client.list_connections()

# Get a specific connection
conn = client.get_connection("conn-id")

# Create a connection
conn = client.create_connection(
    name="my-db",
    connection_type_id="dct-a1b2c3d4",
    data_format="tabular",  # DataFormat: "tabular" | "binary"
    credentials_ref=CredentialsRef(secret="secret/my-db"),
)

# Query data via Flight SQL
table = client.read("SELECT * FROM prompts", connection_id="conn-uuid")
df = table.to_pandas()
```

## API Reference

The REST API is the source of truth for every model below. See the [REST API reference](https://opendatahub-io.github.io/data-connect-hub/) for the full request/response schemas.

### Connection Types (REST)

Connection types describe a category of data source (e.g. PostgreSQL). They define the provider backend and the credential fields required to connect.

```python
client.list_connection_types() -> list[ConnectionType]
client.get_connection_type(type_id) -> ConnectionType
client.create_connection_type(name=..., provider=..., description=..., credentials_fields=...) -> ConnectionType
client.update_connection_type(type_id, name=..., provider=..., description=..., credentials_fields=...) -> ConnectionType
client.delete_connection_type(type_id) -> None
```

#### `ConnectionType`

| Field | Type | Description |
|---|---|---|
| `id` | `str` | Unique identifier |
| `name` | `str` | Display name |
| `provider` | `str` | Backend driver (e.g. `"postgres"`) |
| `description` | `str \| None` | Optional description |
| `tenant_id` | `str` | Owning namespace |
| `created_at` | `datetime \| None` | Creation timestamp |
| `updated_at` | `datetime \| None` | Last update timestamp |
| `credentials_fields` | `list[CredentialField]` | Credential fields required to connect |
| `status` | `ConnectionTypeStatus` | Transports the provider supports |

Pass `id` as the `type_id` argument to `get_connection_type`, `update_connection_type`, and `delete_connection_type` — and as `connection_type_id` to `create_connection`.

`status.capabilities` reports which transports the provider supports (`flight` and `rest`, both `bool`), so you can check before issuing a Flight SQL query:

```python
ct = client.get_connection_type("dct-a1b2c3d4")
if ct.status.capabilities.flight:
    table = client.read("SELECT * FROM prompts", connection_id=conn.id)
```

#### `CredentialField`

Describes a single input field in the connection credential form.

| Field | Type | Description |
|---|---|---|
| `name` | `str` | Field key (used as the secret key) |
| `label` | `str` | Human-readable label |
| `description` | `str \| None` | Optional help text |
| `required` | `bool` | Whether the field must be provided |
| `type` | `str` | Rendering hint for the form (see below) |
| `enum_values` | `list[EnumValue] \| None` | Allowed values when `type` is `"enum"` |
| `default_value` | `str \| None` | Optional default value |

`EnumValue` has two fields: `value` (the stored string) and `label` (the display string).

**`type` values:**

| Value | Meaning |
|---|---|
| `"string"` | Free-text single-line input |
| `"enum"` | One of `enum_values` |

`type` is a free-form string that only tells a client how to render the input — the server neither validates nor interprets it. Its one credential check is that every field with `required=True` is present in the submitted secret. Every connection type shipped in [`config/connection-types/`](../../config/connection-types/) uses `"string"`; your own may use any other value (e.g. `"password"` to hint that input should be masked), and clients that do not recognize it should treat it as `"string"`. The authoritative definition is the `Field` schema in the [REST API reference](https://opendatahub-io.github.io/data-connect-hub/).

### Connection Management (REST)

A connection pairs a connection type with the actual credentials (stored in a Kubernetes secret) and tracks the live status of the data source.

```python
client.list_connections() -> list[DataConnection]
client.get_connection(connection_id) -> DataConnection
client.create_connection(name=..., connection_type_id=..., data_format=..., credentials_ref=..., properties=...) -> DataConnection
client.update_connection(connection_id, name=..., connection_type_id=..., data_format=..., credentials_ref=...) -> DataConnection
client.delete_connection(connection_id) -> None
```

#### `DataConnection`

| Field | Type | Description |
|---|---|---|
| `id` | `str` | Unique identifier |
| `name` | `str` | Display name |
| `data_connection_type_id` | `str` | `id` of the associated `ConnectionType` |
| `format` | `"tabular" \| "binary"` | Data format of the source (see below) |
| `tenant_id` | `str` | Owning namespace |
| `created_at` | `datetime` | Creation timestamp |
| `updated_at` | `datetime` | Last update timestamp |
| `credentials_ref` | `CredentialsRef` | Credential secret reference |
| `properties` | `dict[str, str]` | Driver-specific properties (values masked in repr) |
| `status` | `DataConnectionStatus` | Live connection health |

Pass `id` as the `connection_id` argument to `get_connection`, `update_connection`, `delete_connection`, and the Flight SQL methods.

**`format` values:**

| Value | Meaning | Providers |
|---|---|---|
| `"tabular"` | Queried with SQL, returns rows | `postgres`, `sqlite`, `elasticsearch`, `milvus`, `neo4j`, `uri`, `s3` |
| `"binary"` | Opaque objects addressed by path | `s3`, `uri` |

Tabular connections are read with the [Flight SQL methods](#tabular-data-queries-flight-sql). Binary connections are managed through the same REST methods as tabular ones, but reading their contents uses a separate Flight download path that **this SDK does not wrap yet** — there is no `client.download(...)`. Until it is added, use `pyarrow.flight` directly; see [`hack/py-tools/samples/binary_download.py`](../../hack/py-tools/samples/binary_download.py).

You normally set `format` once, at `create_connection`, but it is not immutable: `update_connection(connection_id, data_format=...)` changes it, and the server accepts the new value without checking it against the provider or re-evaluating `status`. So switching a `postgres` connection to `binary` succeeds, leaves `status` reporting `ready`, and fails only when you try to read.

`credentials_ref` is a reference to a Kubernetes secret containing the connection credentials. Use `CredentialsRef(secret="secret-name")` where `secret-name` is the **name** of an existing secret in the tenant namespace (the namespace named by the connection's `tenant_id` that you passed to `DataConnectClient`). This is a bare secret name, not a `namespace/name` pair; cross-namespace references are not supported. If the secret is missing or unreadable, `status.state` becomes `"not_ready"`. The secret's keys must cover every `CredentialField` on the connection type that has `required=True`.

**`DataConnectionStatus`:**

| Field | Type | Description |
|---|---|---|
| `state` | `"ready" \| "ingestion_not_ready" \| "not_ready"` | Connection health (see below) |
| `message` | `str \| None` | Status detail message |
| `updated_at` | `datetime \| None` | When the status was last evaluated |

**`state` values:**

| Value | Meaning |
|---|---|
| `"ready"` | Credentials are valid and the source is queryable |
| `"ingestion_not_ready"` | Credentials are valid, but the source cannot be queried |
| `"not_ready"` | The referenced secret is missing or invalid |

### Tabular Data Queries (Flight SQL)

```python
client.read(sql, connection_id) -> pyarrow.Table          # full result as Arrow Table
client.read_pandas(sql, connection_id) -> pd.DataFrame    # full result as pandas DataFrame
client.read_batches(sql, connection_id) -> Generator[RecordBatch]  # stream of Arrow RecordBatches
client.get_tables(connection_id) -> pyarrow.Table         # table metadata
client.server_info() -> dict                              # server metadata
```

`read_batches` returns a generator that streams results instead of buffering the full result set in memory. The underlying cursor and connection are closed automatically when the generator is exhausted or garbage-collected:

```python
for batch in client.read_batches("SELECT * FROM prompts", "conn-uuid"):
    process(batch)
```

A server-side failure surfaced mid-stream raises `DCHQueryError`. Automatic token refresh applies when the stream is opened; an authentication failure that occurs after the stream is open is not retried.

These require the `flight` extra. On a REST-only install the client still imports and all REST calls work; the first Flight call raises `DCHConfigError` telling you to install `data-connect-hub[flight]`.

## Error Handling

Every failure raised by the SDK derives from `DCHError`, so a single `except` covers transport failures, HTTP errors, and malformed responses alike:

```python
from data_connect_hub import DCHError, DCHNotFoundError

try:
    conn = client.get_connection("conn-uuid")
except DCHNotFoundError:
    ...
except DCHError as exc:  # connection, timeout, auth, schema drift, ...
    ...
```

| Exception | Raised when |
|---|---|
| `DCHConfigError` | Invalid client configuration or argument (e.g. a blank id) |
| `DCHConnectionError` | The server was unreachable or the transport failed |
| `DCHTimeoutError` | The request exceeded `rest_timeout` |
| `DCHAuthenticationError` / `DCHForbiddenError` | HTTP 401 / 403 |
| `DCHNotFoundError` | HTTP 404 |
| `DCHValidationError` | HTTP 400 / 422 |
| `DCHServerError` | HTTP 5xx |
| `DCHResponseError` | The response was not JSON, or did not match the expected schema |
| `DCHQueryError` | A Flight SQL query failed |

Transient failures — HTTP 429/502/503/504, timeouts, and network or protocol errors — are retried automatically with exponential backoff on idempotent methods. See `max_retries`, `backoff_base`, and `backoff_max`.

## Requirements

- Python 3.11+
- Core dependencies: httpx, pydantic
- Flight SQL extras: adbc-driver-flightsql, pyarrow, pandas (`pip install "data-connect-hub[flight]"`)

## Releases

The package version is derived from Git history by `setuptools-scm` — there is no
version file to bump. Building therefore requires a full clone with tags; a shallow
clone or a tree without usable package metadata fails to build.

**TestPyPI (pre-release).** Run the *Publish Python SDK to TestPyPI* workflow manually
from the Actions tab. The version is a development release derived from the distance
since the last tag, for example `0.1.devN` before the first tag exists and
`0.1.1.dev12` after `sdk-v0.1.0`. Re-dispatching on an already-published commit produces
the same version and fails on the duplicate upload; land a commit first.

**PyPI (tagged release).** Push an SDK-specific tag of `sdk-v` followed by the PEP 440
version, for example `sdk-v0.1.0`; the tag is what defines the published version. The
*Release Python SDK* workflow builds and validates the distribution, publishes it to PyPI
using trusted publishing, then creates the GitHub Release.

Locally, `make sdk-package-check` builds and validates the distribution the same way CI does.

## Contributing

See [CONTRIBUTING.md](../../CONTRIBUTING.md) for development setup, commands, and contribution guidelines.
