Metadata-Version: 2.1
Name: dlc-sql-python
Version: 1.0.1
Summary: A Python DBAPI 2.0 client for Tencent Cloud DLC (Data Lake Compute) via HiveServer2 Thrift
Author: DLC Team
License: Apache-2.0
Keywords: dlc,tencentcloud,thrift,hiveserver2,dbapi,dbt
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Topic :: Database
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: dev

# dlc-sql-python

A Python **DBAPI 2.0** ([PEP 249](https://peps.python.org/pep-0249/)) client for
**Tencent Cloud DLC (Data Lake Compute)**, connecting to its
HiveServer2-compatible Thrift endpoint (TCLIService).

It is the connection layer used by [`dbt-dlc`](https://github.com/dbt-labs) (a
dbt adapter that extends `dbt-spark`), and is the Python counterpart of the
TypeScript `dlc-sql-nodejs` reference implementation.

## Features

- **DBAPI 2.0 compliant** — `apilevel='2.0'`, `paramstyle='pyformat'`, full
  `connect` / `cursor` / `execute` / `fetchone` / `fetchmany` / `fetchall` /
  `description` / `rowcount` API and PEP 249 exception hierarchy.
- **Three transport modes**
  - `binary` (default) — TCP socket + custom SASL PLAIN handshake + framed
    `TBinaryProtocol`.
  - `http` — HTTP POST with `X-DLC-*` auth headers.
  - `https` — HTTPS POST with `X-DLC-*` auth headers (production / public
    endpoints).
- **Two authentication modes**
  - `AccessKey` — Tencent Cloud SecretId / SecretKey.
  - `RoleBasedSSO` — role-based SSO with a pre-obtained access token (Manual),
    plus an `OAuthManager` helper for **M2M** (client_credentials) and **U2M**
    (authorization_code + PKCE, browser login) token acquisition.
- **Async execution + polling** — `ExecuteStatement(runAsync=true)` → poll
  `GetOperationStatus` → `GetResultSetMetadata` → `FetchResults`.
- **Columnar result parsing** — `TColumn` union (bool/byte/i16/i32/i64/double/
  string/binary) + nulls bitmap → Python values.

## Installation

```bash
pip install dlc-sql-python
```

From source (development):

```bash
cd dlc-sql-python
pip install -e ".[dev]"
pytest
```

## Quick start

```python
import dlc_sql

conn = dlc_sql.connect(
    host="dlc-thrift.example.com",
    engine_name="my-spark-engine",
    resource_group_name="my-resource-group",
    catalog="hive",
    database="default",
    auth_mode="AccessKey",
    transport_mode="https",      # or "binary" / "http"
    port=443,
    secret_id="AKIDxxxx",        # via env var in production
    secret_key="xxxx",
)
cursor = conn.cursor()
cursor.execute("SELECT id, name FROM users WHERE id = %(id)s", {"id": 1})
print(cursor.description)
print(cursor.fetchall())
cursor.close()
conn.close()
```

## Connection parameters

| Parameter | Required | Default | Description |
|---|---|---|---|
| `host` | yes | — | DLC Thrift endpoint host |
| `engine_name` | yes | — | Spark engine name |
| `resource_group_name` | yes | — | Resource group name |
| `catalog` | yes | — | Catalog (e.g. `hive`) |
| `database` | yes | — | Database (`schema` is accepted as an alias) |
| `auth_mode` | yes | — | `AccessKey` or `RoleBasedSSO` |
| `transport_mode` | no | `binary` | `binary` / `http` / `https` |
| `port` | no | `10009` (binary/http), `443` (https) | Thrift port |
| `http_path` | no | `/cliservice` | HTTP path (http/https) |
| `socket_timeout_ms` | no | `60000` | Transport timeout |
| `secret_id` / `secret_key` | AccessKey | — | Tencent Cloud AK/SK |
| `role_arn` / `identity_provider_name` / `access_token` | RoleBasedSSO | — | SSO role + token (Manual) |
| `query_timeout_seconds` | no | `0` (unlimited) | Server-side query timeout |
| `poll_interval_ms` | no | `500` | Status polling interval |
| `max_rows` | no | `10000` | Rows per `FetchResults` batch |
| `start_session_props` | no | `{}` | Extra OpenSession configuration |
| `use_arrow_result` / `arrow_batch_size` | no | `False` / `10000` | Kyuubi Arrow result format (enhancement) |

> **Security:** never hard-code credentials. Pass `secret_id` / `secret_key` /
> `access_token` via environment variables. HTTPS mode verifies server
> certificates with the system CA store.

## Authentication

### AccessKey (binary)

The SASL username encodes `engine&rg&AKSK[&catalog[&database]]` (`AKSK` is the
wire tag for the `AccessKey` auth mode) and the password encodes
`secretId&secretKey`; the SDK assembles these automatically.

### RoleBasedSSO

Pass a pre-obtained `access_token`, or resolve one with `OAuthManager`:

```python
from dlc_sql.auth import OAuthManager, RoleBasedSSOAuth

# M2M (client_credentials) — CI/CD
mgr = OAuthManager(
    token_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
    client_id="<client_id>",
    client_secret="<client_secret>",
    scopes=["api://dlc/.default"],
)
token = mgr.get_access_token()

# U2M (authorization_code + PKCE, browser) — local dev
mgr = OAuthManager(
    token_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
    authorize_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/authorize",
    client_id="<client_id>",
    scopes=["api://dlc/.default", "offline_access"],
    callback_port=8030,
    persistence_path="~/.dlc/token_cache.json",
)
token = mgr.get_access_token()
```

Then connect with `auth_mode="RoleBasedSSO"`, `role_arn=...`,
`identity_provider_name=...`, `access_token=token`.

## Architecture

```
dlc_sql.connect(...)
   └── Connection ── open_session ──┐
         └── Cursor ── execute / fetch
               │
   ┌───────────┴───────────────┐
   │  ThriftTransport          │  binary | http | https
   │   └ send_request(bytes)   │
   └───────────┬───────────────┘
               │
   thrift_api/codec.create_client(transport)
        → TCLIService.Client over TBinaryProtocol
               │
   dlc_sql/thrift_api/TCLIService/   (vendored from PyHive)
```

The codec bridges our custom `ThriftTransport` (which carries raw
`TBinaryProtocol` request/response bytes — handling the custom SASL PLAIN
handshake and framing, or HTTP POST) onto a Thrift `TTransport` so the vendored
`TCLIService.Client` drives the RPCs directly. This avoids the `thrift_sasl`
dependency (DLC's SASL is a custom variant) while keeping the full Thrift
message envelope correct.

### Module layout

```
dlc_sql/
├── __init__.py          # DBAPI 2.0 entry point (apilevel, connect, exceptions)
├── connection.py        # Connection + connect()
├── cursor.py            # Cursor (execute / poll / fetch)
├── session.py           # OpenSession/CloseSession + config assembly
├── exceptions.py        # PEP 249 + DLC-specific exceptions
├── types.py             # options, type map, columnar result + schema parsing
├── auth/
│   ├── base.py          # AuthProvider ABC
│   ├── access_key.py    # AccessKeyAuth (AK/SK)
│   ├── role_sso.py      # RoleBasedSSOAuth (Manual token)
│   ├── oauth.py         # OAuthManager (M2M + U2M/PKCE)
│   └── __init__.py      # factory + assemble_sasl_credentials
├── transport/
│   ├── base.py          # ThriftTransport ABC
│   ├── binary.py        # BinaryTransport (SASL + framed)
│   ├── http.py          # HttpTransport (HTTP/HTTPS + X-DLC-* headers)
│   ├── factory.py       # create_transport()
│   └── __init__.py
└── thrift_api/
    ├── codec.py         # TTransport adapter → TCLIService.Client
    └── TCLIService/     # vendored from PyHive (HiveServer2 IDL)
```

## DBAPI 2.0 conformance

- `apilevel = "2.0"`, `paramstyle = "pyformat"`, `threadsafety = 1`.
- `Connection`: `cursor()`, `commit()` / `rollback()` (no-ops, DLC has no
  transactions), `close()`, context-manager support.
- `Cursor`: `execute()`, `executemany()`, `fetchone()`, `fetchmany()`,
  `fetchall()`, `cancel()`, `close()`, `description`, `rowcount` (-1, DLC does
  not report affected rows), `arraysize`, context-manager support.
- Exceptions: `Error`, `Warning`, `InterfaceError`, `DatabaseError`,
  `InternalError`, `OperationalError`, `ProgrammingError`, `IntegrityError`,
  `DataError`, `NotSupportedError` (+ `DlcSqlConnectionError`,
  `DlcSqlAuthError`, `DlcSqlOperationError` subclasses).

## Relationship to dbt-dlc

`dbt-dlc` extends `dbt-spark`, reusing all Spark SQL macros and overriding only
the connection layer to use `dlc-sql-python`. See the companion spec in
`docs/specs/` for the `dbt-dlc` design.

## License

Apache License 2.0.
