Metadata-Version: 2.1
Name: dbt_dlc
Version: 1.0.0
Summary: A dbt adapter for Tencent Cloud DLC (Data Lake Compute), built on dbt-spark
Author: DLC Team
License: Apache-2.0
Keywords: dbt,dlc,tencentcloud,spark,dataplatform
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Topic :: Database
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: dev

# dbt-dlc

A [dbt](https://www.getdbt.com/) adapter for **Tencent Cloud DLC (Data Lake Compute)** — a serverless Spark SQL engine.

`dbt-dlc` extends [dbt-spark](https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-spark), replacing the connection layer with `dlc-sql-python` (a DBAPI 2.0 Thrift client) while inheriting the full Spark SQL dialect and macros.

---

## Installation

Requires Python **3.9+** and dbt **1.8+**.

```bash
pip install dbt-dlc
```

Verify:

```bash
dbt --version
# Should include: dlc: 0.1.0
```

---

## Quick Start

### 1. Create a `profiles.yml` (`~/.dbt/profiles.yml`)

```yaml
my_dlc_project:
  target: dev
  outputs:
    dev:
      type: dlc
      host: dlc-thrift.example.com           # DLC Thrift endpoint
      port: 10009                             # default: 10009
      engine_name: my-engine                  # DLC engine name
      resource_group_name: default-rg-xxxx    # resource group
      catalog: DataLakeCatalog                # catalog name (default: DataLakeCatalog)
      schema: my_database                     # DLC database name
      auth_mode: AccessKey                    # "AccessKey" or "RoleBasedSSO"
      transport_mode: binary                  # "binary" | "http" | "https"
      http_path: /cliservice                  # HTTP path (default)
      secret_id: AKIDxxxxxxxxxxxxxxxx         # Tencent Cloud SecretId
      secret_key: xxxxxxxxxxxxxxxx            # Tencent Cloud SecretKey
      socket_timeout_ms: 300000               # 5 min for engine cold start
      threads: 4
```

### 2. Create `dbt_project.yml`

```yaml
name: my_dlc_project
version: '1.0'
config-version: 2
profile: my_dlc_project

model-paths: ["models"]

models:
  my_dlc_project:
    materialized: table
    +file_format: iceberg               # recommended for DLC
```

### 3. Write your first model (`models/my_first_model.sql`)

```sql
select
    1 as id,
    'hello' as name,
    current_timestamp() as created_at
```

### 4. Run

```bash
dbt debug           # verify connection
dbt run             # build models
dbt test            # run data quality tests
```

---

## Authentication

### AccessKey (Development)

```yaml
auth_mode: AccessKey
secret_id: AKIDxxxxxxxxxx
secret_key: xxxxxxxxxxxxxx
```

### RoleBasedSSO — OAuth M2M (CI/CD)

```yaml
auth_mode: RoleBasedSSO
auth_flow: m2m
role_arn: qcs::cam::uin/xxxxxxxxxx:roleName/role-name
identity_provider_name: dlc_connect
oauth_token_url: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
oauth_client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
oauth_client_secret: xxxxxxxxxxxxxx
```

### RoleBasedSSO — OAuth U2M (Local Dev)

```yaml
auth_mode: RoleBasedSSO
auth_flow: u2m
role_arn: qcs::cam::uin/xxxxxxxxxx:roleName/role-name
identity_provider_name: dlc_connect
oauth_token_url: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
oauth_authorize_url: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/authorize
oauth_client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
oauth_client_secret: xxxxxxxxxxxxxx
oauth_persistence_path: ~/.dlc/token_cache.json
```

---

## Model Configuration

### Table

```yaml
models:
  my_dlc_project:
    materialized: table
    +file_format: iceberg
```

### View

```yaml
models:
  my_dlc_project:
    staging:
      materialized: view
```

### Incremental (Merge / Append / Insert Overwrite)

```sql
{{
  config(
    materialized='incremental',
    file_format='iceberg',
    unique_key='event_id',
    strategy='merge'
  )
}}

select event_id, event_name, event_date
from source_events
{% if is_incremental() %}
  where event_date > (select max(event_date) from {{ this }})
{% endif %}
```

Supported strategies:

| Strategy | Description |
|---|---|
| `merge` | `MERGE INTO` — upsert based on `unique_key` |
| `append` | `INSERT INTO` — append-only |
| `insert_overwrite` | `INSERT OVERWRITE` — replace partition or entire table |

### Table Properties

```sql
{{
  config(
    materialized='table',
    file_format='iceberg',
    partition_by=['event_date'],
    clustered_by=['user_id'],
    buckets=16,
    location_root='cosn://my-bucket/path',
    tblproperties={'write.format.default': 'parquet'},
    persist_docs={'relation': true, 'columns': true},
    comment="My table description"
  )
}}
```

### Seed Configuration

```yaml
seeds:
  my_dlc_project:
    +quote_columns: false
    +file_format: iceberg
```

---

## Three-Level Namespace

DLC uses **catalog.database.table** naming. `dbt-dlc` maps these as follows:

| dbt Term | DLC Layer | Profile Field |
|---|---|---|
| `database` | Catalog | `catalog` (default: `DataLakeCatalog`) |
| `schema` | Database | `schema` |
| `identifier` | Table/View | model name / alias |

---

## Iceberg Procedures (Convenience Macros)

Available as post-hook macros for managing Iceberg tables:

### Expire Old Snapshots

```sql
{{ config(
    post_hook="
      {{ dlc__expire_snapshots(this, retain_last=100) }}
    "
) }}
```

### Compaction / Re-sort

```sql
{{ config(
    post_hook="
      {{ dlc__rewrite_data_files(this, strategy='sort', sort_order='id DESC NULLS LAST') }}
    "
) }}
```

### Remove Orphan Files

```sql
{{ dlc__remove_orphan_files(this, older_than=\"TIMESTAMP '2026-06-01'\") }}
```

### Rollback to a Snapshot

```sql
{{ dlc__rollback_to_snapshot(this, snapshot_id=123456789) }}
```

---

## Known Limitations

| Item | Detail |
|---|---|
| Python models | Not yet supported |
| `ALTER TABLE RENAME` | DLC does not support `ALTER TABLE RENAME`; `dbt-dlc` uses `CREATE OR REPLACE` as a workaround |
| `USE CATALOG` | DLC does not support `USE CATALOG`; catalog is set via session property `spark.sql.defaultCatalog` |
| `clustered_by` / `buckets` | DLC may not support these options; use `sort_order` + Iceberg compaction instead |
| Transactions | Not supported (DLC/Spark has no transactions) |

---

## Testing

```bash
# Unit tests (no DLC connection required)
pytest tests/ -v

# Integration tests (requires DLC connection — see tests/integration/.env.example)
cd tests/integration
cp .env.example .env        # fill in your credentials
./run_tests.sh full dev     # full lifecycle: debug → seed → run → run → test → docs
```

---

## Project Structure

```
dbt-dlc/
├── dbt/
│   ├── adapters/dlc/
│   │   ├── __init__.py         # Plugin registration
│   │   ├── connections.py      # DLCCredentials + DLCConnectionManager
│   │   ├── impl.py             # DLCAdapter
│   │   └── relation.py         # DLCRelation (3-level namespace)
│   └── include/dlc/
│       ├── dbt_project.yml
│       └── macros/
│           └── adapters.sql    # Spark SQL macros + DLC overrides
├── tests/
│   ├── test_connections.py
│   ├── test_relation.py
│   ├── test_macros.py
│   └── integration/
└── docs/plans/
```


