Metadata-Version: 2.4
Name: dbt-spark-livy-iceberg
Version: 1.3.3
Summary: The dbt-spark-livy-iceberg adapter plugin for Spark in Cloudera with the Livy interface, with Apache Iceberg support and dynamic Spark session configuration
Home-page: https://github.com/cloudera/dbt-spark-livy
Author: Cloudera
Author-email: innovation-feedback@cloudera.com
Classifier: Development Status :: 5 - Production/Stable
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Requires-Dist: dbt-core~=1.3.0
Requires-Dist: sqlparams>=3.0.0
Requires-Dist: requests_kerberos==0.14
Requires-Dist: requests-toolbelt>=0.9.1
Requires-Dist: python-decouple>=3.6
Provides-Extra: odbc
Requires-Dist: pyodbc>=4.0.30; extra == "odbc"
Provides-Extra: pyhive
Requires-Dist: PyHive[hive]<0.7.0,>=0.6.0; extra == "pyhive"
Requires-Dist: thrift<0.16.0,>=0.11.0; extra == "pyhive"
Provides-Extra: session
Requires-Dist: pyspark<4.0.0,>=3.0.0; extra == "session"
Provides-Extra: all
Requires-Dist: pyodbc>=4.0.30; extra == "all"
Requires-Dist: PyHive[hive]<0.7.0,>=0.6.0; extra == "all"
Requires-Dist: thrift<0.16.0,>=0.11.0; extra == "all"
Requires-Dist: pyspark<4.0.0,>=3.0.0; extra == "all"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# dbt-spark-livy-iceberg

The `dbt-spark-livy-iceberg` adapter lets you use [dbt](https://www.getdbt.com/) with [Apache Spark](https://spark.apache.org/) on [Cloudera Data Platform](https://cloudera.com) through the Livy interface. It is based on the [dbt-spark](https://github.com/dbt-labs/dbt-spark) project and the Cloudera `dbt-spark-livy` adapter, extended with:

- **Full Apache Iceberg support** — `file_format='iceberg'` for tables, incremental models (`append`, `merge`, `insert_overwrite`), seeds, and snapshots.
- **`incremental_predicates` for merges** — pruning predicates are now passed into `MERGE INTO ... ON`, which the base adapter dropped.
- **Iceberg table properties** — a `tblproperties` config for setting Iceberg options like `format-version` and `write.merge.mode`.
- **Dynamic Spark session configuration** — a per-model `spark_session_config` config that issues `SET key=value` statements before the model runs.

The internal adapter type is still `spark_livy`, so existing profiles (`method: livy`) keep working. Only the pip distribution name changed.

## Getting started

- [Install dbt](https://docs.getdbt.com/docs/installation)
- Read the [introduction](https://docs.getdbt.com/docs/introduction/) and [viewpoint](https://docs.getdbt.com/docs/about/viewpoint/)

## Requirements

- Python >= 3.8
- dbt-core ~= 1.3.0
- pyspark, sqlparams, requests_kerberos, requests-toolbelt, python-decouple

### Installing

```
pip install dbt-spark-livy-iceberg
```

### Profile setup

```yaml
demo_project:
  target: dev
  outputs:
    dev:
     type: spark_livy
     method: livy
     schema: my_db
     host: https://spark-livy-gateway.my.org.com/dbt-spark/cdp-proxy-api/livy_for_spark3/
     user: my_user
     password: my_pass
     # Optional: Spark conf applied once at Livy session creation.
     # Use this for catalog wiring (e.g. Iceberg catalog + extensions).
     livy_session_parameters:
       spark.sql.extensions: "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
       spark.sql.catalog.spark_catalog: "org.apache.iceberg.spark.SparkSessionCatalog"
       spark.sql.catalog.spark_catalog.type: "hive"
```

## Iceberg models

### Incremental merge

```sql
{{ config(
    materialized='incremental',
    file_format='iceberg',
    incremental_strategy='merge',
    unique_key='id',
    incremental_predicates=[
      "DBT_INTERNAL_DEST.event_date >= current_date - interval 7 days"
    ],
    tblproperties={
      'format-version': '2',
      'write.merge.mode': 'merge-on-read'
    }
) }}

select id, event_date, payload
from {{ ref('source_events') }}
{% if is_incremental() %}
where event_date >= current_date - interval 7 days
{% endif %}
```

- `incremental_strategy` supports `append`, `merge`, and `insert_overwrite` for Iceberg.
- `incremental_predicates` are appended to the merge `ON` clause to enable partition pruning.
- `on_schema_change: sync_all_columns` is supported for Iceberg (including dropping columns).

## Dynamic Spark session configuration

Set Spark conf per model with `spark_session_config`. This issues `SET key=value` before the model builds.

```sql
{{ config(
    materialized='incremental',
    file_format='iceberg',
    incremental_strategy='insert_overwrite',
    partition_by=['event_date'],
    spark_session_config={
      'spark.sql.shuffle.partitions': '400',
      'spark.sql.sources.partitionOverwriteMode': 'dynamic'
    }
) }}
select ...
```

You can also set project-wide defaults in `dbt_project.yml`:

```yaml
models:
  my_project:
    +spark_session_config:
      spark.sql.shuffle.partitions: '200'
    +tblproperties:
      write.merge.mode: 'merge-on-read'
```

> **Note:** The Livy connection reuses a single shared SQL session for the whole dbt run, so `SET` values persist across models. Set the properties each model needs (or a project-level default) rather than assuming they reset between models.

### Caveats
- While using livy, in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section.
- Actions > Manage Access > IDBroker Mappings. [Reference](https://docs.cloudera.com/cdf-datahub/7.2.15/flink-analyzing-data/topics/cdf-datahub-sa-create-idbroker-mapping.html)
- Also make sure the workload password is set either through UI or CLI. [Reference](https://docs.cloudera.com/management-console/cloud/user-management/topics/mc-setting-the-ipa-password.html)
- Iceberg catalog/extension wiring must be provided via Spark conf (`livy_session_parameters` in the profile, or `spark_session_config` per model).
- Python models currently write with the Delta writer; Iceberg support here targets SQL models.

## Supported features
See the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile
