Metadata-Version: 2.4
Name: tentaclio-databricks
Version: 2.1.0
Summary: A python project containing all the dependencies for schema databricks for tentaclio.
Author-email: Octopus Energy <nerds@octopus.energy>
License-Expression: MIT
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: tentaclio
Requires-Dist: databricks-sql-connector
Requires-Dist: pandas<3
Requires-Dist: pyarrow
Provides-Extra: polars
Requires-Dist: polars>=0.16; extra == "polars"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: Flake8-pyproject; extra == "dev"
Requires-Dist: isort>=5.11.2; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Dynamic: license-file

# tentaclio-databricks

A package containing all the dependencies for the `databricks+thrift` tentaclio schema .

## Quick Start

This project comes with a `Makefile` which is ready to do basic common tasks

```
$ make help
install                       Initalise the virtual env installing deps
clean                         Remove all the unwanted clutter
lock                          Lock dependencies
update                        Update dependencies (whole tree)
sync                          Install dependencies as per the lock file
lint                          Lint files with flake and mypy
format                        Run black and isort
test                          Run unit tests
circleci                      Validate circleci configuration (needs circleci cli)
```

## Configuring access to Databricks

Your connection url should be in the following format:

```
databricks+thrift://<token>@<host>?HTTPPath=<http_path>
```

Example values:

- token: dapi1213456789abc
- host: myhost.databricks.com
- http_path: /sql/1.0/endpoints/123456789

## Query Comments

Queries can be annotated with comments for observability using the `query_annotations` parameter.

### Example: Basic usage

```python
import os
from tentaclio import URL
from tentaclio_databricks.clients.databricks_client import DatabricksClient

url = URL("databricks+thrift://token@host.databricks.com?HTTPPath=/sql/1.0/endpoints/123")
client = DatabricksClient(
    url,
    query_annotations={
        "app_name": "JupyterHub",
        "user": os.environ.get("USER_NAME", "unknown"),
        "pipeline_id": "456"
    }
)
```

### Result

All queries executed by the client will have prepended comments:

```sql
/* app_name='JupyterHub', user='john', pipeline_id='456' */
SELECT * FROM table
```

## Arrow Usage

`DatabricksClient` enables Arrow's usage for faster data transfer and more faithful type handling.

### Example: Default row-based fetch

```python
from tentaclio import URL
from tentaclio_databricks.clients.databricks_client import DatabricksClient

url = URL("databricks+thrift://token@host.databricks.com?HTTPPath=/sql/1.0/endpoints/123")
with DatabricksClient(url) as client:
    df = client.get_df("SELECT * FROM sample_table")
```

When Arrow is disabled, the client falls back to `fetchall()` and uses `arraysize` for the cursor.

### Example: Arrow enabled

```python
from tentaclio import URL
from tentaclio_databricks.clients.databricks_client import DatabricksClient

url = URL("databricks+thrift://token@host.databricks.com?HTTPPath=/sql/1.0/endpoints/123")
with DatabricksClient(url, use_arrow=True) as client:
    df = client.get_df("SELECT * FROM sample_table")
```

When Arrow is enabled, the client configures the connection with:

- `use_arrow_native_complex_types`
- `use_arrow_native_decimals`
- `use_arrow_native_timestamps`


