Metadata-Version: 2.3
Name: airbyte-databricks-cache
Version: 0.1.19
Summary: 
Author: ossmht
Author-email: ossmht@gmail.com
Requires-Python: >=3.11,<3.12
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: airbyte (>=0,<1)
Requires-Dist: databricks-sql-connector (>=3.6.0,<4.0.0)
Description-Content-Type: text/markdown

# airbyte_databricks_cache

Databricks cache implementation for PyAirbyte

## Installation

```sh
pip install airbyte # pre-requisite
pip install airbyte-databricks-cache

```


## Usage

```py

import airbyte as ab
import airbyte_databricks_cache  # Must import to inject the module into airbyte.caches.databricks
from airbyte.caches.databricks import DatabricksCache # pylint: disable=E0401:import-error

# create airbyte source
source = ab.get_source(...)

# create DatabricksCache
cache_dbks = DatabricksCache(
    access_token = ab.get_secret("databricks_access_token"),
    server_hostname = ab.get_secret("databricks_server_hostname"),
    http_path= ab.get_secret("databricks_http_path"),
    catalog = ab.get_secret("databricks_catalog"),
    schema_name = ab.get_secret("databricks_target_schema"),
    staging_volume_w_location = ab.get_secret("databricks_staging_volume_w_location")
)

# read into cache
result = source.read(
    cache=cache_dbks,
    streams=['xxx'],
    force_full_refresh=False,
    write_strategy="append"
)

### OR
# write to destination
destination = ab.get_destination("destination-databricks", cache_dbks)


destination.write(source, ...)
```


## Build and deploy

Happens via github workflow.

### If building locally, manually:

```py
poetry run python build.py
poetry build --format wheel
```

Run locally to test that import works:
`poetry run python -c "import airbyte_databricks_cache"`

### Release mechanics:

```sh

# release
git fetch --tags origin
git describe --tags --abbrev=0
gh release create v0.1.8 --generate-notes
# gh release create v0.1.8 --target feat-workflow --generate-notes
git fetch --tags origin


# delete local tags if remote is deleted
git fetch origin --prune --prune-tags

## manual addons
# check long desc for pypi
twine check dist/*
```

