Metadata-Version: 2.5
Name: meltanolabs-tap-snowflake
Version: 0.5.3
Summary: Singer tap for Snowflake, built with the Meltano SDK for Singer Taps.
Author-email: Meltano Team <hello@meltano.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ELT,Meltano,Snowflake
Requires-Python: >=3.10
Requires-Dist: cryptography>=50
Requires-Dist: pyarrow>=25.0.1
Requires-Dist: singer-sdk~=0.54.5
Requires-Dist: snowflake-connector-python[secure-local-storage]~=4.7.3
Requires-Dist: snowflake-sqlalchemy==1.11.0
Requires-Dist: sqlalchemy~=2.0.52
Description-Content-Type: text/markdown

# tap-snowflake

`tap-snowflake` is a Singer tap for Snowflake.

Built with the [Meltano Tap SDK](https://sdk.meltano.com) for Singer Taps.

## Installation

```bash
pipx install git+https://github.com/MeltanoLabs/tap-snowflake.git
```

## Configuration

### Accepted Config Options

A full list of supported settings and capabilities for this
tap is available by running:

```bash
tap-snowflake --about
```

### `tables` configuration parameter.

An array of the table names that you want to sync. The table names should be fully qualified, including schema and table name.

NOTES:
* This limits discovery to the tables specified for performance reasons. Do not specify `tables` if you intend to discover the entire available catalog.
* Syntax to specify `tables` differs slightly from `select` (`schema_name.table_name` vs `schema_name-table_name.*`)

:bulb: When adding more elements to `select` ensure the table is specified in `tables` if using `tables`.

Example:
```yaml
  ...
  tables:
    - schema_name1.table1
    - schema_name1.table2
```

### Configure using environment variables

This Singer tap will automatically import any environment variables within the working directory's
`.env` if the `--config=ENV` is provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the `.env` file.

### Source Authentication and Authorization

This tap supports two authentication methods:
- Standard `username` and `password` auth
- [Key Pair Authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth)

#### Key Pair Authentication

Key Pair authentication can be enabled by specifing either the `private_key` of the `private_key_path`, and optionally the `private_key_passphrase` if required.

### Enabling Batch Messaging

This tap is built using the Meltano SDK and therefore supports a `BATCH` [message type](https://sdk.meltano.com/en/latest/batch.html), in
addition to the `RECORD` messages of the Singer spec. This can be enabled either by adding the following to your `config.json`:

```json
{
  // ...
  "batch_config": {
    "encoding": {
      "format": "jsonl",
      "compression": "gzip"
    },
    "storage": {
      "root": "file://tests/core/resources",
      "prefix": "test-batch"
    }
  }
}
```

or its equivalent to your `meltano.yml`

```yaml
config:
  plugins:
    extractors:
      - name: tap-snowflake
        config:
          batch_config:
            encoding:
              format: jsonl
              compression: gzip
            storage:
              root: "file://tests/core/resources"
              prefix: test-batch
```

**Note:** This variant of `tap-snowflake` does not yet support the `INCREMENTAL` replication strategy in `BATCH` mode. Follow [here](https://github.com/meltano/sdk/issues/976#issuecomment-1257848119) for updates.

#### `arrow` encoding

In addition to `jsonl`, `encoding.format` also accepts `arrow`:

```json
{
  "batch_config": {
    "encoding": {
      "format": "arrow"
    },
    "storage": {
      "root": "file:///tmp/tap-snowflake-batches"
    },
    "batch_size": 500000
  }
}
```

| Replication method | With `batch_config` set                |
| ------------------- | --------------------------------------- |
| `FULL_TABLE`         | `BATCH` messages                        |
| `INCREMENTAL`        | `BATCH` messages                        |

Unlike `jsonl`, which unloads rows to the Snowflake-managed internal user
stage via `COPY INTO` and then downloads them (`LIST`/`GET`), `arrow` reads
query results directly using `snowflake-connector-python`'s native Arrow
result format (Snowflake's wire format is already Arrow) -- no stage,
intermediate JSON encoding, or extra download round trip. `batch_size`
controls how many rows accumulate before a new Arrow IPC file is flushed
(default `10000`, matching the SDK's default). As with `jsonl`, batch files
are written to local storage only and are not cleaned up by the tap -- the
consumer owns their lifecycle.

## Usage

You can easily run `tap-snowflake` by itself or in a pipeline using [Meltano](https://meltano.com/).

### Executing the Tap Directly

```bash
tap-snowflake --version
tap-snowflake --help
tap-snowflake --config CONFIG --discover > ./catalog.json
```

## Developer Resources

### Initialize your Development Environment

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh # https://docs.astral.sh/uv/getting-started/installation/
uv sync
```

### Create and Run Tests

Create tests within the `tap_snowflake/tests` subfolder and
then run:

```bash
uv run pytest
```

You can also test the `tap-snowflake` CLI interface directly using `uv run`:

```bash
uv run tap-snowflake --help
```

### Testing with [Meltano](https://www.meltano.com)

_**Note:** This tap will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios._

Your project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any _"TODO"_ items listed in
the file.

Next, install Meltano (if you haven't already) and any needed plugins:

```bash
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-snowflake
meltano install
```

Now you can test and orchestrate using Meltano:

```bash
# Test invocation:
meltano invoke tap-snowflake --version
# OR run a test `elt` pipeline:
meltano elt tap-snowflake target-jsonl
```

### SDK Dev Guide

See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the SDK to
develop your own taps and targets.
