Metadata-Version: 2.4
Name: fivetran-connector-sdk-core
Version: 0.1.0
Summary: The core that runs a Fivetran Connector SDK connector on your machine and writes what it emits to DuckDB
Author-email: Fivetran <developers@fivetran.com>
License: Fivetran Connector SDK Core License Agreement
Project-URL: Homepage, https://fivetran.com/docs/connectors/connector-sdk
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: duckdb==1.5.6
Requires-Dist: requests==2.34.2
Dynamic: license-file

# **fivetran-connector-sdk-core**
[![Downloads](https://static.pepy.tech/badge/fivetran-connector-sdk-core)](https://pepy.tech/project/fivetran-connector-sdk-core)

The *fivetran-connector-sdk-core* is the engine behind `fivetran debug` and `fivetran run` in the [Fivetran Connector SDK](https://fivetran.com/docs/connectors/connector-sdk). It runs your connector on your own machine, in the current Python process, and writes what the connector emits to a local [DuckDB](https://duckdb.org/) file (or, with `fivetran run`, to an Iceberg catalog), applying the rules Fivetran applies in production: types, renaming, primary keys, deletes, truncates, files, checkpoints and state.

You do not install or call it yourself. It is installed with `fivetran-connector-sdk` and has no command of its own.

## **Install**
Installed automatically with `fivetran-connector-sdk`. To install it on its own:

    pip install fivetran-connector-sdk-core

## **Requirements**
- Python version ≥3.10 and ≤3.14
- Operating system:
  - Windows: 10 or later (64-bit only)
  - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
  - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)

## **Getting started**
See [Quickstart guide](https://fivetran.com/docs/connectors/connector-sdk/quickstart-guide) to get started, then run your connector locally:

    fivetran debug

The data lands in `files/warehouse.db` in your project folder, which you can open with `duckdb -ui files/warehouse.db`.

## **Writing to an Iceberg catalog**
`fivetran run` runs your connector like `fivetran debug`, with the same options and the same logs, and writes what it emits to an [Apache Iceberg](https://iceberg.apache.org/) REST catalog instead of the local DuckDB file. It reads the catalog from the file given with `--destination-file <path>`; when that is not given, from the `destination.json` in your project folder, beside your `connector.py`. If neither exists, the command stops and says so. The file looks like this:

```json
{
  "uri": "https://my-catalog.example.com/api/catalog",
  "warehouse": "my_warehouse",
  "namespace": "my_connector",
  "client_id": "${CATALOG_CLIENT_ID}",
  "client_secret": "${CATALOG_CLIENT_SECRET}",
  "header.X-Iceberg-Access-Delegation": "vended-credentials",
  "s3.region": "us-east-1"
}
```

The names are the ones Iceberg clients (PyIceberg, Spark) give the properties of a REST catalog and of its S3 storage; `namespace` is the only setting of our own. A catalog that does not vend credentials is given the storage instead:

```json
{
  "uri": "http://localhost:8181/api/catalog",
  "warehouse": "local_catalog",
  "namespace": "tester",
  "client_id": "${CATALOG_CLIENT_ID}",
  "client_secret": "${CATALOG_CLIENT_SECRET}",
  "s3.region": "us-east-1",
  "s3.endpoint": "http://localhost:8333",
  "s3.force-virtual-addressing": false,
  "s3.access-key-id": "${S3_ACCESS_KEY}",
  "s3.secret-access-key": "${S3_SECRET_KEY}"
}
```

| Setting | |
|---|---|
| `uri`, `warehouse`, `namespace` | Required. The catalog, the warehouse in it, and the namespace your tables go in: it is what `tester` is in a local run. A table that declares a schema goes in `<namespace>_<schema>` (lower-cased) under Fivetran naming, and in a namespace named as declared under source naming. A namespace that is not there is created. |
| `token`, or `client_id` and `client_secret` | Required: one or the other. `scope` and `oauth2-server-uri` go with the client credentials; the token is asked for at `<uri>/v1/oauth/tokens` unless you say where. |
| `header.X-Iceberg-Access-Delegation` | `vended-credentials`: the catalog gives out the credentials for its storage, and only `s3.region` may be given besides, for a catalog whose credentials do not carry one. `none`, or left out (the default): it does not, and the `s3.*` settings say how to reach the storage. |
| `s3.region` | Required unless the catalog vends credentials. |
| `s3.access-key-id` and `s3.secret-access-key` (and `s3.session-token`) | Static keys. Without them the credential chain of the machine is used, with `s3.profile-name` if you name one. |
| `s3.endpoint`, `s3.force-virtual-addressing` | For storage that is not AWS: its address (an `http://` scheme means no SSL), and whether buckets are addressed as host names (the default) or by path (`false`). |

A value may read an environment variable: `"${NAME}"`, in the whole value or in a part of it. A credential written in the file can be committed with it, so `fivetran run` warns about one unless `.gitignore` lists the file.

What to know:

- **Same as a local run.** Tables, types, primary keys, schema changes, deletes and truncates are what `fivetran debug` does. Each checkpoint is one commit, and the state is saved to `files/state.json` after it, so run from the same project folder. `fivetran reset` deletes that folder's `files/`, not the tables in the catalog.
- **Primary keys** are recorded in the table: the key columns are required and named as its identifier fields. A key that changes rewrites the table if its rows fit the new key, and the sync fails if they do not.
- **Types.** Iceberg has no `SMALLINT` and no JSON: they are stored as integer and string. An integer becomes a bigint, a float a double and a decimal one of more digits in place; any other change of a column's type rewrites the table.
- **Naming.** The naming strategy of the first run is kept as the property `fivetran.naming_strategy` of the namespace, so a run from another machine uses the same one.
- **A schema change is a commit of its own.** A sync that fails part way can leave a table or a column it had already added, never rows.
- **Not supported yet:** file uploads (`op.upsert(..., file=...)`); with source naming, table and namespace names that have characters other than ASCII (DuckDB's iceberg extension cannot write them: use Fivetran naming); compaction, so data files and delete files pile up (use your catalog's own maintenance); several syncs writing one table at once.
- **Catalogs.** It is tested against Apache Polaris (locally, and with vended credentials) and Project Nessie; `test-resources/iceberg` in the package's source has a stack of each (`./setup.sh` starts one and writes its `destination.json`). A catalog that cannot commit several tables at once writes a checkpoint table by table. DuckDB downloads its iceberg extension the first time it is used.

## **How it fits with the SDK**
The core knows nothing of the SDK: the SDK hands it a `Source` that declares tables and emits operations, and the core gives that source a `Sink` to emit into. A change that breaks this contract is a new major version of the core, and the SDK asks for the core as `>=0.1,<1`. `fivetran configuration` is in the SDK, not here.

## **License**
The core is licensed under the Fivetran Connector SDK Core License Agreement. The core is not open source: its source is proprietary to Fivetran.

## **Maintenance**
This package is actively maintained by Fivetran Developers. Please reach out to our [Support team](https://support.fivetran.com/hc/en-us) for any inquiries.
