Metadata-Version: 2.4
Name: datacontract-x
Version: 0.1.9
Summary: Data Contract eXtended — AI-native, platform-extensible data contracts: LLM enrichment (descriptions, tags, data quality), live import, and apply. Built on datacontract-cli.
Author: MickaelBZH
License-Expression: MIT
Project-URL: Homepage, https://github.com/MickaelBZH/data-contract-x
Project-URL: Repository, https://github.com/MickaelBZH/data-contract-x
Project-URL: Issues, https://github.com/MickaelBZH/data-contract-x/issues
Keywords: data-contract,odcs,data-quality,data-governance,data-catalog,llm,ai,datacontract-cli,metadata,tagging,snowflake
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: datacontract-cli[avro]<0.13,>=0.12.4
Requires-Dist: typer<0.26,>=0.18.0
Requires-Dist: open-data-contract-standard<4.0.0,>=3.1.2
Requires-Dist: fastapi<0.137.0,>=0.115.0
Requires-Dist: uvicorn<1.0,>=0.30
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: snowflake-connector-python<4.0,>=3.2
Requires-Dist: litellm<2.0,>=1.50
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: httpx<1.0,>=0.27; extra == "dev"
Dynamic: license-file

<p align="center">
  <img src="assets/logo.svg" alt="dcx — Data Contract eXtended" width="520">
</p>

<h3 align="center">Data Contract e<strong>X</strong>tended — AI-native, platform-extensible data contracts</h3>

<p align="center">
  Author data contracts with an LLM, sync them with your live platforms.<br>
  A lean, no-fork extension of <a href="https://github.com/datacontract/datacontract-cli">datacontract-cli</a>, built on the <a href="https://bitol.io/">Open Data Contract Standard (ODCS)</a>.
</p>

<p align="center">
  <img alt="PyPI" src="https://img.shields.io/pypi/v/datacontract-x?color=6366F1&label=pypi">
  <img alt="Python" src="https://img.shields.io/badge/python-3.10%20|%203.11%20|%203.12-3776AB?logo=python&logoColor=white">
  <img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-22c55e">
  <img alt="ODCS" src="https://img.shields.io/badge/ODCS-v3.1-0EA5E9">
  <img alt="Built on datacontract-cli" src="https://img.shields.io/badge/built%20on-datacontract--cli-6366F1">
</p>

---

## What is dcx?

**dcx (Data Contract eXtended)** adds three things to the Open Data Contract Standard workflow that plain datacontract-cli doesn't do:

1. **AI authoring** — use an LLM to enrich a contract with column descriptions, validation constraints, governance **tags** from your own catalog, and an executable **data-quality** suite.
2. **Live import** — build a contract *from* a running system (its real columns, keys, comments, tags).
3. **Apply** — push the contract's governance *back* to the platform (comments, tags, data-quality, and the table itself).

The pipeline:

```
import  ──→  enrich  ──┬──→  apply     push governance back to the platform
                       └──→  export    to SQL / docs / schemas

  import   a live schema into an ODCS contract
  enrich   columns · tags · quality
```

Everything is available both as a **CLI** and as a **REST API** (`dcx api`).

It's **platform-extensible by design** — each platform is a small importer / exporter / apply module that plugs into datacontract-cli's factories. **Snowflake is the first end-to-end platform** (import → enrich → apply), with Kafka import today and more platforms built to slot in the same way.

## Why dcx?

- 🧠 **AI authoring that's safe to ship.** Forced tool-calling, `temperature=0`, and strict server-side validation against the ODCS schema — the model can only produce spec-valid output, never free-form guesses.
- 🏷️ **A tag *manager*, not a tag guesser.** You define a controlled [tag catalog](#the-tag-catalog) (names, allowed values, examples); the LLM classifies columns into *your* vocabulary, with optional defaults.
- ✅ **Executable, portable data quality.** Quality rules prefer ODCS `library` metrics (portable, mappable to platform-native checks) and fall back to portable `sql` checks — across all seven ODCS dimensions.
- 🔌 **Any LLM provider.** Powered by [litellm](https://github.com/BerriAI/litellm) — Anthropic, OpenAI, Azure, Bedrock, Gemini, Ollama, … behind one `--model` flag.
- 🧩 **Pluggable platforms, no fork.** You keep all 30+ upstream importers/exporters and `lint` / `test` / `changelog`, and gain the AI + platform layer on top.
- 🔐 **Auth that makes sense per surface.** Live platform operations over the API use **caller-supplied credentials** — OAuth, key-pair, or password, never the server's; on the CLI, secrets are never flags.

## Install

```bash
pip install datacontract-x
```

The import package and CLI are both `dcx`:

```bash
dcx --help
dcx info
```

From source (for development):

```bash
git clone https://github.com/MickaelBZH/data-contract-x.git
cd data-contract-x
pip install -e ".[dev]"
```

> Requires Python 3.10–3.12. Installing pulls in `datacontract-cli`, `litellm`, FastAPI, and the platform connectors automatically.

## Quickstart

The full loop — import a live schema, enrich it with an LLM, sync it back. Snowflake here is the example platform.

```bash
# 1. Import an existing schema into a contract (real columns, PKs, comments, tags)
dcx import snowflake --database MY_DB --schema LOAD --authenticator externalbrowser --output contract.yaml

# 2. Enrich with an LLM: descriptions + constraints + tags + data-quality tests
export ANTHROPIC_API_KEY=...           # or OPENAI_API_KEY / AZURE_API_KEY / ...
dcx enrich all contract.yaml --catalog tags_catalog.yaml --output contract.enriched.yaml

# 3. Preview exactly what will run — no connection needed
dcx apply snowflake contract.enriched.yaml --include-quality --dry-run

# 4. Apply it: creates the table if missing, governs it (comments + tags + DQ) if it exists
dcx apply snowflake contract.enriched.yaml --include-quality
```

---

## Commands

Every command is `dcx <command>`, and most are mirrored to a REST endpoint when you run [`dcx api`](#rest-api). Each section below lists the sub-commands, a CLI example, and the matching API call. Run `dcx <command> --help` for the full option list.

### `import` — build a contract from a source

| Sub-command | Source |
|---|---|
| `dcx import snowflake` | A live Snowflake schema — tables **and views** (columns, primary keys, comments, tags; `physicalType` records the asset type, and a view's SELECT body is captured as a `viewDefinition`). `--quality` additionally reads attached data metric functions back into `quality` / `slaProperties` |
| `dcx import kafka` | A Kafka topic's value schema (Confluent Schema Registry) |
| `dcx import <format>` | A file/document — `sql`, `avro`, `dbml`, `glue`, `bigquery`, `unity`, `jsonschema`, `json`, `odcs`, `parquet`, `csv`, `protobuf`, `spark`, `iceberg`, `excel`, `dbt` |

```bash
dcx import snowflake --database MY_DB --schema LOAD --authenticator externalbrowser --output contract.yaml
dcx import snowflake --database MY_DB --schema LOAD --quality --output contract.yaml   # + attached DMFs
dcx import kafka --schema-registry https://sr:8081 --topic orders --output contract.yaml
dcx import sql --source schema.sql --dialect snowflake --output contract.yaml
```

**API**
- `POST /import/snowflake` — live import, authenticated by the caller's own credentials: an `auth` block (`oauth` · `key_pair` · `password`) or an `Authorization: Bearer` token. See [Connecting to Snowflake](#connecting-to-snowflake).
- `POST /import/{format}` — file-based importers; send the document inline as `source_content`.
- *(Kafka import is CLI-only.)*

### `enrich` — AI authoring with an LLM

| Sub-command | Adds |
|---|---|
| `dcx enrich columns` | Business descriptions, `logicalTypeOptions` constraints, `required` / `unique` flags |
| `dcx enrich tags` | Governance tags, classified against your [tag catalog](#the-tag-catalog) |
| `dcx enrich quality` | An executable data-quality suite across all ODCS dimensions |
| `dcx enrich all` | columns → tags → quality, in that order so each stage grounds the next |

Each sub-command is independent and **idempotent** — existing values are preserved unless you pass `--overwrite`.

| Option | Effect |
|---|---|
| `--model` | any litellm model (`claude-opus-4-8`, `gpt-4o`, `ollama/llama3`, …) |
| `--base-url` | a proxy / Azure / Ollama endpoint |
| `--overwrite` | replace existing values instead of preserving them |

The provider key is read from the environment — there is no `--api-key` flag.

```bash
dcx enrich columns contract.yaml --output contract.enriched.yaml
dcx enrich tags    contract.yaml --catalog tags_catalog.yaml --output contract.tagged.yaml
dcx enrich quality contract.yaml --model gpt-4o --output contract.dq.yaml
dcx enrich all     contract.yaml --catalog tags_catalog.yaml --output contract.full.yaml
```

**API** (the LLM key comes from the *server's* environment)
- `POST /enrich/columns` · `POST /enrich/quality`
- `POST /enrich/tags` · `POST /enrich/all` — take the tag catalog inline in the request body.

### `export` — convert a contract to a target format

| Sub-command | Output |
|---|---|
| `dcx export snowflake-full` | A Snowflake setup script: DDL + tags + Data Metric Functions, in one file |
| `dcx export dbt` | dbt `models` / `sources` / `staging`, with ODCS governance mapped to `config.meta` / `config.tags` |
| `dcx export <format>` | Any upstream format — `sql`, `jsonschema`, `html`, `markdown`, `mermaid`, `dbt-*`, `avro`, `protobuf`, `bigquery`, `spark`, `sqlalchemy`, `iceberg`, `sodacl`, `great-expectations`, `dbml`, `pydantic-model`, `odcs`, `rdf`, `go`, `excel`, … |

#### `snowflake-full`

Emits the exact script [`apply --dry-run`](#apply--push-governance-to-a-live-platform) would, and shares its SQL-generation knobs:

`--ddl-mode` · `--structured-types` · `--comments` · `--include-tags` · `--include-quality` · `--create-tags` · `--tag-namespace` · `--tag-namespace-filter`

See the `apply` option table below for what each does. Only `--strict` has no export equivalent — drift detection needs a live connection.

#### `dbt`

Unifies upstream's `dbt-models` / `dbt-sources` / `dbt-staging-sql` under one command via `--kind` (those upstream commands remain available, unchanged), and maps ODCS governance the idiomatic dbt way:

| ODCS | → dbt | Why |
|---|---|---|
| `NAME=VALUE` tags | `config.meta` | key/value metadata for docs + catalogs |
| `classification`, `businessName`, `criticalDataElement` | `config.meta` | same |
| bare tags | `config.tags` | dbt selection labels |
| schema-level tags | model `config` | upstream's models exporter drops these |

| Option | Effect |
|---|---|
| `--kind models\|sources\|staging` | which artifact to emit (default `models`) |
| `--meta-key-style full\|sanitized\|short` | how a qualified Snowflake tag `DB.SCHEMA.NAME` appears in the meta key: `db.schema.name` · `db_schema_name` · `name` |
| `--tag-namespace-filter DB.SCHEMA` | repeatable — emit only tags from these namespaces |

```bash
dcx export snowflake-full contract.yaml --include-quality --create-tags --output setup.sql
dcx export snowflake-full contract.yaml --ddl-mode never --output govern.sql   # alter-only
dcx export dbt contract.yaml --kind models --server production --output schema.yml
dcx export html contract.yaml --output contract.html
```

**API**
- `POST /export/{format}` — including `POST /export/snowflake-full` and `POST /export/dbt` (`{options: {kind: "models"}}`). The response media type depends on the format (JSON / YAML / text / binary).

### `apply` — push governance to a live platform

| Sub-command | Target |
|---|---|
| `dcx apply snowflake` | A live Snowflake account |

With the default `--ddl-mode auto` you don't need to know whether the table exists:

- **missing** → created with `CREATE TABLE IF NOT EXISTS`
- **existing** → governed: column/table comments, tags, and (with `--include-quality`) data-quality metrics

For existing tables dcx also compares the live schema to the contract and reports **drift** as warnings — or, with `--strict`, an error that aborts before any change. The check uses `DESCRIBE TABLE`, so it needs no active warehouse.

| Option | Effect |
|---|---|
| `--ddl-mode auto\|always\|never` | create-if-missing-then-govern (default) · always `CREATE TABLE` · govern existing only |
| `--strict` | fail instead of warn on schema drift |
| `--structured-types` | typed nested `OBJECT(...)` / `ARRAY(...)` |
| `--include-quality` · `--create-tags` · `--tag-namespace` | data-metric functions · `CREATE TAG IF NOT EXISTS` · qualify *bare* tag refs (already-namespaced `DB.SCHEMA.NAME` tags are left as-is) |
| `--tag-namespace-filter DB.SCHEMA` | repeatable — apply only tags from these namespaces (skip centrally-managed/inherited ones); un-namespaced tags are skipped |
| `--dry-run` | print the SQL without connecting |

```bash
dcx apply snowflake contract.yaml --dry-run            # preview
dcx apply snowflake contract.yaml --include-quality    # create-or-govern
```

#### Views

Objects with `physicalType: view` are governed as views — tags, comments and DQ use `ALTER VIEW` / `COMMENT ON VIEW`. This holds for both `apply snowflake` and `export snowflake-full`.

Column comments are the catch. Snowflake persists them **only** inside the `CREATE VIEW` column list — there is no `ALTER` path (Snowsight uses the same trick). So dcx has to recreate the view, which needs the `viewDefinition` captured on `import`:

| `--ddl-mode` | With a `viewDefinition` | Without one |
|---|---|---|
| `always` | `CREATE OR REPLACE VIEW` — **column comments updated** | view comment + column tags only |
| `auto` (default) | `CREATE VIEW IF NOT EXISTS` — column comments land on first creation only | view comment + column tags only |
| `never` | view comment + column tags only | view comment + column tags only |

> **To update an existing view's column comments, use `--ddl-mode always`.** Every other combination leaves them as they are, and dcx notes each skip.

Selective imports query `INFORMATION_SCHEMA.VIEWS` only when Snowflake reports
`TABLE_TYPE = 'VIEW'`. This includes regular secure views: security is exposed as a
separate view attribute, not a different table type. Snowflake reports materialized
views as `MATERIALIZED VIEW`, and they do not appear in `INFORMATION_SCHEMA.VIEWS`;
their definition requires `SHOW MATERIALIZED VIEWS` or `GET_DDL`. dcx therefore
imports materialized and external tables with their real `physicalType`, but does not
capture a materialized-view `viewDefinition` and currently governs both as tables.

**API**
- `POST /apply/snowflake` — authenticated by the caller's own credentials (see [Connecting to Snowflake](#connecting-to-snowflake); `dry_run` needs none). Supports `dry_run`, `ddl_mode`, `strict`, `structured_types`, `tag_namespace_filter`, … (all under `options`) and returns the executed SQL plus any drift `warnings`.

### `target` — bind a contract to a platform

`dcx target <type>` does two things: sets the contract's **server block**, and resolves each column's **`physicalType`** for that platform.

~30 types — `snowflake`, `bigquery`, `databricks`, `postgres`, `redshift`, `mysql`, `sqlserver`, `oracle`, `s3`, `kafka`, `trino`, `athena`, `glue`, `duckdb`, `local`, …

```bash
dcx target snowflake contract.yaml --output contract.snowflake.yaml
```

**API**
- `POST /target/{type}` — one route per supported platform type.

### From datacontract-cli

These commands work unchanged — `dcx <command>` behaves exactly like `datacontract <command>`.

| Command | Sub-commands | Purpose | API |
|---|---|---|---|
| `dcx init` | — | Create an empty data contract | — |
| `dcx lint` | — | Validate a contract against the ODCS schema | `POST /lint` |
| `dcx test` | — | Run schema + data-quality tests against a configured server | `POST /test` |
| `dcx ci` | — | `test` for CI/CD — emits GitHub Actions annotations | — |
| `dcx changelog` | — | Semantic changelog between two contract versions | `POST /changelog` |
| `dcx catalog` | — | Render an HTML catalog of many contracts | — |
| `dcx publish` | — | Publish a contract to Entropy Data | — |
| `dcx dbt` | `sync` | Sync contracts into a dbt project | — |

### `api` / `info`

```bash
dcx api --port 4242      # start the REST server (Swagger UI at /docs)
dcx info                 # show dcx + datacontract-cli versions   (API: GET /info)
```

---

## Connecting to Snowflake

Two rules cover everything below:

1. **Credentials differ per surface.** The CLI takes secrets from the environment or your Snowflake connection profile — never from a flag. The API takes them from the request, so the server acts as the caller rather than with one shared identity.
2. **Where the objects land never depends on the credentials.** Every generated statement is qualified `DATABASE.SCHEMA.OBJECT` from the contract's **server block**. A connection profile, an env var, or an OAuth token decides *who you are*, never *what you touch*.

To apply the same contract to a different database, name a different server block — `--server dev` on the CLI, `"server_name": "dev"` in the API — or edit the contract. There is deliberately no `--database` / `--schema` on `apply`: those could not retarget anything, and only pointed the drift check at a different database than the one being written.

### CLI

Non-secret connection context resolves **CLI flag → env var → contract server block**:

| Flag | Env var | Also in the contract |
|---|---|---|
| `--account` | `SNOWFLAKE_ACCOUNT` | ✅ |
| `--user` | `SNOWFLAKE_USER` | — |
| `--role` | `SNOWFLAKE_ROLE` | — |
| `--secondary-roles` | `SNOWFLAKE_SECONDARY_ROLES` | — |
| `--warehouse` | `SNOWFLAKE_WAREHOUSE` | ✅ |
| `--authenticator` | `SNOWFLAKE_AUTHENTICATOR` | — |

**Secrets are environment-only — there is no `--password` flag and there never will be** (shell history, `ps aux`, CI logs):

| Env var | Used for |
|---|---|
| `SNOWFLAKE_PASSWORD` | password auth |
| `SNOWFLAKE_PRIVATE_KEY_PATH` | key-pair — `--authenticator snowflake_jwt` |
| `SNOWFLAKE_PRIVATE_KEY_PASSPHRASE` | encrypted private key |
| `SNOWFLAKE_TOKEN` | OAuth — `--authenticator oauth` |

`--authenticator` selects the method: `snowflake` (password, the default), `externalbrowser` (SSO), `oauth`, `snowflake_jwt` (key-pair). The connector auto-detects when you omit it.

`--secondary-roles` accepts `ALL` or `NONE` alone, or one or more comma-separated
Snowflake role names (for example, `DATA_READER,DATA_STEWARD`). It resolves as **CLI
flag → `SNOWFLAKE_SECONDARY_ROLES` → unset**. `ALL` and `NONE` are case-insensitive;
named roles must use valid Snowflake identifier syntax. A role name with spaces must
preserve its Snowflake identifier quotes in the shell, for example
`--secondary-roles '"Finance Reader"'`. When set, dcx executes `USE SECONDARY ROLES ...`
immediately after connecting, before metadata reads, drift checks, or DDL. When omitted,
dcx does not execute a `USE SECONDARY ROLES` statement. This is session configuration
only: it is not stored in the Data Contract server block, and dcx never selects,
switches, or retries roles. Snowflake determines whether named roles are granted to the
user or permitted by a session policy. This setting applies to dcx's Snowflake `import`
and `apply` commands; upstream `dcx test` uses datacontract-cli's connection path and
does not honor `SNOWFLAKE_SECONDARY_ROLES`.

```bash
export SNOWFLAKE_ACCOUNT=xy12345.eu-central-1 SNOWFLAKE_USER=me SNOWFLAKE_PASSWORD=...
dcx import snowflake --database MY_DB --schema LOAD --output contract.yaml

dcx apply snowflake contract.yaml --authenticator externalbrowser --role TRANSFORMER --secondary-roles NONE
```

### Connection profiles (`config.toml`)

A ready-to-edit template with all four methods lives in [`examples/snowflake_config.example.toml`](examples/snowflake_config.example.toml).

`--connection-name` uses a named profile from **the connector's own config file**. Snowflake Connector remains responsible for locating and parsing the TOML file. When dcx can inspect the named profile, it copies every profile value into explicit connection kwargs and resolves exact `${ENV_VAR}` values before connecting:

```toml
# config.toml — see "Where the file lives" below
[connections.dev]
account = "xy12345.eu-central-1"
user = "SVC_DCX"
authenticator = "SNOWFLAKE_JWT"
private_key_file = "/home/me/.snowflake/rsa_key.p8"   # absolute: `~` is NOT expanded
private_key_file_pwd = "${DEV_PRIVATE_KEY_PASSPHRASE}"
role = "TRANSFORMER"
warehouse = "DEV_WH"
```

```bash
export DEV_PRIVATE_KEY_PASSPHRASE='...'
dcx import snowflake --connection-name dev --database MY_DB --schema LOAD
dcx apply snowflake contract.yaml --connection-name dev --server dev
```

The profile supplies the whole connection; only `--user`, `--role`, `--warehouse`, `--account` and `--authenticator` layer on top of it — plus `--database` / `--schema` on `import`, which name what to read. These explicit values continue to override profile values.

Environment interpolation is deliberately narrow:

- Only a whole string of the form `${ENV_VAR}` is resolved.
- Every inspectable profile follows the same explicit-kwargs path; interpolation changes values, never the connection path.
- Literal values remain unchanged. If dcx cannot inspect the connector configuration or the named profile does not exist, it preserves the connector's legacy `connection_name` handling.
- Prefix/suffix interpolation (`prefix-${ENV_VAR}`), `$ENV_VAR`, `${VAR:-default}`, `$(command)` and backticks are not expanded or executed.
- Every reference is looked up independently for every connection, so profiles can use different variables without sharing resolved credentials.
- A missing variable fails before connecting with `Environment variable 'ENV_VAR' is not defined`.
- dcx resolves a copied profile and never mutates the connector's global config or caches resolved secrets.
- Error messages include a missing variable's name, never its resolved value or neighboring profile values.

This supports deployment systems that inject secrets into the process environment without adding platform-specific secret-management logic to dcx:

```toml
[connections.account_a]
account = "${ACCOUNT_A_SNOWFLAKE_ACCOUNT}"
user = "SVC_ACCOUNT_A"
authenticator = "SNOWFLAKE_JWT"
private_key_file = "/vault/account_a/key.p8"
private_key_file_pwd = "${ACCOUNT_A_PRIVATE_KEY_PASSPHRASE}"

[connections.account_b]
account = "${ACCOUNT_B_SNOWFLAKE_ACCOUNT}"
user = "SVC_ACCOUNT_B"
authenticator = "SNOWFLAKE_JWT"
private_key_file = "/vault/account_b/key.p8"
private_key_file_pwd = "${ACCOUNT_B_PRIVATE_KEY_PASSPHRASE}"
```

```bash
export ACCOUNT_A_SNOWFLAKE_ACCOUNT=org-account-a
export ACCOUNT_A_PRIVATE_KEY_PASSPHRASE='...'
export ACCOUNT_B_SNOWFLAKE_ACCOUNT=org-account-b
export ACCOUNT_B_PRIVATE_KEY_PASSPHRASE='...'
```

Profiles may equally live in `connections.toml` alongside it (same tables, without the `connections.` prefix) — the connector reads both.

#### Where the file lives

The connector resolves the directory, in this order:

1. **`$SNOWFLAKE_HOME`** if that directory exists — defaults to `~/.snowflake/`.
2. Otherwise the platform config dir: `~/.config/snowflake/` on Linux, `~/Library/Application Support/snowflake/` on macOS, `%LOCALAPPDATA%\snowflake\` on Windows.

Note the first rule tests for *existence*: with no `~/.snowflake/` directory, `~/.config/snowflake/config.toml` is the file in play. To check which one your install uses:

```bash
python -c "from snowflake.connector.constants import CONFIG_FILE; print(CONFIG_FILE)"
```

**If you set nothing at all, your default profile is used.** When no flag, no `SNOWFLAKE_*` variable and no contract server block identify a connection, dcx falls back to the profile named by `default_connection_name` — the same one `connect()` uses with no arguments — instead of erroring:

```toml
# config.toml
default_connection_name = "dev"
```

```bash
dcx apply snowflake contract.yaml        # no credentials anywhere: uses [connections.dev]
```

The contract's `account` is still layered on top, so a default profile can never silently move an apply to another account. If no default profile is configured, you get the usual `Cannot determine Snowflake account/user` error.

> Snowflake requires the file to be private: `chmod 0600` it, or the connector warns `Bad owner or permissions` on every connect.

> **`private_key_file` must be an absolute path.** The connector opens it with a bare `open()` and does not expand `~`, so a tilde path fails at connect time with `No such file or directory: '~/...'`.

> SnowSQL's `~/.snowsql/config` is a different file in a different format and is **not** read. If your profiles live there, port them to `config.toml` (`snow connection add`) or use the env vars above.

**A profile authenticates; the contract targets.** A `dev` profile applied to a contract whose server block names `PROD_DB` will authenticate as dev and still write to `PROD_DB`. Pair a profile with the matching `--server` block.

### API

The live endpoints (`POST /import/snowflake`, `POST /apply/snowflake`) take an `auth` object in the request body — the server holds no credentials of its own:

| `auth.type` | Fields |
|---|---|
| `oauth` | `token` |
| `key_pair` | `user`, `private_key`, `private_key_passphrase?` |
| `password` | `user`, `password` |
| `config` | `connection_name` (omit for the server's default profile) — **off by default**, see below |

```jsonc
POST /import/snowflake
{
  "account": "xy12345.eu-central-1", "database": "MY_DB", "schema": "LOAD",
  "secondary_roles": "NONE",
  "auth": {
    "type": "key_pair",
    "user": "SVC_DCX",
    "private_key": "-----BEGIN ENCRYPTED PRIVATE KEY-----\n...",
    "private_key_passphrase": "..."
  }
}
```

With `config` the server's profile carries the account, user and credentials, so the body needs only what to read:

```jsonc
POST /import/snowflake
{
  "database": "MY_DB", "schema": "LOAD",
  "auth": {"type": "config", "connection_name": "dev"}
}
```

Omit `connection_name` to use the server's `default_connection_name`; if it has none configured, that is a 400.

Server-side `config` auth uses the same profile interpolation described above. Missing environment variables return HTTP 400 before Snowflake is contacted. Resolved values are scoped to that request and are not stored in the connector's global configuration.

`private_key` is PEM text or base64-encoded PKCS#8 DER. There is no `private_key_file`: a path would make the server read *its own* filesystem on a caller's behalf.

`Authorization: Bearer <token>` is shorthand for `{"type": "oauth"}` and still works on its own; when both are sent, the body wins. `dry_run` on `/apply/snowflake` needs no credentials at all.

Both endpoints accept optional `secondary_roles`: use it at the top level for
`/import/snowflake`, or in the `options` object for `/apply/snowflake`. Values may be
`"ALL"`, `"NONE"`, a role name, or a comma-separated list of role names. API requests
use only this explicit request value; they never inherit
`SNOWFLAKE_SECONDARY_ROLES` from the API host.

Errors: **401** no credentials · **400** unusable key material · **403** method disabled on this server · **502** Snowflake refused.

#### Server-side profiles — `dcx api --allow-local-credentials`

`auth.type: config` is the one method that reads the **API host's** own connection config instead of the caller's credentials, so it is disabled unless you start the server with it:

```bash
dcx api --allow-local-credentials                              # only when the server is yours alone
dcx api --allow-local-credentials --snowflake-config /etc/dcx  # profiles from a specific location
```

`--snowflake-config` takes the directory holding `config.toml` / `connections.toml` (or the file itself) and points the connector at it via `SNOWFLAKE_HOME`, so a service account's profiles need not live in the server user's home. It is validated at startup — a missing path, an unreadable filename, or a directory with no config in it fails immediately rather than silently falling back to the default location. It requires `--allow-local-credentials`, since without that the server never reads a Snowflake config at all.

Whenever server-side profiles are enabled, startup reports which config is in play and what it contains (names only, never values):

```
Snowflake config: /etc/dcx/config.toml (4 profiles: dev, sso, pw, oauth; default: dev)
Server-side profiles enabled: any caller may authenticate as this host via `auth.type: config`.
```

A config that cannot be read, or that defines no profiles, is reported as a warning at startup rather than surfacing later as a failed request.

`dcx api` has no authentication of its own. On a shared instance a server-side profile would let anyone who can reach the port connect as whoever runs the server — so it stays off by default, and requests using it get a `403`. On a personal localhost server it is the safest option available, since no secret crosses the wire. Only a profile *name* is accepted, never a path.

## The tag catalog

`dcx enrich tags` does **controlled-vocabulary** tagging. Instead of letting the model invent tags, you give it a catalog of allowed names and values, and it classifies each column into that vocabulary.

The catalog is a small YAML (or JSON) file — the only extra input auto-tagging needs.

```yaml
# tags_catalog.yaml
tags:
  - name: DATA_CLASSIFICATION          # the tag name (becomes the platform TAG name)
    description: >                      # tells the model what this tag is for
      Data sensitivity level. Assign exactly one — the highest level that applies.
    multiple: false                    # false = at most one value per column; true = many
    values:
      - value: PUBLIC                   # the model may only pick from these values
        description: Non-sensitive data that can be shared freely.
        examples: [country_code, currency, language, product_category]   # guide classification
      - value: INTERNAL
        description: Internal business data, not for public release. The default.
        default: true                  # assigned when the model picks nothing else
        examples: [order_id, status, created_at, loyalty_points]
      - value: CONFIDENTIAL
        description: Personal data or sensitive business data; need-to-know access.
        examples: [full_name, email, phone, home_address, date_of_birth]
      - value: RESTRICTED
        description: Highly sensitive data under legal/regulatory controls (financial, health, credentials, IDs).
        examples: [national_id, passport_number, iban, credit_card_number, health_status]

  - name: DATA_DOMAIN                   # you can define several tags
    description: The business domain that owns the column.
    multiple: false
    values:
      - value: CUSTOMER
        examples: [customer_id, email, loyalty_points]
      - value: FINANCE
        examples: [amount, currency, invoice_id, iban]
```

| Field | Meaning |
|---|---|
| `name` | Tag name. Required. Becomes the tag key everywhere downstream. |
| `description` | What the tag means — given to the model as classification guidance. |
| `multiple` | `false` (default): at most one value per column. `true`: a column may carry several. |
| `values[].value` | An allowed value. **The model may only assign values listed here** — anything else is dropped. |
| `values[].description` | What the value means — strongly improves accuracy. |
| `values[].examples` | Example column names that fit this value — the model's strongest signal. |
| `values[].default` | If `true`, assigned to columns the model leaves unclassified for this tag. At most one per tag. |

Assigned tags are written on each column as `NAME=VALUE` (e.g. `DATA_CLASSIFICATION=CONFIDENTIAL`) — the convention `export snowflake-full` and `apply snowflake` consume. A worked catalog and example contracts live in [`examples/`](examples/).

## REST API

```bash
dcx api --port 4242      # Swagger UI at http://127.0.0.1:4242/docs
```

Every command above is mirrored to an endpoint, with request **and** response schemas in the OpenAPI spec. Auth model:

- **Live platform operations** (`/import/snowflake`, `/apply/snowflake`) act *as the caller* — credentials ride on the request (an `auth` block or a bearer token), so the server never uses ambient credentials for someone else's data. The one exception, server-side connection profiles, is off unless you start the server with `--allow-local-credentials`. Details: [Connecting to Snowflake](#connecting-to-snowflake).
- **Enrichment** (`/enrich/*`) uses the **server's** LLM key (from the environment). Put service-level auth/quota in front of it before exposing it publicly.
- **The CLI never takes secrets as flags** — platform secrets come from env vars or the connector's own `config.toml`; LLM keys from the provider's standard env var. (Over HTTP a caller *does* send credentials in the request body — that is the point: they are the caller's, not the server's. Terminate TLS in front of it.)

## How it fits with datacontract-cli

dcx is a **separate package that depends on datacontract-cli as a library** — no fork. It plugs into upstream's own extension points:

| dcx adds | Where it plugs in |
|---|---|
| importers `snowflake`, `kafka` | upstream's `importer_factory` |
| exporter `snowflake-full` | upstream's `exporter_factory` |
| `target` / `enrich` / `apply` sub-apps | upstream's Typer app |
| REST routes for every command | FastAPI, via `dcx api` |

So you keep all of upstream's importers, exporters, `lint`, `test` and `changelog`, and gain the AI + platform layer on top.

## Development

```bash
pip install -e ".[dev]"
pytest          # 375 tests
ruff check dcx  # lint
```

Tests never hit live services or real LLMs — platform connections, the Schema Registry, and every LLM call are mocked, so the suite stays fast and offline. See [`RELEASING.md`](RELEASING.md) for the PyPI release process.

## Contributing

Issues and PRs welcome. Please run `pytest` and `ruff check dcx` before opening a PR, and add tests for new behavior.

## License

[MIT](LICENSE) © MickaelBZH.

<p align="center"><sub>Built on <a href="https://github.com/datacontract/datacontract-cli">datacontract-cli</a> · <a href="https://bitol.io/">Open Data Contract Standard</a> · <a href="https://github.com/BerriAI/litellm">litellm</a></sub></p>
