Metadata-Version: 2.4
Name: dvt-core
Version: 0.2.8
Summary: Cross-engine data transformation built on dbt — federate SQL models across any database, warehouse, or bucket
Home-page: https://www.getdvt.net
Author: DVT Team
Project-URL: Homepage, https://www.getdvt.net
Project-URL: Documentation, https://www.getdvt.net/docs/quickstart
Project-URL: Architecture, https://www.getdvt.net/docs/federation-engine
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Database
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9,<4
Description-Content-Type: text/markdown
Requires-Dist: dbt-core<2.0,>=1.9
Requires-Dist: sqlglot>=26.0
Requires-Dist: duckdb>=1.0
Requires-Dist: sling>=1.5.15
Requires-Dist: pyyaml>=6.0
Requires-Dist: cryptography>=41.0
Requires-Dist: requests>=2.28.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=3.0; extra == "dev"
Requires-Dist: black>=22.0; extra == "dev"
Requires-Dist: mypy>=0.950; extra == "dev"
Requires-Dist: flake8>=4.0; extra == "dev"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<p align="center">
  <img src="https://www.getdvt.net/dvt-logo.svg" width="130" alt="DVT logo" />
</p>

# DVT — Data Virtualization Tool

**Any source. Any target. One SQL project.**

DVT is a cross-engine data transformation CLI built on [dbt](https://github.com/dbt-labs/dbt-core). Write SQL models that JOIN tables living on *different* database engines — MySQL with Snowflake, Oracle with PostgreSQL, anything with anything — and DVT handles extraction, federation, and loading automatically.

```sql
{{ config(materialized='f_table') }}   -- "f" = federated

select
    c.customer_name,
    o.order_date,
    o.total_amount
from {{ source('crm', 'customers') }} c      -- lives on MySQL
join {{ source('erp', 'orders') }} o          -- lives on Snowflake
  on c.customer_id = o.customer_id
where o.order_date >= '2024-01-01'
```

```bash
pip install dvt-core
dvt sync       # installs every adapter + driver your profiles.yml needs
dvt run        # dbt runs what dbt can; DVT federates the rest
dvt version    # (worth running once — you'll see)
```

Website & full docs: **[getdvt.net](https://www.getdvt.net)** · [Quickstart](https://www.getdvt.net/docs/quickstart) · [PyPI](https://pypi.org/project/dvt-core/)

---

## Architecture

DVT is a **wrapper around stock dbt-core**, not a fork. dbt compiles the project and runs every standard model; DVT picks up the models dbt can't express — cross-engine federation models and locally-executed Python models — and runs them through its own pipeline:

```mermaid
flowchart TB
    CLI([dvt run / build]) --> COMPILE["dbt compile → manifest.json"]
    COMPILE --> STD["standard models<br/>table · view · incremental"]
    COMPILE --> FED["federated models<br/>f_table · f_incremental · .py"]
    STD --> DBT["stock dbt executes them<br/>on the default target"]
    FED --> HOMO{"all sources on<br/>ONE connection?"}
    HOMO -- "yes — Sling direct" --> DIRECT["whole query transpiled to the
    source engine, streamed straight
    to the target — no staging"]
    HOMO -- "no — federation" --> DECOMP["1 · DECOMPOSE — SQLGlot maps every
    table to a profiles.yml connection"]
    DECOMP --> PUSH["2 · TRANSPILE + PUSH DOWN — minimal
    per-source queries, each engine's dialect"]
    PUSH --> EXTRACT["3 · EXTRACT — Sling → Parquet, parallel"]
    EXTRACT --> COMPUTE["4 · COMPUTE — DuckDB joins locally"]
    COMPUTE --> LOAD["5 · LOAD — Sling delivers to any
    database, warehouse, or bucket"]
    DIRECT --> TARGET[("target")]
    LOAD --> TARGET
    DBT --> TARGET
```

### The Sling direct path (0.2.5)

When every source in an `f_table` model lives on **one** connection, steps 3–5 collapse into a single hop: the whole query — joins and aggregations included — is transpiled to that engine's dialect and submitted to Sling as a custom-SQL stream loaded straight to the target. No Parquet staging, no DuckDB. The check is automatic and per-run; any failure falls back to the standard pipeline. Add a source on another engine tomorrow and the same model silently switches to the federation pipeline — the model never changes.

### Execution paths

| Your model | Who runs it | How |
|---|---|---|
| `table` / `view` / `incremental` | stock dbt | natively on the default target — whole-query pushdown by definition |
| `f_table` / `f_incremental`, sources on one connection | DVT | **Sling direct**: full query pushed to the source engine, result streamed to the target |
| `f_table` / `f_incremental`, sources span connections | DVT | decompose → per-source pushdown extraction → DuckDB compute → load |
| `.py` models | DVT | locally executed; result loads to any target |

Standard models that set a non-default `target=` are **refused** with guidance (dbt silently ignores the kwarg — your model would land on the wrong engine).

### Design principles

- **Wrapper, not fork.** DVT shells out to the dbt CLI and reads `manifest.json` — it never imports dbt internals. Your dbt version can move; DVT keeps working. Every dbt feature (Jinja, macros, packages, tests, snapshots, docs) works because it *is* dbt running it.
- **Dual-dialect by design.** Federation models are written in DuckDB dialect and are transpiled per source — they never depend on any engine's syntax. Standard models compile in the default target's dialect via the official dbt adapter. Result: switching your default target (e.g. PostgreSQL → Databricks) is a profile change, not a rewrite.
- **Pushdown first.** The fastest row to move is the one you never extract. Filters and column pruning push down to source engines; when every table in a model lives on one engine, the entire query pushes down and nothing moves at all.
- **Your hardware does the joining.** Cross-engine compute happens in DuckDB on the machine running DVT — source databases only ever serve simple filtered SELECTs, and no warehouse bills you for the join.

## What DVT adds on top of dbt

| Capability | How |
|---|---|
| Cross-engine models | `materialized='f_table'` / `'f_incremental'` (long aliases: `federated_*`, `federation_*`) |
| Incremental federation | watermark-based delta extraction; persistent local cache between runs |
| Per-model targets | `config(target='other_output')` on federated models — results land on any profile output (standard models are refused a foreign target) |
| Bucket targets | materialize models to S3 / GCS / Azure as Parquet or CSV |
| Python models, locally | dbt-style `.py` models run on your machine (APIs, pandas, ML) — results land on any target |
| Bulk seeds | `dvt seed` loads csv / tsv / parquet / json / jsonl / avro / arrow / xlsx via Sling to **any** engine, with enforced snake_case columns |
| One-command setup | `dvt sync` installs every adapter and driver your `profiles.yml` requires |
| Connection diagnostics | `dvt debug --all` checks every connection on both layers (dbt adapter + Sling) |
| Cross-engine catalog | `dvt docs` builds one catalog + lineage UI across all engines in your project |

Everything else — `profiles.yml` in `~/.dbt/`, `sources.yml`, ref/source/config, tests, snapshots — is standard dbt. Existing dbt projects work unchanged; sources gain an optional `meta.connection` to bind them to other engines.

## Supported engines

Every engine supported by **both dbt and Sling** is a DVT adapter — as federation source, federation target, seed target, and default target:

PostgreSQL · MySQL · MariaDB · SQL Server · Oracle · SQLite · DuckDB · Snowflake · BigQuery · Redshift · Databricks · ClickHouse · Trino · Athena · Microsoft Fabric — plus S3, GCS, and Azure Blob as bucket storages.

Anything else (MongoDB, Elasticsearch, REST APIs, …) is reachable through locally-executed Python models. Details per adapter: [getdvt.net/docs/adapters](https://www.getdvt.net/docs/adapters).

## Learn more

- [The federation engine, step by step](https://www.getdvt.net/docs/federation-engine)
- [Predicate pushdown deep dive](https://www.getdvt.net/docs/pushdown)
- [Switching the default target (lakehouse portability)](https://www.getdvt.net/docs/switching-targets)
- [Pricing & plans](https://www.getdvt.net/pricing)

## License

DVT is distributed as compiled wheels on PyPI. Free tier covers local files, transactional databases, and Python models; Pro adds cloud buckets (S3/GCS/Azure); Max adds cloud warehouses. See [getdvt.net/pricing](https://www.getdvt.net/pricing).
