Metadata-Version: 2.4
Name: exmergo-dex-core
Version: 1.6.0
Summary: Analytics engineering for Claude Code and any agent: data warehouse exploration, dbt transformation and semantic modeling, and schema-drift maintenance on dbt.
Project-URL: Homepage, https://www.exmergo.com/dex
Project-URL: Repository, https://github.com/exmergo/dex
Author-email: "Exmergo, Inc." <support@exmergo.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agent,analytics-engineering,claude-code,data-engineering,dbt,duckdb,semantic-layer,text-to-sql
Requires-Python: >=3.11
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml>=6
Provides-Extra: all
Requires-Dist: boto3>=1.34; extra == 'all'
Requires-Dist: databricks-sdk>=0.30; extra == 'all'
Requires-Dist: databricks-sql-connector>=3; extra == 'all'
Requires-Dist: dbt-bigquery>=1.9; extra == 'all'
Requires-Dist: dbt-databricks>=1.9; extra == 'all'
Requires-Dist: dbt-duckdb>=1.9; extra == 'all'
Requires-Dist: dbt-postgres>=1.9; extra == 'all'
Requires-Dist: dbt-redshift>=1.9; extra == 'all'
Requires-Dist: dbt-snowflake>=1.9; extra == 'all'
Requires-Dist: duckdb>=1; extra == 'all'
Requires-Dist: google-cloud-bigquery>=3; extra == 'all'
Requires-Dist: httpx>=0.27; extra == 'all'
Requires-Dist: metricflow<0.212,>=0.211; extra == 'all'
Requires-Dist: psycopg[binary]>=3; extra == 'all'
Requires-Dist: redshift-connector>=2.1; extra == 'all'
Requires-Dist: scikit-learn>=1.4; extra == 'all'
Requires-Dist: snowflake-connector-python>=3.17; extra == 'all'
Requires-Dist: sqlglot<31,>=28.6; extra == 'all'
Provides-Extra: bigquery
Requires-Dist: dbt-bigquery>=1.9; extra == 'bigquery'
Requires-Dist: google-cloud-bigquery>=3; extra == 'bigquery'
Requires-Dist: sqlglot<31,>=28.6; extra == 'bigquery'
Provides-Extra: cluster
Requires-Dist: scikit-learn>=1.4; extra == 'cluster'
Provides-Extra: databricks
Requires-Dist: databricks-sdk>=0.30; extra == 'databricks'
Requires-Dist: databricks-sql-connector>=3; extra == 'databricks'
Requires-Dist: dbt-databricks>=1.9; extra == 'databricks'
Requires-Dist: sqlglot<31,>=28.6; extra == 'databricks'
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff==0.15.19; extra == 'dev'
Requires-Dist: scikit-learn>=1.4; extra == 'dev'
Requires-Dist: sqlglot<31,>=28.6; extra == 'dev'
Provides-Extra: duckdb
Requires-Dist: dbt-duckdb>=1.9; extra == 'duckdb'
Requires-Dist: duckdb>=1; extra == 'duckdb'
Requires-Dist: sqlglot<31,>=28.6; extra == 'duckdb'
Provides-Extra: postgres
Requires-Dist: dbt-postgres>=1.9; extra == 'postgres'
Requires-Dist: psycopg[binary]>=3; extra == 'postgres'
Requires-Dist: sqlglot<31,>=28.6; extra == 'postgres'
Provides-Extra: project-conformance
Requires-Dist: pytest>=8; extra == 'project-conformance'
Provides-Extra: redshift
Requires-Dist: boto3>=1.34; extra == 'redshift'
Requires-Dist: dbt-redshift>=1.9; extra == 'redshift'
Requires-Dist: redshift-connector>=2.1; extra == 'redshift'
Requires-Dist: sqlglot<31,>=28.6; extra == 'redshift'
Provides-Extra: semantic
Requires-Dist: metricflow<0.212,>=0.211; extra == 'semantic'
Requires-Dist: sqlglot<31,>=28.6; extra == 'semantic'
Provides-Extra: semantic-api
Requires-Dist: httpx>=0.27; extra == 'semantic-api'
Provides-Extra: snowflake
Requires-Dist: dbt-snowflake>=1.9; extra == 'snowflake'
Requires-Dist: snowflake-connector-python>=3.17; extra == 'snowflake'
Requires-Dist: sqlglot<31,>=28.6; extra == 'snowflake'
Provides-Extra: sql
Requires-Dist: sqlglot<31,>=28.6; extra == 'sql'
Provides-Extra: storage-conformance
Requires-Dist: pytest>=8; extra == 'storage-conformance'
Requires-Dist: sqlglot<31,>=28.6; extra == 'storage-conformance'
Description-Content-Type: text/markdown

# exmergo-dex-core

The portable, Apache-2.0 analytics-engineering engine behind
[Dex](https://github.com/exmergo/dex). All non-trivial logic lives here; the
Claude Code skills and the cross-agent `AGENTS.md` are thin wrappers that drive it
through one stable command contract.

Dex is the agent-native analytics engineering toolkit: explore an unfamiliar
warehouse, transform raw data into clean dbt models and a semantic layer on top,
and maintain all of it as the data underneath changes. Read-only against your data;
every change is a reviewable diff.

## Install

```
pip install "exmergo-dex-core"
```

Connector client libraries live behind extras. DuckDB is an in-memory data warehouse,
so you can start from there if you want to test Dex locally. We aim to support all major
data warehouses. Please suggest any missing connectors on [GitHub](https://github.com/exmergo/dex)!

```
exmergo-dex-core[duckdb]       # the on-ramp and the eval/benchmark engine
exmergo-dex-core[snowflake]
exmergo-dex-core[bigquery]
exmergo-dex-core[databricks]
exmergo-dex-core[redshift]
exmergo-dex-core[postgres]
exmergo-dex-core[all]          # every optional capability at once
```

Two capabilities sit behind their own extras rather than a connector's:
`[semantic]` and `[semantic-api]` for the local and hosted semantic-layer query
backends, and `[cluster]` for `explore cluster`. `[all]` covers all of these too.

`[semantic-api]` is the one extra that stands completely alone: dbt Cloud owns the
warehouse connection and executes server-side, so a deployment that only queries a
hosted semantic layer needs no connector, no dbt-core, and no SQL parser. Every
other command validates SQL before running it, which is why the connector extras
carry the dialect engine; run one without a connector installed and dex refuses
with the install to use rather than guessing.

## Two surfaces, one engine

### The Python API

```python
from exmergo_dex_core import DexEngine

with DexEngine(connector="duckdb", path="shop.duckdb") as eng:
    mapped = eng.map()
    rows = eng.query("select status, count(*) from orders group by status")
    print(eng.diagram().mermaid)  # the map as a Mermaid ER diagram
```

Methods return domain objects (`DexCache`, `Dataset`, `Snapshot`) and result
records carrying the counts, notes, and warnings that explain them. The stdout
envelope never crosses this boundary.

Nothing above touches disk. The default store keeps state in the process, so
importing this package cannot leave a `.dex/` directory in a consumer's repo;
pass `store=` for anything durable, or use `DexEngine.from_repo(repo_root)` to get
the CLI's behavior (config read from `.dex/config.yml`, and the backend that
config selects, which defaults to plain files under `.dex/`). The `Store` protocol
is public, so a host can back state with its own session store or database
instead, and a backend published as its own package is selectable by name from
`cache.backend` without a change to dex. See
[`references/storage.md`](../../references/storage.md).

[`examples/quickstart.py`](examples/quickstart.py) is the whole flow in one
runnable file: map a warehouse, read the inferred joins, see PII flagged, ask a
question, and watch the firewall refuse one it should. It builds its own
throwaway DuckDB file, so it runs anywhere:

```
pip install "exmergo-dex-core[duckdb]"
python quickstart.py
```

The test suite runs that file against a freshly built wheel, so the usage
documented here is the usage that is verified.

Every guarantee below holds here too, because it is the same code. An
unconfirmed billed call raises `ConfirmationRequiredError` carrying the estimate
and the payload needed to re-issue; an over-ceiling one raises `OverCeilingError`
and cannot be confirmed through.

Three rules matter the moment a process serves more than one user, and all three
are in `DexEngine`'s docstring: scope one engine to one principal and one session,
know that an engine given an explicit `config=` never reads one from disk (so a
stray `.dex/config.yml` above the working directory cannot silently supply someone
else's connector, budget, or PII overrides), and supply the connection when the
request's identity is not the container's.

That last one is what makes per-end-user access control expressible. By default
dex discovers the credential from process-ambient state, which is right for one
person at a terminal and process-wide everywhere else. Pass a `ConnectionSource`
and the host owns authentication:

```python
from exmergo_dex_core import ConnectionSource, DexEngine

with DexEngine(
    connector="snowflake",
    config=cfg,
    store=store,
    connection=ConnectionSource(connect=lambda: user_conn),
) as eng:
    eng.inventory()
```

It is a zero-argument factory rather than a live connection, so a free metadata
command never opens a billed session. Two things stay dex's. The cost gate is
still built here from your `store`, so the per-command ceiling and the cumulative
session ceiling bind exactly as they do on a discovered connection; handing that
to an integrator would let a fumbled figure disarm the brake in the deployment
where a runaway agent loop costs the most. And dex closes nothing it reached
through the source, because the caller that opened a connection is the one still
holding it. Nothing is persisted either way: dex never stores, caches, or
refreshes a credential.

A hosted dbt Cloud Semantic Layer is a second service with its own credential, so
it has its own parameter. Non-secret coordinates go in the config, where they can
be committed; the service token never can, so it arrives separately:

```python
from exmergo_dex_core import DexConfig, DexEngine, SemanticSource

config = DexConfig(
    semantic={"backend": "dbt_cloud", "host": host, "environment_id": env_id}
)

with DexEngine(
    config=config,
    semantic_source=SemanticSource(token=lambda: token_for(user)),
) as eng:
    catalog = eng.semantic_list()
    result = eng.semantic_query("revenue", group_by=["metric_time__month"])
```

That is the one surface needing nothing on the filesystem at all: no dbt project,
no store, no connector, no credential file. The token callable runs once per
semantic command rather than once per HTTP request, so a metric query that polls
dbt Cloud while it runs costs you one token read. Note that dbt Cloud owns the
warehouse connection on this path and executes server-side, so dex's cost guard
cannot apply and every hosted result says so; the PII dimension gate still does.

### The command contract

Every subcommand prints exactly one sanitized JSON envelope to stdout and nothing
else; nothing reaches agent context except through that envelope. Credentials
never cross it, and data values cross only from profiled, PII-cleared columns,
bounded and capped by the query firewall. State persists in `.dex/`, so
subcommands are stateless and the agent orchestrates multi-step flows.

```
dex connect test --path data.duckdb
```

The CLI is the API's first consumer rather than a parallel implementation: it
parses arguments, builds an engine, and wraps the result it gets back. See
[`references/command-contract.md`](../../references/command-contract.md) for the
full surface and the envelope spec.

## Status

Early and under active development; open issues on [GitHub](https://github.com/exmergo/dex)! Today the engine
runs Explore, Transform, and Maintain end to end on every connector: DuckDB,
BigQuery, Snowflake, Databricks, Amazon Redshift, and Postgres, through either
the command contract or the Python API.

### Commands

`explore`: ranks what matters in an unfamiliar warehouse, profiles columns
selectively, flags PII, surfaces grain and data-quality warnings, infers joins
and verifies them with overlap probes (`--verify`), and executes agent-authored
ad-hoc SELECTs behind a PII-aware query firewall (`explore query`), all
read-only. `explore diagram` serializes the map it built as a Mermaid
`erDiagram`, free and without opening a connection, drawing declared joins solid
and inferred joins dotted and claiming a cardinality only where the cache proved
one. It starts bare by default; with `--use-project` it reads an existing
dbt project, promoting declared `relationships` joins, honoring declared grain
and `unique` tests, and letting metric-backing models surface first in the
ranking. A repeatable `--scope` narrows the source scope per command without
writing back to `.dex/config.yml`. It also queries the dbt semantic layer
(`explore semantic list` / `query`): metric queries run either locally through
MetricFlow and dex's own cost handshake (`--local`), or against a hosted dbt Cloud
deployment (`--api`), where dbt Cloud executes server-side and every result warns
that dex's cost guard does not apply there.

`transform`: bootstraps a dbt project where none exists (`transform init`, with an
explicit connector, never a default), turns agent-authored edits and
deterministic staging scaffolds into reviewable, conflict-checked diffs
(`transform plan` / `apply`, with human edits authoritative on conflict), runs
gated dev-target-only builds with cost surfaced before any spend
(`transform build`), and authors the semantic layer as MetricFlow-validated dbt
semantic models (`semantic define|update|plan`, applied with `transform apply`).

`maintain`: detects drift against the `.dex/` snapshot on four axes and proposes
the fix: schema (structure), volume (freshness), grain (uniqueness and fanout),
and semantic (definitions, dangling references, and dimension cardinality).
`maintain check` sweeps all of them, ranked by blast radius; `reconcile`
proposes reviewable diffs tagged mechanical or advisory, applied through
`transform apply`. Detection is read-only on every connector; on billed
connectors the metadata axes (schema, volume, references) stay free while the
scanning axes (grain, dimension cardinality) take the `--confirm --budget`
handshake, so `check` is two-phase.

### Connectors

Every connector below discovers its own credentials and never asks for a key or a
password, which is the right default for a CLI one person runs. A process serving
several end users supplies the connection instead (see the Python API above), and
dex still builds the cost gate, narrows scope inward only, and keeps the session
read-only. The connector extra is required either way, since each adapter reads
its driver's error types to translate refusals.

BigQuery: connects through Application Default Credentials
(`gcloud auth application-default login`; dex discovers credentials, it never
asks for keys). Metadata is free; every scan is dry-run first, returned as a
`needs_confirmation` estimate, and runs only with `--confirm --budget <bytes>`,
capped server-side by `maximum_bytes_billed` and recorded in a local
`.dex/spend.jsonl` ledger. dbt builds go to a dedicated dev dataset via
dbt-bigquery, which the `[bigquery]` extra carries. See
[`references/bigquery.md`](../../references/bigquery.md).

Snowflake: connects through discovered credentials (`connections.toml`,
`SNOWFLAKE_*` env, or a dbt profile; dex never asks for or persists a
password). The cost inversion from BigQuery: metadata is free (SHOW commands,
no warehouse), while scans bill warehouse time, so budgets are
**warehouse-seconds** with credits shown alongside. Estimates are an honestly
labeled heuristic (Snowflake has no dry-run), floored by the 60-second resume
minimum on a cold warehouse; the budget is hard-enforced anyway by a
per-statement server-side `STATEMENT_TIMEOUT_IN_SECONDS`, and actual seconds
land in the same `.dex/spend.jsonl` ledger. Billed work runs only on the
warehouse the config pins. dbt builds go to a dedicated dev database.schema
via dbt-snowflake, which the `[snowflake]` extra carries. See
[`references/snowflake.md`](../../references/snowflake.md).

Databricks: the lakehouse connector. Connects through the Databricks SDK's
unified auth chain (`databricks auth login`, `DATABRICKS_*` env, or a dbt
profile; dex never asks for or persists a token). Metadata is free through
the Unity Catalog REST API, and the SQL session opens lazily on the first
billed statement, so free commands never touch (or wake) the warehouse.
Budgets are **warehouse-seconds** with DBUs shown alongside. Estimates start
as an honestly labeled floor (no dry-run, no free table sizes) and refine
in-budget via `DESCRIBE DETAIL`; the budget is hard-enforced anyway by a
per-statement server-side `STATEMENT_TIMEOUT`, and actual seconds land in the
same `.dex/spend.jsonl` ledger. Billed work runs only on the SQL warehouse
the config pins. dbt builds go to a dedicated dev catalog.schema via
dbt-databricks, which the `[databricks]` extra carries. See
[`references/databricks.md`](../../references/databricks.md).

Amazon Redshift: Serverless-first and provisioned-compatible. Connects through
the AWS default credential chain (a pinned Serverless `workgroup` or provisioned
`cluster_identifier` mints IAM temporary database credentials), the `REDSHIFT_*`
environment, the committed non-secret target (password via `REDSHIFT_PASSWORD`),
or a dbt profile; dex never asks for or persists a password. Metadata comes from
the Postgres catalog (`pg_class` merged with `SVV_TABLE_INFO` and `SVV_COLUMNS`,
so empty tables still appear). The guarded quantity is compute time, so budgets
are **compute-seconds** with RPU-hours shown alongside (dollars when
`redshift.rpu_price_usd` is set), floored once by the 60-second Serverless wake
minimum; the budget is hard-enforced by a per-statement server-side
`statement_timeout`, and actual seconds land in the same `.dex/spend.jsonl`
ledger. Profiling uses `HLL(...)` approximate distincts with exact escalation
in-budget; the session is read-only at the server. dbt builds go to a dedicated
dev schema via dbt-redshift, which the `[redshift]` extra carries. See
[`references/redshift.md`](../../references/redshift.md).

PostgreSQL: the operational-database connector. Connects through discovered
credentials (`pg_service.conf`, `DATABASE_URL`, the `PG*` environment, or a
dbt profile; dex never asks for or persists a password). Nothing is billed in
dollars; the guarded quantity is load on what is often a production primary,
so budgets are **database-seconds** through the same confirm handshake. Query
estimates come from the genuinely free planner preflight (`EXPLAIN`), profile
estimates from relation sizes, both labeled heuristic; the budget is
hard-enforced anyway by a per-statement server-side `statement_timeout`, and
actual seconds land in the same ledger. The session is read-only at the
server (`default_transaction_read_only = on`), profiling leans on the
planner's own statistics instead of scanning distincts, and dbt builds go to
a dedicated dev schema via dbt-postgres, which the `[postgres]` extra
carries, with the ceiling injected as a statement timeout through
`PGOPTIONS`. See [`references/postgres.md`](../../references/postgres.md).

## License

Apache-2.0.
