Metadata-Version: 2.4
Name: dbt-costgate
Version: 1.1.0
Summary: BigQuery cost gate for dbt pull requests: dry-run changed models, diff the bytes, price the damage before it merges.
Project-URL: Homepage, https://github.com/Drichards124/dbt-costgate
Project-URL: Repository, https://github.com/Drichards124/dbt-costgate
Project-URL: Issues, https://github.com/Drichards124/dbt-costgate/issues
Project-URL: Changelog, https://github.com/Drichards124/dbt-costgate/blob/main/CHANGELOG.md
Author: Dashan Richards
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: analytics-engineering,bigquery,ci,cost,dbt,dry-run,finops
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Requires-Dist: google-cloud-bigquery>=3.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

<div align="center">

<img src="docs/assets/logo.svg" alt="dbt-costgate logo" width="96" height="96"/>

# dbt-costgate

**Know what a dbt pull request will cost — before you merge it.**

Dry-run what changed, price the diff, and catch the $500-a-day model<br/>*before* it merges — not on next month's bill.

[![Dry-run only](https://img.shields.io/badge/queries-dry--run%20only-2ea043)](#safe-to-run-next-to-production-credentials)
[![No telemetry](https://img.shields.io/badge/telemetry-none-2ea043)](#safe-to-run-next-to-production-credentials)
[![PyPI](https://img.shields.io/pypi/v/dbt-costgate?label=pypi&color=0b7285)](https://pypi.org/project/dbt-costgate/)
[![Python](https://img.shields.io/badge/python-3.9%20%E2%80%93%203.13-3776AB?logo=python&logoColor=white)](pyproject.toml)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![CI](https://github.com/Drichards124/dbt-costgate/actions/workflows/ci.yml/badge.svg)](https://github.com/Drichards124/dbt-costgate/actions/workflows/ci.yml)
[![Status](https://img.shields.io/badge/status-beta-blue)](#project-maturity)

[Quick start](#quick-start) ·
[See it on a PR](#see-it-on-a-real-pull-request) ·
[Security](#safe-to-run-next-to-production-credentials) ·
[Pricing accuracy](#pricing-you-can-audit) ·
[Documentation](#documentation) ·
[Maturity](#project-maturity)

</div>

---

## What it does

A dbt model's cost lives in its SQL, and nothing in code review shows you that
number. A dropped partition filter, a widened incremental window, or one changed
join can multiply the bytes a model scans — and the team finds out days later on
the bill, or when finance escalates.

BigQuery will tell you the **exact** bytes a query would scan, for free, before
anything runs. dbt-costgate turns that into a pull-request gate.

- 💸 **Free to run.** Dry-run only — it never executes a query, never reads a row, and never appears on your bill.
- 🎯 **Exact, not estimated.** The byte count comes from BigQuery's own query planner, not from a heuristic over your SQL.
- ⚡ **Four lines to a first answer.** No config file, no baseline, no CI setup, no account to create.
- 🔒 **Nothing to hand over.** No credential flags, no telemetry, and no compiled SQL in any report.

## See it on a real pull request

A sticky comment, updated in place on every push. This is the comment itself —
GitHub is rendering the same markdown dbt-costgate produces:

<!-- BEGIN GENERATED: pr-comment -->
<!-- Generated by scripts/gen_samples.py from the real renderers. Do not edit by hand. -->
### 💸 dbt-costgate — cost impact of this change (2 models)

❌ **Gate: FAIL**
- fct_orders_daily: USD +13.19/run exceeds USD 5.00
- fct_orders_daily: +264% exceeds 25%

**Net increase:** USD 13.19/run · USD 395.70/month

| Model | Baseline | This change | Δ % | Δ / run | Δ / month |
|---|--:|--:|--:|--:|--:|
| `fct_orders_daily` _full-refresh_ | 819.20 GiB | 2.91 TiB | +264% | USD +13.19 | USD +395.63 |
| `dim_customers` _new_ | — | 412.50 MiB | — | USD +0.00 | USD +0.07 |

> ⚠ full-refresh — rows tagged full-refresh show what it costs to build the whole table from scratch. A normal incremental run scans much less, so read this as the ceiling rather than the nightly bill.

<sub>Pricing: US USD 6.25/TiB · built-in table (table 2026.07, verified 2026-07-25)<br/>Priced from the first byte scanned: BigQuery's 1 TiB/month on-demand free tier is per billing account, so it is disclosed here and never deducted.<br/>Estimates from BigQuery dry-run — nothing executed, no bytes billed, no SQL shown.</sub>
<!-- END GENERATED: pr-comment -->

<details>
<summary><b>💻 The same check in your terminal (real output)</b></summary>
<br/>

```bash
dbt-costgate check --baseline path/to/main/manifest.json
```

<!-- BEGIN GENERATED: diff-terminal -->
<!-- Generated by scripts/gen_samples.py from the real renderers. Do not edit by hand. -->
```text
dbt-costgate — region: US · on-demand USD 6.25/TiB · built-in table

  MODEL                             BASELINE     CURRENT    Δ %     Δ / RUN    Δ / MONTH  RUNS
  ────────────────  ────────────  ──────────  ──────────  ─────  ──────────  ───────────  ────
  fct_orders_daily  full-refresh  819.20 GiB    2.91 TiB  +264%  USD +13.19  USD +395.63    30
  dim_customers     new                    —  412.50 MiB      —   USD +0.00    USD +0.07    30

  Net increase: USD 13.19/run · USD 395.70/month

  GATE: FAIL
    - fct_orders_daily: USD +13.19/run exceeds USD 5.00
    - fct_orders_daily: +264% exceeds 25%

  NOTES
    ⚠ full-refresh — rows tagged full-refresh show what it costs to build the whole table from
      scratch. A normal incremental run scans much less, so read this as the ceiling rather than the
      nightly bill.

  Pricing: US USD 6.25/TiB · built-in table (table 2026.07, verified 2026-07-25)
  Priced from the first byte scanned: BigQuery's 1 TiB/month on-demand free tier is per billing
    account, so it is disclosed here and never deducted.
  Estimates from BigQuery dry-run — nothing executed, no bytes billed, no SQL shown.
```
<!-- END GENERATED: diff-terminal -->

</details>

## Quick start

```bash
pip install dbt-costgate
gcloud auth application-default login

dbt compile
dbt-costgate check
```

That is the entire local setup — no baseline, no CI, no config file. You get each
changed model's scan cost immediately.

**To make it block a pull request,** add a threshold. `dbt-costgate init` writes a
config file documenting every setting with all of them commented out, so it
changes nothing until you uncomment one:

```yaml
# .dbt-costgate.yml
thresholds:
  max_usd_total: 20.00     # fail if any model costs more than this per run
```

**Other ways to run it:** a [GitHub Action](docs/usage.md#github-action) with the
sticky comment above, a [pre-commit hook](docs/usage.md#pre-commit-hook) that
catches it before you push, or the published container image
`ghcr.io/drichards124/dbt-costgate:v1.1.0` for
[CI that isn't GitHub Actions](docs/usage.md#docker-and-ci-that-isnt-github-actions).
Every [release](https://github.com/Drichards124/dbt-costgate/releases) also ships a
wheel, an sdist and `SHA256SUMS` if you would rather pin to an artifact.

## Safe to run next to production credentials

This tool runs in CI beside your warehouse credentials, so the design is
deliberately boring. Each row below is an invariant, not a setting you have to
get right:

| Threat | Design answer |
|---|---|
| Billable or data-reading queries | **Dry-run only.** The single warehouse interaction is `jobs.insert` with `dryRun=true` — free, executes nothing, reads no table data |
| Credential theft or mishandling | **No credential surface.** Auth delegates entirely to [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials); in CI the documented path is keyless [Workload Identity Federation](https://github.com/google-github-actions/auth). There are no credential flags to misuse |
| Compromised CI runner | **Least privilege.** BigQuery Job User + metadata read — no data access, no writes; the docs ship the exact IAM setup |
| Malicious fork pull requests | **Fork-safe by default.** Documented workflows use the `pull_request` trigger; fork PRs degrade to "no report", never to exposed secrets |
| Secrets templated into SQL | **No compiled SQL in reports** — model names, bytes and dollars only. This covers BigQuery's own error text, which quotes the query it was given |
| Phone-home | **No telemetry.** The only network call is to the BigQuery API |

Full threat model in [SECURITY.md](SECURITY.md) · design notes in
[docs/architecture.md](docs/architecture.md).

## Pricing you can audit

BigQuery on-demand rates differ by region — a gate that prices every byte at the
US rate is silently wrong for half the world. So every figure shows its work:

- 🌍 **Versioned per-region pricing table** with a `last_verified` date, auto-selected from your job's detected region.
- 🧾 **Every report discloses its math** — region, rate, and where that rate came from. Never a silent assumption:

  ```text
  region: US (multi-region) · on-demand USD 6.25/TiB · source: built-in table 2026.07
  ```

- ⚙️ **Overridable** — `pricing.region` to force a region, `pricing.usd_per_tib` for a negotiated or Editions rate, `pricing.currency` to label amounts in your own currency (an ISO 4217 code — dbt-costgate labels, it never converts).
- ⚠️ **Honest limits, stated up front** — under capacity/Editions pricing, bytes scanned is a proxy signal rather than your invoice; set a rate of `0` and reports drop money entirely and measure bytes instead. Every priced report discloses the 1 TiB/month on-demand free tier it does **not** deduct, because that allowance belongs to the whole billing account and a dry-run cannot see it.

Not sure which applies to you? [Which pricing setup are you?](docs/explained.md#which-pricing-setup-are-you)
· [When the number can be wrong](docs/explained.md#when-the-number-can-be-wrong)

## How it works

<div align="center">

![How dbt-costgate works: pull request → compile both versions → BigQuery dry-run → price the diff → gate](docs/assets/flow.svg)

</div>

| Step | What happens | Cost to you |
|------|--------------|-------------|
| 1 · **Find what changed** | dbt's `state:modified` selector against a baseline manifest (your production artifacts), with a git-diff fallback | free |
| 2 · **Compile both versions** | The baseline and PR-branch versions of each changed model | free |
| 3 · **Dry-run each** | BigQuery `dryRun=true` returns exact bytes scanned — executes nothing, reads no table data | **free** |
| 4 · **Price the diff** | Region-aware on-demand rates; optionally × run frequency for $/month | free |
| 5 · **Gate** | Markdown PR comment, machine-readable JSON, policy-driven exit code (fail on a $ and/or % increase, or an absolute $/run or TiB/run ceiling) | free |

It also catches the changes that do not touch a model's own `.sql` file — a
macro edit or a `dbt_project.yml` config change that alters the compiled SQL
underneath it.

## Documentation

Written for people who have to review this before it runs anywhere near
production. So you know where to look before opening anything:

| Document | What's inside | Go here when |
|---|---|---|
| **[Explained](docs/explained.md)** | Plain-English guide: how it works, what it costs to run, which pricing setup you're in, **every config key**, the deliberate non-goals, and when a number can be wrong | You're new, or you want to know what a setting does |
| **[Usage guide](docs/usage.md)** | The how-to: install, CI setup, baselines, thresholds, the GitHub Action, worked examples for on-demand / negotiated / slot pricing | You're setting it up or changing how it runs |
| **[Architecture](docs/architecture.md)** | Why it's built this way, the invariants, the hard edges | You're contributing or reviewing a change |
| **[Security](SECURITY.md)** | Threat model, and what counts as a vulnerability | You're reviewing it for use next to production credentials |
| **[Changelog](CHANGELOG.md)** | What changed in each release, in operator terms | You're upgrading |

In the tool itself, `dbt-costgate config` lists every setting one line each, and
`dbt-costgate config <key>` explains one in full and prints the YAML that sets it.

Every example report in this README and across the docs is **generated from the
real renderers** by `scripts/gen_samples.py`, and CI fails if one drifts from
what the code actually prints — the figures are illustrative, the output is not.

## Where it fits

dbt-costgate is the **preventive** half of BigQuery cost control. It deliberately
does not compete with the excellent retrospective tools:

| The question you're asking | Reach for |
|---|---|
| "What *did* our warehouse cost, by model / user / query?" | [dbt-bigquery-monitoring](https://github.com/bqbooster/dbt-bigquery-monitoring) |
| "What does the dbt platform estimate my models cost?" | [dbt Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights) |
| "What is **this PR about to do** to our bill?" | **dbt-costgate** |

## Project maturity

**Feature-complete, and honest about mileage.** Everything on the
[roadmap](#roadmap) ships today. What this has not yet had is a year across many
different warehouses, which is the only thing that finds the last class of bug.

Every release is gated on a production-like run that installs the *built wheel*
on the full OS and Python matrix and exercises it there — not just the source
tree — before it can be promoted and tagged.

**If a number looks wrong, that is the single most useful bug report this project
can get.** [Open one here](https://github.com/Drichards124/dbt-costgate/issues/new/choose).

## Roadmap

Every item below ships as of
[v1.1.0](https://github.com/Drichards124/dbt-costgate/releases/latest).

- [x] **`dbt-costgate check`** — local (zero-setup) + CI diff, region-aware pricing, threshold gating
- [x] **One-command local diff** — `dbt-costgate check --against main` (isolated git worktree)
- [x] **GitHub Action** wrapper with a sticky PR comment
- [x] **Absolute cost ceilings** — gate on total `$/run` or `TiB/run`, not just the increase (works without a baseline, so it gates local mode too)
- [x] **Config- and macro-only change detection** — catch a change that reaches a model without touching its `.sql` file
- [x] **`pre-commit` hook** — catch it on your own machine, at pre-push
- [x] **Docker image** — published on every release, for CI that isn't GitHub Actions

**What comes next is not another feature.** That list was written before anyone
had run this against a real warehouse for a month. The useful next step is use —
finding where the numbers, the defaults or the docs are wrong — and the features
after it should be the ones people actually ask for rather than the ones that
looked obvious from here.

## Non-goals

- **Not a monitoring tool** — retrospective observability belongs to [dbt-bigquery-monitoring](https://github.com/bqbooster/dbt-bigquery-monitoring).
- **BigQuery first** — one warehouse done accurately beats three done approximately. Other warehouses come only once BigQuery is genuinely finished, and only where the cost model actually transfers.
- **Never runs billable queries** — features that require executing real queries are out of scope by design.
- **No IDE/editor integration** (for now).

---

<div align="center">

[Explained](docs/explained.md) ·
[Usage guide](docs/usage.md) ·
[Contributing](CONTRIBUTING.md) ·
[Security policy](SECURITY.md) ·
[Changelog](CHANGELOG.md) ·
[Code of Conduct](CODE_OF_CONDUCT.md) ·
[Apache-2.0](LICENSE) · [NOTICE](NOTICE)

Built by [Dashan Richards](https://github.com/Drichards124) — hard invariants apply:<br/>
**dry-run only · no credential handling · no telemetry**

</div>
