Metadata-Version: 2.4
Name: dbt-costgate
Version: 1.0.2
Summary: BigQuery cost gate for dbt pull requests: dry-run changed models, diff the bytes, price the damage before it merges.
Project-URL: Homepage, https://github.com/Drichards124/dbt-costgate
Project-URL: Repository, https://github.com/Drichards124/dbt-costgate
Project-URL: Issues, https://github.com/Drichards124/dbt-costgate/issues
Project-URL: Changelog, https://github.com/Drichards124/dbt-costgate/blob/main/CHANGELOG.md
Author: Dashan Richards
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: analytics-engineering,bigquery,ci,cost,dbt,dry-run,finops
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Requires-Dist: google-cloud-bigquery>=3.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

<div align="center">

<img src="docs/assets/logo.svg" alt="dbt-costgate logo" width="96" height="96"/>

# dbt-costgate

**The BigQuery cost gate for dbt pull requests.**

Dry-run what changed, price the diff, and catch the $500-a-day model<br/>*before* it merges — not on next month's bill.

[![CI](https://github.com/Drichards124/dbt-costgate/actions/workflows/ci.yml/badge.svg)](https://github.com/Drichards124/dbt-costgate/actions/workflows/ci.yml)
[![PLE](https://github.com/Drichards124/dbt-costgate/actions/workflows/ple.yml/badge.svg)](https://github.com/Drichards124/dbt-costgate/actions/workflows/ple.yml)
[![Python](https://img.shields.io/badge/python-3.9%20%E2%80%93%203.13-3776AB?logo=python&logoColor=white)](pyproject.toml)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230)](https://docs.astral.sh/ruff/)
[![Status](https://img.shields.io/badge/status-MVP-brightgreen)](#roadmap)

[Quick start](#quick-start) ·
[How it works](#how-it-works) ·
[What you get](#what-you-get-on-every-pr) ·
[Where it fits](#where-it-fits) ·
[Pricing accuracy](#accurate-transparent-pricing) ·
[Security](#security-model) ·
[Roadmap](#roadmap) ·
[Contributing](CONTRIBUTING.md)

**New here? Start with [dbt-costgate, explained](docs/explained.md)** — plain
English, ten minutes, no prior context assumed.

</div>

> [!NOTE]
> **MVP — feature-complete, not yet battle-tested.** Everything on the
> [roadmap](#roadmap) ships: the CLI, the GitHub Action, a pre-commit hook, a
> published container image, and config scaffolding. What it has not had is
> mileage across many real projects, which is the only thing that finds the last
> class of bug. **If it does something wrong or confusing, that is worth a
> [bug report](https://github.com/Drichards124/dbt-costgate/issues/new/choose)** —
> including "the number looks wrong", which is the most useful report this tool
> can get.
>
> Every report shown here is **generated from the real renderers** by
> `scripts/gen_samples.py`, and CI fails if any of them drifts from what the code
> actually produces — the figures are illustrative, the output is not. See the
> [usage guide](docs/usage.md) and [changelog](CHANGELOG.md).

---

## The problem

On dbt + BigQuery teams, SQL changes merge with **zero visibility into their cost
impact**. A changed join, a dropped partition filter, or a widened incremental
window can multiply a model's bytes scanned — and the team finds out days later
on the bill, or when finance escalates.

BigQuery's dry-run API returns the *exact* bytes a query would scan — **for
free, before running anything**. dbt-costgate packages that into a first-class PR
gate:

<div align="center">

![How dbt-costgate works: pull request → compile both versions → BigQuery dry-run → price the diff → gate](docs/assets/flow.svg)

</div>

## What you get on every PR

A sticky comment on the pull request, updated in place on every push. This is the
comment itself — GitHub renders it from the same markdown dbt-costgate produces:

<!-- BEGIN GENERATED: pr-comment -->
<!-- Generated by scripts/gen_samples.py from the real renderers. Do not edit by hand. -->
### 💸 dbt-costgate — cost impact of this change (2 models)

| Model | Baseline | This change | Δ % | Δ / run | Δ / month |
|---|--:|--:|--:|--:|--:|
| `fct_orders_daily` _full-refresh_ | 819.20 GiB | 2.91 TiB | +264% | USD +13.19 | USD +395.63 |
| `dim_customers` _new_ | — | 412.50 MiB | — | USD +0.00 | USD +0.07 |

> ⚠ full-refresh — rows tagged full-refresh show what it costs to build the whole table from scratch. A normal incremental run scans much less, so read this as the ceiling rather than the nightly bill.

**Net increase:** USD 13.19/run · USD 395.70/month

❌ **Gate: FAIL**
- fct_orders_daily: USD +13.19/run exceeds USD 5.00
- fct_orders_daily: +264% exceeds 25%

<sub>Pricing: US USD 6.25/TiB · built-in table (table 2026.07, verified 2026-07-25)<br/>Priced from the first byte scanned: BigQuery's 1 TiB/month on-demand free tier is per billing account, so it is disclosed here and never deducted.<br/>Estimates from BigQuery dry-run — nothing executed, no bytes billed, no SQL shown.</sub>
<!-- END GENERATED: pr-comment -->

<details open>
<summary><b>💻 The same check, in your terminal (real output)</b></summary>
<br/>

```bash
dbt-costgate check --baseline path/to/main/manifest.json
```

<!-- BEGIN GENERATED: diff-terminal -->
<!-- Generated by scripts/gen_samples.py from the real renderers. Do not edit by hand. -->
```text
dbt-costgate — region: US · on-demand USD 6.25/TiB · built-in table

  MODEL                             BASELINE     CURRENT    Δ %     Δ / RUN    Δ / MONTH  RUNS
  ────────────────  ────────────  ──────────  ──────────  ─────  ──────────  ───────────  ────
  fct_orders_daily  full-refresh  819.20 GiB    2.91 TiB  +264%  USD +13.19  USD +395.63    30
  dim_customers     new                    —  412.50 MiB      —   USD +0.00    USD +0.07    30

  Net increase: USD 13.19/run · USD 395.70/month

  GATE: FAIL
    - fct_orders_daily: USD +13.19/run exceeds USD 5.00
    - fct_orders_daily: +264% exceeds 25%

  NOTES
    ⚠ full-refresh — rows tagged full-refresh show what it costs to build the whole table from
      scratch. A normal incremental run scans much less, so read this as the ceiling rather than the
      nightly bill.

  Pricing: US USD 6.25/TiB · built-in table (table 2026.07, verified 2026-07-25)
  Priced from the first byte scanned: BigQuery's 1 TiB/month on-demand free tier is per billing
    account, so it is disclosed here and never deducted.
  Estimates from BigQuery dry-run — nothing executed, no bytes billed, no SQL shown.
```
<!-- END GENERATED: diff-terminal -->

Or run it with no baseline at all for an instant local read of what your changed
models scan — and fail the run there on an absolute `--max-usd-total` /
`--max-tib-total` ceiling (no baseline required) — or get the full before/after
locally in one command with `dbt-costgate check --against main` (dbt-costgate compiles
`main` for you in a throwaway worktree). See the [usage guide](docs/usage.md).

</details>

## Quick start

```bash
pip install dbt-costgate
gcloud auth application-default login

dbt compile
dbt-costgate check
```

That's the entire local setup — no baseline, no CI, no config file. Add a
baseline and thresholds when you want it to *block* a PR; see the
[usage guide](docs/usage.md).

When you do want a config file, `dbt-costgate init` writes one documenting every
setting, all commented out — so it changes nothing until you uncomment
something.

Every [release](https://github.com/Drichards124/dbt-costgate/releases) also ships a
wheel, an sdist, and `SHA256SUMS` if you'd rather pin to an artifact.

## Documentation

So you know where to look before you open anything:

| Document | What's inside | Go here when |
|---|---|---|
| **[Explained](docs/explained.md)** | Plain-English guide: how it works, what it costs to run, which pricing setup you're in, **every config key**, the deliberate non-goals, and when a number can be wrong | You're new, or you want to know what a setting does |
| **[Usage guide](docs/usage.md)** | The how-to: install, CI setup, baselines, thresholds, the GitHub Action, worked examples for on-demand / negotiated / slot pricing | You're setting it up or changing how it runs |
| **[Architecture](docs/architecture.md)** | Why it's built this way, the invariants, the hard edges | You're contributing or reviewing a change |
| **[Security](SECURITY.md)** | Threat model, and what counts as a vulnerability | You're reviewing it for use next to production credentials |
| **[Changelog](CHANGELOG.md)** | What changed in each release, in operator terms | You're upgrading |

Every example report in these docs is generated from the real renderers, and CI
fails if one drifts — so what you read is what the tool actually prints.

## How it works

| Step | What happens | Cost to you |
|------|--------------|-------------|
| 1 · **Find what changed** | dbt's `state:modified` selector against a baseline manifest (your production artifacts), with a git-diff fallback | free |
| 2 · **Compile both versions** | The baseline and PR-branch versions of each changed model | free |
| 3 · **Dry-run each** | BigQuery `dryRun=true` returns exact bytes scanned — executes nothing, reads no table data | **free** |
| 4 · **Price the diff** | Region-aware on-demand rates; optionally × run frequency for $/month | free |
| 5 · **Gate** | Markdown PR comment, machine-readable JSON, policy-driven exit code (fail on a $ and/or % increase, or an absolute $/run or TiB/run ceiling) | free |

## Where it fits

dbt-costgate is the **preventive** half of BigQuery cost control — it deliberately
does not compete with the excellent retrospective tools:

| The question you're asking | Reach for |
|---|---|
| "What *did* our warehouse cost, by model / user / query?" | [dbt-bigquery-monitoring](https://github.com/bqbooster/dbt-bigquery-monitoring) |
| "What does the dbt platform estimate my models cost?" | [dbt Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights) |
| "What is **this PR about to do** to our bill?" | **dbt-costgate** |

## Accurate, transparent pricing

BigQuery on-demand rates differ by region — a gate that prices every byte at
the US rate is silently wrong for half the world. dbt-costgate treats pricing
accuracy as a feature:

- 🌍 **Versioned per-region pricing table** with a `last_verified` date, auto-selected from your job's detected region.
- 🧾 **Every report discloses its math** — region, rate, and rate source. Never a silent assumption:

  ```text
  region: US (multi-region) · on-demand USD 6.25/TiB · source: built-in table 2026.07
  ```

- ⚙️ **Overridable** — `pricing.region` to force a region, `pricing.usd_per_tib` for negotiated or editions rates, `pricing.currency` to label amounts in your own currency (an ISO 4217 code — dbt-costgate labels, it never converts).
- ⚠️ **Honest limits, stated up front** — under capacity/editions pricing, bytes scanned is a proxy signal, not your invoice; set a rate of `0` and reports drop money entirely and measure bytes instead. Every priced report's footer discloses the 1 TiB/month on-demand free tier it does **not** deduct — the allowance is per billing account, which a dry-run cannot see, so figures are priced from the first byte.

## Security model

This tool runs in CI next to warehouse credentials, so the design is
deliberately boring:

| Threat | Design answer |
|---|---|
| Billable or data-reading queries | **Dry-run only.** The single warehouse interaction is `jobs.insert` with `dryRun=true` — free, executes nothing |
| Credential theft / mishandling | **No credential surface.** Auth delegates entirely to [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials); in CI the documented path is keyless [Workload Identity Federation](https://github.com/google-github-actions/auth). There are no credential flags to misuse |
| Compromised CI runner | **Least privilege.** BigQuery Job User + metadata read — no data access, no writes; docs ship the exact IAM setup |
| Malicious fork PRs | **Fork-safe by default.** Documented workflows use the `pull_request` trigger; fork PRs degrade to "no report", never to exposed secrets |
| Secrets templated into SQL | **No compiled SQL in reports** — model names, bytes, and dollars only; snippets are strictly opt-in |
| Phone-home | **No telemetry.** The only network call is to the BigQuery API |

Details in [SECURITY.md](SECURITY.md) · deeper design notes in [docs/architecture.md](docs/architecture.md).

## Roadmap

**The MVP roadmap is complete** — every item below ships as of
[v1.0.2](https://github.com/Drichards124/dbt-costgate/releases/latest).

- [x] **`dbt-costgate check`** — local (zero-setup) + CI diff, region-aware pricing, threshold gating
- [x] **One-command local diff** — `dbt-costgate check --against main` (isolated git worktree)
- [x] **GitHub Action** wrapper with a sticky PR comment
- [x] **Absolute cost ceilings** — gate on total `$/run` or `TiB/run`, not just the increase (works without a baseline, so it gates local mode too)
- [x] **Config- and macro-only change detection** — catch a change that reaches a model without touching its `.sql` file
- [x] **`pre-commit` hook** — catch it on your own machine, at pre-push
- [x] **Docker image** — for CI that isn't GitHub Actions; build it yourself, or
- [x] **pull the published image** — `ghcr.io/drichards124/dbt-costgate:v1.0.2`, pushed on every release

**What's next is not another feature.** The list above was written before anyone
had run this against a real warehouse for a month. The useful next step is use —
finding where the numbers, the defaults or the docs are wrong — and the next
features should be the ones that use actually asks for, rather than the ones that
looked obvious from here. Two places that already know what they don't do:
[when the number can be wrong](docs/explained.md#when-the-number-can-be-wrong)
and the non-goals below.

## Non-goals

- **Not a monitoring tool** — retrospective observability belongs to [dbt-bigquery-monitoring](https://github.com/bqbooster/dbt-bigquery-monitoring).
- **BigQuery first** — one warehouse done accurately beats three done approximately. Other warehouses come only once BigQuery is genuinely finished, and only where the cost model actually transfers.
- **Never runs billable queries** — features that require executing real queries are out of scope by design.
- **No IDE/editor integration** (for now).

---

<div align="center">

[Explained](docs/explained.md) ·
[Usage guide](docs/usage.md) ·
[Contributing](CONTRIBUTING.md) ·
[Security policy](SECURITY.md) ·
[Changelog](CHANGELOG.md) ·
[Code of Conduct](CODE_OF_CONDUCT.md) ·
[Apache-2.0](LICENSE) · [NOTICE](NOTICE)

Built by [Dashan Richards](https://github.com/Drichards124) — DCO sign-off required, hard invariants apply:<br/>
**dry-run only · no credential handling · no telemetry**

</div>
