Metadata-Version: 2.4
Name: dbt-costgate
Version: 0.9.0
Summary: BigQuery cost gate for dbt pull requests: dry-run changed models, diff the bytes, price the damage before it merges.
Project-URL: Homepage, https://github.com/Drichards124/dbt-costgate
Project-URL: Repository, https://github.com/Drichards124/dbt-costgate
Project-URL: Issues, https://github.com/Drichards124/dbt-costgate/issues
Project-URL: Changelog, https://github.com/Drichards124/dbt-costgate/blob/main/CHANGELOG.md
Author: Dashan Richards
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: analytics-engineering,bigquery,ci,cost,dbt,dry-run,finops
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Requires-Dist: google-cloud-bigquery>=3.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

<div align="center">

<img src="docs/assets/logo.svg" alt="dbt-costgate logo" width="96" height="96"/>

# dbt-costgate

**The BigQuery cost gate for dbt pull requests.**

Dry-run what changed, price the diff, and catch the $500-a-day model<br/>*before* it merges — not on next month's bill.

[![CI](https://github.com/Drichards124/dbt-costgate/actions/workflows/ci.yml/badge.svg)](https://github.com/Drichards124/dbt-costgate/actions/workflows/ci.yml)
[![PLE](https://github.com/Drichards124/dbt-costgate/actions/workflows/ple.yml/badge.svg)](https://github.com/Drichards124/dbt-costgate/actions/workflows/ple.yml)
[![Python](https://img.shields.io/badge/python-3.9%20%E2%80%93%203.13-3776AB?logo=python&logoColor=white)](pyproject.toml)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230)](https://docs.astral.sh/ruff/)
[![Status](https://img.shields.io/badge/status-MVP-brightgreen)](#roadmap)

[Quick start](#quick-start) ·
[How it works](#how-it-works) ·
[What you get](#what-you-get-on-every-pr) ·
[Where it fits](#where-it-fits) ·
[Pricing accuracy](#accurate-transparent-pricing) ·
[Security](#security-model) ·
[Roadmap](#roadmap) ·
[Contributing](CONTRIBUTING.md)

**New here? Start with [dbt-costgate, explained](docs/explained.md)** — plain
English, ten minutes, no prior context assumed.

</div>

> [!NOTE]
> **Working MVP.** `dbt-costgate check` and the **GitHub Action** are implemented and
> tested. Every report shown here is **generated from the real renderers** by
> `scripts/gen_samples.py`, and CI fails if any of them drifts from what the code
> actually produces — the figures are illustrative, the output is not. See the
> [usage guide](docs/usage.md) and [changelog](CHANGELOG.md).

---

## The problem

On dbt + BigQuery teams, SQL changes merge with **zero visibility into their cost
impact**. A changed join, a dropped partition filter, or a widened incremental
window can multiply a model's bytes scanned — and the team finds out days later
on the bill, or when finance escalates.

BigQuery's dry-run API returns the *exact* bytes a query would scan — **for
free, before running anything**. dbt-costgate packages that into a first-class PR
gate:

<div align="center">

![How dbt-costgate works: pull request → compile both versions → BigQuery dry-run → price the diff → gate](docs/assets/flow.svg)

</div>

## What you get on every PR

A sticky comment on the pull request, updated in place on every push. This is the
comment itself — GitHub renders it from the same markdown dbt-costgate produces:

<!-- BEGIN GENERATED: pr-comment -->
<!-- Generated by scripts/gen_samples.py from the real renderers. Do not edit by hand. -->
### 💸 dbt-costgate — cost impact of this change (2 models)

| Model | Baseline | This change | Δ % | Δ / run | Δ / month |
|---|--:|--:|--:|--:|--:|
| `fct_orders_daily` _full-refresh_ | 819.20 GiB | 2.91 TiB | +264% | USD +13.19 | USD +395.63 |
| `dim_customers` _new_ | — | 412.50 MiB | — | USD +0.00 | USD +0.07 |

> ⚠ full-refresh — for the rows tagged above, the figure is the cost of rebuilding the table, not of one incremental run.

**Net increase:** USD 13.19/run · USD 395.70/month

❌ **Gate: FAIL**
- fct_orders_daily: USD +13.19/run exceeds USD 5.00
- fct_orders_daily: +264% exceeds 25%

<sub>Pricing: US USD 6.25/TiB · built-in table (table 2026.07, verified 2026-07-25)<br/>Priced from the first byte scanned: BigQuery's 1 TiB/month on-demand free tier is per billing account, so it is disclosed here and never deducted.<br/>Estimates from BigQuery dry-run — nothing executed, no bytes billed, no SQL shown.</sub>
<!-- END GENERATED: pr-comment -->

<details open>
<summary><b>💻 The same check, in your terminal (real output)</b></summary>
<br/>

```bash
dbt-costgate check --baseline path/to/main/manifest.json
```

<!-- BEGIN GENERATED: diff-terminal -->
<!-- Generated by scripts/gen_samples.py from the real renderers. Do not edit by hand. -->
```text
dbt-costgate — region: US · on-demand USD 6.25/TiB · built-in table

  fct_orders_daily  (full-refresh): 819.20 GiB → 2.91 TiB   +264%   USD +13.19/run   USD +395.63/month (30 runs)
  dim_customers  (new): — → 412.50 MiB   —   USD +0.00/run   USD +0.07/month (30 runs)

  ⚠ full-refresh — for the rows tagged above, the figure is the cost of rebuilding the table, not of one incremental run.

  Net increase: USD 13.19/run · USD 395.70/month

  GATE: FAIL
    - fct_orders_daily: USD +13.19/run exceeds USD 5.00
    - fct_orders_daily: +264% exceeds 25%

  Pricing: US USD 6.25/TiB · built-in table (table 2026.07, verified 2026-07-25)
  Priced from the first byte scanned: BigQuery's 1 TiB/month on-demand free tier is per billing account, so it is disclosed here and never deducted.
  Estimates from BigQuery dry-run — nothing executed, no bytes billed, no SQL shown.
```
<!-- END GENERATED: diff-terminal -->

Or run it with no baseline at all for an instant local read of what your changed
models scan — and fail the run there on an absolute `--max-usd-total` /
`--max-tib-total` ceiling (no baseline required) — or get the full before/after
locally in one command with `dbt-costgate check --against main` (dbt-costgate compiles
`main` for you in a throwaway worktree). See the [usage guide](docs/usage.md).

</details>

## Quick start

```bash
pip install dbt-costgate
gcloud auth application-default login

dbt compile
dbt-costgate check
```

That's the entire local setup — no baseline, no CI, no config file. Add a
baseline and thresholds when you want it to *block* a PR; see the
[usage guide](docs/usage.md).

When you do want a config file, `dbt-costgate init` writes one documenting every
setting, all commented out — so it changes nothing until you uncomment
something.

Every [release](https://github.com/Drichards124/dbt-costgate/releases) also ships a
wheel, an sdist, and `SHA256SUMS` if you'd rather pin to an artifact.

## Documentation

So you know where to look before you open anything:

| Document | What's inside | Go here when |
|---|---|---|
| **[Explained](docs/explained.md)** | Plain-English guide: how it works, what it costs to run, which pricing setup you're in, **every config key**, the deliberate non-goals, and when a number can be wrong | You're new, or you want to know what a setting does |
| **[Usage guide](docs/usage.md)** | The how-to: install, CI setup, baselines, thresholds, the GitHub Action, worked examples for on-demand / negotiated / slot pricing | You're setting it up or changing how it runs |
| **[Architecture](docs/architecture.md)** | Why it's built this way, the invariants, the hard edges | You're contributing or reviewing a change |
| **[Security](SECURITY.md)** | Threat model, and what counts as a vulnerability | You're reviewing it for use next to production credentials |
| **[Changelog](CHANGELOG.md)** | What changed in each release, in operator terms | You're upgrading |

Every example report in these docs is generated from the real renderers, and CI
fails if one drifts — so what you read is what the tool actually prints.

## How it works

| Step | What happens | Cost to you |
|------|--------------|-------------|
| 1 · **Find what changed** | dbt's `state:modified` selector against a baseline manifest (your production artifacts), with a git-diff fallback | free |
| 2 · **Compile both versions** | The baseline and PR-branch versions of each changed model | free |
| 3 · **Dry-run each** | BigQuery `dryRun=true` returns exact bytes scanned — executes nothing, reads no table data | **free** |
| 4 · **Price the diff** | Region-aware on-demand rates; optionally × run frequency for $/month | free |
| 5 · **Gate** | Markdown PR comment, machine-readable JSON, policy-driven exit code (fail on a $ and/or % increase, or an absolute $/run or TiB/run ceiling) | free |

## Where it fits

dbt-costgate is the **preventive** half of BigQuery cost control — it deliberately
does not compete with the excellent retrospective tools:

| The question you're asking | Reach for |
|---|---|
| "What *did* our warehouse cost, by model / user / query?" | [dbt-bigquery-monitoring](https://github.com/bqbooster/dbt-bigquery-monitoring) |
| "What does the dbt platform estimate my models cost?" | [dbt Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights) |
| "What is **this PR about to do** to our bill?" | **dbt-costgate** |

## Accurate, transparent pricing

BigQuery on-demand rates differ by region — a gate that prices every byte at
the US rate is silently wrong for half the world. dbt-costgate treats pricing
accuracy as a feature:

- 🌍 **Versioned per-region pricing table** with a `last_verified` date, auto-selected from your job's detected region.
- 🧾 **Every report discloses its math** — region, rate, and rate source. Never a silent assumption:

  ```text
  region: US (multi-region) · on-demand USD 6.25/TiB · source: built-in table 2026.07
  ```

- ⚙️ **Overridable** — `pricing.region` to force a region, `pricing.usd_per_tib` for negotiated or editions rates, `pricing.currency` to label amounts in your own currency (an ISO 4217 code — dbt-costgate labels, it never converts).
- ⚠️ **Honest limits, stated up front** — under capacity/editions pricing, bytes scanned is a proxy signal, not your invoice; set a rate of `0` and reports drop money entirely and measure bytes instead. Every priced report's footer discloses the 1 TiB/month on-demand free tier it does **not** deduct — the allowance is per billing account, which a dry-run cannot see, so figures are priced from the first byte.

## Security model

This tool runs in CI next to warehouse credentials, so the design is
deliberately boring:

| Threat | Design answer |
|---|---|
| Billable or data-reading queries | **Dry-run only.** The single warehouse interaction is `jobs.insert` with `dryRun=true` — free, executes nothing |
| Credential theft / mishandling | **No credential surface.** Auth delegates entirely to [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials); in CI the documented path is keyless [Workload Identity Federation](https://github.com/google-github-actions/auth). There are no credential flags to misuse |
| Compromised CI runner | **Least privilege.** BigQuery Job User + metadata read — no data access, no writes; docs ship the exact IAM setup |
| Malicious fork PRs | **Fork-safe by default.** Documented workflows use the `pull_request` trigger; fork PRs degrade to "no report", never to exposed secrets |
| Secrets templated into SQL | **No compiled SQL in reports** — model names, bytes, and dollars only; snippets are strictly opt-in |
| Phone-home | **No telemetry.** The only network call is to the BigQuery API |

Details in [SECURITY.md](SECURITY.md) · deeper design notes in [docs/architecture.md](docs/architecture.md).

## Roadmap

- [x] **`dbt-costgate check`** — local (zero-setup) + CI diff, region-aware pricing, threshold gating
- [x] **One-command local diff** — `dbt-costgate check --against main` (isolated git worktree)
- [x] **GitHub Action** wrapper with a sticky PR comment
- [x] **Absolute cost ceilings** — gate on total `$/run` or `TiB/run`, not just the increase (works without a baseline, so it gates local mode too)
- [x] **Config- and macro-only change detection** — catch a change that reaches a model without touching its `.sql` file
- [x] **`pre-commit` hook** — catch it on your own machine, at pre-push
- [x] **Docker image** — for CI that isn't GitHub Actions; build it yourself, or
- [x] **pull the published image** — `ghcr.io/drichards124/dbt-costgate:v0.9.0`, pushed on every release

## Non-goals

- **Not a monitoring tool** — retrospective observability belongs to [dbt-bigquery-monitoring](https://github.com/bqbooster/dbt-bigquery-monitoring).
- **BigQuery first** — one warehouse done accurately beats three done approximately. Other warehouses come only once BigQuery is genuinely finished, and only where the cost model actually transfers.
- **Never runs billable queries** — features that require executing real queries are out of scope by design.
- **No IDE/editor integration** (for now).

---

<div align="center">

[Explained](docs/explained.md) ·
[Usage guide](docs/usage.md) ·
[Contributing](CONTRIBUTING.md) ·
[Security policy](SECURITY.md) ·
[Changelog](CHANGELOG.md) ·
[Code of Conduct](CODE_OF_CONDUCT.md) ·
[Apache-2.0](LICENSE) · [NOTICE](NOTICE)

Built by [Dashan Richards](https://github.com/Drichards124) — DCO sign-off required, hard invariants apply:<br/>
**dry-run only · no credential handling · no telemetry**

</div>
