Metadata-Version: 2.4
Name: sofe-dbt
Version: 0.1.0
Summary: ESLint for your data pipelines — FinOps policies as code for dbt, with the real Athena bill attached to every model
Author-email: Carlos Cortez <carlos@cortez.cloud>
License: Apache-2.0
Project-URL: Homepage, https://github.com/breakingthecloud/sofe-dbt
Project-URL: Repository, https://github.com/breakingthecloud/sofe-dbt
Project-URL: Issues, https://github.com/breakingthecloud/sofe-dbt/issues
Keywords: finops,dbt,athena,aws,policy-as-code,cost-optimization,data-engineering
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: sofe>=0.4.4
Requires-Dist: boto3>=1.34
Requires-Dist: pyyaml>=6.0
Requires-Dist: click>=8.0
Requires-Dist: rich>=13.0

# sofe-dbt

**ESLint for your data pipelines.** FinOps policies as code for [dbt](https://www.getdbt.com/) projects — with the **real Athena bill** attached to every model.

`sofe-dbt` reads your dbt artifacts (`manifest.json`, `run_results.json`, `catalog.json`) plus real Athena query statistics, and evaluates FinOps policies against your models. It's built on the [SOFE engine](https://github.com/breakingthecloud/sofe) — *SOFE is ESLint for your cloud; sofe-dbt is ESLint for your data pipelines.*

```bash
$ sofe-dbt check ./
✓ partition-required ...... 42/42 models
⚠ zombie-model ........... stg_legacy_orders (0 consumers)
✗ full-refresh-in-prod ... fct_events (scans 1.2TB daily)
✗ freshness-test-missing . src_salesforce

# 2 errors → PR blocked. Estimated savings: -$380/mo if fixed
```

## Why

Data engineers using dbt know what their models do, whether they pass tests, and whether freshness is met — but they have **zero visibility into cost efficiency**: how much each model costs, whether materialization is optimal, which models are zombies, and which large tables scan full data on every query.

`sofe-dbt` closes that gap. Every dollar shown is derived from real Athena `DataScannedInBytes` — never a heuristic, never invented.

## Install

```bash
pip install sofe-dbt
```

## Quick start

```bash
# 1. Generate dbt artifacts (writes real Athena scan stats into run_results.json)
dbt build

# 2. Evaluate FinOps policies with real cost per model
sofe-dbt check .

# 3. Gate a PR in CI (exit 1 on high-severity findings)
sofe-dbt check . --fail-on high
```

Live Athena cost enrichment (optional, needs AWS creds):

```bash
sofe-dbt check . --profile my-aws-profile
```

## The 10 policies

| Policy | Severity | Category |
|--------|----------|----------|
| `dbt-small-table-should-be-view` | medium | cost |
| `dbt-long-run-not-incremental` | high | cost |
| `dbt-large-table-no-partition` | high | cost |
| `dbt-zombie-model-no-consumers` | medium | governance |
| `dbt-high-blast-radius-no-tests` | critical | reliability |
| `dbt-ephemeral-over-referenced` | medium | cost |
| `dbt-model-no-description` | low | governance |
| `dbt-model-last-run-failed` | high | reliability |
| `dbt-full-refresh-in-prod` | critical | cost |
| `dbt-freshness-test-missing` | medium | reliability |

## Real cost, not estimates

Cost per model comes from **real Athena bytes scanned**:

- **Offline (CI):** dbt-athena writes `data_scanned_in_bytes` into `run_results.json` at `dbt build` time.
- **Live:** Athena `GetQueryExecution` → `QueryExecutionStatistics.DataScannedInBytes`.

```
cost_per_run = max(bytes_scanned, 10 MB) / 1e12 × $5.00/TB   (us-east-1 on-demand)
```

If real bytes aren't available, `sofe-dbt` reports `cost: unknown` — it does not guess.

## Reproducible demo

See [`GUIDE.md`](./GUIDE.md) for an end-to-end walkthrough: deploy the AWS stack (CDK: S3 + Glue + Athena), load golden data, run `dbt build` against real Athena, and see `sofe-dbt check` surface real cost per model. The whole demo costs cents and tears down with `cdk destroy`.

## Setup (development)

```bash
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e ".[dev]"
pytest
```

## License

Apache-2.0
