Metadata-Version: 2.5
Name: ripple-sql
Version: 0.1.7
Summary: Offline column-level SQL lineage. See what breaks before you merge.
Project-URL: Homepage, https://github.com/bteh/ripple
Project-URL: Repository, https://github.com/bteh/ripple
Project-URL: Issues, https://github.com/bteh/ripple/issues
Project-URL: Changelog, https://github.com/bteh/ripple/blob/main/CHANGELOG.md
Author: Brian Teh
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-engineering,dbt,impact-analysis,lineage,sql
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: jinja2<4,>=3.1
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: sqlglot<31,>=30.14
Provides-Extra: benchmark
Requires-Dist: acryl-datahub==1.6.0.16; extra == 'benchmark'
Requires-Dist: openlineage-sql==1.52.0; extra == 'benchmark'
Requires-Dist: sqllineage==1.5.8; extra == 'benchmark'
Provides-Extra: dev
Requires-Dist: lkml>=1.3; extra == 'dev'
Requires-Dist: pytest-timeout>=2.3; extra == 'dev'
Requires-Dist: pytest-xdist>=3.5; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: looker
Requires-Dist: lkml>=1.3; extra == 'looker'
Description-Content-Type: text/markdown

# Ripple

![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![License](https://img.shields.io/badge/license-Apache--2.0-green)

**See what a SQL change breaks before you merge.**

Point it at a dbt project or a folder of `.sql` files. It reads the SQL offline and answers the question you actually have:

```console
$ ripple breaks ecom.raw_orders.order_total

8 columns in 6 models affected by ecom.raw_orders.order_total
Hit hardest: stg_orders (2), orders (2), customers (1)

  stg_orders · 2 columns · 1 hop
    order_total_cents ← ecom.raw_orders.order_total
    order_total ← ecom.raw_orders.order_total
  orders · 2 columns · 2 hops
    order_total_cents ← stg_orders.order_total_cents
    order_total ← stg_orders.order_total
  customers · 1 column · 3 hops
    lifetime_spend ← orders.order_total
  metric:order_total · 1 column · 3 hops
    value ← orders.order_total
  semantic:orders.order_total_dim · 1 column · 3 hops
    order_total_dim ← orders.order_total
  metric:lifetime_spend · 1 column · 4 hops
    value ← customers.lifetime_spend
```

Four hops out it reaches `customers.lifetime_spend` and the two metrics built on it. Now you know before you touch it, instead of after someone's dashboard goes flat.

The other direction, for when you inherit a table and have no idea where its numbers come from:

```console
$ ripple trace customers.lifetime_spend

customers.lifetime_spend comes from:
  orders.order_total → customers.lifetime_spend
    stg_orders.order_total → orders.order_total
      ecom.raw_orders.order_total → stg_orders.order_total
```

Both of those are real output from [dbt's own jaffle-shop](https://github.com/dbt-labs/jaffle-shop). Clone it and run them yourself.

No warehouse connection, no account, no signup. Nothing leaves your machine.

## Try it

```bash
# inside your dbt project or SQL repo
uvx ripple-sql breaks stg_payments.amount
```

Or add it to your MCP client (Claude Code, Cursor, Codex, and others) as a tool:

```bash
claude mcp add ripple -- uvx ripple-sql mcp
```

Then ask: *"what breaks if I change `orders.amount` to cents?"* If the client
lists ripple but the tools do nothing, `ripple doctor` tells you why.

## Is it actually right?

Most SQL lineage tools are confidently wrong, so Ripple ships a benchmark you can rerun instead of a claim you have to trust. Real public dbt projects, hand-verified column lineage, every tool scored by the same function. The corpus and the harness are both in this repo.

| Round | cases | Ripple | DataHub | sqllineage | openlineage-sql |
|---|---|---|---|---|---|
| in-sample | 122 | **122 (100%)** | 98 (80%) | 38 (31%) | 30 (25%) |
| held-out 1 | 54 | **27 (50%)** | 22 | 5 + crash | 8 |
| held-out 2 | 62 | **29 (47%)** | 26 | 0 + crash | 4 |
| held-out 3 | 56 | **26 (46%)** | 19 | 0 + crash | 3 |
| held-out 4 | 56 | **16 (29%)** | 13 | 3 | 4 |
| held-out 5 | 56 | 30 (54%) | **31** | 3 | 3 |
| held-out 6 | 52 | 21 (40%) | **23** | 7 | 6 |
| held-out 7 | 45 | **23 (51%)** | 17 | 0 + crash | 13 |
| held-out 8 | 56 | **45 (80%)** | 24 | 11 | 7 |
| held-out 9 | 42 | **26 (62%)** | 25 | 11 | 8 |
| held-out 10 | 42 | **26 (62%)** | 20 | 10 | 8 |
| held-out 11 | 46 | **7 (15%)** | 5 | 5 | 0 |
| held-out 12 | 42 | **14 (33%)** | 14 | 0 + crash | 9 |

Thirteen numbers, because any one alone would mislead you.

The **in-sample** 100% means Ripple was debugged against those cases, so read it
as "how far ahead on hard SQL shapes," not as an accuracy claim. The
**held-out** rounds are the honest ones. Each is pre-registered: the repositories
and commit SHAs are committed and pushed before anything is cloned, the ground
truth is annotated by reading SQL only and independently re-derived by a second
reader, it is frozen and pushed before any tool runs, and the round is scored
once against a pinned commit and published as-is. Nothing is patched
retroactively. Twelve rounds, eight dialects, and every miss classified in
[`benchmark/HOLDOUT.md`](benchmark/HOLDOUT.md).

Read the spread, not an average. Rounds are not poolable: each one samples a
different corner of the world, so 15% and 80% are both true and neither is "the"
accuracy. Conventional dbt projects score at or near perfect on first contact.
Raw-SQL application repos are where every tool falls down, and where Ripple has
its worst rounds too. Round 11 scored 15% because Ripple hit an engine crash on
a versioned-migration estate, which is published in full along with the fix that
followed. Round 12 was a tie. Round 5 and round 6 were losses.

The consistent finding across twelve rounds is not that Ripple wins, it is that
Ripple is right when it answers: recent rounds carry precision at or near 1.0
with zero wrong edges, because the engine refuses instead of guessing.

Nobody else in this space publishes a held-out score, and the one prior vendor
benchmark kept its corpus private. Rerun ours, including the three tools Ripple
is compared against:

```bash
uv pip install -e ".[benchmark]"   # sqllineage, DataHub, openlineage-sql
python benchmark/fetch_corpus.py
python benchmark/run.py
```

When Ripple isn't sure about an edge it says `review_required` instead of guessing, and that uncertainty follows the path: a result reached through one shaky hop is marked shaky even if the last hop looks clean. A typo'd name gets an error and a suggestion, never a false "nothing downstream."

## Big repos

`ripple doctor` spawns the server exactly the way a client does and tells you
what's wrong: wrong python, something polluting the protocol stream, or a slow
start. `ripple doctor --json` is the thing to attach to a bug report.

Point it at a messy monorepo and it works out the shape itself. On a 2,600-file repo with 16 dbt projects nested at different depths, `ripple` with no arguments and no config found all 16, picked a dialect per project, and mapped 13,155 column links. The first run builds an index and takes a couple of minutes at that size. Every question after that comes from cache in about a second.

## What it does

- `ripple`: scan the project. Models found, links mapped, what needs review.
- `ripple breaks <model.column>`: the blast radius. Every downstream column and model affected, plus row-level impact (filters, joins, and window keys that use it).
- `ripple trace <model.column>`: where a column comes from, hop by hop, back to sources.
- `ripple models` / `ripple columns <model>`: find the right names and see each column's fanout, so you know which columns are load-bearing before you touch one.
- `ripple ci --base origin/main`: the blast radius of the models you changed, as a PR-ready comment. `--fail-on breaks` fails the build so a breaking change can't merge. `--select dbt` prints just the models to rebuild, so CI builds only what the diff actually touches:

  ```bash
  models=$(ripple ci --base origin/main --select dbt)
  [ -n "$models" ] && dbt build --select "$models"
  ```

  Computed from git and column lineage: no state backend, no manifest artifact to stash between runs, no meter. dbt's own `state:modified+` rebuilds every descendant of any changed file; column lineage skips descendants that never read what changed. Anything uncertain (an unparseable diff, a changed row filter, a deleted model's readers) widens to `model+` instead of being skipped, so a missed rebuild can't happen silently.
- `ripple graph -o graph.json`: export the full column-lineage graph.
- `ripple breaks orders.order_total --open` (also `trace`): the answer as one self-contained page you can send, with a box to ask the next question. `--html PATH` writes it where you say.
- `ripple ci --html PATH`: the same page for a whole change, one view per changed column; the GitHub Action attaches it to the run and links it from the comment.
- `ripple serve` (or `ripple serve orders.order_total`): the same answer page, live, opening on the whole project. Every model sits in its layer from the source tables on the left to the dashboards on the right, with the links between them. Hover a model to light its chain, click it for its columns, click a column to ask. Type in the box to filter the map or to ask by name. Under the map, where coverage stands: which external tables still block links and whether query history has been brought in. Each answer comes from your files as they are now, so edit a model and ask again. Localhost only, read-only, nothing sent.
- `ripple doctor`: verify this install can serve your MCP client, and print the exact config command with absolute paths.

## Deterministic answers for AI-written SQL

An agent will write SQL faster than you can review it, and "the run succeeded and the row counts matched" is not evidence that it wrote the right thing. Ripple is a parser, not a model. Ask it the same question twice and you get the same answer, with its uncertainty labelled.

Run `ripple ci` on the pull request, or hand your agent the MCP server so it has to check its own
blast radius before opening one.

Ripple is built to be driven by an agent, so the line between what is parsed and what is predicted
is written down and enforced: the engine never guesses, the agent may, and you always know which one
is talking. [`docs/AI-BOUNDARY.md`](docs/AI-BOUNDARY.md) is the rule and the reasoning.

## Bring your warehouse's schemas

Ripple never connects to your warehouse. But your SQL probably reads tables defined outside the repo, and without their columns those models stop at `SELECT *`. Fix: your agent already has warehouse access, so let it fill the gap. Ask it to "resolve ripple's unresolved tables" and it will call `unresolved_tables`, fetch the columns from `information_schema.columns`, and hand them back via `ingest_schema`. They land in `.ripple/schemas.json`, which you can commit so the whole team gets the coverage.

The credential boundary is the point. Your agent holds the connection; Ripple never does.

No agent? Same thing by hand:

```bash
ripple unresolved                 # which tables block the most coverage, worst first
ripple ingest-schema cols.csv     # CSV with a table,column header (or JSON, or - for stdin)
```

## Bring your query history (optional)

Your SQL files say what is supposed to happen. The warehouse's query log says what
actually ran. If `snow`, `bq`, or `databricks` is already set up on your machine,
one command fetches and ingests it:

```bash
ripple collect-usage            # shows what's installed and configured
ripple collect-usage snowflake  # your own queries, last 7 days, no permission needed
ripple usage
```

Collect runs your warehouse CLI with the login you already have. Ripple reads
connection names from its config, never a password or token, and never opens a
warehouse connection of its own. The raw rows go to a temp file, get aggregated,
and are deleted inside the same call; the query text is never stored and never
passes through an AI model. When several connections are configured and none is
your declared default, collect refuses and lists them instead of picking one.
`--scope account` covers everyone's queries; if that needs a grant, the error
contains the exact statement to hand an admin.

No CLI set up? Export by hand and hand the file over:

```bash
ripple ingest-usage history.json
ripple usage
```

Either way you get: which models actually ran in the window and which didn't, plus
the tables your queries read that live outside this repo, which is activity the SQL
files alone can't see (dashboards, scripts, other teams). `ripple usage` with no
data prints the exact export SQL to run.

Two honesty rules, enforced by tests. Ripple never says a model is unused; it says
"not seen in this window" and prints the window, because in any short export a
quarterly job looks identical to a dead one. And a statement with no database context
is counted as "skipped rather than guessed" instead of being matched to whatever
model shares its table's name.

The export never goes through an AI model, and the query text is never stored: Ripple
keeps per-table counts only. JSONL or a JSON array with a `query_text` field works;
Snowflake's `QUERY_HISTORY` column names are understood as-is.

## Does it fit my stack?

**Reads:** dbt projects (with or without a compiled manifest), plain folders of
`.sql` files, several dbt projects nested anywhere in one monorepo, LookML
views, and dbt semantic models and metrics. Your orchestrator makes no
difference; Ripple reads the repo, not the scheduler.

**Dialects:** Snowflake, BigQuery, Databricks, Redshift, Postgres, DuckDB,
Spark SQL. Detected per project, override with `--dialect`. Powered by
[sqlglot](https://github.com/tobymao/sqlglot).

**Partially:** SQL Server (T-SQL) views and table-valued functions trace;
stored procedure bodies surface as a warning instead of wrong lineage.

**Not today:** SQL written inside Python strings (Airflow operators, Superset,
most Python ETL), DataFrame pipelines (PySpark, pandas), streaming, and
BI-internal lineage (Tableau, Power BI). Pointing Ripple at a repo like that
finds little or nothing, and it will tell you so rather than report an empty
graph as if it were an answer.

## Cost

Ripple is Apache-2.0 and everything here is free. There is no paid version and
no plan for one. The CLI stays account-free and telemetry-free, nothing phones
home, and accuracy is never a paid tier. [`docs/VALIDATION.md`](docs/VALIDATION.md)
shows how to verify that rather than take it on trust.

## License

Apache-2.0. Free to use, self-host, modify, and ship inside your own tooling.
