Metadata-Version: 2.4
Name: dbt-cst
Version: 0.1.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
License-File: LICENSE
Summary: Lossless dbt model parser: read, modify and rewrite Jinja templated SQL with its formatting intact
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Repository, https://github.com/livinlefevreloca/dbt-cst

# dbt-cst

A lossless dbt model parser written in Rust with Python bindings. It parses a model (Jinja
templated SQL) into a concrete syntax tree (CST) that keeps every comment, blank line,
indentation and Jinja tag, so a file can be read, modified and written back out with its
formatting intact:

```python
from dbt_cst import parse

doc = parse(source)
assert str(doc) == source  # byte for byte, for every model in albert-dbt-analytics-transforms
```

The tree groups the SQL into queries, clauses, select columns and CTEs, and the Jinja into
blocks, without interpreting either: a column is the text between two commas, a WHERE clause
is the text between its keyword and the next clause keyword. That is enough to add and remove
columns in a model someone has customized by hand, which rewriting the file from a template
cannot do. The Python package is an abi3 wheel built against Python 3.9, so one wheel works on
Python 3.9 and 3.11.

## Python usage

```python
from dbt_cst import parse, DbtSyntaxError

doc = parse(path.read_text())

    doc.config["materialized"]                  # "view"
    doc.config["materialized"] = "table"        # quoted like the call's other strings
    doc.config["tags"] = ["pii"]                # added in the call's layout: one per line, trailing comma
    doc.config.set_raw("schema", 'var("schema")')
    del doc.config["alias"]

    # Example: Modifying multiple config values
    doc.config["materialized"] = "table"
    doc.config["schema"] = "analytics"
    doc.config["tags"] = ["pii", "finance"]
    doc.config["enabled"] = False

    # Example: Using raw expressions for dynamic values
    doc.config.set_raw("alias", 'var("model_alias")')
    doc.config.set_raw("schema", 'var("schema")')

    # Example: Removing config values
    if "description" in doc.config:
        del doc.config["description"]

    doc.sources                                 # [("trading", "apex_apexentry")]
    doc.refs                                    # [("users",), ("package", "model")]

query = doc.query                           # the model's own SELECT, after any CTEs
select = query.select
select.names                                # ["account_id", "activity_id", "percent"]
"account_id" in select                      # True, compared ignoring case

select.add("user_id")                       # appended in the select's comma style
select.add("amount / 100 AS dollars", index=select.index("account_id") + 1)
select.remove("activity_id")                # with its comma and the comments above it
select.find("percent").text = '"percent" AS pct'

renamed = query.cte("renamed").query.select # CTE bodies are queries too
for query in doc.queries():                 # every query, subqueries included
    ...

path.write_text(str(doc))
```

### API

`parse(source) -> Document` raises `DbtSyntaxError` (a `ValueError`) with `line`, `column` and
`offset` attributes. Only a Jinja tag that is never closed is an error. SQL that does not fit
the expected shape still parses: it ends up as unstructured tokens and still round trips.

| Object     | Members                                                                                         |
| ---------- | ----------------------------------------------------------------------------------------------- |
| `Document` | `config`, `query`, `queries()`, `sources`, `refs`, `str()`                                      |
| `Config`   | `c[key]`, `get(key, default)`, `c[key] = value`, `del c[key]`, `raw(key)`, `set_raw(key, src)`, `keys()`, `in`, `len`, `iter` |
| `Query`    | `select` (the first), `selects` (one per UNION branch), `ctes`, `cte(name)`, `clauses`            |
| `Cte`      | `name`, `query`                                                                                  |
| `Clause`   | `name` (`FROM`, `GROUP BY`, `UNION ALL`, ...), `str()`                                           |
| `Select`   | `columns`, `names`, `find(name)`, `index(name)`, `add(text, index=None)`, `remove(name_or_column)`, `in`, `len`, `iter` |
| `Column`   | `text` (settable), `expr`, `alias`, `name`, `is_templated`, `select`, `detach()`                   |

`Config` values are Jinja literals: strings, numbers, booleans, `None`, lists and dicts.
Reading an argument whose value is an expression such as `var("x")` raises `ValueError`; use
`raw()` for its source text.

`Column.name` is the alias, or the last part of a plain column reference (`t.id` gives `id`),
unquoted. It is `None` for an unaliased expression or a bare Jinja tag such as
`{{ dbt_utils.star(...) }}`.

Handles are live: a `Column` from `select.columns` stays attached to the document, and two
handles to the same node compare equal.

## How it parses

1. **Lexing** (`src/lexer.rs`) splits the file into tokens covering every byte. Jinja tags end
   where Jinja says they do, skipping string literals and nested brackets, so
   `{{ {'a': 1} }}` and `{{ "}}" }}` are one tag each. A tag inside a SQL string or comment is
   absorbed into that token. `{% raw %}` sections are one token.
2. **Grouping** (`src/parser.rs`) pairs `(`/`)`, `[`/`]` and Jinja block tags with one stack.
   Any `{% name %}` whose `{% endname %}` appears in the file is opened tentatively, so tags
   from Jinja, dbt (`materialization`, `test`, `snapshot`) and packages all work without a
   list of names. A tag that never closes, like a one line `{% set %}`, or one whose end would
   cross a parenthesis, is put back as a plain token. `elif` and `else` split blocks into
   branches.
3. **Structure**: a top level `SELECT` or `WITH` starts a query, which is split into clauses at
   top level clause keywords. SELECT items are split at top level commas, and WITH clauses into
   CTEs. "Top level" means outside any bracket or Jinja block, so commas in function calls and
   keywords in subqueries never split anything. A Jinja block that supplies its own commas,
   like `{% for c in cols %}{{ c }},{% endfor %}`, separates the columns around it.

The lexer is written with nom, like looker-cst's parser. The one hand written scan is the end
of a `{{ }}` or `{% %}` tag, which tracks bracket depth and skips string literals the way
Jinja's own lexer does.

## Rust usage

```rust
let doc = dbt_cst::parse(&source)?;
let query = doc.query().unwrap();
let select = dbt_cst::query::clauses_named(&query.read(), "SELECT").remove(0);
dbt_cst::edit::insert_item(&mut select.write(), usize::MAX, "user_id")?;
assert!(doc.to_string().contains("user_id"));
```

`cargo run --example dump -- path/to/model.sql` prints the tree.

## Development

```sh
cargo test                           # Rust tests, cloning the corpus into target/corpus on first run
uv run pytest                        # builds the extension with maturin, then runs the Python tests
DBT_EXTRA_REPOS=../albert-dbt-analytics-transforms cargo test --release --test corpus
```

The corpus is the 14 public dbt projects listed in `tests/corpus_repos.txt` (dbt-utils,
dbt-expectations, automate-dv, the Fivetran packages and others, about 1,700 `.sql` files).
Both test suites shallow clone them into `target/corpus` on first use, or into `DBT_CORPUS`
when set. A failed clone skips those files unless `REQUIRE_DBT_CORPUS` is set, as in CI.
`DBT_EXTRA_REPOS` adds more checkouts, separated by `:`.

- `tests/corpus.rs` round trips every file, adds and removes a column in every SELECT,
  removes each column in turn, and adds and removes a config argument.
- `tests/python/test_corpus.py` does the same read, modify and write cycle through the Python
  API, one test per file.
- `tests/fixtures/` holds awkward inputs (CRLF, unicode, unterminated quotes, unbalanced
  brackets, unpaired Jinja blocks) that both suites round trip, and `edge_cases.sql`, a
  customized model whose structure both suites check.
- `tests/python/test_albert_models.py` checks every model in a sibling
  `albert-dbt-analytics-transforms` checkout (or `ALBERT_DBT_REPO`) against the field list
  DBTGenerator reads today.

