Metadata-Version: 2.5
Name: dbt-dqm
Version: 0.2.1
Summary: Persistent dbt test issue tracking and a local review application
Project-URL: Homepage, https://github.com/a-adamovich/dbt-dqm
Project-URL: Repository, https://github.com/a-adamovich/dbt-dqm
Project-URL: Issues, https://github.com/a-adamovich/dbt-dqm/issues
Project-URL: Changelog, https://github.com/a-adamovich/dbt-dqm/blob/main/CHANGELOG.md
Author: Aliaksei Adamovich
License-Expression: Apache-2.0
License-File: LICENSE.md
Keywords: bigquery,data-observability,data-quality,data-testing,dbt,postgres
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: <3.15,>=3.11
Requires-Dist: dbt-core==1.12.5
Requires-Dist: mashumaro<3.18,>=3.17
Requires-Dist: pandas<4,>=2.2
Requires-Dist: platformdirs<5,>=4
Requires-Dist: pyarrow<26,>=25.0.1
Requires-Dist: pyyaml<7,>=6
Requires-Dist: sqlparse<0.7,>=0.6.0
Requires-Dist: streamlit<2,>=1.52
Requires-Dist: urllib3<3,>=2.8.0
Requires-Dist: xlsxwriter<4,>=3.2
Provides-Extra: bigquery
Requires-Dist: dbt-bigquery==1.12.1; extra == 'bigquery'
Requires-Dist: google-cloud-bigquery<4,>=3.25; extra == 'bigquery'
Requires-Dist: oauthlib<5,>=4.0.0; extra == 'bigquery'
Provides-Extra: postgres
Requires-Dist: dbt-postgres<1.12,>=1.11; extra == 'postgres'
Description-Content-Type: text/markdown

# dbt-dqm

dbt tests tell you when data is wrong. When the same test runs again, it can be hard to tell which
records are still affected, which ones returned, and what someone has already done about them.
**dbt-dqm turns selected failed rows into persistent, reviewable issues.**

Choose the columns that identify a record, such as `customer_id`. dbt-dqm uses them to recognize
the same issue across runs, even when other values change. It keeps the captured row, tracks when
the issue appears, passes, or recurs, and preserves owner assignments and review notes. A local
browser app lets reviewers triage the resulting backlog; CSV and Excel exports let you share it.

Use it for dbt data tests whose failures need follow-up at the record level. Your regular dbt
workflow stays separate: the dedicated `dbt-dqm run` command enables and runs the tests you mark
for DQM, captures their results, and updates the issue history.

## Get started

Start with an existing dbt project, a configured profile, and a supported warehouse adapter.
dbt-dqm has **two installations**: the dbt package supplies warehouse models and macros; the
Python package supplies the `dbt-dqm` command and local review app. Use matching `0.2.1` versions.

1. Add the dbt package to your project's `packages.yml`, then install it:

   ```yaml
   packages:
     - git: "https://github.com/a-adamovich/dbt-dqm.git"
       revision: v0.2.1
   ```

   ```bash
   dbt deps
   ```

   [dbt Hub registration](https://github.com/dbt-labs/hubcap/pull/464) is pending. Use the Git
   revision above until the Hub listing and installation are verified.

2. In the Python environment that runs your dbt project, install the command with the adapter you
   use:

   ```bash
   python -m pip install 'dbt-dqm[bigquery]==0.2.1'
   # or: python -m pip install 'dbt-dqm[postgres]==0.2.1'
   ```

   The Python command supports Python 3.11–3.14; the dbt package supports dbt Core
   1.11–1.12. See [installation and compatibility](docs/operations.md#installation-and-compatibility)
   for adapter-specific requirements.

3. Add a singular test such as `tests/customers_missing_email.sql`. Each returned row is a
   failure. `granularity` names the columns that identify the issue; the other columns are
   captured as values that can change over time.

   ```sql
   {{ config(
       enabled=var('dqm_enabled', false),
       store_failures=true,
       limit=100,
       meta={'dbt_dqm': {
           'granularity': ['customer_id'],
           'owner': 'customer_data'
       }}
   ) }}

   select customer_id, email
   from {{ ref('customers') }}
   where email is null
   ```

   Set retention periods for the captured execution and event history in `dbt_project.yml`:

   ```yaml
   vars:
     dbt_dqm_retention_days: 30
     dbt_dqm_event_retention_days: 90
   ```

   The test is disabled during ordinary dbt commands. `dbt-dqm run` enables metadata-marked DQM
   tests for its own workflow. You can also configure generic tests and assign owners from a
   failure-row column; see [test configuration](docs/using-dbt-dqm.md#configure-tests).

4. Build the upstream models as you normally would, then run the DQM workflow:

   ```bash
   dbt-dqm run
   ```

   The first run creates the tracking relations. Later runs update existing issues, archive
   issues whose tests conclusively pass, and recognize recurrences. A failing dbt test can make
   this command exit `1` **after** its failure evidence has been captured and reconciled. The
   completion line reports test outcomes, captured issues, and collections that reached their
   limit. Add `--summary` to see changes and active issues by owner. See
   [running DQM](docs/using-dbt-dqm.md#run-and-interpret-results) for selection, deferral,
   summaries, and limit behavior.

## Review the results

Run `dbt-dqm app` to open the local browser app. Sync the saved issues, assign owners, record a
review verdict or notes, and use **Apply and sync** to submit edits. You can stage bulk changes
and export filtered saved issues across pages as CSV or Excel. The app uses the same dbt project,
profile, target, and naming variables as the command. See the [review app guide](docs/review-app.md)
for its controls and limits.

## More guidance

- [Configure tests and run DQM](docs/using-dbt-dqm.md): identity, ownership, collection limits,
  CLI options, lifecycle, and owner summaries.
- [Operate and upgrade](docs/operations.md): installation details, naming, authentication,
  maintenance, resets, upgrades, and troubleshooting.
- [Try the demo projects](docs/operations.md#demo-projects) or inspect the
  [warehouse contract](docs/warehouse-interfaces.md) and [scale measurements](docs/scale.md).
- [Stable 0.2.1 release notes](docs/release-notes-0.2.1.md) and
  [verification record](docs/release-0.2.1.md).

dbt-dqm is Apache-2.0 open source. See the [roadmap](docs/roadmap.md) and [license](LICENSE.md).
