Metadata-Version: 2.4
Name: bigtesty
Version: 1.3.0
Summary: BigTesty is an integration testing framework for BigQuery
Author: Mazlum TOSUN
Author-email: Mazlum TOSUN <mazlum.tosun@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Database
Classifier: Framework :: Pytest
Requires-Dist: deepdiff>=7.0
Requires-Dist: google-cloud-bigquery>=3.25
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6
Requires-Dist: typer>=0.12
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/tosun-si/bigtesty
Project-URL: Documentation, https://tosun-si.github.io/bigtesty
Project-URL: Repository, https://github.com/tosun-si/bigtesty
Project-URL: Changelog, https://github.com/tosun-si/bigtesty/releases
Description-Content-Type: text/markdown

![bigtesty_logo_with_slogan.png](https://raw.githubusercontent.com/tosun-si/bigtesty/main/bigtesty-doc/src/assets/bigtesty_logo_with_slogan.png)

[![PyPI](https://img.shields.io/pypi/v/bigtesty?label=PyPI)](https://pypi.org/project/bigtesty/)
[![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://pypi.org/project/bigtesty/)
[![Docker](https://img.shields.io/docker/v/groupbees/bigtesty?sort=semver&label=Docker)](https://hub.docker.com/r/groupbees/bigtesty)
[![CI](https://github.com/tosun-si/bigtesty/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/tosun-si/bigtesty/actions/workflows/ci.yml)
[![Documentation](https://img.shields.io/badge/docs-tosun--si.github.io%2Fbigtesty-blue)](https://tosun-si.github.io/bigtesty/)
[![License](https://img.shields.io/github/license/tosun-si/bigtesty)](https://github.com/tosun-si/bigtesty/blob/main/LICENSE)

BigTesty is an integration testing framework for BigQuery, running the tests on a real and short-lived
infrastructure.

**Documentation: [tosun-si.github.io/bigtesty](https://tosun-si.github.io/bigtesty/)**

- **The real BigQuery engine**, no emulator: the SQL behaves exactly as in production.
- **Given / When / Then** test files in YAML or JSON: load test data, run any SQL (queries, DML, `MERGE`,
  scripts, procedures), then assert on query results or table contents.
- **Isolated and ephemeral**: each scenario of each run gets its own datasets, created and destroyed by BigTesty.
  Several developers and CI pipelines can run the tests at the same time.
- **Production SQL unchanged**: the references to your datasets are rewritten to the isolated datasets.

![BigTesty architecture](https://raw.githubusercontent.com/tosun-si/bigtesty/main/bigtesty-doc/src/assets/bigtesty_architecture.png)

## Installation

Python 3.11+ is required.

```bash
# as a dev dependency of a uv project, the version is locked in uv.lock
uv add --dev bigtesty
# or as a standalone CLI
uv tool install bigtesty
```

BigTesty uses [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials):

```bash
gcloud auth application-default login
```

## Quick start

A BigTesty project is a `bigtesty.yaml` config file declaring the datasets and tables, and test files named
`*.bigtesty.yaml` (or `.yml` / `.json`):

```text
bigtesty.yaml
schema/monitoring/job_failure.json
ddl/monitoring/v_failures_count_by_feature.sql
tests/monitoring/
├── find_failures.bigtesty.yaml
├── merge_failures_summary.bigtesty.yaml
├── assertion_functions/failures_assertions.py
├── given/input_failures_feature_name.json
├── queries/find_failures_by_feature_name.sql
├── then/expected_failures_feature_name.json
└── when/merge_failures_summary.sql
```

**`bigtesty.yaml`**: the datasets and tables created for each scenario.

```yaml
project: my-project        # or --project / GOOGLE_CLOUD_PROJECT
location: EU
tests: tests

datasets:
  - id: monitoring
    tables:
      - id: job_failure
        schema_file: schema/monitoring/job_failure.json   # BigQuery JSON schema
        partitioning: { type: DAY, field: dwhCreationDate }
        clustering: [featureName, jobName]
      - id: job_failure_summary
        schema:                                            # or inline
          - { name: featureName, type: STRING }
          - { name: failuresCount, type: INT64 }
          - { name: lastFailureDate, type: TIMESTAMP }
    ddl_files:                                             # views, functions, procedures...
      - ddl/monitoring/*.sql
```

**`tests/monitoring/merge_failures_summary.bigtesty.yaml`**: one or several scenarios.

```yaml
scenarios:
  - name: merge_updates_existing_and_inserts_new_features
    given:
      - table: monitoring.job_failure
        rows:
          - { featureName: featurePSG, jobName: job1, dwhCreationDate: "2026-01-10 10:00:00" }
          - { featureName: featurePSG, jobName: job2, dwhCreationDate: "2026-01-11 10:00:00" }
          - { featureName: featureOM, jobName: job3, dwhCreationDate: "2026-01-12 10:00:00" }
      - table: monitoring.job_failure_summary
        rows:                                   # or rows_file, a JSON or YAML file
          - { featureName: featurePSG, failuresCount: 5, lastFailureDate: "2025-12-31 23:00:00" }
    when:
      sql_file: when/merge_failures_summary.sql
    then:
      - table: monitoring.job_failure_summary
        expected:
          - { featureName: featurePSG, failuresCount: 7, lastFailureDate: "2026-01-11 10:00:00" }
          - { featureName: featureOM, failuresCount: 1, lastFailureDate: "2026-01-12 10:00:00" }
      - query: SELECT featureName FROM monitoring.job_failure_summary GROUP BY featureName HAVING COUNT(*) > 1
        row_count: 0
```

Run the tests:

```bash
bigtesty test
```

```text
PASSED  monitoring/find_failures.bigtesty.yaml::find_failures_by_feature_name  (4.7s, 10.0 MiB billed)
PASSED  monitoring/find_failures.bigtesty.yaml::find_failures_by_feature_name_with_functions  (4.1s, 10.0 MiB billed)
PASSED  monitoring/find_failures.bigtesty.yaml::find_failures_by_job_name  (3.8s, 10.0 MiB billed)
FAILED  monitoring/merge_failures_summary.bigtesty.yaml::merge_updates_existing_and_inserts_new_features  (8.2s, 40.0 MiB billed)

________ monitoring/merge_failures_summary.bigtesty.yaml::merge_updates_existing_and_inserts_new_features ________
then[0] / rows:
    expected 2 rows, got 2: 1 changed
      changed     {featureName: "featurePSG", lastFailureDate: "2026-01-11T10:00:00Z"}  failuresCount: 8 → 7
rerun: bigtesty test -k "monitoring/merge_failures_summary.bigtesty.yaml::merge_updates_existing_and_inserts_new_features"

3 passed, 1 failed, 70.0 MiB billed (run zivarf)
```

A failed `then` shows the changed rows (matched on their equal columns), the missing and the unexpected rows. An
error shows the failed step, its SQL file and the executed SQL at the position of the error:

```text
when (tests/monitoring/when/merge_failures_summary.sql): BadRequest: Unrecognized name: failureCount at [3:3]
job: 2007a27c-0e0e-4ad8-b8f9-194a77adfeec

  1 | UPDATE `my-project.monitoring_zivarf_3.job_failure_summary`
  2 | SET
  3 |   failureCount = failureCount + 1
    |   ^
```

The bytes billed are the ones of the query jobs (DDL, `when` and `then`), BigQuery bills at least 10 MiB per
query. The test data is loaded with load jobs, which are free.

Runnable examples, from the basics to a uv project with its CI, are in the [examples](examples) folder.

## Test files

A scenario has 3 blocks:

- `given`: the rows loaded into the tables, inline (`rows`) or from a JSON / YAML file (`rows_file`).
  The rows are loaded with load jobs: they are immediately available for DML statements.
- `when` (optional): the SQL under test (`sql` or `sql_file`), executed as a BigQuery script: queries, DML,
  `MERGE`, `CALL` of a procedure, multi-statements...
- `then`: one or several assertions on a `query`, a `query_file` or a whole `table`:
  - `expected` / `expected_file`: the expected rows,
  - `row_count`: the expected number of rows,
  - `functions`: custom Python assertion functions, called with `(actual_rows, expected_rows)`.

Options of the `then` blocks:

| Option              | Default | Description                                                                 |
|---------------------|---------|-----------------------------------------------------------------------------|
| `ignore_order`      | `true`  | Compare the rows in any order.                                              |
| `compare_columns`   | `all`   | `expected_only` compares only the columns present in the expected rows.     |
| `ignore_columns`    | `[]`    | Columns excluded from the comparison.                                       |
| `float_tolerance`   |         | Absolute tolerance for the float comparison.                                |

The actual and expected values are compared **with the types of the query result**: a `TIMESTAMP` written
`"2026-01-11 10:00:00"`, `"2026-01-11T10:00:00Z"` or `"2026-01-11T12:00:00+02:00"` is the same value, a `NUMERIC`
can be written `1.5` or `"1.50"`, a `DATE` `"2026-01-11"`...

In YAML, `yes` / `no` / `on` / `off` and the dates stay strings: only `true` / `false` are booleans.

### IDE validation

JSON Schemas are published for the config and the test files. With the VS Code YAML extension or JetBrains IDEs:

```yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/tosun-si/bigtesty/main/schemas/bigtesty-test.schema.json
```

## Production SQL

The production SQL is tested as is, from its own folder, even when it is a template with query parameters:

```yaml
# bigtesty.yaml: the template engine of the team (Jinja, Airflow macros, in-house...), called by BigTesty
sql_renderer: { module: tests/render.py, function: render }
```

```yaml
# test file
scenarios:
  - name: daily_failures
    vars: { ds: "2026-01-10" }                        # rendered by the sql_renderer
    when:
      sql_file: ../../src/monitoring/sql/build_daily_failures.sql
      parameters:                                    # BigQuery query parameters (@name)
        feature_names: [featurePSG, featureOM]
        since: { type: DATE, value: "2026-01-01" }
```

See [Production SQL](https://tosun-si.github.io/bigtesty/writing-tests/test-files/#production-sql) and the
[`06-production-sql`](examples/06-production-sql/) example.

## Isolation and SQL rewrite

For each run, BigTesty creates one dataset per configured dataset and per scenario, e.g. `monitoring_hlnyks_0`,
labelled with `bigtesty_run_id`. The references to the configured datasets in the SQL (DDL files, `when` and `then`)
are rewritten to these datasets, the formatting of the SQL is kept:

- `monitoring.job_failure`, `` `monitoring.job_failure` ``, `` `monitoring`.`job_failure` ``
- `my-project.monitoring.job_failure`, when `my-project` is the test project or listed in `project_aliases`
  (e.g. the production project hardcoded in the SQL).

The comments, string literals and references to other datasets or projects are not touched.
Don't alias a table with the name of a configured dataset (`FROM x AS monitoring`), it would be rewritten.

The datasets are destroyed at the end of the run (`cleanup: always` by default, or `on_success` / `never` to analyse
the result in BigQuery). As a safety net, their tables expire after `table_expiration_hours` (6 by default), and
`bigtesty gc` deletes the datasets left by killed runs.

The config can set `labels` on the datasets and the query jobs (cost per team in the billing export), and a
`kms_key_name` to encrypt the datasets with a customer-managed key (CMEK), see the
[project config](https://tosun-si.github.io/bigtesty/writing-tests/project-config/).

## CLI

```bash
bigtesty test                      # run all the scenarios
bigtesty test -k merge             # only the scenarios whose ID contains "merge"
bigtesty test --cleanup never -v   # keep the datasets, show the executed SQL
bigtesty test --update            # rewrite the expected files that don't match from the actual rows
bigtesty validate                  # validate the config and test files, no call to BigQuery
bigtesty gc --older-than-hours 6   # delete the datasets left by killed runs
bigtesty infra create -o json      # create the infra for the tests of another language
bigtesty infra destroy --run-id x  # and delete it
bigtesty schema -o schemas         # write the JSON Schemas
bigtesty skills install            # copy the Agent Skills into the project
```

| Option           | Env var                                    | Description                                       |
|------------------|--------------------------------------------|---------------------------------------------------|
| `--config`, `-c` | `BIGTESTY_CONFIG`                          | Config file, `bigtesty.yaml` found from the current directory by default. |
| `--project`      | `BIGTESTY_PROJECT`, `GOOGLE_CLOUD_PROJECT` | GCP project, overrides the config.                |
| `--location`     | `BIGTESTY_LOCATION`                        | Default location of the datasets, overrides the config. |

The exit code is `0` when all the scenarios pass, `1` when a scenario fails and `2` for an invalid config.

## Run with pytest

BigTesty is also a pytest plugin, installed with the package: pytest collects the `*.bigtesty.yaml` / `.json` files,
one test per scenario. The pytest features come for free: `-k`, `--lf`, JUnit XML reports for the CI, IDE
integration...

```bash
pytest tests --bigtesty-project my-project
pytest tests -k merge --junitxml=report.xml
```

```text
tests/monitoring/find_failures.bigtesty.yaml::find_failures_by_feature_name PASSED              [ 25%]
tests/monitoring/find_failures.bigtesty.yaml::find_failures_by_feature_name_with_functions PASSED [ 50%]
tests/monitoring/find_failures.bigtesty.yaml::find_failures_by_job_name PASSED                  [ 75%]
tests/monitoring/merge_failures_summary.bigtesty.yaml::merge_updates_existing_and_inserts_new_features PASSED [100%]
```

The ephemeral infra is created once for the selected scenarios, and the scenarios run in parallel on the first
BigTesty test executed: this test shows the duration of the whole run. pytest-xdist is not needed.

| Option                 | Env var                                    | Description                                              |
|------------------------|--------------------------------------------|----------------------------------------------------------|
| `--bigtesty-config`    | `BIGTESTY_CONFIG`                          | Config file, `bigtesty.yaml` found from each test file directory by default. |
| `--bigtesty-project`   | `BIGTESTY_PROJECT`, `GOOGLE_CLOUD_PROJECT` | GCP project, overrides the config.                       |
| `--bigtesty-location`  | `BIGTESTY_LOCATION`                        | Default location of the datasets, overrides the config.  |
| `--bigtesty-cleanup`   |                                            | `always`, `on_success` or `never`, overrides the config. |
| `--bigtesty-update`    |                                            | Rewrite the expected files that don't match from the actual rows. |

The plugin does nothing in a project without BigTesty test files.

## Use the ephemeral infra in Python tests

To test Python code reading or writing BigQuery, `BigTestyInfra` creates the datasets and tables of a `bigtesty.yaml`
once in a setup and deletes them once in a teardown, around as many tests as needed, with pytest or unittest:

```python
import pytest

from bigtesty.testing import BigTestyInfra


@pytest.fixture(scope="class")  # the scope is the lifecycle of the infra
def infra():
    with BigTestyInfra.from_config("bigtesty.yaml") as infra:
        infra.load("monitoring.job_failure", [{"featureName": "featurePSG", "jobName": "job1"}])
        build_failures_summary(infra.client, infra.dataset_id("monitoring"))  # the code under test
        yield infra


class TestFailuresSummary:
    def test_one_row_per_feature(self, infra):
        rows = infra.query("SELECT featureName, failuresCount FROM monitoring.job_failure_summary")
        assert rows == [{"featureName": "featurePSG", "failuresCount": 1}]
```

`infra.query` rewrites the configured datasets as in the test files, `infra.reset()` empties the tables between the
scenarios sharing an infra, and anything callable from Python (a job, a service, dbt...) can run on the infra. The
plugin also provides a session-scoped `bigtesty_infra` fixture. See the [Python tests](https://tosun-si.github.io/bigtesty/running-tests/python-tests/) page
and the [`05-python-tests`](examples/05-python-tests/) example.

The tests of the other languages (Java / JUnit 5, Go...) get the same infra with `bigtesty infra create --output json`
in their setup and `bigtesty infra destroy` in their teardown, with the CLI or the Docker image, Testcontainers
included: see the [Other languages](https://tosun-si.github.io/bigtesty/running-tests/other-languages/) page.

## Run with Docker

```bash
docker run -it \
   -e GOOGLE_CLOUD_PROJECT=$PROJECT_ID \
   -v $(pwd)/examples/01-queries:/usr/local/src/app/project \
   -v $HOME/.config/gcloud:/usr/local/src/app/.config/gcloud \
   groupbees/bigtesty test --config project/bigtesty.yaml
```

Mounting the local `gcloud` configuration authenticates the container with your Application Default Credentials,
without a long-lived service account key.

## Run in CI

The [`04-uv-project`](examples/04-uv-project/) example runs the tests with uv and pytest in
[GitHub Actions](examples/04-uv-project/ci/github-actions.yaml), [GitLab CI](examples/04-uv-project/ci/gitlab-ci.yml)
and [Cloud Build](examples/04-uv-project/ci/cloudbuild.yaml).

## Agent Skills

Four [Agent Skills](https://tosun-si.github.io/bigtesty/start-here/agent-skills/) help a coding agent set BigTesty
up, write the test files and the Python tests, and debug a failed run:

```bash
bigtesty skills install                                   # .claude/skills and .agents/skills of the project
```

```
/plugin install bigtesty --marketplace tosun-si/bigtesty
```

## Migrating from 0.x

BigTesty 1.0 replaces Pulumi by the BigQuery client and introduces a new config and test file format:
`tables.json` becomes a `bigtesty.yaml` config file, the `definition_*.json` files become `*.bigtesty.yaml` /
`*.bigtesty.json` test files, and the `--iac-backend-url` and folder options of the CLI are replaced by `--config`.
Python 3.11+ is required.

The [migration guide](https://tosun-si.github.io/bigtesty/start-here/migration/) gives the mapping of every option
and field, with before / after examples.

## Documentation

The full documentation is on the [BigTesty website](https://tosun-si.github.io/bigtesty/).
