# DBWarden Documentation
> Full documentation for DBWarden - a SQL-first database migration system
> Source: https://dbwarden.emiliano-go.com
> Pages: 132

========================================================================
PAGE: https://dbwarden.emiliano-go.com/advanced/checksum-integrity/
========================================================================

# Checksum Integrity

DBWarden stores a SHA-256 checksum of each migration file at apply time. On subsequent runs, it recalculates the checksum and compares. A mismatch means the file changed after it was applied.

## What checksums protect against

- Accidentally editing an applied migration file
- Copy-paste errors that silently modify historical SQL
- Merge conflicts that land inside already-applied migration files
- Tooling (formatters, editors) modifying migration files in place

## What triggers a mismatch error

```
ChecksumMismatchError: Migration '0004_add_indexes' checksum has changed.
  stored:   a3f8c2e1...
  current:  7b91d40c...
  database: primary

The migration file was modified after it was applied.
Use 'dbwarden history --database primary' to inspect applied migrations.
```

This error blocks `migrate` and `status` from running. DBWarden will not proceed while a checksum is inconsistent.

## Repeatable migrations

Migrations prefixed `RA__` (repeatable, always) or `ROC__` (repeatable on change) behave differently: they are designed to be re-applied. Checksum changes on `ROC__` files trigger re-application on the next `migrate` run. This is expected behavior, not an error.

Checksum mismatch errors only apply to versioned migrations (`V__` prefix).

## Diagnosing a mismatch

```bash
# See which migrations are applied and their checksums
$ dbwarden history --database primary

# Check the current status
$ dbwarden status --database primary
```

Common causes:

1. **Editor auto-format**: your editor reformatted whitespace in the file
2. **Merge conflict**: conflict markers were added/removed inside a migration file
3. **Intentional edit**: someone changed the migration to fix a typo or add a comment

## Resolution: dev environment

In development where the migration has not been applied to shared data:

1. Reset the database state and re-run migrations from scratch, **or**
2. If the change is trivial (whitespace, comment), revert the file to its original state:

```bash
git diff migrations/primary/V__0004_add_indexes.sql
git checkout migrations/primary/V__0004_add_indexes.sql
```

After reverting, the stored checksum and file checksum will match again.

## Resolution: production environment

In production, never modify applied migration files. The resolution is:

1. **Revert the file** to its exact applied state (use git history)
2. Create a **new migration** for any schema changes you need to make

If the file change was accidental and the schema is correct, reverting the file is safe; no data or schema change occurs.

If the file was intentionally changed to fix an error in a migration that was already applied in production, the database schema may already reflect the original (wrong) SQL. Coordinate carefully:

```bash
# 1. Revert the migration file to what was actually applied
git checkout <commit-before-edit> -- migrations/primary/V__0004_add_indexes.sql

# 2. Verify status is clean
$ dbwarden status --database primary

# 3. Create a corrective migration for the actual schema fix
$ dbwarden new "fix index on users" --database primary
```

## When is it safe to ignore?

Never. A checksum mismatch means recorded history diverges from what is on disk. Even if the change appears harmless, proceeding without resolving the mismatch means your migration history is untrustworthy.

## Schema snapshot checksums

DBWarden also writes a **schema snapshot** after each migration:
a JSON file at `.dbwarden/schemas/<migration_id>.schema.json`. Each
snapshot contains a `checksum` field computed from the full snapshot
content via SHA-256, plus a `previous_checksum` field linking it to
the prior snapshot:

```json
{
  "tables": { ... },
  "checksum": "abc123...",
  "previous_checksum": "def456..."
}
```

Snapshots are written atomically (write to temp file, verify, rename)
and read with integrity validation; if the file content doesn't match
the stored checksum, the snapshot is rejected.

The snapshot checksum chain serves a different purpose from the
migration checksum. A migration checksum tells you a specific SQL
file hasn't changed. A snapshot checksum tells you the full schema
state at a given point is intact. If a snapshot is corrupted or
manually edited, `find_latest_snapshot()` falls back to the previous
intact snapshot.

## Preventing mismatches

- Treat versioned migration files as immutable once applied to any shared environment
- Configure your editor to exclude `migrations/` from auto-format
- Use a pre-commit hook to detect changes to applied migration files:

```bash
# .git/hooks/pre-commit (example, adapt to your setup)
$ dbwarden check --database primary
```

`dbwarden check` compares models to the live schema. While not a direct checksum check, it surfaces drift that often accompanies unintended migration file edits.

See also: [Migration Locking](migration-locking.md) | [Migration Files](../migration-files.md) | [Cookbook: Schema Inspection](../cookbook/05-schema-inspection.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/advanced/ci-cd-patterns/
========================================================================

# CI/CD Patterns

Patterns for running DBWarden migrations in automated pipelines.

## Core principle

Run migrations from exactly one job. Serialize migration and deploy. Never run `migrate` in parallel across multiple agents or containers targeting the same database.

## GitHub Actions

### Minimal migration job

```yaml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  migrate:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - run: uv add -e ".[migrations]"

      - name: Check migration status
        run: dbwarden status --database primary
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - name: Apply migrations
        run: dbwarden migrate --database primary
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - name: Verify post-migration status
        run: dbwarden status --database primary
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

  deploy:
    needs: migrate
    runs-on: ubuntu-latest
    steps:
      - name: Deploy application
        run: ...
```

The `needs: migrate` dependency ensures migrations are fully applied before the application starts.

### Preventing concurrent migration runs

```yaml
concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: false
```

`cancel-in-progress: false` queues duplicate runs instead of cancelling mid-flight, which avoids leaving a stale lock on the database.

### Multi-database migration

```yaml
- name: Apply all migrations
  run: dbwarden migrate --all
  env:
    PRIMARY_DATABASE_URL: ${{ secrets.PRIMARY_DATABASE_URL }}
    ANALYTICS_DATABASE_URL: ${{ secrets.ANALYTICS_DATABASE_URL }}
```

Or migrate databases sequentially to control order:

```yaml
- name: Migrate primary
  run: dbwarden migrate --database primary
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}

- name: Migrate analytics
  run: dbwarden migrate --database analytics
  env:
    ANALYTICS_DATABASE_URL: ${{ secrets.ANALYTICS_DATABASE_URL }}
```

### With backup before migration

```yaml
- name: Apply migrations with backup
  run: |
    dbwarden migrate --database primary \
      --with-backup \
      --backup-dir ./migration-backups
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}

- name: Upload backup artifact
  uses: actions/upload-artifact@v4
  with:
    name: migration-backup-${{ github.sha }}
    path: ./migration-backups/
    retention-days: 30
```

## GitLab CI

```yaml
stages:
  - migrate
  - deploy

migrate:
  stage: migrate
  image: python:3.12
  script:
    - uv add -e ".[migrations]"
    - dbwarden status --database primary
    - dbwarden migrate --database primary
    - dbwarden status --database primary
  variables:
    DATABASE_URL: $DATABASE_URL  # set in GitLab CI/CD settings as masked variable
  resource_group: production-database  # prevents concurrent runs

deploy:
  stage: deploy
  needs: [migrate]
  script:
    - ...
```

`resource_group` serializes the migrate job across concurrent pipelines.

## Sandbox testing in PR pipelines

Instead of running against a shared staging database, use `--sandbox`
to apply migrations to a temporary in-memory SQLite database or a
Docker-backed instance. This isolates PR checks from each other:

```yaml
sandbox-check:
  runs-on: ubuntu-latest
  if: github.event_name == 'pull_request'
  steps:
    - uses: actions/checkout@v4
    - run: uv add -e ".[migrations,testcontainers]"
    - name: Apply migrations to sandbox
      run: dbwarden migrate --sandbox --database primary
```

The sandbox starts a fresh database, applies all pending migrations,
reports results, and tears down. It never touches the real database.

## Dry-run check in PR pipelines

Use `--dry-run` to preview SQL without any database access:

```yaml
migration-check:
  runs-on: ubuntu-latest
  if: github.event_name == 'pull_request'
  steps:
    - uses: actions/checkout@v4
    - run: uv add -e ".[migrations]"
    - name: Check for pending migrations
      run: dbwarden status --database primary
      env:
        DATABASE_URL: ${{ secrets.STAGING_DATABASE_URL }}
```

This surfaces "pending migrations exist" warnings in PR checks without
modifying the database.

For a deeper check that validates the SQL actually runs, chain
`--dry-run` before `--sandbox`:

```yaml
- name: Preview SQL
  run: dbwarden migrate --dry-run --database primary

- name: Validate in sandbox
  run: dbwarden migrate --sandbox --database primary
```

## Plan output in deploy pipelines

The `make-migrations --plan` flag prints the generated migration plan
as JSON without writing files. Use it in deploy pipelines to capture
what would be generated as a deploy artifact:

```yaml
- name: Generate migration plan
  run: dbwarden make-migrations --database primary --plan > plan.json

- name: Upload plan artifact
  uses: actions/upload-artifact@v4
  with:
    name: migration-plan-${{ github.sha }}
    path: plan.json
```

The plan JSON includes detected changes, operation counts, and
auto-generated migration names.

## Exit codes

DBWarden exits non-zero on:

- Migration failure
- Checksum mismatch
- Lock acquisition failure
- Configuration error

CI pipelines treat non-zero as job failure by default. No extra configuration needed.

## Recommendations

- Store `DATABASE_URL` as an encrypted secret, not a plain environment variable
- Archive migration output logs as artifacts for audit trails
- Use `dbwarden history` output as a post-migration artifact
- Run `dbwarden status` before and after `migrate`; before confirms what will run, after confirms nothing is pending

See also: [Safe Deployment](safe-deployment.md) | [Credentials and Secrets](../configuration/credentials.md) | [Migration Locking](migration-locking.md) | [Cookbook: Offline & CI](../cookbook/04-offline-ci.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/advanced/migration-locking/
========================================================================

# Migration Locking

DBWarden uses a database-level lock to prevent concurrent schema mutation. This page explains how it works, what happens when it fails, and how to recover from a stuck lock.

## How locking works

When `dbwarden migrate` runs, it:

1. Acquires a lock row in the `dbwarden_lock` table (created on first use; the table name is `dbwarden_lock`, not `_dbwarden_lock`)
2. Executes all pending migrations within that lock
3. Releases the lock on success or failure

The lock is stored in the target database itself: no external service (Redis, filesystem) is required.

If a second `migrate` invocation starts while the first holds the lock, it fails immediately with:

```
LockError: Migration lock is already held. Another migration process may be running.
Use 'dbwarden unlock' to release the lock if necessary.
```

DBWarden does not retry on lock failure. The calling process (CI job, deploy script) must decide whether to retry or abort.

## Inspecting lock state

```bash
$ dbwarden lock-status --database primary
```

Output when unlocked:

```
Migration lock: INACTIVE
```

Output when locked:

```
Migration lock: ACTIVE
Another migration process may be running.
```

Use the `locked_at` timestamp in the lock table to determine whether the lock is held by a live process or is stale.

## When a migration fails mid-run

If a migration raises an error after partial execution:

1. DBWarden rolls back the in-flight transaction (if the database supports transactional DDL, PostgreSQL does, MySQL does not)
2. The lock is released
3. The CLI exits non-zero

For PostgreSQL, partial application within a migration file is rolled back atomically. The migration remains in "pending" state.

For MySQL and databases without transactional DDL, partial application is possible. Inspect the database state manually before retrying.

## Stuck lock recovery

A lock becomes stale when:

- The migration process was killed (SIGKILL, OOM, machine restart)
- A CI job was cancelled mid-run
- A deploy container was stopped before migrate completed

**Before unlocking, confirm no migration is running:**

```bash
# Check if the PID from lock-status is still alive
ps aux | grep <pid>

# Or check your deployment logs / CI job status
```

If the process is genuinely dead:

```bash
# 1. Confirm lock state
$ dbwarden lock-status --database primary

# 2. Inspect migration history to see what ran last
$ dbwarden history --database primary

# 3. Check pending migrations
$ dbwarden status --database primary

# 4. Release the stale lock
$ dbwarden unlock --database primary

# 5. Retry migration
$ dbwarden migrate --database primary
```

## When NOT to use `unlock`

Do not run `unlock` if:

- You are unsure whether a migration process is still running
- The `locked_at` timestamp is recent (within seconds or minutes); the process may still be alive
- Multiple processes share a database and you cannot confirm all are idle

Releasing a lock held by a live migration process will allow a second migration to start concurrently, which can corrupt schema state.

## Preventing concurrent migration in CI

In CI/CD, run migrations from a single job with no parallelism:

```yaml
# GitHub Actions: serialize via job dependency
jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - run: dbwarden migrate --database primary
  deploy:
    needs: migrate
    ...
```

If your pipeline can trigger multiple concurrent deploys, add a concurrency group:

```yaml
concurrency:
  group: migrate-${{ github.ref }}
  cancel-in-progress: false
```

`cancel-in-progress: false` queues the second run instead of cancelling it, which avoids orphaned locks from killed jobs.

### 6. Confirm status

Run `dbwarden status` to verify no pending migrations remain:

```bash
$ dbwarden status --database primary
```

## Distributed locking with Redis

For multi-instance deployments where multiple application replicas could
trigger migrations concurrently, DBWarden provides a Redis-backed
distributed lock through `dbwarden_fastapi.lock`:

```python
from dbwarden_fastapi import migration_lock

# Within a FastAPI route or lifespan:
async with migration_lock() as locked:
    if locked:
        await run_migration()
```

The Redis lock uses `SETNX` + `EXPIRE` with a default TTL of 60 seconds.
If the application crashes while holding the lock, Redis releases it
automatically after the TTL expires. Long-running migrations should
specify a custom TTL or implement lock extension.

The lock is also used internally by the `POST /migrate` FastAPI endpoint
to serialize migration requests across application instances.

### Database-level vs Redis lock

| Aspect | Database lock | Redis lock |
|--------|---------------|------------|
| Scope | CLI commands (`migrate`, `seed`) | FastAPI `POST /migrate` endpoint |
| Storage | `dbwarden_lock` table in the target database | Redis key |
| TTL | No TTL: manual `unlock` required after crash | 60-second default TTL |
| Failure mode | Blocks other CLI commands until released | Auto-released after TTL |
| External dependency | None (uses the database itself) | Redis required |

Both locks can be used independently or together; they guard different
entry points. The database lock protects the CLI; the Redis lock
protects the FastAPI endpoint.

## Lifespan integration

The `dbwarden_lifespan` context manager wraps migration logic and
engine disposal into a single FastAPI-compatible lifespan. When using
the Redis lock in a lifespan, acquire the lock before entering the
migration context:

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from dbwarden_fastapi import dbwarden_lifespan, migration_lock

@asynccontextmanager
async def lifespan(app: FastAPI):
    async with migration_lock():
        async with dbwarden_lifespan(mode="migrate", allow_in_production=True):
            yield
```

See also: [Safe Deployment](safe-deployment.md) | [CI/CD Patterns](ci-cd-patterns.md) | [`lock` commands](../commands/lock.md)

The FastAPI lifespan helper lives in the `dbwarden-fastapi` plugin rather than core: [dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/advanced/safe-deployment/
========================================================================

# Safe Deployment

How to deploy schema changes with minimal risk and a clear recovery path.

## Pre-flight checklist

Before running migrations in production:

- [ ] Confirm no other migration is running (`dbwarden lock-status`)
- [ ] Review pending migrations (`dbwarden status`)
- [ ] Confirm migrations have been tested in staging
- [ ] Take a backup if your database does not have point-in-time recovery

## Standard deploy sequence

```bash
# 1. Verify lock is free
$ dbwarden lock-status --database primary

# 2. Check what will run
$ dbwarden status --database primary

# 3. Apply with backup
$ dbwarden migrate --database primary --with-backup --backup-dir ./backups

# 4. Confirm clean state post-migration
$ dbwarden status --database primary
$ dbwarden history --database primary
```

For multi-database deployments:

```bash
$ dbwarden migrate --all --with-backup --backup-dir ./backups
```

## What happens when a migration fails mid-run

### PostgreSQL (transactional DDL)

PostgreSQL wraps DDL in transactions. If a migration file fails partway through, the entire file is rolled back. The migration remains in "pending" state. The lock is released. You can safely fix the SQL and retry.

```bash
# After a failed migration:
$ dbwarden status --database primary     # confirm migration is still pending
$ dbwarden lock-status --database primary # confirm lock was released

# Fix the migration file, then:
$ dbwarden migrate --database primary
```

### MySQL / databases without transactional DDL

DDL cannot be rolled back. A failed migration may have partially applied changes (e.g., a table was created but the index was not). Manual inspection is required before retrying.

```bash
# Check what the migration was supposed to do
cat migrations/primary/V__0012_add_payment_tables.sql

# Inspect current schema state via your database client
# Determine what was applied and what was not

# Either:
# a) Manually apply the remaining SQL
# b) Create a corrective migration
# c) Roll back manually and retry from scratch
```

## Recovery: stuck lock

If a migration process was killed and the lock was not released:

```bash
# 1. Confirm no migration process is running
$ dbwarden lock-status --database primary

# 2. Inspect history to see the last applied migration
$ dbwarden history --database primary

# 3. Inspect pending state
$ dbwarden status --database primary

# 4. Only if the process is confirmed dead:
$ dbwarden unlock --database primary

# 5. Retry
$ dbwarden migrate --database primary
```

See [Migration Locking](migration-locking.md) for full lock recovery guidance.

## Recovery: failed migration, data is wrong

If a migration applied successfully but produced incorrect data or schema:

**Option A: Rollback** (if the migration has a `-- rollback` section):

```bash
$ dbwarden rollback --database primary
```

This executes the rollback SQL defined in the migration file. Verify the rollback SQL was written when the migration was created; not all migrations include one.

**Option B: Forward fix** (preferred for data migrations):

```bash
# Create a corrective migration
$ dbwarden new "fix column type on payments" --database primary
# Edit the generated file with the corrective SQL
$ dbwarden migrate --database primary
```

Forward fixes are safer than rollbacks for data migrations, as rollback SQL is harder to write correctly after the fact.

## Baseline migrations

For databases that already have a schema (migrating from another tool or brownfield setup):

```bash
$ dbwarden migrate --database primary --baseline --to-version 0005
```

`--baseline` marks migrations as applied without executing them. Use this to tell DBWarden "this database already has schema up to version 0005."

## Smoke test after deploy

After migrations complete, run a quick connectivity and schema check:

```bash
$ dbwarden check-db --database primary
```

`check-db` inspects the live database schema and reports what tables and columns exist. Use this to confirm the schema matches what your application expects.

See also: [Migration Locking](migration-locking.md) | [CI/CD Patterns](ci-cd-patterns.md) | [`rollback` command](../commands/rollback.md) | [Cookbook: Safety & Impact](../cookbook/06-safety-impact.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/architecture-deep-dive/
========================================================================

# Architecture

This page explains DBWarden internals for contributors and advanced debugging.

## Layered architecture

```text
CLI (Typer)
  -> Commands layer
    -> Engine layer (planning/parsing/version/checksum/model discovery)
      -> Repository layer (migration + lock records)
        -> Database layer (SQLAlchemy connection + SQL execution)
```

## Responsibilities

- CLI: parse args, global flags (`--dev`, `--strict-translation`, `--help`)
- Commands: orchestrate workflows (`migrate`, `rollback`, `make-migrations`, `status`, `history`, `check`, `diff`, `generate-models`, `export-models`, `seed`, `lock-status`, `unlock`, `init`, `snapshot`, `settings`, `database`, `version`)
- Engine: parse files, resolve ordering, checksums, model discovery
- Repositories: read/write migration and lock metadata
- Database: execute SQL with backend-aware connections

## Configuration resolution pipeline

When runtime config is requested:

1. discover one config source (`dbwarden.py` or single callsite)
2. fallback to `DBWARDEN_CONFIG_MODULE` when configured
3. import source and execute `database_config(...)` calls
4. validate uniqueness/default/model-path rules
5. resolve selected database and apply `--dev` swap when enabled

Ambiguous sources fail fast.

## Migration execution lifecycle

For `migrate`:

1. ensure migrations metadata table exists
2. ensure lock table exists
3. acquire lock
4. build pending execution plan
5. execute SQL statements
6. record migration metadata/checksums
7. release lock

## Rollback lifecycle

Rollback uses the same lock discipline, selecting rollback SQL from applied files in reverse order.

## Model-to-SQL generation lifecycle

`make-migrations` pipeline:

1. discover model paths
2. import model modules
3. extract table/column metadata
4. load latest schema snapshot (`.dbwarden/schemas/*.schema.json`) if one exists
5. if snapshot exists: **snapshot-diff path**
   - diff model tables against snapshot tables
   - auto-detect table renames from dropped↔added table pairs (column overlap ≥ 0.6)
   - apply user `--rename-table` flags and/or interactive prompts to confirm table renames
   - emit `ALTER TABLE ... RENAME TO` (confirmed) or `DROP TABLE` + `CREATE TABLE` (not confirmed)
   - apply confirmed table renames to snapshot before column processing
   - auto-detect column renames from dropped↔added pairs of the same type
   - apply user `--rename` flags and/or interactive prompts to confirm renames
   - detect column-level changes: type, nullability, default (same-name columns)
   - emit `RENAME COLUMN` (confirmed) or `DROP` + `ADD` (not confirmed)
   - emit `ALTER COLUMN TYPE` / `SET NOT NULL` / `DROP NOT NULL` / `SET DEFAULT` / `DROP DEFAULT`
   - optionally use multi-step safe type change (`--safe-type-change`)
   - order all operations by `StatementOrder` (RENAME_TABLE first) and assemble upgrade/rollback
   - generate upgrade and rollback SQL from the ops
 6. if no snapshot: **live-DB fallback path**
    - take a full schema snapshot from the live database via `extract_full_schema_snapshot()`
    - run standard snapshot-diff pipeline against it (type, nullability, default, FK, index changes)
    - only rename detection is unavailable without a cached snapshot
7. deduplicate against existing migration statements
8. write migration file
9. write companion `.plan.json` metadata file (with `resolved_from` on rename ops)

## PostgreSQL Handler Pipeline

PostgreSQL support is implemented through `dbwarden.engine.backends.postgresql.handlers`. Each handler exposes a small contract: `extract`, `model_spec_from_tables`, `canonicalize`, `diff`, and `emit`.

The `RegistryDriver` runs that contract in order:

1. extract snapshot state into a handler specific shape
2. derive model state from `ModelTable` objects, or config for preamble handlers
3. canonicalize both sides
4. diff into `Op` objects
5. emit backend SQL from the ops

The SQL assembly layer then sorts statements by `StatementOrder` and joins upgrade and rollback SQL into migration files.

### Handler Groups

| Handler | Purpose |
|---------|---------|
| `ColumnHandler` | Column add, drop, type, nullability, default, autoincrement, comment, and backend specific column metadata |
| `ConstraintHandler` | Unique, check, and foreign key constraints |
| `IndexHandler` | PostgreSQL and ClickHouse index operations |
| `TableHandler` | Table create, drop, and table comments |
| `RenameTableHandler` | Table rename operations |
| `SchemaHandler` | Schema create and drop |
| `PgTableHandler` | PostgreSQL table options, inheritance, and exclude constraints |
| `PartitionHandler` | Native PostgreSQL partitioning and partition attachment |
| `StorageParamsHandler` | PostgreSQL storage parameter changes |
| `EnumHandler` | Enum create, drop, and add value |
| `DomainHandler` | Domain create and drop |
| `SequenceHandler` | Sequence create and drop |
| `ViewHandler` | Regular and materialized view changes |
| `PoliciesHandler` | RLS enablement and policy lifecycle |
| `GrantsHandler` | Table and schema grants |
| `RoleHandler` | Role create and alter |
| `DefaultPrivilegesHandler` | Default privilege grants and revokes |
| `FunctionHandler` | Function create, replace, and drop |
| `TriggerHandler` | Trigger create and drop |
| `EventTriggerHandler` | Event trigger lifecycle |
| `ExtendedStatisticsHandler` | `CREATE STATISTICS` and drop |
| `StatisticsHandler` | Extended statistics variants and compatibility helpers |
| `MyTableHandler` | MySQL table metadata |
| `ChTableHandler` | ClickHouse table options and engine recreate |

### Online and Offline Paths

The online path uses `diff_models_against_snapshot` and the registry driver directly.

The offline path uses `diff_model_states`, which still keeps a few raw dict comparisons for state only fields such as column diffs, PostgreSQL table scalars, MySQL table metadata, and table comments. Those raw ops still flow through the same emit layer, so the SQL output stays aligned with the handler path.

The equivalence tests in `tests/test_pg_registry.py` and `tests/engine/snapshot/test_backend.py` lock that behavior in.

### Snapshot write lifecycle (in `migrate`)

After applying versioned migrations, `migrate` calls `_write_migration_snapshot()`:

1. connect to database (respecting sandbox override)
2. extract full schema: tables, columns, types, indexes, constraints, enums
3. compute SHA-256 checksum
4. write `<migration_id>.schema.json` to `.dbwarden/schemas/`
5. on failure: log warning (non-blocking)

Snapshots are not written during `--dry-run`, `--sandbox`, or for repeatable migrations.

## Repeatable migration model

Supported classes:

- versioned (`NNNN_`): run once in ordered sequence
- runs always (`RA__`): run each migrate execution
- runs on change (`ROC__`): run only when checksum changes

## Integrity model

Checksums are recorded for migration content and used for:

- repeatable migration change detection
- migration consistency checks
- audit/debug confidence

## Concurrency model

Migration-mutating commands are serialized by lock state stored in database tables.

Recovery commands:

- `dbwarden lock-status`
- `dbwarden unlock`

## Dev translation path

With `--dev` and SQLite target:

1. extract model types/defaults
2. translate backend-specific constructs
3. fallback behavior in non-strict mode
4. fail-fast behavior with `--strict-translation`

Translation happens during SQL generation, not by mutating existing migration files.

## Error propagation strategy

- config/load validation errors fail early
- execution errors abort current run with context
- lock release is guarded in cleanup paths

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cli-reference/
========================================================================

# CLI Reference

Pure command lookup for DBWarden CLI.

## Syntax

```bash
$ dbwarden [GLOBAL_OPTIONS] COMMAND [ARGS] [COMMAND_OPTIONS]
```

## Global options

| Option | Description |
|---|---|
| `--dev` | Use `dev_database_url` and `dev_database_type` for selected database |
| `--strict-translation` | Fail on unsupported/lossy dev SQLite translation |
| `--debug` | Enable DEBUG-level logging (shows per-file model scanning on `make-migrations`) |
| `--debug-level <LEVEL>` | Set an exact log level: `debug`, `info`, `warning`, `error`, `critical`, or `10`/`20`/`30`/`40`/`50` |
| `--help` | Show help |

`--debug`/`--debug-level` set the logging severity and compose with the
per-command `--verbose`/`-v` flag, which controls INFO-level verbosity. Use both
(`--debug` plus `-v`) for DEBUG diagnostics with verbose INFO output. If both
`--debug` and `--debug-level` are given, `--debug-level` wins.

## Configuration

### `settings show`

```bash
$ dbwarden settings show
$ dbwarden settings show primary
$ dbwarden settings show --all
```

### `database list`

```bash
$ dbwarden database list
```

## Migration authoring

### `make-migrations`

```bash
$ dbwarden make-migrations "create users table" --database primary
$ dbwarden make-migrations --verbose --database primary
$ dbwarden --debug make-migrations --database primary
$ dbwarden --debug-level warning make-migrations --database primary
$ dbwarden make-migrations --plan --database primary
$ dbwarden make-migrations --rename users.username:email --database primary
$ dbwarden make-migrations --rename-table users:accounts --database primary
$ dbwarden make-migrations --safe-type-change --database primary
```

Options:

- `--database`/`-d`: Target database
- `--plan`: Print migration plan JSON without writing files
- `--offline`: Use model state file instead of live database (run `export-models` first)
- `--verbose`/`-v`: Verbose output
- `--debug`/`--debug-level <LEVEL>`: Global options (see Global options table). DEBUG shows every model file scanned during discovery.
- `--rename`: Repeatable. Declare a column rename in format `table.old_name:new_name`.
- `--rename-table`: Repeatable. Declare a table rename in format `old_table:new_table`.
- `--safe-type-change`: Multi-step safe type change strategy.
- `--clickhouse-engine-recreate`: Allow automatic ClickHouse table rebuild on engine change.
- `--drop-preserved-clickhouse-table` / `--keep-preserved-clickhouse-table`: Drop or keep the preserved old ClickHouse table after engine-recreate swap.
- `--type`/`-t`: Output prefix: `versioned` (default), `ra`/`runs_always`, or `roc`/`runs_on_change`.

See [make-migrations](commands/make-migrations.md) for full documentation including rename detection, column-level changes, schema snapshots, and plan format.

### `new`

```bash
$ dbwarden new "manual hotfix" --database primary
$ dbwarden new "backfill" --database primary --version 0042
$ dbwarden new "seed data" --database primary --type ra
```

Options: `--database`, `--version`, `--type`/`-t`

### `generate-models`

```bash
$ dbwarden generate-models --output ./models/ --database primary
$ dbwarden generate-models --database primary --single-file
$ dbwarden generate-models --database primary --tables users,posts
$ dbwarden generate-models --database primary --exclude-tables logs,audit
```

Options: `--output`/`-o` (default `models`), `--tables`, `--exclude-tables`, `--clickhouse-engines`, `--relationships`, `--dialect`, `--single-file`, `--base`, `--database`/`-d`

### `export-models`

```bash
$ dbwarden export-models --database primary
$ dbwarden export-models --database primary --output .dbwarden/model_state.json
```

Exports current model definitions to a JSON state file for offline migration diffs.

> **Important:** The model state file is used for offline migration generation. It is auto-generated and committed to version control. If accidentally deleted, restore it from git or regenerate it by running `dbwarden export-models --database <db>` against a live database. Without it, offline commands like `make-migrations --offline` will not work, but online operations are unaffected.

Options: `--output`/`-o` (default `.dbwarden/model_state.json`), `--database`/`-d`

### `diff`

```bash
$ dbwarden diff --database primary
$ dbwarden diff --database primary --out json
$ dbwarden diff --database primary --out sql
$ dbwarden diff --database primary --offline
```

Read-only model-vs-database comparison. No files are written.

Options: `--database`/`-d`, `--out`/`-o` (`table`, `json`, `sql`), `--offline`, `--verbose`/`-v`

### `check-impact`

```bash
$ dbwarden check-impact 0042 --database primary
$ dbwarden check-impact 0042 --database primary --out json
$ dbwarden check-impact 0042 --database primary --scan-path app/
$ dbwarden check-impact path/to/primary__0042_add_bio.plan.json
```

Scans your codebase for references to schema elements affected by a migration.

| Option | Description |
|--------|-------------|
| `migration` | Migration version (e.g. `0042`) or plan file path (required) |
| `--out`/`-o` | Output format: `text` (default) or `json` |
| `--scan-path` | Directory to scan for affected code (default: `.`) |
| `--deep` | Enable deep introspection (imports models live) |
| `--verbose`/`-v` | Include INFO-level operations in the scan |
| `--database`/`-d` | Target database name |

## Migration execution

### `migrate`

```bash
$ dbwarden migrate --database primary
$ dbwarden migrate --all
$ dbwarden migrate --database primary --to-version 0010
$ dbwarden migrate --database primary --count 2
$ dbwarden migrate --database primary --with-backup
$ dbwarden migrate --database primary --baseline --to-version 0005
```

Options:

- `--database`, `--all`
- `--to-version`, `--count`
- `--baseline`
- `--with-backup`, `--backup-dir`
- `--dry-run` (show what would be applied without executing)
- `--sandbox` (apply in a temporary sandbox database)
- `--apply-seeds` (apply pending seeds after migrations, overrides config)
- `--verbose`

### `rollback`

```bash
$ dbwarden rollback --database primary
$ dbwarden rollback --database primary --count 2
$ dbwarden rollback --database primary --to-version 0007
```

Options: `--database`, `--count`, `--to-version`, `--verbose`

### `downgrade`

```bash
$ dbwarden downgrade --to 0005 --database primary
```

Options: `--to` (required), `--database`, `--verbose`

### `make-rollback`

```bash
$ dbwarden make-rollback migrations/primary__0005_add_table.sql
```

Generates a `.rollback.sql` file for the given migration file.

### `snapshot`

```bash
$ dbwarden snapshot users --database primary
```

Outputs the DDL schema of the specified table.

## Seed management

### `seed create`

```bash
$ dbwarden seed create "seed initial data" --database primary
$ dbwarden seed create "populate lookup tables" --database primary --type python
```

Options: `--database`, `--type` (`sql` or `python`, default `sql`), `--verbose`

### `seed apply`

```bash
$ dbwarden seed apply --database primary
$ dbwarden seed apply --database primary --version 0003
$ dbwarden seed apply --database primary --dry-run
$ dbwarden seed apply --all
```

Options: `--database`, `--all` (`-a`), `--version`, `--dry-run`, `--verbose`

### `seed list`

```bash
$ dbwarden seed list --database primary
$ dbwarden seed list --all
$ dbwarden seed list --prune
```

Options: `--database`, `--all`, `--prune`, `--verbose`

### `seed rollback`

```bash
$ dbwarden seed rollback --database primary
$ dbwarden seed rollback --database primary --count 2
$ dbwarden seed rollback --database primary --to-version 0003
$ dbwarden seed rollback --all
```

Options: `--database`, `--count`, `--to-version`, `--all`, `--verbose`

### `seed export`

```bash
$ dbwarden seed export --database primary
$ dbwarden seed export --all
$ dbwarden seed export --database clickhouse --output-dir ./seeds
```

Export code seeds to ROC SQL files for stateless production application.

Options: `--database`/`-d`, `--all`/`-a`, `--output-dir`/`-o` (default `seeds/`)

## Inspection and diagnostics

### `status`

```bash
$ dbwarden status --database primary
$ dbwarden status --all
```

### `history`

```bash
$ dbwarden history --database primary
```

### `check-db`

```bash
$ dbwarden check-db --database primary
$ dbwarden check-db --database primary --out json
```

Output formats: `txt`, `json`, `yaml`, `sql`

### `check`

```bash
$ dbwarden check --database primary
$ dbwarden check --database primary --force
$ dbwarden check --database primary --out json
```

Output formats: `txt`, `json`

## Locking

### `lock-status`

```bash
$ dbwarden lock-status --database primary
```

### `unlock`

```bash
$ dbwarden unlock --database primary
```

## Plugin management

See the [Plugins guide](plugins/index.md) for the trust model and development docs.

### `plugin list`

```bash
$ dbwarden plugin list
$ dbwarden plugin list --format json
```

Shows discovered plugins with tier, trust/load state, registered hooks, object handlers, and lock status.

Options: `--format`/`-f` (`table` or `json`, default `table`)

### `plugin info`

```bash
$ dbwarden plugin info dbwarden-fastapi
$ dbwarden plugin info dbwarden-fastapi --format json
```

Shows entry point, tier, trust/load state, hooks, official repository, approved minimum version, and lockfile provenance. Exits `1` if the plugin is not found.

Options: `--format`/`-f` (`table` or `json`, default `table`)

### `plugin add`

```bash
$ dbwarden plugin add dbwarden-fastapi
$ dbwarden plugin add dbwarden-fastapi --version 0.2.0 --uv
$ dbwarden plugin add dbwarden-example --dry-run
```

Installs a plugin. Official plugins are provenance-verified and fail closed if verification is unavailable; community plugins are installed but not trusted (run `plugin trust` next).

Options: `--uv` (use `uv add` instead of pip), `--version` (pin an exact version), `--dry-run` (print the plan without installing)

### `plugin remove`

```bash
$ dbwarden plugin remove dbwarden-example
$ dbwarden plugin remove dbwarden-example --dry-run --uv
```

Uninstalls the distribution and removes its consent and lockfile entries.

Options: `--uv` (use `uv remove` instead of pip), `--dry-run` (print the plan without uninstalling)

### `plugin trust`

```bash
$ dbwarden plugin trust dbwarden-example
```

Records consent for the installed version of a community plugin in `.dbwarden/consent.toml`. Consent is version-specific.

### `plugin untrust`

```bash
$ dbwarden plugin untrust dbwarden-example
```

Revokes consent for a community plugin.

## Utility

### `config`

```bash
$ dbwarden config
```

### `version`

```bash
$ dbwarden version
```

For worked command examples, see the [Cookbook & Examples](cookbook/index.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/codebase/
========================================================================

# Codebase Organization

## Top-Level Layout

```
dbwarden/        # The package itself
tests/           # Test suite (~40 modules)
docs/            # Documentation site (MkDocs)
examples/        # Runnable example projects
scripts/         # Development and CI tooling
assets/          # Images, icons, branding
site/            # Built documentation output (gitignored)
```

## Package Layout (`dbwarden/`)

| Directory / Module | Responsibility |
|---|---|
| `cli/` | Typer CLI definitions, argument parsing |
| `commands/` | Command orchestration (migrate, generate-models, check, etc.) |
| `engine/` | Core logic: model discovery, snapshot extraction, diff, offline migration, safety checks |
| `database/` | Connection management, SQL queries by dialect |
| `databases/` | Concrete backend specs: ClickHouse, MySQL, PostgreSQL, MariaDB, SQLite |
| `schema/` | Dialect-agnostic metadata layer: table/column/field metadata classes |
| `repositories/` | Migration and lock metadata persistence |
| `fastapi/` | FastAPI integration (lifespan, health checks) |
| `config*.py` | Configuration loading and resolution |
| `constants.py` | Shared constants |
| `exceptions.py` | Exception hierarchy |
| `seed.py` | Seed data infrastructure |
| `sandbox.py` | Module loading sandbox for user model files |

## The `schema/` vs `databases/` Boundary

The `dbwarden/schema/` package is the abstract metadata layer. It defines dialect-agnostic constructs that make no assumptions about the target database:

- `TableMeta` and `*ColumnMeta` classes (e.g. `PGColumnMeta`, `CHColumnMeta`, `MyColumnMeta`)
- `DBWardenMeta`: the runtime metadata container attached to each model
- `_MetaValidator`: metaclass that validates `class Meta` attribute names at import time
- `IndexSpec`, `CheckSpec`, `UniqueSpec`: cross-database object specs
- `_meta_reader.py`: logic that reads `class Meta` from user models and populates `DBWardenMeta`

The `dbwarden/databases/` package is the concrete backend layer. It contains dialect-specific specs and helpers:

- `clickhouse/`: `ChEngineSpec`, `ProjectionSpec`, `ChIndexSpec`, `ChTableSpec`, merge-tree helpers, `ChFieldSpec`
- `mysql/`: `MyFieldSpec`, `MyTableSpec`
- `pgsql/`: `PgFieldSpec`, `PgIndexSpec`, `PgTableSpec`, exclude/partition helpers
- `mariadb/`: `MdbFieldSpec`, `MdbTableSpec`
- `sqlite/`: `SqFieldSpec`, `SqTableSpec`

### The Import Contract

The single most important rule in the codebase is:

> **`schema/` must never import from `databases/`.**

This keeps the metadata layer database-agnostic. `databases/` may import from `schema/` (and does, for `TableMeta`, `DBWardenMeta`, `IndexSpec`, etc.), but the reverse dependency is forbidden.

Consequences of this boundary:

- **`ChEngineSpec` and `ProjectionSpec` live in `databases/clickhouse/`**, not `schema/`. They are ClickHouse-specific types, not abstract schema concepts.
- Backend specs (`ChTableSpec`, `MyTableSpec`, etc.) are defined per-database, not in `schema/`.
- The `schema/__init__.py` only re-exports classes from `schema/` submodules. It does not re-export backend-specific types from `databases/`.
- Users import backend types through `from dbwarden.databases.clickhouse import ChEngineSpec` or the top-level `from dbwarden import ChEngineSpec`.

### What Changed in the v0.13.0 Refactor

The refactor tightened this boundary. Previously, `ChEngineSpec`, `ProjectionSpec`, and the `*FieldMeta` hierarchy lived in `schema/`. They were moved to their correct locations:

- `ChEngineSpec`, `_split_engine_args`: now in `databases/clickhouse/engine.py`
- `ProjectionSpec`: now in `databases/clickhouse/projection.py`
- `*FieldMeta` classes (`PGFieldMeta`, `CHFieldMeta`, etc.): deleted; fields inlined directly into `*ColumnMeta` in `table_meta.py`

The orphan `__pycache__` directories under `schema/{clickhouse,mysql,pgsql,mariadb,sqlite}/` were removed.

## Contribution Guidelines

### Before submitting a PR

1. Ensure your changes respect the `schema/` vs `databases/` import boundary (see above).
2. Run the full test suite before pushing:
   ```
   python -m pytest tests/ -x -q
   ```
3. If you add or remove a public export, update the corresponding `__all__` list in the module's `__init__.py`.
4. If you introduce a new top-level directory, add it to the table in this document.

### Code style

- No comments in production code unless the logic is genuinely subtle.
- Mimic existing patterns: same typing style, same docstring conventions, same import organization.
- Prefer `from __future__ import annotations` at the top of every module.
- Use `metaclass=_MetaValidator` for any new `class Meta`-like user-facing configuration class.

### Adding a new database backend

1. Create a new subpackage under `databases/<name>/` with `__init__.py`, `field.py`, and any backend-specific specs.
2. Define a `*TableSpec` dataclass and a `*FieldSpec` dataclass matching the existing backends.
3. Register the backend in `databases/__init__.py` and add the shortcut import (`sq`, `my`, etc.).
4. If the backend needs no column-level `Meta` attributes (like SQLite), add no `*ColumnMeta` class.
5. Do not touch files in `schema/` unless you are adding cross-database metadata fields.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/check-db/
========================================================================

# `check-db`

Inspect live database schema.

## Usage

```bash
$ dbwarden check-db --database primary
$ dbwarden check-db --database primary --out json
$ dbwarden check-db --database primary --out yaml
```

## Options

- `--database`, `-d`
- `--out`, `-o` (`txt`, `json`, `yaml`, `sql`)

## Notes

- useful for schema inspection and diagnostics
- complements `status` and `history`

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/check/
========================================================================

# `check`

Analyze schema differences between your SQLAlchemy models and the live database.

## Usage

```bash
$ dbwarden check --database primary
$ dbwarden check --database primary --force
$ dbwarden check --database primary --out json
```

## Options

- `--database`, `-d` - Target database
- `--out`, `-o` - Output format: `txt` or `json`
- `--force` - Allow warning-level changes to pass

## Severity model

- `INFO` - safe changes like adding a projection or adding a new object
- `WARNING` - risky changes that require `--force`
- `ERROR` - blocked changes such as partition/order key changes

## Current behavior

DBWarden runs generic safety checks for all backends, covering column type changes, nullability changes, default changes, and table operations. For ClickHouse specifically, additional checks classify changes for:

- added or removed columns
- type changes
- engine changes
- TTL changes
- `ORDER BY` changes
- `PARTITION BY` changes
- materialized view query changes
- projection additions/removals

## Notes

- warning-level changes exit non-zero unless `--force` is provided
- error-level changes remain blocking even with `--force`
- output is based on live database inspection plus current model metadata

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/database/
========================================================================

# `database`

Display configured databases. Config is defined in Python code via `database_config()`, so
`database list` is a read-only command for viewing what's registered.

## Usage

```bash
$ dbwarden database list
```

## See also

- [`settings show`](./settings.md): detailed view of all configuration
- [Configuration docs](../configuration/index.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/diff/
========================================================================

# `diff`

Show structural differences between SQLAlchemy models and a live database.
Read-only: no files are written.

## Usage

```bash
$ dbwarden diff --database primary
$ dbwarden diff --database primary --out json
$ dbwarden diff --database primary --out sql
$ dbwarden diff --database primary --offline
```

## Options

| Option | Description |
|--------|-------------|
| `--database`, `-d` | Target database name |
| `--out`, `-o` | Output format: `table` (default), `json`, `sql` |
| `--offline` | Use exported model state file instead of live DB snapshot |
| `--verbose`, `-v` | Enable verbose logging |

## Output formats

### `table` (default)

Displays a Rich table with columns: Operation, Table, Target, Severity.

```text
          Schema Diff           
┏━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
┃ Operation    ┃ Table ┃ Target ┃ Severity┃
┡━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│ add_column   │ users │ email  │ INFO    │
│ drop_column  │ users │ name   │ WARNING │
└──────────────┴───────┴────────┴─────────┘
```

### `json`

```json
[
  {"operation": "add_column", "table": "users", "target": "email", "severity": "INFO"},
  {"operation": "drop_column", "table": "users", "target": "name", "severity": "WARNING"}
]
```

### `sql`

Prints the raw migration SQL that would be generated.

## Offline mode

Requires a model state file created by `dbwarden export-models`:

```bash
$ dbwarden export-models --database primary
# Switch to offline machine
$ dbwarden diff --database primary --offline
```

## See also

- [`make-migrations`](./make-migrations.md): generates migration files from diffs
- [`check`](./check.md): safety analyzer for schema changes
- [`check-db`](./check-db.md): inspect live database schema

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/downgrade/
========================================================================

# `downgrade`

Revert applied migrations to reach a specific target version.

## Usage

```bash
$ dbwarden downgrade --to 0005 --database primary
```

## Options

- `--to`, `-t` (required) - Target version to downgrade to
- `--database`, `-d`
- `--verbose`, `-v`

## Notes

- reads `-- rollback` sections from migration files and applies them in reverse order
- only reverts versions after the target version; versions at or before the target are preserved
- same lock discipline as `migrate` and `rollback`
- fails if the target version has not been applied

See also: [rollback](rollback.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/generate-models/
========================================================================

# `generate-models`

Reverse-engineer SQLAlchemy model code from a live database.

## Usage

```bash
$ dbwarden generate-models --output ./models/ --database primary
$ dbwarden generate-models --output ./models/ --database primary --single-file
$ dbwarden generate-models --output ./models/ --database primary --base app.database:Base
$ dbwarden generate-models --database primary --tables users,posts
$ dbwarden generate-models --database primary --exclude-tables logs,audit
```

> **Note:** `generate-models` works for all supported databases: PostgreSQL, MySQL, MariaDB, ClickHouse, and SQLite. For ClickHouse, use `--clickhouse-engines` or rely on auto-detection from `database_type="clickhouse"`. SQLite produces basic table models without backend-specific metadata.

## Options

| Option | Description |
|--------|-------------|
| `--output`, `-o` | Output directory (default: `models`) |
| `--tables` | Comma-separated list of tables to include |
| `--exclude-tables` | Comma-separated list of tables to exclude |
| `--clickhouse-engines` | Include ClickHouse engine metadata. Auto-detected when `database_type="clickhouse"` |
| `--relationships` | Generate `relationship()` attributes for foreign keys |
| `--dialect` | SQL dialect for type mapping (auto-detected from database type) |
| `--single-file` | Generate a single `models.py` instead of one file per table |
| `--base` | Custom Base class import path (e.g. `app.database:Base`). Default: generates `declarative_base()` in each file |
| `--database`, `-d` | Target database name |

## Output rules

- **Default**: one `.py` file per table (e.g., `users.py`, `posts.py`)
- **`--single-file`**: generates `models.py` with all models
- Each file imports `declarative_base()` and defines `Base` (or imports from the path given by `--base`)

## Type mapping

Database column types are mapped to SQLAlchemy types:

| Database Type | SQLAlchemy Type |
|---------------|----------------|
| `INTEGER` | `Integer` |
| `VARCHAR(N)` | `String(length=N)` |
| `TEXT` | `Text` |
| `BOOLEAN` / `TINYINT(1)` | `Boolean` |
| `DECIMAL(P,S)` | `Numeric(precision=P, scale=S)` |
| `DATETIME` / `TIMESTAMP` | `DateTime` |
| `BIGINT` | `BigInteger` |
| `FLOAT` / `DOUBLE` | `Float` |
| `Nullable(...)` (ClickHouse) | Inner type (nullable is explicit) |

## PostgreSQL First-Class Output

For PostgreSQL databases, `generate-models` reverse-engineers all supported metadata and emits it as `class Meta` inner classes with `PGTableMeta` and `PGColumnMeta`:

```python
from sqlalchemy.orm import DeclarativeBase
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id:    Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True)
    bio:   Mapped[str | None] = mapped_column(Text, nullable=True)

    class Meta(PGTableMeta):
        comment = "Core user accounts"
        pg_fillfactor = 80

        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100, identity_increment=1)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", collation="en_US.UTF-8")
```

The following metadata is reverse-engineered:

- **Identity columns**: `GENERATED ALWAYS/BY DEFAULT AS IDENTITY` with sequence options
- **Collation**: per-column `COLLATE` setting
- **Storage**: per-column `STORAGE` (PLAIN, MAIN, EXTERNAL, EXTENDED)
- **Generated columns**: `GENERATED ALWAYS AS (...) STORED`
- **Table fillfactor**: `WITH (fillfactor = N)`
- **Tablespace**: `SET TABLESPACE`
- **Inheritance**: `INHERITS (parent)`
- **EXCLUDE constraints**: `EXCLUDE USING ...`
- **FK options**: `ondelete`, `onupdate`, `deferrable` on `ForeignKey()`
- **Index options**: `USING`, `WHERE`, `INCLUDE`, `WITH`, `TABLESPACE`, `NULLS NOT DISTINCT`
- **Table and column comments**

For the complete feature reference, see [PostgreSQL Deep Dive](../databases/postgresql/index.md).

## ClickHouse First-Class Output

For ClickHouse databases, `generate-models` reverse-engineers all supported metadata and emits it as `class Meta` inner classes with `CHTableMeta`, `CHColumnMeta`, `ChEngineSpec`, and `ProjectionSpec`. Engine metadata is included automatically when `database_type="clickhouse"` (no `--clickhouse-engines` flag required).

```python
from sqlalchemy.orm import DeclarativeBase
from dbwarden.databases.clickhouse import CHTableMeta, CHColumnMeta, ChEngineSpec, ProjectionSpec, ch

class Base(DeclarativeBase):
    pass

class Event(Base):
    __tablename__ = "events"

    id:    Mapped[int] = mapped_column(Int64, primary_key=True)
    event_date: Mapped[date] = mapped_column(Date)
    payload: Mapped[str] = mapped_column(String)

    class Meta(CHTableMeta):
        ch_engine = ChEngineSpec("MergeTree")
        ch_order_by = ["event_date", "id"]
        ch_partition_by = "toYYYYMM(event_date)"
        ch_ttl = ["event_date + toIntervalYear(1)"]
        ch_settings = {"index_granularity": "8192"}
        ch_projections = [
            ProjectionSpec("by_date", "SELECT event_date, sum(amount) GROUP BY event_date"),
        ]

        class payload(CHColumnMeta):
            ch = ch.field(codec="ZSTD(3)")
```

The following metadata is reverse-engineered:

- **Engine spec**: engine name, arguments, ZooKeeper path, replica name, settings via `ChEngineSpec`
- **Ordering and partitioning**: `ch_order_by`, `ch_primary_key`, `ch_partition_by`, `ch_sample_by`
- **TTL**: table-level TTL expressions
- **Projections**: named projections via `ProjectionSpec`
- **Materialized views**: `ch_select_statement`, `ch_to_table`
- **Dictionaries**: `ch_dictionary`, `ch_dict_layout`, `ch_dict_source`, `ch_dict_lifetime`, `ch_dict_primary_key`
- **Column metadata**: codec, default expression, LowCardinality/Nullable wrappers via `CHColumnMeta`
- **Skip indexes**: `ChIndexSpec` entries in `ch_indexes`
- **Table and column comments**

For the complete feature reference, see [ClickHouse Deep Dive](../databases/clickhouse/index.md).

## Use cases

- **Bootstrapping**: start a new project from an existing database
- **Documentation**: generate model stubs to document the schema
- **Recovery**: regenerate models when migration scripts are missing

## Warnings

- Generated code requires manual review and cleanup
- ClickHouse engine metadata is auto-detected; review the generated `ChEngineSpec` to ensure correctness

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/history/
========================================================================

# `history`

Show migration execution history.

## Usage

```bash
$ dbwarden history --database primary
```

## Options

- `--database`, `-d`

## Notes

- shows applied migrations, order, and timestamps
- useful for audit and incident analysis

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/init/
========================================================================

# `init`

Initialize DBWarden project scaffolding.

## Usage

```bash
$ dbwarden init
$ dbwarden init --database primary
```

## What it does

- creates `migrations/` and `migrations/<database>/` if missing
- creates/updates config scaffold (`dbwarden.py`) if needed
- does not mutate your database schema

## Notes

- safe to run multiple times
- first command to run in a new project

See also: [Configuration](../configuration/index.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/lock/
========================================================================

# `lock-status` and `unlock`

Inspect and recover migration lock state.

## Usage

```bash
$ dbwarden lock-status --database primary
$ dbwarden unlock --database primary
```

## Options

- `--database`, `-d`

## Notes

- use `lock-status` to inspect lock state
- use `unlock` only when lock is stale and no migration is running

See also: [Migration Locking](../advanced/migration-locking.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/make-migrations/
========================================================================

# `make-migrations`

Generate SQL migration file(s) from SQLAlchemy models.

## Usage

```bash
# Auto-generated name from schema changes
$ dbwarden make-migrations

# User-provided description
$ dbwarden make-migrations "create users table"

# With database option
$ dbwarden make-migrations --database primary --verbose

# Output plan JSON only
$ dbwarden make-migrations --database primary --plan

# Explicitly declare column renames
$ dbwarden make-migrations --rename users.username:email --rename posts.title:headline

# Explicitly declare table renames
$ dbwarden make-migrations --rename-table users:accounts

# Safe multi-step type changes
$ dbwarden make-migrations --database primary --safe-type-change
```

## Options

- `description` (optional): Custom migration name. If not provided, automatically generated from schema changes.
- `--database`, `-d`: Target database.
- `--plan`: Print the migration plan JSON without writing files.
- `--verbose`, `-v`: Verbose output.
- `--rename`: Repeatable. Declare a column rename in the format `table.old_name:new_name`. See [Rename Detection](#rename-detection) below.
- `--rename-table`: Repeatable. Declare a table rename in the format `old_table:new_table`. See [Table Rename Detection](#table-rename-detection) below.
- `--safe-type-change`: Use a multi-step strategy for type changes: add a temporary column, data migration comment, verification step, then drop-and-rename. Useful for databases where `ALTER COLUMN TYPE` would lock the table.
- `--concurrent` / `--no-concurrent`: Enable or disable `CREATE INDEX CONCURRENTLY` for PostgreSQL (default: `--concurrent`). Use `--no-concurrent` when the migration runs inside a transaction block.
- `--offline`: Use a model state file (`.dbwarden/model_state.json`) instead of a live database or schema snapshot. Run `dbwarden export-models` first to establish a baseline. Useful for CI pipelines without a database service.
- `--clickhouse-engine-recreate`: Allow automatic ClickHouse table rebuild when engine changes require recreation. Required to generate `recreate_ch_table` operations. See [ClickHouse Engine Recreate](#clickhouse-engine-recreate) below.
- `--drop-preserved-clickhouse-table` / `--keep-preserved-clickhouse-table`: Control whether the preserved old ClickHouse table is dropped after the engine-recreate swap. If omitted, interactive terminals are prompted; non-TTY preserves by default.
- `--postgres-auto-using`: Emit an active `USING col::newtype` clause on PostgreSQL `ALTER COLUMN TYPE` statements. Default is a commented-out line for manual review. See the PostgreSQL docs section on Column Type Changes for details.
- `--type`, `-t`: Output prefix for the generated migration file: `versioned` (default), `ra` / `runs_always`, or `roc` / `runs_on_change`. Use `ra` for SQL that should run every migration cycle (e.g. grants, materialized view refreshes) and `roc` for SQL that should re-run when the file changes (e.g. stored procedures, triggers).

## Schema Snapshots

After each migration is applied, DBWarden writes a **schema snapshot** to `.dbwarden/schemas/<migration_id>.schema.json`. These snapshots capture the full DDL state (tables, columns, types, indexes, constraints, enums) at that point in time.

`make-migrations` diffs your SQLAlchemy models against the **latest** snapshot instead of the live database. This means:

- You don't need a running database to generate migrations (after the first `migrate`).
- Rename detection works by comparing dropped and added columns between the snapshot and your models.
- If no snapshot exists, `make-migrations` falls back to diffing against the live database.

See [Schema Snapshots](schema-snapshots.md) for details.

## Rollback contract

`make-migrations` enforces rollback correctness for generated migrations. Each emitted rollback statement is classified before the file is accepted:

| Kind | Behavior |
|------|----------|
| `real` | Generated normally. |
| `conditional` | Generated only when prior state exists and the operation is statically proven safe. Verbose mode prints a warning. |
| `irreversible` | Generated only for known irreversible operations or explicit acknowledgement. Normal mode prints a warning. |
| `placeholder` | Refused by default. |

A rollback SQL comment is not treated as a successful rollback. If rollback cannot be generated safely, DBWarden fails generation instead of writing a migration that appears reversible.

Explicit irreversible declaration for committed migrations:

```sql
-- dbwarden: irreversible
```

See [Rollback Coverage](../correctness/rollback-coverage-matrix.md) for PostgreSQL and ClickHouse operation coverage.

## Rename Detection

When a column is dropped from the snapshot and a new column of the same type is added to the model, DBWarden auto-detects it as a potential **rename** and emits `ALTER TABLE ... RENAME COLUMN` instead of `DROP` + `ADD`.

### Auto-detection rules

| Condition | Outcome |
|-----------|---------|
| Same table, 1 dropped + 1 added, same normalized type | Auto-detected as rename |
| Different types | Never auto-detected (emits drop + add) |
| Same name kept | Skipped (no op) |
| 2+ dropped + 2+ added of same type | Paired sequentially (positional) |

### Rename detection edge cases

- **Ambiguous multi-rename**: When 2+ dropped columns and 2+ added columns share the same normalized type, all are treated as renames (paired in insertion order). This is intentionally permissive; false positives can be declined interactively or overridden with `--rename`.
- **Drop-add conversion with `resolved_from`**: A confirmed drop+add pair is converted to a `rename_column` op with `resolved_from` tracking the confirmation source (`"rename_flag"` or `"prompt"`).
- **Non-matching confirmed set**: If a confirmed rename tuple does not match any op (e.g., table name mismatch), it is silently ignored.
- **Table + column rename interaction**: Table renames are processed first (statement order 0). After the snapshot is updated with the new table name, column renames are detected against the renamed table. The column rename's `resolved_from` is independent of the table rename's `resolved_from`.

### Interactive prompt (TTY)

When auto-detected renames are found and you're in an interactive terminal, `make-migrations` prompts you to confirm each one:

**Single rename:**
```
Detected rename: users.username → email. Confirm rename? [Y/n]:
```

**Multiple renames:**
```
Detected column renames:
  [1] users.username → email
  [2] posts.title → headline
  [s] Skip all
  [a] Accept all
Select renames to confirm (e.g. 1,3 or a or s):
```

### CI / non-TTY behavior

When not in an interactive terminal, auto-detected renames are **not applied**. Instead, a warning is printed suggesting the `--rename` flag:

```
The following auto-detected column renames were not confirmed:
  users.username → email (use --rename users.username:email to confirm)
  posts.title → headline (use --rename posts.title:headline to confirm)
These will be emitted as DROP + ADD instead of RENAME.
```

To apply them in CI, pass the corresponding `--rename` flags.

### `--rename` flag

The `--rename` flag explicitly tells `make-migrations` to treat a drop+add pair as a rename. It is required in non-TTY environments (CI) and can also be used to force renames that auto-detection would miss (e.g., type changes).

Format: `--rename <table>.<old_name>:<new_name>`

Examples:

```bash
# Single rename
$ dbwarden make-migrations --rename users.username:email

# Multiple renames
$ dbwarden make-migrations --rename users.username:email --rename posts.title:headline

# Force rename even when types differ
$ dbwarden make-migrations --rename users.phone:mobile_phone
```

### Resolution order

1. **`--rename` flags** are always applied as `RENAME COLUMN` with `resolved_from: "rename_flag"`.
2. **Auto-detected renames** confirmed via interactive prompt get `resolved_from: "prompt"`.
3. **Auto-detected renames** not in `--rename` flags and not confirmed (or in CI) are emitted as `DROP` + `ADD` instead.

## Column-Level Diff

When a schema snapshot exists, `make-migrations` does more than detect new and dropped columns. It also compares columns that exist in both the snapshot and the model for three kinds of change:

| Change | Snapshot vs Model | Generated SQL |
|--------|-------------------|---------------|
| **Type change** | `snapshot.type != model.type` (normalized) | `ALTER COLUMN ... TYPE ...` |
| **Nullability change** | `snapshot.nullable != model.nullable` | `ALTER COLUMN ... SET NOT NULL` / `DROP NOT NULL` |
| **Default change** | `snapshot.default != model.default` | `ALTER COLUMN ... SET DEFAULT` / `DROP DEFAULT` |

### Type change detection

Types are normalized before comparison (see [Schema Snapshots](schema-snapshots.md)). If the normalized type differs between the snapshot and the model, an `alter_column_type` operation is emitted.

Example: model changes `VARCHAR` to `TEXT`:

```sql
-- upgrade

ALTER TABLE users ALTER COLUMN bio TYPE TEXT

-- rollback

ALTER TABLE users ALTER COLUMN bio TYPE VARCHAR
```

### Nullability change

When a model column changes nullable, the corresponding `SET NOT NULL` or `DROP NOT NULL` is generated:

```sql
-- upgrade

ALTER TABLE users ALTER COLUMN email SET NOT NULL

-- rollback

ALTER TABLE users ALTER COLUMN email DROP NOT NULL
```

### Default change

```sql
-- upgrade

ALTER TABLE users ALTER COLUMN role SET DEFAULT 'user'

-- rollback

ALTER TABLE users ALTER COLUMN role DROP DEFAULT
```

### Safe type change (`--safe-type-change`)

For databases that don't support in-place `ALTER COLUMN TYPE` (or when you want to avoid table locks), pass `--safe-type-change` to generate a multi-step strategy:

1. Add a temporary column with the new type
2. Comment indicating a data migration (`UPDATE ... SET temp = CAST(...)`)
3. Verification step comment
4. After manual verification, drop the old column and rename the temporary column

**Limitations:**

| Backend | Supported | Notes |
|---------|-----------|-------|
| PostgreSQL | Yes | Multi-step temp column strategy |
| MySQL / MariaDB | Yes | Multi-step temp column strategy |
| SQLite | No | Comment emitted (SQLite cannot drop columns before 3.35.0 and has limited ALTER TABLE) |
| ClickHouse | No | Comment emitted |

## ClickHouse Engine Recreate

When a ClickHouse table's engine changes, for example `MergeTree` to `ReplicatedMergeTree`, it cannot be altered in-place. DBWarden supports two strategies depending on the table type.

### Table strategy (CREATE + INSERT + RENAME)

For regular `MergeTree`-family tables, DBWarden generates a multi-step operation:

1. Create the new table with the new engine as `<table>__dbw_new`
2. Copy data: `INSERT INTO __dbw_new SELECT ... FROM <table>`
3. Swap: `RENAME TABLE <table> TO <table>__dbw_old, <table>__dbw_new TO <table>`
4. (optional) Drop the preserved old table

**Materialized view targets:** If a materialized view targets the table being recreated (via `TO <table>`), the MV is automatically detached before and reattached after the swap:

```sql
DETACH TABLE events_mv;
CREATE TABLE events__dbw_new (...);
INSERT INTO events__dbw_new SELECT ... FROM events;
RENAME TABLE events TO events__dbw_old, events__dbw_new TO events;
ATTACH TABLE events_mv;
```

**Column renames:** If the table also has column renames, these should be performed in a separate migration (engine recreate and column rename in the same migration is not supported).

### Dictionary strategy (DROP + CREATE)

Dictionaries are recreated with a simple DROP + CREATE since ClickHouse does not support `RENAME DICTIONARY`:

```sql
-- upgrade
DROP DICTIONARY my_dict;
CREATE DICTIONARY my_dict (... ReplicatedMergeTree() ...);

-- rollback
DROP DICTIONARY my_dict;
CREATE DICTIONARY my_dict (... MergeTree() ...);
```

> ⚠️ Dictionaries lose their cached data on recreation. The data will be re-fetched from the source.

### Materialized views and unsupported objects

Engine recreation is **blocked** for tables that are themselves materialized views (`ch_select_statement` or `ch_object_type = materialized_view`). Handle these manually with a DROP/CREATE migration.

Projections (`ch_projections`) are automatically preserved through the table rebuild and do not block it.

### Safety

Rollback for ClickHouse table recreation is deliberately conservative:

| Transition class | Rollback behavior |
|------------------|-------------------|
| Row-preserving engines | Reverse recreate rollback is generated as conditional. |
| Lossy engines | Rollback is irreversible because row-level detail can be lost. |
| Unknown engines | Rollback is irreversible because DBWarden cannot prove safety. |

Lossy engines include `ReplacingMergeTree`, `SummingMergeTree`, `AggregatingMergeTree`, `CollapsingMergeTree`, `VersionedCollapsingMergeTree`, and replicated variants.

| Object type | Safety | Notes |
|------------|--------|-------|
| Regular table with engine change | INFO | Preserved old table by default |
| Table with dependent MVs | INFO | MVs detached before, reattached after |
| Dictionary | CRITICAL | Cached data lost on DROP/CREATE |

### Flags

#### `--clickhouse-engine-recreate`

**Required** to generate engine recreation operations. Without this flag, any detected engine change raises an error:

```
ClickHouse table 'events' cannot be automatically recreated:
is a dictionary (current). This operation requires manual DROP/CREATE,
or use --force to skip this check.
```

#### `--drop-preserved-clickhouse-table` / `--keep-preserved-clickhouse-table`

Controls whether the preserved old table (renamed to `<table>__dbw_old`) is dropped immediately after the swap:

- `--drop-preserved-clickhouse-table`: Drop the old table after successful swap
- `--keep-preserved-clickhouse-table`: Keep the old table (default in non-TTY)

Interactive terminals are prompted to confirm. The preserved table name always ends with `__dbw_old` for easy identification.

### DROP COLUMN warning

All `DROP COLUMN` statements are prefixed with a warning comment:

```sql
-- WARNING: DROPPING COLUMN users.legacy_field

ALTER TABLE users DROP COLUMN legacy_field
```

## Table Rename Detection

When a table is dropped from the snapshot and a new table with similar columns is added to the model, DBWarden auto-detects it as a potential **table rename** using a column-overlap heuristic.

### Auto-detection

| Condition | Outcome |
|-----------|---------|
| A table present in the snapshot but absent from models AND a table absent from the snapshot but present in models | Overlap computed by matching column names and normalized types |
| Overlap ratio ≥ 0.6 | Prompted as a rename candidate |
| Overlap ratio < 0.6 | Emitted as drop+add with a warning comment |

The overlap ratio is `matching_columns / max(len(snapshot_cols), len(model_cols))`. A 0.6 threshold is intentionally conservative: a 10-column table with 6 matching columns is a plausible rename, while a 2-column table with 1 match is not.

### Interactive prompt (TTY)

**Single candidate:**
```
Possible table rename detected:
  users → accounts  (78% columns match)

Treat as rename? [Y/n]:
```

**Multiple candidates:**
```
Possible table renames detected:
  [1] users → accounts     (78% columns match)
  [2] posts → articles     (100% columns match)

Treat as renames? (default: all yes)
  - Press Enter to rename all
  - Type numbers to drop+add instead (e.g. "1" or "1 2"):
```

Table rename prompts appear **before** column rename prompts to ensure table names are resolved before column-level changes.

### CI / non-interactive path

```
Warning: table rename candidates detected but running non-interactive. Emitting drop+add.
  users → accounts  (78% columns match)
Rerun with --rename-table users:accounts to resolve.
```

### `--rename-table` flag

Format: `--rename-table <old_table>:<new_table>`

```bash
# Single table rename
$ dbwarden make-migrations --rename-table users:accounts

# Multiple renames
$ dbwarden make-migrations --rename-table users:accounts --rename-table posts:articles

# Combined with column rename
$ dbwarden make-migrations --rename-table users:accounts --rename accounts.username:email
```

Note: when combining table and column renames, the column rename references the **new** table name. Table renames are applied to the snapshot before column-level processing.

### Table rename edge cases

- **Empty tables**: If either the snapshot table or the model table has zero columns, the overlap ratio is `0.0` and the pair is not a rename candidate.
- **Zero overlap**: If no columns match by name and normalized type, the ratio is `0.0`: emitted as drop+add.
- **Exact match**: If all columns match, the ratio is `1.0`: always a rename candidate.
- **Table rename + column changes in the same table**: After the table rename is applied to the snapshot, column diffs are computed against the new table name. Column renames, type changes, nullable changes, and default changes are all detected on the renamed table.
- **ClickHouse**: `ALTER TABLE RENAME` emits a comment-only placeholder since ClickHouse does not support it.

### SQL generation

All four supported backends (SQLite, PostgreSQL, MySQL, MariaDB) use the same syntax:

```sql
-- upgrade
ALTER TABLE users RENAME TO accounts;

-- rollback
ALTER TABLE accounts RENAME TO users;
```

ClickHouse emits `RENAME TABLE old TO new;` (ClickHouse supports this as a standalone statement).

## Foreign Key and Index Diff

When a schema snapshot exists, `make-migrations` also detects changes to foreign keys and indexes by comparing the snapshot's stored constraints and indexes against the model's declared relationships and indexes.

### Foreign Key vs Index limitations and edge cases

- **Silent skip on missing ref**: If an FK references a table that does not exist in the snapshot, the FK is silently skipped (no error, no SQL). This prevents generating broken SQL but can be surprising. To ensure the FK is emitted, make sure the referenced table exists in the snapshot before running `make-migrations`.
- **Content-based comparison (not name-based)**: Both FKs and indexes are compared by their structural properties, not their names. Renaming a constraint or index does not produce a drop+add.
- **ClickHouse**: FK and index operations emit comment-only placeholders (not supported).
- **SQLite FKs**: Not directly alterable. A comment suggesting table recreation is emitted.

### Foreign Key Detection

| Change | Detection | Generated SQL |
|--------|-----------|---------------|
| **FK added** | FK present in model columns but absent from snapshot constraints | `ALTER TABLE ... ADD CONSTRAINT ... FOREIGN KEY ...` |
| **FK dropped** | FK present in snapshot constraints but absent from model columns | `ALTER TABLE ... DROP CONSTRAINT ...` (or `DROP FOREIGN KEY` on MySQL/MariaDB) |

FKs are compared by content (columns, referenced table, referenced columns), not by name. This means renaming an FK constraint is not treated as a drop+add.

**Validation:** Before emitting an `ADD FOREIGN KEY`, the diff engine verifies that the referenced table and columns exist in the snapshot. If they don't, the FK is silently skipped to avoid generating broken SQL.

**Deferrable constraints (Postgres only):** When detected, `DEFERRABLE INITIALLY DEFERRED` is appended to the constraint SQL.

**SQLite:** FK constraints are not directly alterable. A comment is emitted suggesting table recreation.

### Index Detection

| Change | Detection | Generated SQL |
|--------|-----------|---------------|
| **Index added** | Index present in model but absent from snapshot indexes | `CREATE [UNIQUE] INDEX ... ON table (columns)` |
| **Index dropped** | Index present in snapshot but absent from model indexes | `DROP INDEX ...` |

**Full-content comparison:** Indexes are compared by **all** of these attributes, not just columns + unique. Any difference triggers a drop+add:

| Attribute | SQL Clause | Backend | Example |
|-----------|-----------|---------|---------|
| `using` | `USING <method>` | PostgreSQL, SQLite (partial) | `USING gin`, `USING gist`, `USING hash` |
| `unique` | `UNIQUE` | All | `CREATE UNIQUE INDEX` |
| `where` | `WHERE <predicate>` | PostgreSQL | `WHERE status = 'active'` |
| `include` | `INCLUDE (<cols>)` | PostgreSQL | `INCLUDE (email, name)` |
| `with_params` | `WITH (<params>)` | PostgreSQL | `WITH (fillfactor = 70)` |
| `tablespace` | `TABLESPACE <name>` | PostgreSQL | `TABLESPACE fast_space` |
| `nulls_not_distinct` | `NULLS NOT DISTINCT` | PostgreSQL 15+ | On unique indexes |
| `column_sorting` | Per-column `ASC/DESC NULLS FIRST/LAST` | PostgreSQL | `col1 DESC NULLS LAST, col2 ASC` |
| `type` | `TYPE <type>` | ClickHouse | `TYPE minmax`, `TYPE bloom_filter` |
| `granularity` | `GRANULARITY <n>` | ClickHouse | `GRANULARITY 3` |
| `concurrently` | `CONCURRENTLY` | PostgreSQL | `--concurrent` / `--no-concurrent` |

Omitted attributes or `None`-valued attributes are treated as defaults (btree, no partial, no INCLUDE, etc.), so a plain `Index("ix", "col")` produces the same signature across versions.

**Name generation:** Auto-generated index names follow the pattern:
- `idx_{table}_{col1}_{col2}` for non-unique indexes
- `uq_{table}_{col1}_{col2}` for unique indexes
- Non-btree `USING` methods append a suffix: `idx_{table}_{col}_{method}`

**Backend specifics:**
- PostgreSQL uses `CREATE INDEX CONCURRENTLY` by default (`--concurrent`). Use `--no-concurrent` inside transaction blocks.
- SQLite and MySQL use standard `CREATE INDEX`.
- ClickHouse generates `ALTER TABLE ... ADD INDEX ... TYPE <type> GRANULARITY <n>` for `ChIndexSpec` entries in `ch_indexes`; standard SQL indexes still emit a comment.

## Statement ordering

Operations in the generated migration are ordered consistently:

```
RENAME TABLE        (0)  : table renames first (all subsequent ops use new name)
RENAME COLUMN       (1)
ALTER COLUMN TYPE   (2)
ALTER COLUMN NULLABLE (3)
ALTER COLUMN DEFAULT  (4)
CREATE TABLE        (5)
ADD COLUMN          (6)
ALTER FOREIGN KEY   (7)  : FK adds and drops
ALTER INDEX         (8)  : index adds and drops
DROP COLUMN         (9)
DROP TABLE          (10)
```

Table renames are ordered first so that all subsequent statements reference the new table name.

## Generated artifacts

When a migration is generated, DBWarden writes two files side by side:

- `{database_name}__{version}_{description}.sql`
- `{database_name}__{version}_{description}.plan.json`

The companion plan file contains machine-readable metadata about the generated migration:

- `migration_id`
- `operations`: each operation includes `type`, `table`, `severity` and optionally `resolved_from` (for rename operations)
- `required_flags`
- `checksum`

Example with rename:

```json
{
  "migration_id": "primary__0003_rename_column_users_username",
  "operations": [
    {
      "type": "rename_column",
      "table": "users",
      "new_name": "email",
      "severity": "INFO",
      "resolved_from": "rename_flag"
    },
    {
      "type": "add_column",
      "table": "users",
      "column": "phone",
      "severity": "INFO"
    }
  ],
  "required_flags": [],
  "checksum": "sha256..."
}
```

Possible `resolved_from` values:

| Value | Meaning |
|-------|---------|
| `"rename_flag"` | Explicitly declared via `--rename` or `--rename-table` CLI flag |
| `"prompt"` | Confirmed interactively by the user |
| (absent) | Auto-detected rename kept without prompt (currently unused, reserved) |

`--plan` switches the command into JSON-output mode. In that mode DBWarden prints the plan to stdout and does not write the `.sql` or `.plan.json` files.

## Auto-Generated Names

When no description is provided, DBWarden automatically generates a descriptive name from the schema changes:

| Change | Generated Name |
|--------|----------------|
| Single CREATE TABLE | `create_table_tablename` |
| Multiple CREATE TABLE | `create_tables_users_posts` |
| Single ADD COLUMN | `add_column_tablename_columnname` |
| Multiple ADD COLUMN (same table) | `add_columns_tablename_col1_col2` |
| Single RENAME COLUMN | `rename_column_tablename_new_name` |
| Multiple RENAME COLUMN (same table) | `rename_columns_tablename_col1_col2` |
| Single RENAME TABLE | `rename_table_oldname_newname` |
| Multiple RENAME TABLE | `rename_tables_old1_old2` |
| Single ALTER COLUMN TYPE | `alter_column_type_tablename_col` |
| Single ALTER COLUMN NULLABLE | `alter_column_nullable_tablename_col` |
| Single ALTER COLUMN DEFAULT | `alter_column_default_tablename_col` |
| Single ADD FOREIGN KEY | `add_foreign_key_tablename_ref_table` |
| Single DROP FOREIGN KEY | `drop_foreign_key_tablename` |
| Single ADD INDEX | `add_index_tablename_col` |
| Single DROP INDEX | `drop_index_tablename` |
| Single RECREATE CH TABLE | `recreate_ch_table_tablename` |
| ADD + DROP (same table) | `alter_tablename_col1_col2` |
| Changes across tables | `add_column_users_email_and_1_more_tables` |
| Many targets | `add_columns_tablename_col1_col2_and_3_more` |

### Name Rules

- Snake case throughout.
- Operation words pluralized for multiple targets (e.g., `add_column` → `add_columns`).
- Mixed operations use `alter`.
- Max 72 characters (table/target names truncated as needed).

## Examples

```bash
# Creates primary__0001_create_table_users.sql + .plan.json
$ dbwarden make-migrations --database primary

# Creates primary__0002_add_column_users_email.sql + .plan.json
$ dbwarden make-migrations --database primary

# Creates primary__0003_rename_column_users_email.sql with a confirmed rename
$ dbwarden make-migrations --database primary --rename users.username:email

# Creates primary__0003_alter_column_type_users_bio.sql with type change
$ dbwarden make-migrations --database primary

# Creates primary__0004_add_columns_users_email_name.sql + .plan.json
$ dbwarden make-migrations --database primary

# Creates primary__0003_rename_table_users_accounts.sql with a confirmed table rename
$ dbwarden make-migrations --database primary --rename-table users:accounts

# Uses safe multi-step type change for PostgreSQL
$ dbwarden make-migrations --database primary --safe-type-change

# Uses custom name
$ dbwarden make-migrations "initial_schema" --database primary

# Preview plan JSON without writing files
$ dbwarden make-migrations --database primary --plan
```

## Notes

- Generated file includes both `-- upgrade` and `-- rollback`.
- Generated `.plan.json` files are useful for CI checks and debugging.
- If no models are discovered, configure `model_paths` explicitly.
- With `--dev`, translation can target dev SQLite behavior.
- Schema snapshots are written to `.dbwarden/schemas/` after each successful `migrate`: see [Schema Snapshots](schema-snapshots.md).
- Column-level diff (type/null/default changes) works with a cached schema snapshot, or via a live snapshot taken automatically by `make-migrations` when no cached snapshot exists.
- Without a cached snapshot, `make-migrations` takes a full schema snapshot from the live database internally and detects column-level changes. Only rename detection requires a cached snapshot.
- For authoring guidelines and the review checklist, see [Migration File Format](../migration-files.md).

See also: [Migration File Format](../migration-files.md), [Schema Snapshots](schema-snapshots.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/make-rollback/
========================================================================

# `make-rollback`

Generate a rollback SQL file for a given migration file.

## Usage

```bash
$ dbwarden make-rollback migrations/primary__0005_add_table.sql
```

## Arguments

- `MIGRATION_FILE` (required) - Path to the migration SQL file

## Output

Creates a `.rollback.sql` file next to the given migration file with auto-generated rollback statements.

## Supported reverse transformations

| Upgrade Pattern | Generated Rollback |
|----------------|-------------------|
| `CREATE TABLE t (...)` | `DROP TABLE IF EXISTS t;` |
| `CREATE MATERIALIZED VIEW v AS ...` | `DROP VIEW IF EXISTS v;` |
| `CREATE DICTIONARY d (...)` | `DROP DICTIONARY IF EXISTS d;` |
| `ALTER TABLE t ADD COLUMN c ...` | `ALTER TABLE t DROP COLUMN c;` |
| `CREATE INDEX i ON t (...)` | `DROP INDEX IF EXISTS i;` |
| `CREATE UNIQUE INDEX i ON t (...)` | `DROP INDEX IF EXISTS i;` |
| Other patterns | Refused unless the migration is explicitly irreversible |

## Irreversible annotation

If the command cannot derive executable rollback SQL, it refuses to create a placeholder rollback file. To acknowledge that a migration cannot be rolled back automatically, add this comment to the migration file:

```sql
-- dbwarden: irreversible
```

With that annotation, `make-rollback` may create a rollback file that contains a clear comment instead of executable SQL. This is an intentional declaration, not a successful rollback.

## Notes

- Generated rollback is conservative and may not handle all edge cases.
- Always review the generated rollback before using it.
- For best results, write executable rollback SQL in the `-- rollback` section of the original migration.
- Do not commit placeholder rollback unless the migration is explicitly declared irreversible.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/migrate/
========================================================================

# `migrate`

Apply pending migrations.

## Usage

```bash
$ dbwarden migrate --database primary
$ dbwarden migrate --all
$ dbwarden migrate --database primary --to-version 0010
$ dbwarden migrate --database primary --count 2
$ dbwarden migrate --database primary --with-backup --backup-dir ./backups
$ dbwarden migrate --database primary --baseline --to-version 0005
```

## Options

- `--database`, `-d`
- `--all`, `-a`
- `--count`, `-c`
- `--to-version`, `-t`
- `--baseline`
- `--with-backup`, `-b`
- `--backup-dir`
- `--dry-run`: preview changes without applying
- `--sandbox`: apply in a temporary sandbox database
- `--apply-seeds`: apply pending seeds after migrations
- `--verbose`, `-v`

## Notes

- creates metadata/lock tables if needed
- executes versioned + repeatable migrations
- uses lock protection to prevent concurrent migration mutation

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/new/
========================================================================

# `new`

Create a manual migration file.

## Usage

```bash
$ dbwarden new "manual hotfix" --database primary
$ dbwarden new "backfill users" --database primary --version 0042
$ dbwarden new "seed data" --database primary --type ra
$ dbwarden new "update view" --database primary --type roc
```

## Options

- positional `description`
- `--database`, `-d`
- `--version`
- `--type`, `-t`: Migration type: `versioned` (default), `ra` / `runs_always`, or `roc` / `runs_on_change`

## Notes

- use when change is not model-driven
- file is scaffolded with `-- upgrade` and `-- rollback` sections

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/rollback/
========================================================================

# `rollback`

Rollback applied migrations using `-- rollback` SQL sections.

## Usage

```bash
$ dbwarden rollback --database primary
$ dbwarden rollback --database primary --count 2
$ dbwarden rollback --database primary --to-version 0007
```

## Options

- `--database`, `-d`
- `--count`, `-c`
- `--to-version`, `-t`
- `--verbose`, `-v`

## Notes

- rollback runs in reverse order
- same lock discipline as migrate

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/schema-snapshots/
========================================================================

# Schema Snapshots

Schema snapshots are JSON files that record the full DDL state of a database at the point a migration was applied. They enable offline migration generation, intelligent rename detection, and column-level change detection (type, nullability, default).

## How they work

After each versioned migration is successfully applied by `migrate`, DBWarden extracts the complete schema from the live database and writes it as a `<migration_id>.schema.json` file:

```
.dbwarden/schemas/
  primary__0001_init.schema.json
  primary__0002_add_email.schema.json
  primary__0003_create_posts.schema.json
```

### Snapshot contents

Each snapshot captures:

```json
{
  "format_version": 2,
  "migration_id": "primary__0003_create_posts",
  "database_name": "primary",
  "database_type": "postgresql",
  "applied_at": "2026-06-07T10:30:00Z",
  "checksum": "sha256...",
  "tables": {
    "users": {
      "object_type": "table",
      "columns": {
        "id": {
          "name": "id",
          "type": "integer",
          "nullable": false,
          "primary_key": true,
          "default": null,
          "comment": null,
          "pg_column": {}
        },
        "email": {
          "name": "email",
          "type": "varchar",
          "nullable": false,
          "primary_key": false,
          "default": null,
          "comment": null,
          "pg_column": {}
        }
      },
      "backend_table_spec": {"backend": "postgresql"},
      "comment": null
    }
  },
  "enums": {},
  "indexes": {},
  "constraints": {}
}
```

Fields added in format v2:

| Field | Level | Purpose |
|-------|-------|---------|
| `backend_table_spec` | Per-table | Backend-specific table options (e.g., `ch_engine`, `pg_fillfactor`, `my_engine`) |
| `pg_column` / `ch_column` / `my_column` | Per-column | Backend-specific column metadata (e.g., `ch_type`, `ch_codec`, `pg_type`, `my_unsigned`) |
| `object_type` | Per-table | `"table"`, `"materialized_view"`, or `"dictionary"` |
| `name` | Per-column | Column name (v1 stored names as keys; v2 stores them in both key and value) |

**Backward compatibility:** Snapshots with `format_version: 1` are automatically normalized to v2 when read. V1 keys like `clickhouse_options`, `pg_table`, `my_table`, and per-table `indexes`/`primary_key` are mapped to their v2 equivalents.

### Integrity

Snapshots are self-integrity-checked with a SHA-256 checksum. If a snapshot file is tampered with (manually edited, corrupted), `read_snapshot()` returns `None` and `make-migrations` falls back to the live database.

## Why snapshots?

### Offline diffing

Once the first snapshot exists, `make-migrations` can generate new migrations **without a live database connection**. This is useful in:

- CI pipelines where only model files are available
- Air-gapped environments
- Development setups without a full database

### Rename detection (columns)

Rename detection relies on comparing the snapshot's columns against the model's columns:

1. A column present in the snapshot but absent from the model → **dropped**.
2. A column present in the model but absent from the snapshot → **added**.
3. If a dropped column and an added column share the same normalized type, they are candidates for a **rename**.

Without snapshots (the legacy live-DB fallback path), rename detection is not possible because the live DB already reflects the current state and has no record of dropped columns.

### Rename detection (tables)

Table renames are detected by comparing tables present in the snapshot but absent from models (dropped) against tables absent from the snapshot but present in models (added). A **column overlap heuristic** computes the ratio of matching column names+types between the two tables. If the ratio is ≥ 0.6, the pair is a rename candidate.

Detected table renames are prompted interactively (TTY) or suggested via the `--rename-table` flag (CI). Confirmed renames emit `ALTER TABLE ... RENAME TO` and are applied to the snapshot before column-level processing, so subsequent column renames reference the new table name.

### Column-level change detection

Snapshots also enable `make-migrations` to detect when a column's **type**, **nullability**, or **default** has changed. For columns present in both the snapshot and the model with the same name, the diff engine compares:

- **Normalized type**: If `varchar` in the snapshot but `text` in the model, an `ALTER COLUMN TYPE` operation is emitted.
- **Nullable flag**: If nullable differs, `SET NOT NULL` or `DROP NOT NULL` is generated.
- **Default value**: If the default differs, `SET DEFAULT` or `DROP DEFAULT` is generated.

When a cached snapshot exists, these operations are compared against the historical schema. Without a cached snapshot, `make-migrations` takes a full schema snapshot from the live database internally and detects column-level changes (type, nullability, default) from that live snapshot. Only rename detection requires a cached snapshot.

### Foreign key and index change detection

Snapshots also store the database's foreign keys (in `constraints`) and indexes (in `indexes`). The diff engine compares these against the model's declared FK relationships (parsed from `ModelColumn.foreign_key`) and indexes (extracted from `__table__.indexes`). Comparisons are content-based (columns, referenced table, referenced columns for FKs; columns + unique flag for indexes), so constraint/index name changes are not treated as drop+add.

### Audit trail

Every schema snapshot is an immutable record of the database schema at a specific migration version. You can inspect any historical snapshot to see exactly what the schema looked like.

## How they are created

Snapshots are created automatically by the `migrate` command after applying a versioned migration:

```
$ dbwarden migrate --database primary
```

The snapshot is written **after** all pending migrations have been applied. If the write fails (permission issue, disk full, etc.), a warning is logged but the migration itself is not rolled back. Failure to write a snapshot is non-fatal.

### Snapshots are NOT created for

- `--dry-run` or `--sandbox` runs (no real schema change)
- Rollback operations (snapshot remains as-is for audit)
- Repeatable migrations (`RA__`, `ROC__`)

## Rollback and re-apply

- **Rollback does not delete the snapshot.** The snapshot stays as an audit record of what was applied.
- **Re-applying a migration** overwrites the snapshot with the current schema state.

## Finding the latest snapshot

`make-migrations` uses `find_latest_snapshot()` which scans `.dbwarden/schemas/` for snapshot files matching the current database name and picks the one with the highest version prefix (e.g., `0003` > `0002`).

If no snapshot exists for the database, `make-migrations` takes a full schema snapshot from the live database internally and runs the standard diff pipeline against it. This enables column-level change detection (type, nullability, default). Only rename detection requires a cached snapshot.

## Snapshot lifecycle summary

| Event | Snapshot |
|-------|----------|
| First `migrate` | Created after apply |
| Subsequent `migrate` | Overwritten with latest schema |
| `rollback` | Unchanged (kept as audit) |
| Re-apply same version | Overwritten |
| `--dry-run` / `--sandbox` | Not written |
| `make-migrations` (snapshot exists) | Read for diff + rename detection |
| `make-migrations` (no snapshot) | Fallback to live DB diff |

## DB-agnostic type normalization

Column types in the snapshot are normalized to a canonical set so that equivalent types across databases are treated the same:

| Canonical type | Matches |
|----------------|---------|
| `integer` | INT, INTEGER, INT4, TINYINT, SMALLINT |
| `biginteger` | BIGINT, INT8 |
| `varchar` | VARCHAR, CHARACTER VARYING |
| `text` | TEXT, LONGTEXT, CLOB |
| `boolean` | BOOLEAN, BOOL |
| `timestamp` | TIMESTAMP, DATETIME |
| `numeric` | NUMERIC, DECIMAL (with precision/scale) |
| `float` | FLOAT, REAL, DOUBLE |
| `bytes` | BYTEA, BLOB, BINARY |
| `uuid` | UUID |
| `enum` | ENUM |
| (unknown) | Stored as-is with `"raw": true` |

This normalization is what powers the rename detection: two columns with the same normalized type are candidates for rename, even if their raw SQL type strings differ.

## Edge Cases and Restrictions

### Rename detection
- **Ambiguous renames**: When multiple columns of the same type are dropped and added, all possible pairs are treated as renames (not just one). This maximizes detection but may produce false positives that must be confirmed interactively or via `--rename`.
- **Type change prevents rename**: If a dropped column and an added column have different normalized types, they are never auto-detected as renames. Use `--rename` to force the rename anyway.
- **Same name**: If a column with the same name exists in both the snapshot and the model, no rename is detected even if its type changes (that is handled by type-change detection).

### Table rename detection
- **Column-overlap heuristic**: The ratio is `matching_columns / max(len(snapshot_cols), len(model_cols))`. The 0.6 threshold is intentionally conservative.
- **Empty tables**: Either table having zero columns results in a ratio of `0.0` (no candidate).
- **Table rename + column diff interaction**: Confirmed table renames are applied to the snapshot before column diffs are computed. This ensures column renames and other column-level changes reference the new table name.

### Foreign key and index detection
- **Silent skip on missing ref**: If an FK references a table that does not exist in the snapshot, the FK is silently skipped (no error, no SQL emitted). This prevents broken SQL but can be surprising. Ensure the referenced table exists in the snapshot first.
- **Content-based comparison**: FKs are compared by `(columns, referenced_table, referenced_columns)`. Indexes are compared by `(frozenset(columns), unique)`. Renaming a constraint or index does not produce a drop+add.
- **ClickHouse**: FK and index operations emit comment-only placeholders (not supported by ClickHouse).
- **SQLite FKs**: A comment is emitted suggesting table recreation (not directly alterable).

### Column-level change detection
- **Cached snapshot not required**: Column-level diff works with a live snapshot taken automatically by `make-migrations`. A cached snapshot enables rename detection in addition to column-level diff.
- **Backend limits**: Type changes emit different SQL per backend. SQLite emits comment-only placeholders for type and nullable changes. ClickHouse auto-generates `MODIFY COLUMN` for type, nullable, and LowCardinality changes. Default changes work uniformly across all backends.

### Integrity
- **Tampered snapshots**: If the checksum does not match, `read_snapshot()` returns `None`, and `make-migrations` falls back to the live database. A warning is logged.
- **Checksum-excluded fields**: The `checksum` field itself is excluded from the hash computation, so checksum updates do not cascade.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/seed/
========================================================================

# `seed`

Manage seed data for a database.

## Subcommands

- `seed create`: create a new file seed (legacy)
- `seed apply`: apply pending seeds (file + code seeds)
- `seed list`: list seeds and their status
- `seed rollback`: roll back applied seeds
- `seed export`: export code seeds to ROC SQL files for stateless production application

---

## `seed create`

Create a new file-based seed file (SQL or Python). For new projects, prefer [code seeds](../seeds.md#code-seeds-recommended) instead.

### Usage

```bash
$ dbwarden seed create "seed initial data" --database primary
$ dbwarden seed create "populate lookup tables" --database primary --type python
```

### Options

- `--database`, `-d`: target database handle
- `--type`: `sql` (default) or `python`
- `--verbose`, `-v`

---

## `seed apply`

Apply pending seeds. Both file seeds and [code seeds](../seeds.md#code-seeds-recommended) are discovered and applied.

### Usage

```bash
$ dbwarden seed apply --database primary
$ dbwarden seed apply --database primary --version 0003
$ dbwarden seed apply --database primary --dry-run
$ dbwarden seed apply --all
```

### Options

- `--database`, `-d`
- `--all`, `-a`: apply across all configured databases
- `--version`: apply up to this seed version
- `--dry-run`: preview without executing
- `--verbose`, `-v`

---

## `seed list`

List seeds and their applied status. Includes both file seeds and code seeds.

### Usage

```bash
$ dbwarden seed list --database primary
$ dbwarden seed list --all
$ dbwarden seed list --prune              # clean up orphaned tracking records
```

### Options

- `--database`, `-d`
- `--all`, `-a`
- `--prune`: remove tracking records for seed files that no longer exist on disk
- `--verbose`, `-v`

---

## `seed rollback`

Roll back applied seeds. Removes the tracking record, allowing the seed to be re-applied. Does **not** reverse data changes.

### Usage

```bash
$ dbwarden seed rollback --database primary
$ dbwarden seed rollback --database primary --count 2
$ dbwarden seed rollback --database primary --to-version 0003
```

### Options

- `--database`, `-d`
- `--all`, `-a`: rollback on all databases
- `--count`, `-c`: number of seeds to roll back (default: 1)
- `--to-version`, `-t`: roll back to this seed version
- `--verbose`, `-v`

See also: [Seed Management](../seeds.md)

---

## `seed export`

Export code seeds to ROC (runs-on-change) SQL files for stateless application. The generated file contains `INSERT ... ON CONFLICT` statements rendered in the target database dialect. ROC files are re-applied when their content checksum changes.

### Usage

```bash
$ dbwarden seed export --database primary
$ dbwarden seed export --all
$ dbwarden seed export --database clickhouse --output-dir ./seeds
```

### Options

- `--database`, `-d`: target database handle
- `--all`, `-a`: export seeds for all configured databases
- `--output-dir`, `-o`: output directory (default: `seeds/`)

### Behavior

- **Row-based seeds** (`rows = [...]`): each row is rendered as an `INSERT` statement with `ON CONFLICT` matching the seed's `__seed_on_conflict__`
- **Logic-based seeds** (`generate(session)`): executed in a temporary SQLite database with FK-closure tables created and preceding row-based seeds pre-loaded. The resulting rows are exported as INSERT statements
- Seeds are ordered by FK dependency (topological sort) so foreign-key-safe insert order is preserved

### Dialect requirement

Exporting requires the same dialect packages as connecting to that database. For ClickHouse, install `clickhouse-sqlalchemy`. Missing packages produce a clear error at export time.

### Non-handled problems

- Removed rows are not deleted (no purge on re-export)
- Logic seeds that depend on other logic seeds' output are unsupported
- Non-deterministic `generate()` methods produce a new checksum every export

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/settings/
========================================================================

# `settings`

View DBWarden configuration. All database settings are defined in Python code via
`database_config()`, so `settings show` is a read-only command for inspecting
the current configuration.

## `settings show`

### Usage

```bash
$ dbwarden settings show
$ dbwarden settings show primary
$ dbwarden settings show --all
```

### Options

- `--all`, `-a`: show all configured databases

### Example output

```
Database: PRIMARY (default)
  • Default: True
  • Type: SQLite
  • URL: sqlite:///./app.db
  • Migrations Directory: migrations/primary
  • Migration Table: _dbwarden_migrations
  • Seed Table: _dbwarden_seeds
  • Model Paths: ['app']
  • Dev Database Type: None
  • Dev Database URL: None
  • Overlap Models: False
```

## See also

- [`database list`](./database.md)
- [Configuration docs](../configuration/index.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/snapshot/
========================================================================

# `snapshot`

Output the DDL schema of a specific database table.

## Usage

```bash
$ dbwarden snapshot users --database primary
```

## Options

- `TABLE` (required) - Name of the table to snapshot
- `--database`, `-d`

## Output

For standard SQL databases (SQLite, PostgreSQL, MySQL, MariaDB):

- `CREATE TABLE` statement with column types, nullability, and defaults
- `CREATE INDEX` statements
- Foreign key constraints

For ClickHouse:

- The raw `CREATE TABLE` query from `system.tables`

## Notes

- output is printed to stdout
- useful for debugging schema differences or documenting table structure
- internally uses `sqlalchemy.inspect()` for generic databases

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/status/
========================================================================

# `status`

Show migration status (applied vs pending).

## Usage

```bash
$ dbwarden status --database primary
$ dbwarden status --all
```

## Options

- `--database`, `-d`
- `--all`, `-a`

## Notes

- run before and after migration execution
- supports multi-database status with `--all`

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/commands/version/
========================================================================

# `version`

Show DBWarden version.

## Usage

```bash
$ dbwarden version
```

## Notes

- useful for support/debug and release verification

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/concepts/
========================================================================

# Configuration Concepts

Understand how DBWarden configuration works under the hood.

## What is Configuration?

Configuration tells DBWarden:
- **Where** your databases are (connection URLs)
- **What** kind of databases they are (PostgreSQL, SQLite, etc.)
- **Where** your SQLAlchemy models live (for migration generation)
- **Where** to store migrations (directories)

## Why Python Configuration?

### Type Safety

Your IDE can help you:

```python
primary = database_config(
    database_name="primary",  #  IDE suggests parameter names
    default=True,             #  IDE knows this is boolean
    database_type="sqlite",   #  IDE can validate enum values
    database_url_sync="...",
)
```

### Dynamic Configuration

You can use Python logic:

```python
import os

# Different config per environment
environment = os.getenv("ENV", "dev")

if environment == "production":
    database_url = "postgresql://prod-host/myapp"
else:
    database_url = "sqlite:///./dev.db"

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql" if environment == "production" else "sqlite",
    database_url_sync=database_url,
)
```

### Multiple Databases

Easy to configure multiple databases:

```python
DATABASES = {
    "primary": "postgresql://localhost/main",
    "analytics": "postgresql://localhost/analytics",
    "logging": "postgresql://localhost/logs",
}

for name, url in DATABASES.items():
    db = database_config(
        database_name=name,
        default=(name == "primary"),
        database_type="postgresql",
        database_url_sync=url,
        model_paths=[f"app.models.{name}"],
    )
```

## Configuration Loading

### Discovery Order

DBWarden searches for configuration in this order:

```
1. dbwarden.py in current directory
      not found
2. dbwarden.py in parent directories
      not found
3. Full scan for files with database_config()
      not found
4. DBWARDEN_CONFIG_MODULE environment variable
      not found
Error: No configuration found
```

### When Configuration Loads

Configuration loads when you run **any** DBWarden command:

```bash
$ dbwarden migrate    #  Config loads here
$ dbwarden status     #  Config loads here
$ dbwarden history    #  Config loads here
```

**Load process:**
1. Python imports your config module
2. `database_config()` calls execute
3. Databases register in internal registry
4. Validation runs
5. Command executes with loaded config

### Validation Rules

DBWarden validates configuration at load time:

| Rule | Why It Matters |
|------|----------------|
| Exactly one `default=True` | CLI needs to know which DB to use when `--database` is omitted |
| Unique `database_name` | Commands target databases by name |
| Unique `database_url` | Prevents accidental duplicate configurations |
| Unique physical targets | Prevents two configs pointing to same DB with different credentials |
| Required `model_paths` in multi-DB | Keeps model discovery boundaries clear |
| No overlapping `model_paths` | Prevents ambiguous model ownership (unless `overlap_models=True`) |

### Validation Timing

```python
# dbwarden.py
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
)

primary = database_config(
    database_name="primary",  #  Duplicate!
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/other",
)
```

When you run `dbwarden migrate`:

```
Error: Duplicate database_name 'primary'
```

Validation happens **before** any commands execute.

### Config Source Precedence

When looking for config, DBWarden uses this precedence:

1. **Top-level `dbwarden.py`**: the conventional standalone config file at your project root. This is the default scaffold created by `dbwarden init`, not the only valid location. Always sandboxed (only `dbwarden` imports allowed).

2. **`DBWARDEN_CONFIG_MODULE`**: an explicit environment variable override. Always imported normally as a Python module (no sandbox). This is the escape hatch for projects with ambiguous full-scan results or non-standard layouts.

3. **Full-scan discovery**: if neither of the above produces a config source, DBWarden walks your project tree looking for any `database_config(...)` call. This means `database_config(...)` can live in any discovered Python file inside your project. Files directly at the project root are sandboxed; files inside subdirectories are imported normally.

### Config Loading Security (Sandbox)

DBWarden applies import restrictions only to config files that are **isolated** (sandboxed) vs **in-package** (normal import):

| Mode | Import behavior | Applies to |
|------|----------------|------------|
| `isolated` | Sandboxed: only `dbwarden.*` imports allowed | Top-level `dbwarden.py`; any full-scan-discovered file at the project root |
| `in-package` | Normal Python import | Full-scan-discovered files inside subdirectories; `DBWARDEN_CONFIG_MODULE` modules |

An isolated config file runs in a sandbox that prevents accidental escalation of file-read access to arbitrary code execution. Only `dbwarden` and its submodules can be imported.

An in-package config file is imported as a normal Python module, with full access to `app.*` and any other project imports. This is the correct path when your `database_config(...)` call lives in an application package that imports other project modules.

**Import root detection.** For full-scan-discovered files, DBWarden tries to resolve the dotted module path. It checks two common import roots in order:

- `src/` (PEP 517/518, setuptools, poetry)
- The project root itself

For example, `src/myapp/databases.py` resolves as `myapp.databases` with import root `src/`. If neither root produces an importable path, the file falls back to `isolated` (sandboxed). Projects with other layouts should set `DBWARDEN_CONFIG_MODULE` explicitly.

**Path validation** (path traversal blocking) applies to all file-based sources regardless of mode.

For debugging, set `DBWARDEN_DISABLE_SANDBOX=1` to disable the sandbox for isolated files:

```bash
DBWARDEN_DISABLE_SANDBOX=1 dbwarden status  # Skip sandbox (debug only)
```

Disabling the sandbox also removes import restrictions for isolated config files, which can be useful in development.  Keep it enabled in production.

## The `default` Database

### Why `default=True` Exists

Consider these commands:

```bash
# Explicit database
$ dbwarden migrate --database primary

# Implicit database (uses default)
$ dbwarden migrate
```

Without `default=True`, DBWarden wouldn't know which database to use for the second command.

### Only One Default

```python
#  Good
analytics = database_config(
analytics = database_config(database_name="analytics", default=False, ...)  # or omit default

#  Bad - two defaults
analytics = database_config(
analytics = database_config(database_name="analytics", default=True, ...)  # Error!
```

### Default Affects CLI Behavior

```bash
# These are equivalent when primary is default:
$ dbwarden migrate
$ dbwarden migrate --database primary

# These are NOT equivalent:
$ dbwarden migrate
$ dbwarden migrate --database analytics  # Targets analytics, not primary
```

## Model Discovery

### What Are `model_paths`?

`model_paths` tells DBWarden where your SQLAlchemy models live:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    model_paths=["app.models"],  #  Look here for models
)
```

### How Discovery Works

```
1. Import each module in model_paths
     
2. Find all classes inheriting from DeclarativeBase
     
3. Extract table metadata (__tablename__, columns, etc.)
     
4. Build internal representation for migration generation
```

### Filtering by Table Name

When two databases share the same `model_paths` but should own different
subsets of tables, use `model_tables`:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://clickhouse-host:8123/analytics",
    model_paths=["app.models"],
    model_tables=["analytics_events", "analytics_sessions"],
)
```

This is useful when all models live under one shared package but each
database only owns a subset.  DBWarden validates every name in
`model_tables` exists among the discovered tables and prevents overlap
between databases (unless `overlap_models=True`).

### When Is It Required?

**Single database:** Optional (DBWarden scans entire codebase)

```python
# This works
primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    # No model_paths needed
)
```

**Multiple databases:** Required for each database

```python
# This is required
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],  #  Required
)

analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    model_paths=["app.models.analytics"],  #  Required
)
```

**Why?** To prevent ambiguity about which models belong to which database.

## Dev Mode

### What Is Dev Mode?

Dev mode lets you use a different database for local development:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",              # Production
    database_url_sync="postgresql://prod/myapp",
    dev_database_type="sqlite",              # Development
    dev_database_url="sqlite:///./dev.db",
)
```

Run commands with `--dev`:

```bash
$ dbwarden --dev migrate  # Uses SQLite
$ dbwarden migrate        # Uses PostgreSQL
```

### How It Works

```
Command: dbwarden --dev migrate
    
Check for --dev flag
    
Swap database_type → dev_database_type
Swap database_url → dev_database_url
    
Connect to dev database
    
Execute command
```

### Why Use It?

**Speed:**
- SQLite is faster than PostgreSQL for local iteration
- No network latency
- No server setup

**Safety:**
- Can't accidentally affect production
- Each developer has their own isolated database
- Easy to reset (just delete the file)

**Simplicity:**
- No Docker containers needed
- No database server installation
- Works on all platforms

## Multi-Database Configuration

### Why Multiple Databases?

Common scenarios:
- **Separation of concerns** - Transactions vs analytics
- **Performance** - Offload reporting to separate database
- **Compliance** - Audit logs in separate database
- **Legacy systems** - New and old databases coexist

### How It Works

Each `database_config()` call registers an independent database:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],
)

analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    model_paths=["app.models.analytics"],
)
```

They're completely independent:
- Separate migration histories
- Separate migration directories
- Separate model sets
- Can use different database types

### Model Path Boundaries

```python
app/
  models/
    primary/
      user.py         #  Goes to primary database
      order.py
    analytics/
      event.py        #  Goes to analytics database
      metric.py
```

Configuration:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],  #  Only primary models
    ...
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],  #  Only analytics models
    ...
)
```

## Secure Values

### What Is `secure_values`?

Prevents credentials from appearing in terminal output:

```python
import os

DATABASE_URL = os.getenv("DATABASE_URL")

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=DATABASE_URL,
    secure_values=True,  #  Hide credentials
)
```

### Without `secure_values`:

```bash
$ dbwarden settings show
Database: primary
  URL: postgresql://user:SECRET_PASSWORD@prod-host/myapp
```

### With `secure_values`:

```bash
$ dbwarden settings show --all
Database: primary
  URL: DATABASE_URL (expression)
```

Shows the variable name instead of resolved value.

## Configuration vs Runtime

### Configuration Time

When config loads:
- `database_config()` calls execute
- Validation runs
- Internal registry populates
- **No database connections made**

### Runtime

When commands run:
- DBWarden reads from registry
- **Connects to database**
- Executes command logic

**Key point:** Configuration errors are caught early, before any database operations.

## What's Next?

- **[Connection URLs](connection-urls.md)** - URL format reference
- **[Model Discovery](model-discovery.md)** - Deep dive into model paths
- **[Dev Mode](dev-mode.md)** - Local development workflows
- **[Multi-Database](multi-database.md)** - Multi-database patterns

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/connection-urls/
========================================================================

# Connection URLs

Complete reference for database connection URL formats.

## URL Format

All database URLs follow this general structure:

```
[dialect[+driver]]://[username[:password]@][host][:port][/database][?option=value&...]
```

## PostgreSQL

### Basic Format

```
postgresql://[user[:password]@][host][:port]/database[?options]
```

### Examples

**Local default:**
```python
database_url_sync="postgresql://localhost/myapp"
```

**With credentials:**
```python
database_url_sync="postgresql://user:password@localhost:5432/myapp"
```

**Remote host:**
```python
database_url_sync="postgresql://user:password@db.example.com:5432/myapp"
```

**With SSL:**
```python
database_url_sync="postgresql://user:password@localhost/myapp?sslmode=require"
```

**With connection pool:**
```python
database_url_sync="postgresql://user:password@localhost/myapp?pool_size=20&max_overflow=10"
```

### SSL Modes

| Mode | Description |
|------|-------------|
| `disable` | No SSL |
| `allow` | Try SSL, fall back to non-SSL |
| `prefer` | Try SSL first (default) |
| `require` | Require SSL, fail if unavailable |
| `verify-ca` | Require SSL + verify CA |
| `verify-full` | Require SSL + verify CA + hostname |

**Example:**
```python
database_url_sync="postgresql://user:pass@host/db?sslmode=verify-full&sslrootcert=/path/to/ca.pem"
```

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `sslmode` | SSL connection mode | `sslmode=require` |
| `sslcert` | Client certificate | `sslcert=/path/to/cert.pem` |
| `sslkey` | Client key | `sslkey=/path/to/key.pem` |
| `sslrootcert` | CA certificate | `sslrootcert=/path/to/ca.pem` |
| `connect_timeout` | Connection timeout (seconds) | `connect_timeout=10` |
| `application_name` | App name in pg_stat_activity | `application_name=myapp` |

### Cloud Providers

**AWS RDS:**
```python
database_url_sync="postgresql://user:pass@mydb.abc123.us-east-1.rds.amazonaws.com:5432/myapp?sslmode=require"
```

**Google Cloud SQL:**
```python
database_url_sync="postgresql://user:pass@/myapp?host=/cloudsql/project:region:instance"
```

**Azure Database:**
```python
database_url_sync="postgresql://user@server:pass@server.postgres.database.azure.com:5432/myapp?sslmode=require"
```

**Heroku:**
```python
import os
database_url_sync=os.getenv("DATABASE_URL")  # Provided by Heroku
```

## SQLite

### Basic Format

```
sqlite:///[path]
```

### Examples

**Relative path:**
```python
database_url_sync="sqlite:///./app.db"
database_url_sync="sqlite:///./data/app.db"
```

**Absolute path:**
```python
database_url_sync="sqlite:////absolute/path/to/app.db"
```

**In-memory (testing only):**
```python
database_url_sync="sqlite:///:memory:"
```

In-memory databases are lost when the connection closes. Only use for testing.

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `timeout` | Lock timeout (seconds) | `?timeout=20` |
| `check_same_thread` | Thread safety check | `?check_same_thread=false` |

**Example:**
```python
database_url_sync="sqlite:///./app.db?timeout=20"
```

## MySQL / MariaDB

### Basic Format

```
mysql://[user[:password]@][host][:port]/database[?options]
```

### Examples

**Local:**
```python
database_url_sync="mysql://root:password@localhost:3306/myapp"
```

**With charset:**
```python
database_url_sync="mysql://user:pass@localhost/myapp?charset=utf8mb4"
```

**With SSL:**
```python
database_url_sync="mysql://user:pass@localhost/myapp?ssl_ca=/path/to/ca.pem"
```

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `charset` | Character set | `charset=utf8mb4` |
| `ssl_ca` | CA certificate | `ssl_ca=/path/to/ca.pem` |
| `ssl_cert` | Client certificate | `ssl_cert=/path/to/cert.pem` |
| `ssl_key` | Client key | `ssl_key=/path/to/key.pem` |

### MariaDB

MariaDB uses the same URL format as MySQL:

```python
database_url_sync="mysql://user:pass@localhost:3306/myapp"
```

Configure with `database_type="mariadb"`:

```python
primary = database_config(
    database_name="primary",
    database_type="mariadb",
    database_url_sync="mysql://localhost/myapp",
)
```

## ClickHouse

### Basic Format

```
http://[user[:password]@]host[:port]/database[?options]
```

### Examples

**Local:**
```python
database_url_sync="http://default:@localhost:8123/myapp"
```

**With authentication:**
```python
database_url_sync="http://user:password@localhost:8123/myapp"
```

**With HTTPS:**
```python
database_url_sync="https://user:password@clickhouse.example.com:8443/myapp"
```

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `compression` | Enable compression | `compression=1` |
| `connect_timeout` | Connection timeout | `connect_timeout=10` |
| `send_timeout` | Send timeout | `send_timeout=300` |
| `receive_timeout` | Receive timeout | `receive_timeout=300` |

**Example:**
```python
database_url_sync="http://user:pass@localhost:8123/myapp?compression=1&connect_timeout=10"
```

## Environment Variables

### Basic Pattern

```python
import os

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
)
```

### With Fallback

```python
import os

database_url = os.getenv("DATABASE_URL", "sqlite:///./dev.db")

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql" if "postgresql" in database_url else "sqlite",
    database_url_sync=database_url,
)
```

### Required Environment Variables

```python
import os

DATABASE_URL = os.getenv("DATABASE_URL")
if not DATABASE_URL:
    raise ValueError("DATABASE_URL environment variable is required")

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=DATABASE_URL,
)
```

### Multiple Databases

```python
import os

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("PRIMARY_DATABASE_URL"),
)

analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync=os.getenv("ANALYTICS_DATABASE_URL"),
)
```

## URL Encoding

### Special Characters

If your password contains special characters, URL-encode them:

| Character | Encoded |
|-----------|---------|
| `@` | `%40` |
| `:` | `%3A` |
| `/` | `%2F` |
| `?` | `%3F` |
| `#` | `%23` |
| `&` | `%26` |
| `%` | `%25` |

**Example:**

Password: `p@ss:word`

```python
database_url_sync="postgresql://user:p%40ss%3Aword@localhost/myapp"
```

### Python URL Encoding

```python
from urllib.parse import quote_plus

username = "user"
password = "p@ss:word"
host = "localhost"
database = "myapp"

database_url = f"postgresql://{username}:{quote_plus(password)}@{host}/{database}"
# Result: postgresql://user:p%40ss%3Aword@localhost/myapp
```

## Connection Pools

### PostgreSQL Pool Options

```python
database_url_sync="postgresql://user:pass@localhost/myapp?pool_size=20&max_overflow=10&pool_timeout=30"
```

| Option | Description | Default |
|--------|-------------|---------|
| `pool_size` | Max connections in pool | 5 |
| `max_overflow` | Extra connections if pool full | 10 |
| `pool_timeout` | Wait time for connection (seconds) | 30 |
| `pool_recycle` | Recycle connections after (seconds) | -1 (never) |

### Connection Lifetime

Recycle connections after 1 hour:

```python
database_url_sync="postgresql://user:pass@localhost/myapp?pool_recycle=3600"
```

## Testing Connections

### Verify URL Format

```python
from sqlalchemy import create_engine

try:
    engine = create_engine("postgresql://user:pass@localhost/myapp")
    with engine.connect() as conn:
        result = conn.execute("SELECT 1")
        print("Connection successful!")
except Exception as e:
    print(f"Connection failed: {e}")
```

### Test with DBWarden

```bash
# Check configuration
$ dbwarden settings show

# Test connection
$ dbwarden check-db
```

## Common Mistakes

### Forgetting Port

**Wrong:**
```python
database_url_sync="postgresql://user:pass@localhost/myapp"  # Uses default port 5432
```

**If you need a different port:**
```python
database_url_sync="postgresql://user:pass@localhost:5433/myapp"
```

### Missing Slashes

**Wrong:**
```python
database_url_sync="sqlite://./app.db"  # Only 2 slashes
```

**Correct:**
```python
database_url_sync="sqlite:///./app.db"  # 3 slashes for relative path
database_url_sync="sqlite:////absolute/path/app.db"  # 4 slashes for absolute path
```

### Special Characters Not Encoded

**Wrong:**
```python
database_url_sync="postgresql://user:p@ss@localhost/myapp"  # @ not encoded
```

**Correct:**
```python
database_url_sync="postgresql://user:p%40ss@localhost/myapp"  # @ encoded as %40
```

## What's Next?

- **[Model Discovery](model-discovery.md)** - Configure model paths
- **[Dev Mode](dev-mode.md)** - Local development URLs
- **[Production Patterns](production-patterns.md)** - Real-world examples
- **[Troubleshooting](troubleshooting.md)** - Connection issues

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/credentials/
========================================================================

# Credentials and Secrets

Never hardcode database credentials in `dbwarden.py`. This page covers how to inject secrets safely.

## The problem

The quick-start examples show inline connection strings:

```python
# Do not ship this
primary = database_config(
    database_url_sync="postgresql://admin:s3cr3t@localhost:5432/myapp",
    ...
)
```

`dbwarden.py` is Python source. It ends up in version control. Credentials in source are a liability.

## Environment variables

The standard pattern: read from the environment at config load time.

```python
import os
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    model_paths=["app.models"],
    secure_values=True,
)
```

`secure_values=True` tells DBWarden to redact the URL in CLI output and logs.

### Fail fast on missing env var

```python
import os
from dbwarden import database_config

DATABASE_URL = os.getenv("DATABASE_URL")
if not DATABASE_URL:
    raise ValueError("DATABASE_URL is required")

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=DATABASE_URL,
    model_paths=["app.models"],
    secure_values=True,
)
```

Without the guard, a missing env var silently passes `None` to `database_url_sync`, which fails later with a confusing error.

## `.env` files with python-dotenv

For local development, use a `.env` file to avoid setting vars manually each session.

Install:

```bash
uv add python-dotenv
```

Create `.env` in your project root:

```
DATABASE_URL=postgresql://dev_user:dev_pass@localhost:5432/myapp_dev
```

Load it at the top of `dbwarden.py`:

```python
import os
from dotenv import load_dotenv
from dbwarden import database_config

load_dotenv()

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    model_paths=["app.models"],
    secure_values=True,
)
```

Add `.env` to `.gitignore`. Commit a `.env.example` with placeholder values:

```
DATABASE_URL=postgresql://user:password@localhost:5432/myapp
```

## Multi-database with separate secrets

Each database gets its own env var:

```python
import os
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("PRIMARY_DATABASE_URL"),
    model_paths=["app.models"],
    secure_values=True,
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync=os.getenv("ANALYTICS_DATABASE_URL"),
    model_paths=["app.models.analytics"],
    secure_values=True,
)
```

## Dev mode with secrets

Dev mode can also use env vars, keeping SQLite paths out of source:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    dev_database_type="sqlite",
    dev_database_url=os.getenv("DEV_DATABASE_URL", "sqlite:///./dev.db"),
    model_paths=["app.models"],
    secure_values=True,
)
```

`DEV_DATABASE_URL` defaults to a local SQLite path if not set, which is reasonable for development.

## CI/CD environments

In GitHub Actions, set secrets in the repository settings and reference them in the workflow:

```yaml
- name: Run migrations
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
  run: dbwarden migrate --database primary
```

In GitLab CI, use masked CI/CD variables:

```yaml
migrate:
  script:
    - dbwarden migrate --database primary
  variables:
    DATABASE_URL: $DATABASE_URL  # set in GitLab CI/CD settings
```

## Third-party secret managers

For production systems using Vault, AWS Secrets Manager, or Infisical, fetch the secret before passing it to `database_config()`:

```python
import os
import boto3
import json
from dbwarden import database_config

def get_secret(secret_name: str) -> str:
    client = boto3.client("secretsmanager")
    response = client.get_secret_value(SecretId=secret_name)
    secret = json.loads(response["SecretString"])
    return secret["database_url"]

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=get_secret("myapp/production/database"),
    model_paths=["app.models"],
    secure_values=True,
)
```

`dbwarden.py` is plain Python, so any secret retrieval logic is valid here.

## `secure_values=True`

When set, DBWarden redacts the connection URL in:

- `dbwarden settings show` output
- `dbwarden settings show` output
- Log lines

The URL is still used internally for connections. This prevents accidental credential exposure in terminal output shared in screenshots or logs.

See also: [Production Patterns](production-patterns.md) | [CI/CD Patterns](../advanced/ci-cd-patterns.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/dev-mode/
========================================================================

# Dev Mode

Use SQLite for local development and PostgreSQL in production with the same codebase.

## What Is Dev Mode?

Dev mode lets you configure **two database URLs**:
- **Production URL** - Used by default
- **Dev URL** - Used when you pass `--dev` flag

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",    # Production
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",           # Development
)
```

Run commands with `--dev`:

```bash
$ dbwarden --dev migrate        # Uses SQLite
$ dbwarden --dev status         # Uses SQLite
$ dbwarden migrate              # Uses PostgreSQL
```

## Why Use Dev Mode?

### Speed

SQLite is **much faster** for local development:
- No network latency
- No authentication overhead
- File-based, not server-based
- Instant startup

### Simplicity

No PostgreSQL server required:
- No Docker setup
- No installation
- No configuration
- Works on all platforms

### Safety

Can't accidentally affect production:
- Dev database is a local file
- Each developer has their own database
- Easy to reset (`rm dev.db`)
- No shared state

### Portability

Easy to share between developers:
- One config file works everywhere
- No server setup instructions
- Fresh developers can start immediately

## Basic Setup

### Step 1: Configure Dev Database

```python
# dbwarden.py
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    dev_database_type="sqlite",                    #  Add this
    dev_database_url="sqlite:///./dev.db",         #  Add this
    model_paths=["app.models"],
)
```

### Step 2: Use Dev Mode

```bash
# Development workflow
$ dbwarden --dev make-migrations "create users"
$ dbwarden --dev migrate
$ dbwarden --dev status

# Production workflow
$ dbwarden make-migrations "create users"
$ dbwarden migrate
$ dbwarden status
```

## Dev Mode Workflow

### Daily Development

```bash
# Morning: pull latest code
git pull

# Run migrations against dev database
$ dbwarden --dev migrate

# Work on features...
# Add new models

# Generate migration
$ dbwarden --dev make-migrations "add orders table"

# Test migration
$ dbwarden --dev migrate

# Verify
$ dbwarden --dev status

# Commit
git add migrations/primary/0002_add_orders_table.sql
git commit -m "Add orders table"
```

### Testing Rollbacks

```bash
# Apply migration
$ dbwarden --dev migrate

# Test rollback
$ dbwarden --dev rollback

# Re-apply
$ dbwarden --dev migrate
```

### Fresh Start

Reset your dev database anytime:

```bash
# Delete dev database
rm dev.db

# Re-run migrations
$ dbwarden --dev migrate
```

## Production Workflow

Dev mode only affects **local development**. Production uses the main URL:

### CI/CD Pipeline

```yaml
# .github/workflows/deploy.yml
- name: Run migrations
  run: dbwarden migrate  # No --dev flag
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
```

### Production Deployment

```bash
# Staging
$ dbwarden migrate --database primary

# Production
$ dbwarden migrate --database primary
```

Dev mode is **never used** in CI/CD or production.

## SQLite Limitations

### What Works

Most features work in SQLite:
-  Tables, columns, indexes
-  Primary keys, foreign keys
-  Unique constraints
-  Basic data types
-  Transactions

### What Doesn't Work

Some PostgreSQL features aren't available in SQLite:
-  Advanced types (JSONB, arrays, enums)
-  Partial indexes
-  Generated columns (in older SQLite)
-  Multiple schemas
-  Concurrent writes

### Translation

DBWarden **doesn't translate** SQL between databases. Your migrations should work on both SQLite and PostgreSQL.

**Approach 1:** Write portable SQL

```sql
--  Works on both
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    email VARCHAR(255) NOT NULL
);

--  PostgreSQL-specific
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    metadata JSONB
);
```

**Approach 2:** Use PostgreSQL for dev too

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    dev_database_type="postgresql",                         # Same as prod
    dev_database_url="postgresql://localhost/myapp_dev",    # Different database
)
```

## Environment-Based Configuration

### Automatic Dev Mode

Use environment variables to automatically detect dev:

```python
import os

is_dev = os.getenv("ENV", "dev") in ["dev", "development", "local"]

if is_dev:
    database_url = "sqlite:///./dev.db"
    database_type = "sqlite"
else:
    database_url = os.getenv("DATABASE_URL")
    database_type = "postgresql"

primary = database_config(
    database_name="primary",
    default=True,
    database_type=database_type,
    database_url_sync=database_url,
)
```

Run commands:

```bash
# Dev
ENV=dev dbwarden migrate

# Production
ENV=production dbwarden migrate
```

## Multiple Dev Databases

If you have multiple databases, configure dev mode for each:

```python
# Primary database
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_primary.db",
    model_paths=["app.models.primary"],
)

# Analytics database
analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_analytics.db",
    model_paths=["app.models.analytics"],
)
```

Run against all dev databases:

```bash
$ dbwarden --dev migrate --all
$ dbwarden --dev status --all
```

## Common Patterns

### Pattern 1: SQLite for Dev, PostgreSQL for Prod (Recommended)

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",
)
```

**Pros:**
- Fast local iteration
- No server setup
- Easy to reset

**Cons:**
- SQL must be portable
- Some features unavailable in dev

### Pattern 2: PostgreSQL for Both

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://prod-host/myapp",
    dev_database_type="postgresql",
    dev_database_url="postgresql://localhost/myapp_dev",
)
```

**Pros:**
- Identical environments
- Use all PostgreSQL features
- Catches more bugs in dev

**Cons:**
- Requires PostgreSQL server locally
- Slower than SQLite

### Pattern 3: Dynamic Based on Environment

```python
import os

environment = os.getenv("ENV", "dev")

if environment == "production":
    database_url = os.getenv("DATABASE_URL")
    database_type = "postgresql"
elif environment == "staging":
    database_url = os.getenv("STAGING_DATABASE_URL")
    database_type = "postgresql"
else:
    database_url = "sqlite:///./dev.db"
    database_type = "sqlite"

primary = database_config(
    database_name="primary",
    default=True,
    database_type=database_type,
    database_url_sync=database_url,
)
```

## Testing with Dev Mode

### Unit Tests

Use SQLite for fast unit tests:

```python
# tests/conftest.py
import pytest
from sqlalchemy import create_engine
from app.models import Base

@pytest.fixture(scope="function")
def db():
    engine = create_engine("sqlite:///:memory:")
    Base.metadata.create_all(engine)
    yield engine
    Base.metadata.drop_all(engine)
```

### Integration Tests

Use `--dev` for integration tests:

```bash
# Run integration tests
$ dbwarden --dev migrate
pytest tests/integration/
```

## Troubleshooting

### "SQL syntax error in SQLite"

**Cause:** Using PostgreSQL-specific SQL.

**Solution:** Make SQL portable or use PostgreSQL for dev:

```python
dev_database_type="postgresql"
dev_database_url="postgresql://localhost/myapp_dev"
```

### "dev_database_url is required"

**Cause:** Set `dev_database_type` without `dev_database_url`.

**Solution:** Add both:

```python
dev_database_type="sqlite"
dev_database_url="sqlite:///./dev.db"  #  Add this
```

### Dev database not updating

**Cause:** Forgot `--dev` flag.

**Solution:** Use `--dev`:

```bash
$ dbwarden --dev migrate  #  Add --dev
```

## What's Next?

- **[Multi-Database](multi-database.md)** - Multiple databases with dev mode
- **[Production Patterns](production-patterns.md)** - Deploy to production
- **[Troubleshooting](troubleshooting.md)** - Common issues

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/
========================================================================

# Configuration

DBWarden uses Python-based configuration with `database_config()` to define your databases.

**One configuration source** for migrations, CLI tools, and runtime: no split configs.

## Quick Start

The simplest configuration possible:

```python
# dbwarden.py
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
)
```

That's it! **4 parameters** to get started.

Run your first migration:

```bash
$ dbwarden init
$ dbwarden make-migrations "initial schema"
$ dbwarden migrate
```

## Learning Path

### New to DBWarden?
Start here to understand configuration basics:

1. **[Quick Start](quick-start.md)** - Your first configuration in 2 minutes
2. **[Concepts](concepts.md)** - How configuration works
3. **[Connection URLs](connection-urls.md)** - Database connection formats

### Building Your Configuration
Learn specific features:

- **[Model Discovery](model-discovery.md)** - How DBWarden finds your SQLAlchemy models
- **[Dev Mode](dev-mode.md)** - Local development with SQLite
- **[Multi-Database](multi-database.md)** - Configure multiple databases

### Production Ready
Deploy with confidence:

- **[Production Patterns](production-patterns.md)** - Real-world examples
- **[Troubleshooting](troubleshooting.md)** - Common issues and solutions

### Complete Reference
- **[Configuration API](../reference/configuration-api.md)** - Complete function signature and parameters

## Key Features

###  Simple Configuration

Define once, use everywhere:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    model_paths=["app.models"],
)
```

###  Dev Mode

Use SQLite locally, PostgreSQL in production:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",
)
```

Run commands with `--dev`:

```bash
$ dbwarden --dev migrate
$ dbwarden --dev status
```

###  Multi-Database

Configure as many databases as you need:

```python
# Primary database
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],
)

# Analytics database
analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://localhost:8123/analytics",
    model_paths=["app.models.analytics"],
)
```

###  Security First

Keep credentials out of code:

```python
import os

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    secure_values=True,  # Hide credentials in output
)
```

###  Validation

DBWarden validates your configuration:

-  Exactly one `default=True`
-  Unique database names
-  No duplicate URLs
-  Required `model_paths` for multi-database
-  Consistent dev mode configuration

## Configuration Loading

DBWarden discovers your configuration automatically:

1. **Looks for `dbwarden.py`** in current directory or parents
2. **Checks `DBWARDEN_CONFIG_MODULE`** environment variable
3. **Scans for `database_config()` calls** in your codebase (full project tree walk)
4. **Looks for `warden.toml`** as an alternative TOML-based config file

`dbwarden.py` is the default convention and the file created by `dbwarden init`, but `database_config(...)` can live in any discovered Python file inside your project.

## Common Patterns

### Single Database (Minimal)

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
)
```

### With Dev Mode (Recommended)

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",
    model_paths=["app.models"],
)
```

### Multiple Databases

```python
from dbwarden import database_config

# Primary
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],
)

# Analytics
analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    model_paths=["app.models.analytics"],
)
```

### Production with Environment Variables

```python
import os
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    model_paths=["app.models"],
    secure_values=True,
)
```

## Why Python Configuration?

**vs TOML/YAML/INI:**
-  Type checking with your IDE
-  Dynamic configuration (loops, conditionals)
-  Environment variable integration
-  No schema mismatches
-  Can compute values

**vs Environment Variables Only:**
-  Version controlled
-  Self-documenting
-  Validation at load time
-  Multiple databases easy
-  Can reference code structures

## What's Next?

Ready to configure your first database? Start here:

- **[Quick Start](quick-start.md)** - Build your first configuration
- **[Concepts](concepts.md)** - Understand how it works
- **[Production Patterns](production-patterns.md)** - Real-world examples

Already familiar with configuration? Jump to:

- **[Connection URLs](connection-urls.md)** - URL format reference
- **[Troubleshooting](troubleshooting.md)** - Common issues
- **[Configuration API](../reference/configuration-api.md)** - Complete reference

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/model-discovery/
========================================================================

# Model Discovery

Learn how DBWarden discovers your SQLAlchemy models for migration generation.

## What Is Model Discovery?

Model discovery is the process where DBWarden:
1. Imports Python modules
2. Finds SQLAlchemy model classes
3. Extracts table metadata
4. Uses metadata to generate migrations

## The `model_paths` Parameter

### Basic Usage

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    model_paths=["app.models"],  #  Discover models here
)
```

### What Gets Discovered

DBWarden looks for classes that inherit from:
- `DeclarativeBase` (SQLAlchemy 2.0+)
- `declarative_base()` return value (SQLAlchemy 1.4)

**Example models:**

```python
# app/models.py
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from sqlalchemy import String, Integer

class Base(DeclarativeBase):
    pass

class User(Base):  #  Discovered
    __tablename__ = "users"
    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))

class Order(Base):  #  Discovered
    __tablename__ = "orders"
    id: Mapped[int] = mapped_column(Integer, primary_key=True)
```

## When Is It Required?

### Single Database (Optional)

For single-database projects, `model_paths` is optional:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    # No model_paths - DBWarden scans entire codebase
)
```

DBWarden will scan your entire codebase for models.

Even for single-database projects, specifying `model_paths` makes discovery faster and more predictable.

### Multiple Databases (Required)

For multi-database projects, `model_paths` is **required** for each database:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],  #  Required
)

analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    model_paths=["app.models.analytics"],  #  Required
)
```

**Why required?** To prevent ambiguity about which models belong to which database.

## Discovery Algorithm

### Step 1: Import Modules

DBWarden imports each module in `model_paths`:

```python
model_paths=["app.models", "app.legacy.models"]
```

Becomes:

```python
import app.models
import app.legacy.models
```

### Step 2: Recursive Discovery

For each imported module, DBWarden recursively imports submodules:

```
app/
  models/
    __init__.py       #  Imported
    user.py           #  Imported
    order.py          #  Imported
    admin/
      __init__.py     #  Imported
      admin_user.py   #  Imported
```

### Step 3: Find Model Classes

For each module, DBWarden inspects all classes and finds those inheriting from `DeclarativeBase`.

### Step 4: Extract Metadata

For each model class, DBWarden extracts:
- Table name
- Columns (name, type, constraints)
- Indexes
- Foreign keys
- Check constraints
- Unique constraints

## Module Path Examples

### Single Module

```python
model_paths=["app.models"]
```

Discovers models from:
- `app/models.py` (if it's a file)
- `app/models/__init__.py` (if it's a package)
- `app/models/*.py` (all submodules)

### Multiple Modules

```python
model_paths=["app.models", "app.legacy"]
```

Discovers models from both `app.models` and `app.legacy`.

### Nested Modules

```python
model_paths=["app.models.api", "app.models.admin"]
```

Discovers models from:
- `app/models/api.py` or `app/models/api/*.py`
- `app/models/admin.py` or `app/models/admin/*.py`

### Absolute vs Relative

**Absolute (recommended):**
```python
model_paths=["app.models"]  # From project root
```

**Not supported:**
```python
model_paths=["./models"]  # Relative paths don't work
model_paths=["models"]     # May work if on PYTHONPATH
```

## Common Patterns

### Pattern 1: Single Module

```
app/
  models.py    # All models in one file
```

```python
model_paths=["app.models"]
```

### Pattern 2: Module Package

```
app/
  models/
    __init__.py
    user.py
    order.py
    product.py
```

```python
model_paths=["app.models"]
```

### Pattern 3: Multi-Database

```
app/
  models/
    primary/
      __init__.py
      user.py
      order.py
    analytics/
      __init__.py
      event.py
      metric.py
```

```python
# Primary database
model_paths=["app.models.primary"]

# Analytics database  
model_paths=["app.models.analytics"]
```

### Pattern 4: Legacy + New

```
app/
  models/      # New models
    __init__.py
    user.py
  legacy/      # Legacy models
    models.py
```

```python
model_paths=["app.models", "app.legacy.models"]
```

## Model Path Validation

### No Overlap (Default)

By default, model paths cannot overlap between databases:

```python
#  Error: overlap detected
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models"],  # Same path!
)
```

### Allow Overlap

If models genuinely belong to multiple databases:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.shared"],
    overlap_models=True,  #  Allow overlap
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.shared"],
    overlap_models=True,  #  Allow overlap
)
```

Both databases will include the same tables. Make sure this is intentional.

## Troubleshooting

### "No SQLAlchemy models found"

**Symptom:** DBWarden can't find your models.

**Causes:**

1. **Models not imported**

```python
# app/models/__init__.py
#  Wrong - models not imported
from sqlalchemy.orm import DeclarativeBase

class Base(DeclarativeBase):
    pass

#  Correct - import models
from app.models.user import User
from app.models.order import Order
```

2. **Wrong module path**

```python
#  Wrong
model_paths=["models"]  # Not on PYTHONPATH

#  Correct
model_paths=["app.models"]
```

3. **Circular imports**

```python
# app/models/user.py
from app.models.order import Order  #  Circular import

# app/models/order.py
from app.models.user import User  #  Circular import
```

**Solution:** Use forward references:

```python
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from app.models.order import Order
```

### "model_paths is required"

**Symptom:** Error when running commands with multiple databases.

**Cause:** Multiple databases configured without `model_paths`.

**Solution:** Add `model_paths` to each database:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],  #  Add this
    ...
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],  #  Add this
    ...
)
```

### "model_paths overlap detected"

**Symptom:** Two databases have overlapping model paths.

**Cause:** Same path used for multiple databases.

**Solution 1:** Use separate paths:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],  # Different path
    ...
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],  # Different path
    ...
)
```

**Solution 2:** Allow overlap (if intentional):

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.shared"],
    overlap_models=True,  #  Allow overlap
    ...
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.shared"],
    overlap_models=True,  #  Allow overlap
    ...
)
```

### Import Errors

**Symptom:** `ModuleNotFoundError` or `ImportError` when running commands.

**Cause:** DBWarden tries to import module but it doesn't exist.

**Solution:** Verify the module path:

```bash
python -c "import app.models"  # Test import
```

If import fails, fix your module structure or PYTHONPATH.

## Performance Considerations

### Slow Discovery

If discovery is slow, reduce the search space:

**Before (slow):**
```python
model_paths=["app"]  # Scans entire app
```

**After (fast):**
```python
model_paths=["app.models"]  # Only scans models
```

### Import Side Effects

Models should be pure:

```python
#  Bad - side effects on import
class User(Base):
    __tablename__ = "users"
    ...

print("User model loaded!")  # Side effect

#  Good - no side effects
class User(Base):
    __tablename__ = "users"
    ...
```

## Advanced: Dynamic Model Paths

You can compute `model_paths` dynamically:

```python
import os

environment = os.getenv("ENV", "dev")

if environment == "production":
    model_paths = ["app.models.production"]
else:
    model_paths = ["app.models.dev"]

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="...",
    model_paths=model_paths,
)
```

## What's Next?

- **[Dev Mode](dev-mode.md)** - Local development workflows
- **[Multi-Database](multi-database.md)** - Organize multi-database models
- **[Troubleshooting](troubleshooting.md)** - Common configuration issues

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/multi-database/
========================================================================

# Multi-Database

Configure and manage multiple databases in a single project.

## When to Use Multiple Databases

Common scenarios:
- **Microservices** - Each service has its own database
- **Read/Write Split** - Primary for writes, replica for reads
- **Domain Separation** - Transactions, analytics, logs in separate databases
- **Legacy Integration** - New and old databases coexist
- **Multi-Tenancy** - One database per tenant

## Basic Setup

Configure each database with `database_config()`:

```python
# dbwarden.py
from dbwarden import database_config

# Primary database
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],
)

# Analytics database
analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    model_paths=["app.models.analytics"],
)

# Logging database
logging = database_config(
    database_name="logging",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/logs",
    model_paths=["app.models.logging"],
)
```

## Model Organization

### Pattern 1: Separate Modules

```
app/
  models/
    primary/
      __init__.py
      user.py
      order.py
    analytics/
      __init__.py
      event.py
      metric.py
    logging/
      __init__.py
      audit_log.py
```

Configuration:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],
    ...
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],
    ...
)

logging = database_config(
    database_name="logging",
    model_paths=["app.models.logging"],
    ...
)
```

### Pattern 2: Shared Base Classes

```python
# app/models/base.py
from sqlalchemy.orm import DeclarativeBase

class PrimaryBase(DeclarativeBase):
    pass

class AnalyticsBase(DeclarativeBase):
    pass

# app/models/primary/user.py
from app.models.base import PrimaryBase

class User(PrimaryBase):
    __tablename__ = "users"
    ...

# app/models/analytics/event.py
from app.models.base import AnalyticsBase

class Event(AnalyticsBase):
    __tablename__ = "events"
    ...
```

## CLI Usage

### Target Specific Database

```bash
# Migrate primary
$ dbwarden migrate --database primary

# Migrate analytics
$ dbwarden migrate --database analytics

# Status for logging
$ dbwarden status --database logging
```

### Target All Databases

```bash
# Migrate all
$ dbwarden migrate --all

# Status for all
$ dbwarden status --all

# Rollback all
$ dbwarden rollback --all
```

### Default Database

The database with `default=True` is used when `--database` is omitted:

```bash
# These are equivalent when primary is default:
$ dbwarden migrate
$ dbwarden migrate --database primary
```

## Migration Directories

Each database has its own migration directory:

```
migrations/
  primary/
    0001_create_users.sql
    0002_create_orders.sql
  analytics/
    0001_create_events.sql
    0002_create_metrics.sql
  logging/
    0001_create_audit_logs.sql
```

Configure custom directories:

```python
primary = database_config(
    database_name="primary",
    migrations_dir="migrations/primary",  # Custom path
    ...
)
```

## Independent Migration Histories

Each database maintains its own migration history:

```bash
# Check primary history
$ dbwarden history --database primary
Applied Migrations (primary)
  0001_create_users (2024-01-15 10:30:00)
  0002_create_orders (2024-01-16 11:00:00)

# Check analytics history
$ dbwarden history --database analytics
Applied Migrations (analytics)
  0001_create_events (2024-01-15 10:35:00)
```

Migrations are **completely independent** - you can migrate one database without affecting others.

## Dev Mode with Multiple Databases

Configure dev mode for each database:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_primary.db",
    model_paths=["app.models.primary"],
)

analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_analytics.db",
    model_paths=["app.models.analytics"],
)
```

Use dev mode:

```bash
# Dev mode for all databases
$ dbwarden --dev migrate --all

# Dev mode for specific database
$ dbwarden --dev migrate --database analytics
```

## Common Patterns

### Pattern 1: Read/Write Split

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://primary-host/myapp",
    model_paths=["app.models"],
)

replica = database_config(
    database_name="replica",
    database_type="postgresql",
    database_url_sync="postgresql://replica-host/myapp",
    model_paths=["app.models"],  # Same models
    overlap_models=True,          # Allow overlap
)
```

**Note:** Run migrations only against primary; replica replicates automatically.

### Pattern 2: Domain Separation

```python
# Transactions
transactions = database_config(
    database_name="transactions",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/transactions",
    model_paths=["app.models.transactions"],
)

# Analytics
analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://localhost:8123/analytics",
    model_paths=["app.models.analytics"],
)

# Audit logs
audit = database_config(
    database_name="audit",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/audit",
    model_paths=["app.models.audit"],
)
```

### Pattern 3: Multi-Tenant

```python
tenants = ["tenant_a", "tenant_b", "tenant_c"]

for tenant in tenants:
    db = database_config(
        database_name=tenant,
        default=(tenant == "tenant_a"),
        database_type="postgresql",
        database_url_sync=f"postgresql://localhost/{tenant}",
        model_paths=["app.models"],  # Same models for all tenants
    )
```

## Validation Rules

### Required: `model_paths`

When you have multiple databases, each **must** specify `model_paths`:

```python
#  Error: model_paths required
analytics = database_config(
analytics = database_config(database_name="analytics", ...)  # Missing model_paths

#  Correct
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],
    ...
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],
    ...
)
```

### No Overlap (Default)

Model paths cannot overlap:

```python
#  Error: overlap detected
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    ...
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models"],  # Same path
    ...
)
```

### Allow Overlap

For read replicas or shared models:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    overlap_models=True,  #  Allow overlap
    ...
)
replica = database_config(
    database_name="replica",
    model_paths=["app.models"],
    overlap_models=True,  #  Allow overlap
    ...
)
```

## Troubleshooting

### "model_paths is required"

**Solution:** Add `model_paths` to all databases:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],  #  Add this
    ...
)
```

### "model_paths overlap detected"

**Solution 1:** Use separate paths:
```python
model_paths=["app.models.primary"]
model_paths=["app.models.analytics"]
```

**Solution 2:** Allow overlap:
```python
overlap_models=True
```

### Wrong database targeted

**Check default:**
```bash
$ dbwarden settings show  # Shows which is default
```

**Be explicit:**
```bash
$ dbwarden migrate --database analytics  # Specify database
```

## What's Next?

- **[Production Patterns](production-patterns.md)** - Deploy multi-database apps
- **[Troubleshooting](troubleshooting.md)** - Common issues

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/production-patterns/
========================================================================

# Production Patterns

Real-world configuration patterns for production deployments.

## Environment Variables

### Basic Pattern

```python
import os
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    model_paths=["app.models"],
    secure_values=True,
)
```

### With Validation

```python
import os

DATABASE_URL = os.getenv("DATABASE_URL")
if not DATABASE_URL:
    raise ValueError("DATABASE_URL environment variable is required")

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=DATABASE_URL,
    model_paths=["app.models"],
    secure_values=True,
)
```

### With Defaults

```python
import os

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv(
        "DATABASE_URL",
        "postgresql://localhost/myapp"  # Fallback
    ),
    model_paths=["app.models"],
)
```

## Docker

### Docker Compose

```yaml
# docker-compose.yml
services:
  app:
    build: .
    environment:
      DATABASE_URL: postgresql://user:password@db:5432/myapp
    depends_on:
      - db
  
  db:
    image: postgres:15
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: myapp
```

```python
# dbwarden.py
import os

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    model_paths=["app.models"],
)
```

### Dockerfile

```dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY pyproject.toml uv.lock .
RUN uv sync

COPY . .

# Run migrations on container start
CMD ["sh", "-c", "dbwarden migrate && python app/main.py"]
```

## Kubernetes

### Secrets

```yaml
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: database-secret
type: Opaque
stringData:
  url: postgresql://user:password@postgres-service:5432/myapp
```

### Deployment with Init Container

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      initContainers:
      # Run migrations before app starts
      - name: migrate
        image: myapp:latest
        command: ["dbwarden", "migrate"]
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-secret
              key: url
      
      containers:
      - name: app
        image: myapp:latest
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-secret
              key: url
```

### ConfigMap for Model Paths

```yaml
# config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  model_paths: "app.models"
```

## AWS

### RDS with Secrets Manager

```python
import os
import json
import boto3

def get_database_url():
    secret_name = os.getenv("DB_SECRET_NAME")
    region = os.getenv("AWS_REGION", "us-east-1")
    
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    secret = json.loads(response["SecretString"])
    
    return (
        f"postgresql://{secret['username']}:{secret['password']}"
        f"@{secret['host']}:{secret['port']}/{secret['dbname']}"
    )

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=get_database_url(),
    model_paths=["app.models"],
    secure_values=True,
)
```

### RDS Connection via IAM

```python
import os
import boto3

def get_rds_auth_token():
    rds_client = boto3.client("rds")
    return rds_client.generate_db_auth_token(
        DBHostname=os.getenv("DB_HOST"),
        Port=5432,
        DBUsername=os.getenv("DB_USER"),
    )

database_url = (
    f"postgresql://{os.getenv('DB_USER')}:{get_rds_auth_token()}"
    f"@{os.getenv('DB_HOST')}:5432/{os.getenv('DB_NAME')}"
)

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=database_url,
    model_paths=["app.models"],
)
```

## Multi-Environment

### Environment-Based Configuration

```python
import os

ENVIRONMENT = os.getenv("ENVIRONMENT", "dev")

if ENVIRONMENT == "production":
    database_url = os.getenv("PROD_DATABASE_URL")
    database_type = "postgresql"
elif ENVIRONMENT == "staging":
    database_url = os.getenv("STAGING_DATABASE_URL")
    database_type = "postgresql"
else:
    database_url = "sqlite:///./dev.db"
    database_type = "sqlite"

primary = database_config(
    database_name="primary",
    default=True,
    database_type=database_type,
    database_url_sync=database_url,
    model_paths=["app.models"],
    secure_values=(ENVIRONMENT != "dev"),
)
```

### Separate Config Files

```python
# dbwarden.py
import os
from importlib import import_module

environment = os.getenv("ENVIRONMENT", "dev")
config_module = import_module(f"config.{environment}")
config_module.setup_databases()
```

```python
# config/production.py
import os
from dbwarden import database_config

def setup_databases():
    primary = database_config(
        database_name="primary",
        default=True,
        database_type="postgresql",
        database_url_sync=os.getenv("DATABASE_URL"),
        model_paths=["app.models"],
        secure_values=True,
    )
```

## Connection Pools

### PostgreSQL with Pooling

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=(
        "postgresql://user:pass@localhost/myapp"
        "?pool_size=20"
        "&max_overflow=10"
        "&pool_timeout=30"
        "&pool_recycle=3600"
    ),
    model_paths=["app.models"],
)
```

### External Pooler (PgBouncer)

```python
# Connection through PgBouncer
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@pgbouncer:6432/myapp",
    model_paths=["app.models"],
)
```

## SSL/TLS

### PostgreSQL with SSL

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=(
        "postgresql://user:pass@host/myapp"
        "?sslmode=require"
        "&sslrootcert=/path/to/ca.pem"
        "&sslcert=/path/to/client-cert.pem"
        "&sslkey=/path/to/client-key.pem"
    ),
    model_paths=["app.models"],
)
```

### Environment-Based SSL

```python
import os

ssl_mode = os.getenv("DB_SSL_MODE", "prefer")
ca_cert = os.getenv("DB_CA_CERT_PATH", "")

ssl_params = f"?sslmode={ssl_mode}"
if ca_cert:
    ssl_params += f"&sslrootcert={ca_cert}"

database_url = f"postgresql://user:pass@host/myapp{ssl_params}"

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=database_url,
    model_paths=["app.models"],
)
```

## High Availability

### Multiple Replicas

```python
# Primary (writes)
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("PRIMARY_DATABASE_URL"),
    model_paths=["app.models"],
)

# Replica (reads)
replica = database_config(
    database_name="replica",
    database_type="postgresql",
    database_url_sync=os.getenv("REPLICA_DATABASE_URL"),
    model_paths=["app.models"],
    overlap_models=True,
)
```

### Automatic Failover

```python
import os

# Try primary, fallback to replica
primary_url = os.getenv("PRIMARY_DATABASE_URL")
replica_url = os.getenv("REPLICA_DATABASE_URL")

# Application logic handles failover
database_url = primary_url  # Start with primary

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=database_url,
    model_paths=["app.models"],
)
```

## Monitoring

### Application Name

```python
import os

app_name = os.getenv("APP_NAME", "myapp")
hostname = os.getenv("HOSTNAME", "unknown")

database_url = (
    f"postgresql://user:pass@host/myapp"
    f"?application_name={app_name}-{hostname}"
)

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=database_url,
    model_paths=["app.models"],
)
```

Check active connections:

```sql
SELECT application_name, count(*)
FROM pg_stat_activity
GROUP BY application_name;
```

## Security Best Practices

### Never Commit Credentials

```python
#  Bad
database_url_sync="postgresql://user:password@localhost/myapp"

#  Good
database_url_sync=os.getenv("DATABASE_URL")
```

### Use Least Privilege

Create application user with minimal permissions:

```sql
CREATE USER myapp_user WITH PASSWORD 'secret';
GRANT CONNECT ON DATABASE myapp TO myapp_user;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO myapp_user;
-- Don't grant DROP, TRUNCATE, CREATE, etc.
```

### Rotate Credentials

```python
# Use short-lived tokens
database_url = get_temporary_database_credentials()

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=database_url,
    model_paths=["app.models"],
)
```

## CI/CD Integration

### GitHub Actions

```yaml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Install dependencies
        run: uv add dbwarden
      
      - name: Run migrations
        run: dbwarden migrate --database primary
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
```

### GitLab CI

```yaml
migrate:
  stage: deploy
  script:
    - uv add dbwarden
    - dbwarden migrate --database primary
  environment:
    name: production
  only:
    - main
  variables:
    DATABASE_URL: $DATABASE_URL
```

## What's Next?

- **[Troubleshooting](troubleshooting.md)** - Common production issues
- **[Configuration API Reference](../reference/configuration-api.md)** - Complete parameter docs

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/quick-start/
========================================================================

# Quick Start

Configure your first database in **2 minutes**.

## Prerequisites

You should have:
- Python 3.10+ installed
- DBWarden installed (`uv add dbwarden`)
- A database to connect to (or use SQLite)

## Step 1: Initialize

Create project structure:

```bash
$ dbwarden init
```

This creates:
- `migrations/` directory
- `dbwarden.py` configuration file

## Step 2: Your First Configuration

Open `dbwarden.py` and add:

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
)
```

That's it! **4 required parameters**:
- `database_name` - What to call this database
- `default` - Is this the default?
- `database_type` - What kind of database?
- `database_url_sync` - How to connect? (sync URL for CLI/migrations)

Start with SQLite for the simplest setup. Switch to PostgreSQL later.

## Step 3: Test the Configuration

Verify DBWarden can read your config:

```bash
$ dbwarden settings show
```

You'll see:

```
Database Configuration
════════════════════════════════════════

primary (default)
  Type: sqlite
  URL: sqlite:///./app.db
  Migrations: migrations/primary
```

## Step 4: Add Model Paths (Optional)

If you have SQLAlchemy models, tell DBWarden where they are:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    model_paths=["app.models"],  #  Add this
)
```

DBWarden will discover models from `app.models` and its submodules.

## Step 5: Upgrade to PostgreSQL

When you're ready for PostgreSQL:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/myapp",
    model_paths=["app.models"],
)
```

## Step 6: Add Dev Mode (Recommended)

Keep SQLite for local dev, use PostgreSQL in production:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/myapp",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",
    model_paths=["app.models"],
)
```

Now you can run commands against SQLite locally:

```bash
$ dbwarden --dev migrate
$ dbwarden --dev status
```

And against PostgreSQL in production:

```bash
$ dbwarden migrate
$ dbwarden status
```

## What Just Happened?

### `database_config` registered your database

When Python loads `dbwarden.py`, it executes `database_config()` which:
1. Validates your parameters
2. Registers the database in DBWarden's internal registry
3. Sets up migration directories

### DBWarden Can Now Find Your Database

All CLI commands now know about your database:

```bash
$ dbwarden make-migrations "create users"
$ dbwarden migrate
$ dbwarden status
$ dbwarden history
```

## Common First-Time Issues

### "No configuration found"

**Cause:** DBWarden can't find `dbwarden.py`

**Solution:** Ensure you're in the project directory and `dbwarden.py` exists.

### "No SQLAlchemy models found"

**Cause:** DBWarden can't discover your models

**Solution:** Add `model_paths` to your config:

```python
model_paths=["app.models"]
```

### "Exactly one default=True required"

**Cause:** Multiple databases without one marked as default

**Solution:** Set one database to `default=True`

## Complete Minimal Example

```python
# dbwarden.py
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    model_paths=["app.models"],
)
```

## Complete Production Example

```python
# dbwarden.py
import os
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",
    model_paths=["app.models"],
    secure_values=True,
)
```

## What's Next?

- **[Concepts](concepts.md)** - Understand how configuration works
- **[Connection URLs](connection-urls.md)** - Learn URL formats for different databases
- **[Dev Mode](dev-mode.md)** - Deep dive into dev workflows
- **[Production Patterns](production-patterns.md)** - Real-world examples

========================================================================
PAGE: https://dbwarden.emiliano-go.com/configuration/troubleshooting/
========================================================================

# Troubleshooting

Solutions to common configuration issues.

## "No configuration found"

### Symptom

```
DBWardenConfigError: No configuration found
```

### Causes & Solutions

**Cause 1: No `dbwarden.py` file**

```bash
# Check if file exists
ls dbwarden.py
```

**Solution:** Create `dbwarden.py`:

```bash
$ dbwarden init
```

**Cause 2: Wrong directory**

DBWarden looks in current directory and parents.

**Solution:** Navigate to project root:

```bash
cd /path/to/project
$ dbwarden migrate
```

**Cause 3: No `database_config()` calls**

**Solution:** Add configuration:

```python
# dbwarden.py
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
)
```

**Cause 4: Import error in config file**

```python
# dbwarden.py
from app.models import Base  #  Import fails
```

**Solution:** Fix imports or use lazy loading:

```python
# Don't import models in config file
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],  #  Use model_paths instead
    ...
)
```

## "Exactly one default=True required"

### Symptom

```
ConfigurationError: Exactly one default=True required
```

### Causes & Solutions

**Cause 1: No default database**

```python
#  Wrong
analytics = database_config(
analytics = database_config(database_name="analytics", default=False, ...)
```

**Solution:** Set one database as default:

```python
#  Correct
analytics = database_config(
analytics = database_config(database_name="analytics", default=False, ...)
```

**Cause 2: Multiple defaults**

```python
#  Wrong
analytics = database_config(
analytics = database_config(database_name="analytics", default=True, ...)
```

**Solution:** Only one default:

```python
#  Correct
analytics = database_config(
analytics = database_config(database_name="analytics", ...)  # default=False implied
```

## "Duplicate database_name"

### Symptom

```
ConfigurationError: Duplicate database_name 'primary'
```

### Cause

Same `database_name` used twice:

```python
primary = database_config(
primary = database_config(database_name="primary", ...)  #  Duplicate
```

### Solution

Use unique names:

```python
analytics = database_config(
analytics = database_config(database_name="analytics", ...)  #  Different name
```

## "No SQLAlchemy models found"

### Symptom

```
Warning: No SQLAlchemy models found
```

### Causes & Solutions

**Cause 1: Wrong `model_paths`**

```python
#  Wrong
model_paths=["models"]  # Not on PYTHONPATH
```

**Solution:** Use correct Python path:

```python
#  Correct
model_paths=["app.models"]
```

**Cause 2: Models not imported**

```python
# app/models/__init__.py
#  Wrong - models not imported
from sqlalchemy.orm import DeclarativeBase

class Base(DeclarativeBase):
    pass
```

**Solution:** Import models:

```python
# app/models/__init__.py
#  Correct
from sqlalchemy.orm import DeclarativeBase
from app.models.user import User
from app.models.order import Order

class Base(DeclarativeBase):
    pass
```

**Cause 3: Missing `model_paths`**

**Solution:** Add `model_paths`:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],  #  Add this
    ...
)
```

**Cause 4: Circular imports**

```python
# app/models/user.py
from app.models.order import Order  #  Circular

# app/models/order.py
from app.models.user import User  #  Circular
```

**Solution:** Use TYPE_CHECKING:

```python
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from app.models.order import Order
```

## "model_paths is required"

### Symptom

```
ConfigurationError: model_paths is required when more than one database is configured
```

### Cause

Multiple databases without `model_paths`:

```python
#  Wrong
analytics = database_config(
analytics = database_config(database_name="analytics", ...)  # No model_paths
```

### Solution

Add `model_paths` to all databases:

```python
#  Correct
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],
    ...
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],
    ...
)
```

## "model_paths overlap detected"

### Symptom

```
ConfigurationError: model_paths overlap detected between 'primary' and 'analytics'
```

### Cause

Same model paths for different databases:

```python
#  Wrong
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    ...
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models"],  #  Same path
    ...
)
```

### Solutions

**Solution 1: Use separate paths**

```python
#  Correct
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],
    ...
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],
    ...
)
```

**Solution 2: Allow overlap (if intentional)**

```python
#  Correct for read replicas
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    overlap_models=True,
    ...
)
replica = database_config(
    database_name="replica",
    model_paths=["app.models"],
    overlap_models=True,
    ...
)
```

## "model_tables overlap detected"

### Symptom

```
ConfigurationError: model_tables overlap detected: table 'users' in 'primary' is also in 'analytics'
```

### Cause

Two databases have `model_tables` lists that share table names:

```python
#  Wrong
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    model_tables=["users", "posts"],
    ...
)
analytics = database_config(
    database_name="analytics",
    model_paths=["other_models"],
    model_tables=["users"],  # 'users' already owned by primary
    ...
)
```

### Solutions

**Solution 1: Remove duplicate table name**

```python
#  Correct
analytics = database_config(
    database_name="analytics",
    model_paths=["other_models"],
    model_tables=["analytics_events"],  # No overlap with primary
    ...
)
```

**Solution 2: Allow overlap (if intentional)**

```python
#  Correct for shared tables
analytics = database_config(
    database_name="analytics",
    model_paths=["other_models"],
    model_tables=["users", "analytics_events"],
    overlap_models=True,  # Allow overlap
    ...
)
```

## "dev_database_url is required"

### Symptom

```
ConfigurationError: dev_database_url is required when dev_database_type is set
```

### Cause

Set `dev_database_type` without `dev_database_url`:

```python
#  Wrong
primary = database_config(
    database_name="primary",
    dev_database_type="sqlite",
    # Missing dev_database_url
    ...
)
```

### Solution

Add both dev parameters:

```python
#  Correct
primary = database_config(
    database_name="primary",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",  #  Add this
    ...
)
```

## Connection Errors

### "could not connect to server"

**Cause:** Database server not running or unreachable.

**Solutions:**

1. **Check database is running:**

```bash
# PostgreSQL
sudo systemctl status postgresql

# Docker
docker ps | grep postgres
```

2. **Check connection URL:**

```python
# Verify host, port, credentials
database_url_sync="postgresql://user:pass@localhost:5432/myapp"
```

3. **Test connection:**

```bash
# PostgreSQL
psql -h localhost -U user -d myapp

# MySQL
mysql -h localhost -u user -p myapp
```

### "authentication failed"

**Cause:** Wrong credentials.

**Solutions:**

1. **Check credentials:**

```bash
# PostgreSQL
psql -h localhost -U user -d myapp
```

2. **Verify environment variable:**

```bash
echo $DATABASE_URL
```

3. **URL encode special characters:**

```python
from urllib.parse import quote_plus

password = "p@ss:word"
encoded = quote_plus(password)  # "p%40ss%3Aword"
```

### "database does not exist"

**Cause:** Database not created.

**Solution:** Create database:

```sql
-- PostgreSQL
CREATE DATABASE myapp;

-- MySQL
CREATE DATABASE myapp CHARACTER SET utf8mb4;
```

## Import Errors

### "ModuleNotFoundError"

**Cause:** Python can't find module.

**Solutions:**

1. **Check PYTHONPATH:**

```bash
export PYTHONPATH=/path/to/project:$PYTHONPATH
```

2. **Install package:**

```bash
uv add -e .  # Editable install
```

3. **Verify import:**

```bash
python -c "import app.models"
```

## Performance Issues

### Slow configuration loading

**Cause:** Large codebase scan.

**Solution:** Specify `model_paths`:

```python
#  Slow - scans everything
primary = database_config(

#  Fast - targeted scan
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    ...
)
```

### Slow imports

**Cause:** Heavy imports in config file.

**Solution:** Avoid imports in `dbwarden.py`:

```python
#  Slow
from app.models import Base
from app.services import setup

#  Fast
from dbwarden import database_config

db = database_config(
```

## Debugging Tips

### Enable verbose output

```bash
$ dbwarden --verbose migrate
```

### Check configuration

```bash
# Show all configuration
$ dbwarden settings show

# Show specific database
$ dbwarden settings show --database primary
```

### Test imports

```bash
python -c "import dbwarden; print('OK')"
python -c "from dbwarden import database_config; print('OK')"
```

### Verify database connection

```bash
$ dbwarden check-db
$ dbwarden check-db --database primary
```

## What's Next?

- **[Configuration API Reference](../reference/configuration-api.md)** - Complete parameter docs
- **[Quick Start](quick-start.md)** - Start fresh with correct setup

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/01-project-setup/
========================================================================

# 1. Project Setup

## What You'll Learn

- How to initialize a DBWarden project with `dbwarden init`
- How configuration is structured via `database_config()`
- How to inspect your loaded configuration

## Prerequisites

- Python 3.12+ with `uv add dbwarden sqlalchemy`
- The `examples/core/` directory (see [Cookbook Index](index.md))

## Step 1: Initialize the Project

```bash
cd examples/core/
bash scripts/01-setup.sh
```

The `dbwarden init` command creates the directory structure DBWarden expects:

```
migrations/
  primary/          # Migration files for the 'primary' database
```

It also writes a starter `dbwarden.py` if one doesn't exist. In our case, we already have one with our configuration.

## Step 2: Understanding the Configuration

Our `examples/core/dbwarden.py`:

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    model_paths=["app"],
    model_tables=["users", "posts"],
)
```

Each parameter has a specific role:

| Parameter | Value | Purpose |
|-----------|-------|---------|
| `database_name` | `"primary"` | Logical name used in `--database primary` CLI flags |
| `default` | `True` | Used when no `--database` flag is given |
| `database_type` | `"sqlite"` | Dialect for SQL generation and connection |
| `database_url_sync` | `"sqlite:///./app.db"` | Synchronous connection URL |
| `model_paths` | `["app"]` | Python module paths to scan for SQLAlchemy models |
| `model_tables` | `["users", "posts"]` | Optional table-name filter for this database |

The return value `primary` is a `DatabaseHandle` object. It's also used later for FastAPI dependency injection: the same object provides `primary.async_session` and `primary.sync_session`.

## Step 3: Viewing the Configuration

```text
$ dbwarden config
Configuration:
  Databases:
    primary (default):
      Type: sqlite
      Sync URL: sqlite:///./app.db
      Model Paths: app
      Migrations Dir: migrations/primary
```

This confirms DBWarden has discovered and loaded your configuration. The `(default)` marker means `--database` can be omitted when targeting this database.

## What Happens Under the Hood

When you import `dbwarden` and call `database_config()`:

1. The function call is registered in DBWarden's internal registry
2. On first CLI command, DBWarden discovers `dbwarden.py` via AST scanning
3. It imports the module and executes each `database_config()` call
4. It validates uniqueness, default rules, and model path resolution
5. The resolved configuration is cached for the session

## Key Takeaways

- `dbwarden init` creates the directory skeleton: run it once per project
- `dbwarden config` shows what DBWarden actually resolved (useful for debugging)
- `database_config()` is the single entry point for all configuration
- `model_paths` controls which Python modules are scanned for models
- We chose SQLite here so the example runs with zero external services

## Next

[Section 2: Models & Migrations](02-models-and-migrations.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/02-models-and-migrations/
========================================================================

# 2. Defining Models and Generating Migrations

## What You'll Learn

- How to define SQLAlchemy models with `class Meta` annotations
- How `make-migrations` generates SQL from model changes
- How the generated SQL maps to database DDL
- How to create manual migrations with `dbwarden new`
- How to extract rollback SQL from an existing migration

## Prerequisites

- Completed [Section 1: Project Setup](01-project-setup.md)
- `examples/core/` with `app/models.py`

## Step 1: The Models

Our example project defines four models in `examples/core/app/models.py`. Here they are with explanations:

### User

```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    username: Mapped[str] = mapped_column(String(100), unique=True, nullable=False)
    full_name: Mapped[str | None] = mapped_column(String(200), nullable=True)
    is_active: Mapped[bool] = mapped_column(Boolean, default=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("CURRENT_TIMESTAMP"))

    class Meta(TableMeta):
        comment = "Core user accounts"
        indexes = [
            IndexSpec(name="ix_users_created_at", columns=["created_at"]),
        ]
```

Key points:

- `unique=True` on `email` and `username` generates `UNIQUE` constraints
- `nullable=True` (the default) allows `NULL`; `nullable=False` adds `NOT NULL`
- `server_default=text(...)` becomes a database-level `DEFAULT` clause in the DDL; `default=` is a Python-level default and is not rendered in SQL
- `class Meta(TableMeta)` is how we attach table-level metadata
- `IndexSpec` generates a named `CREATE INDEX` statement

### Post

```python
class Post(Base):
    __tablename__ = "posts"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String(255), nullable=False)
    body: Mapped[str] = mapped_column(Text, nullable=False)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("CURRENT_TIMESTAMP"))

    class Meta(TableMeta):
        comment = "User blog posts"
        indexes = [
            IndexSpec(name="ix_posts_user_id", columns=["user_id"]),
            IndexSpec(name="ix_posts_created_at", columns=["created_at"]),
        ]
```

Key points:

- `ForeignKey("users.id")` generates a `REFERENCES` clause
- Foreign key targets are rendered inline in `CREATE TABLE`

### Product (with CHECK constraint)

```python
class Product(Base):
    __tablename__ = "products"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    name: Mapped[str] = mapped_column(String(200), nullable=False)
    price: Mapped[float] = mapped_column(Float, nullable=False)
    description: Mapped[str | None] = mapped_column(Text, nullable=True)
    in_stock: Mapped[bool] = mapped_column(Boolean, default=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("CURRENT_TIMESTAMP"))

    class Meta(TableMeta):
        comment = "Product catalog"
        checks = [
            {"name": "ck_products_price_positive", "sql": "price > 0"},
        ]
```

Key points:

- `checks` in `class Meta` generates `CHECK` constraints
- Each check needs a `name` (constraint name) and `sql` (the expression)
- This prevents negative prices at the database level

### Tag

```python
class Tag(Base):
    __tablename__ = "tags"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    name: Mapped[str] = mapped_column(String(50), unique=True, nullable=False)

    class Meta(TableMeta):
        comment = "Taxonomy tags for products"
```

The simplest model: just an ID and a unique name.

## Step 2: Generating the Migration

```bash
cd examples/core
bash scripts/02-models-migrations.sh
```

The first script step runs:

```bash
$ dbwarden make-migrations "create core tables" --database primary
```

This compares the current model state against the database (or a stored snapshot). Since this is a fresh project, it detects four new tables and generates:

```sql
-- upgrade

CREATE TABLE IF NOT EXISTS posts (
    id INTEGER NOT NULL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    user_id INTEGER NOT NULL REFERENCES users(id),
    created_at DATETIME
)

CREATE TABLE IF NOT EXISTS products (
    id INTEGER NOT NULL PRIMARY KEY,
    name VARCHAR(200) NOT NULL,
    price FLOAT NOT NULL,
    description TEXT,
    in_stock BOOLEAN DEFAULT TRUE,
    created_at DATETIME
)

CREATE TABLE IF NOT EXISTS tags (
    id INTEGER NOT NULL PRIMARY KEY,
    name VARCHAR(50) NOT NULL UNIQUE
)

CREATE TABLE IF NOT EXISTS users (
    id INTEGER NOT NULL PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    username VARCHAR(100) NOT NULL UNIQUE,
    full_name VARCHAR(200),
    is_active BOOLEAN DEFAULT TRUE,
    created_at DATETIME
)

-- rollback

DROP TABLE users
DROP TABLE tags
DROP TABLE products
DROP TABLE posts
```

> **Note:** The example uses SQLite, which has limited DDL support. With PostgreSQL, DBWarden generates additional features:
> - **`CREATE INDEX IF NOT EXISTS ...`**: from `IndexSpec` entries in `class Meta`
> - **`COMMENT ON TABLE ...`**: from `Meta.comment` attributes
> - **`CONSTRAINT ... CHECK (...)`**: from `Meta.checks`
> - **`server_default`** expressions rendered as native SQL defaults
> - Inline `REFERENCES` become table-level `FOREIGN KEY` constraints
>
> The generated SQL is always backend-specific. DBWarden adapts to the `database_type` configured in `dbwarden.py`.

### Reading the Generated SQL

Let's walk through what each section does:

**`-- upgrade`**: Applied when you run `dbwarden migrate`

1. **`CREATE TABLE IF NOT EXISTS posts (...)`**: Creates posts with a foreign key reference to `users(id)` (inline `REFERENCES` style for SQLite). The foreign key originates from `ForeignKey("users.id")` on the `user_id` column.

2. **`CREATE TABLE IF NOT EXISTS products (...)`**: Creates products with a `CHECK` constraint defined in `class Meta`. In SQLite, CHECK constraints must be inline; with PostgreSQL they become `CONSTRAINT ... CHECK (...)`.

3. **`CREATE TABLE IF NOT EXISTS tags (...)`**: Simple table with a unique constraint on `name`.

4. **`CREATE TABLE IF NOT EXISTS users (...)`**: Creates the users table with all columns, primary key, and unique constraints inline.

Note that with this SQLite backend the table order differs from the order in our Python models, and some features are omitted:
- **IndexSpec entries** generate `CREATE INDEX` only on PostgreSQL and ClickHouse
- **`COMMENT ON TABLE`** is only generated for PostgreSQL
- **`server_default`** expressions render as native SQL defaults on PostgreSQL

**`-- rollback`**: Applied when you run `dbwarden rollback`

1. Drops tables. Order may vary by backend; DBWarden handles dependency ordering automatically.

### Auto-generated Migration Name

The migration file is named automatically:

```
primary__0001_create_core_tables.sql
```

The naming pattern is:

```
{database_name}__{4-digit-version}_{auto-generated-description}.sql
```

### PostgreSQL-Specific Model Metadata

When your `database_type` is `"postgresql"`, DBWarden supports PostgreSQL-specific table and column metadata. The following model shows tablespace, fillfactor, identity columns, and column compression:

```python
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class Order(Base):
    __tablename__ = "orders"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    total: Mapped[float] = mapped_column(Float)
    created_at: Mapped[datetime] = mapped_column(TIMESTAMP)

    class Meta(PGTableMeta):
        pg_tablespace = "fast_space"
        pg_fillfactor = 90
        comment = "Customer orders"

        class id(PGColumnMeta):
            comment = "Order ID"
            pg = pg.field(identity="ALWAYS")

        class created_at(PGColumnMeta):
            pg = pg.field(compression="pglz")
```

The generated PostgreSQL DDL includes tablespace, fillfactor, identity columns, and column-level options:

```sql
CREATE TABLE IF NOT EXISTS orders (
    id INTEGER GENERATED ALWAYS AS IDENTITY NOT NULL,
    total FLOAT NOT NULL,
    created_at TIMESTAMP NOT NULL COMPRESSION pglz
) TABLESPACE fast_space WITH (fillfactor=90);

COMMENT ON TABLE orders IS 'Customer orders';
COMMENT ON COLUMN orders.id IS 'Order ID';
```

For PostgreSQL views, use `PGViewMeta` instead of `PGTableMeta`:

```python
from dbwarden.databases.pgsql import PGViewMeta

class ActiveUser(Base):
    __tablename__ = "active_users"

    id: Mapped[int] = mapped_column(Integer)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(PGViewMeta):
        pg_view_query = "SELECT id, email FROM users WHERE active = true"
        pg_view_materialized = False
```

The generated DDL creates a regular view:

```sql
CREATE OR REPLACE VIEW active_users AS SELECT id, email FROM users WHERE active = true;
```

For a materialized view with auto-refresh:

```python
class OrderSummary(Base):
    __tablename__ = "order_summary"

    user_id: Mapped[int] = mapped_column(Integer)
    total: Mapped[float] = mapped_column(Integer)

    class Meta(PGViewMeta):
        pg_view_query = "SELECT user_id, count(*) AS total FROM orders GROUP BY user_id"
        pg_view_materialized = True
        pg_view_auto_refresh = True
```

The first migration generates `CREATE MATERIALIZED VIEW order_summary AS ...`. Subsequent migrations include `REFRESH MATERIALIZED VIEW order_summary;`.

To scope a table or view to a PostgreSQL schema, set `pg_schema`:

```python
class Meta(PGTableMeta):
    pg_schema = "app"
```

The generated DDL uses the fully qualified name (e.g. `app.users`). Set `pg_schema` at the config level in `database_config(...)` to set the connection `search_path` for all unqualified references.

## Step 3: Creating a Manual Migration

Sometimes you need a migration that isn't model-driven: a data backfill, a stored procedure, or a complex SQL operation.

```bash
$ dbwarden new add_custom_table --database primary
```

This creates a blank migration:

```sql
-- upgrade

-- TODO: write your upgrade SQL here

-- rollback

-- TODO: write your rollback SQL here
```

You fill in both sections manually. Manual migrations follow the same file naming convention and are tracked alongside auto-generated ones.

## Step 4: Extracting Rollback SQL

If you have a migration file and need to extract just its rollback section:

```bash
$ dbwarden make-rollback migrations/primary/primary__0001_create_core_tables.sql
```

This prints the rollback SQL to stdout. Useful for quickly verifying what a rollback will do before running it.

## Key Takeaways

- DBWarden generates explicit, reviewable SQL: no hidden runtime behavior
- Every migration has both `-- upgrade` and `-- rollback` sections
- `class Meta(TableMeta)` is where table-level metadata (comments, indexes, checks) lives
- `IndexSpec` produces named `CREATE INDEX` statements; always prefer named indexes
- `dbwarden new` creates blank migrations for non-model-driven changes
- `dbwarden make-rollback` extracts rollback SQL for review

## Related Documentation

- [SQLAlchemy Models Reference](../models.md)
- [Modeling Guide](../getting-started/modeling.md)
- [Migration File Format](../migration-files.md)
- [`make-migrations` command](../commands/make-migrations.md)

## Next

[Section 3: Apply & Inspect](03-apply-and-inspect.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/03-apply-and-inspect/
========================================================================

# 3. Applying and Inspecting Migrations

## What You'll Learn

- How `dbwarden migrate` applies pending SQL
- How to roll back and downgrade to specific versions
- How to inspect migration history and status
- How to validate schema integrity and database connectivity

## Prerequisites

- Completed [Section 2](02-models-and-migrations.md) (migration file exists)
- `examples/core/` project

## Step 1: Apply Migrations

```bash
cd examples/core
bash scripts/03-apply-inspect.sh
```

### The Migrate Command

```bash
$ dbwarden migrate --database primary
```

When you run `migrate`, DBWarden:

1. Creates the metadata table (`_dbwarden_migrations`) if it doesn't exist
2. Creates the lock table (`_dbwarden_lock`) if it doesn't exist
3. Acquires a migration lock (prevents concurrent runs)
4. Reads all migration files and filters to pending (unapplied) ones
5. Executes the `-- upgrade` SQL of each pending migration
6. Records each migration's version, checksum, and timestamp
7. Writes a schema snapshot file for future diffs
8. Releases the lock

```
[DBWarden] Applying primary__0001_create_core_tables...
[DBWarden] Migration applied successfully (42ms)
[DBWarden] All migrations applied. Pending: 0
```

### Verify Status

```bash
$ dbwarden status --database primary
```

Output:

```
Database: primary
  Applied:  1
  Pending:  0
  Status:   up-to-date
```

### View History

```bash
$ dbwarden history --database primary
```

Output:

```
 Migration History (primary)
  V0001  create_core_tables  2025-01-15 10:30:00  a1b2c3d4...
```

The checksum (`a1b2c3d4...`) is a SHA-256 hash of the migration file content. This detects tampering or accidental edits after apply.

## Step 2: Rollback

```bash
$ dbwarden rollback --database primary --count 1
```

Rollback executes the `-- rollback` section of the most recently applied migration. After rollback:

```
[DBWarden] Rolling back primary__0001_create_core_tables...
[DBWarden] Rollback complete
```

```bash
$ dbwarden status --database primary
```

```
Database: primary
  Applied:  0
  Pending:  1
  Status:   pending
```

### Rollback Mechanics

- Rollback always executes the `-- rollback` section of the file; never auto-generates reverse SQL
- `--count` controls how many migrations to roll back (default: 1)
- Rollbacks are also lock-protected
- After rollback, the migration is considered "pending" again and can be re-applied

## Step 3: Re-apply

```bash
$ dbwarden migrate --database primary
```

Re-applies the migration. Since rollback removed the tracking record, the migration runs again.

## Step 4: Downgrade to a Version

```bash
$ dbwarden downgrade --to 0000 --database primary
```

`downgrade` is a bulk rollback: it rolls back all migrations down to (but not including) the target version. `--to 0000` rolls back everything.

```
[DBWarden] Rolling back primary__0001_create_core_tables...
[DBWarden] Downgrade complete. At version: 0000
```

### migrate vs rollback vs downgrade

| Command | What it does | Safe to run twice? |
|---------|-------------|-------------------|
| `migrate` | Applies pending migrations | Yes (idempotent) |
| `rollback` | Reverses the last N applied migrations | Yes (tracks what's applied) |
| `downgrade` | Rolls back to a specific target version | Yes |

## Step 5: Re-apply All

```bash
$ dbwarden migrate --database primary
$ dbwarden status --database primary
```

After the final apply, status should show:

```
Database: primary
  Applied:  1
  Pending:  0
  Status:   up-to-date
```

## Step 6: Schema Validation

```bash
$ dbwarden check --database primary
```

`check` scans each migration file and classifies operations by safety level:

- **SAFE**: Adding a nullable column, creating an index
- **INFO**: Table comment changes
- **WARN**: Dropping a default, changing column type
- **CRITICAL**: Dropping a table or column, removing a NOT NULL

```
Checking migrations for 'primary'...
  primary__0001_create_core_tables:
    CREATE TABLE users           SAFE
    CREATE TABLE posts           SAFE
    CREATE TABLE products        SAFE
    CREATE TABLE tags            SAFE
    CREATE INDEX                 SAFE
    COMMENT ON TABLE             INFO
  Result: 5 SAFE, 1 INFO, 0 WARN, 0 CRITICAL
```

## Step 7: Database Connectivity Check

```bash
$ dbwarden check-db --database primary
```

`check-db` connects to the live database and reports its schema:

```
Database: primary
  Connection: OK
  Tables:
    users (6 columns)
    posts (5 columns)
    products (6 columns)
    tags (2 columns)
  Migration table: _dbwarden_migrations (present)
  Lock table: _dbwarden_lock (present)
```

This confirms the database is reachable and has the expected schema.

## Key Takeaways

- `migrate` applies pending SQL with lock protection and checksum recording
- `rollback` and `downgrade` give you precise control over reversal
- `status` and `history` are your windows into migration state
- `check` classifies each operation by safety before it runs
- `check-db` validates database connectivity and schema existence

## Related Documentation

- [`migrate` command](../commands/migrate.md)
- [`rollback` command](../commands/rollback.md)
- [`downgrade` command](../commands/downgrade.md)
- [`status` command](../commands/status.md)
- [`history` command](../commands/history.md)
- [`check` command](../commands/check.md)
- [`check-db` command](../commands/check-db.md)

## Next

[Section 4: Offline & CI Workflows](04-offline-ci.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/04-offline-ci/
========================================================================

# 4. Offline & CI Workflows

## What You'll Learn

- How to export model state to JSON for offline use
- How to generate migrations without a live database
- How to integrate this into CI/CD pipelines

## Prerequisites

- Completed [Section 3](03-apply-and-inspect.md) (migrations applied, models in sync)
- `examples/core/` project

## The Problem

In CI/CD pipelines, you often need to generate migrations as part of your build, but your CI runner may not have a database connection. DBWarden's offline mode solves this by serializing model state to a JSON file.

## Step 1: Export Model State

```bash
cd examples/core
bash scripts/04-offline-ci.sh
```

The key command:

```bash
$ dbwarden export-models --database primary
```

This connects to the live database, introspects the current schema, and writes a JSON file to `.dbwarden/model_state.json`:

```json
{
  "version": "1.0",
  "exported_at": "2025-01-15T10:30:00",
  "database": "primary",
  "tables": {
    "users": {
      "columns": {
        "id": {"type": "INTEGER", "nullable": false, "primary_key": true},
        "email": {"type": "VARCHAR(255)", "nullable": false, "unique": true},
        "username": {"type": "VARCHAR(100)", "nullable": false, "unique": true},
        "full_name": {"type": "VARCHAR(200)", "nullable": true},
        "is_active": {"type": "BOOLEAN", "nullable": true, "default": "1"},
        "created_at": {"type": "DATETIME", "nullable": true, "default": "CURRENT_TIMESTAMP"}
      },
      "indexes": [
        {"name": "ix_users_created_at", "columns": ["created_at"]}
      ],
      "checks": [],
      "comment": "Core user accounts"
    }
  }
}
```

This file becomes your source of truth for future diffs; no database required.

## Step 2: Commit the State File

```bash
git add .dbwarden/model_state.json
git commit -m "Update model state snapshot"
```

## Step 3: Generate Migrations Offline

On any machine (including CI without a database):

```bash
$ dbwarden make-migrations "offline schema change" --offline --database primary
```

The `--offline` flag tells DBWarden to:

1. Read the model state from `.dbwarden/model_state.json` instead of querying a live database
2. Introspect the current model definitions in your Python code
3. Diff the two and generate migration SQL
4. Write the migration file AND update the snapshot file

This means the snapshot is always in sync after each generation.

## CI/CD Integration

In a GitHub Actions workflow:

```yaml
jobs:
  generate-migrations:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: uv add dbwarden sqlalchemy

      # Generate migrations using the committed state file
      - name: Check for new migrations
        run: dbwarden make-migrations "ci change" --offline --database primary

      # Commit any newly generated migrations
      - uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: "Auto-generate migration"
```

The full CI pipeline can then `dbwarden migrate` against staging/production using the generated SQL files.

## Key Takeaways

- `export-models` serializes the current database schema to JSON
- `make-migrations --offline` generates migrations using the snapshot instead of a live database
- Offline mode enables migration generation in CI without database access
- The snapshot file should be committed and kept in sync

## Related Documentation

- [CI/CD Patterns](../advanced/ci-cd-patterns.md)
- [`export-models` command](../cli-reference.md) (see CLI reference)

## Next

[Section 5: Schema Inspection](05-schema-inspection.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/05-schema-inspection/
========================================================================

# 5. Schema Inspection

Schema inspection allows you to compare your SQLAlchemy model definitions against the live database, capture DDL snapshots of individual tables, and reverse-engineer models from an existing database.

For complete documentation see the [`diff`](../commands/diff.md), [`snapshot`](../commands/snapshot.md), and [`generate-models`](../commands/generate-models.md) command references.

## What You'll Learn

- How to diff models against the live database
- How to capture DDL snapshots of individual tables
- How to reverse-engineer models from a live database

## Prerequisites

- Completed [Section 3](03-apply-and-inspect.md) (migrations applied)
- `examples/core/` project

## Step 1: Diff Models vs Database

```bash
cd examples/core
bash scripts/05-schema-inspection.sh
```

The key command:

```bash
$ dbwarden diff --database primary
```

`diff` compares your SQLAlchemy model definitions against the current database schema and reports any discrepancies:

```
No differences found between models and database.
```

If you add a column to a model without running `make-migrations`, `diff` would report a schema diff table:

```
Schema Diff
┌───────────┬───────┬────────┬──────────┐
│ Operation │ Table │ Target │ Severity │
├───────────┼───────┼────────┼──────────┤
│ add_column│ users │  bio   │ INFO     │
└───────────┴───────┴────────┴──────────┘
Total changes: 1
```

This is useful for catching drift before deployments.

## Step 2: Capture a DDL Snapshot

```bash
$ dbwarden snapshot users --database primary
```

The `snapshot` command captures the DDL for a specific table:

```sql
CREATE TABLE users (
    id INTEGER NOT NULL,
    email VARCHAR(255) NOT NULL,
    username VARCHAR(100) NOT NULL,
    full_name VARCHAR(200),
    is_active BOOLEAN DEFAULT true,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (id),
    UNIQUE (email),
    UNIQUE (username)
);

-- Indexes:
CREATE INDEX ix_users_created_at ON users (created_at);
```

Useful for:
- Documenting schema for code reviews
- Comparing schemas across environments
- Debugging migration issues

## Step 3: Reverse-Engineer Models

```bash
$ dbwarden generate-models -d primary --tables users,posts
```

This connects to the live database and generates SQLAlchemy model code:

```python
from sqlalchemy import Integer, String, Boolean, DateTime, Text, ForeignKey
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), nullable=False, unique=True)
    username: Mapped[str] = mapped_column(String(100), nullable=False, unique=True)
    full_name: Mapped[str | None] = mapped_column(String(200), nullable=True)
    is_active: Mapped[bool] = mapped_column(Boolean, default=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.now(UTC))


class Post(Base):
    __tablename__ = "posts"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String(255), nullable=False)
    body: Mapped[str] = mapped_column(Text, nullable=False)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False)
```

This is the fastest way to bootstrap models from an existing database. You can review and annotate the output with `class Meta` afterward.

Options:
- `--tables users,posts`: limit to specific tables
- `--exclude-tables`: exclude tables by pattern
- `--single-file`: output all models in one file
- `--output ./models/`: write to a directory instead of stdout

## Key Takeaways

- `diff` detects drift between models and the live database
- `snapshot` captures table DDL for documentation or debugging
- `generate-models` reverse-engineers live tables into SQLAlchemy model code
- These three commands form your schema inspection toolkit

## Related Documentation

- [`diff` command](../commands/diff.md)
- [`snapshot` command](../commands/snapshot.md)
- [`generate-models` command](../commands/generate-models.md)
- [SQLAlchemy Models Reference](../models.md)

## Next

[Section 6: Safety & Impact Analysis](06-safety-impact.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/06-safety-impact/
========================================================================

# 6. Safety & Impact Analysis

Schema changes are the highest-risk operation in most deployments. Dropping a column that application code still references causes runtime errors. Changing a column type can break queries. DBWarden provides two tools to detect these issues before deploy: `check` classifies every migration operation by danger level, and `check-impact` finds affected code references.

For complete documentation see the [`check`](../commands/check.md) and [`check-impact`](../cli-reference.md) command references.

## What You'll Learn

- How `dbwarden check` classifies operations by danger level
- How `dbwarden check-impact` finds code references affected by a migration
- How to detect destructive changes before they reach production

## Prerequisites

- Completed [Section 3](03-apply-and-inspect.md) (migrations applied)
- `examples/core/` project

## Step 1: Safety Check

```bash
cd examples/core
bash scripts/06-safety-impact.sh
```

The key command:

```bash
$ dbwarden check --database primary
```

This scans every migration file and classifies each SQL operation by safety level:

| Level | Meaning |
|-------|---------|
| **SAFE** | No data loss risk (add table, add nullable column, create index) |
| **INFO** | Metadata changes (comments, renames) |
| **WARN** | Potential impact (change column type, drop default) |
| **CRITICAL** | Destructive (drop table, drop column, remove NOT NULL) |

Output for our baseline migrations:

```
Safety Check - primary
┌──────────┬──────────────┬──────────┬────────┬─────────┬───────────────┐
│ Severity │ Change       │ Table    │ Column │ Message │ Required Flag │
├──────────┼──────────────┼──────────┼────────┼─────────┼───────────────┤
│ SAFE     │ create_table │ users    │        │         │               │
│ SAFE     │ create_table │ posts    │        │         │               │
│ SAFE     │ create_table │ products │        │         │               │
│ SAFE     │ create_table │ tags     │        │         │               │
│ SAFE     │ create_index │          │        │         │               │
│ INFO     │ comment_on   │ users    │        │         │               │
└──────────┴──────────────┴──────────┴────────┴─────────┴───────────────┘
```

A migration with a destructive change would show:

```
┌──────────┬──────────────┬───────┬──────────┬─────────┬───────────────┐
│ Severity │ Change       │ Table │ Column   │ Message │ Required Flag │
├──────────┼──────────────┼───────┼──────────┼─────────┼───────────────┤
│ CRITICAL │ drop_column  │ users │ username │         │               │
└──────────┴──────────────┴───────┴──────────┴─────────┴───────────────┘
```

This gives you a quick visual signal during code review: if a migration contains CRITICAL operations, it needs extra scrutiny.

## Step 2: Code Impact Analysis

```bash
$ dbwarden check-impact 0001 --database primary
```

`check-impact` scans your application code (not just migration files) for references that would be affected by a migration. It uses AST analysis with a grep fallback:

```
No impact detected
Scanned: .
```

A more realistic scenario with a destructive change:

```
Migration: 0002_drop_username
Impact detected: 1 operation(s) affect code

drop_column on users.username
  References: 2
    app/routes/users.py:34  attribute_access
      .username
    app/templates/profile.jinja2:12  grep
      user.username
```

The scan finds each reference, identifies the access pattern (attribute access in Python, grep match in templates), and reports the file and line number.

### How It Works

1. Reads the migration's plan file and parses the schema changes
2. Identifies schema changes (DROP COLUMN, ALTER COLUMN TYPE, etc.)
3. Scans `.py` files using Python's `ast` module for attribute access patterns
4. Falls back to grep for non-Python files (templates, configs, etc.)
5. Reports all references grouped by change type

### Flags

- `--scan-path app/`: limit scanning to a specific directory (default: project root)
- `--deep`: also scan dependencies (imported packages)
- `--out json`: output as JSON for CI processing
- `--verbose`: show scan progress

## Pre-Deploy Workflow

Combine both tools for a safe deploy sequence:

```bash
# 1. Check migration safety
$ dbwarden check --database primary

# 2. Check code impact
$ dbwarden check-impact 0042 --database primary

# 3. Only proceed if no unexpected CRITICAL or WARN items
$ dbwarden migrate --database primary
```

In CI:

```yaml
- name: Safety check
  run: dbwarden check --database primary
- name: Impact analysis
  run: dbwarden check-impact 0042 --database primary
- name: Apply (only if previous steps succeeded)
  run: dbwarden migrate --database primary
```

## Key Takeaways

- `check` classifies every migration operation by safety level using a severity table
- `check-impact` finds code references affected by a migration using AST + grep
- Together they catch breaking changes before deploy
- CRITICAL operations aren't blocked; they're flagged for human review
- Use `--out json` for CI integration

## Related Documentation

- [`check` command](../commands/check.md)
- [`check-impact` command](../cli-reference.md) (see CLI reference)
- [Safe Deployment](../advanced/safe-deployment.md)

## Next

[Section 7: Seeds](07-seeds.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/07-seeds/
========================================================================

# 7. Seeds

## What You'll Learn

- How to define code seeds using the `Seed` base class
- How to create and apply file-based SQL/Python seeds (legacy)
- How to list, apply, and roll back seeds
- How to auto-apply seeds after migrations

## Prerequisites

- Completed [Section 3](03-apply-and-inspect.md) (migrations applied, tables exist)
- `examples/core/` project
- Seed plugin installed: `dbwarden plugin add dbwarden-seeds`

## Step 1: Define a Code Seed

Code seeds are the recommended way to seed data. They live alongside your models and keep seed logic close to the schema it populates.

Create a seed file in your models directory (e.g. `models/seeds.py`):

```python
from dbwarden.seed import Seed

class CountrySeed(Seed):
    __seed_database__ = "primary"
    __seed_description__ = "initial countries"
    __seed_on_conflict__ = "update"
    __seed_conflict_columns__ = ["code"]

    model = Country
    rows = [
        Country(code="UY", name="Uruguay"),
        Country(code="AR", name="Argentina"),
    ]
```

Notice:

- **No `version`**: versions are auto-assigned (`C0001`, `C0002`, ...)
- **Model instances** in `rows`: your IDE gives full autocompletion on column names
- **Keyword arguments required**: SQLAlchemy 2.0's `DeclarativeBase` does not accept positional args; always use `Model(col=val)` syntax
- **`__seed_database__`**: route the seed to the correct database

### Logic-Based Seeds

Define a `generate(session)` method for programmatic data:

```python
class PermissionSeed(Seed):
    __seed_database__ = "primary"
    __seed_description__ = "load permissions"

    model = Permission

    @staticmethod
    def generate(session):
        for resource in ["users", "orders"]:
            for action in ["read", "write", "delete"]:
                session.add(Permission(name=f"{resource}:{action}"))
```

## Step 2: Apply Seeds

```bash
$ dbwarden seed apply --database primary
```

Output:

```
Applying code seed C0001: initial countries
```

Code seeds and file seeds are both discovered and applied together. Each seed version is tracked in `_dbwarden_seeds` and can only be applied once until rolled back.

## Step 3: List Applied Seeds

```bash
$ dbwarden seed list --database primary
```

Output:

```
Seeds for database 'primary':
  C0001  initial countries                 applied  2025-01-15 10:30:00  (code seed)
```

## Step 4: Auto-Apply Seeds After Migrations

Configure seeds to be applied automatically after `dbwarden migrate`:

```python
database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    model_paths=["models"],
    auto_apply_seeds=True,
)
```

Now running `dbwarden migrate` will also apply any pending seeds.

Or apply seeds once without changing config:

```bash
$ dbwarden migrate --apply-seeds
```

## Step 5: Traditional File Seeds (Legacy)

For complex multi-statement SQL, you can still use file-based seeds.

### Create a SQL Seed

```bash
$ dbwarden seed create "initial admin users" --database primary
```

This creates `seeds/V0001__initial_admin_users.sql`. Fill it with data:

```sql
INSERT INTO users (email, username, full_name, is_active, created_at)
VALUES ('admin@example.com', 'admin', 'Admin User', 1, CURRENT_TIMESTAMP);

INSERT INTO users (email, username, full_name, is_active, created_at)
VALUES ('moderator@example.com', 'moderator', 'Moderator User', 1, CURRENT_TIMESTAMP);
```

### Apply and List

```bash
$ dbwarden seed apply --database primary
$ dbwarden seed list --database primary
```

Output:

```
Seeds for database 'primary':
  C0001  initial countries                 applied  2025-01-15 10:30:00  (code seed)
  V0001  initial_admin_users              applied  2025-01-15 10:31:00
```

### Python File Seeds

```bash
$ dbwarden seed create "generate sample data" --database primary --type python
```

Creates `seeds/V0002__generate_sample_data.py` with a `seed(connection, session)` function.

## Step 6: Roll Back a Seed

```bash
$ dbwarden seed rollback --database primary --count 1
```

Seed rollback removes the tracking record, allowing the seed to be re-applied. It does **not** reverse the data changes; that is your responsibility if needed.

After rollback:

```
Seeds for database 'primary':
  C0001  initial countries                 applied  2025-01-15 10:30:00  (code seed)
  V0001  initial_admin_users              pending
```

## Step 7: Prune Orphaned Records

Remove tracking records for seed files that no longer exist on disk:

```bash
$ dbwarden seed list --prune
```

## Key Takeaways

- **Code seeds (`Seed` base class) are the recommended approach**: no manual versions, full IDE support, stays in sync with models
- `auto_apply_seeds: True` or `dbwarden migrate --apply-seeds` applies seeds automatically after migrations
- File seeds (`.sql` / `.py`) are still available for complex multi-statement SQL
- `seed list --prune` cleans up orphaned tracking records
- Seed rollback removes the tracking record; it does not undo data

## Related Documentation

- [Seeds Reference](../seeds.md)
- [`seed` command](../commands/seed.md)
- [CLI Reference: Seed Management](../cli-reference.md#seed-management)

## Next

[Section 8: Multi-Database](08-multi-database.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/08-multi-database/
========================================================================

# 8. Multi-Database & Configuration

DBWarden supports managing multiple databases in a single project; each with its own migration directory, lock, tracking table, and model paths. You can mix PostgreSQL, MySQL, and ClickHouse backends in the same codebase.

For complete documentation see the [Multi-Database Configuration](../configuration/multi-database.md) reference.

## What You'll Learn

- How to configure multiple databases in one project
- How to target specific databases with CLI flags
- How to manage PostgreSQL + MySQL + ClickHouse in the same codebase
- How to use `dbwarden settings` for runtime configuration changes

## Prerequisites

- Docker (for PostgreSQL, MySQL, and ClickHouse containers)
- `examples/multi-database/` directory

## Scenario

A project with three databases:

- **primary** (PostgreSQL): transactional user data
- **legacy** (MySQL): legacy CRM and reporting data
- **analytics** (ClickHouse): page view events for analysis

## Step 1: The Configuration

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/primary",
    database_url_async="postgresql+asyncpg://user:password@localhost:5432/primary",
    model_paths=["app.models.primary"],
)

legacy = database_config(
    database_name="legacy",
    database_type="mysql",
    database_url_sync="mysql+pymysql://user:password@localhost:3306/legacy",
    model_paths=["app.models.legacy"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://localhost:8123/analytics",
    model_paths=["app.models.analytics"],
)
```

Key rules:
- Exactly one database must have `default=True` (used when `--database` is omitted)
- Each database must have separate `model_paths` (no overlap by default)
- Each database gets its own migration directory under `migrations/`
- MySQL models use `MyTableMeta` / `MyColumnMeta` for engine, charset, and column metadata

## Step 2: Start the Databases

```bash
cd examples/multi-database
docker compose up -d
```

## Step 3: Initialize and Migrate

```bash
$ dbwarden init
$ dbwarden migrate --all
```

This applies migrations to both databases in sequence. Each has its own lock, its own tracking table, and its own migration history.

## Step 4: Target a Specific Database

```bash
# Generate migrations for primary only
$ dbwarden make-migrations "add user table" --database primary

# Apply to analytics only
$ dbwarden migrate --database analytics

# Check status of one database
$ dbwarden status --database primary
```

## Step 5: Check Status of All Databases

```bash
$ dbwarden status --all
```

Output:

```
Database: primary
  Applied:  1
  Pending:  0
  Status:   up-to-date

Database: analytics
  Applied:  1
  Pending:  0
  Status:   up-to-date
```

## Step 6: Using `dbwarden settings`

The settings commands allow runtime configuration changes without editing `dbwarden.py` directly:

```bash
# View current configuration
$ dbwarden settings show --all

# Set a default database
$ dbwarden settings default-database primary

# Add a new database entry
$ dbwarden settings database-add reporting postgresql://localhost:5432/reporting \
    --type postgresql \
    --model-path app.models.reporting

# Or add a MySQL database
$ dbwarden settings database-add legacy mysql+pymysql://localhost:3306/legacy \
    --type mysql \
    --model-path app.models.legacy

# Remove a database
$ dbwarden settings database-remove reporting

# Rename a database
$ dbwarden settings database-rename analytics analytics_v2
```

Settings commands modify the `dbwarden.py` file directly using AST-based mutation. The changes are permanent and committed to version control.

## Step 7: Dev Mode with Multiple Databases

Each database can independently configure dev mode:

```python
primary = database_config(
    database_name="primary",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/primary",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_primary.db",
    model_paths=["app.models.primary"],
)

legacy = database_config(
    database_name="legacy",
    database_type="mysql",
    database_url_sync="mysql+pymysql://localhost:3306/legacy",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_legacy.db",
    model_paths=["app.models.legacy"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://localhost:8123/analytics",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_analytics.db",
    model_paths=["app.models.analytics"],
)
```

```bash
# Dev mode for all databases
$ dbwarden --dev migrate --all

# Dev mode for a specific database
$ dbwarden --dev migrate --database analytics
```

## Step 8: Legacy Database with MySQL Metadata

The legacy MySQL database uses `MyTableMeta` and `MyColumnMeta` for MySQL-specific features. Here is a sample model from `app/models/legacy/customer.py`:

```python
from sqlalchemy import Integer, String, TIMESTAMP, Text
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta, MyColumnMeta, my

class Customer(Base):
    __tablename__ = "customers"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    name: Mapped[str] = mapped_column(String(200), nullable=False)
    notes: Mapped[str | None] = mapped_column(Text)
    created_at: Mapped[str] = mapped_column(TIMESTAMP)

    class Meta(MyTableMeta):
        my_engine = "InnoDB"
        my_charset = "utf8mb4"
        my_collate = "utf8mb4_unicode_ci"
        comment = "Legacy CRM customers"

        class id(MyColumnMeta):
            my = my.field(unsigned=True)

        class created_at(MyColumnMeta):
            my = my.field(on_update="CURRENT_TIMESTAMP")
```

Migrations for the legacy database work identically to other databases:

```bash
# Generate migration for MySQL legacy database
$ dbwarden make-migrations "add customer table" --database legacy

# Apply to legacy only
$ dbwarden migrate --database legacy
```

The generated DDL will target MySQL-native syntax:

```sql
CREATE TABLE IF NOT EXISTS customers (
    id INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
    name VARCHAR(200) NOT NULL,
    notes TEXT,
    created_at TIMESTAMP NOT NULL ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT='Legacy CRM customers';
```

## Key Takeaways

- Multiple `database_config()` calls create independent database targets
- Each database has its own migration directory, lock, and history
- `--database` targets a specific database; `--all` targets every database
- `default=True` controls which database is used when `--database` is omitted
- `settings` commands modify `dbwarden.py` at runtime without manual editing
- Dev mode can be configured independently per database

## Related Documentation

- [Multi-Database Configuration](../configuration/multi-database.md)
- [Dev Mode](../configuration/dev-mode.md)
- [Settings Command](../commands/settings.md)

## Next

[Section 9: FastAPI Integration](09-fastapi-integration.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/09-fastapi-integration/
========================================================================

# 9. FastAPI Integration

## What You'll Learn

- How to wire DBWarden into a FastAPI application lifecycle
- How to use `primary.async_session` as a dependency injection
- How to expose health check and migration endpoints
- How to validate schema on startup

## Prerequisites

- Docker (for PostgreSQL)
- `examples/fastapi-app/` directory

## Step 1: Configuration with Session Handles

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/myapp",
    database_url_async="postgresql+asyncpg://user:password@localhost:5432/myapp",
    model_paths=["app.models"],
)
```

The `primary` object is a `DatabaseHandle`. It exposes `primary.async_session` and `primary.sync_session` as FastAPI-compatible dependency annotations; no separate dependency module needed.

## Step 2: Lifespan Hook

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from dbwarden_fastapi import dbwarden_lifespan


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with dbwarden_lifespan(app, mode="check"):
        yield


app = FastAPI(lifespan=lifespan)
```

`dbwarden_lifespan` runs on every startup:

1. **Schema validation** (mode `"check"`): verifies all pending migrations exist and the database is in a known state
2. **Readiness gate**: the app won't accept traffic until validation passes
3. **Connection pool warmup**: pre-connects to the database
4. **On shutdown**: disposes all engine pools and ClickHouse clients

Available modes:
- `"check"`: validate schema, fail on pending migrations (recommended for production)
- `"migrate"`: apply pending migrations automatically on startup
- `"skip"`: no startup checks

## Step 3: Session Dependency in Routes

```python
from config import primary
from app.models import User
from app.schemas import UserResponse


@router.get("/{user_id}", response_model=UserResponse)
async def get_user(user_id: int, session: primary.async_session):
    result = await session.execute(
        select(User).where(User.id == user_id)
    )
    user = result.scalar_one_or_none()
    if not user:
        raise HTTPException(status_code=404, detail="User not found")
    return user
```

`primary.async_session` is a type alias for `Annotated[AsyncSession, Depends(...)]`. FastAPI resolves it to an actual database session using the engine configured in `database_config()`.

The session is automatically:
- Opened when the route handler starts
- Committed (or rolled back on exception) when the handler finishes
- Closed and returned to the pool

## Step 4: Health Endpoints

```python
from dbwarden_fastapi import DBWardenHealthRouter

app.include_router(DBWardenHealthRouter(), prefix="/health")
```

This adds:

| Endpoint | Description |
|----------|-------------|
| `GET /health/` | Overall health status across all databases |
| `GET /health/liveness` | Is the app alive? (lightweight) |
| `GET /health/readiness` | Is the app ready for traffic? (checks DB connectivity) |
| `GET /health/{database_name}` | Per-database health status |

Sample response:

```json
{
  "status": "ok",
  "databases": {
    "primary": {
      "status": "ok",
      "connected": true,
      "pending_migrations": 0,
      "applied_migrations": 5,
      "lock_active": false
    }
  }
}
```

## Step 5: Migration Endpoints

```python
from dbwarden_fastapi import DBWardenRouter

app.include_router(DBWardenRouter(), prefix="/db")
```

| Endpoint | Description |
|----------|-------------|
| `GET /db/status` | JSON representation of `dbwarden status` |
| `POST /db/migrate` | Trigger migration execution at runtime |

These endpoints are useful for management UIs or automated deployment tooling.

## Step 6: The Complete App

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from dbwarden_fastapi import (
    DBWardenHealthRouter,
    DBWardenRouter,
    dbwarden_lifespan,
)
from app.routes import users


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with dbwarden_lifespan(app, mode="check"):
        yield


app = FastAPI(
    title="DBWarden FastAPI Example",
    lifespan=lifespan,
)

app.include_router(users.router, prefix="/api/v1")
app.include_router(DBWardenHealthRouter(), prefix="/health")
app.include_router(DBWardenRouter(), prefix="/db")
```

## Step 7: Run and Test

```bash
# Install dependencies
uv add dbwarden sqlalchemy fastapi uvicorn asyncpg
dbwarden plugin add dbwarden-fastapi

# Start PostgreSQL
docker run -d --name pg -e POSTGRES_USER=user \
    -e POSTGRES_PASSWORD=password -e POSTGRES_DB=myapp \
    -p 5432:5432 postgres:16

# Initialize and migrate
$ dbwarden init
$ dbwarden make-migrations "create users table"
$ dbwarden migrate

# Start the app
uvicorn app.main:app --reload
```

```bash
# Health check
curl http://localhost:8000/health/

# Create a user
curl -X POST http://localhost:8000/api/v1/users/ \
    -H "Content-Type: application/json" \
    -d '{"email": "alice@example.com", "username": "alice"}'

# Migration status
curl http://localhost:8000/db/status
```

## Key Takeaways

- `database_config()` returns a `DatabaseHandle` with built-in FastAPI dependencies
- `dbwarden_lifespan` integrates schema validation into the app lifecycle
- `primary.async_session` works directly as a route parameter type annotation
- `DBWardenHealthRouter` exposes liveness, readiness, and per-database health
- `DBWardenRouter` exposes migration status and execution as HTTP endpoints

## Related Documentation

FastAPI support is not part of core. It ships as the `dbwarden-fastapi` plugin, which owns the session dependencies, health endpoints, migration routes, and metrics middleware shown above. Install it with `dbwarden plugin add dbwarden-fastapi`.

Full reference and tutorials live in the plugin's repository: [dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi).

## Next

[Section 10: Auto Schemas](10-auto-schemas.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/10-auto-schemas/
========================================================================

# 10. Auto-Generated Pydantic Schemas

## What You'll Learn

- How `@auto_schema` generates Pydantic schemas from model annotations
- How `public = False` controls field visibility
- How `CreateSchema`, `UpdateSchema`, and `PublicSchema` differ

## Prerequisites

- `dbwarden-fastapi` plugin installed: `dbwarden plugin add dbwarden-fastapi`
- `uv add dbwarden sqlalchemy`

## The Problem

In FastAPI applications, you typically define SQLAlchemy models for the database and Pydantic schemas for the API. This means maintaining two parallel definitions for every entity: the ORM layer and the API layer. They drift apart over time.

DBWarden's `@auto_schema` eliminates this duplication by deriving Pydantic schemas directly from model annotations.

## Step 1: Define a Model with @auto_schema

```python
from dbwarden.databases import TableMeta
from dbwarden_fastapi import auto_schema


@auto_schema
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    username: Mapped[str] = mapped_column(String(100), unique=True, nullable=False)
    full_name: Mapped[str | None] = mapped_column(String(200), nullable=True)
    password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
    is_active: Mapped[bool] = mapped_column(Boolean, default=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("CURRENT_TIMESTAMP"))

    class Meta(TableMeta):
        comment = "User accounts with auto-generated Pydantic schemas"

        class email:
            comment = "Login email"

        class password_hash:
            public = False  # Excluded from PublicSchema
```

## Step 2: What Gets Generated

The decorator creates four schema classes on the model:

### `User.CreateSchema`

Used for POST requests: includes all fields that the client should provide. Server-defaulted fields (like auto-increment `id`) are excluded.

```python
create = User.CreateSchema(
    email="alice@example.com",
    username="alice",
    password_hash="secret",
    full_name="Alice Smith",
    is_active=True,
)
```

### `User.UpdateSchema`

All fields optional: used for PATCH requests.

```python
update = User.UpdateSchema(full_name="Alice Johnson")
```

### `User.PublicSchema`

Excludes fields marked `public = False`. Perfect for API responses where you never want to leak `password_hash`.

```python
user = User(
    id=1,
    email="alice@example.com",
    username="alice",
    password_hash="secret",
    is_active=True,
)
public = User.PublicSchema.model_validate(user)
print(public.model_dump())
# {
#     "id": 1,
#     "email": "alice@example.com",
#     "username": "alice",
#     "full_name": None,
#     "is_active": True,
#     "created_at": ...,
# }
# password_hash is NOT included
```

### `User.Schema`

All mapped columns, including those marked `public = False`.

## Step 3: Controlling Visibility

| Technique | Effect |
|-----------|--------|
| `class Meta: class field: public = False` | Excluded from PublicSchema |
| Field name starting with `_` | Implicitly `public = False` |
| `SchemaConfig(exclude_public=["field"])` | Excluded from PublicSchema |
| `SchemaConfig(exclude_create=["field"])` | Excluded from CreateSchema |

## Step 4: Customizing Schema Generation

```python
from dbwarden_fastapi import auto_schema, SchemaConfig


@auto_schema(config=SchemaConfig(
    exclude_public=["internal_note"],
    exclude_create=["created_at"],
    field_overrides={
        "email": EmailStr,
    },
))
class User(Base):
    ...
```

`SchemaConfig` supports:

| Option | Description |
|--------|-------------|
| `exclude_always` | Excluded from all schemas |
| `exclude_create` | Excluded from CreateSchema only |
| `exclude_update` | Excluded from UpdateSchema only |
| `exclude_public` | Excluded from PublicSchema only |
| `field_overrides` | Override Pydantic field types |
| `required_always` | Fields always required |
| `optional_always` | Fields always optional |

## Key Takeaways

- `@auto_schema` generates CreateSchema, UpdateSchema, PublicSchema, and Schema
- `public = False` in `class Meta` controls API visibility; no manual filtering in routes
- Fields starting with `_` are implicitly non-public
- Use `User.PublicSchema.model_validate(instance)` to convert model instances to API responses
- Customize with `SchemaConfig` for advanced use cases

## Related Documentation

- [Modeling Guide: Auto-Generated Schemas](../getting-started/modeling.md#auto-generated-pydantic-schemas-with-auto_schema)
- [SQLAlchemy Models Reference](../models.md)

## Next

[Section 11: Observability](11-observability.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/11-observability/
========================================================================

# 11. Observability

## What You'll Learn

- How to enable Prometheus metrics for DBWarden
- How to use structured JSON logging
- How to add query tracing middleware to FastAPI
- How to monitor connection pool health

## Prerequisites

- `examples/observability/` directory
- Docker (for optional Prometheus + Grafana)

## Step 1: Enable Prometheus Metrics

Install with metrics support:

```bash
uv add "dbwarden[metrics]"
dbwarden plugin add dbwarden-fastapi
```

DBWarden exposes six Prometheus metric families:

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `dbwarden_migrations_total` | Counter | `database`, `status` | Migration count |
| `dbwarden_migration_duration_seconds` | Histogram | `database` | Execution time |
| `dbwarden_schema_version` | Gauge | `database` | Current version |
| `dbwarden_pending_migrations` | Gauge | `database` | Pending count |
| `dbwarden_errors_total` | Counter | `database`, `error_type` | Error count |
| `dbwarden_seed_version` | Gauge | `database` | Current seed version |

## Step 2: Add Metrics to FastAPI

```python
from dbwarden_fastapi import MetricsMiddleware, MetricsRouter

# Middleware captures request duration and counts
app.add_middleware(MetricsMiddleware)

# Router exposes /metrics endpoint
app.include_router(MetricsRouter(), prefix="/metrics")
```

```bash
curl http://localhost:8000/metrics
```

Output:

```
# HELP dbwarden_migrations_total Total number of migrations
# TYPE dbwarden_migrations_total counter
dbwarden_migrations_total{database="primary",status="applied"} 5

# HELP dbwarden_pending_migrations Number of pending migrations
# TYPE dbwarden_pending_migrations gauge
dbwarden_pending_migrations{database="primary"} 0
```

## Step 3: Structured Logging

```python
import os
os.environ["DBWARDEN_LOG_JSON"] = "1"
```

Or via environment variable:

```bash
DBWARDEN_LOG_JSON=1 uvicorn app.main:app
```

This switches from colored human-readable output to JSON:

```json
{"timestamp": "2025-01-15T10:30:00Z", "level": "INFO", "event": "migration_applied", "database": "primary", "duration_ms": 42, "version": "0005"}
```

JSON logs are easier to ingest into ELK, Datadog, or other log aggregators.

## Step 4: Query Tracing

```python
from dbwarden_fastapi import QueryTracingMiddleware

app.add_middleware(QueryTracingMiddleware)
```

This logs every SQL query with its duration:

```json
{"event": "query", "duration_ms": 3, "database": "primary", "statement": "SELECT ..."}
```

Useful for:
- Identifying slow queries in development
- Building a query performance baseline
- Debugging N+1 query patterns

## Step 5: Pool Metrics Collector

```python
from dbwarden_fastapi import PoolMetricsCollector
```

This monitors SQLAlchemy connection pool health and exposes:

- Pool size (current/total)
- Connections in use
- Connections overflow
- Pool timeouts

## Step 6: Full Setup

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from dbwarden_fastapi import (
    DBWardenHealthRouter,
    dbwarden_lifespan,
    MetricsMiddleware,
    MetricsRouter,
    QueryTracingMiddleware,
)


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with dbwarden_lifespan(app, mode="check"):
        yield


app = FastAPI(
    title="DBWarden Observability Example",
    lifespan=lifespan,
)

app.add_middleware(QueryTracingMiddleware)
app.add_middleware(MetricsMiddleware)
app.include_router(MetricsRouter(), prefix="/metrics")
app.include_router(DBWardenHealthRouter(), prefix="/health")
```

## Step 7: Prometheus + Grafana (Optional)

```yaml
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
```

```bash
docker compose up -d
```

- Prometheus: http://localhost:9090
- Grafana: http://localhost:3000

In Grafana, add Prometheus data source (`http://prometheus:9090`) and create dashboards using the `dbwarden_*` metrics.

## Key Takeaways

- Metrics are opt-in via `uv add "dbwarden[metrics]"`
- Six metric families cover migration, schema, and error tracking
- `DBWARDEN_LOG_JSON=1` switches to structured JSON logging
- `QueryTracingMiddleware` logs every SQL query with duration
- `PoolMetricsCollector` monitors connection pool health
- Metrics are compatible with standard Prometheus + Grafana setup

## Related Documentation

- [Observability Guide](../observability.md)

The FastAPI metrics middleware and routers are part of the `dbwarden-fastapi` plugin, documented in its own repository: [dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/cookbook/
========================================================================

# Cookbook & Examples

Practical, runnable examples that walk through the entire DBWarden workflow: from project setup through advanced observability patterns.

## How to Use

Each cookbook section links to code under the [`examples/`](https://github.com/emiliano-gandini-outeda/DBWarden/tree/main/examples) directory. The **core examples** (sections 1-7) use SQLite and require only `uv add dbwarden`. Advanced examples may need Docker for PostgreSQL, ClickHouse, or Prometheus.

```
examples/
├── core/                 # Sections 1-7: progressive SQL workflow
├── multi-database/       # Section 8
├── fastapi-app/          # Section 9
├── fastapi-app/          # Sections 9-10 (FastAPI + auto-schema examples)
└── observability/        # Section 11
```

## Sections

| # | Section | What You'll Learn | Example Dir |
|---|---------|-------------------|-------------|
| 1 | [Project Setup](01-project-setup.md) | `init`, `config`, understanding `database_config()` | `examples/core/` |
| 2 | [Models & Migrations](02-models-and-migrations.md) | Model definitions, `make-migrations`, `new`, `make-rollback` | `examples/core/` |
| 3 | [Apply & Inspect](03-apply-and-inspect.md) | `migrate`, `rollback`, `downgrade`, `history`, `status`, `check`, `check-db` | `examples/core/` |
| 4 | [Offline & CI](04-offline-ci.md) | `export-models`, `make-migrations --offline` | `examples/core/` |
| 5 | [Schema Inspection](05-schema-inspection.md) | `diff`, `snapshot`, `generate-models` | `examples/core/` |
| 6 | [Safety & Impact](06-safety-impact.md) | `check`, `check-impact`, destructive change detection | `examples/core/` |
| 7 | [Seeds](07-seeds.md) | `seed create/apply/rollback/list`, SQL seeds, `@seed_data` | `examples/core/` |
| 8 | [Multi-Database](08-multi-database.md) | Multiple `database_config()`, PG + ClickHouse, `--all` flag | `examples/multi-database/` |
| 9 | [FastAPI Integration](09-fastapi-integration.md) | Lifespan hooks, health endpoints, session DI, migration endpoints | `examples/fastapi-app/` |
| 10 | [Auto Schemas](10-auto-schemas.md) | `@auto_schema`, `CreateSchema`, `UpdateSchema`, `PublicSchema` (requires `dbwarden-fastapi` plugin) | `examples/fastapi-app/` |
| 11 | [Observability](11-observability.md) | Prometheus metrics, structured logging, query tracing | `examples/observability/` |

## Quick Start (Core)

```bash
cd examples/core
uv add dbwarden sqlalchemy
bash scripts/01-setup.sh
bash scripts/02-models-migrations.sh
bash scripts/03-apply-inspect.sh
```

Each section in the cookbook explains what these commands do, what SQL they produce, and why it matters.

## Database-Specific Examples

The core examples use SQLite for zero-dependency setup. For production, DBWarden fully supports PostgreSQL, MySQL, and ClickHouse; each with its own deep-dive guide and dedicated example patterns.

### PostgreSQL

PostgreSQL is a first-class backend with full round-trip support (read and write schema). The FastAPI integration example in [Section 9](09-fastapi-integration.md) uses PostgreSQL, and [Section 8](08-multi-database.md) shows PostgreSQL + ClickHouse together.

For the complete reference on PostgreSQL-specific metadata (identity columns, collation, compression, generated columns, tablespace, inheritance, exclusion constraints, deferrable FKs, advanced index options), see the [PostgreSQL Deep Dive](../databases/postgresql/index.md).

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/myapp",
    database_url_async="postgresql+asyncpg://user:password@localhost:5432/myapp",
    model_paths=["app.models"],
)
```

### MySQL / MariaDB

MySQL and MariaDB are first-class backends with full round-trip support. All MySQL-specific metadata (engine, charset, collation, row format, auto_increment, unsigned columns, ON UPDATE, column comments) is captured by the snapshot, diffed correctly, and emitted as valid DDL.

See the [MySQL Deep Dive](../databases/mysql.md) for the complete reference, including MySQL-specific model metadata via `class Meta(MyTableMeta)`.

```python
from dbwarden import database_config

legacy = database_config(
    database_name="legacy",
    database_type="mysql",
    database_url_sync="mysql+pymysql://user:password@localhost:3306/legacy",
    model_paths=["app.legacy_models"],
)
```

### ClickHouse

ClickHouse is supported with partial round-trip (read schema and auto-generate most DDL). DBWarden uses the ClickHouse HTTP client directly for DDL execution and supports full engine metadata via `class Meta(CHTableMeta)` with `ChEngineSpec`, `ProjectionSpec`, and `CHColumnMeta`.

See the [ClickHouse Deep Dive](../databases/clickhouse/index.md) for full details on materialized views, projections, dictionaries, replicated engines, and ClickHouse-specific metadata.

ClickHouse is typically configured alongside a transactional database (see [Section 8](08-multi-database.md) for a PostgreSQL + ClickHouse example).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/convergence-gate/
========================================================================

# Convergence Gate

The convergence gate is the strongest correctness check in DBWarden. It proves that the complete migration history can reproduce the schema declared by the current models.

The gate answers one question:

> If a new environment starts with an empty database and applies every migration, does the resulting schema exactly match the models in the repository?

If the answer is no, CI should fail.

## Definition

Convergence means:

```text
Empty database
  |
  v
Apply every migration
  |
  v
Extract live schema
  |
  v
Compare live schema with model spec
  |
  v
Pass only if there is zero drift
```

This is an end-to-end, model-history-consistency check. It does not only test the newest migration. It tests the entire path from an empty database to the current model state.

## Pipeline

The typical CI pipeline has four phases.

### 1. Spin Up an Ephemeral Database

CI starts a disposable database service. For PostgreSQL this is usually a container. For ClickHouse this can be a ClickHouse service container. The database should be empty at the start of the job.

The important property is isolation. The gate should never run against a shared developer database, because manual changes in that database would hide or create drift that is unrelated to the repository.

### 2. Apply the Full Migration History

Run the same command used in deployment:

```bash
dbwarden migrate --database primary
```

This applies versioned migrations and any repeatable migration behavior supported by the project configuration. The gate tests the migration files that will run in production, not a simplified test fixture.

### 3. Check the Resulting Schema

Run DBWarden's check command:

```bash
dbwarden check --database primary
```

The check compares the database state with the model state. If the command reports differences, CI fails. Some teams also run `dbwarden diff` in table or JSON mode for diagnostic output:

```bash
dbwarden diff --database primary --out table
dbwarden diff --database primary --out json
```

Use the diff output to see which object drifted. The gate should not ignore drift. If the models are correct, generate or write a migration. If the migration is correct, update the model.

### 4. Fail Fast on Drift

The final rule is simple:

```text
No drift -> merge allowed
Any drift -> merge blocked
```

The migration history and the models must agree.

## GitHub Actions Example

This example uses PostgreSQL. The same pattern applies to any backend that can run in CI.

```yaml
name: dbwarden-convergence

on:
  pull_request:
  push:
    branches: [main]

jobs:
  convergence:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: dbwarden
          POSTGRES_PASSWORD: dbwarden
          POSTGRES_DB: app
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    env:
      DATABASE_URL: postgresql://dbwarden:dbwarden@localhost:5432/app

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install project
        run: pip install -e .

      - name: Apply migrations
        run: dbwarden migrate --database primary

      - name: Check convergence
        run: dbwarden check --database primary

      - name: Print drift diagnostics on failure
        if: failure()
        run: dbwarden diff --database primary --out table
```

The `--health-cmd` option belongs to Docker service configuration. The DBWarden commands are the two important checks: `migrate` and `check`.

## Example Error Caught by the Gate

Assume a model declares an index:

```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(255), nullable=False)

    class Meta:
        indexes = [IndexSpec(name="idx_users_email", columns=["email"])]
```

But the migration only creates the table:

```sql
-- upgrade
CREATE TABLE users (
    id INTEGER NOT NULL PRIMARY KEY,
    email VARCHAR(255) NOT NULL
);

-- rollback
DROP TABLE IF EXISTS users CASCADE;
```

The migration applies successfully. A unit test that only checks syntax might pass. Production would have a table without the declared index.

The convergence gate catches this:

```text
Model state:
  users has idx_users_email

Extracted database state after migrations:
  users has no idx_users_email

Result:
  drift detected
```

The fix is to generate or add a migration that creates the missing index.

## Other Errors It Catches

### Leftover Column

A column is removed from the model, but no migration drops it. The gate applies all migrations, extracts the schema, and sees that the database still contains the old column.

### Reversed Migration

A migration accidentally renames `customer_id` to `client_id` while the model still declares `customer_id`. The migration applies, but the final schema disagrees with the model.

### Missing Backend Metadata

PostgreSQL table storage parameters, identity options, RLS policies, or ClickHouse engine settings are easy to miss in hand-written SQL. The gate compares the extracted backend-specific shape, not only the visible column list.

## Why This Is the Gold Standard

The convergence gate tests exactly the behavior that matters in production:

- It uses the real migration files.
- It uses the real backend SQL emitters.
- It uses a real database engine.
- It verifies the final schema against the models.
- It catches drift introduced anywhere in the history.

This is stronger than checking that a single generated migration looks plausible. A migration can be syntactically valid and still fail to converge.

## Relationship to Offline Integrity

Offline workflows are useful when CI cannot reach a database service. They let DBWarden generate migrations from a checked-in model state instead of a live database. That is deterministic, but it does not prove the SQL applies to a real engine.

A strong pipeline uses both:

```text
Offline integrity check
  |
  v
Generate or verify migration files deterministically
  |
  v
Convergence gate on a live database
  |
  v
Prove the SQL reproduces the model state
```

See [Offline Integrity](offline-integrity.md) for the first half of that workflow.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/deterministic-diff/
========================================================================

# Deterministic Diff

The deterministic diff is the reason DBWarden can generate stable, reviewable migrations. Given the same model state and the same database snapshot, DBWarden should produce the same operation list every time.

Without deterministic diffing, automated migrations become noisy. A pull request might alternate between two equivalent SQL forms. Reviewers would waste time checking fake changes instead of real schema changes.

## The Problem

Databases and model code often describe the same schema in different ways.

Examples:

- A model says `String(255)` while PostgreSQL reports `character varying(255)`.
- A model says `Boolean` while a database default is rendered as `false` or `FALSE`.
- A PostgreSQL identifier may appear quoted in SQL and unquoted in model metadata.
- ClickHouse engine parameters may be returned in a different order than they were declared.
- Whitespace in view definitions may differ while the query is logically unchanged.

If DBWarden compared raw strings directly, it would produce false diffs.

## Canonicalization

DBWarden handlers canonicalize specs before diffing. Each handler owns the rules for its object type.

The shape is:

```text
Model spec
  |
  v
ObjectHandler.canonicalize(model_spec)
  |
  v
Canonical model spec

Database snapshot
  |
  v
ObjectHandler.canonicalize(snapshot_spec)
  |
  v
Canonical snapshot spec

Canonical model spec + canonical snapshot spec
  |
  v
ObjectHandler.diff(...)
  |
  v
Typed operations
```

The important property is that equivalent states normalize to the same representation.

## Handler Contract

Backend handlers expose a small contract:

```text
extract(snapshot) -> raw backend state
model_spec_from_tables(models) -> raw model state
canonicalize(spec) -> stable comparable state
diff(snapshot_spec, model_spec) -> operations
emit(operation) -> SQL statements
```

The `canonicalize` step sits between raw state and diffing. It is the boundary where noisy representation differences are removed.

## PostgreSQL Normalization Examples

### Type Aliases

PostgreSQL may report type names differently than SQLAlchemy models declare them.

```text
Model declaration:
  String(255)

Database extraction:
  character varying(255)

Canonical form:
  varchar(255)
```

After canonicalization, DBWarden does not emit a type-change migration for this column.

### Boolean Defaults

Different sources may use different boolean spelling.

```text
Model default:
  true

Database default:
  TRUE

Canonical meaning:
  true
```

The diff should compare the normalized meaning, not the original spelling.

### Identifier Quoting

PostgreSQL accepts both quoted and unquoted identifiers, but quoted identifiers preserve case. DBWarden normalizes safe names while preserving names that require quotes.

```text
users.email
"users"."email"

Canonical object identity:
  users.email
```

The goal is not to remove all quotes from emitted SQL. The goal is to compare object identity consistently before SQL generation.

## ClickHouse Normalization Examples

### Engine Metadata

ClickHouse table metadata is not just the engine name. A `MergeTree` table can include partition keys, primary keys, order keys, sample keys, TTL expressions, and settings.

```text
Model:
  engine = MergeTree()
  order_by = ["event_date", "id"]
  settings = {"index_granularity": "8192"}

Snapshot:
  ENGINE = MergeTree()
  ORDER BY (event_date, id)
  SETTINGS index_granularity = 8192

Canonical form:
  engine family: MergeTree
  order_by: [event_date, id]
  settings.index_granularity: 8192
```

The canonical form lets DBWarden decide whether a real engine change occurred.

### Projection and Skip Index Specs

ClickHouse projections and skip indexes contain expressions. DBWarden normalizes the supported metadata fields so the diff sees stable names, expressions, types, and granularities.

```text
Projection model:
  name: by_date
  select: SELECT event_date, count() GROUP BY event_date

Snapshot projection:
  name: by_date
  query returned by system tables

Canonical comparison:
  same projection identity and same normalized query body
```

## Tricky Case Walkthrough

Consider a model column:

```python
name: Mapped[str] = mapped_column(String(255), nullable=False, default="hello")
```

PostgreSQL may report:

```sql
name character varying(255) NOT NULL DEFAULT 'hello'::character varying
```

The raw strings differ:

```text
String(255)
character varying(255)

"hello"
'hello'::character varying
```

The canonical comparison should reduce them to the same meaning:

```text
type: varchar(255)
nullable: false
default: 'hello'
```

Result:

```text
No operation emitted
```

If the model changes to `String(500)`, canonicalization no longer hides the difference:

```text
Canonical model type: varchar(500)
Canonical snapshot type: varchar(255)
Result: alter column type operation
```

## Why Determinism Matters

Determinism gives DBWarden four practical properties.

### Reviewability

The same input produces the same migration. Reviewers can trust that a migration is caused by a real schema change, not by formatter noise.

### Auditability

Generated SQL can be compared across CI runs. If the output changes, the inputs changed or the generator changed.

### No Flapping

Flapping happens when a tool alternates between two equivalent forms. For example:

```text
Run 1 emits: ALTER COLUMN name TYPE VARCHAR(255)
Run 2 emits: ALTER COLUMN name TYPE character varying(255)
Run 3 emits: ALTER COLUMN name TYPE VARCHAR(255)
```

Canonicalization prevents this class of migration noise.

### Safe Automation

CI can block drift because the diff is stable. A nondeterministic diff would make CI untrustworthy.

## Link to SQL Generation

The canonical spec is not the final SQL. It is the stable state that the diff engine compares. Once the diff produces typed operations, backend emitters render SQL.

See [SQL Generation](sql-generation.md) for the next stage in the pipeline.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/
========================================================================

# Correctness

DBWarden generates SQL automatically. This section explains the mechanical checks that make that safe: how DBWarden proves the generated SQL matches the declared model state, how it prevents noisy diffs, how it warns before risky changes, and how rollback SQL is produced under a strict contract.

The claim is not "trust the tool because it is convenient". The claim is stronger: DBWarden exposes a chain of independent checks. Each check catches a different class of failure. Together they make generated migrations reviewable, reproducible, and suitable for CI enforcement.

Start with the [convergence gate](convergence-gate.md). It is the single most powerful correctness check because it tests the full migration history against an empty database and verifies that the resulting schema matches the models.

## Chain of Trust

```text
Models
  |
  v
Canonical model state
  |
  v
Deterministic diff
  |
  v
Typed operations in a migration plan
  |
  +--> Safety classifier
  |
  v
Backend-native SQL emitters
  |
  v
Plain SQL migration file
  |
  +--> Strict rollback contract
  |
  v
Apply to database
  |
  v
Extract schema snapshot
  |
  v
Convergence and round-trip verification
```

Each stage has a different responsibility. Canonicalization removes representation noise. Diffing decides what changed. SQL emitters turn typed operations into backend-native SQL. Safety classification highlights risky operations before execution. Rollback generation checks whether the reverse path is executable or explicitly irreversible. Convergence checks prove that the final database state matches the model state.

## Defense in Depth

### Convergence Gate

The convergence gate applies the complete migration history to an empty database, extracts the resulting schema, and compares it with the current model specification. The gate passes only when there is zero schema drift.

This catches errors that unit tests can miss, such as a migration file that forgot an index, a model rename that was emitted as a drop and add, or a rollback edit that accidentally changed the upgrade section. See [Convergence Gate](convergence-gate.md).

### Round-Trip Verification

Round-trip verification checks extract and emit consistency. DBWarden reads a real database with `generate-models`, emits models, generates migrations from those models, and verifies that extraction after apply produces the same schema shape.

This is especially important for backend-specific features such as PostgreSQL identity columns, partitioning, exclusion constraints, ClickHouse engines, projections, skip indexes, and RBAC objects. See [Round-Trip Verification](round-trip-verification.md).

### Deterministic Diff

DBWarden canonicalizes model specs and database snapshots before comparing them. The same inputs produce the same diff every time.

Canonicalization prevents representation noise from turning into fake migrations. For example, PostgreSQL may report `character varying(255)` while a SQLAlchemy model says `String(255)`. ClickHouse may report engine metadata in an order that differs from the declaration. DBWarden normalizes those forms before diffing. See [Deterministic Diff](deterministic-diff.md).

### Safety Classifier

The safety classifier scans migration plans and identifies risky changes before execution. It distinguishes safe changes from warnings and blocking changes, then requires explicit acknowledgement for risky operations.

The goal is simple: DBWarden should never silently drop data. See [Safety Classifier](safety-classifier.md).

### Offline Integrity

Offline mode lets CI generate migrations without a live database by using a checked-in model state file. Schema snapshots and model state files make migration generation deterministic and protect the pipeline from accidental drift in a developer database.

Offline integrity is not a replacement for a live convergence gate. It is the deterministic first half of the workflow. See [Offline Integrity](offline-integrity.md).

### SQL Generation

DBWarden does not hide runtime logic inside migration files. It emits plain SQL. The diff becomes typed operations, operations are ordered, and backend-specific handlers render native SQL for PostgreSQL, MySQL, SQLite, MariaDB, and ClickHouse.

The output is designed for review. See [SQL Generation](sql-generation.md).

### Rollback Generation

Rollback SQL is generated with the upgrade SQL under a strict contract. Executable rollback SQL is accepted. Placeholder rollback is refused by default. Operations that cannot be rolled back automatically must be explicitly declared irreversible.

This means rollback correctness is not an afterthought and not a manually maintained parallel file. See [Rollback Generation](rollback-generation.md) and [Rollback Coverage Matrix](rollback-coverage-matrix.md).

## Recommended Reading Order

1. [Convergence Gate](convergence-gate.md)
2. [Deterministic Diff](deterministic-diff.md)
3. [SQL Generation](sql-generation.md)
4. [Rollback Generation](rollback-generation.md)
5. [Safety Classifier](safety-classifier.md)
6. [Offline Integrity](offline-integrity.md)
7. [Round-Trip Verification](round-trip-verification.md)

The pages are independent, but the trust model is cumulative. The convergence gate proves the final database shape. Deterministic diffing explains why generated changes are stable. SQL generation explains why the emitted file is backend-native and reviewable. Rollback generation explains the reverse path. Safety classification and offline integrity make the workflow suitable for CI.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/object-handler-protocol/
========================================================================

# ObjectHandler Protocol

The `ObjectHandler` protocol is the mechanical contract that makes deterministic diffs, backend-native SQL generation, and symmetric rollbacks possible. Every backend object - tables, columns, indexes, views, roles, policies, grants, ClickHouse projections, materialized views, and more - has a dedicated handler that owns its full lifecycle: extraction, canonicalization, diffing, and SQL emission.

Without understanding the handler contract, the other correctness guarantees feel like magic. With it, they become inspectable engineering.

## The Problem It Solves

Every database backend has a different DDL dialect, different catalog tables, and different type systems. PostgreSQL uses `pg_catalog`, MySQL uses `information_schema`, ClickHouse uses `system.tables`, and SQLite uses `sqlite_master`. Column types that look similar (`VARCHAR(255)`, `String`, `TEXT`) have different internal representations. Engine metadata - partitioning, sorting keys, TTL expressions - exists only in ClickHouse. Roles and policies are structured differently across PostgreSQL and ClickHouse.

Core must treat all of these uniformly without hard-coding backend knowledge into the diff engine. Adding a new object type or a new backend must not require rewriting the diff pipeline.

The handler protocol solves this by defining a narrow interface that each backend implements per object type. Core calls the interface. Backend code provides the implementation.

## The Handler Contract

Every handler implements the `ObjectHandler` protocol defined in `dbwarden/engine/core/protocol.py`:

```python
class ObjectHandler(Protocol):
    object_type: str
    run_phase: RunPhase
    statement_order: StatementOrder

    def extract(self, snapshot: dict[str, Any]) -> dict[str, Any]: ...

    def model_spec_from_config(self, config: Any) -> dict[str, Any]: ...

    def model_spec_from_tables(
        self, model_tables: list[Any]
    ) -> dict[str, Any]: ...

    def canonicalize(self, spec: dict[str, Any]) -> dict[str, Any]: ...

    def diff(
        self,
        snap_spec: dict[str, Any],
        model_spec: dict[str, Any],
    ) -> Tuple[List[Op], List[Op]]: ...

    def emit(
        self, op: Op, db_name: Optional[str] = None, **kwargs: Any,
    ) -> List[MigrationStatement]: ...
```

### Method Roles

| Method | Input | Output | Responsibility |
|---|---|---|---|
| `extract` | Full database snapshot dict | Raw backend state for this object type | Pulls the handler's relevant objects out of a schema snapshot |
| `model_spec_from_config` | Backend config | Raw config-derived spec | Builds spec from configuration (used in PREAMBLE phase for objects not in model tables) |
| `model_spec_from_tables` | List of model tables | Raw model-derived spec | Builds spec from Python model definitions (used in DIFF phase) |
| `canonicalize` | Raw spec | Normalized comparable spec | Removes representation noise so that equivalent states produce identical dicts |
| `diff` | Canonical snapshot spec, canonical model spec | Upgrade ops + rollback ops | Detects differences and produces paired typed operations |
| `emit` | A single Op | List of MigrationStatement objects | Renders backend-native SQL for upgrade and rollback |

### Supporting Types

**`Op`** is the typed operation that `diff` produces:

```python
@dataclass
class Op:
    object_type: str
    upgrade_attrs: dict[str, Any] = field(default_factory=dict)
    rollback_attrs: dict[str, Any] = field(default_factory=dict)
    irreversible: bool = False
```

`upgrade_attrs` stores the data needed to emit the upgrade SQL. `rollback_attrs` stores the data needed to reverse it. Both are carried together so that rollback never needs to re-inspect the database or guess prior state.

**`MigrationStatement`** is the emit output:

```python
@dataclass
class MigrationStatement:
    order: StatementOrder
    upgrade_sql: str
    rollback_sql: str
    rollback_kind: str = "real"
    rollback_reason: str | None = None
```

`StatementOrder` is an `IntEnum` that controls the sequence of SQL emission. Upgrade statements are sorted by this value. Rollback statements are emitted in reverse order.

**`RunPhase`** controls when the handler runs:

```python
class RunPhase(IntEnum):
    PREAMBLE = 0   # runs before table-diff phase; uses model_spec_from_config
    DIFF = 1       # runs during table-diff phase; uses model_spec_from_tables
```

PREAMBLE handlers cover objects like roles and schemas that exist outside the table model tree. DIFF handlers cover objects that are derived from table metadata.

## How This Enables Correctness

### Canonicalize → Deterministic Diffs

`canonicalize` is the noise-removal step. Two specs that represent the same logical state must produce identical dictionaries after canonicalization. This guarantees that the diff only fires on real changes, not on formatting differences.

For example, PostgreSQL may report `character varying(255)` while a model says `String(255)`. The column handler's canonicalization reduces both to `varchar(255)`. A model default of `true` and a database default of `TRUE` both normalize to `true`.

If the canonicalized specs match, no diff is produced. If they differ, the diff is a real schema change.

### Diff Returns Paired Ops → Symmetric Rollbacks

`diff` returns two lists: upgrade ops and rollback ops. They are produced at the same time from the same inputs. This means rollback is not a separate best-effort pass.

Each op carries `upgrade_attrs` (what `emit` needs for the forward SQL) and `rollback_attrs` (what `emit` needs for the reverse SQL). For a column type change:

```python
Op(
    object_type="alter_column_type",
    upgrade_attrs={"table": "users", "column": "name", "model_type": "varchar(500)"},
    rollback_attrs={"table": "users", "column": "name", "model_type": "varchar(255)"},
)
```

The same `emit` method uses `upgrade_attrs` to render the upgrade statement and `rollback_attrs` to render the rollback. The reverse is structurally paired.

### Emit → Backend-Native SQL

`emit` produces raw SQL strings wrapped in `MigrationStatement` objects. Different backends implement `emit` differently for the same object type. A column type change in PostgreSQL produces:

```sql
ALTER TABLE users ALTER COLUMN name TYPE VARCHAR(500);
```

The same logical change in ClickHouse produces:

```sql
ALTER TABLE users MODIFY COLUMN name String;
```

The diff pipeline does not know or care which dialect is being generated. Core calls `emit`, and the backend handler owns the SQL.

## How Handlers Are Registered

Handlers are registered with the `RegistryDriver` in `dbwarden/engine/core/registry.py`:

```python
class RegistryDriver:
    def __init__(self):
        self._handlers: dict[str, ObjectHandler] = {}

    def register(self, handler: ObjectHandler) -> None:
        self._handlers[handler.object_type] = handler
```

Registration happens during backend initialization. Each backend module (PostgreSQL, ClickHouse, MySQL) creates its set of handlers and registers them:

```python
driver.register(ColumnHandler())
driver.register(IndexHandler())
driver.register(TableHandler())
driver.register(ConstraintHandler())
driver.register(EnumHandler())
driver.register(DomainHandler())
driver.register(RoleHandler())
driver.register(PoliciesHandler())
driver.register(GrantsHandler())
driver.register(ViewHandler())
driver.register(TriggerHandler())
driver.register(SequenceHandler())
driver.register(PartitionHandler())
driver.register(SchemaHandler())
driver.register(FunctionHandler())
driver.register(CompositeTypeHandler())
driver.register(DefaultPrivilegesHandler())
driver.register(EventTriggerHandler())
driver.register(ExtendedStatisticsHandler())
driver.register(StorageParamsHandler())
driver.register(RenameTableHandler())
driver.register(PgTableHandler())
# ClickHouse handlers
driver.register(ChTableHandler())
driver.register(ChColumnHandler())
driver.register(ChProjectionHandler())
driver.register(ChSkipIndexHandler())
driver.register(ChMaterializedViewHandler())
driver.register(ChCommentHandler())
driver.register(ChDictionaryHandler())
driver.register(ChRoleHandler())
driver.register(ChUserHandler())
driver.register(ChRowPolicyHandler())
driver.register(ChSettingsProfileHandler())
driver.register(ChQuotaHandler())
driver.register(ChNamedCollectionHandler())
driver.register(ChGrantHandler())
driver.register(ChAggTargetHandler())
driver.register(ChDataOpHandler())
```

Core's `RegistryDriver.run()` iterates over all registered handlers, extracts their specs, canonicalizes, diffs, and collects the paired ops. The caller then orders them by `statement_order` and emits SQL.

Because registration is just a method call, third-party plugins can register handlers for custom object types using the same protocol.

## Example Walk-Through: ALTER TABLE ADD COLUMN

This trace follows a single `ALTER TABLE ADD COLUMN` through the full pipeline.

### 1. Extract

The database snapshot is a large dict of all extracted schema objects. The `ColumnHandler.extract` pulls only the column data:

```python
# ColumnHandler.extract
def extract(self, snapshot):
    result = {}
    for tname, tdata in snapshot.get("tables", {}).items():
        result[tname] = dict(tdata.get("columns", {}))
    return result
```

Input snapshot (abbreviated):
```json
{
  "tables": {
    "users": {
      "columns": {
        "id": {"type": "integer", "nullable": false, "primary_key": true},
        "name": {"type": "character varying(255)", "nullable": true}
      }
    }
  }
}
```

Output:
```json
{
  "users": {
    "id": {"type": "integer", "nullable": false, "primary_key": true},
    "name": {"type": "character varying(255)", "nullable": true}
  }
}
```

### 2. Model Spec

`model_spec_from_tables` reads the same columns from the Python model definitions:

```json
{
  "users": {
    "id": {"type": "INTEGER", "nullable": false, "primary_key": true},
    "name": {"type": "VARCHAR(255)", "nullable": true},
    "display_name": {"type": "VARCHAR(255)", "nullable": true}
  }
}
```

The model has a new column `display_name` that does not exist in the snapshot.

### 3. Canonicalize

Both specs are run through `canonicalize`. The column handler currently passes through directly, but in general this step normalizes type names, default expressions, and metadata formats so that the diff is stable.

After canonicalization, `character varying(255)` and `VARCHAR(255)` both become `varchar(255)`.

### 4. Diff

`diff` compares the canonical specs. It detects:

- `display_name` exists in the model spec but not in the snapshot spec → **add column**
- `display_name` does not exist in the snapshot spec → no column to drop → the rollback of add is **drop column**

It produces paired ops:

```python
# Upgrade op
Op(
    object_type="add_column",
    upgrade_attrs={
        "table": "users",
        "column": "display_name",
        "model_column": <ModelColumn: display_name VARCHAR(255) nullable>,
    },
    rollback_attrs={"table": "users", "column": "display_name"},
)

# Rollback op
Op(
    object_type="drop_column",
    upgrade_attrs={
        "table": "users",
        "column": "display_name",
        "definition": {"type": "VARCHAR(255)", "nullable": True},
    },
    rollback_attrs={"table": "users", "column": "display_name"},
)
```

### 5. Emit

During SQL generation, `RegistryDriver.emit_all` dispatches each op to the correct handler's `emit` method. For `add_column`, the `ColumnHandler.emit` renders:

```python
sql = generate_add_column_sql("users", model_col, db_name)
# → "ALTER TABLE users ADD COLUMN display_name VARCHAR(255);"

stmt = MigrationStatement(
    order=StatementOrder.ADD_COLUMN,
    upgrade_sql=sql,
    rollback_sql="ALTER TABLE users DROP COLUMN display_name",
)
```

The `MigrationStatement` is then included in the final migration file:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

The same protocol handles column type changes, nullable changes, default changes, comment changes, indexes, constraints, roles, policies, and every other supported object type across all backends.

## The Ordering Constraint

Each handler declares a `statement_order` that controls when its emitted SQL appears relative to other statements. The `StatementOrder` enum defines anchor points such as `ADD_COLUMN`, `DROP_TABLE`, `ALTER_CONSTRAINT`, and `CREATE_TYPE`.

Core sorts all `MigrationStatement` objects by their `order` value before writing the migration file. Rollback statements are emitted in reverse order so that dependencies are satisfied in both directions.

This ordering mechanism is intentionally simple. Future work may introduce a topological sort where handlers declare edges against named anchors instead of numeric enum values. For now, the integer enum provides enough granularity for correct migration ordering across all supported backends.

## How to Add a New Handler

### 1. Implement the Protocol

Create a new class that implements `ObjectHandler`:

```python
from dbwarden.engine.core.protocol import ObjectHandler, Op, RunPhase
from dbwarden.engine.core.statement_order import MigrationStatement, StatementOrder

class MyCustomHandler(ObjectHandler):
    object_type: str = "my_custom_object"
    op_types: tuple[str, ...] = ("my_custom_op",)
    run_phase: RunPhase = RunPhase.DIFF
    statement_order: StatementOrder = StatementOrder.ALTER_TABLE_OPTIONS

    def extract(self, snapshot):
        ...

    def model_spec_from_config(self, config):
        return {}

    def model_spec_from_tables(self, model_tables):
        ...

    def canonicalize(self, spec):
        ...

    def diff(self, snap_spec, model_spec):
        upgrade_ops = []
        rollback_ops = []
        ...
        return upgrade_ops, rollback_ops

    def emit(self, op, db_name=None, **kwargs):
        ...
```

### 2. Register It

```python
from dbwarden.engine.core.registry import RegistryDriver

driver = RegistryDriver()
driver.register(MyCustomHandler())
```

### 3. Write Round-Trip Tests

- Verify that `extract` + `canonicalize` applied to a known snapshot produces the expected spec.
- Verify that `diff` with identical canonical specs produces no ops.
- Verify that `diff` with differing specs produces the expected upgrade/rollback pair.
- Verify that `emit` produces the expected SQL for each operation variant.
- Verify that the full round trip (extract → canonicalize → diff → emit → apply → extract) returns to the original state.

### 4. Add Extraction Support

If the new handler reads from the database, add extraction logic in the appropriate backend extractor (e.g., `extract.py` for PostgreSQL, `extract_ch.py` for ClickHouse). The extractor populates the snapshot dict that `extract` reads.

### 5. Add Safety Classification

If the handler produces operations that could drop data or require user acknowledgement, add entries in the safety classifier in `dbwarden/engine/safety/snapshot.py`.

## Relation to Other Correctness Pages

The handler protocol is the bridge between the abstract guarantees and the concrete pipeline:

| Correctness property | Role of the handler protocol |
|---|---|
| **Deterministic diff** | `canonicalize` removes representation noise before comparison. Two equivalent states produce identical canonical forms, so diffs only fire on real changes. |
| **SQL generation** | `emit` produces backend-native SQL. The diff pipeline generates typed operations; the handler renders them for one specific backend and object family. |
| **Symmetric rollback** | `diff` returns paired upgrade and rollback ops. Each Op carries both forward and reverse attributes, so rollback is structurally paired with upgrade from the start. |

- [Deterministic Diff](deterministic-diff.md) - the handler's `canonicalize` method is what makes diffs stable
- [SQL Generation](sql-generation.md) - the handler's `emit` method is what produces backend-native SQL
- [Rollback Generation](rollback-generation.md) - the handler's `diff` method returns paired ops that make symmetric rollbacks possible

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/offline-integrity/
========================================================================

# Offline Integrity

Offline integrity lets DBWarden generate migrations without connecting to a live database. It does this by comparing current models to a checked-in model state file or schema snapshot instead of querying the database at generation time.

This is a correctness feature because it removes accidental dependence on a developer's local database. CI can generate or verify migrations from repository state alone.

## Snapshot and Model State Concepts

DBWarden uses two related state files.

### Schema Snapshots

After migrations are applied, DBWarden can write checksummed schema snapshots under `.dbwarden/schemas/`. These snapshots represent the database schema after a migration point.

They support:

- Rename detection
- Offline comparisons
- Column-level diffing without querying a live database
- Auditability of schema history

### Model State

`export-models` writes a model state JSON file, usually under `.dbwarden/model_state.json` or a database-specific variant. This file records the model-derived schema state that offline migration generation uses as a baseline.

Create it with:

```bash
dbwarden export-models --database primary
```

Commit it with the repository:

```bash
git add .dbwarden/model_state.json
git commit -m "Update DBWarden model state"
```

The model state file records the schema that DBWarden expects the database to be in after the last migration. It is the baseline for offline diffing and can be regenerated at any time from a live database.

## Offline Migration Generation

Offline migration generation compares the checked-in state file with the current models:

```text
Checked-in model state
  |
  v
Normalize baseline

Current SQLAlchemy models
  |
  v
Extract model state

Baseline + current model state
  |
  v
Offline diff
  |
  v
Generated migration SQL and plan
```

Run it with:

```bash
dbwarden make-migrations "add profile fields" --offline --database primary
```

If the state file is missing, DBWarden tells you to run `export-models` first. If the state file is invalid, DBWarden refuses to use it.

> **If accidentally deleted:** restore it from git (`git checkout .dbwarden/model_state.json`) or regenerate it by running `dbwarden export-models --database <db>` against a live database. Offline commands will work again immediately.

## Integrity Check

Model state and schema snapshots are checksummed. Before DBWarden uses a state file, it validates that the file content matches the stored checksum.

The reason is straightforward:

```text
State file on disk
  |
  v
Recompute SHA-256 checksum
  |
  v
Compare with stored checksum
  |
  +--> match: use the state file
  |
  +--> mismatch: refuse and ask for regeneration
```

This protects against accidental edits, merge corruption, and stale generated files. A modified JSON file should not silently become the baseline for migration generation.

## CI Workflow Example

A deterministic CI workflow can use offline generation first, then live convergence when a database service is available.

```yaml
name: dbwarden-offline-integrity

on:
  pull_request:

jobs:
  offline-integrity:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install project
        run: pip install -e .

      - name: Generate migrations from checked-in state
        run: dbwarden make-migrations "ci offline check" --offline --database primary

      - name: Show changed files
        run: git status --short
```

In a strict repository, CI should fail if offline generation creates unexpected changes. That means a developer changed models without committing the corresponding migration or model state update.

The live convergence job can run after this step:

```text
Offline integrity
  |
  v
Generated files are stable
  |
  v
Live convergence gate
  |
  v
Generated SQL applies and matches models
```

See [Convergence Gate](convergence-gate.md).

## Example Failure

Assume the committed model state says `users` has two columns:

```json
{
  "tables": {
    "users": {
      "columns": {
        "id": {"type": "INTEGER", "primary_key": true},
        "email": {"type": "VARCHAR(255)", "nullable": false}
      }
    }
  }
}
```

A developer adds a model field:

```python
display_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
```

Then CI runs:

```bash
dbwarden make-migrations "ci offline check" --offline --database primary
```

DBWarden detects the difference between the committed state and current models:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

If this migration was expected, commit it. If it was not expected, the model change is accidental and should be reverted or corrected.

## Why Offline Integrity Improves Correctness

Offline integrity decouples migration generation from live database accidents.

Without offline state, a developer's local database could contain manual changes. DBWarden might diff against that local drift and produce a migration that looks correct on one machine but fails in CI or production.

With offline state:

- The baseline is versioned in git.
- The same inputs produce the same diff.
- CI can detect missing migrations without a database service.
- Tampered state files are rejected by checksum validation.

Offline integrity is not the final proof. The final proof is still a live database convergence gate. Offline integrity ensures the inputs to that gate are deterministic.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/rollback-coverage-matrix/
========================================================================

# Rollback Coverage Matrix

This matrix describes rollback behavior implemented by the PostgreSQL and ClickHouse handlers. It distinguishes real inverse SQL from conditional rollback, explicit irreversible changes, and placeholder or manual rollback comments.

## Classification

- `real`: Emits executable reverse SQL for the normal operation.
- `conditional`: Emits executable reverse SQL only when dbwarden can prove the operation is structurally reversible from static metadata and all required prior state is present. Conditional must be explicit and limited. If rollback safety is not secured, the operation must not be classified as conditional.
- `irreversible`: The operation is intentionally marked or treated as not safely reversible.
- `placeholder`: Rollback is a comment, manual instruction, or incomplete placeholder.
- `no-op`: No reverse action is expected for the operation.

## Warning Policy

- `irreversible` rollback emits a warning during standard migration generation.
- `conditional` rollback emits a warning only in verbose mode.
- `placeholder` rollback is a hard generation error by default.
- Placeholder rollback comments are allowed only for explicitly irreversible or manual cases, not as a successful rollback claim.
- The committed-migration escape hatch is an explicit annotation: `-- dbwarden: irreversible`.
- `--allow-placeholder-rollback` is intentionally not the primary escape hatch and should be treated only as a future local-development convenience.

## ClickHouse Recreate Policy

ClickHouse table recreation uses a detect-and-refuse rollback policy. Unknown means unsafe.

| Case | Classification | Behavior |
| --- | --- | --- |
| Structurally lossy transitions | irreversible | Engines such as `ReplacingMergeTree`, `SummingMergeTree`, `AggregatingMergeTree`, `CollapsingMergeTree`, and `VersionedCollapsingMergeTree` may collapse, deduplicate, aggregate, or otherwise lose row-level detail. Reverse recreate cannot restore data that no longer exists. |
| Structurally reversible transitions | conditional | Both engine families must be known row-preserving, such as `MergeTree`, `ReplicatedMergeTree`, `Memory`, `Log`, `TinyLog`, and `StripeLog`. Rollback emits reverse recreate SQL, and should be promoted only when covered by live round-trip tests. |
| Cannot-tell transitions | irreversible | Custom engines, unknown engines, unparsed engine behavior, or transitions whose safety depends on data are treated as irreversible. |

## PostgreSQL

| Operation | Classification | Notes |
| --- | --- | --- |
| `create_schema` | real | Rollback drops the schema. |
| `drop_schema` | real | Rollback recreates the schema. |
| `create_table` | real | Rollback drops the table. |
| `drop_table` | conditional | Recreates from captured `state_table`; strict generation fails if the prior table definition is unavailable. |
| `alter_table_comment` | real | Restores prior comment or `NULL`. |
| `add_column` | real | Rollback drops the column. |
| `drop_column` | conditional | Re-adds the column when definition exists; data is not restored. |
| `rename_column` | real | Renames back to the original column name. |
| `alter_column_type` | conditional | Emits inverse type change; safety depends on database castability and helper behavior. |
| `alter_column_nullable` | real | Restores previous nullability. |
| `alter_column_autoincrement` | real | Toggles sequence/default behavior back. |
| `alter_column_default` | real | Restores previous default from `rollback_attrs`. |
| `alter_column_comment` | real | Restores prior column comment or `NULL`. |
| `alter_pg_column_meta` | conditional | Depends on PostgreSQL column metadata helper support. |
| `alter_enum_add_value` | irreversible | PostgreSQL enum values cannot be removed directly. |
| `create_type` | real | Rollback drops enum type. |
| `drop_type` | real | Rollback recreates enum from saved values. |
| `create_domain` | real | Rollback drops domain. |
| `drop_domain` | conditional | Recreates domain when definition is present; strict generation fails if the prior definition is unavailable. |
| `create_composite_type` | real | Rollback drops composite type. |
| `drop_composite_type` | conditional | Recreates type when definition is present; strict generation fails if the prior definition is unavailable. |
| `create_function` | real | Rollback drops the created function. |
| `drop_function` | conditional | Recreates only when saved definition exists. |
| `create_event_trigger` | real | Rollback drops event trigger. |
| `drop_event_trigger` | conditional | Recreates only when full trigger info exists. |
| `create_extended_statistics` | real | Rollback drops statistics. |
| `drop_extended_statistics` | conditional | Recreates only when statistic info exists. |
| `add_index` | conditional | Diff carries inverse state; executable SQL depends on `_build_index_sql`. |
| `drop_index` | conditional | Recreates when index attributes are present. |
| `add_unique_constraint` | real | Rollback drops unique constraint. |
| `drop_unique_constraint` | real | Rollback recreates unique constraint. |
| `rename_unique_constraint` | real | Rollback renames back. |
| `add_check_constraint` | real | Rollback drops check constraint. |
| `drop_check_constraint` | real | Rollback recreates check constraint. |
| `add_foreign_key` | real | Rollback drops foreign key. |
| `drop_foreign_key` | real | Rollback recreates foreign key. |
| `add_grant` | real | Rollback revokes grant. |
| `revoke_grant` | real | Rollback grants privilege back. |
| `add_schema_grant` | real | Rollback revokes schema grant. |
| `revoke_schema_grant` | real | Rollback grants schema privilege back. |
| `alter_default_privileges` | conditional | Privilege grant/revoke is reversible when the previous privilege state is represented in the plan. |
| `create_role` | real | Rollback drops role. |
| `drop_role` | conditional | Recreates the role from attributes captured in `rollback_attrs` from the live schema or baseline snapshot. |
| `alter_role` | conditional | Restores prior role attributes captured in `rollback_attrs` from the live schema or baseline snapshot. |
| `create_sequence` | real | Rollback drops sequence. |
| `drop_sequence` | conditional | Recreates only when sequence info exists. |
| `alter_pg_rls` | real | Restores previous RLS/force setting. |
| `add_policy` | real | Rollback drops policy. |
| `drop_policy` | conditional | Recreates policy from `rollback_attrs` when available. |
| `alter_policy` | conditional | Restores prior policy from `rollback_attrs` when available. |
| `alter_pg_storage_param` | real | Restores previous table storage parameter or resets it. |
| `alter_pg_table` | conditional | Fillfactor and logged/unlogged are reversible; unsupported physical rewrites fail strict generation unless handled manually. |
| `add_exclude_constraint` | real | Rollback drops exclusion constraint. |
| `drop_exclude_constraint` | real | Rollback recreates exclusion constraint. |
| `alter_column_statistics` | real | Restores prior statistics target or resets to the PostgreSQL default target with `SET STATISTICS -1`. |
| `alter_pg_partition` | irreversible | Partition strategy changes require a table rebuild or a hand-authored migration. DBWarden refuses to claim automatic rollback. |
| `attach_partition` | real | Rollback detaches partition. |
| `detach_partition` | real | Rollback attaches partition with saved bound. |
| `rename_table` | conditional | Delegates to rename SQL helper. |
| `create_trigger` | conditional | Rollback drops trigger; create is placeholder if definition is missing. |
| `drop_trigger` | conditional | Recreates only when trigger definition exists. |
| `alter_trigger` | placeholder | Emit path is not implemented and uses comments. |
| `alter_view` | real | Drops and recreates view or materialized view with previous definition. |
| `refresh_matview` | no-op | Refresh has no stateful rollback. |

## ClickHouse

| Operation | Classification | Notes |
| --- | --- | --- |
| `alter_ch_options` | conditional | TTL/settings/order-by can emit inverse ALTERs; immutable/recreate-required keys can require manual handling. |
| `recreate_ch_table` | conditional or irreversible | Reverse recreate is emitted only when both engines are known row-preserving. Lossy or unknown engine transitions emit an irreversible rollback comment. |
| `alter_ch_column` | conditional | Emits inverse ALTERs for supported type/default/codec/TTL/materialized/alias/nullability/LowCardinality changes. |
| `modify_mv_query` | real | Restores previous materialized view SELECT. |
| `modify_mv_refresh` | conditional | Restores previous refresh when present; absent refresh is no-op, refresh removal is manual. |
| `alter_ch_dict` | conditional | Create/drop reversible when options exist; unsupported alter keys emit manual comments. |
| `create_ch_named_collection` | real | Rollback drops collection. |
| `drop_ch_named_collection` | conditional | Recreates the named collection from prior entries and overridable flags captured in `rollback_attrs`. |
| `alter_ch_named_collection` | conditional | Drops and recreates from full target/prior state so entry and overridable changes are reversible when prior state is captured. |
| `alter_ch_projection` | conditional | Add/drop/replace are reversible when prior definitions are in rollback attrs. |
| `alter_ch_skip_index` | conditional | Add/drop/replace are reversible when prior definitions are in rollback attrs. |
| `apply_data_op` | irreversible | Uses authored rollback when provided; otherwise remains explicitly irreversible because arbitrary data mutations cannot be inferred. |
| `create_ch_agg_target` | real | Rollback drops aggregate target table. |
| `drop_ch_agg_target` | conditional | Recreates from captured options; incomplete options can produce incomplete SQL. |
| `alter_ch_comment` | conditional | Restores table and column comments when prior values exist; otherwise may no-op. |
| `grant_ch_privilege` | real | Rollback revokes privilege. |
| `revoke_ch_privilege` | real | Rollback grants privilege back. |
| `create_ch_role` | real | Rollback drops role. |
| `drop_ch_role` | conditional | Recreates the role and captured settings from `rollback_attrs`. |
| `alter_ch_role` | conditional | Restores captured role settings with drop and recreate from `rollback_attrs`. |
| `create_ch_user` | real | Rollback drops user. |
| `drop_ch_user` | conditional | Recreates the user from captured auth, host, roles, default roles, and settings profile in `rollback_attrs`. |
| `alter_ch_user` | conditional | Restores captured user state with drop and recreate from `rollback_attrs`. |
| `create_ch_quota` | real | Rollback drops quota. |
| `drop_ch_quota` | conditional | Recreates quota from captured interval, limits, and role assignments in `rollback_attrs`. |
| `alter_ch_quota` | conditional | Restores captured quota state with drop and recreate from `rollback_attrs`. |
| `create_ch_row_policy` | real | Rollback drops row policy. |
| `drop_ch_row_policy` | conditional | Recreates the row policy from captured table, predicate, roles, and mode in `rollback_attrs`. |
| `alter_ch_row_policy` | conditional | Restores the row policy from prior state using drop and recreate from `rollback_attrs`. |
| `create_ch_settings_profile` | real | Rollback drops settings profile. |
| `drop_ch_settings_profile` | conditional | Recreates settings profile from captured settings and role assignments in `rollback_attrs`. |
| `alter_ch_settings_profile` | conditional | Restores prior settings and role assignments from `rollback_attrs`. |
| `ChRbacHandler` | no-op | No op types are emitted by this wrapper. |

## Manual and Irreversible Boundaries

These cases are not open rollback gaps. They are explicit policy boundaries where DBWarden either emits no rollback because no schema state changes, requires authored SQL, or marks the operation irreversible.

| Case | Policy |
| --- | --- |
| PostgreSQL partition strategy changes | Require a table rebuild or hand-authored migration. DBWarden refuses automatic rollback because changing partition strategy is not a metadata-only inverse. |
| PostgreSQL `REFRESH MATERIALIZED VIEW` | Rollback is intentionally no-op because refresh does not change schema definition. |
| PostgreSQL enum value additions | Irreversible in PostgreSQL because enum values cannot be removed directly. |
| ClickHouse data operations | Require authored rollback SQL when a reverse data operation exists; otherwise they remain explicitly irreversible. |
| ClickHouse engine transitions | Conditional reverse recreate is allowed only for known row-preserving engine families. Lossy and unknown transitions are irreversible by policy. |

For ordinary generated drop and alter operations, prior state is captured at generation time from the live schema snapshot or from the baseline snapshot used by offline and squash workflows. The captured definition is serialized through `rollback_attrs`, preserved in the migration plan, and consumed by the backend handler that emits rollback SQL. If the required prior state is missing, strict generation fails instead of accepting placeholder rollback.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/rollback-generation/
========================================================================

# Rollback Generation

DBWarden generates rollback SQL at the same time it generates upgrade SQL. Rollback is not a separate best-effort file that developers maintain by hand after the fact.

The current rollback system uses a strict contract:

- Executable rollback SQL is accepted.
- Conditional rollback is accepted only when DBWarden has captured enough prior state.
- Irreversible operations must be explicitly acknowledged.
- Placeholder rollback is refused by default.

See the [Rollback Coverage Matrix](rollback-coverage-matrix.md) for backend-specific operation coverage.

## Core Principle

For every upgrade operation, the diff engine computes the reverse operation at the same time.

```text
Diff detects model change
  |
  v
Upgrade operation + rollback operation
  |
  v
SQL generation
  |
  v
Migration file with upgrade and rollback sections
```

This keeps upgrade and rollback logic synchronized. The same handler that knows how to create an object also knows what information is needed to remove or restore it.

## Symmetric Pairs

Many operations have direct structural inverses.

| Upgrade | Rollback |
|---------|----------|
| `CREATE TABLE` | `DROP TABLE` |
| `ADD COLUMN` | `DROP COLUMN` |
| `CREATE INDEX` | `DROP INDEX` |
| `ADD CONSTRAINT` | `DROP CONSTRAINT` |
| `CREATE POLICY` | `DROP POLICY` |
| `CREATE ROLE` | `DROP ROLE` |

Example:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

This rollback removes the column introduced by the upgrade. If the upgrade is applied and then rolled back immediately, the table shape returns to its previous form.

## Conditional Rollback

Some operations are reversible only if DBWarden captured previous state.

Example: changing a PostgreSQL column default.

```sql
-- upgrade
ALTER TABLE users ALTER COLUMN status SET DEFAULT 'active';

-- rollback
ALTER TABLE users ALTER COLUMN status SET DEFAULT 'pending';
```

The rollback is correct only if DBWarden knows the previous default was `'pending'`. If the previous default was unknown, DBWarden must not invent one.

Conditional rollback appears in operations such as:

- Restoring prior PostgreSQL role attributes.
- Restoring prior PostgreSQL policy definitions.
- Restoring prior ClickHouse RBAC object state.
- Reversing a ClickHouse table recreate only when the engine transition is classified as row-preserving and the prior definition is available.

## Irreversible Operations

Some operations cannot be automatically reversed in a way that restores data.

Example:

```sql
-- upgrade
ALTER TABLE users DROP COLUMN legacy_code;
```

DBWarden can add `legacy_code` back if it knows the column definition, but it cannot reconstruct the deleted values. That means a structural rollback is not the same as data recovery.

For generated migrations, DBWarden refuses placeholder rollback by default. If a migration is intentionally irreversible, it must be explicit.

Use the committed migration annotation:

```sql
-- dbwarden: irreversible
```

That annotation tells reviewers and automation that the migration is intentionally not automatically reversible.

## Rollback Ordering

Rollback statements run in the reverse of upgrade order.

Upgrade order:

```text
1. Add column
2. Create index on the new column
3. Add constraint that uses the new column
```

Rollback order:

```text
1. Drop constraint
2. Drop index
3. Drop column
```

This matters because dependencies point in the opposite direction during rollback. You cannot drop a column while an index or constraint still depends on it.

## Complex Example

Model change:

- Add `display_name` to `users`.
- Drop an old index.
- Add a new uniqueness constraint.

Generated migration:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

DROP INDEX IF EXISTS idx_users_name;

ALTER TABLE users ADD CONSTRAINT uq_users_display_name UNIQUE (display_name);

-- rollback
ALTER TABLE users DROP CONSTRAINT IF EXISTS uq_users_display_name;

CREATE INDEX idx_users_name ON users (name);

ALTER TABLE users DROP COLUMN display_name;
```

Why this rollback order is correct:

1. The constraint must be removed before the column can be dropped.
2. The old index is recreated before the rollback finishes because it existed before the upgrade.
3. The added column is removed last after dependencies are gone.

## Manual SQL and Data Changes

DBWarden can generate rollback for schema operations it understands. It cannot infer the inverse of arbitrary manual data changes.

Example:

```sql
-- upgrade
UPDATE users SET status = 'active' WHERE status IS NULL;
```

The inverse is not generally knowable. Which rows were originally `NULL`? Which rows were already `active`? Without captured data, rollback cannot answer that.

For manual data changes, write a manual rollback section or explicitly mark the migration irreversible if rollback cannot be made correct.

## Rollback Convergence Pattern

Teams that want stronger rollback assurance can add a rollback convergence job:

```text
Empty database
  |
  v
Apply all migrations
  |
  v
Rollback all versioned migrations
  |
  v
Verify expected empty or baseline schema
```

This does not prove that dropped data can be recovered. It proves that rollback SQL is syntactically valid and structurally consistent for the schema objects under test.

## Why Rollbacks Should Not Be Hand-Maintained

Hand-written rollback sections drift. A developer changes an upgrade statement and forgets to update the rollback. A reviewer checks the upgrade and misses the reverse path.

DBWarden avoids that by computing both directions from the same operation model. When rollback cannot be computed safely, the generator refuses placeholder rollback rather than producing a misleading comment.

The result is stricter than convenience-oriented migration tools: either DBWarden emits executable rollback SQL, or the migration explicitly declares why automatic rollback is not available.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/round-trip-verification/
========================================================================

# Round-Trip Verification

Round-trip verification checks whether DBWarden can extract a database schema, represent it as models, generate SQL from those models, and arrive back at the same schema.

It is a consistency check between extractors, model metadata, diffing, and emitters.

## What Round-Tripping Means

The loop is:

```text
Existing database
  |
  v
dbwarden generate-models
  |
  v
SQLAlchemy models with backend-specific Meta
  |
  v
dbwarden make-migrations
  |
  v
Generated SQL migration
  |
  v
Apply migration to another empty database
  |
  v
Extract schema again
  |
  v
Compare extracted schema with original shape
```

If the second extraction matches the first, the handler round trip is stable for those object types.

## Why It Matters

Every supported database has features that are not visible in simple `CREATE TABLE` syntax.

PostgreSQL examples:

- Identity column options
- Generated columns
- Collation
- Per-column storage and compression
- Table storage parameters
- Exclusion constraints
- Row-level security policies
- Materialized view settings

ClickHouse examples:

- `MergeTree` engine families
- `ORDER BY`, `PRIMARY KEY`, `PARTITION BY`, and `SAMPLE BY`
- Engine settings
- Projections
- Skip indexes
- Materialized views
- Dictionaries
- RBAC objects

Round-trip verification proves that DBWarden does not lose those details when moving between database state, model state, and SQL.

## It Is Not a Full Mathematical Proof

Round-tripping is strong evidence, not a complete proof of all possible database behavior.

It proves internal consistency for supported schema objects. It does not prove:

- Application data is preserved.
- Manually written SQL has an inverse.
- Runtime behavior such as locks, query plans, replication lag, or trigger side effects is harmless.
- A database extension behaves identically across all versions.

That is why round-trip verification complements the [Convergence Gate](convergence-gate.md). Round-trip verification checks extractor and emitter consistency. The convergence gate checks that the repository's migration history reproduces the current model state.

## PostgreSQL Example

Start with a PostgreSQL schema that uses backend-specific features:

```sql
CREATE TABLE accounts (
    id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    tenant_id integer NOT NULL,
    email character varying(255) NOT NULL,
    created_at timestamp with time zone DEFAULT now() NOT NULL
) WITH (fillfactor = 80);

CREATE INDEX idx_accounts_tenant_email
    ON accounts USING btree (tenant_id, email);
```

DBWarden extracts the schema:

```bash
dbwarden generate-models --database primary --output generated_models.py
```

The generated model should preserve the meaningful PostgreSQL metadata:

```python
class Account(Base):
    __tablename__ = "accounts"

    id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
    tenant_id: Mapped[int] = mapped_column(Integer, nullable=False)
    email: Mapped[str] = mapped_column(String(255), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)

    class Meta(PGTableMeta):
        pg_storage_params = {"fillfactor": 80}
        indexes = [
            IndexSpec(
                name="idx_accounts_tenant_email",
                columns=["tenant_id", "email"],
            ),
        ]
```

The exact generated Python may differ by import style, but the important part is that schema-relevant metadata is represented. The identity column, storage parameters, and index are not discarded.

Then generate migrations from that model against an empty database:

```bash
dbwarden make-migrations "recreate accounts" --database primary
dbwarden migrate --database primary
dbwarden diff --database primary --out table
```

The expected diff is empty. If DBWarden emits a migration every time from the generated model, some part of extraction, canonicalization, or emission is losing information.

## ClickHouse Example

Start with a ClickHouse table that uses a `MergeTree` engine, partitioning, a projection, and a skip index:

```sql
CREATE TABLE events (
    id UInt64,
    event_date Date,
    user_id UInt64,
    path String,
    INDEX idx_path path TYPE bloom_filter(0.01) GRANULARITY 64,
    PROJECTION by_date (
        SELECT event_date, count()
        GROUP BY event_date
    )
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_date, id)
SETTINGS index_granularity = 8192;
```

The generated model must preserve engine and table metadata:

```python
class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    user_id: Mapped[int] = mapped_column()
    path: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            partition_by="toYYYYMM(event_date)",
            order_by=["event_date", "id"],
            settings={"index_granularity": "8192"},
            projections=[
                ProjectionSpec(
                    name="by_date",
                    select="SELECT event_date, count() GROUP BY event_date",
                ),
            ],
            indexes=[
                ChIndexSpec(
                    name="idx_path",
                    expr="path",
                    clickhouse_type="bloom_filter(0.01)",
                    granularity=64,
                ),
            ],
        )
```

The verification flow is the same:

```bash
dbwarden generate-models --database analytics --output generated_analytics.py
dbwarden make-migrations "recreate clickhouse schema" --database analytics
dbwarden migrate --database analytics
dbwarden diff --database analytics --out table
```

The diff should be empty. If the engine settings, projection, or skip index are missing after extraction, the round trip fails and the handler needs a fix.

## Manual Round-Trip Checklist

Use this checklist when validating a backend feature:

1. Create the object directly in a disposable database.
2. Run `dbwarden generate-models` for that database.
3. Inspect the generated `class Meta` block.
4. Apply the generated model to an empty disposable database with `make-migrations` and `migrate`.
5. Run `dbwarden diff` and confirm no schema drift remains.
6. Add a test for the feature so future changes cannot regress it.

## What Round-Trip Verification Does Not Cover

Round-trip verification does not cover data migration semantics. A table can round-trip perfectly while a manual `UPDATE` statement still needs human review.

It also does not replace backend integration tests. For example, ClickHouse may accept syntax but behave differently depending on engine family. PostgreSQL may require a lock level that matters in production. Those concerns belong in safety review and integration testing.

Use round-trip verification to prove schema representation fidelity. Use the [Convergence Gate](convergence-gate.md) to prove the repository history lands on the declared model state.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/safety-classifier/
========================================================================

# Safety Classifier

The safety classifier detects risky schema changes before migration execution. It reads the migration plan, classifies each operation, and forces the operator to acknowledge changes that may affect data or availability.

The principle is simple: DBWarden should not silently drop data.

## What It Scans

`make-migrations` writes a companion `.plan.json` file next to generated SQL migrations. The plan records the typed operations that produced the SQL. A safety check can inspect that plan before the SQL is applied.

The flow is:

```text
Models or manual migration request
  |
  v
Diff engine
  |
  v
Typed operations
  |
  v
.plan.json
  |
  v
Safety classifier
  |
  v
Safe, warning, or blocking result
```

Run the check command before applying migrations:

```bash
dbwarden check --database primary
```

If the command reports warnings that require acknowledgement, use the documented command option only after reviewing the plan:

```bash
dbwarden check --database primary --force
```

`--force` is an acknowledgement. It should not be used as a default in CI.

## Severity Levels

DBWarden safety classifications map to three operational meanings.

### Info

Info-level changes are expected to be safe from a schema perspective.

Examples:

- Create a table.
- Create an index.
- Add a nullable column.
- Add a column with a safe default on an empty table.

Example operation:

```sql
ALTER TABLE users ADD COLUMN nickname VARCHAR(255);
```

Why it is low risk:

- Existing rows remain valid because the column is nullable.
- No data is removed.
- The operation is visible in the migration file and plan.

### Warning

Warning-level changes may be valid, but they need human review.

Examples:

- Add `NOT NULL` to a column when existing data may violate it.
- Change a column type where conversion may fail.
- Change storage parameters or backend-specific physical settings.
- Recreate a ClickHouse table for an engine transition that is classified as row-preserving but still operationally sensitive.

Example operation:

```sql
ALTER TABLE users ALTER COLUMN status SET NOT NULL;
```

Questions to answer before allowing it:

- Are there existing rows with `NULL` in `status`?
- Does the migration include a backfill if needed?
- Does the backend require a lock that could affect production traffic?

### Blocking

Blocking changes are destructive or ambiguous enough that they must not be hidden inside an ordinary migration review.

Examples:

- Drop a table.
- Drop a column.
- Rename a table without explicit rename intent.
- Change a ClickHouse engine in a way that can collapse, aggregate, or otherwise lose row-level data.

Example operation:

```sql
DROP TABLE audit_log;
```

Why it blocks:

- The table data is removed.
- Rollback may recreate the table shape, but it cannot recover deleted rows.
- The operator must explicitly confirm this is intended.

## Examples by Severity

### Info: Add a Nullable Column

Model change:

```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(primary_key=True)
    display_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
```

Generated SQL:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

The upgrade does not invalidate existing rows. The rollback removes the newly added column.

### Warning: Add NOT NULL

Model change:

```python
status: Mapped[str] = mapped_column(String(32), nullable=False)
```

Generated SQL may include:

```sql
ALTER TABLE users ALTER COLUMN status SET NOT NULL;
```

This is structurally valid but operationally risky if existing rows contain `NULL`. The safe approach is to backfill first, verify, then enforce `NOT NULL`.

### Blocking: Drop a Column

Model change:

```python
# legacy_code was removed from the model
```

Generated SQL may include:

```sql
ALTER TABLE users DROP COLUMN legacy_code;
```

This removes stored data. A rollback can add the column back, but it cannot reconstruct the dropped values. The plan should be reviewed before applying.

## How It Fits in CI

A practical CI sequence is:

```text
Generate migration
  |
  v
Inspect plan and safety classification
  |
  v
Reject unexpected warnings or blocking changes
  |
  v
Apply to ephemeral database
  |
  v
Run convergence gate
```

The safety check runs before the convergence gate. Safety asks whether the operation is acceptable. Convergence asks whether the resulting schema is correct.

See [Convergence Gate](convergence-gate.md).

## Philosophy

DBWarden does not try to guess business intent. It can tell that a column drop destroys stored values, but it cannot know whether those values are obsolete. It can identify that a ClickHouse engine transition is lossy, but it cannot know whether the application already copied the data elsewhere.

For that reason, the safety classifier makes risk visible and requires acknowledgement. It is a correctness mechanism because it prevents accidental data loss from being treated as ordinary schema drift.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/correctness/sql-generation/
========================================================================

# SQL Generation

SQL generation is the stage where DBWarden turns a typed diff into a plain SQL migration file. The generated file contains backend-native SQL and is meant to be reviewed by humans.

There is no hidden runtime library inside the migration. Once generated, the migration is SQL.

## Pipeline

```text
Canonical model state + canonical snapshot state
  |
  v
Diff
  |
  v
Typed operations
  |
  v
Statement ordering
  |
  v
Backend ObjectHandler.emit()
  |
  v
MigrationStatement objects
  |
  v
Plain .sql migration file
```

Each stage has a narrow job.

## 1. Diff to Typed Operations

The diff engine does not start by writing SQL strings. It creates typed operations such as:

```text
create_table
drop_table
add_column
drop_column
alter_column_type
alter_column_nullable
add_index
drop_index
alter_ch_options
recreate_ch_table
```

Typed operations make safety checks, ordering, rollback generation, and backend dispatch possible.

Example operation shape:

```json
{
  "type": "add_column",
  "table": "users",
  "column": "display_name",
  "definition": {
    "type": "VARCHAR(255)",
    "nullable": true
  }
}
```

The exact internal representation may include backend-specific metadata, but the key idea is that the operation is structured before it becomes SQL.

## 2. Operation Ordering

Not every valid operation order is safe. DBWarden uses statement ordering to make generated migrations executable.

Examples:

- Create schemas before creating tables inside them.
- Rename tables before applying column changes to the renamed table.
- Create tables before creating indexes on those tables.
- Drop dependent objects before dropping objects they depend on.
- Reverse upgrade order for rollback statements.

The ordering layer uses `StatementOrder` and backend handler constraints so SQL is emitted in a stable sequence.

## 3. Backend Dispatch

Each backend owns SQL rendering for its supported object types. The dispatch step sends operations to the relevant `ObjectHandler.emit()` implementation.

```text
Operation: add_column
  |
  +--> PostgreSQL ColumnHandler.emit()
  |
  +--> MySQL column renderer
  |
  +--> SQLite-compatible renderer
  |
  +--> ClickHouse ChColumnHandler.emit()
```

This is why the same model can produce different SQL for different database configurations.

## PostgreSQL Example: Identity Column

A PostgreSQL model can declare identity behavior through backend metadata:

```python
class Account(Base):
    __tablename__ = "accounts"

    id: Mapped[int] = mapped_column(BigInteger, primary_key=True)

    class Meta(PGTableMeta):
        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=1000)
```

The PostgreSQL emitter understands that this is not a generic integer column. It must render PostgreSQL-native identity syntax:

```sql
CREATE TABLE accounts (
    id BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1000) PRIMARY KEY
);
```

Depending on the model metadata and backend version, PostgreSQL auto-increment behavior may use identity syntax or sequence-backed behavior. The important correctness property is that the PostgreSQL handler owns that decision. It is not guessed by a generic string formatter.

## PostgreSQL Example: Add a Column

Model change:

```python
display_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
```

Typed operation:

```text
add_column users.display_name varchar(255) nullable
```

Generated SQL:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

The output is intentionally simple. Reviewers can see exactly what will run.

## ClickHouse Example: MergeTree Table

A ClickHouse model can declare engine metadata:

```python
class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    user_id: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["event_date", "id"],
            partition_by="toYYYYMM(event_date)",
            settings={"index_granularity": "8192"},
        )
```

The ClickHouse emitter writes native engine syntax:

```sql
CREATE TABLE IF NOT EXISTS events (
    id UInt64,
    event_date Date,
    user_id UInt64
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_date, id)
SETTINGS index_granularity = 8192;
```

This is not a generic SQL dialect. ClickHouse engine clauses, settings, projections, skip indexes, and materialized view syntax are rendered by ClickHouse-specific handlers.

## Example Migration File

A generated migration contains both upgrade and rollback sections:

```sql
-- migration: primary__0007_add_user_display_name.sql

-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

CREATE INDEX idx_users_display_name ON users (display_name);

-- rollback
DROP INDEX IF EXISTS idx_users_display_name;

ALTER TABLE users DROP COLUMN display_name;
```

This file is self-contained. Applying it does not require importing DBWarden code inside the database. DBWarden is the generator and executor. The migration artifact is plain SQL.

## Why Emitters Are Trustworthy

The trust comes from separation and tests:

- Canonicalization decides whether a change is real.
- Diffing creates typed operations.
- Safety checks inspect the plan before execution.
- Backend handlers render native SQL for one backend and object family.
- Tests cover handler output, ordering, rollback metadata, and round-trip behavior.

A generic SQL generator would need to understand every backend rule at once. DBWarden instead gives each backend handler a focused responsibility.

## Link to Deterministic Diff

SQL generation depends on deterministic diffing. If canonicalization is stable, the operation list is stable. If the operation list is stable, SQL output is reviewable.

See [Deterministic Diff](deterministic-diff.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/aggregating-views/
========================================================================

# Aggregating views (`AggregatingMergeTree`)

## Overview

An aggregating view is a coherent triad:

1. An **AggregatingMergeTree target table** whose columns have
   `AggregateFunction(...)` types derived from aggregate expressions.
2. A **materialized view** that uses `<func>State(...)` combinators in its
   SELECT, `TO` the target.
3. The **source table** (referenced, not created; it must already exist).

Because the target column types and the MV combinators both derive from the
same single list of `AggExpr`, they are guaranteed consistent; the
correspondence that is manual and drift-prone in the string-SELECT form is
here derived and safe.

## Declaring aggregating views

Use `AggregatingView` as the base class, `aggregating_view()` in `Meta`:

```python
from sqlalchemy import func
from dbwarden.databases.clickhouse import AggregatingView, CHViewMeta, aggregating_view, agg

class EventsHourly(AggregatingView):
    __tablename__ = "events_hourly"

    class Meta(CHViewMeta):
        ch = aggregating_view(
            source=Event,
            group_by=[func.toStartOfHour(Event.event_time).label("hour")],
            aggregates=[
                agg.sum(Event.amount).as_("total_amount"),
                agg.count().as_("event_count"),
            ],
            order_by=["hour"],
            partition_by="toYYYYMM(hour)",
        )
```

This generates:

1. The target table `events_hourly` with `AggregatingMergeTree` engine and
   columns `hour DateTime`, `total_amount AggregateFunction(sum, Float64)`,
   `event_count AggregateFunction(count)`.
2. A materialized view `events_hourly_mv TO events_hourly` whose SELECT uses
   `sumState`, `countState`; automatically derived from the `AggExpr` list.
3. The MV reads FROM `Event`.

### The source parameter

`source` may be:

- A model class (preferred); its `__tablename__` is resolved at spec
  construction time.
- A string class name (forward reference); resolved at discovery time when
  all models are loaded.
- A bare table name string; used as-is.

When you pass a model class, rename safety is automatic: if the source table
is renamed, regeneration picks up the new name.

### Aggregate expressions

Every aggregate must have an alias (`.as_(name)`):

```python
aggregates=[
    agg.count().as_("event_count"),
    agg.sum("amount", "Float64").as_("total_amount"),
    agg.uniq("user_id").as_("unique_users"),
]
```

Supported aggregate functions include `sum`, `count`, `min`, `max`, `avg`,
`uniq`, `uniq_exact`, `any`, `any_last`, `groupArray`, `groupUniqArray`,
`quantile`, and the `raw` escape hatch for combinators not enumerated above.

## Configuring the target table

The `AggregatingViewSpec` is a frozen dataclass. Configure it via
`aggregating_view()` keyword arguments:

| Parameter | Description |
|-----------|-------------|
| `source` | Source model class or table name |
| `group_by` | GROUP BY keys: ColumnElement or string |
| `aggregates` | AggExpr list (each with `.as_()`) |
| `order_by` | ORDER BY for the target |
| `partition_by` | Optional PARTITION BY |
| `ttl` | Optional TTL expression(s) |
| `settings` | Optional engine SETTINGS |

## Full example

```python
from sqlalchemy import func, String
from dbwarden.databases.clickhouse import (
    AggregatingView, CHViewMeta, aggregating_view, agg,
)

class EventStats(AggregatingView):
    __tablename__ = "event_stats"

    class Meta(CHViewMeta):
        ch = aggregating_view(
            source=PageView,
            group_by=[
                PageView.url.label("url"),
                func.toDate(PageView.viewed_at).label("day"),
            ],
            aggregates=[
                agg.count().as_("views"),
                agg.uniq(PageView.session_id).as_("unique_sessions"),
                agg.sum(PageView.duration).as_("total_duration"),
            ],
            order_by=["url", "day"],
            partition_by="toYYYYMM(day)",
            ttl="day + INTERVAL 90 DAY DELETE",
        )
```

## Populating an aggregating view

Use the `populate()` helper from `data_ops` to generate an `INSERT ... SELECT`
DataOp that backfills the target table:

```python
from dbwarden.databases.clickhouse import data_ops, AggregatingView

pop = data_ops.populate(EventStats.Meta.ch)
```

This produces a `DataOp` whose forward SQL is equivalent to:

```sql
INSERT INTO event_stats
SELECT
    url,
    toDate(viewed_at) AS day,
    countState() AS views,
    uniqState(session_id) AS unique_sessions,
    sumState(duration) AS total_duration
FROM page_view
GROUP BY url, toDate(viewed_at)
```

## Discoverability

`AggregatingView` subclasses are automatically registered in
`ChView._ch_view_registry` and discovered by `ch_view_tables_from_models()`.
They contribute both the aggregating target model and the materialized view to
the model list.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/columns-types/
========================================================================

# Columns and types

## Column-level Meta

Use `CHColumnMeta` inner classes on the model, named after the column:

```python
from dbwarden.databases.clickhouse import CHTableMeta, CHColumnMeta, ch

class Event(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    payload: Mapped[str] = mapped_column()
    event_time: Mapped[datetime] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="event_time")

        class payload(CHColumnMeta):
            ch = ch.field(codec="ZSTD(3)")

        class event_time(CHColumnMeta):
            ch = ch.field(default_expression="now()")
```

Generated DDL:

```sql
CREATE TABLE events (
    id Int64,
    payload String CODEC(ZSTD(3)),
    event_time DateTime DEFAULT now()
) ENGINE = MergeTree() ORDER BY event_time
```

## Additional model examples

### Column with codec and LowCardinality

```python
class UserSession(Base):
    __tablename__ = "user_sessions"

    user_id: Mapped[int] = mapped_column(primary_key=True)
    browser: Mapped[str] = mapped_column()
    ip: Mapped[str] = mapped_column()
    duration: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="user_id")

        class browser(CHColumnMeta):
            ch = ch.field(
                codec="ZSTD(3)",
                low_cardinality=True,
            )

        class ip(CHColumnMeta):
            ch = ch.field(
                codec="LZ4HC(9)",
                nullable=True,
            )

        class duration(CHColumnMeta):
            ch = ch.field(
                default_expression="0",
                ttl="now() - toIntervalDay(30)",
            )
```

Generated DDL:

```sql
CREATE TABLE user_sessions (
    user_id Int64,
    browser LowCardinality(String) CODEC(ZSTD(3)),
    ip Nullable(String) CODEC(LZ4HC(9)),
    duration Int64 DEFAULT 0 TTL now() - toIntervalDay(30)
) ENGINE = MergeTree() ORDER BY user_id
```

### Column with alias and comment

```python
class Metrics(Base):
    __tablename__ = "metrics"

    width: Mapped[float] = mapped_column()
    height: Mapped[float] = mapped_column()
    area: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="width")

        class area(CHColumnMeta):
            ch = ch.field(
                alias="width * height",
                comment="Calculated area in pixels",
            )
```

### REMOVE clause example

```python
# Removing a codec from a column
class Meta(CHTableMeta):
    ch = ch_table(engine=merge_tree(), order_by="id")

    class payload(CHColumnMeta):
        ch = ch.field(codec=None)  # emits MODIFY COLUMN payload REMOVE CODEC
```

## `ch.field()` options

| Parameter | Type | SQL | REMOVE form |
|-----------|------|-----|-------------|
| `codec` | `str` | `CODEC(ZSTD(3))` | `MODIFY COLUMN c REMOVE CODEC` |
| `default_expression` | `str` | `DEFAULT now()` | `MODIFY COLUMN c REMOVE DEFAULT` |
| `materialized` | `str` | `MATERIALIZED expr` | `MODIFY COLUMN c REMOVE MATERIALIZED` |
| `alias` | `str` | `ALIAS expr` | `MODIFY COLUMN c REMOVE ALIAS` |
| `ephemeral` | `str` | `EPHEMERAL expr` | `MODIFY COLUMN c REMOVE EPHEMERAL` |
| `ttl` | `str` | `TTL expr` | `MODIFY COLUMN c REMOVE TTL` |
| `low_cardinality` | `bool` | `LowCardinality(String)` | Wrapped into type |
| `nullable` | `bool` | `Nullable(String)` | Wrapped into type |
| `comment` | `str` | `COMMENT ON COLUMN` | `MODIFY COLUMN c REMOVE COMMENT` |

Setting a property to `None` (or omitting it) and then setting it to a value emits `MODIFY COLUMN ... <property>`. The reverse (removing a property) emits the `REMOVE` form. This is a write-only asymmetry in ClickHouse that dbwarden handles for you.

## Type normalization

SQLAlchemy types are normalized to ClickHouse native types:

| SQLAlchemy type | ClickHouse type |
|----------------|-----------------|
| `Integer`, `BIGINT` | `Int32`, `Int64` |
| `VARCHAR`, `String` | `String` |
| `FLOAT(53)`, `REAL` | `Float64`, `Float32` |
| `NUMERIC(p,s)` | `Decimal(p,s)` |
| `BOOLEAN` | `Bool` |
| `ARRAY(Integer)` | `Array(Int32)` |
| `Enum` | `Enum8` / `Enum16` |
| `UUID` | `UUID` |
| `JSON` | `JSON` |
| `DATETIME`, `DATETIME64` | `DateTime`, `DateTime64` |

## What changes are allowed

| Change | Safety | Notes |
|--------|--------|-------|
| Add column | INFO | Standard `ALTER TABLE ADD COLUMN` |
| Drop column | WARN | Requires `--force` |
| Type change (compatible) | INFO | e.g. `Int32` → `Int64` |
| Type change (incompatible) | CRITICAL | e.g. `String` → `Int64`, requires `--force` + recreate |
| Codec change | WARN | Relatively cheap |
| Default / Materialized / Alias change | WARN | |
| TTL change | WARN | |
| LowCardinality / Nullable toggle | CRITICAL | Requires `--force` |
| Comment change | INFO | |
| REMOVE any property | INFO | Emitted as `MODIFY COLUMN c REMOVE ...` |

## Rollback behavior

Column changes emit inverse operations. `ADD COLUMN` rolls back as `DROP COLUMN`. A `MODIFY COLUMN` rolls back as `MODIFY COLUMN` with the previous state.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/conventions/
========================================================================

# Conventions

## Canonicalization rules

The canonicalizer normalizes DDL before diffing. These normalizations apply:

| Rule | Example |
|------|---------|
| Whitespace normalization | `ORDER BY (a,b)` → `ORDER BY (a, b)` |
| Trailing comma removal | `(a, b,)` → `(a, b)` |
| Engine parameter normalization | ReplicatedMergeTree arguments |
| Settings key normalization | Underscore-vs-hyphen normalization |
| Type alias expansion | `INT` → `Int32`, `VARCHAR` → `String` |
| Nullable/LowCardinality wrapper normalization | Wrapping order |
| `ENGINE = Distributed(cluster, db, table)` | Shard/key delimiters |

## Defaults-as-absence

If a property matches the ClickHouse default, it is omitted from the emitted DDL. For example:

- `index_granularity = 8192` is the default and is not emitted unless explicitly set to a non-default value
- `SETTINGS` block is omitted entirely when all settings are at their defaults

This means the diff is clean: only non-default values appear in the DDL and the field is absent from declarations until overridden.

## Secrets-declare-only

Credentials are never diffed. See [Named collections](named-collections.md).

- Values of `password`, `sasl_password`, `secret_access_key` etc. are not compared between model and server
- Named collection declarations list the **keys** that should exist, not the secret values
- RBAC `identified_by` values are not stored in the model at all

## Additional model examples

### Builder vs raw dict comparison

```python
# Builder path (preferred): typed, validated, autocompletable
ch = ch_table(
    engine=replicated_merge_tree("/zk/path", "{replica}"),
    order_by=["ts", "id"],
    partition_by="toYYYYMM(ts)",
    settings=MergeTreeSettings({
        "index_granularity": 4096,
        "min_bytes_for_wide_part": "10485760",
    }),
)

# Raw dict path (escape hatch): same output, no validation
ch = {
    "engine": {
        "type": "ReplicatedMergeTree",
        "params": ["/zk/path", "{replica}"],
    },
    "order_by": ["ts", "id"],
    "partition_by": "toYYYYMM(ts)",
    "settings": {
        "index_granularity": "4096",
        "min_bytes_for_wide_part": "10485760",
    },
}
```

### Defaults-as-absence example

```python
# These two models emit identical DDL:
class Explicit(Base):
    __tablename__ = "t"
    x: Mapped[int] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="x",
                       settings=MergeTreeSettings(index_granularity=8192))

class Implicit(Base):
    __tablename__ = "t"
    x: Mapped[int] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="x")
        # index_granularity=8192 is the default, omitted from DDL
```

Both produce:

```sql
CREATE TABLE t (x Int64) ENGINE = MergeTree() ORDER BY x
```

## Two-door API

Every operation has two paths:

| Path | Purpose |
|------|---------|
| Builder (e.g., `ch_table()`, `ChRoleSpec()`) | Primary. Provides type-checking, autocomplete, and validation. Covers 95% of use cases. |
| Raw dict (e.g., `{"type": "MergeTree"}`) | Escape hatch. For infrequent or edge-case settings the builder doesn't expose. |

Raw dicts are passed through without validation:

```python
# Builder path
ch = ch_table(
    engine=merge_tree(),
    settings={"allow_experimental_inverted_index": 1},
)

# Raw path
ch = {"engine": {"type": "MergeTree"}, "settings": {"allow_experimental_inverted_index": "1"}}
```

Raw dicts are validated for structure (must match a known schema) but not for content correctness. The builder path is always preferred.

## Version support evidence

Every assertion about version support is backed by a test case in the audit harness:

```
tests/databases/clickhouse/audit/
├── 24.3/       # 31 cases, zero drift
└── 26.6/       # same 31 cases, zero drift
```

New ClickHouse versions are added by running the existing test suite against the new version. Zero branching in the canonicalizer means there is no per-version code path to update.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/data-operations/
========================================================================

# Data operations

Data operations (`data_op()`) are non-DDL statements that mutate data or trigger maintenance. They are not declared in the model: they are ad-hoc operations.

## Additional model examples

### Batch partition cleanup

```python
# Freeze, back up, then drop old partitions
from dbwarden.databases.clickhouse import data_op

# Freeze for backup
for m in ["2023-01", "2023-02", "2023-03"]:
    data_op(f"ALTER TABLE events FREEZE PARTITION '{m}'")

# After backup verified, drop
for m in ["2023-01", "2023-02", "2023-03"]:
    data_op(f"ALTER TABLE events DROP PARTITION '{m}'")
```

### Conditional mutation with setting override

```python
# Large mutation with timeout
data_op("""
    ALTER TABLE events
        UPDATE status = 'archived'
        WHERE event_date < '2020-01-01'
        SETTINGS mutations_sync = 2
""")
```

### OPTIMIZE with deduplicate

```python
# Force full merge and deduplication on all parts
data_op("OPTIMIZE TABLE events FINAL DEDUPLICATE")

# With column-specific deduplication
data_op("OPTIMIZE TABLE events FINAL DEDUPLICATE BY id, event_date")
```

### Multi-step migration with data ops

```python
def migrate_events():
    # 1. Create new table via migrate
    # 2. Backfill from old partition
    data_op("""
        ALTER TABLE events_v2
            REPLACE PARTITION '2024-01'
            FROM events_v1
    """)
    # 3. Drop old partition
    data_op("ALTER TABLE events_v1 DROP PARTITION '2024-01'")
    # 4. Verify
    data_op("OPTIMIZE TABLE events_v2 FINAL")
```

## Partition operations

```python
from dbwarden.databases.clickhouse import data_op

# Attach a detached partition
data_op("ALTER TABLE events ATTACH PARTITION '2024-01'")

# Replace one partition with another
data_op("ALTER TABLE events REPLACE PARTITION '2024-02' FROM staging_events")

# Drop a partition
data_op("ALTER TABLE events DROP PARTITION '2024-01'")

# Clear column in partition
data_op("ALTER TABLE events CLEAR COLUMN payload IN PARTITION '2024-01'")

# Freeze partition for backup
data_op("ALTER TABLE events FREEZE PARTITION '2024-01'")

# Unfreeze
data_op("ALTER TABLE events UNFREEZE PARTITION '2024-01'")
```

## Mutations

```python
# DELETE
data_op("ALTER TABLE events DELETE WHERE event_date < '2023-01-01'")

# UPDATE
data_op("ALTER TABLE events UPDATE payload = 'redacted' WHERE id = 123")
```

## OPTIMIZE

```python
# Merge parts
data_op("OPTIMIZE TABLE events FINAL")

# With partition
data_op("OPTIMIZE TABLE events PARTITION '2024-01' FINAL")

# Deduplicate
data_op("OPTIMIZE TABLE events FINAL DEDUPLICATE")
```

## POPULATE

```python
# Populate a materialized view
data_op("ALTER TABLE mv_name POPULATE")
```

This is a data-op rather than a DDL property because it is a write concern, not structural. See [Materialized views](materialized-views.md).

## Secret rotation

Named collection secrets are rotated through ClickHouse's secret store:

```python
# Refresh credentials from secret store
data_op("ALTER NAMED COLLECTION kafka_prod UPDATE sasl_password = SECRET 'new_secret_id'")
```

## Safety

| Operation | Safety | Notes |
|-----------|--------|-------|
| ATTACH PARTITION | INFO | Cheap metadata operation |
| REPLACE PARTITION | WARN | Overwrites target |
| DROP PARTITION | WARN | Data loss within a partition |
| CLEAR COLUMN | WARN | Data cleared for partition |
| DELETE mutation | WARN | Async, causes part rewrites |
| UPDATE mutation | WARN | Async, causes part rewrites |
| OPTIMIZE FINAL | INFO | Heavy IO |
| POPULATE | INFO | Inserts current data |
| Secret rotation | INFO | |

## Rollback behavior

Data operations are **not reversible** by dbwarden: they are ad-hoc mutations. Plan accordingly: test on staging, back up partitions before DROP or REPLACE.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/declaring-tables/
========================================================================

# Declaring tables

## The `ch_table()` builder (preferred path)

Use `ch_table()` inside a `class Meta(CHTableMeta)` block. This is the primary, recommended API:

```python
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.clickhouse import (
    CHTableMeta, merge_tree, ch_table,
)

class Base(DeclarativeBase):
    pass

class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    payload: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["event_date", "id"],
            partition_by="toYYYYMM(event_date)",
            ttl=["event_date + toIntervalYear(1)"],
            settings={"index_granularity": "8192"},
        )
```

`ch_table()` returns a `ChTableSpec` dataclass. Full signature:

| Parameter | Type | SQL |
|-----------|------|-----|
| `engine` | `ChEngineSpec` | `ENGINE = MergeTree()` |
| `order_by` | `str` or `list[str]` | `ORDER BY (col1, col2)` |
| `primary_key` | `str` or `list[str]` | `PRIMARY KEY (col1)` |
| `partition_by` | `str` | `PARTITION BY toYYYYMM(col)` |
| `sample_by` | `str` | `SAMPLE BY intHash64(col)` |
| `ttl` | `str` or `list[str]` | `TTL expr1, expr2` |
| `settings` | `MergeTreeSettings` dict | `SETTINGS key=value` |
| `projections` | `list[ProjectionSpec]` | `PROJECTION name (SELECT ...)` |
| `indexes` | `list[ChIndexSpec]` | `ALTER TABLE ... ADD INDEX ...` |

## Generated DDL

```sql
CREATE TABLE IF NOT EXISTS events (
    id Int64,
    event_date Date,
    payload String
) ENGINE = MergeTree()
ORDER BY (event_date, id)
PARTITION BY toYYYYMM(event_date)
TTL event_date + toIntervalYear(1)
SETTINGS index_granularity = 8192;
```

## Loose `ch_*` attrs (legacy)

Individual `ch_engine`, `ch_order_by`, etc. class attributes on `Meta` still work but are **deprecated**:

```python
class Meta(CHTableMeta):
    ch_engine = ChEngineSpec("MergeTree")       # deprecated
    ch_order_by = ["event_date", "id"]           # deprecated
    ch_partition_by = "toYYYYMM(event_date)"     # deprecated
```

**Migration path:** Replace all loose `ch_*` attrs with a single `ch = ch_table(...)` assignment. Both forms coexist in the same codebase during migration; the loose path emits a `DeprecationWarning`.

## Additional model examples

### Replicated table with custom settings

```python
class ReplicatedEvents(Base):
    __tablename__ = "replicated_events"

    id: Mapped[int] = mapped_column(primary_key=True)
    ts: Mapped[datetime] = mapped_column()
    value: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=replicated_merge_tree(
                "/clickhouse/tables/{shard}/replicated_events",
                "{replica}",
            ),
            order_by=["ts", "id"],
            partition_by="toYYYYMM(ts)",
            ttl=["ts + toIntervalMonth(3)"],
            settings={
                "index_granularity": 4096,
                "min_bytes_for_wide_part": 10485760,
            },
        )
```

### Table with multiple projections and indexes

```python
class AnalyticsEvents(Base):
    __tablename__ = "analytics_events"

    dt: Mapped[date] = mapped_column()
    user_id: Mapped[int] = mapped_column()
    event_type: Mapped[str] = mapped_column()
    amount: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=summing_merge_tree("amount"),
            order_by=["dt", "user_id", "event_type"],
            partition_by="toYYYYMM(dt)",
            settings={"allow_experimental_full_text_index": 1},
            projections=[
                ch_projection(
                    name="user_summary",
                    select=["user_id", "sum(amount)", "count()"],
                    group_by=["user_id"],
                ),
            ],
            indexes=[
                ch_index(
                    name="type_bloom",
                    expression="event_type",
                    type="bloom_filter(0.02)",
                ),
            ],
        )
```

### Distributed table

```python
class DistributedEvents(Base):
    __tablename__ = "distributed_events"

    id: Mapped[int] = mapped_column(primary_key=True)

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=distributed_engine(
                cluster="analytics_cluster",
                database="analytics",
                table="events",
                sharding_key="rand()",
            ),
        )
```

## What changes are allowed

See [Immutability](immutability.md) for the full rules.

| Property | Allowed changes |
|----------|----------------|
| `order_by` | Append-only extension |
| `settings` | Any key-value change |
| `ttl` | Any expression change |
| `projections` | Add/drop by name |
| `indexes` | Add/drop by name |
| `engine` | Only with `--force` (recreate) |
| `partition_by` | Never |
| `primary_key` | Only with `--force` (recreate) |
| `sample_by` | Never |

## Rollback behavior

Every `ALTER` and `CREATE` statement has a rollback. The rollback for a CREATE is DROP. The rollback for an ALTER is the inverse ALTER. For recreates, the rollback restores the original table.

See [Safety](safety.md) for the full rebuild pipeline.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/dictionaries/
========================================================================

# Dictionaries

## Declaration

Dictionaries are declared with flat `ch_dict_*` attributes on `class Meta`, not with builder functions. Set `ch_dictionary = True` to mark the model as a dictionary:

```python
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.clickhouse import CHTableMeta

class CountryLookup(Base):
    __tablename__ = "country_lookup"

    code: Mapped[str] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_primary_key = "code"
        ch_dict_layout = "flat"
        ch_dict_source = {
            "clickhouse": {"query": "SELECT code, name FROM source_countries"},
        }
        ch_dict_lifetime = 300
```

The five recognised attributes are `ch_dictionary`, `ch_dict_primary_key`, `ch_dict_layout`, `ch_dict_source`, and `ch_dict_lifetime`. Because `CHTableMeta` is validated at import time, a misspelled attribute raises `DBWardenConfigError` when the module loads.

There is also a `dictionary()` helper that builds a `DictSpec` directly, for code that constructs specs programmatically rather than declaring them on a model:

```python
from dbwarden.databases.clickhouse import dictionary

spec = dictionary(
    layout="hashed",
    source={"clickhouse": {"table": "dim_users"}},
    lifetime=300,
    primary_key="id",
)
```

## Additional model examples

### MySQL-sourced dictionary

```python
class MySQLCountry(Base):
    __tablename__ = "mysql_country"

    code: Mapped[str] = mapped_column()
    name: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_primary_key = "code"
        ch_dict_layout = "hashed"
        ch_dict_source = {
            "mysql": {
                "named_collection": "mysql_dict",
                "query": "SELECT iso_code, full_name FROM ref.countries",
            },
        }
        ch_dict_lifetime = "MIN 60 MAX 300"
```

### HTTP-sourced dictionary with complex key

```python
class CurrencyRate(Base):
    __tablename__ = "currency_rate"

    currency: Mapped[str] = mapped_column()
    rate: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_primary_key = "currency"
        ch_dict_layout = "cache"
        ch_dict_source = {
            "http": {
                "url": "https://api.example.com/rates",
                "format": "JSONEachRow",
            },
        }
        ch_dict_lifetime = 3600
```

### Range-hashed dictionary for time-based lookup

```python
class TaxRate(Base):
    __tablename__ = "tax_rate"

    region: Mapped[str] = mapped_column()
    rate: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_primary_key = ["region", "valid_from"]
        ch_dict_layout = "range_hashed"
        ch_dict_source = {
            "clickhouse": {
                "query": "SELECT region, valid_from, valid_to, rate FROM ref.tax_rates",
            },
        }
        ch_dict_lifetime = 86400
```

Usage in queries:

```sql
SELECT dictGet('tax_rate', 'rate', ('CA', today()))
```

## Source types

`ch_dict_source` is a dict keyed by source type, whose value carries that source's settings:

| Source type | `ch_dict_source` |
|-------------|------------------|
| ClickHouse | `{"clickhouse": {"query": "..."}}` |
| MySQL | `{"mysql": {...}}` |
| PostgreSQL | `{"postgresql": {...}}` |
| MongoDB | `{"mongodb": {...}}` |
| HTTP(S) | `{"http": {...}}` |
| Local file | `{"file": {...}}` |
| Executable | `{"executable": {...}}` |

For connection secrets, reference a named collection rather than inlining credentials:

```python
ch_dict_source = {"clickhouse": {"named_collection": "clickhouse_dict_source"}}
```

## Layout types

`ch_dict_layout` is the layout name as a string:

```python
ch_dict_layout = "flat"                # One key, single value
ch_dict_layout = "hashed"              # Hash table, all in memory
ch_dict_layout = "sparse_hashed"       # Like hashed but sparse
ch_dict_layout = "cache"               # LRU cache
ch_dict_layout = "complex_key_hashed"  # Composite keys
ch_dict_layout = "ip_trie"             # IP prefix matching
ch_dict_layout = "direct"              # No caching
ch_dict_layout = "range_hashed"        # Time ranges
```

## Lifetime

`ch_dict_lifetime` accepts an integer for a fixed interval, or a string for ClickHouse's ranged form:

```python
ch_dict_lifetime = 300              # Fixed interval, in seconds
ch_dict_lifetime = "MIN 300 MAX 600"  # Ranged
```

## What changes are allowed

| Change | Safety |
|--------|--------|
| Lifetime adjustment | INFO |
| Layout change | CRITICAL: requires recreate |
| Source connection change | INFO (named collection swap) |
| Query/SELECT change | WARN |
| Primary key change | CRITICAL: requires recreate |

## Rollback behavior

Dictionary changes that require a recreate follow the full pipeline. See [Safety](safety.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/engines-integration/
========================================================================

# Integration engines

dbwarden supports 13 integration engines. Credentials use the [named-collections](named-collections.md) declare-only pattern: no secret values are diffed.

## Kafka

```python
from dbwarden.databases.clickhouse import kafka

class Meta(CHTableMeta):
    ch = ch_table(
        engine=kafka(
            named_collection="kafka_prod",
            topic="events",
            format="JSONEachRow",
            group_name="dbwarden_consumer",
        ),
    )
```

Generated DDL:

```sql
CREATE TABLE kafka_events (
    payload String
) ENGINE = Kafka
SETTINGS kafka_named_collection = 'kafka_prod',
         kafka_topic_list = 'events',
         kafka_format = 'JSONEachRow',
         kafka_group_name = 'dbwarden_consumer';
```

Parameters that can be set directly (overriding the named collection):

| Factory parameter | DDL setting |
|------------------|-------------|
| `named_collection` | `kafka_named_collection` |
| `topic` | `kafka_topic_list` |
| `format` | `kafka_format` |
| `group_name` | `kafka_group_name` |
| `num_consumers` | `kafka_num_consumers` |
| `thread_per_consumer` | `kafka_thread_per_consumer` |
| `handle_error_mode` | `kafka_handle_error_mode` |
| `commit_every_batch` | `kafka_commit_every_batch` |

`KafkaSettings` is a fully-typed TypedDict for arbitrary Kafka engine settings:

```python
from dbwarden.databases.clickhouse import KafkaSettings

settings: KafkaSettings = {
    "kafka_max_block_size": 524288,
}
```

## S3

```python
from dbwarden.databases.clickhouse import s3

class Meta(CHTableMeta):
    ch = ch_table(
        engine=s3(
            named_collection="s3_prod",
            pattern="events/*.parquet",
            format="Parquet",
        ),
    )
```

Parameters:

| Parameter | DDL setting |
|-----------|-------------|
| `named_collection` | `s3_named_collection` |
| `pattern` | `url` (first positional) |
| `format` | `format` |
| `compression` | `compression` |

`S3Settings` for arbitrary settings. Key naming variants (`s3_*`, `s3queue_*`) are all typed.

## S3Queue

```python
from dbwarden.databases.clickhouse import s3_queue

class Meta(CHTableMeta):
    ch = ch_table(
        engine=s3_queue(
            named_collection="s3_prod",
            pattern="incoming/*.json",
            format="JSONEachRow",
        ),
    )
```

`S3QueueSettings` covers all s3queue-specific settings.

## RabbitMQ

```python
from dbwarden.databases.clickhouse import rabbitmq

class Meta(CHTableMeta):
    ch = ch_table(
        engine=rabbitmq(
            named_collection="rabbit_prod",
            format="JSONEachRow",
        ),
    )
```

`RabbitMQSettings`, typed TypedDict.

## NATS

```python
from dbwarden.databases.clickhouse import nats

class Meta(CHTableMeta):
    ch = ch_table(
        engine=nats(
            named_collection="nats_prod",
            format="JSONEachRow",
        ),
    )
```

`NatsSettings`, typed TypedDict.

## MySQL, PostgreSQL, MongoDB, Redis

```python
from dbwarden.databases.clickhouse import mysql_engine, postgresql_engine, mongodb, redis

# MySQL engine
engine = mysql_engine(
    named_collection="mysql_prod",
    query="SELECT * FROM source_db.table",
)

# PostgreSQL engine
engine = postgresql_engine(
    named_collection="pg_prod",
    query="SELECT * FROM source_schema.source_table",
)

# MongoDB engine
engine = mongodb(
    named_collection="mongo_prod",
    collection="source_collection",
)

# Redis engine
engine = redis(
    named_collection="redis_prod",
    key="prefix:*",
)
```

Each has an associated `*Settings` TypedDict for engine-specific settings.

## Additional model examples

### Named collection for multi-engine reuse

```python
# Single named collection reused by Kafka and S3 engines
named_collection(
    name="aws_prod",
    keys={
        "region": "us-east-1",
        "access_key_id": "AKIA...",
        # secret_access_key from secret store
    },
)

engine = kafka(
    named_collection="aws_prod",
    topic="events",
    format="JSONEachRow",
    group_name="ch_consumer",
)

engine2 = s3(
    named_collection="aws_prod",
    pattern="data/*.parquet",
    format="Parquet",
)
```

### S3Queue with complex settings

```python
engine = s3_queue(
    named_collection="aws_prod",
    pattern="incoming/*.json",
    format="JSONEachRow",
)

# With custom settings
settings: S3QueueSettings = {
    "s3queue_processing_threads": 8,
    "s3queue_polling_min_timeout_ms": 1000,
    "s3queue_polling_max_timeout_ms": 30000,
    "s3queue_tracked_files_limit": 100000,
}
```

### PostgreSQL engine with query

```python
engine = postgresql_engine(
    named_collection="pg_prod",
    query="SELECT id, name, created_at FROM public.users WHERE active = 1",
)
```

### URL engine with multiple formats

```python
# CSV
engine = url_engine(
    named_collection="http_data",
    format="CSV",
)

# With specific compression
engine = url_engine(
    named_collection="http_data",
    format="JSONEachRow",
    compression="gzip",
)
```

## URL

```python
from dbwarden.databases.clickhouse import url_engine

class Meta(CHTableMeta):
    ch = ch_table(
        engine=url_engine(
            named_collection="http_prod",
            format="CSV",
        ),
    )
```

`URLSettings`.

## File

```python
from dbwarden.databases.clickhouse import file_engine

class Meta(CHTableMeta):
    ch = ch_table(
        engine=file_engine(
            path="/var/lib/clickhouse/user_files/export.csv",
            format="CSV",
        ),
    )
```

## HDFS

```python
from dbwarden.databases.clickhouse import hdfs

class Meta(CHTableMeta):
    ch = ch_table(
        engine=hdfs(
            named_collection="hdfs_prod",
            format="Parquet",
        ),
    )
```

`HDFSSettings`.

## Per-engine settings TypedDicts

Every integration engine has its own `*Settings` TypedDict for arbitrary settings. These are all defined in `dbwarden.databases.clickhouse`:

| Engine | TypedDict |
|--------|-----------|
| Kafka | `KafkaSettings` |
| S3 | `S3Settings` |
| S3Queue | `S3QueueSettings` |
| RabbitMQ | `RabbitMQSettings` |
| NATS | `NatsSettings` |
| MySQL | `MySQLSettings` |
| PostgreSQL | `PostgreSQLSettings` |
| MongoDB | `MongoDBSettings` |
| Redis | `RedisSettings` |
| URL | `URLSettings` |
| HDFS | `HDFSSettings` |

## What changes are allowed

| Change | Safety |
|--------|--------|
| Named collection swap | CRITICAL (metadata only, not data) |
| Any setting | INFO: `ALTER TABLE MODIFY SETTING` |
| Format change | WARN: requires data re-ingestion |
| Pattern / query / key change | WARN |

## Rollback behavior

Settings changes revert via `RESET SETTING`. Named collection swaps require reversing the collection reference.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/engines-mergetree/
========================================================================

# MergeTree engine family

## Factories

All MergeTree variants have typed factory functions in `dbwarden.databases.clickhouse`:

```python
from dbwarden.databases.clickhouse import (
    merge_tree, replacing_merge_tree, replicated_merge_tree,
    summing_merge_tree, aggregating_merge_tree,
    collapsing_merge_tree, versioned_collapsing_merge_tree,
    graphite_merge_tree,
)
```

| Factory | Engine name | Signature |
|---------|-------------|-----------|
| `merge_tree()` | `MergeTree` | `()` |
| `replacing_merge_tree(ver?)` | `ReplacingMergeTree` | `(version_col: str \| None = None)` |
| `replicated_merge_tree(zk, replica, ...)` | `ReplicatedMergeTree` | `(zookeeper_path, replica_name, *args)` |
| `summing_merge_tree(...)` | `SummingMergeTree` | `(*columns: str)` |
| `aggregating_merge_tree()` | `AggregatingMergeTree` | `()` |
| `collapsing_merge_tree(sign)` | `CollapsingMergeTree` | `(sign_col: str)` |
| `versioned_collapsing_merge_tree(sign, ver)` | `VersionedCollapsingMergeTree` | `(sign_col, version_col)` |
| `graphite_merge_tree(section?)` | `GraphiteMergeTree` | `(config_section: str = "default")` |

Example:

```python
class Meta(CHTableMeta):
    ch = ch_table(
        engine=replicated_merge_tree(
            "/clickhouse/tables/events",
            "{replica}",
            "ver",
        ),
        order_by=["event_date", "id"],
    )
```

Generated DDL:

```sql
CREATE TABLE events (
    event_date Date,
    id Int64
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/events', '{replica}', 'ver')
ORDER BY (event_date, id)
```

## MergeTreeSettings

`ch_table(settings=MergeTreeSettings(...))` type-checks known MergeTree settings:

```python
from dbwarden.databases.clickhouse import MergeTreeSettings

settings: MergeTreeSettings = {
    "index_granularity": 8192,
    "ttl_only_drop_parts": True,
    "min_bytes_for_wide_part": 10485760,
}
```

Boolean values are automatically converted to `"0"` / `"1"`. Integer values are stringified. All values are rendered as strings before reaching the server.

## What changes are allowed

| Property | Allowed |
|----------|---------|
| Engine variant | Only with `--force` (full recreate) |
| ZK path / replica name | Only with `--force` |
| Settings | Any key-value via `MODIFY SETTING` (where supported by server) |
| ORDER BY | Append-only |
| PARTITION BY | Never |

## Additional model examples

### ReplacingMergeTree with version column

```python
class Product(Base):
    __tablename__ = "products"

    sku: Mapped[str] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column()
    price: Mapped[float] = mapped_column()
    updated_at: Mapped[datetime] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=replacing_merge_tree(version_col="updated_at"),
            order_by="sku",
        )
```

Deduplicates by `sku`, keeping the row with the latest `updated_at`.

### CollapsingMergeTree for mutable state

```python
class OrderState(Base):
    __tablename__ = "order_state"

    order_id: Mapped[str] = mapped_column(primary_key=True)
    status: Mapped[str] = mapped_column()
    amount: Mapped[float] = mapped_column()
    sign: Mapped[int8] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=collapsing_merge_tree(sign_col="sign"),
            order_by="order_id",
        )
```

Cancellations emit a row with `sign = -1` that collapses with the original `sign = 1`.

### SummingMergeTree

```python
class DailySummary(Base):
    __tablename__ = "daily_summary"

    dt: Mapped[date] = mapped_column()
    product: Mapped[str] = mapped_column()
    revenue: Mapped[float] = mapped_column()
    units: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=summing_merge_tree("revenue"),
            order_by=["dt", "product"],
        )
```

`revenue` and `units` are summed automatically during merge.

### GraphiteMergeTree

```python
class GraphiteMetrics(Base):
    __tablename__ = "graphite_metrics"

    path: Mapped[str] = mapped_column()
    value: Mapped[float] = mapped_column()
    timestamp: Mapped[datetime] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=graphite_merge_tree(config_section="rollup_default"),
            order_by=["path", "timestamp"],
            partition_by="toYYYYMM(timestamp)",
        )
```

## Rollback behavior

Engine changes with `--force` trigger the full recreate pipeline. See [Safety](safety.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/engines-special/
========================================================================

# Special engines

These engines do not participate in `ORDER BY`: they are storage-format-specific serverside readers.

## Factories

```python
from dbwarden.databases.clickhouse import (
    null, memory, merge,
    set_engine, join_engine, dictionary_engine,
    log, tiny_log, stripe_log,
)
```

## Additional model examples

### Null engine as MV sink

```python
class NullSink(Base):
    __tablename__ = "null_sink"

    payload: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=null(),
        )

class ViewFromSink(Base):
    __tablename__ = "view_from_sink"

    value: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by="value",
            ch_to_table="sink_dest",
            ch_select="SELECT count(*) AS value FROM null_sink",
        )
```

### Merge engine for partitioned read

```python
class AllEvents(Base):
    __tablename__ = "all_events"

    date: Mapped[date] = mapped_column()
    payload: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge(
                source_database="analytics",
                table_regex="events_202[0-9]_*",
            ),
        )
```

### Dictionary engine with explicit dictionary

```python
class CountryDict(Base):
    __tablename__ = "country_dict"

    code: Mapped[str] = mapped_column()
    name: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(engine=dictionary_engine())
```

The dictionary is declared separately via `ch_dictionary()`. See [Dictionaries](dictionaries.md).

## Null

```python
engine = null()
```

DDL: `ENGINE = Null`. Accepts any data and discards it. Used as the target of a materialized view that does its own aggregation.

## Memory

```python
engine = memory()
```

DDL: `ENGINE = Memory`. In-memory storage, lost on restart. Schema management only.

## Merge

```python
engine = merge(
    source_database="analytics",
    table_regex="events_.*",
)
```

DDL: `ENGINE = Merge('analytics', 'events_.*')`. A virtual table that reads from multiple tables whose names match the regex.

## Set

```python
engine = set_engine()
```

DDL: `ENGINE = Set`. Always in-memory. Use for IN-query acceleration.

## Join

```python
engine = join_engine(
    join_type="LEFT",
    strictness="ALL",
)
```

DDL: `ENGINE = Join(LEFT, ALL)`. Specialized for JOIN queries.

## Dictionary

```python
engine = dictionary_engine()
```

DDL: `ENGINE = Dictionary(<dict_name>)`. References a [Dictionary](dictionaries.md) object by name.

## Log, TinyLog, StripeLog

```python
engine = log()
engine = tiny_log()
engine = stripe_log()
```

DDLs: `ENGINE = Log`, `TinyLog`, `StripeLog`. Append-only file-based storage. No ORDER BY, no parts merging. StripeLog is multithreaded on read; TinyLog is the simplest.

## What changes are allowed

These engines have no ORDER BY, so immutability rules don't apply in the same way. An engine change (e.g., Memory → MergeTree) requires `--force` and a recreate.

| Change | Safety |
|--------|--------|
| Engine variant | CRITICAL with `--force` |
| Merge source/target | INFO |
| Join type/strictness | WARN |

## Rollback behavior

Engine changes trigger recreate. See [Safety](safety.md) for the pipeline.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/immutability/
========================================================================

# Immutability: what can never change

This page is the single most important thing to read before writing your first ClickHouse model. PG users expect most properties to be mutable via `ALTER`. ClickHouse is different: many properties are design-time commitments that can never be changed, and others require a full table rebuild.

## Model example: table with immutable properties

```python
class Orders(Base):
    __tablename__ = "orders"

    id: Mapped[int] = mapped_column(primary_key=True)
    created_at: Mapped[datetime] = mapped_column()
    amount: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["created_at", "id"],
            partition_by="toYYYYMM(created_at)",
            primary_key=["id"],
            sample_by="intHash64(id)",
        )
```

Once applied, `partition_by`, `primary_key`, `sample_by` and `engine` can never be altered. Only `order_by` can be extended by appending new columns: `["created_at", "id"]` → `["created_at", "id", "status"]`.

## What can never change

| Property | Constraint | Mechanism |
|----------|------------|-----------|
| `PARTITION BY` | Cannot be altered after creation | ClickHouse does not support `ALTER TABLE MODIFY PARTITION BY`. The only fix is a full table rebuild. |
| `PRIMARY KEY` | Cannot be altered | Unlike PG where you can `ALTER TABLE ... SET WITHOUT CLUSTER`, ClickHouse's primary key is baked into the storage order at creation. |
| `SAMPLE BY` | Cannot be altered | Same as partition by: set at CREATE time only. |
| `ENGINE` | Cannot be `ALTER`ed | Changing `MergeTree` to `ReplicatedMergeTree` (or vice versa) requires a full data copy. See below. |

## What extends only

| Property | Behavior | Example |
|----------|----------|---------|
| `ORDER BY` | New columns can be appended to the end. Existing columns cannot be removed or reordered. | `ORDER BY (a, b)` → `ORDER BY (a, b, c)` is valid. `ORDER BY (b, a)` is not. |

This is enforced by dbwarden: the differ refuses an `ORDER BY` change that is not an append-only extension. If you try, you get a CRITICAL-safety classification and must use `--force` to trigger a recreate.

## Model example: change requiring recreate

This ORDER BY change is refused by dbwarden:

```python
# Current model
class Meta(CHTableMeta):
    ch = ch_table(
        engine=replicated_merge_tree("/zk/orders", "{replica}"),
        order_by=["created_at", "id"],
    )

# Attempted change (reordered, not append-only)
class Meta(CHTableMeta):
    ch = ch_table(
        engine=replicated_merge_tree("/zk/orders", "{replica}"),
        order_by=["id", "created_at"],  # REORDERED: not append-only
    )
```

dbwarden emits: `CRITICAL: Changing ORDER BY from (created_at, id) to (id, created_at) requires --force`. The correct extension:

```python
class Meta(CHTableMeta):
    ch = ch_table(
        order_by=["created_at", "id", "status"],  # append-only: OK
    )
```

## What requires `--force` (recreate)

These properties trigger the recreate pipeline (DETACH source → CREATE new → INSERT INTO ... SELECT → RENAME → ATTACH) when changed:

| Change | Safety |
|--------|--------|
| Engine change (including ZK path / replica name) | CRITICAL |
| ORDER BY non-extension change (remove/reorder columns) | CRITICAL |
| PRIMARY KEY change | CRITICAL |
| PARTITION BY change | CRITICAL |
| SAMPLE BY change | INFO |
| Object type change (`table` ↔ `materialized_view`) | CRITICAL |
| MV target (`ch_to_table`) change | CRITICAL |
| Column type change (incompatible) | CRITICAL |
| LowCardinality / Nullable wrapper change | CRITICAL |

### Model example: full recreate with AggregateFunction

```python
# Source column type change triggers MV recreate
# Current: amount is Float64
class Meta(CHTableMeta):
    ch = ch_table(
        engine=merge_tree(),
        order_by="date",
        ch_select="SELECT date, agg.sumState(amount) AS state FROM events GROUP BY date",
    )

# If amount changes to Float32, the AggregateFunction signature changes:
#   AggregateFunction(sum, Float64) -> AggregateFunction(sum, Float32)
# This is CRITICAL and requires --force + recreate
```

**The 40GB-vs-500GB rebuild argument.** A 40GB table recreates in minutes. A 500GB table needs planning: provision a second table, backfill, verify, swap. dbwarden's recreate pipeline handles the orchestration but the cost is real. Test on a staging environment first.

## AggregateFunction signatures are incompatible state

`AggregateFunction(sum, Float64)` and `AggregateFunction(sum, Float32)` are different types. An MV that selects `sum(value)` where `value` is `Float64` produces `AggregateFunction(sum, Float64)`. If the source column type changes, the MV's aggregate type is locked: it cannot be `ALTER`ed to `Float32`. The only fix is to drop and recreate the MV.

This is not a dbwarden limitation: it is ClickHouse's columnar storage model. dbwarden will flag it as a CRITICAL change requiring a recreate.

## What dbwarden refuses entirely

dbwarden will not emit `ALTER TABLE ... MODIFY ORDER BY` for a non-extension change. It will refuse to emit `ALTER TABLE ... MODIFY PARTITION BY` (ClickHouse doesn't support it). It will refuse to emit an engine change without the recreate flag.

The error messages name the constraint and the flag required to override:
```
CRITICAL: Changing ORDER BY from (a, b) to (c, d) requires --force and
triggers a full table recreate (DETACH -> CREATE -> INSERT -> ATTACH).
```

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/
========================================================================

# ClickHouse

DBWarden treats ClickHouse as a first-class backend. Every natively supported feature is reverse-engineered, diffed, and emitted as correct DDL.

**Before reading further:** ClickHouse's object model is fundamentally different from PostgreSQL. What is a `SET` in PG is often a `CREATE` commitment in CH. Read [Immutability](immutability.md) first.

## Documentation Sections

- [Immutability](immutability.md) : What can never change, and what forces a table recreate
- [Conventions](conventions.md) : Canonicalization, defaults-as-absence, declare-only secrets, the two-door API
- [Declaring Tables](declaring-tables.md) : The `ch_table()` builder, generated DDL, and legacy `ch_*` attrs
- [Columns & Types](columns-types.md) : Column-level Meta, `ch.field()` options, type normalization
- [MergeTree Engines](engines-mergetree.md) : Engine factories, `MergeTreeSettings`, allowed changes, rollback behavior
- [Integration Engines](engines-integration.md) : Kafka, S3, S3Queue, RabbitMQ, NATS, and the other external sources
- [Special Engines](engines-special.md) : Distributed, Buffer, Join, Set, Memory, Null, Merge, and the Log family
- [Materialized Views](materialized-views.md) : The two MV shapes, refreshable MVs, `MODIFY QUERY` vs recreate
- [Aggregating Views](aggregating-views.md) : `AggregatingMergeTree` views, target tables, populating
- [Projections & Indexes](projections-indexes.md) : Projections, skip indexes, `MATERIALIZE` as a data operation
- [Dictionaries](dictionaries.md) : Declaration, source types, layout types, lifetime
- [Named Collections](named-collections.md) : Credential-bearing collections, declare-only by design (needs `dbwarden-ch-rbac`)
- [RBAC](rbac.md) : Roles, users, row policies, quotas, settings profiles, grants (needs `dbwarden-ch-rbac`)
- [Data Operations](data-operations.md) : Partition operations, mutations, `OPTIMIZE`, `POPULATE`
- [Safety Classification](safety.md) : Classification levels, `--force`, and the recreate pipeline

## Quick-start

Set up a table, a materialized view, and an aggregated table:

```python
from datetime import date
from sqlalchemy import func, Date
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.clickhouse import (
    CHTableMeta, CHViewMeta, ch_table, ch,
    merge_tree, materialized_view, kafka, aggregating_view, agg,
    AggregatingView, MaterializedView,
)

class Base(DeclarativeBase):
    pass

# 1. Source table
class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    amount: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["event_date", "id"],
            partition_by=func.toYYYYMM(Event.event_date),
        )

# 2. Materialized view: Mode A (class IS the target, MV is auto-generated)
class EventDaily(MaterializedView):
    __tablename__ = "event_daily"

    date: Mapped[date] = mapped_column(primary_key=True)
    total: Mapped[float] = mapped_column()
    cnt: Mapped[int] = mapped_column()

    class Meta(CHViewMeta):
        ch = materialized_view(
            select="SELECT event_date AS date, sum(amount) AS total, "
                   "count(*) AS cnt FROM events GROUP BY event_date",
            engine=merge_tree(),
            order_by=["date"],
        )

# 3. Aggregating view: sources from EventDaily model
class EventAggregated(AggregatingView):
    __tablename__ = "event_aggregated"

    class Meta(CHViewMeta):
        ch = aggregating_view(
            source=EventDaily,
            group_by=[EventDaily.date],
            aggregates=[
                agg.sum(EventDaily.total, "Float64").as_("state"),
            ],
            order_by=[EventDaily.date],
        )
```

Generate DDL and apply:

```bash
dbwarden make-migrations -d analytics
dbwarden migrate -d analytics
```

This produces: `events` (source MergeTree), `event_daily` (target MergeTree), `event_daily_mv` (MV TO event_daily), `event_aggregated` (AggregatingMergeTree target), and `event_aggregated_mv` (MV TO event_aggregated). Query the final table:

```sql
SELECT date, sumMerge(state) FROM event_aggregated GROUP BY date
```

## Version support

| Version | Status | Evidence |
|---------|--------|----------|
| 24.3 | Verified | 39 audit cases, zero drift |
| 26.6 (latest) | Verified | Same 39 cases, zero drift |

The canonicalizer has **zero version branching**: a single code path covers 24.3–26.6. This is measured fact from the multi-version audit harness, not an assumption.

## Capability matrix

| Category | Feature | Status |
|----------|---------|--------|
| Engines | MergeTree family (8 variants) | Done |
| | Distributed, Buffer | Done |
| | Kafka, S3, S3Queue, RabbitMQ, NATS | Done |
| | MySQL, PostgreSQL, MongoDB, Redis | Done |
| | URL, File, HDFS | Done |
| | Null, Memory, Merge, Set, Join, Dictionary | Done |
| | Log, TinyLog, StripeLog | Done |
| Column features | Codecs, TTL, DEFAULT/MATERIALIZED/ALIAS | Done |
| | LowCardinality, Nullable wrappers | Done |
| | Type normalization | Done |
| | REMOVE clauses (CODEC, TTL, DEFAULT, MATERIALIZED, ALIAS, COMMENT) | Done |
| Compiled expressions | `render_expr()` accepts SQLAlchemy `ColumnElement`/`ChRaw`/`str` | Done |
| | Expression fields in all specs accept `ColumnElement` | Done |
| Table features | ORDER BY, PRIMARY KEY, PARTITION BY, SAMPLE BY | Done |
| | TTL (table + column) | Done |
| | Settings | Done |
| | Comments (table + column) | Done |
| | Projections, skip indexes | Done |
| Materialized views | Class-based API (`materialized_view()` + `CHViewMeta`) | Done |
| | TO target, implicit `.inner`, refreshable | Done |
| | MODIFY QUERY vs recreate | Done |
| | POPULATE (data-op) | Done |
| Dictionaries | CREATE DICTIONARY via ch_dict_* | Done |
| Aggregating views | Class-based API (`aggregating_view()` + `CHViewMeta`) | Done |
| | `agg()` namespace, `-State`/`-Merge` correspondence | Done |
| | Auto-expansion: single declaration → target + MV | Done |
| | `derive_agg_target_columns()` utility | Done |
| RBAC | Roles, users, settings profiles, quotas, row policies, grants | Done |
| | `storage != 'users.xml'` filter | Done |
| | Drop gating (`--clickhouse-allow-drop-rbac`) | Done |
| Named collections | Key-set diffed, values declare-only | Done |
| Class-based views | `ChView`, `MaterializedView`, `AggregatingView` mixin bases | Done |
| | `CHViewMeta`: Meta class for view models | Done |
| | `get_all_ch_views()`: view discovery | Done |
| | `MaterializedViewSpec`: typed spec with expression compilation | Done |
| Safety | Classify options, column, and object changes | Done |
| | --force gating for destructive changes | Done |
| | Recreate pipeline (DETACH → CREATE → INSERT → ATTACH) | Done |
| Data ops | Partition ops, mutations, OPTIMIZE, POPULATE, secret rotation | Done |

## Deliberate exclusions

These are not gaps: they are deliberate boundaries, documented with reasoning so nobody wonders "why doesn't this work" or invents syntax to fill the vacuum.

| Feature | Reason |
|---------|--------|
| **Window View** | Requires `allow_experimental_window_view`. Experimental: DDL surface can change. Building a handler introduces the first version branch into the canonicalizer. Cost of waiting is near zero (MV-handler variant when it stabilizes). |
| **LIVE VIEW** | Experimental (`allow_experimental_live_view`), effectively abandoned, superseded by refreshable MVs (already supported). |
| **ANN (vector similarity) index** | Experimental (`allow_experimental_vector_similarity_index`). When stable it is a skip-index type addition: one Literal member, one audit case. Deferred, not excluded. |
| **Full-text index** | Experimental: the flag was renamed (`allow_experimental_inverted_index` → `allow_experimental_full_text_index`). The rename alone is the argument. Deferred. |
| **Replicated database engine** | dbwarden operates *within* a database, it does not provision databases. That is orchestration, same category as `config.xml`. |
| **SYSTEM commands** | Operational concern, not schema management. Not declarable in a model. |
| **Server config (`config.xml`)** | Infrastructure. Same boundary as PostgreSQL's `postgresql.conf`. |
| **Secret values** | Declare-only by design (see [named-collections](named-collections.md)). Values are not diffed. |

## Config keys

These are the **only** `database_config()` parameters for ClickHouse. Exact key shapes, documented because undocumented config keys are what assistants hallucinate.

**Every key below requires the `dbwarden-ch-rbac` plugin:** `dbwarden plugin add dbwarden-ch-rbac`. The plugin owns both these config keys and the handlers that emit their DDL, so declaring them without it installed raises `DBWardenConfigError` at config load.

```python
from dbwarden import database_config
from dbwarden.databases.clickhouse import (
    NamedCollectionSpec, named_collection,
    ChRoleSpec, ChUserSpec, ChRowPolicySpec,
    ChQuotaSpec, ChSettingsProfileSpec, ChGrantSpec,
)

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://...",
    ch_named_collections=[...],       # list[NamedCollectionSpec | dict]
    ch_roles=[...],                   # list[ChRoleSpec | dict]
    ch_users=[...],                   # list[ChUserSpec | dict]
    ch_row_policies=[...],            # list[ChRowPolicySpec | dict]
    ch_quotas=[...],                  # list[ChQuotaSpec | dict]
    ch_settings_profiles=[...],       # list[ChSettingsProfileSpec | dict]
    ch_grants=[...],                  # list[ChGrantSpec | dict]
)
```

## Model examples

### Partitioned time-series table with TTL

```python
class PageView(Base):
    __tablename__ = "page_views"

    id: Mapped[int] = mapped_column(primary_key=True)
    url: Mapped[str] = mapped_column()
    user_id: Mapped[int] = mapped_column()
    event_time: Mapped[datetime] = mapped_column()
    duration_ms: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=replicated_merge_tree("/zk/pv", "{replica}"),
            order_by=["event_time", "user_id"],
            partition_by="toYYYYMM(event_time)",
            ttl=["event_time + toIntervalMonth(6)"],
            settings={"index_granularity": 4096},
        )
```

### Table with codecs and column TTL

```python
from datetime import datetime
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.clickhouse import (
    CHColumnMeta, CHTableMeta, ch_table, ch,
    merge_tree,
)

class SensorReading(Base):
    __tablename__ = "sensor_readings"

    sensor_id: Mapped[str] = mapped_column()
    ts: Mapped[datetime] = mapped_column()
    temp: Mapped[float] = mapped_column()
    humidity: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["sensor_id", "ts"],
        )

        class temp(CHColumnMeta):
            ch = ch.field(codec="ZSTD(5)")

        class ts(CHColumnMeta):
            ch = ch.field(codec="DoubleDelta", ttl="ts + toIntervalDay(90)")

        class humidity(CHColumnMeta):
            ch = ch.field(ttl="ts + toIntervalDay(30)")
```

### Kafka ingestion with MV

```python
class KafkaEvents(Base):
    __tablename__ = "kafka_events"

    payload: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=kafka(
                named_collection="kafka_prod",
                topic="raw_events",
                format="JSONEachRow",
                group_name="dbwarden",
            ),
        )

class ParsedEvents(MaterializedView):
    __tablename__ = "parsed_events"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select="""
                SELECT
                    JSONExtractString(payload, 'type') AS event_type,
                    JSONExtractFloat(payload, 'value') AS value,
                    JSONExtractDateTime(payload, 'ts') AS ts
                FROM kafka_events
            """,
            to="parsed_events_dest",
        )
```

### RBAC config

```python
database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://localhost:9000",
    ch_named_collections=[
        named_collection("ldap_auth", ldap_server="ldap.example.com"),
    ],
    ch_roles=[ChRoleSpec("analyst"), ChRoleSpec("engineer")],
    ch_users=[
        ChUserSpec(
            name="bob",
            named_collection="ldap_auth",
            default_role="analyst",
        ),
    ],
    ch_grants=[
        ChGrantSpec(privileges=["SELECT"], on="analytics.*", to="analyst"),
    ],
)
```

### S3-backed table with projection

```python
from datetime import datetime
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.clickhouse import (
    CHTableMeta, ch_table, s3, projection,
)

class S3Logs(Base):
    __tablename__ = "s3_logs"

    ts: Mapped[datetime] = mapped_column()
    level: Mapped[str] = mapped_column()
    message: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=s3(
                named_collection="s3_logs",
                path="logs/*.parquet",
                format="Parquet",
            ),
            projections=[
                projection(
                    name="by_level",
                    query="SELECT level, count() GROUP BY level",
                ),
            ],
        )
```

## Workflow examples

### Generate models from existing ClickHouse

```bash
# Point dbwarden at an existing ClickHouse instance
$ dbwarden generate-models -d analytics --url clickhouse://user:pass@host:9000/analytics

# Models are written to models/analytics/*.py with ch_table() / materialized_view()
# declarations that match the live schema exactly
# (verified: 39 audit cases, zero drift)
```

### Diff and preview a migration

```bash
# Change a model (e.g., add a column), then preview
$ cat >> models/analytics.py << 'EOF'
class Event(Base):
    __tablename__ = "events"
    # ... existing columns ...
    user_agent: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["event_date", "id"],
        )
EOF

$ dbwarden make-migrations --plan -d analytics

# Output:
# ALTER TABLE events ADD COLUMN user_agent String   (INFO)
```

### Handle a destructive change

```bash
# Model changes ORDER BY to a non-extension
$ dbwarden make-migrations --plan -d analytics

# Output:
# CRITICAL: Changing ORDER BY from (a, b) to (c) requires --force

# Review the plan, then apply
$ dbwarden make-migrations --plan --force -d analytics
# Shows the full recreate pipeline:
#   DETACH TABLE events
#   CREATE TABLE events_new ...
#   INSERT INTO events_new SELECT * FROM events
#   RENAME TABLE ...

$ dbwarden migrate --force -d analytics
```

### Deploy RBAC changes

```bash
# Add a new role and user, drop an old one
$ dbwarden make-migrations --plan -d analytics

# Output:
# CREATE ROLE IF NOT EXISTS engineer        (INFO)
# CREATE USER IF NOT EXISTS alice ...       (INFO)
# DROP USER bob                             (WARN: gated)

$ dbwarden migrate -d analytics --clickhouse-allow-drop-rbac
```

### Materialize a projection on existing data

```python
from dbwarden.databases.clickhouse import data_op

# After adding a projection to a table with existing data:
data_op(
    name="materialize_daily_agg",
    forward="ALTER TABLE events MATERIALIZE PROJECTION daily_agg",
)
```

## Verification workflow

```bash
# Reverse-engineer a live database
$ dbwarden generate-models -d analytics

# Preview ops without writing files
$ dbwarden make-migrations --plan -d analytics

# Write and apply
$ dbwarden make-migrations -d analytics
$ dbwarden migrate -d analytics
```

Always review auto-generated migrations before applying, especially for destructive changes flagged as CRITICAL.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/materialized-views/
========================================================================

# Materialized views

## Plain MVs: two shapes

### Mode A: Class IS the target table (preferred)

The view class IS the target table; `__tablename__` is the table name.
The MV is auto-generated as `f"{__tablename__}_mv"`.  Columns, engine, and
`order_by` are required for Mode A (engine is enforced; columns and
`order_by` are strongly recommended but not strictly validated).

Columns are declared via `mapped_column` on the class.  Although no
SQLAlchemy `Base` is needed; the class is NOT a SQLAlchemy model, and
`session.query(ClassName)` will *not* work; the column descriptors are
read from `cls.__dict__` by the discovery pipeline.

```python
from datetime import date
from sqlalchemy import func
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.clickhouse import MaterializedView, CHViewMeta, materialized_view, merge_tree

class EventCount(MaterializedView):
    __tablename__ = "event_counts"

    date: Mapped[date] = mapped_column(primary_key=True)
    count: Mapped[int] = mapped_column()

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            engine=merge_tree(),
            order_by=["date"],
        )
```

Generated DDL:

```sql
CREATE TABLE event_counts (date Date, count Int64)
ENGINE = MergeTree() ORDER BY date

CREATE MATERIALIZED VIEW event_counts_mv TO event_counts
AS SELECT sum(amount) AS total FROM events
```

### Mode B: Explicit `to=` target

The class IS the MV; it writes to a pre-existing target table.  No columns,
no engine, no `order_by`; the target owns its own storage.

```python
from sqlalchemy import func
from dbwarden.databases.clickhouse import MaterializedView, CHViewMeta, materialized_view

class EventCountMV(MaterializedView):
    __tablename__ = "event_counts_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            to="events_dest",
        )
```

Generated DDL:

```sql
CREATE MATERIALIZED VIEW event_counts_mv TO events_dest
AS SELECT sum(amount) AS total FROM events
```

## Refreshable MVs (24.3+)

```python
class DailyRollupMV(MaterializedView):
    __tablename__ = "daily_rollup_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            to="rollup_dest",
            refresh="EVERY 3600 SECONDS",
        )
```

Generated DDL:

```sql
CREATE MATERIALIZED VIEW daily_rollup_mv TO rollup_dest
REFRESH EVERY 3600 SECONDS
AS SELECT sum(amount) AS total FROM events
```

Refreshable MVs (introduced in CH 24.3) replace LIVE VIEW and support:

- Periodic refresh (`EVERY n SECONDS`)
- Refresh dependencies (`DEPENDS ON`) embedded in the `refresh=` string
- Empty vs populating initial state

These options are combined in a single `refresh` string:

```python
refresh="EVERY 3600 SECONDS DEPENDS ON my_other_mv"
```

Refresh on cluster is configured at the deployment level via `--cluster-mode`.

## Additional model examples

### Refreshable MV with DEPENDS ON

```python
class HourlyRollupMV(MaterializedView):
    __tablename__ = "hourly_rollup_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            to="hourly_rollup_dest",
            refresh="EVERY 300 SECONDS DEPENDS ON daily_rollup_mv",
        )
```

Generated DDL:

```sql
CREATE MATERIALIZED VIEW hourly_rollup_mv TO hourly_rollup_dest
REFRESH EVERY 300 SECONDS DEPENDS ON daily_rollup_mv
AS SELECT sum(amount) AS total FROM events
```

### MV on clustered setup

```python
class ClusterMV(MaterializedView):
    __tablename__ = "cluster_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select="SELECT hostName() AS node, count(*) AS cnt FROM events",
            to="cluster_mv_dest",
        )
```

### MV chaining (MV reading from MV)

```python
# First MV: raw -> hourly
class RawToHourlyMV(MaterializedView):
    __tablename__ = "raw_to_hourly_mv"
    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Raw.value).label("total"),
            to="hourly_dest",
        )

# Second MV: hourly -> daily
class HourlyToDailyMV(MaterializedView):
    __tablename__ = "hourly_to_daily_mv"
    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(HourlyDest.total).label("total"),
            to="daily_dest",
        )
```

## What the `.inner` table is

When an MV has no `TO target` (deprecated module-level form only), ClickHouse
creates a hidden table named ``.inner.<view_name>`` with the MV's result
schema.  dbwarden reverse-engineers this table during ``generate-models``.

The class API **does not support** implicit ``.inner.`` storage; every
MV must either be the target (Mode A) or name a target (Mode B).  The
``.inner.`` form appears only when reverse-engineering legacy MVs that were
created outside dbwarden.

## `MODIFY QUERY` vs recreate

ClickHouse supports `ALTER TABLE ... MODIFY QUERY` for refreshable MVs. For
plain (non-refreshable) MVs, the query is immutable: any change requires a
DROP + CREATE.

dbwarden classifies this as:

| MV type | `MODIFY QUERY` | Safety |
|---------|----------------|--------|
| Refreshable | Supported | INFO |
| Plain (non-refreshable) | Not supported by CH | CRITICAL: requires `--force` |

## `POPULATE` as data operation

`POPULATE` is a one-time statement that inserts existing source data into the
MV on creation. dbwarden treats it as a
[data operation](data-operations.md), not a DDL property:

```python
from dbwarden.databases.clickhouse import data_ops
from dbwarden.databases.clickhouse.materialized_view import materialized_view

spec = materialized_view(
    name="event_counts_mv",
    select="SELECT sum(amount) AS total FROM events",
    to="events_dest",
)
pop = data_ops.populate(spec)
```

This is because `POPULATE` is a write concern, not a structural declaration:
it runs once during creation and changes nothing about the schema.

## What changes are allowed

| Change | Safety |
|--------|--------|
| Add/drop MV | WARN |
| Change `TO` target | CRITICAL: requires recreate |
| Change SELECT (non-refreshable) | CRITICAL: requires recreate |
| Change SELECT (refreshable) | INFO |
| Change refresh interval | INFO |
| Toggle POPULATE | Data-op, not structural |

## Rollback behavior

MV DROP is a rollback of MV CREATE. Data created by the MV is not restored by
rollback of the DDL: you must re-POPULATE.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/named-collections/
========================================================================

# Named collections

**Requires the `dbwarden-ch-rbac` plugin:** `dbwarden plugin add dbwarden-ch-rbac`. The `ch_named_collection` object handler ships in that plugin, not in core.

Named collections are the mechanism for declaring credentials without exposing secret values. They are declared in the config layer, referenced by name from engine and RBAC specs.

## Declaration

```python
from dbwarden import database_config
from dbwarden.databases.clickhouse import named_collection

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://...",
    ch_named_collections=[
        named_collection(
            name="kafka_prod",
            keys={
                "sasl_username": "kafka_user",
# sasl_password is NOT declared here: it comes from the
        # secret store. See "declare-only" below.
            },
        ),
        named_collection(
            name="s3_prod",
            keys={
                "region": "us-east-1",
                "access_key_id": "AKIA...",
            },
        ),
    ],
)
```

## Key-set diffed, values declare-only

Named collections are diffed on their **key set**: what keys are declared and what values are referenced. The **values themselves are never diffed**. This is the "declare-only" principle:

- `named_collection("kafka_prod", keys={"sasl_username": "kafka_user"})` declares that a collection named `kafka_prod` should have the key `sasl_username`.
- The value `"kafka_user"` is metadata for dbwarden's diff output but is **never compared** to the server state. Secret values (`password`, `sasl_password`, `secret_access_key`) are not declared at all: they come from ClickHouse's secret store.

This means dbwarden will detect that a key exists in a model but is missing from the server, and emit `CREATE NAMED COLLECTION ...`. But it will never emit `ALTER NAMED COLLECTION` with a changed password value: it can't know the real value.

## Additional model examples

### Named collection for Postgres engine

```python
named_collection(
    name="pg_source",
    keys={
        "connection_string": "postgresql://user:pass@pg-host:5432/db",
        "port": "5432",
    },
)
```

### Named collection with cluster and secret reference

```python
named_collection(
    name="kafka_secure",
    keys={
        "bootstrap_servers": "kafka-broker:9092",
        "sasl_mechanism": "SCRAM-SHA-256",
        "sasl_username": "ch_user",
        # sasl_password = SECRET(...) stored in ClickHouse secret store
        "security_protocol": "sasl_ssl",
    },
)
```

Engine usage:

```python
engine = kafka(
    named_collection="kafka_secure",
    topic="events",
    format="Avro",
    group_name="dbwarden",
)
```

### Dict config (raw path)

```python
database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://localhost:9000",
    ch_named_collections=[
        {
            "name": "s3_data",
            "keys": {
                "region": "us-east-1",
                "url": "https://s3.amazonaws.com/mybucket",
            },
        },
    ],
)
```

## Reference from engines

```python
engine = kafka(
    named_collection="kafka_prod",
    topic="events",
)
```

The engine gets credentials from the named collection. The named collection is referenced by name only in the engine settings (`kafka_named_collection`, `s3_named_collection`, etc.).

## Reference from RBAC

```python
ChUserSpec(
    named_collection="ldap_prod",
    ...
)
```

## What changes are allowed

| Change | Safety |
|--------|--------|
| Add named collection | INFO |
| Drop named collection | WARN (may break references) |
| Add key to declaration | INFO |
| Remove key from declaration | WARN |
| Change a non-secret value | INFO |
| Secret values | Not tracked |

## Rollback behavior

`DROP NAMED COLLECTION` rolls back as `CREATE NAMED COLLECTION`. Key changes roll back as inverse key changes.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/projections-indexes/
========================================================================

# Projections & indexes

Both projections and secondary indexes are acceleration structures, declared inside `ch_table()`.

## Projections

```python
from dbwarden.databases.clickhouse import projection

class Meta(CHTableMeta):
    ch = ch_table(
        engine=merge_tree(),
        order_by=["event_date", "id"],
        projections=[
            projection(
                name="daily_agg",
                query="SELECT event_date, count(), sum(amount) GROUP BY event_date",
            ),
        ],
    )
```

Generated DDL (the projection is part of CREATE TABLE or added via ALTER):

```sql
CREATE TABLE events (
    event_date Date,
    id Int64,
    amount Float64,
    PROJECTION daily_agg
    (
        SELECT event_date, count(), sum(amount)
        GROUP BY event_date
        ORDER BY event_date
    )
) ENGINE = MergeTree()
ORDER BY (event_date, id)
```

`ch_projection()` parameters:

| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | `str` | Projection name |
| `select` | `list[str]` | SELECT expressions |
| `group_by` | `list[str]` | GROUP BY columns |
| `order_by` | `list[str]` | ORDER BY within projection |

## Skip indexes

```python
from dbwarden.databases.clickhouse import skip_index

class Meta(CHTableMeta):
    ch = ch_table(
        engine=merge_tree(),
        order_by=["event_date", "id"],
        indexes=[
            skip_index(
                name="payload_idx",
                columns=["payload"],
                type="tokenbf_v1(1024, 3, 0)",
                granularity=1,
            ),
            ch_index(
                name="date_bloom",
                expression="event_date",
                type="bloom_filter(0.05)",
                granularity=4,
            ),
        ],
    )
```

Generated DDL:

```sql
CREATE TABLE events (
    payload String,
    INDEX payload_idx payload TYPE tokenbf_v1(1024, 3, 0) GRANULARITY 1,
    INDEX date_bloom event_date TYPE bloom_filter(0.05) GRANULARITY 4
) ENGINE = MergeTree()
ORDER BY (event_date, id)
```

`ch_index()` parameters:

| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | `str` | Index name |
| `expression` | `str` | Column or expression to index |
| `type` | `str` | Index type + parameters |
| `granularity` | `int` | Number of granules (default 1) |

Supported index types: `minmax`, `set(max_rows)`, `bloom_filter(false_positive)`, `ngrambf_v1(n, size, hashes, seed)`, `tokenbf_v1(size, hashes, seed)`, `hypothesis`, `inverted` (experimental), `vector_similarity` (experimental).

## Additional model examples

### Two projections on one table

```python
class Orders(Base):
    __tablename__ = "orders"

    dt: Mapped[date] = mapped_column()
    product: Mapped[str] = mapped_column()
    category: Mapped[str] = mapped_column()
    revenue: Mapped[float] = mapped_column()
    qty: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["dt", "product"],
            partition_by="toYYYYMM(dt)",
            projections=[
                ch_projection(
                    name="category_summary",
                    select=["category", "sum(revenue)", "sum(qty)"],
                    group_by=["category"],
                ),
                ch_projection(
                    name="product_top",
                    select=["product", "sum(revenue)"],
                    group_by=["product"],
                    order_by=["sum(revenue) DESC"],
                ),
            ],
        )
```

### Multiple index types

```python
class LogSearch(Base):
    __tablename__ = "log_search"

    ts: Mapped[datetime] = mapped_column()
    level: Mapped[str] = mapped_column()
    message: Mapped[str] = mapped_column()
    ip: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["ts", "ip"],
            indexes=[
                ch_index(
                    name="level_idx",
                    expression="level",
                    type="set(10)",
                    granularity=4,
                ),
                ch_index(
                    name="msg_bloom",
                    expression="message",
                    type="bloom_filter(0.01)",
                    granularity=1,
                ),
                ch_index(
                    name="ip_minmax",
                    expression="ip",
                    type="minmax",
                    granularity=8,
                ),
            ],
        )
```

### MATERIALIZE workflow

```python
# 1. Add projection to model
# 2. Generate migration (ADD PROJECTION is INFO)
# 3. Apply migration
dbwarden migrate -d analytics
# 4. Materialize on existing data
from dbwarden.databases.clickhouse import data_op
data_op("ALTER TABLE orders MATERIALIZE PROJECTION category_summary")
```

## MATERIALIZE as data operation

Indexes and projections written to new parts automatically, but existing parts need a `MATERIALIZE` operation:

```sql
ALTER TABLE events MATERIALIZE INDEX payload_idx
ALTER TABLE events MATERIALIZE PROJECTION daily_agg
```

dbwarden treats `MATERIALIZE` as a [data operation](data-operations.md), not DDL:

```python
with data_ops() as ops:
    ops.materialize_index("events", "payload_idx")
    ops.materialize_projection("events", "daily_agg")
```

## What changes are allowed

| Change | Safety |
|--------|--------|
| Add projection | INFO (new parts only; existing need MATERIALIZE) |
| Drop projection | WARN |
| Add index | INFO (new parts only; existing need MATERIALIZE) |
| Drop index | WARN |
| Change projection definition | CRITICAL: requires drop + recreate |
| Change index definition | CRITICAL: requires drop + recreate |

## Rollback behavior

Projections and indexes follow ALTER semantics: ADD rolls back as DROP and vice versa. MATERIALIZE is a data-op that is not structurally reversible: it is idempotent in practice.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/rbac/
========================================================================

# RBAC (roles, users, policies, quotas, profiles, grants)

**Requires the `dbwarden-ch-rbac` plugin:** `dbwarden plugin add dbwarden-ch-rbac`. The plugin owns both the `ch_*` RBAC config keys and the object handlers that emit their DDL. Core ships the spec dataclasses only, so declaring these keys without the plugin installed raises `DBWardenConfigError` when your `dbwarden.py` loads.

All RBAC objects are declared in the config layer via `database_config()`.

## Config keys

```python
from dbwarden import database_config
from dbwarden.databases.clickhouse import (
    ChRoleSpec, ChUserSpec, ChRowPolicySpec,
    ChQuotaSpec, ChSettingsProfileSpec, ChGrantSpec,
)

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://...",
    ch_roles=[...],
    ch_users=[...],
    ch_row_policies=[...],
    ch_quotas=[...],
    ch_settings_profiles=[...],
    ch_grants=[...],
)
```

## Roles

```python
ch_roles=[
    ChRoleSpec(name="analyst"),
    ChRoleSpec(name="admin"),
]
```

Generated DDL:

```sql
CREATE ROLE IF NOT EXISTS analyst;
CREATE ROLE IF NOT EXISTS admin;
```

No diff beyond name: roles are identifiers.

## Users

```python
ch_users=[
    ChUserSpec(
        name="alice",
        # Authentication: declare-only via named collection
        named_collection="ldap_prod",
        # Or directly (but values are not stored/compared):
        # identified_by="sha256_password", password="secret"
        default_role="analyst",
        settings={"max_memory_usage": 10000000000},
    ),
]
```

Generated DDL:

```sql
CREATE USER IF NOT EXISTS alice
DEFAULT ROLE analyst
SETTINGS max_memory_usage = 10000000000;
```

## Row policies

```python
ch_row_policies=[
    ChRowPolicySpec(
        name="analyst_filter",
        table="events",
        as_restriction="event_date >= '2024-01-01'",
    ),
]
```

Generated DDL:

```sql
CREATE ROW POLICY IF NOT EXISTS analyst_filter
ON events
AS PERMISSIVE
FOR SELECT USING event_date >= '2024-01-01'
TO analyst;
```

## Quotas

```python
ch_quotas=[
    ChQuotaSpec(
        name="monthly_reads",
        interval={"month": [1000000, 0, 0]},
    ),
]
```

Generated DDL:

```sql
CREATE QUOTA IF NOT EXISTS monthly_reads
FOR INTERVAL 1 MONTH
MAX QUERIES 1000000, ERRORS 0, RESULT ROWS 0
TO analyst;
```

## Settings profiles

```python
ch_settings_profiles=[
    ChSettingsProfileSpec(
        name="strict",
        settings={"max_memory_usage": 10000000000},
        constraints={"max_memory_usage": "READONLY"},
    ),
]
```

Generated DDL:

```sql
CREATE SETTINGS PROFILE IF NOT EXISTS strict
SETTINGS max_memory_usage = 10000000000 CONSTRAINED READONLY;
```

## Grants

```python
ch_grants=[
    ChGrantSpec(
        privileges=["SELECT", "INSERT"],
        on="analytics.events",
        to="analyst",
    ),
]
```

Generated DDL:

```sql
GRANT SELECT, INSERT ON analytics.events TO analyst;
```

## Additional model examples

### Complete RBAC config with multiple objects

```python
database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://localhost:9000",
    ch_named_collections=[
        named_collection("ldap_corp", keys={"ldap_server": "ldap.corp.example.com"}),
    ],
    ch_roles=[
        ChRoleSpec("readonly"),
        ChRoleSpec("analyst"),
        ChRoleSpec("admin"),
    ],
    ch_settings_profiles=[
        ChSettingsProfileSpec(
            name="strict_read",
            settings={
                "max_memory_usage": 5000000000,
                "max_result_rows": 10000,
            },
            constraints={
                "max_memory_usage": "READONLY",
                "max_result_rows": "READONLY",
            },
        ),
    ],
    ch_users=[
        ChUserSpec(
            name="alice",
            named_collection="ldap_corp",
            default_role="analyst",
            settings_profile="strict_read",
        ),
        ChUserSpec(
            name="bob",
            identified_by="sha256_password",
            password="changeme",  # declare-only: not diffed after creation
            default_role="readonly",
        ),
    ],
    ch_row_policies=[
        ChRowPolicySpec(
            name="analyst_filter",
            table="analytics.events",
            as_restriction="event_date >= '2024-01-01'",
        ),
    ],
    ch_quotas=[
        ChQuotaSpec(
            name="monthly_cap",
            interval={"month": [100000, 0, 0]},
        ),
    ],
    ch_grants=[
        ChGrantSpec(privileges=["SELECT"], on="analytics.*", to="readonly"),
        ChGrantSpec(privileges=["SELECT", "INSERT"], on="analytics.*", to="analyst"),
        ChGrantSpec(privileges=["ALL"], on="analytics.*", to="admin"),
    ],
)
```

### Dict config (raw path)

```python
database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://localhost:9000",
    ch_roles=[{"name": "analyst"}, {"name": "engineer"}],
    ch_users=[{
        "name": "carol",
        "identified_by": "sha256_password",
        "password": "s3cret",
        "default_role": "engineer",
    }],
)
```

## `storage != 'users.xml'` filter

dbwarden refuses to manage roles, users, or any RBAC object stored in `users.xml`:

```
ERROR: Cannot manage RBAC objects stored in users.xml.
Set storage = 'replicated' or use ClickHouse-native RBAC.
```

This is checked at config load time. If the server reports that RBAC storage is `users.xml`, all RBAC operations are skipped with a clear error.

## Drop gating

`DROP USER`, `DROP ROLE`, etc. require `--clickhouse-allow-drop-rbac`:

```bash
dbwarden migrate -d analytics --clickhouse-allow-drop-rbac
```

Without it, RBAC drop statements are skipped:

```
INFO: RBAC drop skipped: use --clickhouse-allow-drop-rbac to enable
```

This prevents accidental deactivation of users during migration runs.

## What changes are allowed

| Change | Safety |
|--------|--------|
| Add RBAC object | INFO |
| Drop RBAC object | WARN (gated by `--clickhouse-allow-drop-rbac`) |
| Modify user settings | INFO |
| Modify grant set | INFO |
| Change row policy expression | WARN |
| Change quota interval | INFO |
| Change settings profile | INFO |
| Named collection swap | INFO |

## Rollback behavior

Every RBAC CREATE has a DROP rollback and vice versa. Settings changes revert via `ALTER USER ... SETTINGS ...`.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/clickhouse/safety/
========================================================================

# Safety classification

Every change dbwarden detects is classified. Destructive operations require `--force` and emit warnings with reasons.

## Classification levels

| Level | Color | Behavior |
|-------|-------|----------|
| `INFO` | Green | Applied automatically. Safe metadata changes: ADD COLUMN, ADD INDEX, ADD PROJECTION, SETTING changes, TTL changes. |
| `WARN` | Yellow | Applied automatically but logged as a warning. Attention recommended: DROP COLUMN, DROP TABLE, DROP INDEX, mutations, partition DROP/REPLACE. |
| `CRITICAL` | Red | Skipped unless `--force` is passed. Requires explicit acknowledgement: engine changes, ORDER BY non-extension, PRIMARY KEY changes, MV TO target change, type incompatibility, LowCardinality/Nullable toggle. |

## The `--force` flag

```bash
# Preview what would run
dbwarden make-migrations --plan --force -d analytics

# Apply with force
dbwarden migrate --force -d analytics
```

`--force` is all-or-nothing for CRITICAL items in the plan. There is no per-item override.

## The recreate pipeline

When a CRITICAL change requires a full table rebuild, dbwarden executes:

```
DETACH TABLE source
CREATE TABLE source_new (...new definition...)
INSERT INTO source_new SELECT * FROM source
RENAME TABLE source TO source_old,
             source_new TO source
ATTACH TABLE source_old
```

Steps:
1. **DETACH**: unmounts the table metadata from the database (data remains on disk)
2. **CREATE new**: creates a table with the new definition
3. **INSERT INTO ... SELECT**: copies all data from old to new (blocking if `source` is still receiving writes)
4. **RENAME**: atomically swaps source and source_old under an exclusive lock
5. **ATTACH**: remounts the old table as a backup under its new name

After the pipeline, `source` has the new definition and `source_old` holds the original data. Verification steps:

```sql
SELECT count(*) FROM source
SELECT engine, create_table_query FROM system.tables WHERE name = 'source'
```

If something went wrong, swap back:

```sql
RENAME TABLE source TO source_broken,
             source_old TO source
```

## Rollback of a recreate

The rollback is the reverse pipeline:

```
DETACH TABLE source_old
CREATE TABLE source (...original definition...)
INSERT INTO source SELECT * FROM source_old
RENAME TABLE source TO source_old2,
             source_old TO source
ATTACH TABLE source_old2
```

This restores the original table and keeps the failed new table as `source_old2`.

## When to use `--force`

- **Staging/testing**: always use `--force` to verify the pipeline works
- **Production small tables** (< 40 GB): `--force` with a maintenance window
- **Production large tables** (> 40 GB): avoid `--force`. Instead, manually plan a zero-downtime migration using `clickhouse-copier` or double-write during backfill

## Additional model examples

### Model that triggers CRITICAL classification

```python
# Current table for reference:
class OldEvents(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by=["event_date", "id"])

# Change: remove a column from ORDER BY (non-extension)
class UpdatedEvents(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by=["event_date"])  # 'id' removed
```

Plan output:

```
CRITICAL: Changing ORDER BY from (event_date, id) to (event_date) requires --force
Apply with: dbwarden migrate --force -d analytics
```

### Safe change that passes without --force

```python
# Add a column and extend ORDER BY
class SafeEvents(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    status: Mapped[str] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by=["event_date", "id", "status"])
```

Plan output:

```
ALTER TABLE events ADD COLUMN status String   (INFO)
ALTER TABLE events MODIFY ORDER BY (event_date, id, status)   (INFO)
```

### Recreate pipeline example with verification

```bash
# 1. Preview the recreate
$ dbwarden make-migrations --plan --force -d analytics

# 2. Apply
$ dbwarden migrate --force -d analytics

# 3. Verify the new table
$ clickhouse-client -q "SELECT count(*) FROM events"
$ clickhouse-client -q "SELECT engine, create_table_query FROM system.tables WHERE name = 'events'"

# 4. If rollback needed:
$ clickhouse-client -q "RENAME TABLE events TO events_broken, events_old TO events"
```

## ClickHouse-native safety features

dbwarden also respects ClickHouse's server-side `allow_ddl` and `readonly` settings. If the server refuses a DDL statement, dbwarden logs the error and continues with the remaining plan items.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/
========================================================================

# Supported Databases

DBWarden supports PostgreSQL (the default and first-class backend), MySQL, MariaDB, SQLite, and ClickHouse.

A **round-trip** backend is one where DBWarden can both read schema (via `generate-models`) and write schema (via `make-migrations` / `migrate`).

## Backend Matrix

| Backend | `database_type` | Typical URL | Round-Trip |
|---------|------------------|-------------|------------|
| PostgreSQL | `postgresql` | `postgresql://user:pass@host:5432/db` | Yes |
| MySQL | `mysql` | `mysql://user:pass@host:3306/db` | Yes |
| MariaDB | `mariadb` | `mariadb://user:pass@host:3306/db` | No |
| ClickHouse | `clickhouse` | `clickhouse://user:pass@host:8123/db` | Yes |
| SQLite | `sqlite` | `sqlite:///./app.db` | Dev only |

## Optional Dependency Groups

When you install `dbwarden`, the `[postgres]` extra is included by default (providing the PostgreSQL driver). For other backends you must specify the corresponding extra:

| Extra | Command | Driver |
|-------|---------|--------|
| `[postgres]` | Included by default | `psycopg2-binary` |
| `[mysql]` | `uv add "dbwarden[mysql]"` | `pymysql` |
| `[mariadb]` | `uv add "dbwarden[mariadb]"` | `pymysql` |
| `[clickhouse]` | `uv add "dbwarden[clickhouse]"` | `clickhouse-connect` |

See [Installation](../installation.md) for full details.

## Config Examples

PostgreSQL:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
)
```

MySQL:

```python
legacy = database_config(
    database_name="legacy",
    database_type="mysql",
    database_url_sync="mysql://user:password@localhost:3306/legacy",
)
```

SQLite:

```python
dev = database_config(
    database_name="dev",
    database_type="sqlite",
    database_url_sync="sqlite:///./development.db",
)
```

ClickHouse:

```python
analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://user:password@localhost:8123/analytics",
)
```

## Internal Connection Handling

DBWarden uses SQLAlchemy engines, with backend-specific URL normalization where needed.

Conceptual flow:

```python
def get_engine(config):
    url = config.sqlalchemy_url
    if config.database_type == "clickhouse":
        url = normalize_clickhouse_dialect(url)
    return create_engine(url)
```

Connections include retry logic: `get_db_connection()` wraps engine connections with up to 5 attempts and exponential backoff when the database is temporarily unavailable (e.g. during a restart or network hiccup). Engines are cached and reused across calls.

For PostgreSQL schema support, set `pg_schema` in `database_config(...)`. DBWarden sets `search_path` on connection so all unqualified references use that schema:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/main",
    pg_schema="app",
)
```

At the model level, set `pg_schema` on `PGTableMeta` or `PGViewMeta` to scope a specific table or view to a schema. This takes precedence over the config-level `search_path`. See [PostgreSQL Deep Dive](postgresql/index.md) for full details.

## Development Database Strategy

Recommended pattern:

- Production-like primary DB (for example PostgreSQL)
- SQLite for dev DB via `dev_database_url`
- Run local commands with `--dev`

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

```bash
$ dbwarden --dev make-migrations "sync models" -d primary
$ dbwarden --dev migrate -d primary
```

## Translation Note

When targeting SQLite in dev mode, DBWarden translates unsupported backend-specific types/defaults.

- Unknown/unsupported types fallback to `TEXT` with warnings
- `--strict-translation` turns those warnings into errors

Details: [SQL Translation](../sql-translation.md)

## Backend-Specific Notes

Each backend has deep-dive documentation:

| Backend | Guide |
|---------|-------|
| PostgreSQL | [PostgreSQL Deep Dive](postgresql/index.md) |
| MySQL / MariaDB | [MySQL Deep Dive](mysql.md) |
| SQLite | [SQL Databases](sql-databases.md) |
| ClickHouse | [ClickHouse Deep Dive](clickhouse/index.md) |

### PostgreSQL

PostgreSQL is a **first-class backend** with full round-trip support. All metadata: identity columns, collation, storage, compression, generated columns, fillfactor, tablespace, inheritance, exclude constraints, deferrable FKs, and advanced index options, is captured by the snapshot, diffed correctly, and emitted as valid DDL.

See [PostgreSQL Deep Dive](postgresql/index.md) for the complete reference.

### MySQL

MySQL is a **first-class backend** with full round-trip support. All metadata: engine, charset, collation, row format, auto_increment, unsigned columns, ON UPDATE, and column comments, is captured by the snapshot, diffed correctly, and emitted as valid DDL.

Key MySQL DDL behavior:

- **DDL is NOT transactional**: each statement auto-commits; partial failure possible
- Column type/nullable changes use `MODIFY COLUMN` (requires full column definition)
- Table comments use `ALTER TABLE t COMMENT = '...'` (not `COMMENT ON`)
- Column comments use `MODIFY COLUMN ... COMMENT '...'` (full column definition preserved)
- Auto-increment toggle uses `MODIFY COLUMN ... AUTO_INCREMENT`
- FK drop uses `DROP FOREIGN KEY` (not `DROP CONSTRAINT`)

See [MySQL Deep Dive](mysql.md) for the complete reference.

### MariaDB

MariaDB is supported as a separate `database_type` (`mariadb`), but it does **not** have round-trip support. You can use MariaDB as a target database for migrations, but `generate-models` and full schema introspection are not available. Use `make-migrations` to write migrations manually.

See [MySQL Deep Dive](mysql.md) for MariaDB-specific notes.

### SQLite

- Great for local tests and dev loops via `--dev` mode
- Limited DDL: no `ALTER COLUMN TYPE`, no `SET/DROP NOT NULL`, no FK alterations
- `--safe-type-change` emits a comment (not supported)
- Type affinity differs from server databases
- See [SQL Translation](../sql-translation.md) for dev-mode type mapping

### ClickHouse

ClickHouse has full round-trip support: `generate-models` reads schema from a live ClickHouse server, and `make-migrations` / `migrate` auto-generates DDL for table operations.

- HTTP-based wire protocol; DBWarden uses ClickHouse client, not SQLAlchemy session
- DDL operations are mostly auto-generated: table rename, column type change, nullable and LowCardinality changes, projections, RBAC objects, and engine recreation. Placeholder rollback is refused by the strict rollback contract unless the migration is explicitly irreversible.
- Full engine metadata support via `class Meta(CHTableMeta)` with `ChEngineSpec`, `ProjectionSpec`, `CHColumnMeta`
- Supports materialized views, projections, dictionaries, replicated engines
- See [ClickHouse Deep Dive](clickhouse/index.md) for full details

## Recommended Verification Workflow

```bash
# local loop on dev DB
$ dbwarden --dev migrate -d primary

# pre-release validation on production-like DB
$ dbwarden migrate -d primary
$ dbwarden status -d primary
```

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/mysql/
========================================================================

# MySQL & MariaDB

DBWarden treats MySQL (and its fork MariaDB) as **first-class backends**: every natively supported feature is reverse-engineered, diffed, and emitted as correct DDL.

## First-Class Features

"First-class" means the round-trip is verified: reverse-engineer a live database with `generate-models`, feed the output back into `make-migrations`, and get **zero diff**.

```bash
# Step 1: reverse-engineer your live MySQL/MariaDB database
$ dbwarden generate-models -d primary

# Step 2: feed the generated models back in, zero diff
$ dbwarden make-migrations -d primary
# -> "No new migrations to generate"  (output is empty; your models match the DB exactly)
```

The following MySQL/MariaDB features are fully supported in this round-trip:

| Category | Features |
|----------|----------|
| Engine | `ENGINE=InnoDB`, `MyISAM`, etc. via `my_engine` |
| Charset & Collation | Table: `DEFAULT CHARACTER SET` / `COLLATE` via `my_charset`, `my_collate`. Column: per-column `CHARACTER SET` / `COLLATE` via `my.field(charset=..., collate=...)` |
| Row Format | `ROW_FORMAT=DYNAMIC`, `COMPACT`, `COMPRESSED`, `REDUNDANT` via `my_row_format` |
| Auto Increment | Table-level `AUTO_INCREMENT=N` via `my_auto_increment`. Column-level toggle via `autoincrement` field |
| Unsigned | `UNSIGNED` on integer columns via `my.field(unsigned=True)` |
| ON UPDATE | `ON UPDATE CURRENT_TIMESTAMP` via `my.field(on_update="CURRENT_TIMESTAMP")` |
| Comments | Table: `ALTER TABLE t COMMENT = '...'`. Column: `MODIFY COLUMN ... COMMENT '...'` (full column definition preserved) |
| Foreign Keys | `ON DELETE` / `ON UPDATE` options; DROP uses `DROP FOREIGN KEY` (MySQL syntax) |
| Indexes | Full index support; `USING BTREE / HASH` preserved |
| Auto-increment Lifecycle | Toggle autoincrement on integer PKs via `autoincrement` field: generates `MODIFY COLUMN ... AUTO_INCREMENT` |
| Type Normalization | `TINYINT(1)` -> `BOOLEAN`, `INT`, `BIGINT`, `VARCHAR(n)`, `TEXT`, `DATETIME`, `TIMESTAMP`, `YEAR`, `DECIMAL(p,s)`, `FLOAT`, `DOUBLE`, `BLOB`, `JSON`, `ENUM`, `SET` |

### MariaDB-Specific Features

| Category | Features |
|----------|----------|
| Page Compression | `PAGE_COMPRESSED=1` / `PAGE_COMPRESSION_LEVEL=N` via `mdb_page_compressed`, `mdb_page_compression_level` on `MdbTableMeta` |
| Invisible Columns | Column invisibility via `mdb.field(invisible=True)` on `MdbColumnMeta` |
| Sequences | `CREATE SEQUENCE` support via `mdb.field(sequence=...)` |

## Installation

Install with the MySQL driver:

```bash
uv add "dbwarden[mysql]"
```

Or with uv:

```bash
uv add "dbwarden[mysql]"
```

## Configuration

The MySQL backend is enabled by setting `database_type="mysql"` (or `database_type="mariadb"`) in your dbwarden config:

```python
from dbwarden import database_config

database_config(
    database_name="primary",
    default=True,
    database_type="mysql",
    database_url_sync="mysql+pymysql://user:password@localhost:3306/mydb",
)
```

The connection URL uses the `mysql+pymysql://` scheme (the `pymysql` driver is included via the `[mysql]` extra). You can also use any SQLAlchemy-compatible MySQL driver such as `mysql+mysqlconnector://`.

## Declaring Metadata

MySQL/MariaDB metadata is declared in a `class Meta` inner class on the model. This is the **only** supported surface: `mapped_column(info=...)` raises `DBWardenConfigError`.

### Table-Level Meta

Inherit from `MyTableMeta` on your `class Meta`:

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(MyTableMeta):
        my_engine = "InnoDB"
        my_charset = "utf8mb4"
        my_collate = "utf8mb4_unicode_ci"
        my_row_format = "DYNAMIC"
        my_auto_increment = 1000
        comment = "Core user accounts"
```

`MyTableMeta` inherits from `TableMeta`, which provides common attributes shared across all backends:

| Attribute | Type | SQL |
|-----------|------|-----|
| `comment` | `str` | `ALTER TABLE t COMMENT = '...'` |
| `indexes` | `list[dict]` | `CREATE INDEX ...` |
| `checks` | `list[dict]` | `ALTER TABLE t ADD CONSTRAINT ... CHECK (...)` |
| `uniques` | `list[dict]` | `ALTER TABLE t ADD CONSTRAINT ... UNIQUE (...)` |

MySQL-specific `MyTableMeta` attributes:

| Attribute | Type | SQL |
|-----------|------|-----|
| `my_engine` | `str` | `ALTER TABLE t ENGINE = name` |
| `my_charset` | `str` | `ALTER TABLE t DEFAULT CHARACTER SET name` |
| `my_collate` | `str` | `ALTER TABLE t COLLATE = name` |
| `my_row_format` | `str` | `ALTER TABLE t ROW_FORMAT = name` |
| `my_auto_increment` | `int` | `ALTER TABLE t AUTO_INCREMENT = N` |

For MariaDB, use `MdbTableMeta`:

```python
from dbwarden.databases.mariadb import MdbTableMeta

class Meta(MdbTableMeta):
    my_engine = "InnoDB"
    mdb_page_compressed = True
    mdb_page_compression_level = 3
```

MariaDB-specific `MdbTableMeta` attributes (in addition to all `MyTableMeta` attributes):

| Attribute | Type | SQL |
|-----------|------|-----|
| `mdb_page_compressed` | `bool` | `PAGE_COMPRESSED=1` |
| `mdb_page_compression_level` | `int` | `PAGE_COMPRESSION_LEVEL=N` |

### Column-Level Meta

Use `MyColumnMeta` inner classes for per-column metadata. The inner class must be named after the column. Use `my = my.field(...)` to set column-level options:

```python
from sqlalchemy import Integer, String, Text, TIMESTAMP
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta, MyColumnMeta, my

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    bio: Mapped[str] = mapped_column(Text)
    updated_at: Mapped[str] = mapped_column(TIMESTAMP)

    class Meta(MyTableMeta):
        class id(MyColumnMeta):
            comment = "Primary key"
            my = my.field(unsigned=True)

        class email(MyColumnMeta):
            my = my.field(charset="utf8mb4", collate="utf8mb4_unicode_ci")

        class updated_at(MyColumnMeta):
            my = my.field(on_update="CURRENT_TIMESTAMP")
```

`MyColumnMeta` includes common column attributes shared across all backends:

| Attribute | Type | SQL |
|-----------|------|-----|
| `comment` | `str` | `MODIFY COLUMN ... COMMENT '...'` |
| `public` | `bool` | Controls field visibility in `@auto_schema` (requires `dbwarden-fastapi` plugin) |
| `my` | `MyFieldSpec` | MySQL-specific column options (see table below) |

MySQL-specific `MyFieldSpec` fields (set via `my.field(...)`):

| Keyword | Type | SQL |
|---------|------|-----|
| `unsigned` | `bool` | `UNSIGNED` on integer columns |
| `charset` | `str` | `CHARACTER SET name` (per-column charset) |
| `collate` | `str` | `COLLATE name` (per-column collation) |
| `on_update` | `str` | `ON UPDATE CURRENT_TIMESTAMP` (typically on TIMESTAMP columns) |

For MariaDB, use `MdbColumnMeta` and `mdb.field(...)`:

```python
from dbwarden.databases.mariadb import MdbColumnMeta
from dbwarden.databases.mariadb import mdb

class Meta(MdbTableMeta):
    class id(MdbColumnMeta):
        mdb = mdb.field(invisible=True)
```

MariaDB-specific `MdbFieldSpec` fields (set via `mdb.field(...)`):

| Keyword | Type | SQL |
|---------|------|-----|
| `invisible` | `bool` | `ALTER TABLE ... ALTER COLUMN c SET INVISIBLE` |
| `sequence` | `str` | Sequence name for MariaDB sequence support |
| `unsigned` | `bool` | `UNSIGNED` on integer columns |
| `charset` | `str` | `CHARACTER SET name` |
| `collate` | `str` | `COLLATE name` |
| `on_update` | `str` | `ON UPDATE CURRENT_TIMESTAMP` |

### Foreign Key Options

Foreign key options (`ondelete`, `onupdate`) are captured from the database by `generate-models` and emitted in the `ForeignKey` constructor:

```python
from sqlalchemy import ForeignKey
from sqlalchemy.orm import Mapped, mapped_column

class OrderItem(Base):
    __tablename__ = "order_items"

    order_id: Mapped[int] = mapped_column(ForeignKey("orders.id", ondelete="CASCADE"), nullable=False)
```

### Model Example (Generated)

Here is the complete generated model output for a MySQL table with engine, charset, unsigned PK, ON UPDATE, and per-column charset:

```python
from sqlalchemy import BigInteger, Column, Integer, String, TIMESTAMP, Text, text
from sqlalchemy.orm import DeclarativeBase

Base = declarative_base()

from dbwarden.databases.mysql import MyColumnMeta, MyTableMeta, my

class User(Base):
    __tablename__ = 'users'

    id = Column('id', Integer, primary_key=True, nullable=False)
    email = Column('email', String(255), nullable=False)
    bio = Column('bio', Text)
    updated_at = Column('updated_at', TIMESTAMP, nullable=False, server_default=text('CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP'))

    class Meta(MyTableMeta):
        my_engine = 'InnoDB'
        my_charset = 'utf8mb4'
        my_collate = 'utf8mb4_unicode_ci'
        my_row_format = 'DYNAMIC'
        comment = 'Core user accounts'

        class id(MyColumnMeta):
            comment = 'Primary key'
            my = my.field(unsigned=True)

        class email(MyColumnMeta):
            my = my.field(charset='utf8mb4', collate='utf8mb4_unicode_ci')

        class updated_at(MyColumnMeta):
            my = my.field(on_update='CURRENT_TIMESTAMP')
```

## DDL Behavior

### DDL Is NOT Transactional

MySQL and MariaDB DDL is **non-transactional**: each DDL statement implicitly commits the current transaction. If a migration file contains multiple statements and one fails, the prior DDL cannot be rolled back. This makes MySQL/MariaDB more fragile than PostgreSQL for automated migration runs.

### Column Type Changes

Emits `ALTER TABLE t MODIFY COLUMN c newtype`. Unlike PostgreSQL, MySQL requires the full column definition on every `MODIFY COLUMN`. DBWarden handles this by re-emitting all column attributes (type, unsigned, nullable, default, comment, charset, collate, auto_increment) in a single statement:

```sql
ALTER TABLE users MODIFY COLUMN email VARCHAR(255) NOT NULL COMMENT 'User email';
```

### Column Nullable Changes

Emits `ALTER TABLE t MODIFY COLUMN c type [NULL | NOT NULL]`, again with the full column type.

### Column Meta Changes

When MySQL-specific column metadata changes (unsigned, charset, collate, on_update), DBWarden generates a full `MODIFY COLUMN` that preserves the column's type, nullable, default, comment, and autoincrement state:

```sql
ALTER TABLE users MODIFY COLUMN id INT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary key';
```

### Table Option Changes

MySQL table-level option changes generate individual `ALTER TABLE` statements:

| Change | Generated SQL |
|--------|---------------|
| Engine | `ALTER TABLE t ENGINE = InnoDB` |
| Charset | `ALTER TABLE t DEFAULT CHARACTER SET utf8mb4` |
| Collation | `ALTER TABLE t COLLATE = utf8mb4_unicode_ci` |
| Row Format | `ALTER TABLE t ROW_FORMAT = DYNAMIC` |
| Auto Increment | `ALTER TABLE t AUTO_INCREMENT = 1000` |

### Auto-increment Lifecycle

DBWarden supports toggling auto-increment on integer primary key columns. The `autoincrement` field in your model controls whether a column uses auto-increment:

```python
class User(Base):
    __tablename__ = "users"

    # Autoincrement enabled: same as default behavior
    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)

    class Meta(MyTableMeta):
        class id(MyColumnMeta):
            pass  # uses default autoincrement from model
```

To explicitly disable auto-increment on a PK column:

```python
class User(Base):
    __tablename__ = "users"

    # Plain integer PK: no auto-increment
    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=False)
```

**What happens when autoincrement changes:**

| Change | Generated SQL |
|--------|---------------|
| Add autoincrement | `ALTER TABLE t MODIFY COLUMN c INT NOT NULL AUTO_INCREMENT` |
| Remove autoincrement | `ALTER TABLE t MODIFY COLUMN c INT NOT NULL` |

### Comments

Unlike PostgreSQL, MySQL has no `COMMENT ON` syntax. DBWarden generates the correct MySQL syntax:

```sql
-- Table comment
ALTER TABLE users COMMENT = 'Core user accounts';

-- Column comment (full MODIFY COLUMN preserving all attributes)
ALTER TABLE users MODIFY COLUMN email VARCHAR(255) NOT NULL COMMENT 'User email address';
```

When a comment is cleared, MySQL syntax is used:

```sql
ALTER TABLE users COMMENT = '';
ALTER TABLE users MODIFY COLUMN email VARCHAR(255) NOT NULL COMMENT '';
```

## Snapshot Format

When `database_type` is `"mysql"` or `"mariadb"`, the snapshot captures MySQL-specific metadata in `my_column` and `my_table` blocks.

### Column Extras

```json
{
  "columns": {
    "id": {
      "name": "id",
      "type": "int",
      "nullable": false,
      "default": null,
      "autoincrement": true,
      "primary_key": true,
      "comment": "Primary key",
      "my_column": {
        "my_unsigned": true,
        "my_charset": null,
        "my_collate": null,
        "my_on_update": null
      }
    },
    "updated_at": {
      "name": "updated_at",
      "type": "timestamp",
      "nullable": false,
      "default": "CURRENT_TIMESTAMP",
      "autoincrement": false,
      "primary_key": false,
      "my_column": {
        "my_unsigned": false,
        "my_on_update": "CURRENT_TIMESTAMP"
      }
    }
  }
}
```

### Table Extras

```json
{
  "my_table": {
    "my_engine": "InnoDB",
    "my_charset": "utf8mb4",
    "my_collate": "utf8mb4_unicode_ci",
    "my_row_format": "Dynamic",
    "my_auto_increment": 1000
  }
}
```

For MariaDB, additional fields appear:

```json
{
  "my_table": {
    "mdb_page_compressed": false,
    "mdb_page_compression_level": null
  }
}
```

## Reverse Engineering

`generate-models` queries `information_schema.TABLES` and `information_schema.COLUMNS` to reverse-engineer all MySQL/MariaDB metadata. The emitted model uses `class Meta` with `MyTableMeta` and `MyColumnMeta` inner classes.

```bash
$ dbwarden generate-models -d primary
```

Generated output includes automatic detection of:
- Engine, charset, collation, row format from `information_schema.TABLES`
- Column unsigned, charset, collation, on_update from `information_schema.COLUMNS`
- Foreign key options (`ON DELETE`, `ON UPDATE`)
- Auto-increment columns
- Column comments

## Safety Classification

DBWarden classifies migration changes using the `Safety` enum:

```python
from dbwarden.engine.safety import Safety

assert Safety.SAFE == "SAFE"
assert Safety.INFO == "INFO"
assert Safety.WARN == "WARN"
assert Safety.CRITICAL == "CRITICAL"
```

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add column | `INFO` | None |
| Drop column | `WARNING` | `--force` |
| Change column type | `WARNING` | `--force` |
| Change column nullable | `WARNING` | `--force` |
| Change column comment | `INFO` | None |
| Change MySQL column meta | `WARNING` | `--force` |
| Change engine | `INFO` | None |
| Change charset | `INFO` | None |
| Change collation | `INFO` | None |
| Change row format | `INFO` | None |
| Change auto_increment | `INFO` | None |
| Change table comment | `INFO` | None |
| Add / drop index | `INFO` / `WARNING` | `--force` |
| Add / drop FK | `INFO` / `WARNING` | `--force` |

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/config-keys/
========================================================================

# Config Keys

PostgreSQL features are configured through keys in your `database_config(...)` call. These keys live alongside the connection URL and model paths.

**Most of these keys are contributed by plugins.** The plugin owns both the config key and the object handler that emits its DDL. Declaring a key whose plugin is not installed raises `DBWardenConfigError` when your `dbwarden.py` loads, naming the plugin to install. Keys that no plugin owns are rejected as unknown arguments, so typos fail immediately instead of being silently ignored.

| Config key | Required plugin |
|---|---|
| `pg_roles`, `pg_default_privileges` | `dbwarden-pgsql-rbac` |
| `pg_domains`, `pg_sequences`, `pg_composite_types` | `dbwarden-pgsql-types` |
| `pg_extensions`, `pg_functions`, `pg_triggers`, `pg_event_triggers`, `pg_extended_statistics` | `dbwarden-pgsql-extensions` |
| `pg_schema`, `pg_migration_lock_timeout` | None, core |

Install with `dbwarden plugin add <name>`.

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/mydb",
    # Config keys below
    pg_schema="app",
    pg_extensions=["uuid-ossp", "pgcrypto"],
    pg_roles=[...],
    pg_domains=[...],
    pg_sequences=[...],
    pg_functions=[...],
    pg_triggers=[...],
    pg_default_privileges=[...],
    pg_composite_types=[...],
    pg_extended_statistics=[...],
    pg_event_triggers=[...],
    pg_migration_lock_timeout=30,
)
```

## `pg_schema`

Default schema for unqualified table references. Sets the connection's `search_path`.

| Type | Default |
|------|---------|
| `str \| None` | `None` |

```python
pg_schema="app"
```

## `pg_extensions`

SQL extensions to create (equivalent to `CREATE EXTENSION IF NOT EXISTS`).

| Type | Default |
|------|---------|
| `list[str]` | `[]` |

```python
pg_extensions=["uuid-ossp", "pgcrypto", "postgis"]
```

Generated DDL: `CREATE EXTENSION IF NOT EXISTS "uuid-ossp";`

## `pg_roles`

Roles to create or alter. Each entry supports PostgreSQL role options.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_roles=[
    {"name": "app_user", "login": True, "password": "encrypted"},
    {"name": "readonly", "login": True, "connection_limit": 5},
]
```

Generated DDL: `CREATE ROLE app_user WITH LOGIN PASSWORD 'encrypted';`

### Role Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Role name |
| `login` | `bool` | `LOGIN` / `NOLOGIN` |
| `password` | `str` | `PASSWORD` (plain or `encrypted`) |
| `superuser` | `bool` | `SUPERUSER` / `NOSUPERUSER` |
| `createdb` | `bool` | `CREATEDB` / `NOCREATEDB` |
| `createrole` | `bool` | `CREATEROLE` / `NOCREATEROLE` |
| `inherit` | `bool` | `INHERIT` / `NOINHERIT` |
| `replication` | `bool` | `REPLICATION` / `NOREPLICATION` |
| `bypassrls` | `bool` | `BYPASSRLS` / `NOBYPASSRLS` |
| `connection_limit` | `int` | `CONNECTION LIMIT n` |
| `valid_until` | `str` | `VALID UNTIL 'timestamp'` |
| `in_role` | `str` | `IN ROLE parent_role` |
| `membership` | `list[str]` | `IN GROUP members` |

## `pg_domains`

Domain type declarations.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_domains=[
    {
        "name": "us_postal_code",
        "type": "text",
        "not_null": True,
        "check": "VALUE ~ '^\d{5}(-\d{4})?$'",
    },
]
```

Generated DDL:
```sql
CREATE DOMAIN us_postal_code AS text NOT NULL CHECK (VALUE ~ '^\d{5}(-\d{4})?$');
```

### Domain Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Domain name |
| `type` | `str` | Base type |
| `schema` | `str` | Schema (optional) |
| `default` | `str` | Default expression |
| `not_null` | `bool` | `NOT NULL` constraint |
| `check` | `str` | `CHECK` expression |

## `pg_sequences`

Sequence declarations.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_sequences=[
    {
        "name": "order_number_seq",
        "start": 1000,
        "increment": 1,
        "minvalue": 1,
        "maxvalue": 999999,
        "cycle": True,
        "owned_by": None,
    },
]
```

Generated DDL:
```sql
CREATE SEQUENCE order_number_seq START WITH 1000 INCREMENT BY 1 MINVALUE 1 MAXVALUE 999999 CYCLE;
```

### Sequence Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Sequence name |
| `schema` | `str` | Schema (optional) |
| `start` | `int` | `START WITH` |
| `increment` | `int` | `INCREMENT BY` |
| `minvalue` | `int` | `MINVALUE` |
| `maxvalue` | `int` | `MAXVALUE` |
| `cycle` | `bool` | `CYCLE` / `NO CYCLE` |
| `owned_by` | `str \| None` | `OWNED BY table.column` |

## `pg_functions`

Function declarations. Supports SQL, PL/pgSQL, and other languages.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_functions=[
    {
        "name": "update_timestamp",
        "language": "plpgsql",
        "body": """
            BEGIN
                NEW.updated_at = NOW();
                RETURN NEW;
            END;
        """,
        "returns": "trigger",
        "args": [],
    },
]
```

### Function Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Function name |
| `schema` | `str` | Schema (optional) |
| `language` | `str` | Language (`sql`, `plpgsql`, `c`, etc.) |
| `body` | `str` | Function body |
| `returns` | `str` | Return type |
| `args` | `list[dict]` | Arguments: `[{"name": "x", "type": "int"}]` |
| `volatility` | `str` | `VOLATILE`, `STABLE`, or `IMMUTABLE` |
| `security_definer` | `bool` | `SECURITY DEFINER` |
| `leakproof` | `bool` | `LEAKPROOF` |
| `parallel` | `str` | `PARALLEL UNSAFE`, `RESTRICTED`, or `SAFE` |
| `cost` | `int` | `COST` |
| `rows` | `int` | `ROWS` (for `RETURNS SETOF`) |

## `pg_triggers`

Trigger declarations. Each trigger references a table and an existing function.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_triggers=[
    {
        "name": "trg_users_updated_at",
        "table": "users",
        "function": "update_timestamp",
        "timing": "BEFORE",
        "events": ["UPDATE"],
        "for_each": "ROW",
    },
]
```

Generated DDL:
```sql
CREATE TRIGGER trg_users_updated_at BEFORE UPDATE ON users FOR EACH ROW EXECUTE FUNCTION update_timestamp();
```

### Trigger Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Trigger name |
| `table` | `str` | Table name |
| `schema` | `str` | Schema (optional) |
| `function` | `str` | Function to execute |
| `func_schema` | `str` | Function schema (optional) |
| `timing` | `str` | `BEFORE`, `AFTER`, or `INSTEAD OF` |
| `events` | `list[str]` | `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE` |
| `for_each` | `str` | `ROW` or `STATEMENT` |
| `condition` | `str` | `WHEN` clause (optional) |
| `args` | `list[str]` | Arguments passed to function |

## `pg_default_privileges`

Default privileges applied per schema, role, or object type.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_default_privileges=[
    {
        "schema": "public",
        "role": "app_user",
        "kind": "TABLES",
        "privileges": "SELECT, INSERT, UPDATE, DELETE",
    },
]
```

Generated DDL:
```sql
ALTER DEFAULT PRIVILEGES FOR ROLE app_user IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_user;
```

### Default Privilege Keys

| Key | Type | Description |
|-----|------|-------------|
| `schema` | `str` | Schema name |
| `role` | `str` | Target role |
| `kind` | `str` | Object type: `TABLES`, `SEQUENCES`, `FUNCTIONS`, `TYPES`, `SCHEMAS` |
| `privileges` | `str` | Comma-separated privileges |

## `pg_composite_types`

Composite type declarations.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_composite_types=[
    {
        "name": "address",
        "columns": [
            {"name": "street", "type": "text"},
            {"name": "city", "type": "text"},
            {"name": "zip", "type": "text"},
        ],
    },
]
```

Generated DDL:
```sql
CREATE TYPE address AS (street text, city text, zip text);
```

### Composite Type Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Type name |
| `schema` | `str` | Schema (optional) |
| `columns` | `list[dict]` | List of `{"name": ..., "type": ...}` |

## `pg_extended_statistics`

Extended statistics objects for the query planner (PG 14+).

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_extended_statistics=[
    {
        "name": "stats_users_email_city",
        "table": "users",
        "kinds": ["d", "f"],
        "columns": "email, city",
    },
]
```

Generated DDL:
```sql
CREATE STATISTICS stats_users_email_city (ndistinct, dependencies) ON email, city FROM users;
```

### Extended Statistics Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Statistics name |
| `table` | `str` | Table name |
| `schema` | `str` | Schema (optional) |
| `kinds` | `list[str]` | Kind codes: `d` (ndistinct), `f` (dependencies), `m` (MCV), `e` (expressions, PG 14+) |
| `columns` | `str` | Comma-separated column names |
| `expressions` | `list[str]` | Expression columns (PG 14+) |

## `pg_event_triggers`

Event triggers fired on DDL events at the database level.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_event_triggers=[
    {
        "name": "trg_ddl_audit",
        "event": "ddl_command_start",
        "function": "audit_ddl",
        "tags": ["CREATE TABLE", "ALTER TABLE"],
    },
]
```

Generated DDL:
```sql
CREATE EVENT TRIGGER trg_ddl_audit ON ddl_command_start WHEN TAG IN ('CREATE TABLE', 'ALTER TABLE') EXECUTE FUNCTION audit_ddl();
```

### Event Trigger Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Trigger name |
| `event` | `str` | Event: `ddl_command_start`, `ddl_command_end`, `sql_drop`, `table_rewrite` |
| `function` | `str` | Function to execute |
| `func_schema` | `str` | Function schema (optional) |
| `tags` | `list[str]` | DDL command tags to filter (optional) |
| `enabled` | `str` | `O` (enabled), `D` (disabled), `R` (replica), `A` (always) |

## `pg_migration_lock_timeout`

Timeout (seconds) for `LOCK TABLE` statements during migration DDL to prevent indefinite blocking.

| Type | Default |
|------|---------|
| `int \| None` | `None` |

```python
pg_migration_lock_timeout=30
```

When set, emits `SET lock_timeout = '30s'` before each migration statement.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/constraints/
========================================================================

# Constraints

Constraints are handled by `ConstraintHandler` during the DIFF phase.

## Foreign Keys

FK comparison uses a signature tuple: `(columns, ref_table, ref_columns, on_delete, on_update, deferrable, match)`.

### MATCH Behaviour

PostgreSQL supports three FK match modes: `MATCH FULL`, `MATCH PARTIAL`, and `MATCH SIMPLE` (default).

- `MATCH FULL` is explicitly emitted in DDL
- `MATCH PARTIAL` is explicitly emitted
- `MATCH SIMPLE` and absent match are canonicalized to the same representation (no match clause) to avoid churn on existing FKs

```sql
ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (ref_id) REFERENCES ref (id) MATCH FULL;
ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (ref_id) REFERENCES ref (id) ON DELETE CASCADE;
```

`MATCH PARTIAL` is defined by the SQL standard but PostgreSQL implements it identically to `MATCH SIMPLE`. Both allow some columns in a multi-column FK to be NULL; `MATCH FULL` requires all columns to be NULL or all to be non-NULL.

### ON DELETE / ON UPDATE

Supported actions: `CASCADE`, `SET NULL`, `SET DEFAULT`, `RESTRICT`, `NO ACTION`.

`NO ACTION` is the default. Unlike `RESTRICT`, `NO ACTION` allows deferrable constraints to defer checking to transaction end.

### DEFERRABLE

Support for `DEFERRABLE` / `NOT DEFERRABLE` and `INITIALLY DEFERRED` / `INITIALLY IMMEDIATE`.

A deferrable FK does not check referential integrity at statement end; checking is deferred to transaction end. Use `SET CONSTRAINTS ALL DEFERRED` at the start of a transaction to restore deferred checking.

```sql
BEGIN;
SET CONSTRAINTS all_fks DEFERRED;
-- Can now delete referenced rows before updating referencing rows
DELETE FROM orders WHERE id = 1;
UPDATE order_items SET order_id = 2 WHERE order_id = 1;
COMMIT; -- Constraints checked here
```

### NOT VALID / VALIDATE

FKs can be created `NOT VALID` and validated later with `VALIDATE CONSTRAINT`. This allows adding an FK to a live table without locking it for a full table scan during creation.

```sql
ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (ref_id) REFERENCES ref (id) NOT VALID;
-- later, during low-traffic window:
ALTER TABLE t VALIDATE CONSTRAINT fk;
```

### ALTER ALTER CONSTRAINT

PostgreSQL 9.4+ supports modifying constraint deferrability without drop/create:

```sql
ALTER TABLE t ALTER CONSTRAINT fk DEFERRABLE INITIALLY DEFERRED;
```

DBWarden detects when only the deferrability changed and emits `ALTER CONSTRAINT` instead of a drop+add cycle.

## Unique Constraints

Support for:

- `DEFERRABLE` / `INITIALLY DEFERRED`
- `INCLUDE` columns
- `NULLS NOT DISTINCT` (PG 15+)

A unique constraint with `NULLS NOT DISTINCT` treats NULLs as equal, so only one row can contain NULL.
If only the constraint name changes, DBWarden emits `ALTER TABLE ... RENAME CONSTRAINT ...` instead of dropping and recreating the unique constraint.

## Primary Key Constraints

Declared via SQLAlchemy's `primary_key=True` on `mapped_column`. The PK constraint is managed automatically:

| Operation | DDL |
|-----------|-----|
| Create | `PRIMARY KEY (col1, col2)` inline in `CREATE TABLE` |
| Add | `ALTER TABLE t ADD PRIMARY KEY (col1)` |
| Drop | `ALTER TABLE t DROP CONSTRAINT t_pkey` |

PK constraint options (set via the column/s in SQLAlchemy):

| Option | SQL | Notes |
|--------|-----|-------|
| Tablespace | `USING INDEX TABLESPACE name` | Separates PK index from table storage |
| Deferrable | `DEFERRABLE` / `INITIALLY DEFERRED` | PK can be deferred (rare) |

## Check Constraints

Support for:

- Arbitrary CHECK expressions
- `NO INHERIT` (constraint not applied to child tables)
- `NOT VALID` + `VALIDATE`

### NO INHERIT Example

```python
class Meta(PGTableMeta):
    pg_checks = [
        {"name": "ck_users_type", "sql": "user_type IN ('admin', 'user')", "no_inherit": True},
    ]
```

```sql
ALTER TABLE users ADD CONSTRAINT ck_users_type CHECK (user_type IN ('admin', 'user')) NO INHERIT;
```

`NO INHERIT` is useful when a parent table has a constraint that should not apply to child tables.

## Exclude Constraints

Declared via `pg_excludes` on `PGTableMeta`. Uses `PgTableHandler`.

```python
class Meta(PGTableMeta):
    pg_excludes = [
        {"name": "excl_room_booking", "expression": "USING gist (room_id WITH =, during WITH &&)"},
    ]
```

Exclusion constraints require a GiST or SP-GiST index. They prevent any two rows from having overlapping values in the specified columns. Common use cases: time-range booking, geo-spatial exclusion.

Diffing compares the full expression; any change in operator, access method, or columns produces a drop+add cycle.

## UniqueSpec vs unique=True on PgIndexSpec

PostgreSQL implements unique constraints as unique indexes under the hood. DBWarden supports both approaches, and the choice is a semantic one.

| Aspect | `UniqueSpec` in `uniques` / `pg_uniques` | `unique=True` on `PgIndexSpec` in `pg_indexes` |
|--------|------------------------------------------|------------------------------------------------|
| SQL generated | `ALTER TABLE ... ADD CONSTRAINT ... UNIQUE (...)` | `CREATE UNIQUE INDEX ... ON ... (...)` |
| Shows in `information_schema.table_constraints` | Yes | No |
| Targetable by FK references | Yes | No (needs the unique index to also be a constraint) |
| Business meaning | A constraint: a rule the data must satisfy | An index: a data structure for performance |

### When to use each

**Use `UniqueSpec`** when the uniqueness is a business rule:

```python
from dbwarden.databases import UniqueSpec

class Meta(PGTableMeta):
    uniques = [
        UniqueSpec(name="uq_users_email", columns=["email"]),
    ]
```

This generates an `ALTER TABLE` constraint. It shows in `information_schema.table_constraints` and can be referenced by foreign keys.

**Use `unique=True` on `PgIndexSpec`** when the uniqueness is a performance concern:

```python
from dbwarden.databases.pgsql import PgIndexSpec

class Meta(PGTableMeta):
    pg_indexes = [
        PgIndexSpec("ix_users_email", ["email"], unique=True),
    ]
```

This generates a `CREATE UNIQUE INDEX`. It enforces uniqueness identically but does not appear in the SQL standard constraint views and cannot be targeted by FK `REFERENCES`.

### Practical guidance

For a business rule like "no two users can share the same username within a branch":

```python
class Meta(PGTableMeta):
    uniques = [
        UniqueSpec(name="uq_user_branch_username",
                   columns=["branch_id", "username"]),
    ]
```

A constraint is semantically correct here: it declares a rule the data must satisfy. It also enables future FK references and works with `ON CONFLICT` in the same way as a unique index.

See [Indexes](indexes.md) for `PgIndexSpec` options.

## Constraint Diffing

Constraints are compared by full attribute content. Any difference in signature (columns, expression, options) produces `DROP` + `ADD`. Constraint name changes are detected as a new constraint.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/ddl-behavior/
========================================================================

# DDL Behavior

## Transactional DDL

PostgreSQL DDL is transactional. If a migration file contains multiple statements and one fails, all prior DDL in that file is rolled back. This makes PostgreSQL the safest backend for automated migration runs.

Operations that cannot run inside a transaction block:

- `CREATE INDEX CONCURRENTLY`
- `REFRESH MATERIALIZED VIEW CONCURRENTLY`
- `REINDEX DATABASE` / `REINDEX INDEX CONCURRENTLY`
- `ALTER TABLE DETACH PARTITION CONCURRENTLY` (PG 16+)
- `VACUUM` / `ANALYZE` (maintenance commands)

## Index Creation

DBWarden defaults to `CREATE INDEX CONCURRENTLY` to avoid table locking. Pass `--no-concurrent` when the migration must run inside a transaction block (PostgreSQL requires `CONCURRENTLY` outside a transaction).

### CONCURRENTLY Behaviour

- Requires more total time (two table scans)
- Allows concurrent reads and writes during index build
- If it fails, an invalid index is left behind (clean up with `DROP INDEX CONCURRENTLY`)
- Not supported for `CREATE INDEX ON ...` with partitioned tables (PG 14+ added partial support)

## Column Type Changes

Emits `ALTER TABLE t ALTER COLUMN c TYPE newtype` with a commented-out `-- USING col::newtype` line. Pass `--postgres-auto-using` to emit an active `USING` clause. Without the flag, uncomment and verify the USING expression before running the migration against production.

### USING Clause Examples

```sql
-- String to enum conversion
ALTER TABLE users ALTER COLUMN status TYPE user_status USING status::user_status;

-- Text to boolean
ALTER TABLE users ALTER COLUMN active TYPE boolean USING active::boolean;

-- JSONB extraction
ALTER TABLE users ALTER COLUMN metadata TYPE text USING metadata->>'name';
```

### Table Rewrite Behaviour

`ALTER COLUMN TYPE` rewrites the entire table (ACCESS EXCLUSIVE lock). Other operations that rewrite the table:

| Operation | Rewrite? | Notes |
|-----------|----------|-------|
| `ALTER COLUMN TYPE` | Yes | Full table rewrite |
| `SET STORAGE` | Yes | Rewrites column data |
| `SET COMPRESSION` | Yes | Rewrites column data (PG 14+) |
| `SET TABLESPACE` | Yes | Moves entire table |
| `ALTER COLUMN SET/DROP NOT NULL` | No | Metadata only |
| `ALTER COLUMN SET/DROP DEFAULT` | No | Metadata only |
| `ADD COLUMN` (no default) | No | Metadata only |
| `ADD COLUMN` (volatile default) | Yes | Rewrites table |
| `ALTER TABLE SET (fillfactor)` | No | Metadata only |
| `DROP COLUMN` | No | Metadata only (mark as dropped) |
| `SET UNLOGGED` | Yes | Writes all data to WAL |

### Safe Type Change

The `--safe-type-change` flag generates a multi-step strategy:
1. Add a temporary column with the new type
2. Emit a `--` comment with an `UPDATE` statement template
3. Emit a verification comment
4. After manual verification, drop the old column and rename the temporary column

### Lock Levels

| Operation | Lock Mode | Concurrent Access |
|-----------|-----------|-------------------|
| `ALTER TABLE t ALTER COLUMN c TYPE` | `ACCESS EXCLUSIVE` | Blocks all reads and writes |
| `ALTER TABLE t ADD COLUMN c` | `ACCESS EXCLUSIVE` (brief) | Very short lock |
| `ALTER TABLE t DROP COLUMN c` | `ACCESS EXCLUSIVE` | Table rewrite avoided, metadata only |
| `CREATE INDEX` (non-concurrent) | `ACCESS EXCLUSIVE` | Blocks all access |
| `CREATE INDEX CONCURRENTLY` | `SHARE UPDATE EXCLUSIVE` | Allows reads and writes |
| `DROP INDEX CONCURRENTLY` | `SHARE UPDATE EXCLUSIVE` | Allows reads and writes |
| `ALTER TABLE t SET (fillfactor)` | `ACCESS EXCLUSIVE` | Brief metadata change |
| `VALIDATE CONSTRAINT` | `SHARE UPDATE EXCLUSIVE` | Allows reads and writes |
| `REFRESH MATERIALIZED VIEW CONCURRENTLY` | `SHARE UPDATE EXCLUSIVE` | Allows reads and writes |

## Generated Columns

Adding a generated column via `ALTER TABLE` is not supported by PostgreSQL. DBWarden emits a comment placeholder noting this limitation. Dropping the generation expression (`ALTER COLUMN c DROP EXPRESSION`) produces real DDL.

PostgreSQL supports `GENERATED ALWAYS AS (expr) STORED` only. Virtual generated columns are not supported.

## Auto-increment Lifecycle

DBWarden supports toggling auto-increment on integer primary key columns:

```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)

    class Meta(PGTableMeta):
        id = ColumnMeta(autoincrement=True)
```

To explicitly disable auto-increment:

```python
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=False)
```

| Change | Generated SQL |
|--------|---------------|
| Adding autoincrement | `CREATE SEQUENCE users_id_seq` + `ALTER COLUMN id SET DEFAULT nextval(...)` + `ALTER SEQUENCE ... OWNED BY` |
| Removing autoincrement | `ALTER COLUMN id DROP DEFAULT` + `DROP SEQUENCE IF EXISTS users_id_seq` |

### GENERATED ALWAYS vs GENERATED BY DEFAULT

| Mode | Behaviour |
|------|-----------|
| `GENERATED ALWAYS AS IDENTITY` | User cannot provide a value; `INSERT` with explicit value fails |
| `GENERATED BY DEFAULT AS IDENTITY` | User can provide a value; auto-generation is skipped when value is supplied |

Override for `ALWAYS`: `INSERT OVERRIDING SYSTEM VALUE`.

### Detection from live databases

DBWarden detects autoincrement by:
1. SERIAL/BIGSERIAL column types
2. SQLAlchemy's `.autoincrement` attribute
3. `nextval(...)` default patterns

### Type mapping

| Condition | Resulting Type |
|-----------|---------------|
| `autoincrement=True` (default) | `SERIAL` / `BIGSERIAL` |
| `autoincrement=False` | `INTEGER` / `BIGINT` |
| `autoincrement=None` (unspecified) | `SERIAL` / `BIGSERIAL` (backward compatible) |

## DDL Limitations

- **Adding generated column**: Not supported via ALTER TABLE; requires table recreation
- **`CONCURRENTLY` in transaction**: `CREATE INDEX CONCURRENTLY`, `REFRESH MATERIALIZED VIEW CONCURRENTLY`, and similar operations cannot run inside a transaction block
- **Volatile default on ADD COLUMN**: `ALTER TABLE t ADD COLUMN c type DEFAULT random()` rewrites the table because each row needs a distinct default value
- **Dropping column with dependencies**: Requires `CASCADE` if views, FKs, or other objects reference the column
- **Removing enum values**: PostgreSQL does not support `ALTER TYPE ... DROP VALUE`; enum values can only be added, renamed, or the type recreated with CASCADE
- **Composite type modification**: No `ALTER TYPE` for composite types; must drop and recreate with CASCADE
- **Constraint rename**: Must use `ALTER TABLE t RENAME CONSTRAINT old TO new` (no `ALTER CONSTRAINT` rename)
- **SET STORAGE after table creation**: Rewrites the column data; cannot be done without a table lock

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/declaring-metadata/
========================================================================

# Declaring Metadata

PostgreSQL metadata is declared in a `class Meta` inner class on the model. This is the **only** supported surface: `mapped_column(info=...)` raises `DBWardenConfigError`.

## Table-Level Meta

Inherit from `PGTableMeta` on your `class Meta`:

```python
from sqlalchemy import Integer
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.pgsql import PGTableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)

    class Meta(PGTableMeta):
        pg_fillfactor = 80
        pg_tablespace = "fastspace"
        pg_storage_params = {
            "fillfactor": 80,
            "autovacuum_enabled": "false",
        }
        pg_inherits = "base_entity"
        pg_excludes = [
            {"name": "excl_room_booking", "expression": "USING gist (room_id WITH =, during WITH &&)"},
        ]
```

`PGTableMeta` inherits from `TableMeta`, which provides common attributes shared across all backends:

| Attribute | Type | SQL |
|-----------|------|-----|
| `comment` | `str` | `COMMENT ON TABLE t IS '...'` |
| `indexes` | `list[IndexSpec]` | `CREATE INDEX ...` |
| `checks` | `list[CheckSpec]` | `ALTER TABLE t ADD CONSTRAINT ... CHECK (...)` |
| `uniques` | `list[UniqueSpec]` | `ALTER TABLE t ADD CONSTRAINT ... UNIQUE (...)` |

PostgreSQL-specific `PGTableMeta` attributes:

| Attribute | Type | SQL |
|-----------|------|-----|
| `pg_fillfactor` | `int` | `ALTER TABLE t SET (fillfactor = N)` |
| `pg_tablespace` | `str` | `ALTER TABLE t SET TABLESPACE name` |
| `pg_storage_params` | `dict[str, Any]` | `ALTER TABLE t SET (param = value)` |
| `pg_unlogged` | `bool` | `CREATE UNLOGGED TABLE ...` / `ALTER TABLE t SET UNLOGGED` |
| `pg_partition` | `dict` | `PARTITION BY RANGE / LIST / HASH (columns)` |
| `pg_inherits` | `str \| list[str]` | `ALTER TABLE t INHERIT parent` |
| `pg_excludes` | `list[ExcludeSpec]` | `ALTER TABLE t ADD CONSTRAINT ... EXCLUDE USING ...` |
| `pg_indexes` | `list[PgIndexSpec]` | `CREATE INDEX ...` (with `USING`, `WHERE`, `INCLUDE`, `NULLS NOT DISTINCT`, column sorting) |
| `pg_checks` | `list[CheckSpec]` | `ALTER TABLE t ADD CONSTRAINT ... CHECK (...)` (with `NO INHERIT`) |
| `pg_uniques` | `list[UniqueSpec]` | `ALTER TABLE t ADD CONSTRAINT ... UNIQUE (...)` (with `DEFERRABLE`, `NULLS NOT DISTINCT`, `INCLUDE`) |

Pythonic model definitions use the typed spec classes with full IDE autocomplete:

```python
from dbwarden.databases import IndexSpec, CheckSpec, UniqueSpec
from dbwarden.databases.pgsql import PgIndexSpec, ExcludeSpec

class Meta(PGTableMeta):
    indexes = [
        IndexSpec(name="ix_users_email", columns=["email"], unique=True),
    ]
    checks = [
        CheckSpec(name="ck_users_age", expression="age >= 0"),
    ]
    uniques = [
        UniqueSpec(name="uq_users_email", columns=["email"],
                   deferrable=True, initially_deferred=True),
    ]
    pg_excludes = [
        ExcludeSpec(name="excl_room_booking",
                    elements=[{"column": "room_id", "with": "="},
                              {"column": "during", "with": "&&"}]),
    ]
```

Plain dicts are also accepted for backwards compatibility, but typed specs are the recommended and idiomatic form.

### INHERITS

`pg_inherits` accepts a single parent or a list for multiple inheritance:

```python
# Single inheritance
pg_inherits = "base_entity"

# Multiple inheritance
pg_inherits = ["base_entity", "audit_mixin"]
```

### ON COMMIT (Temporary Tables)

For `pg_unlogged` tables or temporary tables, use `ON COMMIT` options via `pg_on_commit`:

| Value | Behaviour |
|-------|-----------|
| `PRESERVE ROWS` | Default: rows persist across transaction boundaries |
| `DELETE ROWS` | All rows deleted at transaction end |
| `DROP` | Table dropped at transaction end |

### WITH OPTIONS (Storage Parameters)

Beyond `pg_fillfactor`, additional storage parameters can be set:

```sql
ALTER TABLE users SET (autovacuum_vacuum_threshold = 100, toast_tuple_target = 1024);
```

See [Storage Parameters](storage-params.md) for the complete list.

## JSONB Columns

JSONB columns use `from sqlalchemy.dialects.postgresql import JSONB`:

```python
from sqlalchemy.dialects.postgresql import JSONB

class User(Base):
    __tablename__ = "users"
    metadata = Column(JSONB)

    class Meta(PGTableMeta):
        pg_indexes = [
            PgIndexSpec("ix_users_metadata", ["metadata"], using="gin"),
        ]
```

For smaller indexes on path-based queries, use the `jsonb_path_ops` operator class:

```python
PgIndexSpec("ix_users_metadata", ["metadata"],
    using="gin",
    postgresql_ops={"metadata": "jsonb_path_ops"})
```

JSONB column type changes (e.g., `json` -> `jsonb`) are classified as **SAFE**.

### JSONB Operators for Indexing

GIN indexes on JSONB columns support these default operators:

| Operator | Description |
|----------|-------------|
| `?` | Does the string exist as a top-level key? |
| `?\|` | Do any of these strings exist as keys? |
| `?&` | Do all of these strings exist as keys? |
| `@>` | Does the left JSON contain the right JSON path/value? |
| `<@` | Is the left JSON contained by the right JSON? |

With `jsonb_path_ops`, only `@>` and `<@` are supported, but the index is smaller and faster for path queries.

## Column-Level Meta

Use `PGColumnMeta` inner classes for per-column metadata:

```python
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    bio: Mapped[str] = mapped_column(Text)

    class Meta(PGTableMeta):
        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100, identity_increment=1)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", compression="pglz", collation="en_US.UTF-8")
```

`PGColumnMeta` common attributes:

| Attribute | Type | SQL |
|-----------|------|-----|
| `comment` | `str` | `COMMENT ON COLUMN t.c IS '...'` |
| `public` | `bool` | Controls field visibility |
| `pg` | `PgFieldSpec` | PostgreSQL-specific column options |

PostgreSQL-specific `PgFieldSpec` fields (set via `pg.field(...)`):

| Keyword | Type | SQL |
|---------|------|-----|
| `collation` | `str` | `ALTER COLUMN c TYPE t COLLATE "name"` |
| `storage` | `str` | `ALTER COLUMN c SET STORAGE {PLAIN\|MAIN\|EXTERNAL\|EXTENDED}` |
| `compression` | `str` | `ALTER COLUMN c SET COMPRESSION {pglz\|zstd}` (PG 14+) |
| `generated` | `str` | `GENERATED ALWAYS AS (expr) STORED` |
| `identity` | `str` | `ADD GENERATED {ALWAYS\|BY DEFAULT} AS IDENTITY` |
| `identity_start` | `int` | Sequence `START WITH` |
| `identity_increment` | `int` | Sequence `INCREMENT BY` |
| `identity_min` | `int` | Sequence `MINVALUE` |
| `identity_max` | `int` | Sequence `MAXVALUE` |

### Column Defaults

Column default values are set via SQLAlchemy's `server_default` parameter, not through `PGColumnMeta`. The default is emitted as `ALTER COLUMN c SET DEFAULT expr` in DDL:

```python
from sqlalchemy import text

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("NOW()"))
    slug: Mapped[str] = mapped_column(String, server_default=text("gen_random_uuid()"))
```

`server_default=text("...")` renders the SQL expression directly. Use `server_default="constant"` (string) for literal defaults.

## Foreign Key Options

FK options are captured from the database by `generate-models` and emitted in the `ForeignKey` constructor:

```python
from sqlalchemy import ForeignKey

class OrderItem(Base):
    __tablename__ = "order_items"
    order_id: Mapped[int] = mapped_column(
        ForeignKey("orders.id", ondelete="CASCADE", onupdate="CASCADE", deferrable=True),
        nullable=False,
    )
```

Supported FK options: `ondelete`, `onupdate`, `deferrable`, `initially`, `match` (`FULL`/`PARTIAL`/`SIMPLE`).

See [Constraints](constraints.md) for FK lifecycle and deferrable behaviour.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/event-triggers/
========================================================================

# Event Triggers

**Requires the `dbwarden-pgsql-extensions` plugin:** `dbwarden plugin add dbwarden-pgsql-extensions`. The `event_trigger` object handler ships in that plugin, not in core.

**Handler**: `EventTriggerHandler` (PREAMBLE phase, config-driven)

Event triggers fire on database-level DDL events. They are scoped to the entire database cluster (not per-schema).

```python
pg_event_triggers=[
    {
        "name": "trg_ddl_audit",
        "event": "ddl_command_start",
        "function": "audit_ddl",
        "tags": ["CREATE TABLE", "ALTER TABLE"],
    },
]
```

## Events

| Event | Fires On |
|-------|----------|
| `ddl_command_start` | Before any DDL statement |
| `ddl_command_end` | After any DDL statement |
| `sql_drop` | When objects are dropped |
| `table_rewrite` | When `ALTER TABLE` rewrites a table |

## DDL Command Tags

Available tags for `WHEN TAG IN` filtering (selected common tags):

| Tag Category | Tags |
|--------------|------|
| DDL | `CREATE TABLE`, `ALTER TABLE`, `DROP TABLE`, `CREATE INDEX`, `ALTER INDEX`, `DROP INDEX` |
| Schema | `CREATE SCHEMA`, `ALTER SCHEMA`, `DROP SCHEMA` |
| Type | `CREATE TYPE`, `ALTER TYPE`, `DROP TYPE` |
| Function | `CREATE FUNCTION`, `ALTER FUNCTION`, `DROP FUNCTION` |
| Trigger | `CREATE TRIGGER`, `ALTER TRIGGER`, `DROP TRIGGER` |
| View | `CREATE VIEW`, `ALTER VIEW`, `DROP VIEW` |
| Sequence | `CREATE SEQUENCE`, `ALTER SEQUENCE`, `DROP SEQUENCE` |
| Extension | `CREATE EXTENSION`, `ALTER EXTENSION`, `DROP EXTENSION` |

The full list of supported tags is available in the PostgreSQL documentation under "Server Event Trigger Command Tags".

## Function Context Variables

Event trigger functions access DDL context through special session variables:

| Variable | Type | Description |
|----------|------|-------------|
| `TG_EVENT` | `text` | Event name: `ddl_command_start`, `ddl_command_end`, `sql_drop`, `table_rewrite` |
| `TG_TAG` | `text` | Command tag: `CREATE TABLE`, `ALTER TABLE`, etc. |
| `TG_TABLE_SCHEMA` | `text` | Schema of the target object (when applicable) |
| `TG_TABLE_NAME` | `text` | Name of the target object (when applicable) |

Example function using context variables:

```sql
CREATE FUNCTION audit_ddl()
RETURNS event_trigger
LANGUAGE plpgsql
AS $$
BEGIN
    INSERT INTO ddl_audit_log (event, tag, schema, object, occurred_at)
    VALUES (TG_EVENT, TG_TAG, TG_TABLE_SCHEMA, TG_TABLE_NAME, NOW());
END;
$$;
```

## Function Signature Requirements

Event trigger functions must:
- Take **no arguments**
- Return type `event_trigger`
- Be created before the event trigger that references them

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE EVENT TRIGGER name ON event WHEN TAG IN ('tag1', 'tag2') EXECUTE FUNCTION func();` |
| Alter | `ALTER EVENT TRIGGER name DISABLE;` / `ALTER EVENT TRIGGER name ENABLE;` / `ALTER EVENT TRIGGER name RENAME TO new_name;` |
| Drop | `DROP EVENT TRIGGER IF EXISTS name;` |

## Enabled State

| Value | Meaning |
|-------|---------|
| `O` | Enabled (default) |
| `D` | Disabled |
| `R` | Enabled in replica mode |
| `A` | Always enabled |

## Notes

- Event triggers require a superuser to create
- The backing function must be created first (see [Functions & Triggers](functions-and-triggers.md))
- `DROP EVENT TRIGGER` does not auto-drop the backing function
- Tags filter which DDL commands fire the trigger; absent tags means all DDL commands
- If an event trigger function raises an exception, the DDL command is aborted and rolled back
- Use `sql_drop` with care: objects have already been removed from catalogs, so `TG_TABLE_SCHEMA` and `TG_TABLE_NAME` may be NULL for dropped objects; use `pg_event_trigger_dropped_objects()` to get the list

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/extended-statistics/
========================================================================

# Extended Statistics

**Requires the `dbwarden-pgsql-extensions` plugin:** `dbwarden plugin add dbwarden-pgsql-extensions`. The `extended_statistics` object handler ships in that plugin, not in core.

**Handler**: `ExtendedStatisticsHandler` (PREAMBLE phase, config-driven, PG 14+)

Extended statistics give the query planner better estimates for correlated columns.

```python
pg_extended_statistics=[
    {
        "name": "stats_users_email_city",
        "table": "users",
        "kinds": ["d", "f"],
        "columns": "email, city",
    },
]
```

## Kind Codes

| Code | Kind | Description | PG Version |
|------|------|-------------|------------|
| `d` | ndistinct | Distinct-value counts for column groups | 10+ |
| `f` | dependencies | Functional dependency statistics | 10+ |
| `m` | MCV | Most-common-values lists | 10+ |
| `e` | expressions | Expression statistics (PG 14+) | 14+ |

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE STATISTICS name (kinds) ON columns FROM table;` |
| Alter | `ALTER STATISTICS name SET STATISTICS target;` |
| Drop | `DROP STATISTICS IF EXISTS name;` |
| Analyze | `ANALYZE table;` (required to populate statistics) |

### ALTER STATISTICS

The statistics target controls sample size. Higher values produce better estimates at the cost of longer `ANALYZE` time:

```sql
ALTER STATISTICS stats_users_email_city SET STATISTICS 1000;
```

Values range from `-1` (use system default, typically 100) to `10000`.

### ANALYZE Requirements

`CREATE STATISTICS` defines the statistics object but does not populate it. Run `ANALYZE` on the table against the database directly to collect the first statistics sample:

```sql
ANALYZE users;
```

## Examples

```python
# ndistinct + dependencies on correlated columns
{"name": "stats_order_date_status", "table": "orders",
 "kinds": ["d", "f"], "columns": "order_date, status"}

# MCV on a high-cardinality group
{"name": "stats_product_category", "table": "products",
 "kinds": ["m"], "columns": "category_id, price"}

# Expression statistics (PG 14+)
{"name": "stats_user_email_domain", "table": "users",
 "kinds": ["d", "m"], "expressions": ["lower(email)"]}
```

Generated DDL:
```sql
CREATE STATISTICS stats_users_email_city (ndistinct, dependencies) ON email, city FROM users;
```

## Schema Support

Extended statistics can be scoped to a schema:

```python
{"name": "stats_order_date_status", "table": "orders",
 "schema": "app", "kinds": ["d", "f"], "columns": "order_date, status"}
```

## Migration Safety

| Change | Severity |
|--------|----------|
| Add extended statistic | `INFO` |
| Drop extended statistic | `WARNING` |
| Change kinds / columns | `WARNING` |
| Change statistics target | `INFO` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/functions-and-triggers/
========================================================================

# Functions & Triggers

**Requires the `dbwarden-pgsql-extensions` plugin:** `dbwarden plugin add dbwarden-pgsql-extensions`. The plugin owns the `pg_functions` and `pg_triggers` config keys and their object handlers, so declaring them without it installed raises `DBWardenConfigError` at config load.

Functions and triggers are **config-driven** objects processed during the PREAMBLE phase (before table diffing). They are declared in `database_config(...)`.

## Functions

Declared via `pg_functions`. Each function definition includes the body, language, and options.

```python
pg_functions=[
    {
        "name": "update_timestamp",
        "language": "plpgsql",
        "body": """
            BEGIN
                NEW.updated_at = NOW();
                RETURN NEW;
            END;
        """,
        "returns": "trigger",
    },
    {
        "name": "count_users",
        "language": "sql",
        "body": "SELECT count(*) FROM users",
        "returns": "bigint",
        "volatility": "STABLE",
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE FUNCTION name (...args...) RETURNS type AS $$ body $$ LANGUAGE lang;` |
| Drop | `DROP FUNCTION IF EXISTS name (...args...) CASCADE;` |

Changes are detected as drop-then-create. There is no `CREATE OR REPLACE` for changed bodies.

### Options

| Config Key | SQL |
|------------|-----|
| `volatility` | `VOLATILE` / `STABLE` / `IMMUTABLE` |
| `security_definer` | `SECURITY DEFINER` |
| `leakproof` | `LEAKPROOF` |
| `parallel` | `PARALLEL UNSAFE` / `RESTRICTED` / `SAFE` |
| `cost` | `COST n` |
| `rows` | `ROWS n` |

Volatility semantics:

| Volatility | Behaviour | Optimizer Assumptions |
|------------|-----------|-----------------------|
| `VOLATILE` | Can return different results on each call (default) | No optimizations |
| `STABLE` | Same results within same statement | Can be used in index scan conditions |
| `IMMUTABLE` | Same results for same arguments always | Can be pre-evaluated, used in expression indexes, partition pruning |

### Return Types

| Returns | Description |
|---------|-------------|
| `trigger` | Trigger function (returns `TRIGGER`) |
| `void` | No return value |
| `table (...)` | Set-returning function (use `rows` for estimate) |
| `setof type` | Set-returning function (alternative syntax) |
| Any PG type | Scalar return |

### Arguments

```python
{
    "name": "add_user",
    "args": [
        {"name": "p_name", "type": "text"},
        {"name": "p_email", "type": "text"},
    ],
    "returns": "int",
    "language": "sql",
    "body": "INSERT INTO users (name, email) VALUES (p_name, p_email) RETURNING id",
}
```

### Parameter Modes

| Mode | Description |
|------|-------------|
| `IN` | Default: input parameter |
| `OUT` | Output parameter, acts as return value |
| `INOUT` | Input-output parameter (both accepts and returns) |
| `VARIADIC` | Variable number of arguments (last parameter only) |

```python
{
    "name": "get_user_stats",
    "args": [
        {"name": "p_user_id", "type": "int", "mode": "IN"},
        {"name": "out_count", "type": "bigint", "mode": "OUT"},
    ],
    "returns": "bigint",
    "language": "sql",
    "body": "SELECT count(*) FROM orders WHERE user_id = p_user_id",
}
```

### Function Overloading

PostgreSQL supports multiple functions with the same name but different argument types. DBWarden tracks the full argument signature. Two functions with the same name but different `args` are treated as distinct objects.

### WINDOW Functions

```python
{
    "name": "rank_per_category",
    "returns": "int",
    "language": "sql",
    "body": "SELECT rank() OVER (PARTITION BY category ORDER BY score DESC)",
    "args": [{"name": "category", "type": "text"}],
}
```

Set `RETURNS TABLE (...) ` or `RETURNS SETOF` with `rows` estimate.

## Procedures

PostgreSQL 11+ supports `CREATE PROCEDURE`, distinct from functions in that procedures can use transaction control (`COMMIT` / `ROLLBACK`).

```python
pg_functions=[
    {
        "name": "transfer_funds",
        "language": "plpgsql",
        "body": """
            BEGIN
                UPDATE accounts SET balance = balance - amount WHERE id = from_id;
                UPDATE accounts SET balance = balance + amount WHERE id = to_id;
                COMMIT;
            END;
        """,
        "returns": "void",
        "kind": "procedure",
    },
]
```

Key differences from functions:

| Aspect | Function | Procedure |
|--------|----------|-----------|
| Transaction control | No | Yes (`COMMIT`/`ROLLBACK`) |
| Called via | `SELECT func()` | `CALL proc()` |
| Return value | Required (can be void) | No return value |
| `kind` field | Omit or `"function"` | `"procedure"` |

## Triggers

Declared via `pg_triggers`. Each trigger binds a function to a table event.

```python
pg_triggers=[
    {
        "name": "trg_users_updated_at",
        "table": "users",
        "function": "update_timestamp",
        "timing": "BEFORE",
        "events": ["UPDATE"],
        "for_each": "ROW",
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE TRIGGER name timing event ON table FOR EACH ROW EXECUTE FUNCTION func();` |
| Alter | `ALTER TRIGGER name ON table RENAME TO new_name;` |
| Drop | `DROP TRIGGER IF EXISTS name ON table;` |

### Events

One or more of: `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE`.

### Timing

| Timing | Description |
|--------|-------------|
| `BEFORE` | Fires before the event |
| `AFTER` | Fires after the event |
| `INSTEAD OF` | Replaces the event (views only) |

### UPDATE OF Columns

Fire the trigger only when specific columns are updated:

```python
pg_triggers=[{
    "name": "trg_user_email",
    "table": "users",
    "function": "send_email_verification",
    "timing": "AFTER",
    "events": ["UPDATE OF email"],
    "for_each": "ROW",
}]
```

### Condition

Use `condition` to add a `WHEN` clause:

```python
pg_triggers=[{
    "name": "trg_prevent_delete",
    "table": "users",
    "function": "prevent_delete",
    "timing": "BEFORE",
    "events": ["DELETE"],
    "for_each": "ROW",
    "condition": "OLD.is_protected",
}]
```

### Trigger Function Context

Trigger functions access row data through special variables:

| Variable | Type | Description |
|----------|------|-------------|
| `NEW` | `RECORD` | New row for INSERT/UPDATE (NULL for DELETE) |
| `OLD` | `RECORD` | Old row for UPDATE/DELETE (NULL for INSERT) |
| `TG_OP` | `text` | Operation: `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE` |
| `TG_TABLE_NAME` | `text` | Table that fired the trigger |
| `TG_TABLE_SCHEMA` | `text` | Schema of the table |
| `TG_NAME` | `text` | Trigger name |
| `TG_WHEN` | `text` | Timing: `BEFORE`, `AFTER`, or `INSTEAD OF` |
| `TG_LEVEL` | `text` | `ROW` or `STATEMENT` |
| `TG_NARGS` | `int` | Number of arguments passed to the trigger |
| `TG_ARGV` | `text[]` | Arguments passed to the trigger |

### Constraint Triggers

Constraint triggers are deferred triggers that fire at transaction end. Declared separately from regular triggers:

```python
pg_triggers=[{
    "name": "trg_check_balance",
    "table": "accounts",
    "function": "verify_balance",
    "timing": "AFTER",
    "events": ["UPDATE"],
    "for_each": "ROW",
    "constraint": True,
    "deferrable": True,
    "initially": "DEFERRED",
}]
```

Generated DDL:
```sql
CREATE CONSTRAINT TRIGGER trg_check_balance
AFTER UPDATE ON accounts
DEFERRABLE INITIALLY DEFERRED
FOR EACH ROW
EXECUTE FUNCTION verify_balance();
```

### Trigger Arguments

```python
pg_triggers=[{
    "name": "trg_log_changes",
    "table": "users",
    "function": "log_changes",
    "timing": "AFTER",
    "events": ["UPDATE"],
    "for_each": "ROW",
    "args": ["user_audit", "ignore_columns:password_hash"],
}]
```

Arguments are passed as `TG_ARGV[0]`, `TG_ARGV[1]`, etc. in the trigger function.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/grants-and-roles/
========================================================================

# Grants & Roles

**Requires the `dbwarden-pgsql-rbac` plugin:** `dbwarden plugin add dbwarden-pgsql-rbac`. The plugin owns the `pg_roles` and `pg_default_privileges` config keys along with the handlers described below, so declaring them without it installed raises `DBWardenConfigError` at config load.

Grants are handled by `GrantsHandler` (DIFF phase). Roles are handled by `RoleHandler` (PREAMBLE phase). Default privileges are handled by `DefaultPrivilegesHandler` (PREAMBLE phase).

## Table Grants

Grants are model-derived, declared per-table on the model.

### Grant Types

| Operation | DDL |
|-----------|-----|
| Grant | `GRANT privileges ON TABLE table TO role;` |
| Revoke | `REVOKE privileges ON TABLE table FROM role;` |

Supported privileges: `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE`, `REFERENCES`, `TRIGGER`, `ALL`.

### Column-Level Privileges

Grant access to specific columns only:

```sql
GRANT SELECT (email, name) ON TABLE users TO read_only_role;
GRANT UPDATE (email) ON TABLE users TO app_user;
```

### GRANT OPTION

Allow the grantee to grant the same privilege to others:

```sql
GRANT SELECT ON TABLE users TO admin_user WITH GRANT OPTION;
```

### All Tables in Schema

Bulk grant using `ALL TABLES IN SCHEMA`:

```sql
GRANT SELECT ON ALL TABLES IN SCHEMA public TO read_only_role;
```

## Schema Grants

Schema grants use the same `GrantsHandler` with `object_type="schema"`:

| Operation | DDL |
|-----------|-----|
| Grant | `GRANT USAGE ON SCHEMA schema TO role;` |
| Revoke | `REVOKE USAGE ON SCHEMA schema FROM role;` |

Additional schema privileges:

| Privilege | Description |
|-----------|-------------|
| `USAGE` | Allows access to objects in the schema |
| `CREATE` | Allows creating objects in the schema |
| `ALL` | All schema privileges |

## Database-Level Grants

| Operation | DDL |
|-----------|-----|
| Grant | `GRANT privilege ON DATABASE db TO role;` |
| Revoke | `REVOKE privilege ON DATABASE db FROM role;` |

Supported database privileges: `CREATE`, `CONNECT`, `TEMPORARY` (or `TEMP`), `ALL`.

## Type-Level Grants

```sql
GRANT USAGE ON TYPE my_enum TO app_user;
```

## Roles

**Handler**: `RoleHandler` (PREAMBLE phase, config-driven)

```python
pg_roles=[
    {"name": "app_user", "login": True, "password": "encrypted"},
    {"name": "readonly", "login": True, "connection_limit": 5},
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE ROLE name WITH options` |
| Alter | `ALTER ROLE name WITH options` |
| Drop | `DROP ROLE IF EXISTS name` |

Roles are filtered to non-bootstrap roles (excludes `pg_*` roles and default cluster roles).

### ADMIN OPTION

A role with `ADMIN OPTION` can grant its role membership to others:

```python
{"name": "app_admin", "login": True, "membership": ["app_user"], "admin_option": True}
```

### Role Membership and INHERIT

Role membership grants privileges of the parent role. With `INHERIT` (default), member roles automatically inherit privileges. Without `INHERIT`, use `SET ROLE parent_role` to activate them.

```sql
-- app_user inherits privileges of base_user
CREATE ROLE app_user INHERIT IN ROLE base_user;
```

### SET ROLE / SET SESSION AUTHORIZATION

```sql
SET ROLE app_user;              -- Switch to role within session
SET SESSION AUTHORIZATION app_user;  -- Switch session user
```

## Default Privileges

**Handler**: `DefaultPrivilegesHandler` (PREAMBLE phase, config-driven)

```python
pg_default_privileges=[
    {
        "schema": "public",
        "role": "app_user",
        "kind": "TABLES",
        "privileges": "SELECT, INSERT, UPDATE, DELETE",
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Grant | `ALTER DEFAULT PRIVILEGES FOR ROLE role IN SCHEMA schema GRANT privileges ON kind TO role;` |
| Revoke | `ALTER DEFAULT PRIVILEGES FOR ROLE role IN SCHEMA schema REVOKE privileges ON kind FROM role;` |

### Object Kinds

| Kind | Objects Covered |
|------|----------------|
| `TABLES` | Tables, views, materialized views |
| `SEQUENCES` | Sequences |
| `FUNCTIONS` | Functions, procedures |
| `TYPES` | Types, domains |
| `SCHEMAS` | Schemas |

### REVOKE Behaviour

`REVOKE` supports `CASCADE` and `RESTRICT`:

```sql
REVOKE ALL ON ALL TABLES IN SCHEMA public FROM read_only_role CASCADE;
```

`CASCADE` revokes the privilege from all users who received it through the target role. `RESTRICT` (default) fails if dependent privileges exist.

## BYPASSRLS and RLS Interaction

The `BYPASSRLS` role attribute lets a role bypass Row-Level Security entirely:

```python
{"name": "admin_user", "login": True, "bypassrls": True}
```

Roles without `BYPASSRLS` are subject to all RLS policies on accessed tables. See [RLS & Policies](rls-and-policies.md) for details.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/
========================================================================

# PostgreSQL

DBWarden treats PostgreSQL as a **first-class backend**: every natively supported feature is reverse-engineered, diffed, and emitted as correct DDL.

"First-class" means the round-trip is verified: reverse-engineer a live database with `generate-models`, feed the output back into `make-migrations`, and get **zero diff**.

Implementation note: PostgreSQL diffs and SQL emission flow through the `dbwarden.engine.backends.postgresql.handlers` handler package. The handler pipeline is described in the [Architecture Deep Dive](../../architecture-deep-dive.md#postgresql-handler-pipeline).

```bash
$ dbwarden generate-models -d primary --tables users,orders,items
$ dbwarden make-migrations
# -> "No changes detected"
```

## Feature Matrix

| Category | Features |
|----------|----------|
| Identity Columns | `GENERATED ALWAYS AS IDENTITY`, `GENERATED BY DEFAULT AS IDENTITY`, sequence options |
| Collation | Per-column `COLLATE` via `pg.field(collation=...)` |
| Storage | Per-column `STORAGE` (`PLAIN`, `MAIN`, `EXTERNAL`, `EXTENDED`) |
| Compression | Per-column `COMPRESSION` (`pglz`, `zstd`) via `pg.field(compression=...)` (PG 14+) |
| Generated Columns | `GENERATED ALWAYS AS (...) STORED` |
| Table Properties | Fillfactor, storage params, tablespace, unlogged, partitioning, inheritance |
| Renames | Table rename, column rename |
| Constraints | FK (`MATCH FULL/PARTIAL/SIMPLE`, `ON DELETE/UPDATE`, `DEFERRABLE`), unique (`NULLS NOT DISTINCT`, `INCLUDE`, `DEFERRABLE`), check (`NO INHERIT`, `NOT VALID`), exclude |
| Indexes | B-tree, hash, GiST, GIN, BRIN, SP-GiST; partial, expression, `INCLUDE`, `WHERE`, opclasses, `NULLS NOT DISTINCT`, column sorting, `CONCURRENTLY` |
| RLS & Policies | `ENABLE`/`DISABLE`/`FORCE`/`NO FORCE` row-level security; permissive/restrictive, role-scoped policies |
| Enums | `CREATE TYPE ... AS ENUM`, `ALTER TYPE ... ADD VALUE ... AFTER` |
| Domains | `CREATE DOMAIN` with base type, default, NOT NULL, CHECK |
| Composite Types | `CREATE TYPE ... AS (col1 type1, col2 type2, ...)` |
| Sequences | `CREATE SEQUENCE` with all options |
| Functions | `CREATE FUNCTION` with language, arguments, body |
| Triggers | `CREATE TRIGGER` with timing, events, FOR EACH ROW/STATEMENT |
| Roles | `CREATE ROLE` with login, password, privileges |
| Default Privileges | `ALTER DEFAULT PRIVILEGES` per schema/role/object-type |
| Extended Statistics | `CREATE STATISTICS` with ndistinct, dependencies, MCV, expressions (PG 14+) |
| Event Triggers | `CREATE EVENT TRIGGER` for DDL events |
| Views | Regular `CREATE OR REPLACE VIEW`, materialized views with auto-refresh |
| Schema-level Grants | `GRANT USAGE ON SCHEMA`, `GRANT ALL ON SCHEMA` |
| Table Grants | `GRANT SELECT/INSERT/UPDATE/DELETE` |
| Type Mapping | SQLAlchemy type → PostgreSQL native type normalization |
| Storage Parameters | Table-level and index-level `WITH` options, autovacuum tuning |

## Documentation Sections

- [Config Keys](config-keys.md) : All 12 `pg_*` configuration keys
- [Declaring Metadata](declaring-metadata.md) : Table-level, column-level, JSONB, FK options
- [Tables & Columns](tables-and-columns.md) : Column handler, type changes, auto-increment lifecycle
- [Registry Architecture](../../architecture-deep-dive.md#postgresql-handler-pipeline) : Handler map, phases, online and offline diff flow
- [Constraints](constraints.md) : FK (MATCH FULL, CASCADE), unique, check, exclude
- [Indexes](indexes.md) : B-tree, partial, expression indexes, operator classes, NULLS NOT DISTINCT
- [Types](types.md) : Enums, domains, composite types
- [Functions & Triggers](functions-and-triggers.md) : Function and trigger lifecycle
- [RLS & Policies](rls-and-policies.md) : Row-level security, policy lifecycle, FORCE
- [Grants & Roles](grants-and-roles.md) : Table grants, schema grants, roles, default privileges
- [Partitioning](partitioning.md) : RANGE/LIST/HASH partition strategies, attach/detach
- [Views](views.md) : Regular and materialized views, auto-refresh
- [Sequences](sequences.md) : Sequence creation and ownership
- [Extended Statistics](extended-statistics.md) : ndistinct, dependencies, MCV, expressions
- [Event Triggers](event-triggers.md) : DDL event trigger lifecycle
- [Schemas](schemas.md) : Config-level and model-level schemas, search path
- [DDL Behavior](ddl-behavior.md) : Transactional DDL, CONCURRENTLY, type change strategies
- [Type Mapping](type-mapping.md) : SQLAlchemy type → PostgreSQL type normalization
- [Storage Parameters](storage-params.md) : Table and index storage parameters, autovacuum tuning
- [Migration Safety](migration-safety.md) : Safety classification table

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/indexes/
========================================================================

# Indexes

Indexes are handled by `IndexHandler` during the DIFF phase.

## Supported Index Types

| Method | Description |
|--------|-------------|
| B-tree | Default, for equality and range queries |
| Hash | Equality-only, smaller than btree |
| GiST | Geometric, full-text, and custom data types |
| GIN | JSONB, array, full-text, and tsvector |
| BRIN | Block range indexes for large tables |
| SP-GiST | Space-partitioned GiST |

## Declaring Indexes

Model-level indexes use `PgIndexSpec`:

```python
from dbwarden.databases.pgsql import PgIndexSpec

class Meta(PGTableMeta):
    pg_indexes = [
        PgIndexSpec("ix_users_email", ["email"], unique=True, using="btree"),
        PgIndexSpec("ix_users_data", ["data"], using="gin",
            postgresql_ops={"data": "jsonb_path_ops"}),
    ]
```

`PgIndexSpec` supports:

| Field | Description |
|-------|-------------|
| `name` | Index name (auto-generated if omitted) |
| `columns` | Indexed columns |
| `unique` | `CREATE UNIQUE INDEX` |
| `using` | Access method (`btree`, `gin`, `gist`, `hash`, `brin`, `spgist`) |
| `where` | Partial index predicate |
| `include` | `INCLUDE` columns (covering indexes) |
| `expression` | Expression index (e.g., `lower(email)`) |
| `with_params` | Storage parameters (`fillfactor`, `autovacuum_*`) |
| `tablespace` | `TABLESPACE name` |
| `nulls_not_distinct` | `NULLS NOT DISTINCT` (PG 15+) |
| `column_sorting` | Per-column `ASC` / `DESC` / `NULLS FIRST` / `NULLS LAST` |
| `postgresql_ops` | Per-column operator classes |
| `concurrently` | `CREATE INDEX CONCURRENTLY` (default `True`) |

## Partial Indexes

```python
PgIndexSpec("ix_users_active", ["email"], where="active = true")
```

```sql
CREATE INDEX CONCURRENTLY ix_users_active ON users (email) WHERE active = true;
```

## Expression Indexes

```python
PgIndexSpec("ix_users_lower_email", columns=[], expression="lower(email)")
```

```sql
CREATE INDEX CONCURRENTLY ix_users_lower_email ON users (lower(email));
```

Expression indexes are stored with empty `columns` and the expression text in the `expression` field. Changes to the expression are detected and produce a `DROP INDEX` + `CREATE INDEX` cycle.

## Covering Indexes (INCLUDE)

Non-key columns stored in the index to enable index-only scans:

```python
PgIndexSpec("ix_users_email_covering", ["email"],
    include=["name", "avatar_url"])
```

```sql
CREATE INDEX CONCURRENTLY ix_users_email_covering ON users (email) INCLUDE (name, avatar_url);
```

`INCLUDE` columns are not used for index scans but are available as output columns without a table lookup. This makes index-only scans possible for queries that select only `email`, `name`, and `avatar_url`.

## Multi-Column Indexes

Column order matters for query planning:

```python
PgIndexSpec("ix_orders_user_date", ["user_id", "created_at DESC"])
```

B-tree multi-column indexes support queries on the leftmost columns. This index supports:
- `WHERE user_id = 1` (leftmost prefix)
- `WHERE user_id = 1 AND created_at > '2024-01-01'` (range on second column)
- Does NOT support: `WHERE created_at > '2024-01-01'` (no leftmost column)

## Operator Classes

```python
PgIndexSpec("ix_users_data", ["data"],
    using="gin",
    postgresql_ops={"data": "jsonb_path_ops"})
```

```sql
CREATE INDEX CONCURRENTLY ix_users_data ON users USING GIN (data jsonb_path_ops);
```

### Common Operator Classes

| Access Method | Operator Class | Use Case |
|---------------|---------------|----------|
| GIN | `jsonb_path_ops` | Smaller, faster JSONB path queries |
| GIN | `array_ops` (default) | Array containment queries |
| GiST | `inet_ops` | IP address range queries |
| GiST | `tsvector_ops` | Full-text search |
| BRIN | `bloom_ops` | Bloom filter indexes |

## Column Sorting

```python
PgIndexSpec("ix_orders_date", ["created_at DESC NULLS LAST", "id ASC"])
```

```sql
CREATE INDEX CONCURRENTLY ix_orders_date ON users (created_at DESC NULLS LAST, id ASC);
```

## BRIN Parameters

BRIN indexes accept access-method-specific storage parameters:

```python
PgIndexSpec("ix_logs_created_at", ["created_at"],
    using="brin",
    with_params={"pages_per_range": 64, "autosummarize": True})
```

| Parameter | Default | Description |
|-----------|---------|-------------|
| `pages_per_range` | `128` | Number of pages per block range. Lower values = finer granularity, larger index |
| `autosummarize` | `off` | Automatically summarize new pages on insert |

## REINDEX

DBWarden does not auto-generate `REINDEX` statements. When an index becomes corrupted or bloated, recreate it manually:

```sql
REINDEX INDEX CONCURRENTLY ix_users_email;
REINDEX TABLE CONCURRENTLY users;
REINDEX DATABASE CONCURRENTLY mydb;
```

`CONCURRENTLY` avoids locking, but requires extra resources and can fail if the index is a unique index with duplicates.

## NULLS NOT DISTINCT

```python
PgIndexSpec("ix_users_email_unique", ["email"],
    unique=True, nulls_not_distinct=True)
```

Without `NULLS NOT DISTINCT`, a unique index allows multiple NULL values (PostgreSQL treats NULLs as distinct by default). With `NULLS NOT DISTINCT` (PG 15+), only one NULL is permitted.

## Unique Constraint vs Unique Index

For declaring uniqueness, DBWarden offers two paths:

| Path | API | Use case |
|------|-----|----------|
| **Constraint** | `UniqueSpec` in `uniques` / `pg_uniques` | Business rule, FK-targetable, appears in `information_schema` |
| **Index** | `unique=True` on `PgIndexSpec` | Performance-focused, simpler configuration |

See [Constraints](constraints.md#uniquespec-vs-uniquetrue-on-pgindexspec) for the full comparison and guidance.

## Migration Safety

| Change | Severity |
|--------|----------|
| Add index | `INFO` |
| Drop index | `WARNING` |
| Change index expression / columns | `WARNING` |
| Add/drop INCLUDE column | `INFO` |
| Change storage parameters | `INFO` |
| Change tablespace | `WARNING` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/migration-safety/
========================================================================

# Migration Safety

DBWarden classifies migration changes using the `Safety` enum:

```python
from dbwarden.engine.safety import Safety

assert Safety.SAFE == "SAFE"
assert Safety.INFO == "INFO"
assert Safety.WARN == "WARN"
assert Safety.CRITICAL == "CRITICAL"
```

## Safety Classification by Object Type

### Tables & Columns

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add column | `INFO` | None |
| Drop column | `WARNING` | `--force` |
| Change column type (safe) | `INFO` | None |
| Change column type (warn) | `WARNING` | `--force` |
| Change column type (critical) | `WARNING` | `--force` |
| Change column comment | `INFO` | None |
| Change nullable (SET/DROP NOT NULL) | `INFO` | None |
| Change default (SET/DROP DEFAULT) | `INFO` | None |
| Add autoincrement | `INFO` | None |
| Remove autoincrement | `WARNING` | `--force` |
| Change PG column meta (storage, compression, collation) | `WARNING` | `--force` |
| Rename column | `WARNING` | `--force` |

### Table Properties

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Change fillfactor | `INFO` | None |
| Change tablespace | `WARNING` | `--force` |
| Change inheritance | `WARNING` | `--force` |
| Change exclude constraints | `WARNING` | `--force` |
| Change table comment | `INFO` | None |
| Change object type (logged/unlogged) | `WARNING` | `--force` |
| Change ON COMMIT | `WARNING` | `--force` |

### Indexes

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add index | `INFO` | None |
| Drop index | `WARNING` | `--force` |
| Change index expression / columns | `WARNING` | `--force` |
| Add/drop INCLUDE column | `INFO` | None |
| Change index storage parameters | `INFO` | None |
| Change index tablespace | `WARNING` | `--force` |

### Constraints

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add FK | `INFO` | None |
| Drop FK | `WARNING` | `--force` |
| Change FK options | `WARNING` | `--force` |
| Add unique constraint | `INFO` | None |
| Drop unique constraint | `WARNING` | `--force` |
| Add check constraint | `INFO` | None |
| Drop check constraint | `WARNING` | `--force` |
| Add exclude constraint | `INFO` | None |
| Drop exclude constraint | `WARNING` | `--force` |
| Change constraint deferrability | `INFO` | None |

### Sequences

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add sequence | `INFO` | None |
| Drop sequence | `WARNING` | `--force` |
| Change sequence options (start, increment, etc.) | `WARNING` | `--force` |

### Types

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add enum value | `INFO` | None |
| Add enum type | `INFO` | None |
| Drop enum type | `WARNING` | `--force` |
| Add domain | `INFO` | None |
| Drop domain | `WARNING` | `--force` |
| Change domain definition | `WARNING` | `--force` |
| Add composite type | `INFO` | None |
| Drop composite type | `WARNING` | `--force` |
| Change composite type columns | `WARNING` | `--force` |

### Functions & Procedures

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add function | `INFO` | None |
| Drop function | `WARNING` | `--force` |
| Change function body | `WARNING` | `--force` |
| Add procedure | `INFO` | None |
| Drop procedure | `WARNING` | `--force` |

### Triggers

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add trigger | `INFO` | None |
| Drop trigger | `WARNING` | `--force` |
| Change trigger timing/events | `WARNING` | `--force` |
| Add constraint trigger | `INFO` | None |
| Drop constraint trigger | `WARNING` | `--force` |

### Roles & Grants

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add role | `INFO` | None |
| Drop role | `WARNING` | `--force` |
| Change role options | `WARNING` | `--force` |
| Add grant | `INFO` | None |
| Revoke grant | `WARNING` | `--force` |
| Add default privilege | `INFO` | None |
| Revoke default privilege | `WARNING` | `--force` |

### Event Triggers

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add event trigger | `INFO` | None |
| Drop event trigger | `WARNING` | `--force` |
| Change event trigger tags | `INFO` | None |

### Extended Statistics

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add extended statistic | `INFO` | None |
| Drop extended statistic | `WARNING` | `--force` |
| Change kinds / columns | `WARNING` | `--force` |
| Change statistics target | `INFO` | None |

### Views

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add view | `INFO` | None |
| Drop view | `WARNING` | `--force` |
| Change view query | `WARNING` | `--force` |
| Refresh materialized view | `INFO` | None |
| Change view schema | `WARNING` | `--force` |

### Schemas

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add schema | `INFO` | None |
| Drop schema | `WARNING` | `--force` |

### RLS & Policies

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Enable RLS | `INFO` | None |
| Disable RLS | `WARNING` | `--force` |
| Force RLS | `INFO` | None |
| Add policy | `INFO` | None |
| Drop policy | `INFO` | None |
| Change policy expression | `INFO` | None |

### Partitions

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add partition | `INFO` | None |
| Detach partition | `WARNING` | `--force` |
| Change partition strategy | `CRITICAL` | `--force` |

## Safe / Warn / Critical Type Changes

### Safe Type Changes (INFO)

| Conversion | Example |
|------------|---------|
| `VARCHAR(n)` → `VARCHAR(m)` (widening) | `VARCHAR(50)` → `VARCHAR(100)` |
| `VARCHAR` → `TEXT` | `VARCHAR` → `TEXT` |
| Storing to same type family | No change |
| `TIMESTAMP` → `TIMESTAMPTZ` | Adding timezone info |
| `json` → `jsonb` | JSON to binary JSON |
| Adding `NOT NULL` when column has no NULLs | Safe when data validates |

### Warn Type Changes (WARNING)

| Conversion | Example |
|------------|---------|
| `VARCHAR(n)` → `VARCHAR(m)` (narrowing) | `VARCHAR(100)` → `VARCHAR(50)` |
| `INTEGER` → `BIGINT` | Widening (safe in PG, uses USING) |
| `BIGINT` → `INTEGER` | Narrowing (potential data loss) |
| `TEXT` → `VARCHAR(n)` | Truncation risk |
| `NUMERIC(p,s)` → `NUMERIC(p',s')` with precision loss | May truncate |
| Any type change that requires USING | Needs manual verification |

### Critical Type Changes (CRITICAL)

| Conversion | Example |
|------------|---------|
| Dropping the only column of a table | Always critical |
| Removing enum values | Not supported by PostgreSQL |
| Changing partition key columns | Requires full table rewrite + data migration |
| Converting from `SETOF`/`TABLE` return type | Schema breakage |

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/partitioning/
========================================================================

# Partitioning

**Handler**: `PartitionHandler` (DIFF phase)

Partitioning is declared on `PGTableMeta` and requires the table to be a true PostgreSQL partitioned table (not traditional inheritance).

## Declaring a Partitioned Table

```python
class Meta(PGTableMeta):
    pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
```

### Partition Strategies

| Strategy | Description |
|----------|-------------|
| `RANGE` | Partition by range of values on one or more columns |
| `LIST` | Partition by discrete values |
| `HASH` | Partition by hash of values |

### Partition Columns

```python
# Range partition by date
pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}

# List partition by category
pg_partition = {"strategy": "LIST", "columns": ["category"]}

# Hash partition by id
pg_partition = {"strategy": "HASH", "columns": ["id"]}
```

## Attaching Partitions

Child partitions are declared via `pg_partitions`:

```python
class Meta(PGTableMeta):
    pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
    pg_partitions = [
        {"name": "events_2024_q1", "bound": "FOR VALUES FROM ('2024-01-01') TO ('2024-04-01')"},
        {"name": "events_2024_q2", "bound": "FOR VALUES FROM ('2024-04-01') TO ('2024-07-01')"},
    ]
```

Partitions are automatically attached/detached during migration.

### Partition Bounds Syntax

| Strategy | Bound Syntax | Example |
|----------|-------------|---------|
| RANGE | `FROM (expr) TO (expr)` | `FROM ('2024-01-01') TO ('2024-04-01')` |
| LIST | `IN (value, ...)` | `IN ('active', 'pending')` |
| HASH | `MODULUS m, REMAINDER r` | `MODULUS 4, REMAINDER 0` |

RANGE bounds are half-open: the lower bound is inclusive, the upper bound is exclusive. Use `UNBOUNDED` for open-ended ranges:

```python
{"name": "events_archived", "bound": "FOR VALUES FROM ('2023-01-01') TO (MAXVALUE)"}
```

HASH partitions use modulus/remainder for round-robin distribution:

```python
{"name": "events_hash_0", "bound": "FOR VALUES WITH (MODULUS 4, REMAINDER 0)"},
{"name": "events_hash_1", "bound": "FOR VALUES WITH (MODULUS 4, REMAINDER 1)"},
{"name": "events_hash_2", "bound": "FOR VALUES WITH (MODULUS 4, REMAINDER 2)"},
{"name": "events_hash_3", "bound": "FOR VALUES WITH (MODULUS 4, REMAINDER 3)"},
```

### DEFAULT Partition

```python
{"name": "events_default", "bound": "DEFAULT"}
```

A DEFAULT partition catches all rows that do not match any other partition bound. Only one DEFAULT partition is allowed per partitioned table.

## Sub-Partitioning

Create partitions of partitions by declaring a partition strategy on a partition table:

```python
class Event(Base):
    __tablename__ = "events"
    ...

    class Meta(PGTableMeta):
        pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
        pg_partitions = [
            {"name": "events_2024", "bound": "FOR VALUES FROM ('2024-01-01') TO ('2025-01-01')",
             "sub_partition": {"strategy": "LIST", "columns": ["category"]}},
        ]
```

Sub-partitions can use a different strategy than the parent. Each sub-partition level has its own `pg_partitions` entries.

## Indexes on Partitioned Tables

Creating an index on the parent table automatically creates matching indexes on all partitions:

```python
class Meta(PGTableMeta):
    pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
    pg_indexes = [
        PgIndexSpec("ix_events_user_id", ["user_id"]),
    ]
```

The index is propagated to all existing and future partitions. Unique indexes (including PKs) must include the partition key.

## FK Constraints on Partitioned Tables

Foreign keys can reference a partitioned table (referencing the parent, referencing is routed to partitions). Foreign keys from a partitioned table must include the partition key:

```python
class OrderItem(Base):
    __tablename__ = "order_items"
    ...

    class Meta(PGTableMeta):
        pg_partition = {"strategy": "HASH", "columns": ["order_id"]}
```

FK from `order_items` to `orders` must include `order_id` in the FK columns.

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Declare partition | `ALTER TABLE parent ATTACH PARTITION child FOR VALUES ...;` |
| Detach partition | `ALTER TABLE parent DETACH PARTITION child;` |
| Detach concurrently (PG 16+) | `ALTER TABLE parent DETACH PARTITION child CONCURRENTLY;` |
| Change strategy | Manual rewrite required (commented in DDL) |

### DETACH CONCURRENTLY (PG 16+)

PostgreSQL 16+ supports `DETACH CONCURRENTLY`, which avoids the `ACCESS EXCLUSIVE` lock normally required:

```sql
ALTER TABLE events DETACH PARTITION events_2023 CONCURRENTLY;
```

The partition becomes an independent table. A background worker finalizes the detach. You can monitor with `pg_detach_pending` in `pg_partitioned_table`.

## Maintenance Operations

Individual partitions support the same maintenance as regular tables:

- `ANALYZE partition_name;` on each partition
- `VACUUM partition_name;` on each partition
- Index creation directly on a partition (not propagated to parent)
- Tablespace management per partition

## Inheritance vs Partitioning

PartitionHandler only tracks tables with `pg_partition` declared (true PG partitioning). Traditional inheritance (`pg_inherits`) is handled separately via `PgTableHandler`.

Partitioning vs Inheritance:

| Aspect | Partitioning | Inheritance |
|--------|-------------|-------------|
| Constraint exclusion | Automatic | Manual (CHECK constraints) |
| Unique constraints | Requires partition key | Supported across any columns |
| FK references | To parent only | To each child separately |
| Row movement | Possible (PG 17+) | Not supported |
| Sub-partitioning | Supported | Not supported |

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/rls-and-policies/
========================================================================

# RLS & Policies

**Requires the `dbwarden-pgsql-rbac` plugin:** `dbwarden plugin add dbwarden-pgsql-rbac`. The policies handler ships in that plugin, not in core.

**Handler**: `PoliciesHandler` (DIFF phase)

Row-Level Security and policies are model-derived, declared on `PGTableMeta`.

## Enabling RLS

```python
class Meta(PGTableMeta):
    pg_rls = True
```

Generated DDL:
```sql
ALTER TABLE users ENABLE ROW LEVEL SECURITY;
```

### Default Deny Behaviour

When RLS is enabled but no policy applies to the current user, the default behaviour is to **deny all access**. Every row is invisible for read operations, and all write operations are blocked. At least one permissive policy must exist for the user to access data.

### FORCE / NO FORCE

```python
class Meta(PGTableMeta):
    pg_rls = True
    pg_rls_force = True
```

Generated DDL:
```sql
ALTER TABLE users FORCE ROW LEVEL SECURITY;
```

To disable force (without disabling RLS):

```python
class Meta(PGTableMeta):
    pg_rls = True
    pg_rls_force = False
```

```sql
ALTER TABLE users NO FORCE ROW LEVEL SECURITY;
```

The force flag only emits when it explicitly changes. If `pg_rls_force` is absent or `False` on both the snapshot and model sides, no `FORCE`/`NO FORCE` DDL is generated. This avoids churn on indexes that only toggle RLS without changing force.

`FORCE ROW LEVEL SECURITY` applies RLS to the table owner, who would normally bypass it. `NO FORCE` (default) exempts the owner.

## Policies

Policies are declared as a list on `PGTableMeta`:

```python
class Meta(PGTableMeta):
    pg_rls = True
    pg_policies = [
        {
            "name": "tenant_isolation",
            "using": "tenant_id = current_setting('app.tenant_id')::int",
            "roles": ["app_user"],
            "permissive": True,
        },
    ]
```

### Policy Keys

| Key | Description |
|-----|-------------|
| `name` | Policy name |
| `using` | `USING` expression (row visibility) |
| `with_check` | `WITH CHECK` expression (row modification) |
| `roles` | Roles the policy applies to (absent = all roles) |
| `permissive` | `PERMISSIVE` (default) or `RESTRICTIVE` |
| `command` | `ALL` (default), `SELECT`, `INSERT`, `UPDATE`, `DELETE` |

### Permissive vs Restrictive

Multiple policies interact differently based on their type:

| Policy Type | Combination Logic |
|-------------|-------------------|
| `PERMISSIVE` (default) | **OR**: access is granted if ANY permissive policy allows it |

| `RESTRICTIVE` | **AND**: access is denied if ANY restrictive policy blocks it |
Restrictive policies filter results after permissive policies. Use restrictive policies to implement mandatory access controls that override permissive policies:

```python
pg_policies = [
    # Permissive: tenant-level access
    {"name": "tenant_access", "using": "tenant_id = current_setting('app.tenant_id')::int",
     "permissive": True},
    # Restrictive: block access to sensitive rows regardless of tenant
    {"name": "block_sensitive", "using": "NOT is_sensitive",
     "permissive": False, "command": "SELECT"},
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Add policy | `CREATE POLICY name ON table FOR command USING (expr)` |
| Alter policy | `ALTER POLICY name ON table USING (new_expr)` |
| Drop policy | `DROP POLICY IF EXISTS name ON table` |
| Enable RLS | `ALTER TABLE table ENABLE ROW LEVEL SECURITY` |
| Disable RLS | `ALTER TABLE table DISABLE ROW LEVEL SECURITY` |
| Force RLS | `ALTER TABLE table FORCE ROW LEVEL SECURITY` |
| Remove force | `ALTER TABLE table NO FORCE ROW LEVEL SECURITY` |

## BYPASSRLS Role Attribute

The `BYPASSRLS` role attribute lets a role bypass all RLS policies:

```python
pg_roles=[
    {"name": "admin_user", "login": True, "bypassrls": True},
]
```

This is equivalent to running without RLS for that role. See [Grants & Roles](grants-and-roles.md) for role configuration.

## RLS and COPY

`COPY TO` on a table with RLS respects policies; only rows visible through the user's policies are exported. `COPY FROM` on a table with RLS checks `WITH CHECK` policies for each inserted row.

## RLS and Unique Constraints

Unique constraints (including PKs) are **not** RLS-aware. A user may see a unique constraint violation caused by a row they cannot see. This is a PostgreSQL limitation: RLS filters query results but does not filter constraint enforcement.

## RLS and FK Constraints

Foreign key constraints are enforced server-side regardless of RLS. A user may not be able to `SELECT` a referenced row but can still insert a referencing row if the FK validates.

## Migration Safety

| Change | Severity |
|--------|----------|
| Enable RLS | `INFO` |
| Disable RLS | `WARNING` |
| Force RLS | `INFO` |
| Add policy | `INFO` |
| Drop policy | `INFO` |
| Change policy expression | `INFO` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/schemas/
========================================================================

# Schemas

**Handler**: `SchemaHandler` (PREAMBLE phase)

## Config-Level Schema

Set `pg_schema` in `database_config(...)` to set the connection's `search_path`:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/main",
    pg_schema="app",
)
```

All unqualified table references use this schema. The `_dbwarden_seeds` tracking table is created in the schema specified by `search_path`.

## Model-Level Schema

Set `pg_schema` on `PGTableMeta` or `PGViewMeta`:

```python
class Meta(PGTableMeta):
    pg_schema = "app"
```

When a model has `pg_schema`, all DDL references the fully qualified name (`app.users`). This takes precedence over the config-level `search_path`.

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE SCHEMA IF NOT EXISTS name;` |
| Drop | `DROP SCHEMA IF EXISTS name CASCADE;` |

## Schema Ownership

When a schema is created, it is owned by the role that created it. Set a different owner:

```sql
ALTER SCHEMA app OWNER TO app_admin;
```

## Schema Privileges

Schemas require privileges for access:

| Privilege | Effect |
|-----------|--------|
| `USAGE` | Allows access to objects in the schema |
| `CREATE` | Allows creating new objects in the schema |

```sql
GRANT USAGE ON SCHEMA app TO app_user;
GRANT CREATE ON SCHEMA app TO app_admin;
```

Without `USAGE` on a schema, a user cannot see or access any objects within it, even if they have table-level privileges.

## Search Path Resolution

PostgreSQL resolves unqualified names by searching schemas in `search_path` order:

```
current_schema (first match wins) -> pg_catalog -> public
```

Use the `current_schema` function to check the effective search path:

```sql
SELECT current_schema; -- Returns the first schema in the path
SHOW search_path;      -- Returns the full search path string
```

The config-level `pg_schema` becomes the first entry in `search_path`, which means it takes precedence for all unqualified references.

## pg_catalog vs public

- `pg_catalog` is always searched unless explicitly excluded
- `public` schema is accessible by default to all roles
- Custom schemas require explicit `USAGE` grants

## Temporary Schema

PostgreSQL creates a `pg_temp_*` schema for temporary tables. Temporary tables take precedence over permanent tables when their schema is first in `search_path`. You can reference `pg_temp.schema_name` explicitly.

## Extensions

Extensions are created during the PREAMBLE phase via `pg_extensions`, which requires the `dbwarden-pgsql-extensions` plugin (`dbwarden plugin add dbwarden-pgsql-extensions`). Without it, the key raises `DBWardenConfigError` at config load.

```python
pg_extensions=["uuid-ossp", "pgcrypto"]
```

Generated DDL: `CREATE EXTENSION IF NOT EXISTS "uuid-ossp";`

## Code Seeds and Schema

Code seeds automatically qualify the table name with the model's `pg_schema`. If `User` has `pg_schema = "app"`, the seed INSERT becomes `INSERT INTO app.users (...) VALUES (...)`.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/sequences/
========================================================================

# Sequences

**Requires the `dbwarden-pgsql-types` plugin:** `dbwarden plugin add dbwarden-pgsql-types`. The `sequence` object handler ships in that plugin, not in core.

**Handler**: `SequenceHandler` (PREAMBLE phase, config-driven)

```python
pg_sequences=[
    {
        "name": "order_number_seq",
        "start": 1000,
        "increment": 1,
        "minvalue": 1,
        "maxvalue": 999999,
        "cycle": True,
    },
]
```

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE SEQUENCE name START WITH 1000 INCREMENT BY 1 MINVALUE 1 MAXVALUE 999999 CYCLE;` |
| Alter | `ALTER SEQUENCE name INCREMENT BY 10 MAXVALUE 9999999;` |
| Drop | `DROP SEQUENCE IF EXISTS name;` |

### ALTER SEQUENCE

Modify sequence parameters after creation:

```sql
ALTER SEQUENCE order_number_seq INCREMENT BY 10;
ALTER SEQUENCE order_number_seq RESTART WITH 5000;
ALTER SEQUENCE order_number_seq MAXVALUE 9999999;
ALTER SEQUENCE order_number_seq NO CYCLE;
```

DBWarden detects changes to any sequence option and emits `ALTER SEQUENCE` instead of drop+create when feasible.

## Options

| Key | SQL |
|-----|-----|
| `start` | `START WITH n` |
| `increment` | `INCREMENT BY n` |
| `minvalue` | `MINVALUE n` |
| `maxvalue` | `MAXVALUE n` |
| `cycle` | `CYCLE` / `NO CYCLE` |
| `cache` | `CACHE n` |
| `owned_by` | `OWNED BY table.column` |

### CACHE Option

`CACHE n` pre-allocates sequence values in memory for better performance:

```python
{
    "name": "order_number_seq",
    "start": 1000,
    "increment": 1,
    "cache": 100,
}
```

```sql
CREATE SEQUENCE order_number_seq START WITH 1000 INCREMENT BY 1 CACHE 100;
```

Higher cache values improve multi-session throughput but increase gaps on crashes (cached values are lost).

## Ownership

When `owned_by` is set, dropping the owning table/column automatically drops the sequence. If `owned_by` is `None`, the sequence is standalone and persists after table drops.

Transfer sequence ownership:

```sql
ALTER SEQUENCE order_number_seq OWNED BY orders.order_number;
```

## Schema-Qualified Sequences

Reference a sequence in a specific schema:

```python
{
    "name": "app.order_number_seq",
    "start": 1000,
}
```

## Sequence Functions

| Function | Description |
|----------|-------------|
| `nextval('name')` | Advance and return next value |
| `currval('name')` | Return last value obtained in session |
| `lastval()` | Return last value from any sequence in session |
| `setval('name', n)` | Set current value |
| `setval('name', n, is_called)` | Set value with `is_called` flag |

## Sequence Privileges

| Privilege | Description |
|-----------|-------------|
| `USAGE` | Allows `nextval` and `currval` |
| `SELECT` | Allows `currval` only |
| `UPDATE` | Allows `setval` |
| `ALL` | All sequence privileges |

```sql
GRANT USAGE ON SEQUENCE order_number_seq TO app_user;
```

## Sequence vs Auto-increment

Auto-increment on PK columns (SERIAL/IDENTITY) automatically creates and manages sequences. Independent sequences declared via `pg_sequences` are useful for manual sequence management, custom numbering, or shared sequences across tables.

### Sequence Gaps

Sequence values are not intended to be gap-free. Gaps can occur from:

| Cause | Description |
|-------|-------------|
| `CACHE` loss | Cached values lost on crash |
| Rollback | `nextval` is not rolled back on transaction abort |
| `setval` | Manual value adjustment |
| Concurrent deletes | Rows deleted do not free sequence values |

## Migration Safety

| Change | Severity |
|--------|----------|
| Add sequence | `INFO` |
| Drop sequence | `WARNING` |
| Change sequence options (start, increment, etc.) | `WARNING` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/storage-params/
========================================================================

# Storage Parameters

**Table storage parameters require the `dbwarden-pgsql-extensions` plugin:** `dbwarden plugin add dbwarden-pgsql-extensions`. The `storage_params` object handler ships in that plugin. Index-level storage parameters are part of core index handling and need no plugin.

PostgreSQL supports table-level and index-level storage parameters that control physical storage behaviour, autovacuum tuning, and performance characteristics.

DBWarden tracks storage parameters through the `with_params` field on `PgIndexSpec` and table-level handlers.

## Table Storage Parameters

Set via `ALTER TABLE t SET (param = value, ...)` using the handler infrastructure.

### Autovacuum Parameters

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `autovacuum_enabled` | `bool` | Enables/disables autovacuum for this table | `True` |
| `autovacuum_vacuum_threshold` | `int` | Minimum number of dead tuples before vacuum | `50` |
| `autovacuum_vacuum_scale_factor` | `float` | Fraction of table dead tuples before vacuum | `0.2` |
| `autovacuum_vacuum_ins_threshold` | `int` | Minimum INSERT dead tuples before vacuum (PG 17+) | `1000` |
| `autovacuum_vacuum_ins_scale_factor` | `float` | Fraction of INSERT dead tuples before vacuum (PG 17+) | `0.2` |
| `autovacuum_analyze_threshold` | `int` | Minimum modified tuples before analyze | `50` |
| `autovacuum_analyze_scale_factor` | `float` | Fraction of modified tuples before analyze | `0.1` |
| `autovacuum_vacuum_cost_delay` | `int` | Millisecond delay between vacuum cost units | `2` |
| `autovacuum_vacuum_cost_limit` | `int` | Cost limit before vacuum pauses | `-1` (use global) |
| `autovacuum_freeze_min_age` | `int` | Minimum age before FREEZE | `50000000` |
| `autovacuum_freeze_max_age` | `int` | Maximum age before forced FREEZE | `200000000` |
| `autovacuum_freeze_table_age` | `int` | Age at which whole-table freeze is considered | `150000000` |
| `autovacuum_multixact_freeze_min_age` | `int` | Minimum multixact age before freeze | `5000000` |
| `autovacuum_multixact_freeze_max_age` | `int` | Maximum multixact age before forced freeze | `200000000` |
| `autovacuum_multixact_freeze_table_age` | `int` | Table-level multixact freeze age | `150000000` |

### Tuning Parameters

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `fillfactor` | `int` | Percentage of page space to fill (1-100) | `100` |
| `toast_tuple_target` | `int` | Minimum tuple size before TOAST (PG 11+) | `2048` |
| `parallel_workers` | `int` | Number of parallel workers for scans (PG 11+) | `0` (use global) |
| `vacuum_truncate` | `bool` | Allow vacuum to truncate empty pages (PG 14+) | `True` |
| `log_autovacuum_min_duration` | `int` | Log autovacuum actions exceeding this (ms) (PG 14+) | `-1` (disabled) |

### Table-Level Fillfactor

The `pg_fillfactor` attribute on `PGTableMeta` sets the fillfactor storage parameter:

```python
class Meta(PGTableMeta):
    pg_fillfactor = 80
```

```sql
ALTER TABLE users SET (fillfactor = 80);
```

Fillfactor applies per-table for heap storage and per-index for B-tree indexes. A lower fillfactor leaves more free space for future updates, reducing page splits at the cost of denser storage.

### Toast Tuple Target

Controls when inline values are moved to TOAST storage. Lower values move larger tuples to TOAST earlier:

```sql
ALTER TABLE users SET (toast_tuple_target = 1024);
```

## Index Storage Parameters

Set via the `with_params` field on `PgIndexSpec`:

```python
PgIndexSpec("ix_users_email", ["email"],
    with_params={"fillfactor": 70})
```

```sql
CREATE INDEX CONCURRENTLY ix_users_email ON users (email) WITH (fillfactor = 70);
```

### Access Method-Specific Parameters

| Access Method | Parameter | Type | Default | Description |
|---------------|-----------|------|---------|-------------|
| B-tree | `fillfactor` | `int` | `90` | Page fill percentage |
| GiST | `buffering` | `str \| bool` | `auto` | `on`, `off`, or `auto` |
| BRIN | `pages_per_range` | `int` | `128` | Pages per block range |
| BRIN | `autosummarize` | `bool` | `off` | Auto-summarize on insert |
| Hash | `fillfactor` | `int` | `100` | Page fill percentage |

Example: BRIN parameters:

```python
PgIndexSpec("ix_orders_created_at", ["created_at"],
    using="brin",
    with_params={"pages_per_range": 32, "autosummarize": True})
```

```sql
CREATE INDEX CONCURRENTLY ix_orders_created_at ON orders USING BRIN (created_at) WITH (pages_per_range = 32, autosummarize = on);
```

## Using `with_params`

The `with_params` field on `PgIndexSpec` accepts a dict of parameter name-value pairs:

| Field | Value Type | Description |
|-------|------------|-------------|
| Key | `str` | PostgreSQL storage parameter name (e.g. `fillfactor`, `autosummarize`) |
| Value | `str \| int \| bool` | Parameter value. Booleans render as `on`/`off` |

Parameters are rendered as `WITH (key1 = val1, key2 = val2, ...)` in the `CREATE INDEX` statement.

## Changing Storage Parameters

Storage parameter changes are detected during the DIFF phase. Parameter changes for tables and indexes produce `ALTER TABLE ... SET (...)` or `ALTER INDEX ... SET (...)` DDL. Safety classification varies:

| Change | Severity |
|--------|----------|
| fillfactor change | `INFO` |
| autovacuum setting change | `INFO` |
| BRIN parameter change | `INFO` |
| Index `with_params` change | `INFO` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/tables-and-columns/
========================================================================

# Tables & Columns

**Handlers**: `TableHandler`, `RenameTableHandler`, `ColumnHandler`, `PgTableHandler`, `StorageParamsHandler` (DIFF phase)

## Table Lifecycle

| Operation | DDL |
|-----------|-----|
| Create table | `CREATE TABLE name (...)` |
| Drop table | `DROP TABLE IF EXISTS name CASCADE;` |
| Rename table | `ALTER TABLE old_name RENAME TO new_name;` |
| Alter table comment | `COMMENT ON TABLE name IS 'comment';` |

## Column Lifecycle

| Operation | DDL |
|-----------|-----|
| Add column | `ALTER TABLE t ADD COLUMN c type;` |
| Drop column | `ALTER TABLE t DROP COLUMN c;` |
| Change type | `ALTER TABLE t ALTER COLUMN c TYPE newtype;` |
| Change nullable | `ALTER TABLE t ALTER COLUMN c SET/DROP NOT NULL;` |
| Change default | `ALTER TABLE t ALTER COLUMN c SET/DROP DEFAULT;` |
| Change comment | `COMMENT ON COLUMN t.c IS 'comment';` |
| Change autoincrement | `CREATE/DROP SEQUENCE` + `SET/DROP DEFAULT nextval` |
| Change PG meta | `SET STORAGE`, `SET COMPRESSION`, `SET COLLATION`, identity options |
| Rename column | `ALTER TABLE t RENAME COLUMN c TO new_name;` |

## Table Properties

| Property | Meta Attribute | DDL |
|----------|---------------|-----|
| Fillfactor | `pg_fillfactor` | `ALTER TABLE t SET (fillfactor = N);` |
| Tablespace | `pg_tablespace` | `ALTER TABLE t SET TABLESPACE name;` |
| Unlogged | `pg_unlogged` | `CREATE UNLOGGED TABLE` / `ALTER TABLE t SET UNLOGGED;` |
| Inheritance | `pg_inherits` | `ALTER TABLE t INHERIT parent;` |
| Storage params | (via handler) | `ALTER TABLE t SET (param = value);` |
| ON COMMIT | `pg_on_commit` | `ON COMMIT DELETE ROWS` / `DROP` / `PRESERVE ROWS` |

### Storage Params

`pg_storage_params` stores raw PostgreSQL table storage options. `pg_fillfactor` is kept as a shorthand and is folded into `pg_storage_params["fillfactor"]` during discovery.

```python
class Meta(PGTableMeta):
    pg_storage_params = {
        "fillfactor": 80,
        "autovacuum_enabled": "false",
    }
```

Generated DDL:

```sql
ALTER TABLE users SET (fillfactor = 80, autovacuum_enabled = false);
```

## ALTER TABLE Operations

### SET SCHEMA

Move a table between schemas:

```sql
ALTER TABLE users SET SCHEMA app;
```

### SET LOGGED / SET UNLOGGED

Toggle between logged and unlogged modes:

```sql
ALTER TABLE users SET LOGGED;
ALTER TABLE users SET UNLOGGED;
```

`SET LOGGED` converts an unlogged table back to logged mode (all data is written to WAL).

### ALTER COLUMN SET STATISTICS

Set column-level statistics target:

```sql
ALTER TABLE users ALTER COLUMN email SET STATISTICS 500;
```

Higher values improve query planner estimates for columns with non-uniform distributions. Values range from `-1` (use default) to `10000`.

### ALTER COLUMN SET (attribute_option)

Set column-level attribute options:

```sql
ALTER TABLE users ALTER COLUMN email SET (n_distinct = 0.01);
ALTER TABLE users ALTER COLUMN email RESET (n_distinct);
```

Common attribute options: `n_distinct`, `n_distinct_inherited`.

### CLUSTER

Cluster a table based on an index:

```sql
ALTER TABLE users CLUSTER ON ix_users_email;
```

### ENABLE / DISABLE TRIGGER

Control trigger execution:

```sql
ALTER TABLE users DISABLE TRIGGER trg_users_updated_at;
ALTER TABLE users ENABLE TRIGGER trg_users_updated_at;
ALTER TABLE users ENABLE REPLICA TRIGGER trg_users_updated_at;
ALTER TABLE users ENABLE ALWAYS TRIGGER trg_users_updated_at;
```

### TRUNCATE

```sql
TRUNCATE TABLE users;
TRUNCATE TABLE users, posts CASCADE;
```

`TRUNCATE` is not generated by `make-migrations` but can be run manually for bulk data removal.

### Temporary Tables

Temporary tables use `ON COMMIT` for cleanup behaviour:

```sql
CREATE TEMPORARY TABLE temp_data (id int) ON COMMIT DELETE ROWS;
```

Supported `ON COMMIT` values:

| Value | Behaviour |
|-------|-----------|
| `PRESERVE ROWS` | Default: rows persist across transaction boundaries |
| `DELETE ROWS` | All rows deleted at transaction end |
| `DROP` | Table dropped at transaction end |

### LIKE Clause

Create a table with the same structure as an existing one:

```sql
CREATE TABLE users_archive (LIKE users INCLUDING ALL);
```

`INCLUDING ALL` copies defaults, constraints, indexes, and storage. This is a DDL operation (no data copied).
### Storage Params

`pg_storage_params` stores raw PostgreSQL table storage options. `pg_fillfactor` is kept as a shorthand and is folded into `pg_storage_params["fillfactor"]` during discovery.

```python
class Meta(PGTableMeta):
    pg_storage_params = {
        "fillfactor": 80,
        "autovacuum_enabled": "false",
    }
```

Generated DDL:

```sql
ALTER TABLE users SET (fillfactor = 80, autovacuum_enabled = false);
```
## Snapshot Format

### Column Extras

```json
{
  "name": "bio",
  "type": "text",
  "pg_column": {
    "collation": "en_US.UTF-8",
    "storage": "EXTENDED",
    "compression": "pglz",
    "generated": null,
    "identity": "always",
    "identity_start": 1,
    "identity_increment": 1
  }
}
```

### Table Extras

```json
{
  "pg_table": {
    "pg_fillfactor": 80,
    "pg_tablespace": "fastspace",
    "pg_unlogged": false,
    "pg_inherits": "base_entity",
    "pg_partition": {
      "strategy": "RANGE",
      "columns": ["created_at"]
    },
    "pg_excludes": [
      {"name": "excl_room_booking", "expression": "EXCLUDE USING gist (room_id WITH =, during WITH &&)"}
    ]
  }
}
```

## Reverse Engineering

`generate-models` queries `pg_class`, `pg_attribute`, `pg_constraint`, `pg_inherits`, `pg_tablespace`, `pg_partitioned_table`, and `pg_collation` to reverse-engineer all metadata.

```bash
$ dbwarden generate-models -d primary
```

Generated output for a table with identity, storage, compression, collation, and fillfactor:

```python
class User(Base):
    __tablename__ = "users"

    id:    Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True)
    bio:   Mapped[str | None] = mapped_column(Text, nullable=True)

    class Meta(PGTableMeta):
        comment = "Core user accounts"
        pg_fillfactor = 80

        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100, identity_increment=1)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", compression="pglz", collation="en_US.UTF-8")
```

For a partitioned table:

```python
class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    created_at: Mapped[datetime] = mapped_column(DateTime)

    class Meta(PGTableMeta):
        pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
```

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/type-mapping/
========================================================================

# Type Mapping

DBWarden normalizes SQLAlchemy column types to PostgreSQL native types during snapshot extraction and DDL generation.

## Standard SQLAlchemy Types

| SQLAlchemy Type | PostgreSQL Type | Notes |
|-----------------|-----------------|-------|
| `Integer` | `INTEGER` | |
| `BigInteger` | `BIGINT` | |
| `SmallInteger` | `SMALLINT` | |
| `String(n)` | `VARCHAR(n)` | |
| `Text` | `TEXT` | |
| `Unicode(n)` | `VARCHAR(n)` | |
| `UnicodeText` | `TEXT` | |
| `Boolean` | `BOOLEAN` | |
| `DateTime` | `TIMESTAMP WITHOUT TIME ZONE` | |
| `Date` | `DATE` | |
| `Time` | `TIME WITHOUT TIME ZONE` | |
| `Float` | `FLOAT` | |
| `Double` / `DOUBLE_PRECISION` | `DOUBLE PRECISION` | |
| `Numeric(p, s)` | `NUMERIC(p, s)` | |
| `LargeBinary` | `BYTEA` | |
| `PickleType` | `BYTEA` | |
| `JSON` | `JSON` | |
| `ARRAY(Type)` | `type[]` | e.g. `ARRAY(String)` → `text[]` |
| `Enum(*members)` | `CREATE TYPE ... AS ENUM` | Auto-creates enum type |

## PostgreSQL Dialect-Specific Types

These types from `sqlalchemy.dialects.postgresql` map directly to their PostgreSQL equivalents:

| SQLAlchemy Type | PostgreSQL Type | Notes |
|-----------------|-----------------|-------|
| `JSONB` | `JSONB` | Binary JSON, supports GIN indexes |
| `TIMESTAMP` | `TIMESTAMP WITHOUT TIME ZONE` | |
| `TIMESTAMPTZ` | `TIMESTAMP WITH TIME ZONE` | |
| `TIME` | `TIME WITHOUT TIME ZONE` | |
| `TIMETZ` | `TIME WITH TIME ZONE` | |
| `INTERVAL` | `INTERVAL` | |
| `UUID` | `UUID` | |
| `BYTEA` | `BYTEA` | |
| `OID` | `OID` | |
| `REGCLASS` | `REGCLASS` | |
| `TEXT` | `TEXT` | |
| `BOOLEAN` | `BOOLEAN` | |
| `CIDR` | `CIDR` | IPv4/IPv6 network |
| `INET` | `INET` | IPv4/IPv6 host address |
| `MACADDR` | `MACADDR` | MAC address |
| `MACADDR8` | `MACADDR8` | MAC address (EUI-64) |
| `MONEY` | `MONEY` | Currency amount |
| `TSVECTOR` | `TSVECTOR` | Full-text search document |
| `TSQUERY` | `TSQUERY` | Full-text search query |
| `INT4RANGE` | `INT4RANGE` | Range of integer |
| `INT8RANGE` | `INT8RANGE` | Range of bigint |
| `NUMRANGE` | `NUMRANGE` | Range of numeric |
| `DATERANGE` | `DATERANGE` | Range of date |
| `TSTZRANGE` | `TSTZRANGE` | Range of timestamptz |
| `TSRANGE` | `TSRANGE` | Range of timestamp |
| `BIT(n)` | `BIT(n)` | Fixed-length bit string |
| `VARBIT(n)` | `VARBIT(n)` | Variable-length bit string |
| `XML` | `XML` | XML data |
| `ARRAY(type, dimensions)` | `type[]` | Multi-dimensional array |
| `ENUM(*members)` | `CREATE TYPE ... AS ENUM` | Named enum (creates persistent type) |

## Auto-increment Normalization

| Condition | Resulting Type | Sequence Behavior |
|-----------|---------------|-------------------|
| `Integer` + `autoincrement=True` | `SERIAL` | Auto-creates `tablename_colname_seq` |
| `BigInteger` + `autoincrement=True` | `BIGSERIAL` | Auto-creates sequence |
| `Integer` + `autoincrement=False` | `INTEGER` | No sequence |
| `BigInteger` + `autoincrement=False` | `BIGINT` | No sequence |
| `Integer` (unspecified autoincrement) | `SERIAL` | Backward compatible |
| `GENERATED ALWAYS AS IDENTITY` | `INTEGER` | Creates implicit sequence |
| `GENERATED BY DEFAULT AS IDENTITY` | `INTEGER` | Creates implicit sequence |

See [DDL Behavior](ddl-behavior.md#auto-increment-lifecycle) for the full lifecycle.

## Type Normalization Details

### SERIAL / BIGSERIAL

`SERIAL` and `BIGSERIAL` are syntactic sugar for `INTEGER` / `BIGINT` with an auto-created sequence and a `DEFAULT nextval(...)` expression. DBWarden normalizes them during reverse-engineering:

- On input (`generate-models`): a column typed `INTEGER` with `nextval('seq'::regclass)` default is normalized to `Integer(autoincrement=True)`
- On output (`make-migrations`): a column with `autoincrement=True` emits `SERIAL` / `BIGSERIAL` in `CREATE TABLE`

### TIMESTAMP / TIMESTAMPTZ

| SQLAlchemy Type | Normalized DDL |
|----------------|----------------|
| `DateTime` | `TIMESTAMP WITHOUT TIME ZONE` |
| `TIMESTAMP` | `TIMESTAMP WITHOUT TIME ZONE` |
| `TIMESTAMPTZ` | `TIMESTAMP WITH TIME ZONE` |

### NUMERIC Precision

`Numeric(10, 2)` emits `NUMERIC(10, 2)`. Without precision: `NUMERIC`.

### JSONB vs JSON

- `JSON` → `JSON` (stores exact copy of input text)
- `JSONB` → `JSONB` (stores decomposed binary, supports indexing)

### ARRAY Handling

`ARRAY(String)` emits `text[]`. `ARRAY(Integer)` emits `integer[]`. Multi-dimensional arrays preserve dimensions:

| SQLAlchemy | PostgreSQL |
|------------|------------|
| `ARRAY(String)` | `text[]` |
| `ARRAY(Integer, dimensions=2)` | `integer[][]` |
| `ARRAY(JSONB)` | `jsonb[]` |

### Range Types

Range types accept `Range` objects in Python. DBWarden preserves the range type variant:

| SQLAlchemy | PostgreSQL | Example Value |
|------------|------------|---------------|
| `INT4RANGE` | `INT4RANGE` | `[1, 10)` |
| `TSTZRANGE` | `TSTZRANGE` | `["2024-01-01", "2024-12-31")` |
| `DATERANGE` | `DATERANGE` | `[2024-01-01, 2024-12-31)` |

### Enum Normalization

SQLAlchemy `Enum` types with `create_constraint=True` are extracted as `CREATE TYPE` statements. Enum members are tracked positionally so new values are added with `AFTER` to preserve ordering.

### Domain-Based Columns

When a column uses a domain type (e.g., `us_postal_code`), DBWarden preserves the domain type name in the snapshot rather than expanding to the base type. See [Types](types.md#domains) for domain lifecycle.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/types/
========================================================================

# Types

**Requires the `dbwarden-pgsql-types` plugin:** `dbwarden plugin add dbwarden-pgsql-types`. The `enum`, `domain`, and `composite_type` object handlers ship in that plugin, not in core.

DBWarden supports three object-level type families: enums, domains, and composite types. Enums are model-derived (auto-discovered from table columns). Domains and composite types are config-driven.

## Enums

**Handler**: `EnumHandler` (DIFF phase)

Enums are auto-discovered from column types during snapshot extraction. Enum values are tracked with position data, so `ALTER TYPE ... ADD VALUE ... AFTER ...` preserves ordering.

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE TYPE name AS ENUM ('val1', 'val2', ...)` |
| Add value | `ALTER TYPE name ADD VALUE 'newval' AFTER 'existing'` |
| Rename value | `ALTER TYPE name RENAME VALUE 'old' TO 'new'` (PG 10+) |
| Drop | `DROP TYPE IF EXISTS name` |

### Adding Enum Values

New enum values are inserted after the preceding existing value to preserve sort order:

```python
# Current enum: mood AS ENUM ('sad', 'ok', 'happy')
# Adding 'ecstatic' after 'happy':
# ALTER TYPE mood ADD VALUE 'ecstatic' AFTER 'happy'
```

### Enum Value Renaming (PG 10+)

```sql
ALTER TYPE mood RENAME VALUE 'sad' TO 'unhappy';
```

### Enum Value Deletion

PostgreSQL does **not** support `ALTER TYPE ... DROP VALUE`. Removing an enum value requires:
1. `ALTER TYPE name RENAME TO name_old`
2. `CREATE TYPE name AS ENUM (...)` without the removed value
3. `ALTER TABLE ... ALTER COLUMN c TYPE name USING c::text::name`
4. `DROP TYPE name_old`

This is **not** automated by DBWarden. Value removal is classified as a manual operation.

### Enum Type Normalization

SQLAlchemy's `Enum` type with `create_type=True` creates a persistent PostgreSQL enum type. Without it, the enum is rendered as a `VARCHAR` with a `CHECK` constraint.

## Domains

**Handler**: `DomainHandler` (PREAMBLE phase)

Domains are config-driven objects declared via `pg_domains`:

```python
pg_domains=[
    {
        "name": "us_postal_code",
        "type": "text",
        "not_null": True,
        "check": "VALUE ~ '^\d{5}(-\d{4})?$'",
    },
    {
        "name": "positive_int",
        "type": "int",
        "check": "VALUE > 0",
        "default": "1",
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE DOMAIN name AS base_type [DEFAULT default] [NOT NULL] [CHECK (expr)]` |
| Drop | `DROP DOMAIN IF EXISTS name CASCADE;` |
| Set default | `ALTER DOMAIN name SET DEFAULT expr;` |
| Drop default | `ALTER DOMAIN name DROP DEFAULT;` |
| Set not null | `ALTER DOMAIN name SET NOT NULL;` |
| Drop not null | `ALTER DOMAIN name DROP NOT NULL;` |
| Add constraint | `ALTER DOMAIN name ADD CONSTRAINT c CHECK (expr);` |
| Drop constraint | `ALTER DOMAIN name DROP CONSTRAINT c;` |
| Rename | `ALTER DOMAIN name RENAME TO new_name;` |
| Rename constraint | `ALTER DOMAIN name RENAME CONSTRAINT old TO new;` |
| Set schema | `ALTER DOMAIN name SET SCHEMA new_schema;` |

### Domain Constraint Validation

Domain constraints are checked on every use of the domain, not just at column creation. This means changing a domain's CHECK constraint can invalidate existing data in every table that uses the domain.

Changes to domain definition are detected as drop-then-create. When a domain check is relaxed or tightened, existing rows must satisfy the new constraint or be updated first.

## Composite Types

**Handler**: `CompositeTypeHandler` (PREAMBLE phase)

Composite types are config-driven objects declared via `pg_composite_types`:

```python
pg_composite_types=[
    {
        "name": "address",
        "columns": [
            {"name": "street", "type": "text"},
            {"name": "city", "type": "text"},
            {"name": "zip", "type": "text"},
        ],
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE TYPE name AS (col1 type1, col2 type2, ...)` |
| Drop | `DROP TYPE IF EXISTS name CASCADE;` |

### Composite Type Modification

PostgreSQL does **not** support `ALTER TYPE` for composite types. To modify a composite type:
1. `DROP TYPE name CASCADE;` (automatically drops dependent columns/tables)
2. `CREATE TYPE name AS (...);` with the new definition
3. Recreate any dropped columns referencing this type

This is **not** automated by DBWarden. Composite type changes are detected as drop-then-create with `CASCADE`.

### Schema

Composite types can be scoped to a schema:

```python
{"name": "address", "schema": "app", "columns": [...]}
```

## Range Types

PostgreSQL supports built-in range types that DBWarden normalizes during schema extraction:

| PostgreSQL Type | SQLAlchemy Type | Example Value |
|----------------|-----------------|---------------|
| `INT4RANGE` | `INT4RANGE` | `[1, 10)` |
| `INT8RANGE` | `INT8RANGE` | `[100, 200)` |
| `NUMRANGE` | `NUMRANGE` | `[0.0, 1.0)` |
| `DATERANGE` | `DATERANGE` | `[2024-01-01, 2024-12-31)` |
| `TSTZRANGE` | `TSTZRANGE` | `["2024-01-01 00:00:00+00", "2024-12-31 23:59:59+00")` |
| `TSRANGE` | `TSRANGE` | `[2024-01-01, 2024-12-31)` |

Range type bounds are canonicalized as `[lower, upper)` (inclusive lower, exclusive upper) by PostgreSQL.

## Type Mapping Summary

See [Type Mapping](type-mapping.md) for the complete SQLAlchemy-to-PostgreSQL type normalization reference.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/postgresql/views/
========================================================================

# Views

**Handler**: `ViewHandler` (DIFF phase)

Views are model-derived via `PGViewMeta`.

## Regular Views

```python
from dbwarden.databases.pgsql import PGViewMeta

class ActiveUser(Base):
    __tablename__ = "active_users"

    id: Mapped[int] = mapped_column(Integer)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(PGViewMeta):
        pg_view_query = "SELECT id, email, name FROM users WHERE active = true"
        pg_view_materialized = False
```

Generated DDL:
```sql
CREATE OR REPLACE VIEW active_users AS SELECT id, email, name FROM users WHERE active = true;
```

## Materialized Views

```python
class OrderSummary(Base):
    __tablename__ = "order_summary"

    user_id: Mapped[int] = mapped_column(Integer)
    total: Mapped[float] = mapped_column(Integer)

    class Meta(PGViewMeta):
        pg_view_query = "SELECT user_id, count(*) AS total FROM orders GROUP BY user_id"
        pg_view_materialized = True
```

Generated DDL:
```sql
CREATE MATERIALIZED VIEW order_summary AS SELECT user_id, count(*) AS total FROM orders GROUP BY user_id;
```

## WITH Options

### Security Barrier

Control whether the view acts as a security barrier:

```python
class Meta(PGViewMeta):
    pg_view_query = "SELECT id, email FROM users WHERE active = true"
    pg_view_options = "security_barrier"
```

```sql
CREATE OR REPLACE VIEW active_users WITH (security_barrier) AS ...;
```

Security barrier views prevent leaked subquery optimizations when used for row-level access control. Use this for views that expose a subset of a table based on session variables.

### WITH CHECK OPTION

Control what rows can be inserted/updated through the view:

```python
class Meta(PGViewMeta):
    pg_view_query = "SELECT id, email, name FROM users WHERE active = true"
    pg_view_check_option = "LOCAL"  # or "CASCADED"
```

```sql
CREATE OR REPLACE VIEW active_users AS ... WITH LOCAL CHECK OPTION;
```

| Option | Behaviour |
|--------|-----------|
| `LOCAL` | Check only this view's WHERE clause |
| `CASCADED` (default) | Check this view's WHERE and all underlying views' WHERE clauses |

Without `WITH CHECK OPTION`, rows can be inserted or updated through the view even if they would not satisfy the view's WHERE clause (they become invisible through the view but exist in the underlying table).

## Auto-Refresh

Set `pg_view_auto_refresh = True` to emit `REFRESH MATERIALIZED VIEW` in subsequent migrations:

```python
class Meta(PGViewMeta):
    pg_view_query = "..."
    pg_view_materialized = True
    pg_view_auto_refresh = True
```

```sql
REFRESH MATERIALIZED VIEW order_summary;
```

### REFRESH CONCURRENTLY

To avoid table locking during refresh:

```python
class Meta(PGViewMeta):
    pg_view_query = "..."
    pg_view_materialized = True
    pg_view_auto_refresh = True
    pg_view_refresh_concurrently = True
```

```sql
REFRESH MATERIALIZED VIEW CONCURRENTLY order_summary;
```

`CONCURRENTLY` requires a unique index on the materialized view. It takes longer but allows concurrent reads and writes.

### WITH DATA / WITH NO DATA

Control whether the materialized view is populated on creation:

| Option | Behaviour |
|--------|-----------|
| `WITH DATA` (default) | Populate immediately |
| `WITH NO DATA` | Create empty; must `REFRESH` before querying |

```python
class Meta(PGViewMeta):
    pg_view_query = "..."
    pg_view_materialized = True
    pg_view_with_data = False
```

## Schema-Qualified Views

```python
class Meta(PGViewMeta):
    pg_view_query = "SELECT id, email FROM users"
    pg_view_materialized = False
    pg_schema = "app"
```

```sql
CREATE OR REPLACE VIEW app.active_users AS SELECT id, email FROM users;
```

## Recursive Views

```sql
CREATE RECURSIVE VIEW view_name (col1, col2, ...) AS
    SELECT ...   -- non-recursive term
    UNION ALL
    SELECT ...   -- recursive term
;
```

Recursive views are declared via `pg_view_query` with the full recursive query text.

## Temporary Views

```sql
CREATE TEMPORARY VIEW temp_active_users AS SELECT * FROM active_users;
```

Temporary views are session-scoped and dropped automatically at session end.

## Updatable Views

PostgreSQL automatically makes simple views updatable (single table, no aggregates, no DISTINCT, no set operations). Complex views require `INSTEAD OF` triggers for updates.

## Indexes on Materialized Views

Materialized views support indexes. Create indexes directly on the materialized view table:

```python
class Meta(PGViewMeta):
    pg_view_query = "..."
    pg_view_materialized = True
    bg_view_materialized = True  # Note: use model's Meta if needed
```

Indexes on materialized views are created via `pg_indexes` on `PGTableMeta` (not `PGViewMeta`). For materialized views with `pg_view_auto_refresh`, indexes persist across refreshes.

## INSTEAD OF Triggers

For complex views that need to support INSERT/UPDATE/DELETE, use `INSTEAD OF` triggers. `pg_triggers` requires the `dbwarden-pgsql-extensions` plugin (`dbwarden plugin add dbwarden-pgsql-extensions`).

```python
pg_triggers=[{
    "name": "trg_active_users_insert",
    "table": "active_users",
    "function": "insert_active_user",
    "timing": "INSTEAD OF",
    "events": ["INSERT"],
    "for_each": "ROW",
}]
```

See [Functions & Triggers](functions-and-triggers.md) for trigger configuration.

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create regular view | `CREATE OR REPLACE VIEW name AS query;` |
| Create matview | `CREATE MATERIALIZED VIEW name AS query;` |
| Refresh matview | `REFRESH MATERIALIZED VIEW name;` |
| Refresh matview concurrently | `REFRESH MATERIALIZED VIEW CONCURRENTLY name;` |
| Drop | `DROP VIEW IF EXISTS name CASCADE;` / `DROP MATERIALIZED VIEW IF EXISTS name CASCADE;` |

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/round-trip/
========================================================================

# Round Trip Support

A **round-trip** backend is one where DBWarden can both read schema (via `generate-models`) and write schema (via `make-migrations` / `migrate`).

## Supported Backends

| Backend | `database_type` | Round-Trip |
|---------|------------------|------------|
| PostgreSQL | `postgresql` | Yes |
| MySQL | `mysql` | Yes |
| ClickHouse | `clickhouse` | Yes |
| SQLite | `sqlite` | Dev only |
| MariaDB | `mariadb` | No |

## How Round-Trip Verification Works

"First-class" means the round-trip is verified: reverse-engineer a live database with `generate-models`, feed the output back into `make-migrations`, and get **zero diff**.

## Per-Backend Details

### PostgreSQL

PostgreSQL is a **first-class backend** with full round-trip support. All metadata (identity columns, collation, storage, compression, generated columns, fillfactor, tablespace, inheritance, exclude constraints, deferrable foreign keys, and advanced index options) is captured by the snapshot, diffed correctly, and emitted as valid DDL.

See [PostgreSQL Deep Dive](postgresql/index.md) for the complete list of supported features.

### MySQL

MySQL is a **first-class backend** with full round-trip support. All metadata (engine, charset, collation, row format, auto_increment, unsigned columns, `ON UPDATE`, and column comments) is captured by the snapshot, diffed correctly, and emitted as valid DDL.

See [MySQL Deep Dive](mysql.md) for the complete list of supported features.

### ClickHouse

ClickHouse has full round-trip support: `generate-models` reads schema from a live ClickHouse server, and `make-migrations` / `migrate` auto-generates DDL for table operations.

See [ClickHouse Deep Dive](clickhouse/index.md) for the complete list of supported features.

### SQLite

SQLite is supported for **development workflows only** (`--dev` flag). It uses the same snapshot format as PostgreSQL but with SQLite-compatible DDL. SQLite is ideal for local iteration before running migrations against production.

### MariaDB

MariaDB is supported as a separate `database_type` (`mariadb`), but it does **not** have round-trip support. You can use MariaDB as a target database for migrations, but `generate-models` and full schema introspection are not available. Use `make-migrations` to write migrations manually.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/databases/sql-databases/
========================================================================

# SQL Databases

DBWarden supports PostgreSQL, MySQL, MariaDB, and SQLite. While all four share standard SQL DDL, each backend has distinct behaviors that affect generated migrations. This page documents backend-specific syntax, limitations, and edge cases.

## DDL Transactional Behavior

| Backend | Transactional DDL | Impact |
|---------|------------------|--------|
| PostgreSQL | Yes | Entire migration file succeeds or rolls back atomically |
| MySQL / MariaDB | No | Each DDL auto-commits; partial failure leaves schema inconsistent |
| SQLite | Mostly | Per-statement auto-commit outside explicit transactions |

**PostgreSQL**: DDL is transactional. If a migration file contains multiple statements and one fails, all prior DDL in that file is rolled back. This makes PostgreSQL the safest backend for automated migration runs.

PostgreSQL is also a **first-class backend** with full support for identity columns, collation, storage, compression, generated columns, table fillfactor, tablespace, inheritance, EXCLUDE constraints, deferrable FK options, and advanced index parameters. See [PostgreSQL Deep Dive](postgresql/index.md) for complete details.

**MySQL / MariaDB**: DDL statements auto-commit immediately. If a 5-statement migration fails on the 4th statement, the first 3 are already committed and cannot be rolled back. Manual inspection and recovery may be needed. Always test MySQL/MariaDB migrations in a staging environment first.

**SQLite**: DDL is transactional only within explicit `BEGIN/COMMIT` blocks. DBWarden migration files are executed with each statement as a separate implicit transaction. This means SQLite is similar to MySQL in practice; partial failure is possible.

## Column Rename

| Backend | Syntax | Supported |
|---------|--------|-----------|
| PostgreSQL | `ALTER TABLE t RENAME COLUMN old TO new` | Yes |
| SQLite | `ALTER TABLE t RENAME COLUMN old TO new` | Yes (3.25+) |
| MySQL | `ALTER TABLE t CHANGE old new type` | Workaround needed |
| MariaDB | `ALTER TABLE t CHANGE old new type` | Workaround needed |

PostgreSQL and SQLite (3.25+) support `RENAME COLUMN` natively. DBWarden emits the standard form for all backends. If you need a backend that does not support native column rename (e.g., MySQL < 8.0, MariaDB), you must write a manual migration or verify the generated SQL.

## Column Type Change

| Backend | Syntax | Supported |
|---------|--------|-----------|
| PostgreSQL | `ALTER TABLE t ALTER COLUMN c TYPE newtype` (commented-out `USING` by default, active with `--postgres-auto-using`) | Yes |
| MySQL / MariaDB | `ALTER TABLE t MODIFY COLUMN c newtype` | Yes |
| SQLite | Not supported | Comment emitted |

**PostgreSQL**: Emits `ALTER TABLE t ALTER COLUMN c TYPE newtype` with a commented-out `-- USING col::newtype` line. Pass `--postgres-auto-using` on `make-migrations` to emit an active `USING` clause.

**MySQL / MariaDB**: Emits `ALTER TABLE t MODIFY COLUMN c newtype`. Note that `MODIFY COLUMN` requires specifying the entire column definition, not just the type. DBWarden includes only the type in the `MODIFY` statement; if you need additional attributes (e.g., `NOT NULL`, `DEFAULT`), add them manually.

**SQLite**: `ALTER COLUMN TYPE` is not supported. DBWarden emits a comment:

```sql
-- SQLite does not support ALTER COLUMN TYPE.
-- Use 'dbwarden new' to write a manual migration for:
-- ALTER TABLE users ALTER COLUMN name TYPE TEXT
```

Table recreation is required to change a column's type in SQLite.

## Column Nullable Change

| Backend | Syntax | Supported |
|---------|--------|-----------|
| PostgreSQL | `ALTER TABLE t ALTER COLUMN c [SET/DROP] NOT NULL` | Yes |
| MySQL / MariaDB | `ALTER TABLE t MODIFY COLUMN c coltype [NOT] NULL` | Yes |
| SQLite | Not supported | Comment emitted |

**PostgreSQL**: Uses `SET NOT NULL` / `DROP NOT NULL`. No column type needed.

**MySQL / MariaDB**: Uses `MODIFY COLUMN` which requires the full column type. DBWarden includes the type from the model column definition. If the type is not available, nullable changes for MySQL/MariaDB may produce incomplete SQL.

**SQLite**: Not supported. A comment is emitted:

```sql
-- SQLite: ALTER TABLE users ALTER COLUMN email SET NOT NULL (not supported)
```

## Column Default Change

All four backends support `ALTER TABLE t ALTER COLUMN c SET DEFAULT value` and `ALTER TABLE t ALTER COLUMN c DROP DEFAULT`. Default changes work uniformly across all SQL backends.

## Foreign Key Handling

| Backend | ADD FK | DROP FK | Notes |
|---------|--------|---------|-------|
| PostgreSQL | `ADD CONSTRAINT ... FOREIGN KEY` | `DROP CONSTRAINT ...` | Supports `ON DELETE`, `ON UPDATE`, `DEFERRABLE` |
| MySQL | `ADD CONSTRAINT ... FOREIGN KEY` | `DROP FOREIGN KEY ...` | Uses constraint name, not FK name |
| MariaDB | `ADD CONSTRAINT ... FOREIGN KEY` | `DROP FOREIGN KEY ...` | Same as MySQL |
| SQLite | Not supported (comment) | Not supported (comment) | Recreate table |
| ClickHouse | Not supported (error) | Not supported (error) | `ForeignKey()` raises `DBWardenConfigError` |

**PostgreSQL FK options**: `ON DELETE`, `ON UPDATE`, and `DEFERRABLE INITIALLY DEFERRED` are fully supported. See [PostgreSQL Deep Dive](postgresql/index.md).

**Validation**: Before emitting `ADD FOREIGN KEY`, DBWarden verifies that the referenced table and columns exist in the snapshot. If they don't, the FK is silently skipped to avoid generating broken SQL. This can happen when the referenced table is added in the same migration batch. If an FK is unexpectedly missing from generated SQL, check whether the referenced table exists in the snapshot.

**Content-based comparison**: FKs are compared by a 6-tuple signature `(columns, ref_table, ref_columns, on_delete, on_update, deferrable)`, not by constraint name. Renaming an FK constraint or changing options produces a drop+add.

## Index Handling

| Backend | CREATE INDEX | DROP INDEX | Notes |
|---------|-------------|------------|-------|
| PostgreSQL | `CREATE [UNIQUE] INDEX [CONCURRENTLY] ... USING <method> INCLUDE (<cols>) WITH (<params>) WHERE <pred> TABLESPACE <ts>` | `DROP INDEX` | Full feature support |
| MySQL / MariaDB | `CREATE [UNIQUE] INDEX` | `DROP INDEX` | Standard |
| SQLite | `CREATE [UNIQUE] INDEX` | `DROP INDEX` | Standard |

**PostgreSQL advanced parameters**: all are supported in `_build_index_sql`. See [PostgreSQL Deep Dive](postgresql/index.md) for full coverage.

**PostgreSQL `CONCURRENTLY`**: DBWarden defaults to `CREATE INDEX CONCURRENTLY` to avoid table locking. Pass `--no-concurrent` when the migration must run inside a transaction block (PostgreSQL requires `CONCURRENTLY` outside a transaction).

**Full-content comparison**: Indexes are compared by **all** attributes (using, unique, where, include, with_params, tablespace, nulls_not_distinct, column_sorting, concurrently), not just columns + name. Any difference produces a drop+add; ALTER INDEX is not used.

**Auto-generated names**: `idx_{table}_{col1}_{col2}` (non-unique), `uq_{table}_{col1}_{col2}` (unique). Non-btree `USING` methods append a suffix: `idx_{table}_{col}_{method}`.

**No ALTER INDEX**: All index parameter changes (adding a WHERE clause, switching USING methods, changing sort order) produce `DROP INDEX` + `CREATE INDEX`. This is intentional; `ALTER INDEX` support varies widely across backends and index attribute types.

## Safe Type Change

| Backend | Supported | Behavior |
|---------|-----------|----------|
| PostgreSQL | Yes | Multi-step: add temp column, backfill comment, verify, drop+rename |
| MySQL / MariaDB | Yes | Multi-step: add temp column, backfill comment, verify, drop+rename |
| SQLite | No | Comment emitted |

The `--safe-type-change` flag generates a multi-step strategy:
1. Add a temporary column with the new type
2. Emit a `--` comment with an `UPDATE` statement template
3. Emit a verification comment
4. After manual verification, drop the old column and rename the temporary column

On SQLite, this strategy is not supported because SQLite cannot drop columns (before 3.35.0) and has limited ALTER TABLE support. A comment is emitted instead.

## Table Rename

All four SQL backends support `ALTER TABLE t RENAME TO newname`. The syntax is uniform. ClickHouse is the only supported backend that does not support table rename (see [ClickHouse](clickhouse/index.md)).

## DROP COLUMN Warning

All DROP COLUMN statements are prefixed with a warning comment:

```sql
-- WARNING: DROPPING COLUMN users.legacy_field

ALTER TABLE users DROP COLUMN legacy_field
```

This applies to all SQL backends. The warning is a comment only and does not affect execution. In MySQL/MariaDB, be especially careful with DROP COLUMN since the DDL auto-commits and cannot be rolled back.

## DROP TABLE

`DROP TABLE` emits a rollback comment that references restoring from snapshot. The actual rollback SQL is a placeholder and must be written manually if needed.

## Statement Ordering

```
RENAME TABLE         (0)
RENAME COLUMN        (1)
ALTER COLUMN TYPE    (2)
ALTER COLUMN NULLABLE (3)
ALTER COLUMN DEFAULT (4)
CREATE TABLE         (5)
ADD COLUMN           (6)
ALTER FOREIGN KEY    (7)
ALTER INDEX          (8)
DROP COLUMN          (9)
DROP TABLE           (10)
ALTER TABLE COMMENT  (11)
ALTER COLUMN COMMENT (12)
ALTER TABLE OPTIONS  (13)
ALTER TABLE CONSTRAINT (14)
```

This ordering ensures safe execution across all SQL backends. Table renames come first so all subsequent ops use the new name. Drops come last to minimize risk of referencing dropped objects.

## Migration Name Generation

Auto-generated migration names are truncated to 72 characters. Operation words (`add`, `drop`, `alter`, `create`, `rename`, `add_columns`, etc.) are preserved during truncation; table and column names are shortened as needed. This applies uniformly across all backends.

## Resolved From Values

The plan JSON `resolved_from` field indicates how a rename was confirmed:

| Value | Meaning |
|-------|---------|
| `"rename_flag"` | Explicitly declared via `--rename` or `--rename-table` CLI flag |
| `"prompt"` | Confirmed interactively by the user |
| (absent) | Auto-detected and kept without explicit confirmation |

## Known Backend-Specific Limitations

### PostgreSQL
- `USING` clause for type casts is not auto-generated (write a manual migration)
- `CREATE INDEX CONCURRENTLY` may not work within multi-statement transactions
- `ALTER COLUMN ADD GENERATED ... AS (expr) STORED` is not supported by PostgreSQL; DBWarden emits a comment placeholder

### MySQL / MariaDB
- DDL is not transactional; partial failure leaves schema inconsistent
- `MODIFY COLUMN` requires full column definition; auto-generated nullable change SQL includes the column type but may omit other attributes
- Column rename is not natively supported (requires `CHANGE` syntax with type)
- Foreign key drop uses `DROP FOREIGN KEY` (constraint name is still the auto-generated name)

### SQLite
- `ALTER COLUMN TYPE` is not supported (comment emitted)
- `ALTER COLUMN [SET/DROP] NOT NULL` is not supported (comment emitted)
- `--safe-type-change` is not supported (comment emitted)
- FK constraints are not directly alterable (comment suggesting table recreation)
- Column rename supported since 3.25.0; older versions need manual migration
- Type affinity differs from server databases (e.g., `VARCHAR(255)` becomes `TEXT`)
- Limited to single-writer; no concurrent writes

========================================================================
PAGE: https://dbwarden.emiliano-go.com/features/
========================================================================

# Features

This page gives a compact overview of the main DBWarden features, with short examples. Use it to understand the surface area of the tool before diving into the guides and reference pages.

## SQL-First Migrations

DBWarden writes migrations as plain SQL files. Each file contains both an `--upgrade` section and a `--rollback` section.

```sql
-- upgrade
CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE
);

-- rollback
DROP TABLE users;
```

This keeps schema changes reviewable in code review and runnable without hidden ORM magic.

## Typed Database Configuration

Configure one or many databases with explicit `database_config(...)` calls.

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://default:@localhost:8123/analytics",
    model_paths=["app.analytics_models"],
    model_tables=["events", "page_views"],
)
```

Each entry is validated before use, including database names, table names, `model_paths`, `model_tables`, and duplicate target detection.

## Model-Driven Migration Generation

DBWarden reads SQLAlchemy models, diffs them against the live schema or an offline model state, and emits SQL.

```text
$ dbwarden make-migrations "add posts table" --database primary
Created migration: migrations/primary/primary__0002_add_posts_table.sql
```

## Backend-Specific Metadata

PostgreSQL, MySQL/MariaDB, and ClickHouse support first-class metadata through `class Meta`.

PostgreSQL tables:

```python
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg


class Meta(PGTableMeta):
    pg_fillfactor = 80
    pg_schema = "app"

    class id(PGColumnMeta):
        pg = pg.field(identity="always")
```

PostgreSQL views and materialized views:

```python
from dbwarden.databases.pgsql import PGViewMeta


class Meta(PGViewMeta):
    pg_view_query = "SELECT id, email FROM users WHERE active = true"
    pg_view_materialized = False
    pg_schema = "app"
```

Set `pg_view_auto_refresh = True` for materialized views that need `REFRESH MATERIALIZED VIEW` on every migration cycle.

MySQL

```python
from dbwarden.databases.mysql import MyTableMeta, MyColumnMeta, my


class Meta(MyTableMeta):
    my_engine = "InnoDB"
    my_charset = "utf8mb4"

    class id(MyColumnMeta):
        my = my.field(unsigned=True)
```

ClickHouse example:

```python
from dbwarden.databases.clickhouse import CHTableMeta, ChEngineSpec, ChIndexSpec


class Meta(CHTableMeta):
    ch_engine = ChEngineSpec("MergeTree")
    ch_order_by = ["event_date", "id"]
    ch_indexes = [
        ChIndexSpec("ix_payload", ["payload"], type="bloom_filter", granularity=1),
    ]
```

## Safety Classification

Use `check` to classify changes before generating or applying SQL.

```text
$ dbwarden check --database primary
SAFE      add column users.bio
WARN      shrink varchar users.email
CRITICAL  drop table audit_log
```

Safety levels are `SAFE`, `INFO`, `WARN`, and `CRITICAL`.

## Read-Only Schema Diffing

Use `diff` when you want to inspect differences without writing migration files.

```bash
$ dbwarden diff --database primary
```

This is useful during reviews, debugging, and CI checks.

## Reverse-Engineering Models

Generate SQLAlchemy model code from a live database.

```bash
$ dbwarden generate-models --database primary --tables users,posts
```

This is useful when adopting DBWarden in an existing project, documenting an inherited schema, or recovering model definitions.

## Offline Migrations

DBWarden can generate migrations without connecting to a live database, by diffing current models against an exported model state.

```bash
$ dbwarden export-models --database primary
$ dbwarden make-migrations "offline change" --offline --database primary
```

This is useful for CI pipelines and restricted environments.

## Multi-Database Workflows

Manage multiple backends from one repository, each with its own migration directory and model set.

```bash
$ dbwarden migrate --database primary
$ dbwarden migrate --database analytics
```

You can also show status across all configured databases:

```bash
$ dbwarden status --all
```

## Dev Mode

Use `--dev` to point a configured database at a separate development target.

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

```bash
$ dbwarden --dev make-migrations "test locally" --database primary
$ dbwarden --dev migrate --database primary
```

## Seed Management

DBWarden tracks SQL and Python seed files separately from schema migrations.

```bash
$ dbwarden seed create "load countries" --type sql --database primary
$ dbwarden seed apply --database primary
$ dbwarden seed list --database primary
```

This is useful for reference data, lookup tables, and repeatable environment setup.

## FastAPI Integration

`database_config(...)` returns a `DatabaseHandle` that can be used directly in FastAPI dependencies.

```python
from fastapi import APIRouter
from .dbwarden import primary


router = APIRouter()


@router.get("/users")
async def list_users(session: primary.async_session):
    ...
```

This gives one shared source of truth for migrations, runtime connections, and session injection.

## Sandbox Testing

Apply migrations in a temporary sandbox database before applying them for real.

```bash
$ dbwarden migrate --sandbox --database primary
```

This is useful when validating generated SQL against a throwaway environment.

## Status, History, and Rollback

DBWarden includes built-in commands for operational visibility.

```bash
$ dbwarden status --database primary
$ dbwarden history --database primary
$ dbwarden rollback --count 1 --database primary
```

These commands make the migration lifecycle inspectable and reversible.

## Next Steps

- Follow [Get Started](getting-started/setup.md)
- Explore [Cookbook & Examples](cookbook/index.md)
- Use [CLI Reference](cli-reference.md) for command lookup

========================================================================
PAGE: https://dbwarden.emiliano-go.com/getting-started/developing-locally/
========================================================================

# Developing Locally

This guide covers the local development workflow: using a development database, checking diffs safely, reverse-engineering models, and generating offline migrations.

## Use Dev Mode

Dev mode swaps the configured database target for `dev_database_url` and `dev_database_type`.

Configuration example:

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

Run local commands against the development target:

```text
$ dbwarden --dev make-migrations "test local change" --database primary
Created migration: migrations/primary/primary__0002_test_local_change.sql
$ dbwarden --dev migrate --database primary
Applying migration: primary__0002_test_local_change.sql
Migration applied successfully
```

## SQLite Translation

Using SQLite in dev mode is common, but not every type or default translates perfectly from server databases. DBWarden handles this by translating backend-specific types and warning when fidelity is reduced.

If you want those warnings to become hard failures, use:

```bash
$ dbwarden --dev --strict-translation make-migrations "validate translation" --database primary
```

## Check the Planned Changes

Use `diff` when you want to inspect differences without writing files:

```bash
$ dbwarden diff --database primary
```

Use `check` when you want a safety classification:

```text
$ dbwarden check --database primary
SAFE      add column users.bio
WARN      shrink varchar users.email
CRITICAL  drop table audit_log
```


## Offline Migrations

Offline migrations let you generate SQL without connecting to a live database. The workflow is:

1. Export the current model state.
2. Change your models.
3. Generate a migration with `--offline`.

Commands:

```bash
$ dbwarden export-models --database primary
$ dbwarden make-migrations "offline schema change" --offline --database primary
```

This is useful for CI, restricted environments, and workflows where the migration plan should not depend on a live database connection.

For a full walkthrough, see [Cookbook: Offline & CI](../cookbook/04-offline-ci.md).

## Local Validation Loop

A practical local loop looks like this:

```bash
$ dbwarden --dev diff --database primary
$ dbwarden --dev check --database primary
$ dbwarden --dev make-migrations "local change" --database primary
$ dbwarden --dev migrate --database primary
$ dbwarden --dev status --database primary
```

This keeps feedback fast while still using the same toolchain you use in production.

Next, continue with [Workflows](workflows.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/getting-started/first-migration/
========================================================================

# Your First Migration

This guide walks through the core DBWarden workflow: define models, generate SQL, apply the migration, inspect the result, and roll it back.

## Create the Models

Create `app/models.py`:

```python
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases import IndexSpec, TableMeta


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    bio: Mapped[str | None] = mapped_column(Text, nullable=True)

    class Meta(TableMeta):
        comment = "Core user accounts"


class Post(Base):
    __tablename__ = "posts"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String(255), nullable=False)
    body: Mapped[str] = mapped_column(Text, nullable=False)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False)

    class Meta(TableMeta):
        indexes = [
            IndexSpec(name="ix_posts_created_at", columns=["created_at"]),
        ]
```

## Generate the Migration

Run:

```text
$ dbwarden make-migrations "create core tables" --database primary
Created migration: migrations/primary/primary__0001_create_core_tables.sql
```

DBWarden compares your current models against the live schema, or snapshot state, and writes a new SQL migration file.

The `make-migrations` and `migrate` command names deliberately mirror Django's `makemigrations` and `migrate`, so the generate-then-apply loop is familiar if you have used Django. The rest of the CLI follows its own conventions.

## Review the Generated SQL

Open the new file. It will look roughly like this:

```sql
-- upgrade

CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    bio TEXT
);

CREATE TABLE IF NOT EXISTS posts (
    id SERIAL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    user_id INTEGER NOT NULL REFERENCES users(id),
    created_at TIMESTAMP NOT NULL
);

CREATE INDEX IF NOT EXISTS ix_posts_created_at ON posts (created_at);

-- rollback

DROP INDEX IF EXISTS ix_posts_created_at;
DROP TABLE posts;
DROP TABLE users;
```

The exact SQL depends on the backend, but the structure is always the same:

- `-- upgrade` contains the forward change
- `-- rollback` contains the reverse change

## Apply the Migration

Run:

```text
$ dbwarden migrate --database primary
Applying migration: primary__0001_create_core_tables.sql
Migration applied successfully
```

Internally, DBWarden resolves the config, acquires the migration lock, executes the upgrade SQL, records the checksum, and releases the lock.

## Verify the Result

Run:

```text
$ dbwarden status --database primary
Database: primary
Applied migrations: 1
Pending migrations: 0
$ dbwarden history --database primary
1  primary__0001_create_core_tables.sql  applied
```

Use `status` to see the current state of the migration queue. Use `history` to see what has been applied and in what order.

You can also inspect the live schema directly:

```bash
$ dbwarden check-db --database primary
```

This is useful when you want a read-only view of what the database currently contains.

## Roll Back the Migration

Run:

```text
$ dbwarden rollback --count 1 --database primary
Rolling back migration: primary__0001_create_core_tables.sql
Rollback completed successfully
```

After that, the database is back to its previous schema state.

## Step by Step

### Step 1: Define the Base Class

```python
from sqlalchemy.orm import DeclarativeBase


class Base(DeclarativeBase):
    pass
```

DBWarden does not export a shared `Base`. You define a local SQLAlchemy declarative base in your project.

### Step 2: Define the Models

```python
class User(Base):
    __tablename__ = "users"
```

Every model maps to a table. Columns come from normal SQLAlchemy field declarations. Table-level migration metadata lives in `class Meta`.

### Step 3: Add Table Metadata

```python
class Meta(TableMeta):
    comment = "Core user accounts"
```

`TableMeta` is the cross-database surface for comments, indexes, checks, and unique constraints.

### Step 4: Generate SQL

```bash
$ dbwarden make-migrations "create core tables" --database primary
```

This command inspects the configured models, compares them with the current schema, and emits a SQL file. The file becomes part of your normal code review and deployment workflow.

### Step 5: Review Upgrade and Rollback

Every migration file contains both directions. This is one of DBWarden's core design choices: a migration is not complete until the rollback exists.

### Step 6: Apply the Migration

```bash
$ dbwarden migrate --database primary
```

This executes pending migrations in order and records them in the migration table.

### Step 7: Verify the State

```bash
$ dbwarden status --database primary
$ dbwarden history --database primary
```

Verification is part of the workflow, not optional cleanup.

### Step 8: Roll It Back

```bash
$ dbwarden rollback --count 1 --database primary
```

Rolling back the first migration confirms that the file contains valid reverse SQL, not just valid forward SQL.

## Manual Migrations

Not every schema or data change should be auto-generated. When the change is not model-driven, create a manual migration file:

```bash
$ dbwarden new "manual hotfix" --database primary
```

Use manual migrations for cases like:

- data backfills
- type changes that require custom `USING` expressions
- backend-specific operations that need hand-written SQL

DBWarden will track these files the same way it tracks generated migrations.

Next, continue with [Developing Locally](developing-locally.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/getting-started/first-steps/
========================================================================

# First Steps

This walkthrough is the foundation of the DBWarden workflow.

The goal is not just to run commands, but to understand why each step exists and how it fits the migration lifecycle.

## Step 1: Initialize the project

```bash
$ dbwarden init
```

This creates:

- a migrations directory structure
- a Python configuration scaffold (`dbwarden.py`)

Why it matters: DBWarden expects a project-local migration layout and config source so migration behavior is deterministic per repository.

## Step 2: Define one explicit database entry

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    model_paths=["app.models"],
    model_tables=["users"],
)
```

Why it matters: DBWarden resolves migration targets from explicit typed entries, not inferred environment state.

## Step 3: Add SQLAlchemy models

DBWarden uses model metadata to generate migration SQL. A minimal model example:

```python
from sqlalchemy import DateTime, Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from datetime import datetime


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
```

Why it matters: model metadata is the input to `make-migrations`.

## Step 4: Generate migration SQL

```bash
$ dbwarden make-migrations -d "create users table" --database primary
```

DBWarden creates a versioned SQL file under `migrations/primary/`.

Why it matters: this file is now part of your code review process and deployment artifact.

## Step 5: Review the generated migration

Open the file and validate both sections:

```sql
-- upgrade

-- rollback
```

Why it matters: rollback quality determines recovery quality.

## Step 6: Apply migrations

```bash
$ dbwarden migrate --database primary
```

During execution DBWarden:

1. resolves config and target database
2. acquires migration lock
3. executes pending SQL
4. stores migration record and checksum
5. releases lock

## Step 7: Verify the result

```bash
$ dbwarden status --database primary
$ dbwarden history --database primary
```

Use status to confirm pending/applied counts and history to confirm execution order.

## Common first-run issues

- `No configuration found`: ensure your project has one discovered config source with `database_config(...)`
- `Database '<name>' not found`: ensure `--database` matches configured `database_name`
- `No SQLAlchemy models found`: set `model_paths` explicitly in config

## Next Steps

- [Configuration](../configuration/index.md)
- [Your First Migration](first-migration.md)
- [Developing Locally](developing-locally.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/getting-started/migrating-from-alembic/
========================================================================

# Migrating from Alembic

This guide is for teams with an existing Alembic setup who want to switch to DBWarden. The migration is low-risk by design: your SQLAlchemy models stay exactly where they are, your database is never rebuilt, and your Alembic history remains in git. You are replacing the migration workflow, not the schema.

If you are evaluating rather than migrating, read [Why DBWarden](../index.md#why-dbwarden) first. The short version: Alembic maintains schema truth in a chain of revision scripts, DBWarden maintains it in your models and derives plain SQL migrations from them.

## Concept mapping

Every Alembic concept has a DBWarden counterpart. This table is the mental model for the whole guide.

| Alembic | DBWarden |
|---|---|
| `alembic.ini` + `env.py` | `dbwarden.py` config file |
| Revision script (`.py`) | Migration file (`.sql`) with `-- upgrade` / `-- rollback` sections |
| Revision chain as source of truth | SQLAlchemy models as source of truth |
| `alembic revision --autogenerate` | `dbwarden make-migrations "description"` |
| `alembic upgrade head` | `dbwarden migrate` |
| `alembic downgrade -1` | `dbwarden rollback` |
| `alembic stamp <rev>` | `dbwarden migrate --baseline` |
| `alembic current` | `dbwarden status` |
| `alembic history` | `dbwarden history` |
| `alembic upgrade head --sql` (offline mode) | `dbwarden make-migrations --sql`, or full offline generation via `export-models` + `--offline` |
| `alembic_version` table | DBWarden migration table |
| Hand-written revision | `dbwarden new "description"` (manual SQL migration) |

Two Alembic features have no direct equivalent, and you should know that before starting:

- **Python data migrations.** Alembic revisions can run arbitrary Python. DBWarden manual migrations (`dbwarden new`) are SQL. Most backfills express well in SQL; if yours do not, plan for how you will run them (application scripts, one-off jobs) before switching.
- **Revision branching and merging.** DBWarden uses a linear versioned sequence per database. If your team relies on `alembic merge` workflows, the linear model is a real change to your process.

## Prerequisites

- Python 3.12+ and SQLAlchemy 2.0+
- A database whose schema currently matches your models (run your pending Alembic migrations first, and resolve any drift)
- Your SQLAlchemy models importable from the project

## Step 1: Install and configure

```bash
uv add dbwarden
```

Create `dbwarden.py` in your project root. This replaces `alembic.ini` and `env.py`:

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/myapp",
    database_url_async="postgresql+asyncpg://user:pass@localhost:5432/myapp",
)
```

Model discovery is automatic. If you want explicit control over which modules are scanned, pass `model_paths`. See [Setup](setup.md) and [Model Discovery](../configuration/model-discovery.md).

Then initialize:

```bash
dbwarden init
```

## Step 2: Generate the baseline migration

Your first `make-migrations` run has no prior DBWarden state, so it generates the full schema from your models:

```bash
dbwarden make-migrations "baseline from alembic"
```

Review the generated `.sql` file. It should describe the schema you already have. If something looks wrong here, your models and your database disagree, and it is far better to learn that now than later. Fix the models (or the database) and regenerate before continuing.

## Step 3: Baseline the existing database

Your database already has the schema, so the baseline migration must be recorded as applied without executing:

```bash
dbwarden migrate --baseline
```

This marks the migration as applied in DBWarden's migration table. Nothing runs against the database. This is the equivalent of `alembic stamp head` for a fresh setup.

## Step 4: Verify convergence

```bash
dbwarden status
dbwarden check
dbwarden diff
```

`status` should show no pending migrations. `diff` compares your models against the live database and should report no differences. If `diff` is clean, DBWarden and reality agree, and the migration is effectively done.

For extra confidence, make a trivial model change (add a nullable column), run `dbwarden make-migrations`, inspect the generated SQL and its rollback, then revert the change and delete the generated file. That exercise shows you the full loop before you rely on it.

## Step 5 (optional): Set up offline generation for CI

If your CI previously needed a database service for `alembic revision --autogenerate` checks, you can drop it entirely:

```bash
dbwarden export-models --database primary
git add .dbwarden/model_state.primary.json
```

The state file is named after the database, so a database called `primary` produces `.dbwarden/model_state.primary.json`.

From then on, any machine can generate migrations without a database connection:

```bash
dbwarden make-migrations "description" --offline
```

See [Cookbook: Offline & CI](../cookbook/04-offline-ci.md). Do not delete the model state file; it is the offline source of truth. If it is ever lost, restore it from git or regenerate with `export-models`.

## Step 6: Retire Alembic

Once you trust the new workflow:

- Remove `alembic.ini`, `env.py`, and the `versions/` directory from the active workflow. They stay in git history, which is where migration archaeology belongs.
- Remove `alembic` from your dependencies and any `alembic upgrade head` calls from deploy scripts, replacing them with `dbwarden migrate`.
- The `alembic_version` table in your database is inert. Drop it when convenient, or leave it; DBWarden does not touch it.

If you prefer a transition period, both tools can coexist: they track state in separate tables. Generate the same change with both and compare the SQL until you are confident. Keep the overlap short, since two sources of truth defeats the point.

## Workflow changes to communicate to your team

- **Renames are explicit.** Where autogenerate guessed (usually as drop-and-create), DBWarden requires `--rename table.old:new` or `--rename-table old:new`. A rename never becomes silent data loss, but your team must know the flags exist.
- **Rollback is a contract.** Generated migrations include an executable `-- rollback` section, and placeholder rollbacks are refused. Irreversible changes must be declared with `-- dbwarden: irreversible`. See [Rollback Generation](../correctness/rollback-generation.md).
- **Table metadata moves into `class Meta`.** Comments, advanced indexes, and backend-specific options that previously lived in revision scripts or raw SQL are declared on the model. See [Modeling](modeling.md).
- **Destructive changes can be checked first.** `dbwarden check-impact` reports which code still references a column or table you are about to drop. Make it part of review for destructive migrations.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/getting-started/modeling/
========================================================================

# Modeling Guide

This guide walks through the process of defining SQLAlchemy models that DBWarden can read to generate migration SQL. For the complete reference of all supported Meta attributes, see [SQLAlchemy Models Reference](../models.md).

## How DBWarden Reads Models

DBWarden discovers models in the directories specified by `model_paths` in your `database_config(...)`. It reads two sources of metadata from each model:

1. **Column definitions**: typed SQLAlchemy `Mapped[...] = mapped_column(...)` fields, nullability, defaults, primary keys
2. **`class Meta` inner class**: backend-specific options like engine specs, partitioning, codecs

All backend-specific metadata uses the `class Meta` pattern. The `__table_args__` approach is not supported for PostgreSQL metadata. Using `mapped_column(info=...)` for backend-specific options raises `DBWardenConfigError`.

The `class Meta` convention is borrowed from Django's model metadata, so the pattern should read as familiar if you have used Django. DBWarden's version is typed and validated at import time: unknown attributes raise `DBWardenConfigError` when the module loads rather than producing incorrect DDL later.

## Common Meta Attributes

Every backend supports a core set of cross-database attributes. These work with any `database_type`.

### Table-Level

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases import TableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(TableMeta):
        comment = "Core user accounts"
        indexes = [
            {"name": "ix_users_email", "columns": ["email"]},
        ]
```

Available table-level attributes: `comment`, `indexes`, `checks`, `uniques`. See [Common Meta Attributes](../models.md#common-meta-attributes) for details.

### Column-Level

```python
class Meta(TableMeta):
    class internal_note:
        comment = "Internal system note"
        public = False
```

Available column-level attributes: `comment`, `public`. Fields named with a leading `_` are implicitly `public=False`.

For backend-specific column options, use `pg = pg.field(...)` for PostgreSQL. See [Column-Level Meta Base Class](../models.md#column-level-meta-base-class) for details.

## PostgreSQL Models

When `database_type="postgresql"`, use `class Meta(PGTableMeta)` for table-level metadata and `PGColumnMeta` inner classes for column-level metadata.

```python
from sqlalchemy import Integer, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    bio: Mapped[str] = mapped_column(Text)

    class Meta(PGTableMeta):
        pg_fillfactor = 80

        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", compression="pglz")
```

See the [reference](../models.md#postgresql-model-metadata) for the full list of `PGTableMeta` and `PGColumnMeta` attributes, or the [PostgreSQL Deep Dive](../databases/postgresql/index.md) for DDL behavior and snapshot format.

### PostgreSQL Views and Schemas

DBWarden supports PostgreSQL views and materialized views via `PGViewMeta`. Define a view as a model with `__tablename__` matching the view name:

```python
from dbwarden.databases.pgsql import PGViewMeta

class ActiveUser(Base):
    __tablename__ = "active_users"

    id: Mapped[int] = mapped_column(Integer)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(PGViewMeta):
        pg_view_query = "SELECT id, email FROM users WHERE active = true"
        pg_view_materialized = False
```

For materialized views, set `pg_view_materialized = True` and `pg_view_auto_refresh = True` to emit `REFRESH MATERIALIZED VIEW` automatically:

```python
class OrderSummary(Base):
    __tablename__ = "order_summary"

    user_id: Mapped[int] = mapped_column(Integer)
    total: Mapped[float] = mapped_column(Integer)

    class Meta(PGViewMeta):
        pg_view_query = "SELECT user_id, count(*) AS total FROM orders GROUP BY user_id"
        pg_view_materialized = True
        pg_view_auto_refresh = True
```

Scope tables or views to a PostgreSQL schema with `pg_schema`:

```python
class Meta(PGTableMeta):
    pg_schema = "app"

# DDL uses app.users instead of public.users
```

At the config level, set `pg_schema` in `database_config(...)` to set the connection `search_path`. See [Schema Support](../databases/postgresql/schemas.md#config-level-schema) for details.


## Using `generate-models` as a Starting Point

> **Note**: `generate-models` only works for databases with round trip support (PostgreSQL, SQLite, MySQL, ClickHouse). See [Round Trip Support](../databases/round-trip.md) for details.

The fastest way to get a correct model is to reverse-engineer it from your live database:

```bash
$ dbwarden generate-models -d primary --tables users,orders
```

DBWarden produces one `.py` file per table (or a single `models.py` with `--single-file`). The generated output includes `class Meta` with all detected backend-specific metadata.

Review the generated code before using it:

- Column types are mapped from database types to SQLAlchemy types. Verify the mapping is correct for your use case.
- Generated `class Meta` attributes are complete but may need adjustment (for example, you might want different index names or additional column hints).
- Partitioning, TTL, and engine settings are captured from the live database. If the database schema has drifted from what you intend, edit the model before running `make-migrations`.

## Auto-Generated Pydantic Schemas with `@auto_schema`

Use `@auto_schema` to generate four Pydantic schema classes on your model:

| Attribute | Contents |
|-----------|---------|
| `Model.Schema` | All mapped columns |
| `Model.CreateSchema` | Excludes server-defaulted columns (PKs with identity, `server_default`) |
| `Model.UpdateSchema` | All fields optional |
| `Model.PublicSchema` | Excludes fields where `public=False` or name starts with `_` |

Requires the `dbwarden-fastapi` plugin:

```bash
dbwarden plugin add dbwarden-fastapi
```

```python
from dbwarden_fastapi import auto_schema

@auto_schema
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    password_hash: Mapped[str] = mapped_column(String(255))

    class Meta:
        class email:
            comment = "Primary contact email"
            public = True

        class password_hash:
            public = False

# PublicSchema excludes password_hash and any _prefixed fields
public = User.PublicSchema(email="alice@example.com")
```

The decorator reads `class Meta` to infer `SchemaConfig`, then calls `schemap` to build the Pydantic models. Column `comment` values are injected into Pydantic field descriptions, and backend-specific metadata (`pg_*`, `my_*`, `ch_*`, `mdb_*`, `sq_*`) is included in `json_schema_extra.dbwarden_backend_meta`.

To customize schema generation, pass a `SchemaConfig` explicitly:

```python
from dbwarden_fastapi import auto_schema, SchemaConfig

@auto_schema(config=SchemaConfig(exclude_public=["internal_note"]))
class Order(Base):
    ...
```

`SchemaConfig` supports the following fields:

| Field | Type | Description |
|-------|------|-------------|
| `exclude_always` | `list[str]` | Excluded from all schemas |
| `exclude_create` | `list[str]` | Excluded from CreateSchema only |
| `exclude_update` | `list[str]` | Excluded from UpdateSchema only |
| `exclude_public` | `list[str]` | Excluded from PublicSchema only |
| `field_overrides` | `dict` | Override field types in generated schemas |
| `required_always` | `list[str]` | Fields that are always required |
| `optional_always` | `list[str]` | Fields that are always optional |

## When to Use Manual Migrations

Auto-generated migrations handle most cases, but some schema changes still need manual intervention via `dbwarden new`:

- PostgreSQL `USING` clause for type casts (e.g., casting `TEXT` to `INTEGER`). DBWarden emits `ALTER COLUMN ... TYPE` with a commented-out `-- USING col::newtype` line. Pass `--postgres-auto-using` to emit an active `USING` clause.
- Column renames not caught by the heuristic auto-detection. Use `--rename old_name:new_name` flags for deterministic renames, or rename in a manual migration.
- Data migrations (backfilling, transforming existing data). DBWarden emits a SQL comment placeholder.

For these cases run `dbwarden new` and write the SQL by hand, or use the relevant flag for auto-generation.

## Best Practices

- **One model class per table**: DBWarden discovers models by scanning directories. Each table should have exactly one model class.
- **Use `model_paths`**: always set `model_paths` explicitly in `database_config(...)`. Auto-discovery is available but explicit paths are more predictable.
- **Review generated migrations**: always read the `.sql` file before running `dbwarden migrate`.
- **Use `--dev` for local development**: configure a `dev_database_url` (SQLite works well) and use `dbwarden --dev` to iterate quickly without touching your real database.
- **Keep Meta classes minimal**: only set attributes that differ from the default. Default values are omitted from generated migrations, reducing noise.
- **Use `@auto_schema` for API projects** (requires `dbwarden-fastapi` plugin): generates Pydantic schemas from your model annotations. Fields with `public=False` or a leading `_` are excluded from `PublicSchema`.

See also: [Cookbook: Models & Migrations](../cookbook/02-models-and-migrations.md)

Next, continue with [Your First Migration](first-migration.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/getting-started/setup/
========================================================================

# Setup

This guide shows the initial project setup for DBWarden. By the end, you will have DBWarden installed, a project-local config file, and one verified database entry.

## Requirements

- Python 3.12 or higher
- A project that uses SQLAlchemy models, or plans to
- A supported backend: PostgreSQL, MySQL, MariaDB, SQLite, or ClickHouse

## Install DBWarden

Install the base package:

```bash
uv add dbwarden
```

Optional dependency groups:

| Group | Command | Use case |
|---|---|---|
| `fastapi` | `uv add "dbwarden[fastapi]"` | FastAPI session dependencies and runtime integration |
| `metrics` | `uv add "dbwarden[metrics]"` | Prometheus metrics |
| `sandbox` | `uv add "dbwarden[sandbox]"` | Sandbox migration testing |

You can combine them:

```bash
uv add "dbwarden[fastapi,metrics,sandbox]"
```

## Initialize the Project

```text
$ dbwarden init
Initialized DBWarden project structure
Created migrations directory
Created dbwarden.py
```

`init` creates the local migration layout and a config scaffold. It is safe to run again, DBWarden will not destroy existing `database_config(...)` definitions.

## Create the Configuration File

By default, `dbwarden init` creates `dbwarden.py`, and that is the simplest place to start. However, `database_config(...)` can live in any discovered Python file inside your project.

The simplest `dbwarden.py` looks like this:

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

Copy this into `dbwarden.py`, or another discovered Python module in your project, then adjust the URLs, `model_paths`, and `model_tables` for your project.

## Step by Step

### Step 1: Import `database_config`

```python
from dbwarden import database_config
```

`database_config(...)` is the entry point for defining databases. Every configured database becomes part of the validated runtime config.

### Step 2: Define `database_name`

```python
database_name="primary"
```

This is the stable name you will use in CLI commands such as:

```bash
$ dbwarden status --database primary
```

### Step 3: Mark the default database

```python
default=True
```

Exactly one configured database must be the default. Commands without `--database` use that entry.

### Step 4: Set the backend type

```python
database_type="postgresql"
```

This controls backend-specific SQL generation, schema inspection, and metadata behavior.

Supported values:

- `postgresql`
- `mysql`
- `mariadb`
- `sqlite`
- `clickhouse`

### Step 5: Set the runtime URL

```python
database_url_sync="postgresql://user:password@localhost:5432/main"
```

This is the database URL used by CLI commands such as `make-migrations`, `migrate`, `status`, and `check`.

If your application also uses async SQLAlchemy sessions, you can define an async URL too:

```python
database_url_async="postgresql+asyncpg://user:password@localhost:5432/main"
```

DBWarden keeps sync and async URLs separate so the CLI and FastAPI runtime can share one config source without forcing a single driver choice.

### Step 6: Point to your models

```python
model_paths=["app.models"]
```

This tells DBWarden where to discover SQLAlchemy models. In multi-database projects, explicit `model_paths` are required.

### Step 7: Filter tables for this database

```python
model_tables=["users", "posts", "comments"]
```

Optional. When set, DBWarden only includes the listed tables from the discovered models. All other discovered tables are ignored.

This is useful when multiple databases share the same `model_paths` but own different subsets of tables:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://default:@localhost:8123/analytics",
    model_paths=["app.models"],
    model_tables=["events", "page_views"],
)

logs = database_config(
    database_name="logs",
    database_type="mysql",
    database_url_sync="mysql+pymysql://user:password@localhost:3306/logs",
    model_paths=["app.models"],
    model_tables=["audit_log", "error_log"],
)
```

If `model_tables` is not set, all discovered tables in `model_paths` belong to that database.

### Step 8: Configure a dev database

```python
dev_database_type="sqlite"
dev_database_url="sqlite:///./development.db"
```

This is optional, but recommended. It lets you run commands locally with `--dev`, without touching your main database.

## Verify the Configuration

Run:

```text
$ dbwarden settings show --all
Database: primary
Type: postgresql
Default: true
Sync URL: postgresql://user:password@localhost:5432/main
Model paths: ['app.models']
Model tables: ['users', 'posts', 'comments']
Dev database type: sqlite
Dev database URL: sqlite:///./development.db
```

If this command works, DBWarden can resolve and validate your config.

## Common Problems

### `No configuration found`

DBWarden could not locate a config source. Make sure your project contains at least one discovered file with a `database_config(...)` call. `dbwarden.py` is the default convention, but it does not have to be the only location.

### `Exactly one default=True required`

If you configure more than one database, only one can be the default.

### `model_paths is required when more than one database is configured`

In multi-database setups, each database must declare the models that belong to it.

Next, continue with [Modeling](modeling.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/getting-started/workflows/
========================================================================

# Workflows

This guide covers larger day-to-day workflows once the basics are in place.

## Multi-Database Projects

DBWarden can manage more than one database from one config source.

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://default:@localhost:8123/analytics",
    model_paths=["app.analytics_models"],
    model_tables=["events", "page_views"],
)
```

Apply migrations per database:

```bash
$ dbwarden migrate --database primary
$ dbwarden migrate --database analytics
```

Show status across all configured databases:

```bash
$ dbwarden status --all
```

## Separate Model Sets

Each database should usually own a distinct model set through `model_paths`. When databases share the same models package, use `model_tables` to split ownership by table name. DBWarden validates overlapping paths unless `overlap_models=True` is set explicitly.

This prevents one model tree from being interpreted as belonging to multiple databases by accident.

## CI Workflows

A common CI pattern is:

```bash
$ dbwarden export-models --database primary
$ dbwarden make-migrations "ci validation" --offline --database primary
$ dbwarden check --database primary
```

This keeps schema generation deterministic and avoids depending on a live database in every pipeline step.

For a full example, see [Cookbook: Offline & CI](../cookbook/04-offline-ci.md).

## Sandbox Validation

Before applying migrations to a real environment, you can validate them in a temporary sandbox database.

```bash
$ dbwarden migrate --sandbox --database primary
```

This is especially useful for complex migrations, risky type changes, and CI gates.

See the [Architecture Deep Dive](../architecture-deep-dive.md) for a thorough explanation of sandbox validation.

## Baselines and Partial Applies

When integrating DBWarden into an existing environment, or when applying only part of a migration sequence, these patterns are common:

- `--baseline` marks the target migration as already applied without actually running it, useful for onboarding an existing database.
- `--partial` (via `--count` or `--to-version`) applies a subset of pending migrations instead of all of them.

```bash
$ dbwarden migrate --database primary --baseline --to-version 0005
$ dbwarden migrate --database primary --count 2
$ dbwarden rollback --database primary --to-version 0007
```

See the [CLI Reference](../cli-reference.md) for a full breakdown of these flags. Use these modes carefully. They are operational tools, not everyday authoring commands.

## Operational Command Pattern

A typical production-safe pattern is:

```bash
$ dbwarden check --database primary
$ dbwarden make-migrations "release change" --database primary
$ dbwarden migrate --database primary
$ dbwarden status --database primary
$ dbwarden history --database primary
```

This keeps planning, execution, and verification as separate visible steps.

## Rollback Command Pattern

When validating rollback quality, use a loop like this:

```bash
$ dbwarden migrate --database primary
$ dbwarden rollback --count 1 --database primary
$ dbwarden migrate --database primary
```

This verifies both directions of the migration before a release depends on them.

## Where to Go Next

- Use [Cookbook Overview](../cookbook/index.md) for full working flows
- Use [Configuration](../configuration/index.md) for deeper config behavior
- Use [CLI Reference](../cli-reference.md) for command details

========================================================================
PAGE: https://dbwarden.emiliano-go.com/glossary/
========================================================================

# Glossary

## A

**Auto Schema**
: A feature that generates Pydantic schemas from SQLAlchemy model annotations using `@auto_schema` (from the `dbwarden-fastapi` plugin), eliminating duplication between ORM and API layers in FastAPI applications.

## B

**Backend**
: A supported database type (PostgreSQL, MySQL, MariaDB, SQLite, ClickHouse). Each backend has specific DDL syntax, feature support, and round-trip capability.

**Baseline**
: A migration version used as a starting point, marking the point up to which existing migrations are considered already applied without executing the upgrade SQL.

## C

**Checksum**
: A SHA-256 hash stored when a migration file is applied. On subsequent runs, the checksum is recalculated to detect file tampering or accidental edits.

**Check (command)**
: Analyzes schema differences between SQLAlchemy models and a live database, classifying every operation by danger level.

**Code Seed**
: A seed defined as a Python class extending `Seed`, with `run()` and optional `rollback()` methods. The recommended way to manage seed data.

**Configuration (dbwarden.py)**
: DBWarden uses a Python file (`dbwarden.py`) with `database_config()` calls, providing type safety, runtime flexibility, and IDE support for configuring databases.

## D

**Database Config**
: A single `database_config()` call that defines one database. Each config includes the database name, type, connection URL, model paths, and optional dev mode settings.

**Dev Mode**
: Using a different database type (typically SQLite) for local development while targeting a production database (e.g., PostgreSQL) in deployment, enabled via the `dev_database_type` and `dev_database_url` config options.

**Diff**
: A read-only command showing structural differences between SQLAlchemy models and a live database, with table, json, and sql output formats.

**Downgrade**
: Revert applied migrations to reach a specific target version by reading `-- rollback` sections and applying them in reverse order.

## F

**FastAPI Integration**
: DBWarden's built-in support for FastAPI including async sessions, health endpoints, migration endpoints, Prometheus metrics, and distributed locking. Configured once via `database_config()`.

## G

**generate-models**
: A command that reverse-engineers SQLAlchemy model code from a live database, supporting all backends.

## H

**Health Endpoints**
: Production-ready HTTP endpoints for database connectivity checks, migration state monitoring, and Kubernetes liveness/readiness probes.

## I

**Impact Analysis**
: The ability to find affected Python code references (function calls, class references, variable names) before deploying a migration, via the `check-impact` command.

**Index Spec**
: A class (`IndexSpec`) used in model `Meta` to define database indexes, including columns, uniqueness, and types.

## L

**Lock (Migration Lock)**
: A database-level lock that prevents concurrent schema mutations across multiple application instances or CLI invocations.

## M

**make-migrations**
: The command that generates SQL migration files by comparing current SQLAlchemy model definitions against either a live database or stored schema snapshots.

**Manual Migration**
: A migration file created by hand (via `dbwarden new`) rather than auto-generated. Useful for data migrations, stored procedures, or any DDL outside model diffs.

**Meta (class Meta)**
: An inner class on SQLAlchemy models that provides DBWarden with backend-specific metadata like table comments, indexes, partitioning, engine options, and more.

**Migration File**
: A plain SQL file with `-- upgrade` and `-- rollback` sections. Each file represents one atomic schema change.

**Multi-Database**
: Managing multiple database configurations (e.g., primary + analytics, or microservice per database) from a single `dbwarden.py` config file.

## O

**Observability**
: DBWarden's monitoring capabilities including Prometheus metrics (migration counters, schema version gauges, connection pool health) and structured JSON logging.

**Offline Mode**
: Generating migrations using stored JSON schema snapshots instead of connecting to a live database, enabling CI/CD pipelines without database access.

## P

**Pydantic Schema (auto-generated)**
: Request/response schemas automatically generated from SQLAlchemy model annotations using `@auto_schema` (from the `dbwarden-fastapi` plugin), keeping API contracts in sync with database models.

## R

**Rename Detection**
: DBWarden can detect column and table renames by comparing schema snapshots, generating `ALTER TABLE ... RENAME` instead of `DROP` + `ADD`.

**Rollback**
: The `-- rollback` section of a migration file containing SQL to undo the upgrade. DBWarden enforces that every migration has a corresponding rollback.

**Round-Trip**
: A backend that supports both reading schema (via `generate-models`) and writing schema (via `make-migrations`/`migrate`). Verified when reverse-engineering a database and re-generating produces zero diff.

**RA (Runs-Always) Migration**
: A migration type that runs every time `migrate` is executed, regardless of previous application. Useful for idempotent operations like views or functions.

**ROC (Runs-on-Change) Migration**
: A migration type that runs only when its content has changed (detected via checksum). Useful for stored procedures that evolve over time.

## S

**Safety Check**
: A feature that classifies every migration operation into danger levels (safe, caution, danger, manual) so teams can review high-risk changes before production.

**Sandbox**
: An isolated test environment using Testcontainers to validate migrations against a real database before applying them to production or staging.

**Schema Snapshot**
: A JSON file recording the full DDL state of a database at the point a migration was applied. Enables offline migration generation, rename detection, and CI workflows.

**Seed**
: Data population mechanism using either Python code seeds (recommended) or file-based SQL/Python seeds. Seeds are tracked and versioned like migrations.

**SQL-First**
: A design philosophy where all schema changes are expressed as explicit SQL files that can be reviewed, tested, and rolled back, rather than being abstracted away by an ORM.

**SQL Translation**
: The ability to generate SQL for one dialect (e.g., PostgreSQL) while operating against a different development database (e.g., SQLite), enabling local development without full infrastructure.

## T

**TableMeta**
: The base class for model `Meta` inner classes, providing type-safe configuration of table-level metadata like comments, indexes, partitioning, and engine options.

## U

**Unlock**
: The `dbwarden unlock` command to recover from a stale migration lock when no migration is actually running and the lock was not released properly.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/
========================================================================

<p align="center">
  <img src="https://raw.githubusercontent.com/dbwarden-org/dbwarden/refs/heads/main/assets/icon.png" alt="DBWarden" width="128"/>
</p>
<p align="center">
  <strong style="font-size: 2.5em;">DBWarden</strong>
</p>
<p align="center">
  <em>Your SQLAlchemy models are your migrations.</em>
</p>
<p align="center">
  <a href="https://www.python.org/downloads/">
    <img src="https://img.shields.io/badge/Python-3.12.7%2B-3776AB?logo=python&logoColor=white&style=for-the-badge" alt="Python">
  </a>
  <a href="https://pypi.org/project/dbwarden/">
    <img src="https://img.shields.io/pypi/v/dbwarden?logo=pypi&logoColor=white&style=for-the-badge" alt="PyPI">
  </a>
  <a href="https://github.com/dbwarden-org/dbwarden/blob/main/LICENSE">
    <img src="https://img.shields.io/badge/License-MIT-10AC84?style=for-the-badge" alt="License">
  </a>
  <a href="https://deepwiki.com/dbwarden-org/dbwarden/">
    <img src="https://img.shields.io/badge/DeepWiki-8A2BE2?logo=readthedocs&logoColor=white&style=for-the-badge" alt="DeepWiki">
  </a>
  <a href="https://codecov.io/gh/dbwarden-org/dbwarden">
    <img src="https://img.shields.io/codecov/c/github/dbwarden-org/dbwarden?logo=codecov&logoColor=white&style=for-the-badge" alt="Codecov">
  </a>
</p>

<p align="center">
  <strong><a href="https://dbwarden.emiliano-go.com/">Full documentation</a></strong>
  &nbsp;|&nbsp;
  <strong><a href="https://github.com/dbwarden-org/dbwarden">Source Code</a></strong>
</p>

---

DBWarden is a declarative database migration and schema management tool for SQLAlchemy. You declare the schema you want in your SQLAlchemy models, and DBWarden derives everything else: migration SQL, rollbacks, snapshots, and safety checks.

There are no migration scripts to write or maintain. There is no migration runtime. Your models are the contract. The database is kept in sync with them.

## At a glance
- Generates migration files as plain SQL, with `-- upgrade` and `-- rollback` sections
- Reads SQLAlchemy models and backend-specific metadata from `class Meta`
- Supports PostgreSQL, MySQL, MariaDB, SQLite, and ClickHouse
- Uses a registry driven PostgreSQL pipeline for diffs and SQL emission
- Manages one or many databases from one typed config source
- Adds safety tooling, schema diffing, and status commands

- Migrations generated from your models, not written by hand
- Plain SQL output: reviewable, committable, executable anywhere
- Rollback contract with executable rollback, strict placeholder refusal, and explicit irreversible declarations
- Pre-deploy impact analysis: know what breaks before it ships
- Offline migration generation for CI pipelines without a live database
- Schema snapshots for deterministic diffs and rename detection
- Typed `class Meta` system with import-time validation
- Multi-database support: PostgreSQL, MySQL, ClickHouse, MariaDB, SQLite
- Extensible plugin system with official plugins for seeds, RBAC, FastAPI, sandbox testing, and PostgreSQL/ClickHouse extensions
- Reverse-engineer live databases into models with `generate-models`

## Why DBWarden

Schema management tools fall into two camps. Imperative tools have you author *changes*: revision scripts that describe how to get from one schema version to the next. Declarative tools have you author the *desired state* and derive the changes for you. DBWarden is declarative: your SQLAlchemy models are the single definition of what the schema should be.

Most imperative tools ask you to maintain two representations of your schema: your ORM models and your migration files. When they drift, you find out at deploy time.

DBWarden eliminates the second representation. Your SQLAlchemy models are the schema definition. DBWarden reads them, diffs them against the current database state, and generates the SQL to close the gap (including rollback) without you writing a line of migration code.

This also means:

- No migration runtime to install or version
- No generated Python scripts that quietly do the wrong thing
- No schema drift discovered in production: drift is caught at `make-migrations` time
- Migrations that can be generated in CI without a database connection

DBWarden is not a wrapper around Alembic. It is a different approach to the same problem. Alembic asks you to describe *how* to change the database; DBWarden asks you to describe *what* the schema should be. Alembic can autogenerate revisions, but each one becomes an imperative Python artifact you own, edit, and chain — the revision history is the source of truth. With DBWarden the models stay the source of truth, and the output is plain SQL.

Unlike tools that apply declarative diffs directly to the database, DBWarden still produces versioned, reviewable migration files with explicit rollbacks: declarative authoring without giving up auditable deploy artifacts.

## From zero to production

Typical adoption path in an existing project:

1. Point DBWarden at your existing SQLAlchemy models
2. Run initial `make-migrations` to generate a baseline schema
3. Commit generated migrations as your source of truth
4. Replace your current migration workflow with the DBWarden CLI
5. Optionally enable:
   - Migration impact analysis for safer deploys
   - Offline mode for CI pipelines without a database service

## Installation

```bash
uv add dbwarden
```

Requirements: Python 3.12+, SQLAlchemy 2.0+.

Optional dependency groups:

| Group        | Default | Provides                             |
|--------------|---------|--------------------------------------|
| `[postgres]` | Yes     | `psycopg2-binary`                    |
| `[mysql]`    |         | `pymysql`                            |
| `[clickhouse]` |       | `clickhouse-connect`, `aiohttp`      |
| `[dev]`      |         | `pytest`, `zensical`, `seoslug`, `httpx2` |

## Quick start

### 1. Configure

Create a file named `dbwarden.py` in your project root:

```python
from dbwarden import database_config

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/myapp",
    database_url_async="postgresql+asyncpg://user:pass@localhost:5432/myapp",
)
```

### 2. Define your models

```python
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases import IndexSpec, TableMeta


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    bio: Mapped[str | None] = mapped_column(Text, nullable=True)

    class Meta(TableMeta):
        comment = "Core user accounts"


class Post(Base):
    __tablename__ = "posts"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String(255), nullable=False)
    body: Mapped[str] = mapped_column(Text, nullable=False)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False)

    class Meta(TableMeta):
        indexes = [
            IndexSpec(name="ix_posts_created_at", columns=["created_at"]),
        ]
```

### 3. Generate a migration

```bash
dbwarden init
dbwarden make-migrations "create initial tables"
```

Output: both upgrade and rollback in the same file.

```sql
-- upgrade
CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    bio TEXT
);
COMMENT ON TABLE users IS 'Core user accounts';

CREATE TABLE IF NOT EXISTS posts (
    id INTEGER PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    user_id INTEGER NOT NULL REFERENCES users(id),
    created_at TIMESTAMP NOT NULL
);
CREATE INDEX IF NOT EXISTS ix_posts_created_at ON posts (created_at);

-- rollback
DROP TABLE posts;
DROP TABLE users;
```

### 4. Apply

```bash
dbwarden migrate
```

### 5. Check status

```bash
dbwarden status
```

## Typical workflow

1. Define or update your SQLAlchemy models with `class Meta` annotations
2. Run `dbwarden make-migrations` to generate SQL
3. Review the generated `.sql` file and its rollback section
4. Run `dbwarden migrate` to apply
5. Verify with `dbwarden status`

---

## Migration engine

**Model-driven generation**: DBWarden reads your SQLAlchemy models directly. When you change a model, it diffs the new state against the last snapshot and generates the SQL to reconcile them.

**Plain SQL output**: Generated migrations are `.sql` files. No migration runtime, no generated Python. Review them, commit them, execute them directly against any environment.

**Rollback contract**: Generated migrations carry both upgrade and rollback sections. DBWarden emits executable rollback when it is safe, refuses placeholder rollback by default, and requires an explicit irreversible declaration when rollback cannot be produced.

**Schema snapshots**: After every migration, a checksummed JSON snapshot is written to `.dbwarden/schemas/`. Snapshots power rename detection, offline diffing, and column-level comparisons without querying the live database.

**Column-level diffing**: Type, nullability, default, and comment changes generate precise `ALTER COLUMN` statements.

**Typed `class Meta`**: The `_MetaValidator` metaclass validates every attribute on `class Meta` at import time. Typos that would have silently produced wrong DDL now raise `DBWardenConfigError` immediately.

```python
class Meta(MyTableMeta):
    my_engin = "InnoDB"  # DBWardenConfigError: unknown attr 'my_engin'
```

Supported index features:

- Partial indexes (`WHERE` clause)
- Covering indexes (`INCLUDE` columns)
- `USING` access methods
- `NULLS NOT DISTINCT` (PostgreSQL 15+)
- Per-column sort order
- Storage parameters (`WITH (fillfactor=...)`)
- ClickHouse skip indexes via `ChIndexSpec`

---

## Pre-deploy impact analysis

Before applying schema changes, DBWarden can scan your codebase to identify what will be affected. It uses AST analysis with a grep fallback, so results reflect actual code structure rather than text matches.

```bash
dbwarden check-impact 0042 --database primary
```

Output:

```
drop_column on users.username
  References: 2
    app/routes/users.py:34  attribute_access
      .username
    app/templates/profile.jinja2:12  grep
      user.username
```

Run this before any destructive deploy to surface breaking changes before they reach production.

---

## Offline migrations

Export model state once, then generate migrations on any machine without a database connection. Designed for CI pipelines and local development without a running database.

```bash
dbwarden export-models --database primary
git add .dbwarden/model_state.json
```

Then on any machine, with no database required:

```bash
dbwarden make-migrations "add bio column" --offline
```

The model state file is updated in place after each migration.

> **Important:** The model state file (`.dbwarden/model_state.*.json`) is used for offline migration generation. It is auto-generated and committed to version control. If accidentally deleted, restore it from git (`git checkout .dbwarden/model_state.*.json`) or regenerate it by running `dbwarden export-models --database <db>` against a live database. Without it, offline commands like `make-migrations --offline` will not work, but online operations are unaffected.

---

## Reverse-engineer models

Generate SQLAlchemy models from a live database with round-trip support (PostgreSQL, MySQL, ClickHouse, SQLite):

```bash
dbwarden generate-models --database primary --tables users,posts
dbwarden generate-models --database primary --base app.database:Base
```

By default each generated file declares its own `Base = declarative_base()`. Use `--base` to import a custom Base class from your project instead (e.g. `--base app.database:Base` or `--base app.database:DeclarativeBase`). The generated output includes `class Meta` blocks with all detected backend-specific metadata.

---

## Supported databases

| Database   | Round-trip | Notes                                       |
|------------|------------|---------------------------------------------|
| PostgreSQL | Full       | Primary backend, full schema fidelity       |
| MySQL      | Full       | DDL parity focus                            |
| ClickHouse | Full       | Analytics backend, MergeTree engine family  |
| SQLite     | Dev only   | Local development and SQL translation       |
| MariaDB    | No         | Schema layer complete; snapshot gaps remain |

### PostgreSQL

First-class support with full round-trip schema fidelity. Supported features include identity and generated columns, partitioning, table inheritance, exclusion constraints, deferrable constraints, advanced indexes via `PgIndexSpec`, per-column storage and collation, enum type creation, and full type normalization (SERIAL, TIMESTAMPTZ, NUMERIC, JSONB, UUID, ARRAY, TSTZRANGE).

### MySQL

Full round-trip support with `MyTableMeta` / `MyColumnMeta` and `my.field()` spec objects. Engine-level options (`my_engine`, `my_charset`, `my_collate`, `my_row_format`), column-level options (`unsigned`, `charset`, `collate`, `on_update`), and model reverse-engineering via `generate-models`.

```bash
uv add "dbwarden[mysql]"
```

### ClickHouse

First-class analytics backend support. MergeTree engine family via `ChEngineSpec`, replicated engines, projections, dictionaries, materialized views, skip indexes via `ChIndexSpec`, column codecs, `LowCardinality` and `Nullable` type wrappers.

```bash
uv add "dbwarden[clickhouse]"
```

### MariaDB

Schema layer is complete with `MdbTableMeta` / `MdbColumnMeta` and `mdb.field()` spec objects including MariaDB-specific features (`page_compressed`, `invisible`, `without_overlaps`). Snapshot capture and reverse-engineering of MariaDB-specific features are not yet complete.

---

## Developer experience

**Dev mode**: Run SQLite locally against a PostgreSQL production schema with automatic SQL translation.

**Multi-database**: One project, multiple databases, full isolation between them. Use `model_tables` to assign table ownership per database when sharing model paths.

**Generate models**: Reverse-engineer a live database (PostgreSQL, MySQL, ClickHouse) into SQLAlchemy models with `dbwarden generate-models`.

**`dbwarden diff`**: Read-only comparison tool. Outputs as Rich table, JSON, or raw SQL. Supports `--offline` mode.

**Graceful disconnection**: Automatic retry logic and clear error messages when a database is unreachable.

---

## Official plugins

DBWarden features a plugin system with three trust tiers (official, verified, community). Official plugins extend core with features that were previously built-in, now maintained independently:

| Plugin | PyPI | Purpose |
|---|---|---|
| `dbwarden-ch-rbac` | [`dbwarden-ch-rbac`](https://pypi.org/p/dbwarden-ch-rbac) | ClickHouse RBAC: roles, users, grants, row policies, quotas, settings profiles |
| `dbwarden-fastapi` | [`dbwarden-fastapi`](https://pypi.org/p/dbwarden-fastapi) | FastAPI session dependencies, health endpoints, migration routes |
| `dbwarden-pgsql-extensions` | [`dbwarden-pgsql-extensions`](https://pypi.org/p/dbwarden-pgsql-extensions) | PostgreSQL extensions, event triggers, functions, triggers, storage parameters |
| `dbwarden-pgsql-rbac` | [`dbwarden-pgsql-rbac`](https://pypi.org/p/dbwarden-pgsql-rbac) | PostgreSQL RBAC: roles, grants, default privileges, policies |
| `dbwarden-pgsql-types` | [`dbwarden-pgsql-types`](https://pypi.org/p/dbwarden-pgsql-types) | PostgreSQL custom types: ENUMs, domains, composite types, sequences |
| `dbwarden-sandbox` | [`dbwarden-sandbox`](https://pypi.org/p/dbwarden-sandbox) | Testcontainers sandbox providers for safe migration replay |
| `dbwarden-seeds` | [`dbwarden-seeds`](https://pypi.org/p/dbwarden-seeds) | Seed data management with code seeds and file-based SQL/Python seeds |

See the [plugin documentation](plugins/) for installation, development guides, and the full Verified standard.

---

## License

MIT

---

DBWarden is built for teams that want declarative, reviewable, reproducible database changes, derived from the models they already maintain, not from migration scripts they have to write.

## Next Steps

- Start with [Features](features.md)
- Follow the guides in [Get Started](getting-started/setup.md)
- Explore [Cookbook & Examples](cookbook/index.md)
- Browse [Plugins](plugins/) to extend DBWarden's capabilities
- Use [CLI Reference](cli-reference.md) as command lookup

========================================================================
PAGE: https://dbwarden.emiliano-go.com/installation/
========================================================================

# Installation

This guide covers installing DBWarden in your project and verifying it works correctly.

## Requirements

- Python 3.10 or higher
- A project that uses SQLAlchemy for database models
- uv or another Python package manager

## Install using uv

```bash
uv add dbwarden
```

### Development dependencies

To also install testing and linting tools:

```bash
uv add "dbwarden[dev]"
```

### Optional dependency groups

The `[postgres]` extra is the most commonly used. Install it if you are targeting PostgreSQL.

| Group                  | Command | Provides |
|------------------------|-------|-------|
| `postgres`             | `uv add "dbwarden[postgres]"` | PostgreSQL driver (`psycopg2-binary`) |
| `mysql`                | `uv add "dbwarden[mysql]"` | MySQL/MariaDB driver (`pymysql`) |
| `clickhouse`           | `uv add "dbwarden[clickhouse]"` | ClickHouse driver (`clickhouse-connect`) |
| `fastapi`              | `uv add "dbwarden[fastapi]"` | FastAPI session dependencies, health router, migration router, metrics router, Redis lock |
| `metrics`              | `uv add "dbwarden[metrics]"` | Prometheus metrics endpoint (`prometheus-client`) |
| `sandbox`              | `uv add "dbwarden[sandbox]"` | Sandbox migration testing via testcontainers |

Combine groups as needed:

```bash
uv add "dbwarden[postgres,mysql,fastapi]"
```

## Database drivers

DBWarden uses SQLAlchemy under the hood. The recommended way to install drivers is via the extras above:

```bash
# PostgreSQL
uv add "dbwarden[postgres]"

# MySQL/MariaDB
uv add "dbwarden[mysql]"

# ClickHouse
uv add "dbwarden[clickhouse]"

# SQLite comes bundled with Python
```

You can also install drivers directly if you prefer:

```bash
uv add psycopg2-binary    # PostgreSQL
uv add pymysql            # MySQL / MariaDB
uv add clickhouse-connect # ClickHouse
```

## Verify installation

After installing, confirm DBWarden is available:

```bash
$ dbwarden version
```

You should see output:

```
0.9.4
```

## Initialize in your project

Create the DBWarden structure in your project directory:

```bash
$ dbwarden init
```

This creates:

- a `migrations/` directory structure
- a `dbwarden.py` config scaffold (or discovers your existing config source)

## What happens during init

When you run `init`, DBWarden:

1. Creates `migrations/` if missing
2. Creates `dbwarden.py` (or updates existing config source) with import scaffolding
3. Does not overwrite existing `database_config(...)` definitions you have added

You can run `init` safely on an existing project - it is idempotent.

## Quick configuration

After init, configure your first database by editing `dbwarden.py` (or your existing config file that contains `database_config(...)` calls):

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/mydb",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

The `dev_database_*` fields are optional but recommended - they enable fast local iterations with `--dev`.

## Verify configuration loads

```bash
$ dbwarden settings show --all
```

You should see your database entry printed with type and URL.

## Common installation issues

**Command not found after uv add**

- Ensure your virtual environment is activated
- Try removing and re-adding: `uv remove dbwarden && uv add dbwarden`

**Import errors or missing module warnings**

- Upgrade uv and reinstall: `uv add --upgrade dbwarden`

**Database driver errors**

- Install the appropriate driver for your target database (see Database drivers section above)

## Upgrading

To update to a newer version:

```bash
uv add --upgrade dbwarden
```

Or with poetry:

```bash
poetry update dbwarden
```

Check the release notes when upgrading major versions - there may be configuration or workflow changes.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/migration-files/
========================================================================

# Migration File Format

Migration files are the execution contract in DBWarden.

Everything that changes your database should be represented in explicit SQL files that can be reviewed, tested, and rolled back.

## File naming and location

Versioned migrations are stored under each database migrations directory (default: `migrations/<database_name>`).

Canonical filename pattern:

```text
{database_name}__{version}_{description}.sql
```

A legacy format without the `{database_name}__` prefix (e.g. `0001_create_users_table.sql`) is also accepted for backward compatibility.

Examples:

```text
primary__0001_initial_schema.sql
primary__0002_add_users_table.sql
analytics__0001_create_events.sql
```

When a migration is auto-generated with `make-migrations`, DBWarden also writes a companion plan file:

```text
primary__0001_initial_schema.plan.json
```

That file captures machine-readable metadata for CI and debugging and is not executed by `migrate`.

## Required sections

Each migration file must define both:

```sql
-- upgrade

-- rollback
```

- `-- upgrade`: statements applied during `migrate`
- `-- rollback`: statements applied during `rollback`

If rollback is weak or incomplete, production recovery is weak or incomplete.

## Rollback contract

Generated rollback is part of the migration contract. DBWarden classifies rollback before a migration is accepted:

| Kind | Meaning | Generation behavior |
|------|---------|---------------------|
| `real` | Executable rollback SQL restores the prior schema state for the operation. | Allowed. |
| `conditional` | Executable rollback SQL is allowed only when prior state is captured and DBWarden can prove the inverse is structurally safe. | Allowed with verbose warning. |
| `irreversible` | Rollback is genuinely unsafe or impossible, such as lossy ClickHouse engine changes or PostgreSQL enum value additions. | Allowed only for known irreversible operations or explicit acknowledgement. |
| `placeholder` | Rollback is only a comment or manual instruction. | Refused by default. |

Placeholder rollback is not a working rollback. If DBWarden cannot emit executable rollback SQL, generation fails unless the migration is intentionally declared irreversible.

Explicit irreversible declaration:

```sql
-- dbwarden: irreversible
```

Use this only when the team accepts that the migration cannot be rolled back automatically. See [Rollback Coverage](correctness/rollback-coverage-matrix.md) for the backend operation matrix.

## Migration classes

DBWarden supports three execution classes:

| Prefix | Class | Behavior |
|--------|-------|----------|
| `NNNN_` | Versioned | Runs once in ordered version sequence |
| `RA__` | Runs always | Runs on every `migrate` execution |
| `ROC__` | Runs on change | Runs when checksum changed |

### When to use each

- `NNNN_`: schema evolution (tables, columns, indexes, constraints)
- `RA__`: objects that should always be refreshed (views, grants)
- `ROC__`: routines/policies that should apply only when content changes

## Execution model

At runtime, DBWarden builds an execution plan from file discovery + migration metadata:

1. read versioned files and filter already-applied versions
2. include `RA__` files
3. include changed `ROC__` files
4. execute with lock protection
5. record metadata and checksums

Conceptual plan:

```python
def build_plan(directory, applied_versions):
    versioned = parse_versioned_files(directory)
    repeatable = parse_repeatable_files(directory)
    pending_versioned = [m for m in versioned if m.version not in applied_versions]
    pending_ra = repeatable.runs_always
    pending_roc = changed_only(repeatable.runs_on_change)
    return pending_versioned + pending_ra + pending_roc
```

## Examples

### Versioned migration

```sql
-- upgrade

CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    created_at DATETIME
);

-- rollback

DROP TABLE users;
```

### Runs-always migration (`RA__`)

Filename example: `primary__RA__refresh_active_users_view.sql`

```sql
-- upgrade

CREATE OR REPLACE VIEW active_users AS
SELECT id, email FROM users WHERE is_active = TRUE;

-- rollback

DROP VIEW IF EXISTS active_users;
```

### Runs-on-change migration (`ROC__`)

Filename example: `primary__ROC__update_timestamp_trigger.sql`

```sql
-- upgrade

CREATE OR REPLACE FUNCTION update_updated_at()
RETURNS TRIGGER AS $$
BEGIN
  NEW.updated_at = NOW();
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- rollback

DROP FUNCTION IF EXISTS update_updated_at();
```

## Metadata headers

Headers are parsed from migration file comments. The `-- seed` marker is recognised
by tools; `-- depends_on` parsing is implemented but **not yet enforced** during
migration execution (migrations run in filesystem-sort order by version).

Dependency header (parsed but not enforced):

```sql
-- depends_on: ["0004", "0005"]
```

Seed marker:

```sql
-- seed
```

## Authoring guidelines

- One logical change per migration file
- Keep DDL explicit; avoid hidden application-side schema effects
- Keep rollback idempotent when possible (`IF EXISTS`, safe predicates)
- Do not commit placeholder rollback. Use executable rollback SQL or an explicit irreversible declaration.
- For data migrations, use bounded, reversible operations
- Prefer small migrations over large monolithic SQL scripts

## Review checklist

Before merge:

- upgrade section matches intended schema change
- rollback section restores prior valid state
- indexes/constraints/defaults are explicit
- no environment-specific literals accidentally committed

Before release:

```bash
$ dbwarden status --database primary
$ dbwarden migrate --database primary
$ dbwarden rollback --database primary --count 1
$ dbwarden migrate --database primary
```

========================================================================
PAGE: https://dbwarden.emiliano-go.com/models/
========================================================================

# SQLAlchemy Models Reference

This page is the **reference** for all supported Meta attributes across every backend. For a step-by-step walkthrough of defining models, see the [Modeling Guide](getting-started/modeling.md).

DBWarden reads SQLAlchemy model metadata to generate migration SQL. Use `model_paths` in your `database_config(...)` entries to control discovery.

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/main",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
)
```

## Common Meta Attributes

Every backend supports a core set of cross-database attributes via `class Meta(TableMeta)`:

### Table-level

| Attribute | Type | SQL | Backends |
|-----------|------|-----|----------|
| `comment` | `str` | `COMMENT ON TABLE t IS '...'` | All |
| `indexes` | `list[IndexSpec]` | `CREATE INDEX ...` | All |
| `checks` | `list[CheckSpec]` | `ALTER TABLE t ADD CONSTRAINT ... CHECK (...)` | All |
| `uniques` | `list[UniqueSpec]` | `ALTER TABLE t ADD CONSTRAINT ... UNIQUE (...)` | All |

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases import TableMeta, IndexSpec, CheckSpec, UniqueSpec

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    age: Mapped[int] = mapped_column(Integer)

    class Meta(TableMeta):
        comment = "Core user accounts"
        indexes = [
            IndexSpec(name="ix_users_email", columns=["email"]),
        ]
        checks = [
            CheckSpec(name="ck_users_age", expression="age >= 0"),
        ]
        uniques = [
            UniqueSpec(name="uq_users_email", columns=["email"]),
        ]
```

Typed spec classes (`IndexSpec`, `UniqueSpec`, `CheckSpec`) provide full IDE autocomplete and are the recommended way to declare model metadata. Plain dicts are also accepted for all three lists using the same field names.

### Column-level

| Attribute | Type | SQL | Backends |
|-----------|------|-----|----------|
| `comment` | `str` | `COMMENT ON COLUMN t.c IS '...'` | All |
| `public` | `bool` | Controls field visibility in `@auto_schema` (requires `dbwarden-fastapi` plugin) | All |

```python
class Meta(TableMeta):
    class internal_note:
        comment = "Internal system note"
        public = False
```

These attributes work with any `database_type`. Backend-specific subclasses (`PGTableMeta`, `MyTableMeta`, `CHTableMeta`) inherit all common attributes and add their own.

### Typed Spec Classes (Recommended)

Use `IndexSpec`, `UniqueSpec`, and `CheckSpec` from `dbwarden.databases` for cross-backend model metadata. These provide full IDE autocomplete and type checking:

```python
from dbwarden.databases import IndexSpec, UniqueSpec, CheckSpec

class Meta(TableMeta):
    indexes = [
        IndexSpec(name="ix_users_email", columns=["email"]),
    ]
    checks = [
        CheckSpec(name="ck_users_age", expression="age >= 0"),
    ]
    uniques = [
        UniqueSpec(name="uq_users_email", columns=["email"]),
    ]
```

Plain dicts using the same field names (`name`, `columns`, `expression`, `sql`) are also accepted for backwards compatibility, but typed specs are the idiomatic form.

### Column-Level Meta Base Class

For IDE autocomplete on column-level inner classes, use `PGColumnMeta` for PostgreSQL, `MyColumnMeta` for MySQL, `MdbColumnMeta` for MariaDB, or `CHColumnMeta` for ClickHouse. Each defines the cross-database attributes (`comment`, `public`) alongside its backend-specific spec object (`pg`, `ch`, `my`, `mdb`, `sq`). There is no shared `FieldMeta` base: the `*FieldMeta` classes were removed and their fields inlined into each backend's column Meta.

```python
from dbwarden.databases.pgsql import PGColumnMeta, pg

# Use typed spec objects for backend-specific column attributes:
#   pg = pg.field(collation=..., storage=..., ...)
#   ch = ch.field(codec=..., nullable=..., ...)
```

Backend-specific options are always set via a typed spec object attribute, never as flat attributes. For example, use `pg = pg.field(collation="en_US.UTF-8")` instead of the old `pg_collation = "en_US.UTF-8"`.

### Backend Subpackages

DBWarden organizes backend-specific types into subpackages under `dbwarden.databases`, also available there as short aliases:

| Alias | Subpackage | Key types |
|-------|------------|-----------|
| `pg` | `dbwarden.databases.pgsql` | `PgFieldSpec`, `PgIndexSpec`, `PgTableSpec` |
| `ch` | `dbwarden.databases.clickhouse` | `ChFieldSpec`, `ChIndexSpec`, `ChTableSpec` |
| `my` | `dbwarden.databases.mysql` | `MyFieldSpec`, `MyTableSpec` |
| `mdb` | `dbwarden.databases.mariadb` | `MdbFieldSpec`, `MdbTableSpec` |
| `sq` | `dbwarden.databases.sqlite` | `SqFieldSpec`, `SqTableSpec` |

Only `IndexSpec`, `PgIndexSpec`, and `ChIndexSpec` exist as typed index spec classes. MySQL, MariaDB, and SQLite use the base `IndexSpec` with the `indexes` attribute or plain dicts in their backend-specific index list (`my_indexes`, `sq_indexes`).

```python
from dbwarden.databases.pgsql import pg
from dbwarden.databases.clickhouse import ch
from dbwarden.databases.mysql import my
from dbwarden.databases.mariadb import mdb
from dbwarden.databases.sqlite import sq

# Use pg.field(), ch.field() for column-level metadata
pg_spec = pg.field(collation="en_US.UTF-8", storage="PLAIN")
ch_spec = ch.field(codec="ZSTD(3)", nullable=True)
```

## PostgreSQL Model Metadata

When `database_type="postgresql"`, DBWarden supports first-class PostgreSQL metadata via `class Meta(PGTableMeta)` inner classes. This is the **only** supported surface: `mapped_column(info=...)` raises `DBWardenConfigError`.

### Table-Level Meta

Inherit from `PGTableMeta` on your `class Meta`:

```python
from sqlalchemy import Integer
from sqlalchemy.orm import DeclarativeBase,     Mapped, mapped_column
from dbwarden.databases.pgsql import PGTableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)

    class Meta(PGTableMeta):
        pg_fillfactor = 80
        pg_tablespace = "fastspace"
        pg_storage_params = {
            "fillfactor": 80,
            "autovacuum_enabled": "false",
        }
```

`PGTableMeta` inherits all common `TableMeta` attributes (`comment`, `indexes`, `checks`, `uniques`) and adds PostgreSQL-specific ones (`pg_schema`, `pg_fillfactor`, `pg_tablespace`, `pg_storage_params`, `pg_unlogged`, `pg_partition`, `pg_inherits`, `pg_excludes`, `pg_indexes`, `pg_checks`, `pg_uniques`).

For PostgreSQL-specific indexes, use `PgIndexSpec` in `pg_indexes`:

```python
from dbwarden.databases.pgsql import PgIndexSpec

class Meta(PGTableMeta):
    pg_indexes = [
        PgIndexSpec("ix_users_email", ["email"],
            unique=True, using="gin"),
    ]
```

`PgIndexSpec` supports operator classes via `postgresql_ops` for GIN indexes on JSONB columns:

```python
PgIndexSpec("ix_users_data", ["data"],
    using="gin",
    postgresql_ops={"data": "jsonb_path_ops"})
```

This generates `CREATE INDEX ... ON users USING GIN (data jsonb_path_ops)`.

Full `PgIndexSpec` constructor fields: `name`, `columns`, `unique`, `using`, `where`, `include`, `with_params`, `tablespace`, `nulls_not_distinct`, `column_sorting`, `postgresql_ops`, `concurrently`. See [Indexes](databases/postgresql/indexes.md) for indexing options and [Declaring Metadata](databases/postgresql/declaring-metadata.md) for the full `PGTableMeta` attribute list.

### Column-Level Meta

Use `PGColumnMeta` inner classes named after the column. Use `pg = pg.field(...)` to set column-level options:

```python
from sqlalchemy import Integer, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    bio: Mapped[str] = mapped_column(Text)

    class Meta(PGTableMeta):
        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", collation="en_US.UTF-8")
```

`PGColumnMeta` includes the common `comment` and `public` attributes plus a `pg` attribute of type `PgFieldSpec` that bundles all PostgreSQL-specific column options.

For the full list of supported attributes, see [Tables & Columns](databases/postgresql/tables-and-columns.md) for column handler details and [Declaring Metadata](databases/postgresql/declaring-metadata.md) for PGColumnMeta options.

### PostgreSQL Views

Use `PGViewMeta` for models that represent PostgreSQL views or materialized views. Set `__tablename__` to the view name:

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.pgsql import PGViewMeta

class Base(DeclarativeBase):
    pass

class ActiveUser(Base):
    __tablename__ = "active_users"

    id: Mapped[int] = mapped_column(Integer)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(PGViewMeta):
        pg_view_query = "SELECT id, email FROM users WHERE active = true"
        pg_view_materialized = False
```

`PGViewMeta` attributes:

| Attribute | Type | Default | SQL |
|-----------|------|---------|-----|
| `pg_view_query` | `str` | `None` | `AS <query>` in CREATE VIEW |
| `pg_view_materialized` | `bool` | `False` | `CREATE MATERIALIZED VIEW` instead of `CREATE VIEW` |
| `pg_view_auto_refresh` | `bool` | `False` | Emits `REFRESH MATERIALIZED VIEW` in subsequent migrations |
| `pg_schema` | `str \| None` | `None` | Fully qualifies the view as `schema.view_name` |

Set `pg_view_auto_refresh = True` to keep a materialized view current after each schema change:

```python
class Meta(PGViewMeta):
    pg_view_query = "SELECT user_id, count(*) AS total FROM orders GROUP BY user_id"
    pg_view_materialized = True
    pg_view_auto_refresh = True
```

Each `make-migrations` run after the initial CREATE produces a `REFRESH MATERIALIZED VIEW` statement.

See [Views](databases/postgresql/views.md) for the full view lifecycle and auto-refresh configuration.

## ClickHouse Model Metadata

When `database_type="clickhouse"`, DBWarden supports first-class ClickHouse metadata via `class Meta(CHTableMeta)` inner classes. This is the **only** supported surface. Pass options via `mapped_column(info=...)` raises `DBWardenConfigError`.

### Table-Level Meta

Inherit from `CHTableMeta` on your `class Meta`:

```python
from datetime import date
from sqlalchemy.orm import DeclarativeBase,     Mapped, mapped_column
from dbwarden.databases.clickhouse import CHTableMeta, ChEngineSpec

class Base(DeclarativeBase):
    pass

class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(Int64, primary_key=True)
    event_date: Mapped[date] = mapped_column(Date)
    payload: Mapped[str] = mapped_column(String)

    class Meta(CHTableMeta):
        ch_engine = ChEngineSpec("ReplacingMergeTree", args=("version_column",))
        ch_order_by = ["region", "event_time"]
        ch_primary_key = "region"
        ch_partition_by = "toYYYYMM(event_time)"
        ch_sample_by = "intHash64(user_id)"
        ch_ttl = [
            "event_time + INTERVAL 1 MONTH DELETE",
            "event_time + INTERVAL 1 YEAR TO DISK 'cold'",
        ]
        ch_settings = {"index_granularity": "8192"}
```

`CHTableMeta` inherits all common `TableMeta` attributes (`comment`, `indexes`, `checks`, `uniques`) and adds ClickHouse-specific ones (`ch_engine`, `ch_order_by`, `ch_primary_key`, `ch_partition_by`, `ch_sample_by`, `ch_ttl`, `ch_settings`, `ch_object_type`, `ch_select_statement`, `ch_to_table`, `ch_dictionary`, `ch_dict_layout`, `ch_dict_source`, `ch_dict_lifetime`, `ch_dict_primary_key`, `ch_projections`, `ch_zookeeper_path`, `ch_replica_name`).

For the full list of supported attributes, see [ClickHouse Deep Dive](databases/clickhouse/index.md).

### Column-Level Meta

Use `CHColumnMeta` inner classes named after the column. Use `ch = ch.field(...)` to set column-level options:

```python
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.clickhouse import CHTableMeta, CHColumnMeta, ChEngineSpec, ch

class Base(DeclarativeBase):
    pass

class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(Int64, primary_key=True)
    payload: Mapped[str] = mapped_column(String)
    tags: Mapped[list[str]] = mapped_column(ARRAY(String))

    class Meta(CHTableMeta):
        ch_engine = ChEngineSpec("MergeTree")
        ch_order_by = "event_time"

        class payload(CHColumnMeta):
            ch = ch.field(codec="ZSTD(3)", nullable=False)

        class tags(CHColumnMeta):
            ch = ch.field(low_cardinality=True)
```

`CHColumnMeta` includes the common `comment` and `public` attributes plus a `ch` attribute of type `ChFieldSpec` that bundles all ClickHouse-specific column options.

### Engine Spec

Use `ChEngineSpec` for the table engine:

```python
from dbwarden.databases.clickhouse import ChEngineSpec

# Simple engine
ch_engine = ChEngineSpec("MergeTree")

# Engine with arguments
ch_engine = ChEngineSpec("ReplacingMergeTree", args=("version_column",))

# Replicated engine
ch_engine = ChEngineSpec("ReplicatedMergeTree",
    zookeeper_path="/clickhouse/tables/shard1/events",
    replica_name="{replica}")

# Distributed engine with settings
ch_engine = ChEngineSpec("Distributed",
    args=("cluster", "db", "events", "rand()"),
    settings={"insert_distributed_sync": "1"})
```

For replicated engines, `ch_zookeeper_path` and `ch_replica_name` are injected as the first two engine arguments. If `args` contains existing positional arguments, they come after the ZooKeeper path and replica name.

### Projections

Use `ProjectionSpec` in `ch_projections`:

```python
from dbwarden.databases.clickhouse import ProjectionSpec

class Meta(CHTableMeta):
    ch_order_by = ["author", "created_at"]
    ch_projections = [
        ProjectionSpec("by_author", "SELECT * ORDER BY author"),
        ProjectionSpec("daily_stats",
            "SELECT toDate(created_at) AS day, count() GROUP BY day"),
    ]
```

Current behavior:
- projection definitions are rendered into generated ClickHouse DDL
- safety checks classify added projections as `INFO`
- removed projections are classified as `WARNING`

### Skip Indexes

Use `ChIndexSpec` in `ch_indexes`:

```python
from dbwarden.databases.clickhouse import ChIndexSpec

class Meta(CHTableMeta):
    ch_indexes = [
        ChIndexSpec("ix_payload", ["payload"],
            type="bloom_filter", granularity=1),
    ]
```

### Materialized Views

Use `materialized_view()` builder with `CHViewMeta`:

```python
from dbwarden.databases.clickhouse import CHViewMeta, materialized_view

class EventRollup(Base):
    __tablename__ = "event_rollup_mv"

    event_date: Mapped[date] = mapped_column(Date)
    total: Mapped[int] = mapped_column(Int64)

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=(
                "SELECT toDate(event_time) AS event_date, count() AS total "
                "FROM events GROUP BY event_date"
            ),
            to="mv_target",
        )
```

Two storage modes:

| Shape | `to` | `engine` / `order_by` |
|-------|-----------|----------------------|
| Explicit target | Set | Not needed (target owns storage) |
| Implicit `.inner` | `None` | Required on `materialized_view()` |

Expression fields (select, order_by, partition_by, ttl) accept
SQLAlchemy `ColumnElement`, `ChRaw`, or plain `str`.

### Dictionaries

ClickHouse dictionaries use `ch_dictionary = True` with related `ch_dict_*` fields:

```python
class CountryCode(Base):
    __tablename__ = "country_codes"

    code: Mapped[str] = mapped_column(String)
    name: Mapped[str] = mapped_column(String)

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_layout = "FLAT()"
        ch_dict_source = "CLICKHOUSE(HOST 'localhost' TABLE 'countries')"
        ch_dict_lifetime = "MIN 0 MAX 3600"
        ch_dict_primary_key = "code"
```

Required fields when `ch_dictionary = True`:

| Field | Description | Example |
|-------|-------------|---------|
| `ch_dict_layout` | Dictionary layout | `"FLAT()"`, `"COMPLEX_KEY_HASHED()"` |
| `ch_dict_source` | Source configuration | `"CLICKHOUSE(HOST '...' TABLE '...')"` |
| `ch_dict_lifetime` | Cache lifetime | `"MIN 0 MAX 3600"` or `3600` |

Optional field:

| Field | Description | Default |
|-------|-------------|---------|
| `ch_dict_primary_key` | Primary key expression | First column |

Column types render as CH-native types (`Int64`, `String`).

### Column Hints

Use `CHColumnMeta` inner classes for per-column hints instead of `info={}`:

```python
from dbwarden.databases.clickhouse import ch

class Meta(CHTableMeta):
    class payload(CHColumnMeta):
        ch = ch.field(codec="ZSTD(3)", low_cardinality=True, nullable=False)
```

Supported `ch.field()` options:

| Keyword | Type | Description | Example |
|---------|------|-------------|---------|
| `codec` | `str` | Compression codec | `"ZSTD(3)"` |
| `default_expression` | `str` | Default value expression | `"now()"` |
| `materialized` | `str` | Materialized expression | `"lower(name)"` |
| `alias` | `str` | Alias expression | `"concat(a, b)"` |
| `ttl` | `str` | Column TTL expression | `"event_time + INTERVAL 1 YEAR"` |
| `low_cardinality` | `bool` | Wrap type in LowCardinality | `True` |
| `nullable` | `bool` | Wrap type in Nullable | `True` |

## MySQL Model Metadata

When `database_type="mysql"` (or `"mariadb"`), DBWarden supports first-class MySQL metadata via `class Meta(MyTableMeta)` inner classes. This is the **only** supported surface: `mapped_column(info=...)` raises `DBWardenConfigError`.

### Table-Level Meta

Inherit from `MyTableMeta` on your `class Meta`:

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(MyTableMeta):
        my_engine = "InnoDB"
        my_charset = "utf8mb4"
        my_collate = "utf8mb4_unicode_ci"
        my_row_format = "DYNAMIC"
        my_auto_increment = 1000
        comment = "Core user accounts"
```

`MyTableMeta` inherits all common `TableMeta` attributes (`comment`, `indexes`, `checks`, `uniques`) and adds MySQL-specific ones (`my_engine`, `my_charset`, `my_collate`, `my_row_format`, `my_auto_increment`).

For MariaDB, use `MdbTableMeta` which extends `MyTableMeta` with `mdb_page_compressed` and `mdb_page_compression_level`.

### Column-Level Meta

Use `MyColumnMeta` inner classes named after the column. Use `my = my.field(...)` to set column-level MySQL options:

```python
from sqlalchemy import Integer, String, TIMESTAMP
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta, MyColumnMeta, my

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    updated_at: Mapped[str] = mapped_column(TIMESTAMP)

    class Meta(MyTableMeta):
        class id(MyColumnMeta):
            comment = "Primary key"
            my = my.field(unsigned=True)

        class email(MyColumnMeta):
            my = my.field(charset="utf8mb4", collate="utf8mb4_unicode_ci")

        class updated_at(MyColumnMeta):
            my = my.field(on_update="CURRENT_TIMESTAMP")
```

Supported `my.field()` options:

| Keyword | Type | Description | Example |
|---------|------|-------------|---------|
| `unsigned` | `bool` | `UNSIGNED` on integer columns | `unsigned=True` |
| `charset` | `str` | Per-column character set | `charset="utf8mb4"` |
| `collate` | `str` | Per-column collation | `collate="utf8mb4_unicode_ci"` |
| `on_update` | `str` | `ON UPDATE` expression (typically for TIMESTAMP) | `on_update="CURRENT_TIMESTAMP"` |

For MariaDB, use `MdbColumnMeta` and `mdb.field()` which extends `my.field()` with `invisible` and `sequence` options.

Cross-backend column attributes (`comment`, `public`) are set directly on the inner class, not on the spec object.

See [Type Mapping](databases/postgresql/type-mapping.md) for the full SQLAlchemy-to-PostgreSQL type normalization table.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/observability/
========================================================================

# Observability

DBWarden provides Prometheus metrics and structured JSON logging for monitoring and debugging.

## Prometheus metrics

### Installation

Install the optional metrics dependency:

```bash
uv add "dbwarden[metrics]"
```

This installs `prometheus-client` which is required for metric collection and exposition.

### Enabling metrics

Set the `DBWARDEN_METRICS` environment variable to `true`:

```bash
export DBWARDEN_METRICS=true
```

When enabled, DBWarden instruments the `migrate` and `seed apply` commands with Prometheus metric recording. When disabled (or when `prometheus_client` is not installed), all metric functions are safe no-ops.

### Available metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `dbwarden_migrations_total` | Counter | `database`, `version` | Total migrations applied |
| `dbwarden_migration_duration_seconds` | Histogram | `database` | Duration of migration operations |
| `dbwarden_schema_version` | Gauge | `database` | Current schema version |
| `dbwarden_seed_version` | Gauge | `database` | Current seed version |
| `dbwarden_pending_migrations` | Gauge | `database` | Number of pending migrations |
| `dbwarden_migration_errors_total` | Counter | `database` | Total migration errors |

### FastAPI metrics endpoint

The `MetricsRouter` exposes a `GET /metrics` endpoint in Prometheus text format:

```python
from fastapi import FastAPI
from dbwarden_fastapi import MetricsRouter

app = FastAPI()
app.include_router(MetricsRouter(), prefix="/metrics")
```

The endpoint returns:

```
# HELP dbwarden_pending_migrations Number of pending migrations
# TYPE dbwarden_pending_migrations gauge
dbwarden_pending_migrations{database="primary"} 0
# HELP dbwarden_schema_version Current schema version
# TYPE dbwarden_schema_version gauge
dbwarden_schema_version{database="primary"} 5.0
```

Only active when `prometheus_client` is installed and `DBWARDEN_METRICS=true` is set. Returns 404 when disabled.

### MetricsMiddleware

The `MetricsMiddleware` is an ASGI middleware that refreshes pending-migration gauges on each HTTP request:

```python
from fastapi import FastAPI
from dbwarden_fastapi import MetricsMiddleware, MetricsRouter

app = FastAPI()
app.add_middleware(MetricsMiddleware)
app.include_router(MetricsRouter(), prefix="/metrics")
```

The middleware also records HTTP request duration via the migration duration histogram.

## JSON logging

DBWarden supports structured JSON logging for integration with log aggregation systems (ELK, Loki, Datadog, etc.).

### Enabling JSON logging

Set the `DBWARDEN_LOG_JSON` environment variable to `true`:

```bash
export DBWARDEN_LOG_JSON=true
```

When enabled, all DBWarden log output uses newline-delimited JSON format:

```json
{"timestamp": "2025-06-01T10:00:00.123456", "level": "INFO", "logger": "dbwarden", "message": "Applying migration 0003", "db_name": "primary", "db_type": "postgresql"}
{"timestamp": "2025-06-01T10:00:01.234567", "level": "INFO", "logger": "dbwarden", "message": "Migration 0003 applied successfully", "db_name": "primary", "db_type": "postgresql"}
```

### JSON log fields

| Field | Description |
|-------|-------------|
| `timestamp` | ISO-8601 timestamp with microseconds |
| `level` | Log level (DEBUG, INFO, WARNING, ERROR) |
| `logger` | Logger name |
| `message` | Log message text |
| `db_name` | Database name (when applicable) |
| `db_type` | Database type (when applicable) |
| `exception` | Exception traceback (when applicable) |

## Environment variables reference

| Variable | Value | Effect |
|----------|-------|--------|
| `DBWARDEN_METRICS` | `true` | Enable Prometheus metric recording and exposition |
| `DBWARDEN_LOG_JSON` | `true` | Enable JSON-formatted log output |
| `DBWARDEN_MIGRATE_AUTH` | API key string | Require `X-API-Key` header for `POST /migrate` endpoint |
| `DBWARDEN_HEALTH_AUTH` | API key string | Require `X-API-Key` header for health endpoints |

See also: [Cookbook: Observability](../cookbook/11-observability.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/developing/object-plugins/
========================================================================

# Object Plugins

An object plugin adds a database object **type** to DBWarden's schema convergence pipeline. Core drives your handler through the same extract → canonicalize → diff → emit flow it uses for tables, so the object is diffed against the live database and emitted in the right SQL order.

## Scope Test

An object plugin is the right tool when the feature is **database state declared by the model (or config) and converged by DBWarden**, not a one-off script. Good candidates: extensions, roles, grants, policies, triggers, functions, sequences, and backend-specific declarative objects. If the thing you want isn't schema state DBWarden should diff and emit, use a value hook or a migration instead.

## The `ObjectHandler` Contract

A handler is any object exposing these attributes and methods (see `dbwarden.engine.core.protocol.ObjectHandler`):

```python
object_type: str                 # unique registration key, e.g. "pg_extension"
op_types: tuple[str, ...]        # op names this handler emits, e.g. ("create_pg_extension", ...)
run_phase: RunPhase              # PREAMBLE or DIFF
ordering: OrderingConstraint     # public anchors (optional)

def extract(self, snapshot) -> dict: ...           # DB snapshot -> current spec
def model_spec_from_config(self, config) -> dict: ...  # config -> desired spec
def model_spec_from_tables(self, model_tables) -> dict: ...  # models -> desired spec
def canonicalize(self, spec) -> dict: ...          # normalize for comparison
def diff(self, snap_spec, model_spec) -> tuple[list[Op], list[Op]]: ...  # (upgrade, rollback)
def emit(self, op, db_name=None, **kwargs) -> list[MigrationStatement]: ...  # Op -> SQL
```

## Step By Step: A `CREATE EXTENSION` Handler

The goal: converge the PostgreSQL extensions declared in config (`config.pg_extensions`) with those present in the database. This mirrors the real [`dbwarden-pgsql-extensions`](https://github.com/dbwarden-org/dbwarden-pgsql-extensions) `PgExtensionHandler`.

### 1. Import the public types

```python
from dbwarden.engine.core import (
    Anchor,
    MigrationStatement,
    Op,
    OrderingConstraint,
    RunPhase,
)
```

### 2. Declare identity, phase, and ordering

Extensions must exist before anything that uses them, so pin the handler between `PREAMBLE` and `BEFORE_TABLES`:

```python
class PgExtensionHandler:
    object_type = "pg_extension"
    op_types = ("create_pg_extension", "drop_pg_extension")
    run_phase = RunPhase.PREAMBLE
    ordering = OrderingConstraint(
        after=(Anchor.PREAMBLE,),
        before=(Anchor.BEFORE_TABLES,),
    )
```

### 3. Read current state from the snapshot

```python
    def extract(self, snapshot):
        return snapshot.get("pg_extensions", {})
```

### 4. Read desired state

```python
    def model_spec_from_config(self, config):
        return {name: {} for name in getattr(config, "pg_extensions", [])}

    def model_spec_from_tables(self, model_tables):
        return {}  # extensions come from config, not model tables
```

### 5. Canonicalize and diff

`canonicalize` normalizes both sides so comparison is apples-to-apples (the real handler lowercases names and sorts). `diff` returns `(upgrade_ops, rollback_ops)`, emitting a create for what's missing and a drop for what's extra, with the inverse recorded for rollback:

```python
    def canonicalize(self, spec):
        return {str(n).lower(): dict(v or {}) for n, v in sorted((spec or {}).items())}

    def diff(self, snap_spec, model_spec):
        upgrade_ops, rollback_ops = [], []
        snap, model = snap_spec or {}, model_spec or {}
        for name in sorted(set(model) - set(snap)):        # missing -> create
            attrs = {"name": name}
            upgrade_ops.append(Op("create_pg_extension", attrs, attrs))
            rollback_ops.insert(0, Op("drop_pg_extension", attrs, attrs))
        for name in sorted(set(snap) - set(model)):        # extra -> drop
            attrs = {"name": name}
            upgrade_ops.append(Op("drop_pg_extension", attrs, attrs))
            rollback_ops.insert(0, Op("create_pg_extension", attrs, attrs))
        return upgrade_ops, rollback_ops
```

### 6. Emit SQL

`emit` turns each `Op` into ordered SQL, branching on `op.object_type`. Use `self.statement_order`, which DBWarden sets from your `ordering` anchors (see below):

```python
    def emit(self, op, db_name=None, **kwargs):
        name = '"' + str(op.upgrade_attrs["name"]).replace('"', '""') + '"'
        if op.object_type == "create_pg_extension":
            return [MigrationStatement(
                order=self.statement_order,
                upgrade_sql=f"CREATE EXTENSION IF NOT EXISTS {name};",
                rollback_sql=f"DROP EXTENSION IF EXISTS {name};",
            )]
        if op.object_type == "drop_pg_extension":
            return [MigrationStatement(
                order=self.statement_order,
                upgrade_sql=f"DROP EXTENSION IF EXISTS {name};",
                rollback_sql=f"CREATE EXTENSION IF NOT EXISTS {name};",
            )]
        return []
```

### 7. Register it

`setup` lives in the package `__init__.py` and imports the handler module lazily:

```python
# src/dbwarden_example/__init__.py
def setup(registrar) -> None:
    from dbwarden_example.handler import PgExtensionHandler

    registrar.register_object_handler(PgExtensionHandler())
```

### 8. Declare your config keys

If your handler is driven by `database_config(...)` keys, register them too. A plugin owns its config keys, not just its handlers.

```python
CONFIG_KEYS = ("pg_extensions",)


def setup(registrar) -> None:
    from dbwarden_example.handler import PgExtensionHandler

    registrar.register_object_handler(PgExtensionHandler())
    registrar.register_config_key(*CONFIG_KEYS)
```

Core validates every keyword argument passed to `database_config(...)` against the registered keys. A key no plugin owns is rejected as an unknown argument, and a key belonging to a plugin that is not installed raises `DBWardenConfigError` naming the plugin to install. Both fire when `dbwarden.py` loads, so a missing plugin surfaces immediately rather than as a migration that silently produces nothing.

Read the config values off the `config` object your handler already receives, using the key name directly:

```python
extensions = getattr(config, "pg_extensions", []) or []
```

To stay loadable against cores that predate the config-key registry, guard the call:

```python
    register_config_key = getattr(registrar, "register_config_key", None)
    if register_config_key is not None:
        register_config_key(*CONFIG_KEYS)
```

## Ordering And The DAG

DBWarden orders all object handlers into a single directed acyclic graph, then emits their statements in that order.

- **Anchors** place a handler relative to core milestones. At registration, DBWarden validates your `OrderingConstraint` and derives a private `statement_order` from the anchors: you never touch the integers.
- **Object-to-object** constraints (`after_object`, `before_object`) place your handler relative to *other* handlers by `object_type`:

```python
ordering = OrderingConstraint(after_object=("role",))  # emit after the "role" handler
```

Anchors are the **public contract**: `PREAMBLE`, `BEFORE_TABLES`, `AFTER_TABLES`, `AFTER_CONSTRAINTS`, `AFTER_INDEXES`, `POSTAMBLE`. See [ordering anchors](../reference/ordering-anchors.md) for the full map and failure modes (unknown references, cycles, impossible pairs).

## Shared SQL Helpers

Emitting SQL that matches core's byte-for-byte matters: a plugin that quotes identifiers differently, or that skips `ON CLUSTER`, produces migrations that disagree with the ones core generates for neighbouring objects. Rather than have you copy those rules, the shared implementations are public in `dbwarden.engine.core.plugin_api`:

| Name | Use |
|---|---|
| `quote_pg(name)` | Quote a PostgreSQL identifier the way core does (reserved words only). |
| `qualified_name(name, schema)` | Render `schema.name`, or bare `name` when unqualified. |
| `build_create_policy_sql`, `build_alter_policy_sql` | Render row-level-security policy DDL. |
| `build_grant_sql`, `build_revoke_sql` | Render `GRANT` / `REVOKE`. |
| `emit_with_cluster`, `ClusterableStatement` | Wrap ClickHouse DDL so it honours the configured `ON CLUSTER`. |
| `strip_secret_values`, `has_visible_secrets`, `REDACTED` | Redact secret values before they reach a snapshot. |

The module sits under `dbwarden.engine.core`, so importing from it satisfies the public-API check in the [Verified standard](verified-standard.md).

## Conflicts And Overrides

`object_type` is the registration key. Two *different* plugins registering the same `object_type` raises `ObjectHandlerConflictError`. A plugin handler and a core handler with the same type is allowed: the plugin handler overrides core, which is how official plugins replace built-in fallbacks (e.g. the core `CREATE EXTENSION` preamble).

Overriding is a real transfer of responsibility, so DBWarden announces it rather than swapping silently:

```
WARNING  dbwarden.registry - Plugin 'dbwarden-pgsql-rbac' overrides the built-in
handler for object type 'role'. Migration SQL for 'role' now comes from the
plugin, not DBWarden core.
```

The warning is emitted once per plugin and object type per run. If you see one you did not expect, a plugin has taken over DDL generation for that object type, and the SQL in your migrations for it is the plugin's, not core's. Plugins requesting the Verified tier must declare their overrides in the verification issue.

## Tests

Object plugins run the value-plugin conformance checks plus two handler-specific ones from the shared harness (`dbwarden.plugin_conformance`); see the [Verified standard](verified-standard.md).

- `test_object_handler_conformance`: `assert_object_handler_conformance(handler, config=...)` exercises `extract`, `canonicalize`, `diff`, and `emit` against a minimal fixture and validates the returned types and SQL shape.
- `test_ordering_constraint_satisfiable`: `assert_ordering_constraint_satisfiable(handler)` confirms the constraint is not statically impossible (no impossible anchor pair; no unknown/cyclic object references).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/developing/overview/
========================================================================

# Plugin Development Overview

A DBWarden plugin is a Python package with one entry point in the `dbwarden.plugins` group. The entry point points to a `setup(registry)` callable that registers hooks or object handlers.

```toml
[project.entry-points."dbwarden.plugins"]
example = "dbwarden_example:setup"
```

`setup` is defined in the package's `__init__.py`, so the entry point is `dbwarden_example:setup` (not `...plugin:setup`). Hook functions live at module top level, and heavy or DBWarden-specific imports go inside the function body so importing the package stays side-effect-free:

```python
# src/dbwarden_example/__init__.py
import importlib.util
import sys
from pathlib import Path
from typing import Any


def load_model_module(path: Path, base_dir: Path) -> Any:
    spec = importlib.util.spec_from_file_location(Path(path).stem, path)  # deferred/lazy work
    module = importlib.util.module_from_spec(spec)
    sys.modules[spec.name] = module
    spec.loader.exec_module(module)
    return module


def setup(registrar) -> None:
    registrar.register("load_model_module", load_model_module)
```

## Two Kinds Of Plugin

- **[Value plugins](value-plugins.md)** supply a value at a named hook point: session factories, FastAPI routes, lifespans, seed commands, module loaders. Registered with `registry.register(name, callable)`.
- **[Object plugins](object-plugins.md)** add a database object type to the schema diff pipeline: extensions, roles, grants, triggers, types. Registered with `registry.register_object_handler(handler)`.

A single package may register both.

## Lifecycle

1. DBWarden discovers entry-point metadata (name, version, value) without importing your code.
2. DBWarden classifies the distribution into a trust tier by name.
3. If trust rules allow loading, DBWarden imports the entry point and calls `setup(registry)`.
4. `setup(registry)` registers value hooks and/or object handlers.

The `registry` argument is a `PluginRegistrar` bound to your distribution name, so DBWarden attributes every hook to your plugin (used for conflict reporting and `plugin list`).

## Naming Conventions

All plugins are distributed as `dbwarden-<name>`. The import package and entry-point key derive from the distribution name: `slug = distribution.removeprefix("dbwarden-").replace("-", "_")`, then import package = `dbwarden_<slug>` and entry key = `<slug>`.

| Role | Convention | Example |
|------|------------|---------|
| PyPI project | `dbwarden-<name>` | `dbwarden-audit` |
| Import package | `dbwarden_<slug>` | `dbwarden_audit` |
| Entry-point key | `<slug>` | `audit` |

The `dbwarden-` prefix is shared by every plugin; a plugin's trust tier comes from the curated Official/Verified lists in core, not from its name. See [publishing](publishing.md#naming) for the full rule.

## Project Structure

Start from the [`dbwarden-plugin-template`](https://github.com/dbwarden-org/dbwarden-plugin-template) GitHub template (see [publishing](publishing.md#starting-from-the-template)):

```text
dbwarden-example/
├── pyproject.toml
├── src/dbwarden_example/
│   ├── __init__.py        # hook functions + setup(registrar)
│   └── handler.py         # object plugins only
├── tests/
│   └── test_example_plugin.py
├── README.md
└── LICENSE
```

`setup(registrar)` goes at the bottom of `__init__.py`. There is no `plugin.py`.

## The Core Contract

- **You may import anything from `dbwarden`.** There is no allowlist and no import check to satisfy. If your plugin needs `dbwarden.output` to render like the rest of the CLI, or `dbwarden.repositories.*` to read a tracking table, import it.
- Three surfaces are **stable** and change only on a major version: `dbwarden.plugin` (`PluginRegistrar`, the registries, hook errors), `dbwarden.exceptions`, and `dbwarden.engine.core` including `dbwarden.engine.core.plugin_api`. Prefer them where they cover your need, since that is what you will not have to revisit on a core upgrade.
- Anything deeper is supported but moves faster. Pin your `dbwarden` dependency to the range you test against, and run CI against those versions. `plugin_conformance.core_imports_outside_stable_api(pkg)` lists your deeper imports so an upgrade has a checklist.
- Use public **ordering anchors**, never private `StatementOrder` integers, those are renumbered freely.

## Rules

- **No import-time side effects.** Registering a hook must happen inside `setup()`, never at module import. Importing your package should register nothing.
- **Defer heavy imports to call time.** Value hooks import their DBWarden internals (and optional deps like FastAPI or drivers) inside the hook function body; object plugins import their `handler` module inside `setup()`. That way merely importing the package pulls in nothing heavy.
- Match documented hook signatures exactly; see the [hook catalog](../reference/hook-catalog.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/developing/publishing/
========================================================================

# Publishing Plugins

## Starting From The Template

Use the [`dbwarden-plugin-template`](https://github.com/dbwarden-org/dbwarden-plugin-template) GitHub template repository: click **Use this template** to create your own repo, then rename the placeholders:

```bash
python bootstrap.py dbwarden-yourname
```

`bootstrap.py` applies the naming rule below, renames the package and test, and removes itself. The template is a working value plugin with a passing test and CI; adapt it to your hooks.

Every plugin uses the same shape: a setuptools `src/` layout, `setup(registrar)` defined at the bottom of the package `__init__.py`, and a single `dbwarden.plugins` entry point.

```text
dbwarden-example/
├── pyproject.toml
├── LICENSE
├── README.md
├── .gitignore
├── .github/workflows/test.yml
├── src/dbwarden_example/
│   └── __init__.py        # hook functions + setup(registrar)
└── tests/
    └── test_example_plugin.py
```

Object plugins add a `handler.py` next to `__init__.py` and import it lazily inside `setup()`. The official plugins are the reference implementations: [`dbwarden-fastapi`](https://github.com/dbwarden-org/dbwarden-fastapi) (value), [`dbwarden-pgsql-extensions`](https://github.com/dbwarden-org/dbwarden-pgsql-extensions) (object).

## Naming

All plugins, community and official alike, are distributed as `dbwarden-<name>`. The import package and entry-point key derive from the distribution name by one rule:

```
slug        = distribution_name.removeprefix("dbwarden-").replace("-", "_")
import pkg  = "dbwarden_" + slug
entry key   = slug
```

| Role | Convention | Example (`dbwarden-audit`) |
|------|------------|---------------------------|
| PyPI project | `dbwarden-<name>` | `dbwarden-audit` |
| Import package | `dbwarden_<slug>` | `dbwarden_audit` |
| Entry-point key | `<slug>` | `audit` |

The `dbwarden-` prefix is shared by every plugin. A plugin's trust tier is **not** inferred from its name; it comes from the curated Official and Verified lists in core (`dbwarden/_official.py`, `dbwarden/_verified.py`), and everything else is Community. Use DBWarden-owned names such as `dbwarden-fastapi` only for packages published by the DBWarden organization.

## PyPI Metadata

This mirrors what the official plugins ship (setuptools, `src/` layout, one entry point):

```toml
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "dbwarden-example"
version = "0.2.0"
description = "Example DBWarden plugin"
readme = "README.md"
requires-python = ">=3.12.7"
dependencies = ["dbwarden>=0.15.0"]

[project.entry-points."dbwarden.plugins"]
example = "dbwarden_example:setup"

[tool.setuptools.packages.find]
where = ["src"]

[tool.pytest.ini_options]
testpaths = ["tests"]
```

The entry point is `dbwarden_example:setup` because `setup` is defined in the package's `__init__.py`; there is no separate `plugin.py`. Add `[project.urls]`, `license`, and `keywords` if you want them; the official plugins keep their metadata minimal.

## Compatibility

Depend on DBWarden with a lower bound, as the official plugins do:

```toml
dependencies = ["dbwarden>=0.15.0"]
```

The plugin API is stable within the `0.x` series. To guarantee a core update never silently breaks your plugin, you can also cap the upper bound (`"dbwarden>=0.15.0,<1.0"`), though the official plugins currently pin only the lower bound.

## CI/CD Example

`.github/workflows/test.yml`:

```yaml
name: tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -e . pytest
      - run: pytest -q
```

Link a green run of this workflow when you [submit for verification](verified-standard.md#submit-for-verification).

## Trusted Publishing

Official DBWarden plugins use provenance-backed publishing (PyPI Trusted Publishing via GitHub Actions OIDC), which is what `dbwarden plugin add` verifies at install time. Community plugins can (and should) adopt Trusted Publishing too, but on its own it does not make a plugin Official: the Official tier is the curated list in `dbwarden/_official.py`.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/developing/value-plugins/
========================================================================

# Value Plugins

A value plugin supplies a value at a named hook point. Core calls the hook where optional behavior is needed, and, for single hooks, exactly one plugin may provide it. See the [hook catalog](../reference/hook-catalog.md) for every hook, its signature, and whether it is single or multi.

Common value hooks:

- `session_factory`, `sync_session_factory`: SQLAlchemy session dependency factories
- `clickhouse_session_factory`, `clickhouse_sync_session_factory`: ClickHouse client factories
- `load_model_module`, `load_config_module`: custom module loaders
- `lifespan`: FastAPI lifespan builder
- `health_routes`, `migration_routes`: route bundles (multi hooks)
- `seed_create`, `seed_apply`, `seed_list`, `seed_rollback`, `seed_export`: seed commands

## Registering A Hook

Define the hook function at module top level and register it in `setup()`:

```python
# src/dbwarden_example/__init__.py
def session_factory(database: str | None = None, *, dev: bool = False):
    from dbwarden_fastapi.engines import _async_session_factory  # deferred

    async def _dependency():
        async with _async_session_factory(database, dev=dev)() as session:
            yield session

    return _dependency


def setup(registrar) -> None:
    registrar.register("session_factory", session_factory)
```

`registrar.register(name, callable)` raises `ValueError` if `name` is not a known hook, so typos fail fast at load time.

## Walkthrough: The `dbwarden-fastapi` `setup()`

The official `dbwarden-fastapi` plugin defines each hook as a top-level function in its `__init__.py` and registers them all at the end. Its `setup()` is exactly:

```python
def setup(registrar) -> None:
    registrar.register("session_factory", session_factory)
    registrar.register("sync_session_factory", sync_session_factory)
    registrar.register("clickhouse_session_factory", clickhouse_session_factory)
    registrar.register("clickhouse_sync_session_factory", clickhouse_sync_session_factory)
    registrar.register("lifespan", lifespan)
    registrar.register("health_routes", health_routes)       # multi hook
    registrar.register("migration_routes", migration_routes) # multi hook
```

Each registered function imports what it needs from DBWarden (and FastAPI) **inside its body**, so importing the package registers the hooks without pulling in FastAPI. Core then resolves them where needed: `dbwarden_fastapi.get_session("primary")` calls the registered `session_factory`; if no plugin registered it, core uses its built-in fallback.

## The Import-Deferred Pattern

`setup` is exported from `__init__.py`, and registration happens only when DBWarden calls it. Keep heavy imports out of module scope by deferring them into each hook function:

```python
# src/dbwarden_example/__init__.py
def health_routes(*, auth_mode: str = "open", api_key: str | None = None):
    from fastapi import APIRouter  # deferred: FastAPI loads only when this hook runs

    router = APIRouter()
    # ... define routes ...
    return router


def setup(registrar) -> None:
    registrar.register("health_routes", health_routes)


__all__ = ["health_routes", "setup"]
```

Importing `dbwarden_example` must not import FastAPI or register anything until `setup()` runs.

## Multi vs Single Hooks

`health_routes` and `migration_routes` are **multi** hooks: several plugins may each contribute a router, and core collects them with `execute_all`. Every other value hook is **single**: if two plugins register it, core raises `HookConflictError` when the hook is invoked.

## Tests

Use the shared conformance harness (`dbwarden.plugin_conformance`) so your suite matches the [Verified standard](verified-standard.md) and earns the Verified badge; the [template](publishing.md#starting-from-the-template) wires it up for you. At minimum, value plugins should include:

- `test_setup_registers_hooks`: `setup(PluginRegistrar("dist"))` registers the declared hooks.
- `test_import_has_no_side_effects`: importing the package/module registers nothing and mutates no plugin global state.
- `test_entry_point_is_declared`: `pyproject.toml` exposes a discoverable `dbwarden.plugins` entry point.
- `test_hook_signature_compliance`: each registered callable accepts the documented arguments.

The official plugins use a lightweight fake registrar, matching how core calls `setup`:

```python
from dbwarden.plugin import HookRegistry
from dbwarden_example import setup


def setup_function() -> None:
    HookRegistry.clear()


def test_setup_registers_hooks() -> None:
    class Registrar:
        def register(self, hook_name, fn) -> None:
            HookRegistry.register(hook_name, fn, plugin="dbwarden-example")

    setup(Registrar())

    assert HookRegistry.is_registered("session_factory") is True
```

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/developing/verified-standard/
========================================================================

# Verified Plugin Standard

The **Verified** tier earns a plugin the **Verified** badge: it loads without per-user consent once its installed version meets the verified minimum. To qualify, the plugin must pass a focused, automatable conformance suite that proves it respects the DBWarden contract, does not compromise the user's environment, and (for object plugins) integrates correctly with the diff pipeline.

The suite is deliberately minimal. It does **not** review business logic, performance, or security; it ensures the plugin is a well-behaved citizen.

!!! warning "Verification is not a security audit"
    The Verified badge is a statement of contract adherence, not fitness for purpose. It says the plugin respects DBWarden's rules, will not break core, and will not execute code before you consent. It does **not** certify the plugin is free of vulnerabilities or malicious behavior, and it grants no sandboxing. A loaded plugin runs with full process privileges. Vet the source yourself for anything sensitive.

## The Conformance Harness

DBWarden ships the checks as `dbwarden.plugin_conformance`, so verification is reproducible rather than a one-time manual event. Each function raises `ConformanceError` with an explanation on failure. The [plugin template](publishing.md#starting-from-the-template) already wires them into `tests/test_conformance.py`; copy that file and adjust the distribution/package names.

## Required Tests

| # | Test | Harness call | Protects against |
|---|------|--------------|------------------|
| 1 | `test_entry_point_is_declared` | `assert_entry_point_declared(dist)` | Misconfigured packaging, missing/unresolvable entry point, a plugin that can never be discovered. |
| 2 | `test_import_has_no_side_effects` | `assert_import_has_no_side_effects(pkg)` | The #1 security-model violation: registering at import time bypasses the consent gate. Enforces classify-before-load. |
| 3 | `test_setup_registers_hooks` | `assert_setup_registers(setup, value_hooks=...)` | A broken `setup()` (wrong signature, swallowed exception, forgot to register) that installs but does nothing. |
| 4 | `test_hook_signature_compliance` | `assert_hook_signatures(setup)` | Runtime `TypeError` when core calls a hook with the documented arguments; caught at build time instead of migration time. |
| 5 | `test_core_imports_resolve` | `assert_core_imports_resolve(pkg)` | A plugin importing a DBWarden module the installed core does not have, which would otherwise fail on a user's machine instead of in CI. |
| 6 | `test_api_version_is_declared` | `assert_api_version_declared(pkg)` | A plugin with no declared contract version, which silently keeps loading after the contract changes under it. |
| 7 | `test_object_handler_conformance` (object plugins) | `assert_object_handler_conformance(handler, config=...)` | Handlers that crash the diff pipeline or emit malformed statements: `extract`/`canonicalize`/`diff`/`emit` must return the right types. |
| 8 | `test_ordering_constraint_satisfiable` (object plugins) | `assert_ordering_constraint_satisfiable(handler)` | Ordering constraints that can never be satisfied (impossible anchor pair, unknown/cyclic object references) and crash generation. |
| 9 | `test_idempotent_setup` (recommended) | `assert_idempotent_setup(setup)` | Double-loading in reloads/tests/interactive sessions: calling `setup()` twice must not raise or add new registrations. |

Tests 1 to 6 apply to every plugin. Tests 7 and 8 apply to object plugins; value-only plugins mark them not applicable and say so. Test 9 is recommended and may become required.

### What the harness checks for test 5

**Plugins may import anything from DBWarden core.** There is no allowlist. A plugin that needs `dbwarden.output` for consistent CLI rendering, or `dbwarden.repositories.*` to read tracking tables, should just import them.

What test 5 verifies is that every `dbwarden` module you import actually exists in the core you are built against. That catches the failure that actually bites users: a plugin pinned to `dbwarden>=0.15` importing a module that moved in 0.16, discovered at load time on their machine rather than in your CI.

Three surfaces are documented as stable and change only on a major version:

- `dbwarden.plugin` (`PluginRegistrar`, the registries, hook errors)
- `dbwarden.exceptions`
- `dbwarden.engine.core`, including `dbwarden.engine.core.plugin_api`

Building on those means less to revisit when core releases. Anything deeper is fair game, but pin your `dbwarden` dependency accordingly and keep CI running against the versions you claim to support. `plugin_conformance.core_imports_outside_stable_api(pkg)` lists your deeper imports so you know what to re-check on an upgrade; it reports, it does not fail.

## What This Does Not Cover

- **Business logic**: whether the emitted SQL does what the user wants (their own tests).
- **Performance**: no benchmark required.
- **Security audit**: no static analysis for malicious code beyond the import restriction.
- **Inter-plugin interaction**: the resolver's job.

## Submit For Verification

Open an issue in the DBWarden repository using the **Plugin Verification** template. It asks for the repository, the distribution name and requested minimum version, what the plugin provides, the core versions you test against, a link to a green conformance run, and two things worth explaining here.

### The plugin API version

The plugin contract is versioned separately from DBWarden's release version, as `dbwarden.plugin.PLUGIN_API_VERSION`. Declare the one you target on your package:

```python
DBWARDEN_PLUGIN_API = 1
```

Core refuses to load a plugin declaring a version it does not provide, reporting it as `incompatible` with both versions named, rather than letting it register and generate migrations under assumptions that no longer hold. Declaring nothing means "version 1", so plugins written before the contract was versioned keep working; the conformance suite still requires the declaration, because a plugin that never declares gets no protection when the contract moves.

The version changes only when a plugin could otherwise be *wrong* rather than loudly broken: a hook signature change, different semantics for a registered handler, a rename on the stable surface. Adding a hook or a new `plugin_api` helper does not change it.

### Declaring deep imports

The template asks you to paste the output of:

```bash
python -c "from dbwarden.plugin_conformance import core_imports_outside_stable_api as f; print('\n'.join(f('your_package')) or 'none')"
```

and justify each line. This is a review conversation, not a test: no automated check can tell a good reason from a bad one, which is why it belongs to a reviewer.

It also runs in your favour. A justification that generalises is an argument that the stable surface is missing something, and the outcome is usually that the helper becomes public in `dbwarden.engine.core.plugin_api` rather than that you are told to stop. That is exactly how `plugin_api` acquired `quote_pg`, `qualified_name`, the grant and policy builders, and `emit_with_cluster`: official plugins needed them, so they stopped being internal. If you would switch to a public equivalent, say so.

### Declaring object type overrides

A plugin handler that claims an `object_type` core already handles **replaces** core for that type: migration DDL for those objects comes from the plugin. DBWarden logs a warning when this happens, but a reviewer needs to know up front. Declare any overrides and explain why replacing core is right rather than adding a new type. Overriding is legitimate (it is how official plugins supersede built-in fallbacks) and it raises the review bar.

## CI Enforcement

To be Verified, the plugin must run the conformance suite in public CI (GitHub Actions or similar) on every push, with a visible status badge. The template's `.github/workflows/test.yml` does this. The reviewer confirms CI is green and that `tests/test_conformance.py` matches the standard, then does a quick sanity pass for obviously unsafe behavior (malware, core monkey-patching, network exfiltration). On verification, the distribution name and minimum version are added to `dbwarden/_verified.py` and ship in the next DBWarden release.

## Version Re-Verification

The verified minimum is a **floor**: patch and minor releases above it load automatically. A **new major version** (or any release that changes hooks, signatures, or object semantics) must request re-verification, and the floor in `dbwarden/_verified.py` is updated accordingly. Until then, a version below the recorded floor is treated as **Community** and requires consent.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/faq/
========================================================================

# Plugin FAQ

## Can I Use A Plugin Without The CLI?

Yes. Install it with `uv add` or `pip install`. DBWarden still applies trust rules when loading it, so community plugins installed this way still require consent (`dbwarden plugin trust <name>` or the interactive prompt).

## What Happens If An Official Plugin's Provenance Fails?

Installation aborts and nothing is installed. Official plugin install is fail-closed: if provenance cannot be verified, DBWarden refuses rather than installing unverified code.

## How Do I Know A Community Plugin Is Safe?

There is no automatic guarantee. Review the source, prefer Verified plugins, and only consent to plugins you have vetted yourself. Trust gates whether a plugin loads, not what it can do once loaded.

## Can Two Plugins Provide The Same Hook?

For **multi** hooks (`health_routes`, `migration_routes`), yes: core collects all providers. For **single** hooks, no: a second provider causes a `HookConflictError` when the hook runs. Two plugins registering the same object handler `object_type` raise `ObjectHandlerConflictError`.

## Can Plugins Depend On Each Other?

Yes. Use normal Python package dependencies. Object handlers from different plugins are ordered relative to each other with `OrderingConstraint` (`after_object` / `before_object`).

## Do Plugins Work In CI?

Yes. Install them as project dependencies. Use `--format json` on `plugin list` and `plugin info` for machine-readable output; Rich tables degrade gracefully when output is captured. Note that non-interactive runs never auto-consent, so trust community plugins ahead of time (commit `.dbwarden/consent.toml`).

## How Do I Get My Plugin Verified?

Follow the [Verified Plugin Standard](developing/verified-standard.md): pass the seven mandatory tests and open a review issue in the DBWarden repository.

## What Happens When I Upgrade A Community Plugin?

Consent is version-specific. After upgrading, the recorded consent no longer matches the new version, so the plugin is treated as unconsented until you run `dbwarden plugin trust <name>` again.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/
========================================================================

# DBWarden Plugins

Plugins let DBWarden keep a clean core while still supporting framework integrations, backend-specific schema objects, and community extensions. They are normal Python packages distributed through PyPI (or another package source) and discovered through the `dbwarden.plugins` entry point group.

DBWarden supports two plugin kinds:

- **Value plugins** supply values at named hook points: session factories, FastAPI routes, lifespans, seed commands, and module loaders.
- **Object plugins** add database object types to the schema diff pipeline: PostgreSQL extensions, roles, grants, policies, triggers, and other backend-specific objects.

Before plugin code is imported, DBWarden classifies the package by distribution name. Community plugin code is not imported until you explicitly consent to that exact version.

## Trust Tiers

| Tier | Meaning | Trust guarantee |
|------|---------|-----------------|
| Official | Built and maintained by the DBWarden organization. Published under organization-owned package names with provenance verified at install time. | Cryptographic and organizational. |
| Verified | Community-maintained, but passed the DBWarden plugin test standard and manual review. | Community review and technical compliance. |
| Community | Any `dbwarden.plugins` entry point not listed as Official or Verified. | Explicit consent only. |

Verification is not a security audit. Once loaded, a plugin runs with normal Python process privileges (it is not sandboxed). See [consent and trust](using-plugins/consent-and-trust.md).

## Philosophy

- Core defines stable contracts and migration semantics.
- Plugins use standard Python packaging and entry points.
- DBWarden classifies plugins before loading them (classify-before-load).
- Plugins register through `setup(registry)`, not import-time side effects.
- Object plugins use public ordering anchors, not private statement-order integers.

## Example

```bash
dbwarden plugin add dbwarden-fastapi
dbwarden plugin list
```

Then use the integration exposed by the plugin according to its documentation.

## Next Steps

- Start with the [quickstart](quickstart.md).
- Learn how to [install plugins](using-plugins/installing.md).
- Read the [trust model](using-plugins/consent-and-trust.md).
- Build your first plugin with the [developer overview](developing/overview.md).

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/quickstart/
========================================================================

# Quickstart: Add FastAPI Support

This quickstart walks through installing the official `dbwarden-fastapi` plugin, handling the consent prompt for community plugins, listing what DBWarden discovered, and using a session dependency in an endpoint.

## 1. Install The Plugin

Use DBWarden's plugin installer:

```bash
dbwarden plugin add dbwarden-fastapi
```

`dbwarden-fastapi` is an **Official** plugin, so `plugin add` verifies its provenance before installing and records the result in `.dbwarden/plugins.lock`.

You can also install directly with your package manager:

```bash
uv add dbwarden-fastapi
# or
pip install dbwarden-fastapi
```

To route `plugin add` through `uv` instead of pip, pass `--uv`:

```bash
dbwarden plugin add dbwarden-fastapi --uv
```

Preview the plan without touching your environment:

```bash
dbwarden plugin add dbwarden-fastapi --dry-run
```

```text
              Plugin Add (dry run): dbwarden-fastapi
  Tier            official
  Installer       python -m pip install dbwarden-fastapi
  Version         latest
  After install   provenance-verified, lockfile updated
```

## 2. Consent For Community Plugins

Official and Verified plugins load automatically. A **Community** plugin, any distribution not listed in core, is discovered but not imported until you consent to the exact installed version. When you run a DBWarden command in an interactive terminal, you are prompted:

```text
Enable community plugin 'dbwarden-example' version 0.1.0? [y/N]:
```

Answering `y` records consent in `.dbwarden/consent.toml` and loads the plugin. You can also consent ahead of time:

```bash
dbwarden plugin trust dbwarden-example
```

## 3. List Plugins

```bash
dbwarden plugin list
```

```text
                                     DBWarden Plugins
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Distribution     ┃ Version ┃ Tier     ┃ Trusted ┃ State  ┃ Hooks           ┃ Objects ┃ Lock       ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━┩
│ dbwarden-fastapi │ 0.1.0   │ official │ yes     │ loaded │ health_routes,  │ -       │ provenance │
│                  │         │          │         │        │ session_factory │         │            │
└──────────────────┴─────────┴──────────┴─────────┴────────┴─────────────────┴─────────┴────────────┘
```

For machine-readable output, add `--format json`:

```bash
dbwarden plugin list --format json
```

## 4. Use The Integration

The FastAPI integration exposes session dependencies through DBWarden's FastAPI API. A minimal endpoint that uses the async session for the `primary` database:

```python
from fastapi import Depends, FastAPI
from sqlalchemy.ext.asyncio import AsyncSession

from dbwarden_fastapi import get_session

app = FastAPI()


@app.get("/users")
async def users(session: AsyncSession = Depends(get_session("primary"))):
    result = await session.execute(...)
    return {"users": result.scalars().all()}
```

`get_session` resolves the dependency through the plugin's `session_factory` hook. This import path is provided by the `dbwarden-fastapi` plugin package, not by DBWarden core. See the plugin's README for configuration.

## Next Steps

- [Installing plugins](using-plugins/installing.md)
- [Consent and trust](using-plugins/consent-and-trust.md)
- [Plugin CLI reference](reference/plugin-cli.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/reference/hook-catalog/
========================================================================

# Hook Catalog

Value hooks are registered with `registry.register(name, callable)`. Object handlers are registered with `registry.register_object_handler(handler)`. The known hook names are defined by `KNOWN_VALUE_HOOKS` in `dbwarden.plugin`; registering an unknown name raises `ValueError`.

## Value Hooks

| Hook | Multi | Signature | Returns | Provided by (example) |
|------|-------|-----------|---------|-----------------------|
| `session_factory` | no | `(database: str \| None = None, *, dev: bool = False)` | async session dependency (callable) | `dbwarden-fastapi` |
| `sync_session_factory` | no | `(database: str \| None = None, *, dev: bool = False)` | sync session dependency (callable) | `dbwarden-fastapi` |
| `clickhouse_session_factory` | no | `(database: str \| None = None, *, dev: bool = False)` | async ClickHouse dependency (callable) | `dbwarden-fastapi` |
| `clickhouse_sync_session_factory` | no | `(database: str \| None = None, *, dev: bool = False)` | sync ClickHouse dependency (callable) | `dbwarden-fastapi` |
| `load_model_module` | no | `(path: Path, base_dir: Path)` | loaded module | `dbwarden-sandbox` |
| `load_config_module` | no | `(path: Path, base_dir: Path)` | `None` | `dbwarden-sandbox` |
| `lifespan` | no | `(app=None, *, mode="check", **kwargs)` | async context manager | `dbwarden-fastapi` |
| `health_routes` | yes | `(*, auth_mode: str = "open", api_key: str \| None = None)` | `APIRouter` | `dbwarden-fastapi` |
| `migration_routes` | yes | `(*, auth_mode: str = "open", api_key: str \| None = None)` | `APIRouter` | `dbwarden-fastapi` |
| `seed_create` | no | `(description, *, seed_type="sql", database=None, verbose=False)` | `None` | `dbwarden-seeds` |
| `seed_apply` | no | `(*, version=None, dry_run=False, database=None, all_databases=False, verbose=False)` | `None` | `dbwarden-seeds` |
| `seed_list` | no | `(*, database=None, all_databases=False, verbose=False, prune=False)` | `None` | `dbwarden-seeds` |
| `seed_rollback` | no | `(*, count=None, to_version=None, database=None, all_databases=False, verbose=False)` | `None` | `dbwarden-seeds` |
| `seed_export` | no | `(*, database=None, all_databases=False, output_dir="seeds")` | `None` | `dbwarden-seeds` |

**Multi** hooks (`health_routes`, `migration_routes`) may be provided by several plugins; core collects all of them. Every other value hook is **single**: two providers cause a `HookConflictError` when the hook runs.

## Object Handlers

Object handlers are not named hooks; they register through `registry.register_object_handler(handler)` and are keyed by the handler's `object_type`. See [object plugins](../developing/object-plugins.md). Registering the same `object_type` from two different plugins raises `ObjectHandlerConflictError`.

## How Core Calls Hooks

Core resolves value hooks through the `HookRegistry`:

```python
from dbwarden.plugin import HookRegistry, HookNotRegisteredError

# Single hook: exactly one provider, or fall back to core behavior.
try:
    dependency = HookRegistry.execute_single("session_factory", "primary", dev=False)
except HookNotRegisteredError:
    dependency = core_default_session("primary")

# Multi hook: collect every provider's contribution.
routers = HookRegistry.execute_all("health_routes", auth_mode="open")
```

- `execute_single(name, *args, **kwargs)` raises `HookNotRegisteredError` when no plugin provides the hook and `HookConflictError` when more than one does.
- `execute_all(name, *args, **kwargs)` returns a list of every provider's result (empty if none).

Plugin authors do not call these; they are shown here so you can see exactly how and with what arguments core will invoke your callable.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/reference/ordering-anchors/
========================================================================

# Ordering Anchors

Object plugins position their SQL using public **anchors**, not private statement-order integers. At registration, DBWarden validates your `OrderingConstraint` and derives the internal `statement_order` from the anchors for you.

## The `Anchor` Enum

| Anchor | Meaning |
|--------|---------|
| `PREAMBLE` | Earliest setup statements (extensions, roles, schemas). |
| `BEFORE_TABLES` | Objects that must exist before tables (types, domains). |
| `AFTER_TABLES` | Objects emitted after table creation (added columns). |
| `AFTER_CONSTRAINTS` | Objects emitted after table constraints. |
| `AFTER_INDEXES` | Objects emitted after indexes. |
| `POSTAMBLE` | Final statements (views, grants that depend on everything). |

Anchors run in the order listed above.

## `OrderingConstraint`

```python
@dataclass(frozen=True)
class OrderingConstraint:
    after: tuple[Anchor, ...] = ()
    before: tuple[Anchor, ...] = ()
    after_object: tuple[str, ...] = ()   # by another handler's object_type
    before_object: tuple[str, ...] = ()
```

- `after` / `before` place the handler relative to core milestones.
- `after_object` / `before_object` place it relative to other object handlers by their `object_type`, forming a DAG that DBWarden topologically sorts.

## Examples

Extensions, first thing (PostgreSQL):

```python
ordering = OrderingConstraint(after=(Anchor.PREAMBLE,), before=(Anchor.BEFORE_TABLES,))
```

Roles, before the objects that reference them:

```python
ordering = OrderingConstraint(after=(Anchor.PREAMBLE,), before=(Anchor.BEFORE_TABLES,))
```

Grants, after everything they depend on:

```python
ordering = OrderingConstraint(after=(Anchor.AFTER_INDEXES,), before=(Anchor.POSTAMBLE,))
```

Triggers, after tables and their constraints exist:

```python
ordering = OrderingConstraint(after=(Anchor.AFTER_CONSTRAINTS,))
```

A grant handler that must run after a role handler, by object type:

```python
ordering = OrderingConstraint(after_object=("role",))
```

## Failure Modes

DBWarden rejects invalid ordering at registration or run time by raising `OrderingError`:

- **Impossible anchor pair**, `after` an anchor that comes at or after `before`:

  ```python
  OrderingConstraint(after=(Anchor.POSTAMBLE,), before=(Anchor.PREAMBLE,))
  # OrderingError: Impossible ordering: after postamble and before preamble
  ```

- **Unknown object reference**, naming an `object_type` no registered handler provides:

  ```python
  OrderingConstraint(after_object=("missing",))
  # OrderingError: Unknown object ordering reference 'missing' for '<your type>'
  ```

- **Cycle**, two handlers that each require running after the other:

  ```python
  # first: after_object=("second",);  second: after_object=("first",)
  # OrderingError: Object handler ordering cycle detected: first, second
  ```

## Why `StatementOrder` Is Private

The internal `StatementOrder` integers (and their exact values) are an implementation detail and are renumbered as core evolves. Anchors are the **public, stable contract**. Never read or set `statement_order` yourself; declare `ordering` with anchors and let DBWarden compute the rest.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/reference/plugin-cli/
========================================================================

# Plugin CLI

All commands live under `dbwarden plugin`. Unlike other DBWarden commands, `dbwarden plugin` does not auto-load plugins first, so it can inspect and manage plugins without importing them.

## Subcommands

| Command | Purpose |
|---------|---------|
| `list` | Show discovered plugins as a table or JSON. |
| `info <name>` | Show one plugin's metadata and provenance. |
| `add <name>` | Install a plugin (provenance-verified for official). |
| `remove <name>` | Uninstall a plugin and clean up its consent/lock state. |
| `trust <name>` | Record consent for a community plugin's installed version. |
| `untrust <name>` | Revoke consent for a community plugin. |

## Flags

| Command | Flag | Effect |
|---------|------|--------|
| `list`, `info` | `--format`, `-f` `table\|json` | Output format (default `table`). |
| `add` | `--uv` | Install with `uv add` instead of pip. |
| `add` | `--version <v>` | Pin an exact version (`dist==<v>`). |
| `add` | `--dry-run` | Print the install plan without installing. |
| `remove` | `--uv` | Uninstall with `uv remove` instead of pip. |
| `remove` | `--dry-run` | Print the uninstall plan without uninstalling. |

## list

```bash
dbwarden plugin list
dbwarden plugin list --format json
```

Columns: Distribution, Version, Tier, Trusted, State, Hooks, Objects, Lock.

```text
                                     DBWarden Plugins
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Distribution     ┃ Version ┃ Tier     ┃ Trusted ┃ State  ┃ Hooks           ┃ Objects ┃ Lock       ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━┩
│ dbwarden-fastapi │ 0.1.0   │ official │ yes     │ loaded │ health_routes,  │ -       │ provenance │
│                  │         │          │         │        │ session_factory │         │            │
└──────────────────┴─────────┴──────────┴─────────┴────────┴─────────────────┴─────────┴────────────┘
```

## info

```bash
dbwarden plugin info dbwarden-fastapi
dbwarden plugin info dbwarden-fastapi --format json
```

Shows entry point, tier, trust/load state, registered hooks and object handlers, official repository, verified minimum version, and lockfile provenance fields when present. Exits `1` if the plugin is not found.

## add

```bash
dbwarden plugin add dbwarden-fastapi
dbwarden plugin add dbwarden-fastapi --version 0.2.0 --uv
dbwarden plugin add dbwarden-example --dry-run
```

Official plugin installation verifies provenance and fails closed when verification is unavailable. Community plugins are installed but not trusted; the command prints the follow-up `trust` hint. `--dry-run` prints tier, installer command, and post-install behavior without installing.

## remove

```bash
dbwarden plugin remove dbwarden-example
dbwarden plugin remove dbwarden-example --dry-run --uv
```

Uninstalls the distribution and removes its consent entry and lockfile record.

## trust / untrust

```bash
dbwarden plugin trust dbwarden-example
dbwarden plugin untrust dbwarden-example
```

`trust` records consent for the currently installed version in `.dbwarden/consent.toml`. `untrust` removes it.

## File Formats

### Consent file: `.dbwarden/consent.toml`

Written by `trust` and by accepting the interactive prompt. Consent is version-specific.

```toml
[consent."dbwarden-example"]
version = "0.1.0"
consented_at = "2026-07-22T14:03:11.482915+00:00"
```

### Lockfile: `.dbwarden/plugins.lock`

Written by `plugin add` for official (provenance-verified) installs. The section key is the distribution name with the `dbwarden-` prefix stripped and dashes replaced by underscores (for example `dbwarden-pgsql-extensions` becomes `pgsql_extensions`).

```toml
[pgsql_extensions]
distribution = "dbwarden-pgsql-extensions"
version = "0.3.0"
filename = "dbwarden_pgsql_extensions-0.3.0-py3-none-any.whl"
sha256 = "abc123..."
tier = "official"
verified = "provenance"
identity = "https://github.com/dbwarden-org/dbwarden-pgsql-extensions"
installed_at = "2026-07-22T14:05:40.001122+00:00"
```

Commit both files so teammates and CI inherit your trust and provenance decisions.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/using-plugins/consent-and-trust/
========================================================================

# Consent And Trust

DBWarden classifies every plugin entry point **before importing any plugin code**. Classification is by distribution name, so a plugin's tier is known without running it.

## The Three Tiers

| Tier | Loads automatically? | Basis of trust |
|------|----------------------|----------------|
| Official | Yes | Organization-owned name + provenance verified at install. |
| Verified | Yes, when version ≥ verified minimum | Community review against the test standard. |
| Community | No, consent required | Your explicit, version-specific consent. |

### Official

Official plugins are DBWarden-maintained. `plugin add` verifies provenance and **fails closed** when verification is unavailable or invalid. They load without consent.

### Verified

Verified plugins are community-maintained but reviewed against the DBWarden plugin standard. They load **without consent** only when the installed version is at or above the verified minimum recorded in `dbwarden/_verified.py`. If the installed version is below the floor, DBWarden logs a warning and treats the plugin as **Community** (consent required).

### Community

Community plugins require explicit consent before loading. In an interactive terminal, DBWarden prompts on first use:

```text
Enable community plugin 'dbwarden-example' version 0.1.0? [y/N]:
```

Accepting records consent. Non-interactive runs (CI, scripts) never auto-consent: an unconsented community plugin is skipped with a warning:

```text
Skipping community plugin 'dbwarden-example' (not consented).
Run `dbwarden plugin trust dbwarden-example` to enable it.
```

## Consent Commands

```bash
dbwarden plugin trust dbwarden-example      # record consent for the installed version
dbwarden plugin untrust dbwarden-example    # revoke consent
```

## The Consent File

Consent is stored per-project in `.dbwarden/consent.toml`:

```toml
[consent."dbwarden-example"]
version = "0.1.0"
consented_at = "2026-07-22T14:03:11.482915+00:00"
```

Consent is bound to the exact `version`. If you upgrade the plugin, the recorded version no longer matches and the plugin is treated as unconsented until you trust the new version. Commit this file to share consent decisions with your team.

## Security Boundary

Classify-before-load ensures untrusted community plugin code is **not imported** automatically: the security decision happens before `import`.

!!! warning "Not a sandbox"
    Trust gates *whether* a plugin loads, not *what it can do*. Once a plugin is loaded it runs with **full Python process privileges**: it can read files, open network connections, and call any API your process can. Consent is a statement that you have reviewed and trust the code, not a containment mechanism. Approval is likewise a review, **not a security audit**.

Verified and Official tiers exist to reduce how often you must make that trust decision manually, not to make loading arbitrary code safe.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/using-plugins/finding-plugins/
========================================================================

# Finding Plugins

DBWarden plugins are standard Python packages that expose a `dbwarden.plugins` entry point. DBWarden classifies each discovered distribution into a [trust tier](consent-and-trust.md) before importing it, by matching its distribution name against the curated Official and Verified lists in core.

## Search PyPI

All plugins are distributed as:

```text
dbwarden-<name>
```

Search PyPI for `dbwarden-` to find them. The name alone does not confer trust: Official plugins are the DBWarden-owned packages listed in `dbwarden/_official.py` (for example `dbwarden-fastapi`, `dbwarden-pgsql-extensions`); every other `dbwarden-*` package is Community until it reaches the Verified list.

## Official Plugins

Official plugins are built and maintained by the DBWarden organization under the [`dbwarden` GitHub organization](https://github.com/dbwarden). They are enumerated in core (`dbwarden/_official.py`) and are eligible for provenance verification during `dbwarden plugin add`.

| Package | Repository | Purpose |
|---------|------------|---------|
| `dbwarden-fastapi` | [dbwarden-org/dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi) | FastAPI sessions, health/migration routes, lifespan. |
| `dbwarden-pgsql-extensions` | [dbwarden-org/dbwarden-pgsql-extensions](https://github.com/dbwarden-org/dbwarden-pgsql-extensions) | PostgreSQL `CREATE EXTENSION` diffing. |
| `dbwarden-pgsql-rbac` | [dbwarden-org/dbwarden-pgsql-rbac](https://github.com/dbwarden-org/dbwarden-pgsql-rbac) | PostgreSQL roles, grants, policies. |
| `dbwarden-pgsql-types` | [dbwarden-org/dbwarden-pgsql-types](https://github.com/dbwarden-org/dbwarden-pgsql-types) | PostgreSQL enum/domain/composite types. |
| `dbwarden-ch-rbac` | [dbwarden-org/dbwarden-ch-rbac](https://github.com/dbwarden-org/dbwarden-ch-rbac) | ClickHouse roles and grants. |
| `dbwarden-seeds` | [dbwarden-org/dbwarden-seeds](https://github.com/dbwarden-org/dbwarden-seeds) | Seed command implementations. |
| `dbwarden-sandbox` | [dbwarden-org/dbwarden-sandbox](https://github.com/dbwarden-org/dbwarden-sandbox) | Isolated model/config module loading. |

The authoritative list is always `OFFICIAL_PLUGINS` in `dbwarden/_official.py`.

## Verified Plugins

Verified plugins are community-maintained packages that passed DBWarden's [plugin test standard](../developing/verified-standard.md) and manual review. They load without consent once the installed version meets the verified minimum.

| Package | Minimum verified version | Description |
|---------|--------------------------|-------------|
| _None yet_ – feel free to [be the first](../developing/verified-standard.md#submit-for-verification)! | – | Verified entries are listed in `dbwarden/_verified.py` and this table as the ecosystem grows. |

Verification is not a security audit. See [consent and trust](consent-and-trust.md).

## Community Plugins

Any other package with a `dbwarden.plugins` entry point is treated as **Community**. Community plugins require explicit consent before loading. A community-curated index may be maintained as a Markdown page in the DBWarden repository; until then, discover them via PyPI search and review the source before trusting.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/using-plugins/installing/
========================================================================

# Installing Plugins

## DBWarden CLI

```bash
dbwarden plugin add <distribution-name>
```

What `plugin add` does depends on the [trust tier](consent-and-trust.md) of the distribution:

- **Official**: DBWarden verifies provenance first. On success it installs the package and writes an entry to `.dbwarden/plugins.lock` recording the version, filename, SHA-256, and verifying identity. If provenance cannot be verified, installation aborts and nothing is installed (fail-closed).
- **Verified**: installed like any package; it will load automatically once the installed version meets the verified minimum.
- **Community**: the package is installed but **not** trusted. You must run `dbwarden plugin trust <name>` (or accept the interactive consent prompt) before DBWarden loads it.

```text
              Plugin Installed
  Installed community plugin 'dbwarden-example'.
Run `dbwarden plugin trust dbwarden-example` to allow it to load.
```

### Flags

| Flag | Applies to | Effect |
|------|------------|--------|
| `--uv` | `add`, `remove` | Use `uv add` / `uv remove` instead of pip. |
| `--version <v>` | `add` | Pin an exact version to install (`dist==<v>`). |
| `--dry-run` | `add`, `remove` | Print the plan (tier, installer command, post-install behavior) without changing anything. |

```bash
dbwarden plugin add dbwarden-fastapi --version 0.2.0 --uv
dbwarden plugin add dbwarden-fastapi --dry-run
```

## uv Or pip

You can install plugins directly without the CLI:

```bash
uv add dbwarden-fastapi
```

```bash
pip install dbwarden-fastapi
```

DBWarden still applies its trust model when loading. Community plugins installed this way still require consent, and Official plugins installed this way are **not** provenance-locked (no `.dbwarden/plugins.lock` entry is written unless you use `plugin add`).

## Version Pinning

Pin versions in your project dependencies for repeatable environments:

```bash
uv add "dbwarden-fastapi==0.2.0"
```

Community consent is version-specific: upgrading a community plugin invalidates prior consent and requires consent for the new version.

## What Happens During Official Install

Provenance verification uses PyPI's [PEP 740](https://peps.python.org/pep-0740/) attestations. DBWarden:

1. Looks up the distribution in `OFFICIAL_PLUGINS` for its expected GitHub repository and publishing workflow.
2. Resolves the target version on PyPI (the pinned/`--version` release, or the highest stable release) and selects its distribution file and SHA-256.
3. Fetches that file's attestation from PyPI's Integrity API and checks that the recorded Trusted-Publishing publisher is the expected GitHub repository and workflow, and that the attestation covers the file's exact digest.
4. On success, installs that **exact** version and writes a lockfile entry with `verified = "provenance"`, the identity, filename, and SHA-256.

Any missing attestation, publisher mismatch, digest mismatch, or network error makes the install **fail closed**: nothing is installed. Trust is anchored in PyPI's server-side attestation verification and TLS, the same root pip relies on.

## Updating Plugins

Update with your package manager, then reconcile DBWarden's view:

```bash
uv add "dbwarden-fastapi@latest"
dbwarden plugin list
dbwarden plugin info dbwarden-fastapi
```

For community plugins, re-run `dbwarden plugin trust <name>` after upgrading so consent matches the new version.

## Removing Plugins

```bash
dbwarden plugin remove <distribution-name>
```

This uninstalls the distribution and removes its consent entry and lockfile record. Use `--dry-run` to preview or `--uv` to uninstall via `uv remove`.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/plugins/using-plugins/plugin-list-and-info/
========================================================================

# Plugin List And Info

## List Plugins

```bash
dbwarden plugin list
```

The Rich table shows every discovered distribution with its version, tier, trust state, load state, the hooks and object handlers it registered, and its lockfile verification status.

```text
                                     DBWarden Plugins
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Distribution           ┃ Version ┃ Tier      ┃ Trusted ┃ State   ┃ Hooks           ┃ Objects      ┃ Lock       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ dbwarden-fastapi       │ 0.1.0   │ official  │ yes     │ loaded  │ health_routes,  │ -            │ provenance │
│                        │         │           │         │         │ session_factory │              │            │
│ dbwarden-pgsql-exten…  │ 0.3.0   │ official  │ yes     │ loaded  │ -               │ pg_extension │ provenance │
│ dbwarden-acme          │ 1.0.0   │ community │ no      │ skipped │ -               │ -            │ -          │
└────────────────────────┴─────────┴───────────┴─────────┴─────────┴─────────────────┴──────────────┴────────────┘
```

Column meanings:

- **Trusted**: `yes` if the plugin is allowed to load (official, verified-and-current, or consented community).
- **State**: `loaded`, `skipped` (untrusted community), `failed` (raised during `setup`), `incompatible` (built against a different plugin API version), or `discovered`.

An `incompatible` plugin is not broken: it targets a different version of the plugin contract than this DBWarden provides, so it was refused rather than allowed to register and produce migrations under the wrong assumptions. The **Error** row names both versions. Upgrade the plugin, or pin DBWarden to a release that provides the version it wants.
- **Hooks / Objects**: the value hooks and object handler types the plugin registered. A discovered-but-skipped plugin registers nothing, so these stay empty.
- **Lock**: the lockfile `verified` value (e.g. `provenance`) for provenance-locked installs.

### JSON Output

For CI or scripting, use `--format json`:

```bash
dbwarden plugin list --format json
```

```json
[
  {
    "distribution": "dbwarden-fastapi",
    "version": "0.1.0",
    "tier": "official",
    "trusted": true,
    "state": "loaded",
    "hooks": ["health_routes", "session_factory"],
    "object_handlers": [],
    "error": null,
    "lock": {"verified": "provenance", "identity": "https://github.com/dbwarden-org/dbwarden-fastapi", "...": "..."}
  }
]
```

## Plugin Details

```bash
dbwarden plugin info dbwarden-fastapi
```

```text
Plugin Info
  Distribution       dbwarden-fastapi
  Version            0.1.0
  Entry point        fastapi = dbwarden_fastapi:setup
  Tier               official
  Trusted            yes
  State              loaded
  Hooks              health_routes, session_factory
  Object handlers    -
  Error
  Repository         https://github.com/dbwarden-org/dbwarden-fastapi
  Verified minimum
  Lock verified      provenance
  Lock identity      https://github.com/dbwarden-org/dbwarden-fastapi
```

- For **Official** plugins, details include the repository and lockfile provenance fields.
- For **Verified** plugins, the **Verified minimum** row shows the version floor.
- For **Community** plugins, trust reflects consent state; `Repository` and `Verified minimum` are empty.

`plugin info` also supports `--format json`.

## Checking Registered Hooks

Both `plugin list` and `plugin info` report the hooks and object handlers registered **after load**. If a plugin is discovered but skipped (untrusted) or failed during `setup`, those columns are empty and, for failures, the **Error** row explains why.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/reference/configuration-api/
========================================================================

# Configuration API Reference

Complete reference for the `database_config()` function.

This is a reference page. For step-by-step guides, see
[Quick Start](../configuration/quick-start.md), [Concepts](../configuration/concepts.md),
or [Production Patterns](../configuration/production-patterns.md).

## Function Signature

```python
def database_config(
    *,
    database_name: str,
    database_type: Literal["sqlite", "postgresql", "mysql", "mariadb", "clickhouse"] = "sqlite",
    database_url_sync: str | None = None,
    database_url_async: str | None = None,
    default: bool = False,
    migrations_dir: str | None = None,
    migration_table: str | None = None,
    seed_table: str | None = None,
    auto_apply_seeds: bool = False,
    model_paths: list[str] | None = None,
    model_tables: list[str] | None = None,
    dev_database_type: str | None = None,
    dev_database_url: str | None = None,
    overlap_models: bool = False,
    secure_values: bool = False,
    pg_schema: str | None = None,
    pg_migration_lock_timeout: int | None = None,
    **plugin_config: Any,
) -> DatabaseHandle:
    """Register a database in DBWarden and return a handle with session dependencies."""
```

## Plugin-contributed arguments

Backend object keys such as `pg_roles`, `pg_functions`, and `ch_grants` are **not** part of core. Each is owned by the plugin that also supplies its object handler, and arrives through `**plugin_config`.

Core validates every extra keyword against the plugins actually installed:

- A key owned by a plugin that is **not installed** raises `DBWardenConfigError` naming the plugin to install.
- A key **no plugin owns** raises `DBWardenConfigError` as an unexpected keyword argument, so typos fail immediately.

Both fire when your `dbwarden.py` is loaded, not at migration time.

| Config keys | Plugin |
|---|---|
| `pg_roles`, `pg_default_privileges` | `dbwarden-pgsql-rbac` |
| `pg_domains`, `pg_sequences`, `pg_composite_types` | `dbwarden-pgsql-types` |
| `pg_extensions`, `pg_functions`, `pg_triggers`, `pg_event_triggers`, `pg_extended_statistics` | `dbwarden-pgsql-extensions` |
| `ch_named_collections`, `ch_roles`, `ch_users`, `ch_row_policies`, `ch_quotas`, `ch_settings_profiles`, `ch_grants` | `dbwarden-ch-rbac` |

Install with `dbwarden plugin add <name>`. `pg_schema` and `pg_migration_lock_timeout` are core and need no plugin.

## Required arguments

| Argument | Type | Description |
|----------|------|-------------|
| `database_name` | `str` | unique name for this database in your project |
| `database_type` | `str` | backend type: `sqlite`, `postgresql`, `mysql`, `mariadb`, or `clickhouse` (default: `"sqlite"`) |

At least one of `database_url_sync` or `database_url_async` must be provided.

## Optional arguments

| Argument | Type | Default | Description |
|----------|------|---------|-------------|
| `database_url_sync` | `str | None` | `None` | synchronous connection URL (used by migrations, CLI, and sync sessions) |
| `database_url_async` | `str | None` | `None` | async connection URL (used by async sessions; falls back to `database_url_sync` if omitted) |
| `default` | `bool` | `False` | if `True`, this database is used when `--database` is omitted |
| `migrations_dir` | `str | None` | `None` | custom migration directory path (defaults to `migrations/<database_name>`) |
| `migration_table` | `str | None` | `None` | custom migration tracking table name (defaults to `_dbwarden_migrations`) |
| `seed_table` | `str | None` | `None` | custom seed tracking table name (defaults to `_dbwarden_seeds`) |
| `auto_apply_seeds` | `bool` | `False` | if `True`, automatically apply pending code seeds after `migrate` |
| `model_paths` | `list[str] | None` | `None` | list of Python import paths containing SQLAlchemy models for this database |
| `model_tables` | `list[str] | None` | `None` | optional filter: only these table names are owned by this database |
| `dev_database_type` | `str | None` | `None` | backend type for local development (used with `--dev`) |
| `dev_database_url` | `str | None` | `None` | connection URL for local development (used with `--dev`) |
| `overlap_models` | `bool` | `False` | if `True`, allow model path overlap with other databases |
| `secure_values` | `bool` | `False` | if `True`, display commands show variable names instead of resolved values |

## Field descriptions

### `database_name`

A unique identifier for this database within your project.

**Requirements:**
- Must be unique across all entries in your config source
- Used in CLI `--database` / `-d` flags to select this database
- Becomes part of migration filename prefix (for versioned migrations)

**Examples:**
```python
database_name="primary"
database_name="analytics"
database_name="legacy"
```

Use descriptive names that reflect the database's purpose: `primary`, `analytics`, `audit_logs`, etc.

### `database_type`

The database backend technology. Each value determines:

- URL parsing behavior
- SQL dialect and syntax handling
- Available features (transactions, DDL, constraints)

Valid values: `sqlite`, `postgresql`, `mysql`, `mariadb`, `clickhouse`

### `database_url_sync` and `database_url_async`

Connection URL strings in the format:

```
[dialect+driver://user:password@]host[:port][/database][?options]
```

- **`database_url_sync`**  used by CLI commands (`migrate`, `init`, etc.) and any sync session
- **`database_url_async`**  used by async sessions (FastAPI); falls back to `database_url_sync` if omitted

At least one must be provided. If only `database_url_sync` is given, async sessions will use it (with async driver substitution like `postgresql://...`  `postgresql+asyncpg://...`).

Examples:

```python
# Both sync and async (recommended for FastAPI projects)
database_url_sync = "postgresql://user:password@localhost:5432/mydb"
database_url_async = "postgresql+asyncpg://user:password@localhost:5432/mydb"

# Sync only (CLI-only projects)
database_url_sync = "postgresql://user:password@localhost:5432/mydb"

# SQLite (relative path)
database_url_sync = "sqlite:///./development.db"

# MySQL
database_url_sync = "mysql://user:password@localhost:3306/mydb"

# ClickHouse
database_url_sync = "http://user:password@clickhouse-host:8123/mydb"
```

### `default`

When `True`, this database is selected when `--database` / `-d` is not specified.

**Rule:** Exactly one entry must have `default=True`.

**Example:**
```python
# Primary is default
primary = database_config(
analytics = database_config(database_name="analytics", ...)  # default=False implied
```

Exactly one database must have `default=True`. Having zero or multiple defaults will cause a validation error.

### `migrations_dir`

Path where this database's migration files are stored.

- Defaults to `migrations/<database_name>`
- Each database should have its own directory to avoid collision
- Versioned migration files go here (`NNNN_description.sql`)
- Repeatable migration files go here (`RA__*.sql`, `ROC__*.sql`)

### `model_paths`

A list of Python import paths where DBWarden should discover SQLAlchemy model definitions.

**When required:**
- **Single database:** Optional (DBWarden scans entire codebase)
- **Multiple databases:** Required for each database

**How it works:** DBWarden imports each path and inspects classes inheriting from `DeclarativeBase` or `declarative_base()`.

**Examples:**
```python
# Single module
model_paths=["app.models"]

# Multiple modules
model_paths=["app.models.primary", "app.legacy"]

# Nested modules
model_paths=["app.models.api.v1", "app.models.api.v2"]
```

Specifying `model_paths` makes discovery faster and more predictable, even for single-database projects.

See [Multi-Database Guide](../configuration/multi-database.md) for organizing models across databases.

### `model_tables`

A downstream filter applied after model discovery. Only tables whose string
name appears in this list are owned by this database. All other discovered
tables are ignored.

**When to use:**
- **Multi-database shared `model_paths`:** Two databases share the same
  import path but own different subsets of tables.
- **Selective deployment:** A microservice owns only a few tables from a
  shared models package.

**How it works:**
1. DBWarden discovers all models via `model_paths`
2. If `model_tables` is set, it validates every name exists among the
   discovered tables
3. Only the matching tables participate in migrations, diffs, and exports

**Example:**
```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
)

audit = database_config(
    database_name="audit",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/audit",
    model_paths=["app.models"],
    model_tables=["audit_logs"],
)
```

**Overlap validation:** If two databases both set `model_tables` with
overlapping names, DBWarden raises an error (same behavior as
`model_paths` overlap). Set `overlap_models=True` to allow it.

**Must be valid SQL identifiers.** Dotted (schema-qualified) names are not
supported in the initial release.

### `migration_table`

Name of the table DBWarden uses to record applied migrations and repeatable migration checksums.

- Defaults to `_dbwarden_migrations`
- Must be a valid SQL identifier
- Applies per database entry
- Only affects migration tracking metadata; lock tables are separate

**Example:**

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    migration_table="custom_migrations",
)
```

Use this when:

- integrating with an existing database that already reserves a migrations table name
- isolating DBWarden metadata under a project-specific convention

### `seed_table`

Name of the table DBWarden uses to record applied seeds.

- Defaults to `_dbwarden_seeds`
- Must be a valid SQL identifier
- Applies per database entry

**Example:**

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    seed_table="custom_seeds",
)
```

Use this when integrating with an existing database that already reserves the seed table name.

### `auto_apply_seeds`

When `True`, DBWarden automatically applies pending code seeds after each successful `migrate` run.

- Defaults to `False`
- Applies per database entry
- Can be overridden per-run with `--apply-seeds` / `--no-apply-seeds` CLI flags on `migrate`

**Example:**

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    auto_apply_seeds=True,
)
```

Use this when:

- you want seeds to stay in sync with schema changes without manual `seed apply` steps
- deploying code seeds that define reference data or lookup tables
- running in CI/CD where every migration cycle should also re-seed

### `dev_database_type` and `dev_database_url`

These define an alternate connection for local development workflows.

When `--dev` is passed to any DBWarden command:
- `database_type` is swapped to `dev_database_type`
- `database_url_sync` / `database_url_async` are swapped to `dev_database_url`

**Benefits:**
-  Use SQLite locally for speed (if production is PostgreSQL)
-  Target a separate development database instance
-  Test migrations safely before running against production
-  Each developer has isolated database
-  Easy to reset (just delete the file)

**Example:**
```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://prod-host/myapp",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",
)
```

Use with:
```bash
$ dbwarden --dev migrate  # Uses SQLite
$ dbwarden migrate        # Uses PostgreSQL
```

Use SQLite with `dev_database_url="sqlite:///./dev.db"` for the fastest local iteration loop.

See [Dev Mode](../configuration/dev-mode.md) for complete workflow and patterns.

### `overlap_models`

By default, DBWarden prevents model path overlap between databases.

Set `overlap_models=True` when:

- Two databases legitimately share model definitions
- You understand the behavior implications (both databases will include overlapping tables)

### `secure_values`

When enabled, CLI display commands show the original variable/expression for non-literal arguments instead of resolved values.

**Use when:**
- Your config uses environment variables or expressions for secrets
- You want terminal output to avoid exposing credentials
- Running commands in CI/CD with logged output

**Example:**
```python
import os

DATABASE_URL = os.getenv("DATABASE_URL")

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=DATABASE_URL,
    secure_values=True,  #  Enable secure display
)
```

**Without `secure_values`:**
```bash
$ dbwarden settings show
URL: postgresql://user:SECRET_PASSWORD@prod-host/myapp
```

**With `secure_values=True`:**
```bash
$ dbwarden settings show --all
URL: DATABASE_URL (expression)
```

Always set `secure_values=True` in production to prevent credential exposure in logs.

## Return value: `DatabaseHandle`

`database_config()` returns a `DatabaseHandle` object with two
properties designed as FastAPI dependency annotations:

| Property | Resolves to (SQL) | Resolves to (ClickHouse) |
|----------|-------------------|--------------------------|
| `.async_session` | `Annotated[AsyncSession, Depends(...)]` | `Annotated[AsyncClient, Depends(...)]` |
| `.sync_session` | `Annotated[Session, Depends(...)]` | `Annotated[Client, Depends(...)]` |

Use them as type hints in FastAPI route parameters. The handle serves as
a namespace so you never confuse which database a session belongs to:

```python
# dbwarden.py
from dbwarden import database_config

primary = database_config(database_name="primary", ...)
analytics = database_config(database_name="analytics", ...)
```

```python
# routes.py
from ..dbwarden import primary, analytics

@router.get("/users")
async def get_users(session: primary.async_session):
    return await session.execute(...)

@router.get("/reports")
def get_reports(session: analytics.sync_session):
    return session.execute(...)
```

Use `.async_session` for async route handlers and `.sync_session` for sync
handlers. The deprecated `.session` property (aliased to `.async_session`)
will be removed in a future version.

`DatabaseHandle` is still useful as a typed container even without FastAPI. Access
`handle._name` and `handle._db_type` for the raw config values.

## Configuration rules (enforced at load time)

DBWarden validates your config to prevent dangerous misconfigurations:

| Rule | Error message (if violated) |
|------|---------------------------|
| Exactly one `default=True` | `Exactly one default=True required` |
| Unique `database_name` across all entries | `Duplicate database_name` |
| Unique `database_url_sync` across all entries | `Duplicate database_url_sync` |
| Unique physical target (even across credentials) | `Duplicate database target detected` |
| Required `model_paths` when multiple databases | `model_paths is required when more than one database is configured` |
| Explicit `overlap_models` when paths overlap | `model_paths overlap detected` |
| `model_tables` (if set) must not overlap across databases | `model_tables overlap detected` |
| If `dev_database_type` set, `dev_database_url` also required | `dev_database_url is required when dev_database_type is set` |

## Loading and resolution

Config is loaded by importing your Python config source and executing `database_config(...)` calls.

The resolution priority is:

1. Look for `dbwarden.py` in the current directory or parent directories
2. If `DBWARDEN_CONFIG_MODULE` environment variable is set, use that module
3. Full scan for any file containing `database_config(...)` calls

If more than one discovery source is found, DBWarden fails with an ambiguity error.

`dbwarden.py` is the default convention, but it is not the only valid location. Any discovered Python file inside the project can call `database_config(...)`.

### Security sandbox

Config files are loaded with path traversal protection that ensures the file is within the project tree. See [Configuration Concepts  Config Loading Security](../configuration/concepts.md#config-loading-security-sandbox).

## Examples

### Minimal single-database setup

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/mydb",
)
```

### With local development (recommended)

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/mydb",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

### Multi-database setup

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    model_paths=["app.models.api"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://clickhouse:password@clickhouse-host:8123/analytics",
    model_paths=["app.models.analytics"],
)
```

## Quick Reference

| Parameter | Required? | Default | Use When |
|-----------|-----------|---------|----------|
| `database_name` |  Yes | - | Always |
| `database_type` |  No | `"sqlite"` | Non-SQLite backends |
| `database_url_sync` |  Conditional | `None` | CLI or sync sessions |
| `database_url_async` |  No | `None` | Async sessions (FastAPI) |
| `default` |  No | `False` | Mark one database as default |
| `migrations_dir` |  No | `migrations/<name>` | Custom migration directory |
| `seed_table` |  No | `_dbwarden_seeds` | Custom seed tracking table |
| `auto_apply_seeds` |  No | `False` | Auto-apply seeds after migrate |
| `migration_table` |  No | `_dbwarden_migrations` | Custom migration tracking table |
| `model_paths` |  Conditional | `None` | Multi-database or explicit discovery |
| `model_tables` |  No | `None` | Filter discovered tables by name |
| `dev_database_type` |  No | `None` | Local development |
| `dev_database_url` |  No | `None` | Local development |
| `overlap_models` |  No | `False` | Shared models (read replicas) |
| `secure_values` |  No | `False` | Hide credentials in output |

## Related Documentation

**Getting Started:**
- [Quick Start](../configuration/quick-start.md) - Your first configuration
- [Concepts](../configuration/concepts.md) - How configuration works

**Guides:**
- [Connection URLs](../configuration/connection-urls.md) - Database URL formats
- [Model Discovery](../configuration/model-discovery.md) - How `model_paths` works
- [Dev Mode](../configuration/dev-mode.md) - Local development
- [Multi-Database](../configuration/multi-database.md) - Multiple databases
- [Production Patterns](../configuration/production-patterns.md) - Real-world examples

**Help:**
- [Troubleshooting](../configuration/troubleshooting.md) - Common issues

========================================================================
PAGE: https://dbwarden.emiliano-go.com/reference/migrate-from-toml/
========================================================================

# Migrate from TOML

If your project currently uses `warden.toml` for DBWarden configuration, this guide walks through transitioning to the Python-based `database_config(...)` approach.

## Why migrate

The Python configuration model offers:

- **Type validation** - catches misconfigurations at import time, not runtime
- **Runtime flexibility** - use environment variables, conditional logic, and expressions in your config
- **IDE support** - autocomplete, type hints, and inline documentation
- **Consistency** - same Python codebase powers your app and your migrations

## Before: TOML configuration

```toml
default = "primary"

[database.primary]
database_type = "postgresql"
sqlalchemy_url = "postgresql://user:password@localhost:5432/main"
migrations_dir = "migrations/primary"
model_paths = ["app.models.api"]
```

## After: Python configuration

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    migrations_dir="migrations/primary",
    model_paths=["app.models.api"],
)
```

## What changed in field names

| TOML field | Python field | Notes |
|------------|--------------|-------|
| `default` | `default=True` | Boolean flag per entry instead of top-level |
| `database.<name>.database_type` | `database_type` | Passed directly to function |
| `database.<name>.sqlalchemy_url` | `database_url_sync` | Note: renamed for clarity |
| `database.<name>.migrations_dir` | `migrations_dir` | Same concept, different syntax |
| `database.<name>.model_paths` | `model_paths` | List syntax in Python |
| `database.<name>.dev_database_type` | `dev_database_type` | Optional dev swap entries |
| `database.<name>.dev_database_url` | `dev_database_url` | Optional dev swap entries |

## Migration checklist

### Step 1: Identify current databases

Check your existing `warden.toml`:

```bash
$ dbwarden database list
```

Note each database entry and its configuration.

### Step 2: Create new config source

Create your new config file (or update an existing config file that will contain `database_config(...)` calls).

Typical options:

- `dbwarden.py` (recommended for new projects)
- `app/core/config.py` (if you already have one)
- Any Python file that DBWarden can discover

### Step 3: Map each database entry

For each database in your TOML config, create a corresponding `database_config(...)` call.

**Example transformation:**

```python
# From TOML:
# [database.primary]
# database_type = "postgresql"
# sqlalchemy_url = "postgresql://user:password@localhost:5432/main"
# dev_database_type = "sqlite"
# dev_database_url = "sqlite:///./development.db"

# To Python:
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

### Step 4: Verify configuration loads

```bash
$ dbwarden settings show --all
```

You should see all your migrated databases listed with correct types and URLs.

### Step 5: Test core commands

Confirm the migration works by running key commands:

```bash
$ dbwarden status --database primary
$ dbwarden history --database primary
```

### Step 6: Remove TOML file

Once verified, delete your old `warden.toml`:

```bash
rm warden.toml
```

DBWarden now uses only your Python configuration source.

## Advanced migration patterns

### Using environment variables

```python
import os


DATABASE_URL = os.getenv("DATABASE_URL")


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=DATABASE_URL,
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

This replaces the old pattern of reading URL from environment via TOML (which TOML cannot do natively).

### Using conditional configuration

```python
import os


ENV = os.getenv("ENVIRONMENT", "development")


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db" if ENV == "development" else None,
)
```

This is impossible with TOML but natural with Python.

### Adding `secure_values` for sensitive URLs

If your configuration uses variables or expressions for credentials:

```python
import os


DB_USER = os.getenv("DB_USER")
DB_PASS = os.getenv("DB_PASS")


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=f"postgresql://{DB_USER}:{DB_PASS}@localhost:5432/main",
    secure_values=True,
)
```

With `secure_values=True`, display commands show the expression rather than resolved credentials.

## Common issues during migration

### "Exactly one default=True required"

Ensure exactly one entry has `default=True`. Unlike TOML's top-level `default` field, you must set this on exactly one entry.

### "model_paths is required when more than one database is configured"

When using multiple databases, each needs explicit `model_paths` to keep model discovery boundaries clear.

### "Duplicate database_name" or "Duplicate database_url"

Python loads all `database_config(...)` calls, so ensure each call has a unique `database_name` and no duplicate URLs.

### TOML-specific features not supported

Some TOML features don't map 1:1 to Python:

- Inline tables → use separate key-value dictionary in Python (more verbose but explicit)
- Multiline strings → use triple-quoted strings in Python
- Arrays of tables → use separate `model_paths=[...]` list entries

If you used advanced TOML features, manually translate those to equivalent Python constructs.

## Verification commands after migration

After completing the migration, run these to confirm everything works:

```bash
# Confirm all databases visible
$ dbwarden settings show --all

# Confirm status works
$ dbwarden status --database <name>

# Confirm history works
$ dbwarden history --database <name>

# If using --dev, confirm it works
$ dbwarden --dev status --database <name>
```

## Rollback if needed

If migration causes issues, you can always:

1. Recreate `warden.toml` with the original configuration
2. Run DBWarden version that supports TOML (pre-0.5)

However, the Python-based approach is the recommended direction (0.9+) and offers significant benefits.

========================================================================
PAGE: https://dbwarden.emiliano-go.com/seeds/
========================================================================

# Seed Management

**Requires the `dbwarden-seeds` plugin:** `dbwarden plugin add dbwarden-seeds`. Seed management was previously built into core and now ships as an official plugin. The `dbwarden seed` commands still exist in core, but without the plugin installed each one exits with a message telling you to install it.

DBWarden provides seed data management for populating databases with initial or reference data. Seeds complement migrations by handling data that belongs in version control.

## Overview

There are two ways to define seeds, listed in order of preference:

1. **Code seeds** (recommended): define seeds inline alongside your SQLAlchemy models using the `Seed` base class or `@seed_data` decorator. No separate files, no manual versioning.
2. **File seeds**: traditional `.sql` or `.py` files in a `seeds/` directory, useful for complex multi-statement SQL.

Both are tracked in the `_dbwarden_seeds` table and applied via `dbwarden seed apply`.

---

## Code Seeds (Recommended)

Code seeds live alongside your models in your `model_paths` directories. They are the recommended way to define seed data because they stay in sync with your schema, support IDE autocompletion, and do not require manual version management.

### Seed Base Class

Inherit from `Seed` and set a `model` + `rows`:

```python
from dbwarden.seed import Seed

class CountrySeed(Seed):
    __seed_database__ = "primary"
    __seed_description__ = "initial countries"
    __seed_on_conflict__ = "update"
    __seed_conflict_columns__ = ["code"]

    model = Country
    rows = [
        Country(code="UY", name="Uruguay"),
        Country(code="AR", name="Argentina"),
    ]
```

Key advantages over the old decorator approach:

- **Full IDE autocompletion**: `rows` uses model instances directly, so your editor knows the column names and types
- **No `version` parameter**: versions are auto-assigned (`C0001`, `C0002`, ...) based on deterministic class ordering
- **No manual import of `SeedRow`**: though `SeedRow` is still available if you prefer dict-like rows

### Seed Class Reference

| Attribute | Default | Description |
|-----------|---------|-------------|
| `__seed_database__` | `"default"` | Routes the seed to the named database handle configured in `database_config(...)`. |
| `__seed_description__` | `""` | Human-readable label shown in `dbwarden seed list` output. |
| `__seed_on_conflict__` | `"ignore"` | What to do when a row with matching columns exists: `"ignore"` (skip silently), `"update"` (overwrite), or `"error"` (raise). |
| `__seed_conflict_columns__` | `None` | List of column names used for conflict detection. Required when `__seed_on_conflict__` is `"update"`. |

### Model Instances in Rows

Because `rows` accepts model instances, you get full autocompletion from your `Mapped` annotations:

```python
from dbwarden.seed import Seed

class RepoSeed(Seed):
    __seed_database__ = "clickhouse"
    __seed_description__ = "Tracked Repos"

    model = Repo
    rows = [
        Repo(name="dbwarden", owner="anomalyco", is_org=True, default_branch="main"),
        Repo(name="vigil", owner="anomalyco", is_org=True, default_branch="master"),
    ]
```

Your editor will suggest `name`, `owner`, `is_org`, `default_branch` etc. as you type.

> SQLAlchemy 2.0's `DeclarativeBase` does not accept positional arguments in the constructor. Always use keyword arguments when instantiating models in `rows`: `Repo(name="dbwarden", ...)` instead of `Repo("dbwarden", ...)`.

### SeedRow (Alternative)

If you prefer dict-like rows, `SeedRow` still works:

```python
from dbwarden.seed import Seed, SeedRow

class CountrySeed(Seed):
    __seed_database__ = "primary"
    __seed_description__ = "initial countries"
    __seed_on_conflict__ = "update"
    __seed_conflict_columns__ = ["code"]

    model = Country
    rows = [
        SeedRow(code="UY", name="Uruguay"),
        SeedRow(code="AR", name="Argentina"),
    ]
```

### `on_conflict` Behavior

| Value | Behavior |
|-------|----------|
| `"ignore"` (default) | Skips existing rows silently |
| `"update"` | Updates existing rows with new values |
| `"error"` | Raises an error on conflict |

### PostgreSQL Schema Resolution

When a model uses `pg_schema` in its Meta (via `PGTableMeta` or `PGViewMeta`), code seeds automatically qualify the table name with that schema:

```python
from dbwarden.databases.pgsql import PGTableMeta

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)

    class Meta(PGTableMeta):
        pg_schema = "app"

class UserSeed(Seed):
    __seed_database__ = "primary"
    __seed_description__ = "initial users"

    model = User
    rows = [User(email="alice@example.com", name="Alice")]
```

The generated INSERT becomes:

```sql
INSERT INTO app.users (email, name) VALUES ('alice@example.com', 'Alice')
```

The schema is resolved in this order: `Meta.pg_schema`, then `Meta.backend_table.schema`, then `__table__.schema`. The seed tracking table (default `_dbwarden_seeds`) stays in the schema set by the connection's `search_path` (config-level `pg_schema`).

### Logic-Based Seeds

Define a `generate(session)` static/class method for programmatic data:

```python
class PermissionSeed(Seed):
    __seed_database__ = "primary"
    __seed_description__ = "load permissions"
    __seed_on_conflict__ = "ignore"

    model = Permission

    @staticmethod
    def generate(session):
        for resource in ["users", "orders"]:
            for action in ["read", "write", "delete"]:
                session.add(Permission(name=f"{resource}:{action}"))
```

### `@seed_data` Decorator (Deprecated)

The old decorator still works but is deprecated in favour of the `Seed` base class:

```python
from dbwarden.seed import seed_data, SeedRow

@seed_data(
    database="primary",
    description="initial countries",
    on_conflict="update",
    conflict_columns=["code"],
)
class CountrySeed:
    model = Country
    rows = [SeedRow(code="UY", name="Uruguay")]
```

Note that `version` is **no longer required**; it is auto-assigned.

### Discovery and Ordering

Code seeds are discovered through the same `model_paths` scan as models. They use auto-assigned versions in the `C` namespace (`C0001`, `C0002`, ...) and are sorted deterministically by module and class name. Pending detection compares the class qualified name against the `_dbwarden_seeds` tracking table.

---

## File Seeds (Traditional)

File seeds live in a `seeds/` directory and are useful for complex multi-statement SQL or when you need to hand-craft seed files.

### Directory Structure

```
seeds/
  V0001__seed_initial_users.sql
  V0002__seed_lookup_tables.sql
  V0003__seed_sample_data.py
```

Each file follows the naming convention:

```
V<4-digit-version>__<description>.<sql|py>
```

### Creating File Seeds

```bash
$ dbwarden seed create "seed initial users" --database primary
```

Creates a file like `seeds/V0001__seed_initial_users.sql`:

```sql
-- INSERT statements go here
```

### Python File Seeds

```bash
$ dbwarden seed create "generate sample data" --database primary --type python
```

Creates `seeds/V0001__generate_sample_data.py` with a `seed(connection, session)` function.

The `seed()` function receives both a raw SQLAlchemy `Connection` and an ORM `Session` bound to the same transaction:

```python
# Using raw connection
def seed(connection, session):
    for i in range(100):
        connection.execute(
            "INSERT INTO users (name) VALUES (:name)",
            {"name": f"user_{i}"},
        )

# Using ORM session
def seed(connection, session):
    for i in range(100):
        session.add(User(name=f"user_{i}"))
    session.flush()
```

---

## Applying Seeds

Apply all pending seeds (file + code seeds are both discovered):

```bash
$ dbwarden seed apply --database primary
```

Apply a specific version:

```bash
$ dbwarden seed apply --database primary --version 0003
```

Apply to all databases:

```bash
$ dbwarden seed apply --all
```

### Dry Run

Preview what would be applied without executing:

```bash
$ dbwarden seed apply --database primary --dry-run
```

### Auto-Apply After Migrations

Configure seeds to be applied automatically after each `dbwarden migrate`:

```python
database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    model_paths=["models"],
    auto_apply_seeds=True,
)
```

Or apply seeds once after a migration without changing config:

```bash
$ dbwarden migrate --apply-seeds
```

---

## Listing Seeds

```bash
$ dbwarden seed list --database primary
```

Output:

```
Seeds for database 'primary':
  V0001  seed_initial_users                   applied  2025-06-01 10:00:00
  C0001  initial countries                    pending   (code seed)
```

List across all databases:

```bash
$ dbwarden seed list --all
```

### Pruning Orphaned Records

Remove tracking records for seed files that no longer exist on disk:

```bash
$ dbwarden seed list --prune
```

---

## Rolling Back Seeds

Rollback removes the applied tracking record, allowing the seed to be re-applied. It does **not** reverse data changes.

```bash
# Rollback the most recent seed
$ dbwarden seed rollback --database primary

# Rollback a specific number
$ dbwarden seed rollback --database primary --count 2

# Rollback to a specific version
$ dbwarden seed rollback --database primary --to-version 0002
```

---

## Seed Tracking Table

DBWarden tracks applied seeds in `_dbwarden_seeds` (configurable via `seed_table`):

| Column | Description |
|--------|-------------|
| `version` | 4-digit seed version (`V0001`) or code seed ID (`C0001`) |
| `description` | Human-readable description |
| `filename` | File path or code seed identifier |
| `seed_type` | `sql`, `python`, or `code` |
| `checksum` | SHA-256 hash of file/class source |
| `applied_at` | Timestamp of application |

The tracking table is created automatically on first seed apply. Each version can only be applied once until rolled back.

### Checksum Drift

When a seed file has been modified since it was last applied, DBWarden emits a warning:

```
Warning: Seed V0001 has been modified since last apply (checksum mismatch).
```

This helps detect accidental changes to already-applied seeds.

---

## Exporting Seeds for Production

Code seeds require your full application environment to execute. For Dockerized deployments where you don't want to copy the application code into a container just to seed data, use `dbwarden seed export` to produce stateless ROC (runs-on-change) SQL files.

```bash
$ dbwarden seed export --database clickhouse
```
This writes `seeds/ROC__clickhouse__code_seeds.sql` containing `INSERT ... ON CONFLICT` statements rendered in the target database dialect. In production, apply with:
```bash
$ dbwarden seed apply --database clickhouse
```

Because the file is ROC, updating the code seed and re-exporting produces a new content checksum, which triggers re-application. The `ON CONFLICT DO UPDATE` clause handles updating existing rows; no need to delete and recreate.

**Non-handled problems:**

- Rows removed from a code seed are not automatically deleted in the target database
- Logic seeds that depend on other logic seeds' output are not supported (preceding row-based seeds are pre-loaded, but logic-to-logic ordering is not)
- Non-deterministic `generate()` methods (e.g. using `datetime.now()`) produce a new checksum every export, causing re-apply on every deploy: acceptable for idempotent upserts, wasteful for pure inserts. Use deterministic `generate()` where possible

**Dialect requirement:** Exporting requires the same dialect packages as connecting to that database. For ClickHouse, install `clickhouse-sqlalchemy`. Missing packages produce a clear error at export time.

## Seeds and Migrations

Seeds are independent from migrations. You can:

- Apply migrations without seeds
- Apply seeds without migrations
- Mix both in your workflow

The `dbwarden status` command and the FastAPI `GET /status` endpoint report both pending migrations and pending seeds.

---

## Seeds in FastAPI

The `DBWardenRouter` includes seed status in its `GET /status` response:

```json
{
  "databases": {
    "primary": {
      "status": "ok",
      "connected": true,
      "pending_migrations": 0,
      "applied_migrations": 5,
      "pending_seeds": 2,
      "applied_seeds": 1,
      "lock_active": false,
      "error": null
    }
  }
}
```

FastAPI integration ships separately as the `dbwarden-fastapi` plugin, so its reference lives with the plugin: [dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi).

See also: [Cookbook: Seeds](../cookbook/07-seeds.md)

========================================================================
PAGE: https://dbwarden.emiliano-go.com/sql-translation/
========================================================================

# SQL Translation

DBWarden includes a SQL translation layer to support development workflows where your primary database differs from your development database.

The most common case is:

- Primary database: PostgreSQL/MySQL/MariaDB/ClickHouse
- Development database: SQLite (`--dev` mode)

This keeps local development fast while still allowing production-targeted schemas.

## Why SQL Translation Exists

SQLite does not support all backend-specific SQL types and default expressions used by other databases.

Without translation, generated migrations can fail in local development when they contain backend-specific types like `UUID`, `JSONB`, or default expressions like `now()`.

DBWarden translation solves this by adapting generated SQL for SQLite compatibility.

## When translation is active

Translation is applied when all are true:

- command runs with `--dev`
- selected database resolves to a SQLite `dev_database_url`
- command path generates SQL from models (`make-migrations`)

It is not a runtime SQL proxy for arbitrary manual SQL.

## How It Works

When you run commands in development mode and target a SQLite dev database:

```bash
$ dbwarden --dev make-migrations "sync models" -d primary
```

DBWarden uses this flow:

1. Loads the selected database config and resolves `dev_database_url`.
2. Detects that the active target backend is SQLite.
3. Extracts model metadata from SQLAlchemy models.
4. Translates backend-specific types/defaults to SQLite-compatible SQL.
5. Generates migration SQL with translated definitions.

Translation is applied during migration generation, not as a post-processing regex pass.

## Type conversion behavior

Common conversions:

| Source type | SQLite output |
|-------------|---------------|
| `UUID` | `TEXT` |
| `JSON` / `JSONB` | `TEXT` |
| `TIMESTAMPTZ` | `DATETIME` |
| `SERIAL` / `BIGSERIAL` | `INTEGER` |
| ClickHouse nullable numeric forms | `INTEGER`/`REAL` depending on source |

If a type cannot be translated safely:

- non-strict mode: fallback to `TEXT` + warning
- strict mode: fail migration generation

## Default expression handling

Backend expressions such as `now()` or `gen_random_uuid()` may not have direct SQLite equivalents.

In non-strict mode, unsupported defaults are dropped with warning.

In strict mode, unsupported defaults fail generation.

## Strict Translation Mode

If you want hard failures instead of fallback behavior:

```bash
$ dbwarden --dev --strict-translation make-migrations "sync models" -d primary
```

In strict mode:

- Unknown/unsupported type conversions raise errors
- Unsupported default expression conversions raise errors

Use this when you want to catch every lossy conversion early.

## Recommended team workflow

1. iterate quickly with `--dev` (SQLite)
2. keep strict checks in CI (`--strict-translation`)
3. validate release candidate migrations against production-like database

This balances speed and correctness.

## Troubleshooting

`--dev mode is enabled, but database '<name>' has no dev_database_url configured`:

- add `dev_database_url` for that database entry

Unexpected type fallback to `TEXT`:

- inspect model type for backend-specific declaration
- re-run with `--strict-translation` to fail fast and fix explicitly

Generated SQL differs from production expectations:

- expected in SQLite compatibility mode; validate final release migrations on production-like backend

## Notes and Limitations

- Translation focuses on compatibility for local development.
- Some backend features cannot be represented exactly in SQLite.
- For production accuracy, always test migrations against your production-like database too.
