# dbwarden Documentation
> Full documentation for dbwarden - a SQL-first database migration system
> Source: https://docs.dbwarden.org
> Pages: 150

========================================================================
PAGE: https://docs.dbwarden.org/advanced/checksum-integrity/
========================================================================

# Checksum Integrity

dbwarden stores a SHA-256 checksum of each migration file at apply time. On subsequent runs, it recalculates the checksum and compares. A mismatch means the file changed after it was applied.

## What checksums protect against

- Accidentally editing an applied migration file
- Copy-paste errors that silently modify historical SQL
- Merge conflicts that land inside already-applied migration files
- Tooling (formatters, editors) modifying migration files in place

## What triggers a mismatch error

```
Migration '0004_add_indexes' has been hand-edited (checksum mismatch)
```

This warning is logged when a migration file's checksum doesn't match the stored checksum. dbwarden will skip the migration if it has already been applied (based on the stored checksum).

## Repeatable migrations

Migrations prefixed `RA__` (repeatable, always) or `ROC__` (repeatable on change) behave differently: they are designed to be re-applied. Checksum changes on `ROC__` files trigger re-application on the next `migrate` run. This is expected behavior, not an error.

Checksum mismatch errors only apply to versioned migrations (`V__` prefix).

## Diagnosing a mismatch

```bash
# See which migrations are applied and their checksums
$ dbwarden history --database primary

# Check the current status
$ dbwarden status --database primary
```

Common causes:

1. **Editor auto-format**: your editor reformatted whitespace in the file
2. **Merge conflict**: conflict markers were added/removed inside a migration file
3. **Intentional edit**: someone changed the migration to fix a typo or add a comment

## Resolution: dev environment

In development where the migration has not been applied to shared data:

1. Reset the database state and re-run migrations from scratch, **or**
2. If the change is trivial (whitespace, comment), revert the file to its original state:

```bash
git diff migrations/primary/V__0004_add_indexes.sql
git checkout migrations/primary/V__0004_add_indexes.sql
```

After reverting, the stored checksum and file checksum will match again.

## Resolution: production environment

In production, never modify applied migration files. The resolution is:

1. **Revert the file** to its exact applied state (use git history)
2. Create a **new migration** for any schema changes you need to make

If the file change was accidental and the schema is correct, reverting the file is safe; no data or schema change occurs.

If the file was intentionally changed to fix an error in a migration that was already applied in production, the database schema may already reflect the original (wrong) SQL. Coordinate carefully:

```bash
# 1. Revert the migration file to what was actually applied
git checkout <commit-before-edit> -- migrations/primary/V__0004_add_indexes.sql

# 2. Verify status is clean
$ dbwarden status --database primary

# 3. Create a corrective migration for the actual schema fix
$ dbwarden new "fix index on users" --database primary
```

## When is it safe to ignore?

Never. A checksum mismatch means recorded history diverges from what is on disk. Even if the change appears harmless, proceeding without resolving the mismatch means your migration history is untrustworthy.

## Schema snapshot checksums

dbwarden also writes a **schema snapshot** after each migration:
a JSON file at `.dbwarden/schemas/<db_name>__<migration_id>.schema.json`. Each
snapshot contains a `checksum` field computed from the full snapshot
content via SHA-256:

```json
{
  "tables": { ... },
  "checksum": "abc123..."
}
```

Snapshots are written atomically (write to temp file, verify, rename)
and read with integrity validation; if the file content doesn't match
the stored checksum, the snapshot is rejected.

If a snapshot fails checksum validation, `find_latest_snapshot()` returns
`None` and `make-migrations` falls back to diffing against the live database.

## Preventing mismatches

- Treat versioned migration files as immutable once applied to any shared environment
- Configure your editor to exclude `migrations/` from auto-format
- Use a pre-commit hook to detect changes to applied migration files:

```bash
# .git/hooks/pre-commit (example, adapt to your setup)
$ dbwarden check --database primary
```

`dbwarden check` compares models to the live schema. While not a direct checksum check, it surfaces drift that often accompanies unintended migration file edits.

See also: [Migration Locking](migration-locking.md) | [Migration Files](../migration-files.md) | [Cookbook: Schema Inspection](../cookbook/05-schema-inspection.md)

========================================================================
PAGE: https://docs.dbwarden.org/advanced/ci-cd-patterns/
========================================================================

# CI/CD Patterns

Patterns for running dbwarden migrations in automated pipelines.

## Core principle

Run migrations from exactly one job. Serialize migration and deploy. Never run `migrate` in parallel across multiple agents or containers targeting the same database.

## GitHub Actions

### Minimal migration job

```yaml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  migrate:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - run: uv add -e ".[migrations]"

      - name: Check migration status
        run: dbwarden status --database primary
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - name: Apply migrations
        run: dbwarden migrate --database primary
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - name: Verify post-migration status
        run: dbwarden status --database primary
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

  deploy:
    needs: migrate
    runs-on: ubuntu-latest
    steps:
      - name: Deploy application
        run: ...
```

The `needs: migrate` dependency ensures migrations are fully applied before the application starts.

### Preventing concurrent migration runs

```yaml
concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: false
```

`cancel-in-progress: false` queues duplicate runs instead of cancelling mid-flight, which avoids leaving a stale lock on the database.

### Multi-database migration

```yaml
- name: Apply all migrations
  run: dbwarden migrate --all
  env:
    PRIMARY_DATABASE_URL: ${{ secrets.PRIMARY_DATABASE_URL }}
    ANALYTICS_DATABASE_URL: ${{ secrets.ANALYTICS_DATABASE_URL }}
```

Or migrate databases sequentially to control order:

```yaml
- name: Migrate primary
  run: dbwarden migrate --database primary
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}

- name: Migrate analytics
  run: dbwarden migrate --database analytics
  env:
    ANALYTICS_DATABASE_URL: ${{ secrets.ANALYTICS_DATABASE_URL }}
```

### With backup before migration

```yaml
- name: Apply migrations with backup
  run: |
    dbwarden migrate --database primary \
      --with-backup \
      --backup-dir ./migration-backups
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}

- name: Upload backup artifact
  uses: actions/upload-artifact@v4
  with:
    name: migration-backup-${{ github.sha }}
    path: ./migration-backups/
    retention-days: 30
```

## GitLab CI

```yaml
stages:
  - migrate
  - deploy

migrate:
  stage: migrate
  image: python:3.12
  script:
    - uv add -e ".[migrations]"
    - dbwarden status --database primary
    - dbwarden migrate --database primary
    - dbwarden status --database primary
  variables:
    DATABASE_URL: $DATABASE_URL  # set in GitLab CI/CD settings as masked variable
  resource_group: production-database  # prevents concurrent runs

deploy:
  stage: deploy
  needs: [migrate]
  script:
    - ...
```

`resource_group` serializes the migrate job across concurrent pipelines.

## Sandbox testing in PR pipelines

Instead of running against a shared staging database, use `--sandbox`
to apply migrations to a temporary in-memory SQLite database or a
Docker-backed instance. This isolates PR checks from each other:

```yaml
sandbox-check:
  runs-on: ubuntu-latest
  if: github.event_name == 'pull_request'
  steps:
    - uses: actions/checkout@v4
    - run: uv add -e ".[migrations,testcontainers]"
    - name: Apply migrations to sandbox
      run: dbwarden migrate --sandbox --database primary
```

The sandbox starts a fresh database, applies all pending migrations,
reports results, and tears down. It never touches the real database.

## Dry-run check in PR pipelines

Use `--dry-run` to preview SQL without any database access:

```yaml
migration-check:
  runs-on: ubuntu-latest
  if: github.event_name == 'pull_request'
  steps:
    - uses: actions/checkout@v4
    - run: uv add -e ".[migrations]"
    - name: Check for pending migrations
      run: dbwarden status --database primary
      env:
        DATABASE_URL: ${{ secrets.STAGING_DATABASE_URL }}
```

This surfaces "pending migrations exist" warnings in PR checks without
modifying the database.

For a deeper check that validates the SQL actually runs, chain
`--dry-run` before `--sandbox`:

```yaml
- name: Preview SQL
  run: dbwarden migrate --dry-run --database primary

- name: Validate in sandbox
  run: dbwarden migrate --sandbox --database primary
```

## Plan output in deploy pipelines

The `make-migrations --plan` flag prints the generated migration plan
as JSON without writing files. Use it in deploy pipelines to capture
what would be generated as a deploy artifact:

```yaml
- name: Generate migration plan
  run: dbwarden make-migrations --database primary --plan > plan.json

- name: Upload plan artifact
  uses: actions/upload-artifact@v4
  with:
    name: migration-plan-${{ github.sha }}
    path: plan.json
```

The plan JSON includes detected changes, operation counts, and
auto-generated migration names.

## Exit codes

dbwarden exits non-zero on:

- Migration failure
- Checksum mismatch
- Lock acquisition failure
- Configuration error

CI pipelines treat non-zero as job failure by default. No extra configuration needed.

## Recommendations

- Store `DATABASE_URL` as an encrypted secret, not a plain environment variable
- Archive migration output logs as artifacts for audit trails
- Use `dbwarden history` output as a post-migration artifact
- Run `dbwarden status` before and after `migrate`; before confirms what will run, after confirms nothing is pending

See also: [Safe Deployment](safe-deployment.md) | [Credentials and Secrets](../configuration/credentials.md) | [Migration Locking](migration-locking.md) | [Cookbook: Offline & CI](../cookbook/04-offline-ci.md)

========================================================================
PAGE: https://docs.dbwarden.org/advanced/clickhouse-locking/
========================================================================

# ClickHouse Migration Locking

dbwarden provides production-grade migration locking for ClickHouse. This guide explains why ClickHouse needs custom locking, what the coordination profiles are, and how to set up your deployment.

See [Locking Architecture](../databases/locking.md) for the general locking model across all backends.

## Why ClickHouse needs custom locking

Every other database backend (PostgreSQL, MySQL, SQLite) provides a native lock primitive that dbwarden can use directly:

- PostgreSQL has `pg_advisory_lock()` (session-scoped, auto-releases on crash)
- MySQL has `GET_LOCK()` (session-scoped, auto-releases on crash)
- SQLite has `BEGIN IMMEDIATE` (write lock, released on crash via journal cleanup)

ClickHouse has none of these. Specifically, ClickHouse is missing:

1. **No session-scoped user locks.** There is no `LOCK TABLE` or advisory lock equivalent. You cannot hold a lock for the duration of a session.
2. **No synchronous compare-and-swap on table rows.** The `INSERT ... WHERE NOT EXISTS` pattern is not atomic in the way PostgreSQL's `INSERT ... ON CONFLICT` is.
3. **Non-transactional DDL.** DDL statements (CREATE, ALTER, DROP) cannot be rolled back. If a migration fails midway, there is no automatic undo.

Without native locks, two concurrent `dbwarden migrate` processes could both acquire the "lock" simultaneously, execute conflicting DDL, and corrupt the schema. dbwarden must therefore emulate locking using what ClickHouse does provide: table rows, ZooKeeper, and external infrastructure.

The coordination profiles (CH-0 through CH-4) represent progressively stronger guarantees, each addressing a specific weakness of the previous profile. They are not arbitrary levels; each one exists because the previous level has a concrete failure mode that cannot be solved within ClickHouse alone.

## Coordination profiles

| Profile | Description | Grade | Use case | What it fixes |
|---------|-------------|-------|----------|---------------|
| **CH-0** | Lease row with fencing token | C | Non-production, human-gated | Baseline: prevents concurrent migration via lease table |
| **CH-1** | CH-0 + strict idempotency | C | Production with idempotent migrations | Prevents data corruption from non-idempotent DDL |
| **CH-2** | CH-1 + Keeper-backed lock | C | Replicated with clickhouse-keeper | Better liveness: crash releases lock immediately via znode expiry |
| **CH-3** | Migration proxy choke point | A- | Production with migration proxy | Eliminates stale-lock window: single migration process |
| **CH-4** | Singleton executor | A* | Kubernetes with dedicated executor | Strongest guarantee: no DDL credentials on workers |

---

## CH-0: Lease with fencing token

The baseline profile. dbwarden writes a lease row to `dbwarden_lock` with a fencing token (monotonically increasing integer). A second process cannot acquire the lease while the first is active.

**Why it is grade C:** The fence check and DDL statement are not atomic. A paused worker (GC pause, SIGSTOP, CPU starvation) can lose its lease, another worker can acquire it, and the paused worker resumes and executes DDL against a database it no longer owns. The fencing token detects this, but only after the fact.

**Use when:** Development, staging, or any environment where a human is the only one triggering migrations and you can manually verify no concurrent runs are happening.

### CH-0 prerequisites

- ClickHouse server (any recent version)
- No additional infrastructure required

### CH-0 configuration

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    default = True
    database_type = "clickhouse"
    database_url_sync = "clickhouse://default:@localhost:8123/analytics"
    model_paths = ["app/analytics_models"]
    # CH-0 is the default; no coordination profile setting needed
```

Or with `database_config()`:

```python
from dbwarden import database_config

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://default:@localhost:8123/analytics",
    model_paths=["app/analytics_models"],
)
```

### How the lease works

When you run `dbwarden migrate`, dbwarden:

1. Attempts to insert a lease row into `dbwarden_lock` with a unique `execution_id`, a monotonically increasing `fencing_token`, and metadata
2. The insert uses an atomic INSERT...SELECT...WHERE NOT EXISTS pattern with FINAL to ensure only one lease is active
3. A background heartbeat updates `last_heartbeat_at` every 15 seconds
4. On completion, the lease row is marked as COMPLETE

```sql
INSERT INTO dbwarden_lock
    (namespace, execution_id, owner_id, migration_version, migration_checksum,
     fencing_token, host, pid, state, acquired_at, last_heartbeat_at, expires_at)
SELECT
    :namespace, :execution_id, :owner_id, :migration_version, :migration_checksum,
    ifNull(max(fencing_token), 0) + 1, :host, :pid, 'RUNNING',
    :acquired_at, :last_heartbeat_at, :expires_at
FROM dbwarden_lock FINAL
WHERE namespace = :namespace
  AND (expires_at IS NULL OR expires_at <= now())
LIMIT 1
```

The `expires_at` column is populated with the lease TTL (default 120 seconds) and is used to detect expired leases.

### CH-0 verification

```bash
# Run a migration
dbwarden migrate -d analytics

# Check lock status (should show INACTIVE after migration completes)
dbwarden lock-status -d analytics

# Inspect the lease table directly
clickhouse-client --query "SELECT * FROM analytics.dbwarden_lock"
```

### CH-0 limitations

- If the process crashes after the lease expires but before cleanup, another process must wait for the TTL (default 120 seconds) before acquiring the lock
- A paused worker can lose its lease and resume DDL execution against a database it no longer owns (the fencing token detects this after the fact)
- No automatic recovery from partial migration failure

---

## CH-1: Idempotency enforcement

CH-1 adds strict idempotency checking on top of CH-0. Non-idempotent statements (`RENAME`, `MODIFY COLUMN`, `UPDATE`, `DELETE`) are refused by default.

**Why it exists:** Even with CH-0's lease, a stale worker can execute non-idempotent DDL after losing its lease. Idempotent DDL (`CREATE TABLE IF NOT EXISTS`, `DROP TABLE IF EXISTS`) is safe to re-execute; non-idempotent DDL is not. By refusing non-idempotent statements, CH-1 limits the blast radius of the stale-worker window to idempotent operations only.

!!! note "Concurrent ALTERs on ReplicatedMergeTree"
    Even with idempotent DDL, multiple separate `ALTER TABLE` statements against the same ReplicatedMergeTree table can trigger [CANNOT_ASSIGN_ALTER (Code 517)](../databases/clickhouse/alter-semantics.md#cannot_assign_alter-code-517). See [ALTER semantics](../databases/clickhouse/alter-semantics.md) for batching requirements.

**Use when:** Production environments where all migrations are idempotent (dbwarden generates idempotent DDL by default).

### CH-1 prerequisites

- Everything in CH-0
- All migrations must use idempotent forms (`IF NOT EXISTS`, `IF EXISTS`)

### CH-1 configuration

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    default = True
    database_type = "clickhouse"
    database_url_sync = "clickhouse://default:@localhost:8123/analytics"
    model_paths = ["app/analytics_models"]
```

Or with `database_config()`:

```python
from dbwarden import database_config

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://default:@localhost:8123/analytics",
    model_paths=["app/analytics_models"],
)
```

### Idempotent vs non-idempotent DDL

**Idempotent** (allowed under CH-1):

- `CREATE TABLE/VIEW/DICTIONARY IF NOT EXISTS`
- `DROP TABLE/VIEW/DICTIONARY IF EXISTS`
- `ALTER TABLE ADD COLUMN IF NOT EXISTS`
- `ALTER TABLE DROP COLUMN IF EXISTS`
- Metadata-only alters (COMMENT, TTL)

**Non-idempotent** (refused under CH-1):

- `RENAME` (no idempotent form)
- `MODIFY COLUMN` (rewrites data)
- `UPDATE` / `DELETE` (data mutations)

### CH-1 idempotency checklist

Before enabling CH-1 in production:

1. **Audit existing migrations**:

   ```bash
   grep -E "RENAME|MODIFY COLUMN|UPDATE|DELETE" migrations/analytics/*.sql
   ```

2. **Test idempotency**:

   ```bash
   dbwarden migrate -d analytics
   dbwarden migrate -d analytics  # Should be no-op
   ```

3. **Review generated migrations**:

   - All CREATE statements should use `IF NOT EXISTS`
   - All DROP statements should use `IF EXISTS`
   - All ALTER ADD/DROP COLUMN should use `IF NOT EXISTS`/`IF EXISTS`

### CH-1 verification

```bash
# Run a migration
dbwarden migrate -d analytics

# Run again (should be a no-op with CH-1 enforcement)
dbwarden migrate -d analytics

# Check lock status
dbwarden lock-status -d analytics
```

### Overriding CH-1 for non-idempotent migrations

If you must run a non-idempotent migration (e.g., a one-time RENAME), restructure it to use an idempotent form:

```sql
-- Instead of:
ALTER TABLE events RENAME TO events_v2;

-- Use:
ALTER TABLE IF EXISTS events RENAME TO events_v2;
```

Or drop to CH-0 temporarily by removing the idempotency check in your deployment.

---

## CH-2: Keeper-backed lock

For replicated deployments with clickhouse-keeper (ZooKeeper-compatible), CH-2 replaces the lease table with ephemeral-sequential znodes.

**Why it exists:** CH-0/CH-1 lease expiry depends on TTL, which means there is always a window between "worker died" and "lease expired" where the database is locked but no work is happening. With ZooKeeper ephemeral znodes, the lock is released the instant the ZooKeeper session ends (typically within seconds of a crash), eliminating the stale-lock window for liveness purposes.

**Use when:** Replicated ClickHouse deployments with clickhouse-keeper or ZooKeeper available.

### CH-2 prerequisites

- ClickHouse cluster with clickhouse-keeper or ZooKeeper
- clickhouse-keeper must be reachable from all cluster nodes
- Default keeper path: `/dbwarden/locks`

### CH-2 infrastructure setup

**1. Deploy clickhouse-keeper (if not already running):**

clickhouse-keeper ships with ClickHouse. A minimal 3-node keeper cluster for quorum:

```xml
<!-- /etc/clickhouse-server/config.d/keeper.xml -->
<clickhouse>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <server_id>1</server_id>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>keeper1</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>keeper2</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>keeper3</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>
```

**2. Verify keeper is running:**

```bash
clickhouse-client --host keeper1 --port 9181 --query "SELECT * FROM system.zookeeper WHERE path = '/'"
```

### CH-2 configuration

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    default = True
    database_type = "clickhouse"
    database_url_sync = "clickhouse://default:@clickhouse1:8123/analytics"
    model_paths = ["app/analytics_models"]
    ch_cluster = "production_cluster"
```

Or with `database_config()`:

```python
from dbwarden import database_config

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://default:@clickhouse1:8123/analytics",
    model_paths=["app/analytics_models"],
    ch_cluster="production_cluster",
)
```

### How CH-2 locking works

1. dbwarden creates an ephemeral-sequential znode under the keeper path
2. The znode sequence number serves as the fencing token (monotonic by construction)
3. If the process crashes, the ZooKeeper session ends, and the ephemeral znode is removed immediately
4. The next worker sees no active lock and acquires one instantly

### CH-2 verification

```bash
# Run a migration on a replicated cluster
dbwarden migrate -d analytics

# Check lock status
dbwarden lock-status -d analytics

# Inspect keeper znodes directly
clickhouse-client --host keeper1 --port 9181 --query \
  "SELECT name, value, ctime FROM system.zookeeper WHERE path = '/dbwarden/locks'"
```

### CH-2 limitations

- Requires clickhouse-keeper or ZooKeeper infrastructure
- Keeper availability is a single point of failure for migration locking (but not for data serving)
- Does not prevent two workers from acquiring the lock during a network partition (addressed by CH-3)

---

## CH-3: Migration proxy

All migration DDL flows through a single migration proxy process. Workers connect only to the proxy, not directly to ClickHouse.

**Why it exists:** Even with CH-2's fast lock release, two workers could theoretically acquire the lock in rapid succession during a network partition. CH-3 eliminates this by design: only one process ever has DDL credentials, so concurrent migration is structurally impossible.

**Use when:** Production environments where zero-downtime deploys run from multiple replicas and you need the strongest DDL safety guarantee without Kubernetes.

### CH-3 prerequisites

- Everything in CH-2
- A dedicated proxy host or container
- Network configuration to restrict DDL access to the proxy only

### CH-3 infrastructure setup

**1. Create a migration-only ClickHouse user with DDL privileges:**

```sql
CREATE USER dbwarden_migration IDENTIFIED BY 'secure_password';
GRANT CREATE, ALTER, DROP, RENAME ON *.* TO dbwarden_migration;
```

**2. Create a restricted user for application connections (no DDL):**

```sql
CREATE USER dbwarden_app IDENTIFIED BY 'app_password';
GRANT SELECT, INSERT, UPDATE, DELETE ON analytics.* TO dbwarden_app;
```

**3. Configure ClickHouse to only allow DDL from the proxy IP:**

```xml
<!-- /etc/clickhouse-server/users.d/migration_restriction.xml -->
<clickhouse>
    <users>
        <dbwarden_migration>
            <networks>
                <ip>proxy-host-ip</ip>
            </networks>
        </dbwarden_migration>
    </users>
</clickhouse>
```

The permission model is the core of CH-3:

| User | Privileges | Used by | Purpose |
|------|-----------|---------|---------|
| `dbwarden_app` | `SELECT, INSERT, UPDATE, DELETE` on `analytics.*` | Worker nodes | Data read/write only. No DDL. |
| `dbwarden_migration` | `CREATE, ALTER, DROP, RENAME` on `*.*` | Proxy node only | Schema changes only. No DML. |

Workers can read and write data but cannot create, alter, or drop tables. The proxy can change schema but does not serve application traffic. Since only the proxy has DDL credentials, concurrent migration is structurally impossible.

**4. Run the migration proxy:**

The proxy is a thin process that holds the CH-2 lock and accepts DDL commands:

```python
# migration_proxy.py
import asyncio
from dbwarden import database_config
from dbwarden_redis import migration_lock
from redis.asyncio import Redis

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://dbwarden_migration:secure_password@proxy-host:8123/analytics",
    model_paths=["app/analytics_models"],
    ch_cluster="production_cluster",
)

async def run_migration():
    redis = Redis.from_url("redis://localhost:6379")
    async with migration_lock(redis, key="analytics_migration"):
        # Run migration via CLI
        import subprocess
        subprocess.run(["dbwarden", "migrate", "-d", "analytics"])

if __name__ == "__main__":
    asyncio.run(run_migration())
```

### CH-3 configuration

**Worker nodes (no DDL access):**

```python
from dbwarden import DbwardenDatabase

class AnalyticsWorker(DbwardenDatabase):
    database_name = "analytics"
    default = True
    database_type = "clickhouse"
    # Workers use the restricted user (no DDL privileges)
    database_url_sync = "clickhouse://dbwarden_app:app_password@clickhouse1:8123/analytics"
    model_paths = ["app/analytics_models"]
    ch_cluster = "production_cluster"
    # Workers do NOT run migrations; they only read schema
```

**Proxy node (DDL access):**

```python
from dbwarden import DbwardenDatabase

class AnalyticsProxy(DbwardenDatabase):
    database_name = "analytics"
    default = True
    database_type = "clickhouse"
    # Proxy uses the migration user (has DDL privileges)
    database_url_sync = "clickhouse://dbwarden_migration:secure_password@proxy-host:8123/analytics"
    model_paths = ["app/analytics_models"]
    ch_cluster = "production_cluster"
```

### CH-3 verification

```bash
# On the proxy node: run migration
python migration_proxy.py

# On any node: check lock status
dbwarden lock-status -d analytics

# On any node: verify schema
dbwarden diff -d analytics
```

### CH-3 takeover procedure

If the proxy process dies and you need to run migrations from a different host:

```bash
# 1. Kill any lingering proxy processes
kill $(pgrep -f migration_proxy)

# 2. Wait for the keeper lock to expire (or force release)
clickhouse-client --host keeper1 --port 9181 --query \
  "SELECT * FROM system.zookeeper WHERE path = '/dbwarden/locks'"

# 3. Run migration from the new proxy host
python migration_proxy.py
```

---

## CH-4: Singleton executor

Workers submit migration jobs; a single executor process executes them. The executor is the only process with DDL credentials, enforced by the deployment platform.

**Why it exists:** CH-3 requires running a proxy process, which is additional infrastructure. CH-4 achieves the same guarantee using the deployment platform's native primitives (Kubernetes Job with `parallelism: 1`). Workers submit jobs; the platform serializes them; the executor runs them one at a time.

**Use when:** Kubernetes deployments where you want the platform to enforce migration serialization rather than running a dedicated proxy process.

### CH-4 prerequisites

- Kubernetes cluster
- ClickHouse cluster with clickhouse-keeper or ZooKeeper
- A container image with dbwarden and your models installed

### CH-4 infrastructure setup

**1. Create a Kubernetes Secret for the migration credentials:**

```yaml
# migration-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-migration
  namespace: analytics
type: Opaque
stringData:
  url: "clickhouse://dbwarden_migration:secure_password@clickhouse1:8123/analytics"
```

**2. Create a ConfigMap with your dbwarden config:**

```yaml
# dbwarden-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dbwarden-config
  namespace: analytics
data:
  dbwarden.py: |
    from dbwarden import DbwardenDatabase

    class Analytics(DbwardenDatabase):
        database_name = "analytics"
        default = True
        database_type = "clickhouse"
        database_url_sync = "clickhouse://dbwarden_migration:secure_password@clickhouse1:8123/analytics"
        model_paths = ["app/analytics_models"]
        ch_cluster = "production_cluster"
```

**3. Create the executor Job (runs once, serially):**

```yaml
# migration-executor.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: dbwarden-migrate
  namespace: analytics
spec:
  # Key setting: only one job runs at a time
  parallelism: 1
  completions: 1
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: your-registry/dbwarden:latest
          command: ["dbwarden", "migrate", "-d", "analytics"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: clickhouse-migration
                  key: url
          volumeMounts:
            - name: config
              mountPath: /app/dbwarden.py
              subPath: dbwarden.py
            - name: models
              mountPath: /app/app/analytics_models
          resources:
            limits:
              memory: "256Mi"
              cpu: "500m"
      volumes:
        - name: config
          configMap:
            name: dbwarden-config
        - name: models
          configMap:
            name: analytics-models
```

**4. Trigger migrations from workers:**

Workers submit Jobs instead of running migrations directly:

```python
# In your deploy script or CI pipeline
from kubernetes import client, config

config.load_incluster_config()
batch_api = client.BatchV1Api()

job = client.V1Job(
    metadata=client.V1ObjectMeta(name=f"dbwarden-migrate-{timestamp}"),
    spec=client.V1JobSpec(
        parallelism=1,
        completions=1,
        backoff_limit=1,
        template=client.V1PodTemplateSpec(
            spec=client.V1PodSpec(
                restart_policy="Never",
                containers=[client.V1Container(
                    name="migrate",
                    image="your-registry/dbwarden:latest",
                    command=["dbwarden", "migrate", "-d", "analytics"],
                )],
            ),
        ),
    ),
)

batch_api.create_namespaced_job(namespace="analytics", body=job)
```

### CH-4 configuration

The executor container uses the same dbwarden config as CH-3 (the migration user with DDL privileges):

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    default = True
    database_type = "clickhouse"
    database_url_sync = "clickhouse://dbwarden_migration:secure_password@clickhouse1:8123/analytics"
    model_paths = ["app/analytics_models"]
    ch_cluster = "production_cluster"
```

### CH-4 verification

```bash
# Check that the executor Job ran successfully
kubectl get jobs -n analytics

# Check logs
kubectl logs job/dbwarden-migrate -n analytics

# Check lock status from any pod
kubectl exec -it clickhouse-client-pod -n analytics -- \
  dbwarden lock-status -d analytics
```

### CH-4 cleanup

After a successful migration, the Job remains until manually cleaned up. Use a TTL policy:

```yaml
# Add to the Job spec
spec:
  ttlSecondsAfterFinished: 3600  # Clean up after 1 hour
```

---

## Choosing a profile

| Your situation | Recommended profile |
|----------------|---------------------|
| Local development, single developer | CH-0 |
| Staging, single deploy process | CH-0 |
| Production, all migrations are idempotent | CH-1 |
| Production, replicated ClickHouse with keeper | CH-2 |
| Production, multiple deploy replicas, zero-downtime | CH-3 |
| Kubernetes, want platform-enforced serialization | CH-4 |

## Configuration reference

| Key | Default | Description |
|-----|---------|-------------|
| `ch_cluster` | `None` | ClickHouse cluster name. Appends `ON CLUSTER '<name>'` to every DDL statement. |
| `ch_replicated_database` | `False` | Use ClickHouse `Replicated` database engine. Mutually exclusive with `ch_cluster`. |
| `clickhouse_lock_ttl` | `120` | Lease TTL in seconds for CH-0/CH-1. How long a migration lock is held before auto-expiring. |
| `lock_namespace` | `"default"` | Lock scope. Allows independent lock streams (e.g., separate schema migrations from data operations). |

## Troubleshooting

### "Non-idempotent statement" error

```
Non-idempotent: ALTER TABLE events RENAME TO events_v2
```

**Solution**: Make the statement idempotent:

```sql
-- Instead of:
ALTER TABLE events RENAME TO events_v2;

-- Use:
ALTER TABLE IF EXISTS events RENAME TO events_v2;
-- Or restructure to avoid RENAME
```

Or restructure the migration to use idempotent forms:

### "Lease expired" error

```
ClickHouse lease held by: host=clickhouse1, pid=1234, expires=2026-09-01 12:00:00
```

**Solution**: Wait for lease expiry or force release:

```bash
dbwarden unlock -d analytics --force
```

### Replicated database conflicts

```
ConfigurationError: ch_cluster and ch_replicated_database are mutually exclusive
```

**Solution**: Use one or the other, not both:

```python
# Correct: ON CLUSTER
ch_cluster = "production_cluster"
ch_replicated_database = False

# Correct: Replicated database
ch_cluster = None
ch_replicated_database = True
```

### Keeper connection refused

```
Connection refused: keeper1:9181
```

**Solution**: Verify keeper is running and reachable:

```bash
# Check keeper status
clickhouse-client --host keeper1 --port 9181 --query "SELECT * FROM system.build_options"

# Check network connectivity
nc -zv keeper1 9181
```

### Migration stuck in RUNNING state

Check if the migration process is still alive:

```bash
dbwarden lock-status -d analytics
```

If the process is dead, force release the lock:

```bash
dbwarden unlock -d analytics --force
```

## See Also

- [Locking Architecture](../databases/locking.md): General locking model across all backends
- [ON Cluster](../databases/clickhouse/on-cluster.md): Cluster modes and DDL propagation
- [Immutability](../databases/clickhouse/immutability.md): What can never change in ClickHouse
- [Safety Classification](../databases/clickhouse/safety.md): Change classification levels

========================================================================
PAGE: https://docs.dbwarden.org/advanced/merge-handling/
========================================================================

# Merge Handling

dbwarden supports branch merge reconciliation for teams using git workflows. When two branches generate migrations that diverge from a common model state, dbwarden can detect the merge, generate a reconciliation migration, and mark the branch migrations as superseded.

## Overview

dbwarden is declarative: SQLAlchemy models are the source of truth, and migrations are derived output. When two branches diverge and each generates migrations, the merged schema is defined entirely by the merged models. The correct migration after a merge is `diff(merge_base_state, merged_models)`.

This means:
- Branch migrations carry no schema meaning going forward (only provenance)
- The reconciliation migration is the only new runnable migration
- Branch migrations are marked as superseded (audit-only, never run again)

## Commands

### `dbwarden merge`

Reconciles divergent migration histories after a branch merge.

```bash
dbwarden merge --database primary
```

**Flags:**
- `--database`, `-d`: Target database name
- `--rename-column`: Column rename to confirm (format: `table.old=new`)
- `--rename-table`: Table rename to confirm (format: `old=new`)
- `--force`, `-f`: Force marking hand-edited migrations
- `--commit`: Create a git commit with the changes
- `--verbose`, `-v`: Enable verbose logging

**Algorithm:**
1. Check preconditions (clean working tree, no conflict markers, merge-base resolvable)
2. Resolve merge-base state from git
3. Rebuild current model state from merged models
4. Compute reconciliation diff
5. Probe persistent environments
6. Generate reconciliation migration
7. Mark branch migrations as superseded
8. Report

### `dbwarden rebase`

Recovers a disposable environment after a merge.

```bash
dbwarden rebase --database local
```

**Flags:**
- `--database`, `-d`: Target database name
- `--yes`, `-y`: Skip confirmation prompts
- `--force`, `-f`: Force operation even against persistent environments
- `--check`: Only check what would happen, don't make changes
- `--verbose`, `-v`: Enable verbose logging

**Algorithm:**
1. Snapshot live environment
2. Diff against merged models
3. Generate environment-specific reconciliation
4. Apply with lock discipline
5. Update merge record

### `dbwarden reconcile`

Recovers a persistent environment after a dirty merge.

```bash
dbwarden reconcile staging
```

**Flags:**
- `environment`: Environment name to reconcile (required)
- `--database`, `-d`: Target database name
- `--rename-column`: Column rename to confirm
- `--dry-run`: Only show what would happen
- `--verbose`, `-v`: Enable verbose logging

## File Formats

### Superseded Marker

Prepended to migration files that have been superseded by a merge:

```sql
-- dbwarden:superseded
-- merged-into: 0006
-- merged-at: 2026-08-22T14:03:11Z
-- merge-base: 0004
-- branch: feature/profile-fields
-- applied-persistent: none
-- file-checksum: sha256:7ab1...

-- upgrade
ALTER TABLE users ADD COLUMN bio TEXT;

-- rollback
ALTER TABLE users DROP COLUMN bio;
```

**Rules:**
- Marked files are excluded from the runnable chain
- Marked files are never deleted (audit trail)
- `applied-persistent` tracks which environments applied the migration

### Reconciliation Migration Header

Added to migration files generated at merge time:

```sql
-- dbwarden:merge-reconciliation
-- merge-base: 0004 (state checksum 9f2c...)
-- supersedes: 0005_add_profile.sql, 0005_extend_billing.sql
-- probe: staging=clean, production=clean, qa=unknown
-- generated-by: dbwarden merge (0.18.0)

-- upgrade
...

-- rollback
...
```

## Environment Registry

Configure persistent vs disposable environments in your database config:

```python
from dbwarden import DbwardenDatabase, EnvironmentConfig

class Primary(DbwardenDatabase):
    database_name = "primary"
    database_type = "postgresql"
    database_url_sync = "postgresql://..."
    default = True
    environments = [
        EnvironmentConfig(name="staging", url_env="STAGING_DATABASE_URL", persistent=True),
        EnvironmentConfig(name="production", url_env="PROD_DATABASE_URL", persistent=True),
    ]
```

**Rules:**
- Unregistered environments are disposable by default
- Persistent environments require `reconcile` after a dirty merge
- Disposable environments can be reset with `rebase`

## Merge Detection

dbwarden detects merges by checking for:
1. **Divergent generation base**: Newest migration's base checksum doesn't match current model state
2. **Version collisions**: Multiple migration files share the same version prefix
3. **Snapshot discontinuity**: Latest snapshot doesn't match model state

When detected, `make-migrations` refuses generation and `status` shows `MERGE_PENDING`.

## Workflow

### After pulling a merge

1. **Check status:**
   ```bash
   dbwarden status
   # Shows: Merge: PENDING
   ```

2. **Run merge:**
   ```bash
   dbwarden merge --database primary
   ```

3. **Recover local database:**
   ```bash
   dbwarden rebase --database local
   ```

4. **Recover persistent environments (if dirty):**
   ```bash
   dbwarden reconcile staging
   ```

5. **Commit and push:**
   ```bash
   git add .
   git commit -m "Merge reconciliation"
   git push
   ```

## Known Limitations

- **MariaDB:** Snapshot support is incomplete; merge-base resolution uses `model_state.json` only. Rename detection features degrade as they do in the legacy live-fallback path. The mandatory confirmation rule (§9.1.1) becomes the primary safeguard for MariaDB merges.
- **Git required:** All merge operations require git to be available
- **No executable Python callbacks:** Declarative data migrations use verified, frozen `.data.py` artifacts that are parsed without execution. Arbitrary Python migration callbacks are unsupported; handwritten manual migrations remain SQL-only.
- **No revision branching/merging:** Linear versioned sequence per database

### MariaDB Specifics

MariaDB has limited snapshot support compared to PostgreSQL. When merging on a MariaDB database:

- **Snapshot-based rename detection** is degraded because MariaDB snapshots don't capture all column metadata
- **Merge-base resolution** uses only `model_state.json`, not the full snapshot
- **Mandatory confirmation** is required for all rename candidates during merge (§9.1.1)
- **Column type changes** may not be fully detected without a complete snapshot

For MariaDB projects, consider:
- Using `--rename-column` and `--rename-table` flags explicitly during merge
- Verifying the reconciliation migration carefully before applying
- Running `dbwarden diff` after merge to confirm no unexpected changes

## See Also

- [Migration Locking](migration-locking.md): How migrations are locked
- [Offline Integrity](../correctness/offline-integrity.md): Model state file management
- [CLI Reference](../cli-reference.md): All commands

========================================================================
PAGE: https://docs.dbwarden.org/advanced/migration-locking/
========================================================================

# Migration Locking

dbwarden uses per-engine native locking to prevent concurrent schema mutation. This page explains how it works, what happens when it fails, and how to recover from a stuck lock.

## How locking works

When `dbwarden migrate` runs, it:

1. Acquires a native lock on the database engine (advisory lock for PostgreSQL, named lock for MySQL, BEGIN IMMEDIATE for SQLite, lease for ClickHouse)
2. Writes a status row in the `dbwarden_lock` table for observability
3. Starts a background heartbeat to detect stale workers
4. Executes all pending migrations
5. Stops the heartbeat and releases the lock on success or failure

The lock mechanism varies by engine:

| Engine | Lock Type | Grade | Auto-release on crash |
|--------|-----------|-------|----------------------|
| PostgreSQL | Session-level advisory lock | A | Yes (connection teardown) |
| MySQL/MariaDB | Named user lock (GET_LOCK) | A | Yes (connection teardown) |
| SQLite | BEGIN IMMEDIATE transaction | B | Yes (journal cleanup) |
| ClickHouse | Lease row with fencing token | C | After TTL expiry |

If a second `migrate` invocation starts while the first holds the lock, it fails immediately with:

```
Could not acquire migration lock. <holder diagnostics>
```

dbwarden does not retry on lock failure by default. The calling process (CI job, deploy script) must decide whether to retry or abort.

## Inspecting lock state

```bash
$ dbwarden lock-status --database primary
```

Output when unlocked:

```
Migration lock: INACTIVE
```

Output when locked:

```
Migration lock status
  State:       RUNNING
  Health:      HEALTHY
  Execution:   abc123def456ghi7
  Host:        deploy-runner-7
  PID:         1234
  Migration:   V042
  Acquired:    2026-08-22 12:03:14Z
  Heartbeat:   2026-08-22 12:04:01Z
```

Health verdicts:

- **HEALTHY**: Lock held, heartbeat is fresh
- **STUCK**: Lock held, heartbeat is stale (process may be paused or dead)
- **DEAD**: Lock free, status row shows dead worker
- **AVAILABLE**: No lock held
- **COMPLETE**: Migration completed successfully
- **FAILED**: Migration failed
- **INSPECTING**: Recovery inspection in progress
- **NEEDS_REVIEW**: Human intervention required

## Heartbeat

dbwarden runs a background heartbeat that updates the `last_heartbeat_at` timestamp every 15 seconds. This allows other processes to detect stale or dead workers.

On native-lock engines (PostgreSQL, MySQL), heartbeat failure is an observability event only; the migration continues. On SQLite, there is no heartbeat because the migration transaction holds the write lock. On fallback engines (ClickHouse), heartbeat failure is fatal after a configurable grace period.

The heartbeat runs on a separate connection to avoid interfering with the migration connection.

## Stuck lock recovery

A lock becomes stuck when:

- The migration process was killed (SIGKILL, OOM, machine restart)
- A CI job was cancelled mid-run
- A deploy container was stopped before migrate completed
- The process is paused (GC, SIGSTOP, CPU starvation)

**Before unlocking, confirm no migration is running:**

```bash
# Check if the PID from lock-status is still alive
ps aux | grep <pid>

# Or check your deployment logs / CI job status
```

If the process is genuinely dead:

```bash
# 1. Confirm lock state and holder
$ dbwarden lock-status --database primary

# 2. Inspect migration history to see what ran last
$ dbwarden history --database primary

# 3. Check pending migrations
$ dbwarden status --database primary

# 4. Release the stale lock
$ dbwarden unlock --database primary --force

# 5. Retry migration
$ dbwarden migrate --database primary
```

The `--force` flag skips the confirmation prompt. Without it, `unlock` shows holder diagnostics and requires confirmation.

## Recovery state machine

When a new worker acquires the lock and detects a dead predecessor, it enters the recovery state machine:

```
AVAILABLE → RUNNING → COMPLETE → AVAILABLE (loop back)
                     → FAILED → AVAILABLE (loop back)
                     → DEAD (detected by new acquirer)
                          → INSPECTING
                               → COMPLETE (all steps applied)
                               → RUNNING (resume safely)
                               → NEEDS_REVIEW (human decision)
                                    → AVAILABLE (loop back)
                                    → RUNNING (retry)
```

The INSPECTING state compares the recorded migration checksum with the candidate migration. If they match, the migration can be resumed. If they don't match, the worker transitions to NEEDS_REVIEW and waits for human intervention.

## INSPECTING procedure

When a new worker acquires the lock and detects a dead predecessor (DEAD state), it executes the INSPECTING procedure:

1. **Compare checksums**: The recorded `migration_checksum` is compared with the candidate migration's checksum
2. **Walk history**: The migration history table is checked to determine the last durably recorded step
3. **Verify catalog state**: On transactional engines (PostgreSQL, SQLite), verify catalog state matches history
4. **Reconcile catalog**: On non-transactional engines (MySQL, ClickHouse), reconcile catalog against step list
5. **Apply recovery policy**: Based on the configured `recovery_policy`, transition to RUNNING (resume) or NEEDS_REVIEW (halt)

## Recovery policy

Configure the recovery policy in `dbwarden.py`:

```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    recovery_policy = "halt"  # default
```

Policies:

| Policy | Behavior |
|--------|----------|
| `halt` (default) | Transition to NEEDS_REVIEW, print inspection report, exit with code 78 |
| `resume_idempotent` | Resume only if all remaining statements are idempotent; else NEEDS_REVIEW |
| `force` | Re-run remaining steps unconditionally (audit-logged) |

## Re-entrancy protection

dbwarden prevents nested acquisition of the same advisory lock on PostgreSQL. If a worker attempts to acquire a lock it already holds, the acquisition is refused with an error. This prevents lock corruption from nested migration runs.

## Heartbeat fatal on fallback engines

On ClickHouse (fallback engine), if the heartbeat fails repeatedly (exceeding `fatal_grace`), the migration run is aborted immediately. This prevents a worker from executing DDL after losing its lease.

## ClickHouse per-statement fence check

For ClickHouse, dbwarden performs a per-statement fence check before each DDL statement. This validates that the worker still owns the lease by checking the fencing token matches. If the token has advanced (another worker acquired the lease), the migration is aborted with a `LockError`.

## POSSIBLE_CONCURRENT_EXECUTION detection

After a ClickHouse migration completes, dbwarden re-reads the lease and checks if the fencing token has advanced during the run. If it has, a `POSSIBLE_CONCURRENT_EXECUTION` warning is printed, indicating another worker may have executed DDL concurrently.

## TCP keepalive

dbwarden enables TCP keepalive on migration connections to prevent idle-timeout kills:

- **MySQL/MariaDB**: 60s idle, 3 probes, 30s interval (configurable via `tcp_keepalive` config)
- **PostgreSQL**: Uses default OS settings
- **SQLite**: N/A (file-based)

## SQLite busy_timeout

For SQLite, dbwarden sets a configurable `busy_timeout` to control how long `BEGIN IMMEDIATE` waits for the write lock:

```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    sqlite_busy_timeout = 5000  # Wait up to 5 seconds
```

Default is 0 (fail fast). Set to a positive value to wait for the write lock.

## Prepared transactions prohibition

dbwarden prohibits 2PC (prepared transactions) on the migration connection. Advisory locks interact badly with prepared transactions, which can lead to lock leaks or corruption.

!!! note "Advisory lock operational details"
    For PgBouncer detection (double PID check), replica detection (`pg_is_in_recovery()`), re-entrancy protection, and the session-scoped lock lifecycle, see [DDL Semantics](../databases/postgresql/ddl-semantics.md#advisory-lock-operational-details).

## AUDIT-level logging

dbwarden logs all unlock operations at AUDIT level for compliance and forensics. The audit log includes:

- Operator identity (user, host, PID)
- Target diagnostics (host, PID, execution ID)
- Outcome (success/failure)

Example audit log entry:
```
AUDIT: unlock executed by user=deploy host=ci-runner-1 pid=5678
target_host=deploy-runner-7 target_pid=1234 target_execution=abc123def456 outcome=success
```

## v1 lock phase-out

dbwarden v2 uses native locking with the `dbwarden_lock` table schema. The v1 lock schema (with `locked`, `owner_token` columns) is no longer supported. If a v1 table exists, it will be automatically dropped and recreated with the v2 schema on first use.

## Per-statement history recording

For non-transactional engines (MySQL, ClickHouse), dbwarden can record history per-statement instead of per-migration file. This enables finer-grained recovery but may impact performance.

```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    per_statement_history = True  # Default: False
```

When enabled, each DDL statement is recorded to the history table before the next statement begins. This is the only resumption signal on non-transactional engines.

Default is 0 (fail fast). Set to a positive value to wait for the write lock.

## When NOT to use `unlock`

Do not run `unlock` if:

- You are unsure whether a migration process is still running
- The lock status shows HEALTHY with a recent heartbeat
- Multiple processes share a database and you cannot confirm all are idle

Releasing a lock held by a live migration process will allow a second migration to start concurrently, which can corrupt schema state.

## ClickHouse coordination profiles

ClickHouse has no session-scoped user locks, so dbwarden uses a lease-based approach with coordination profiles. For the full explanation of why each profile exists and how to set up your deployment, see [ClickHouse Locking](clickhouse-locking.md).

| Profile | Description | Grade |
|---------|-------------|-------|
| CH-0 | Lease row with fencing token | C |
| CH-1 | CH-0 + strict idempotency enforcement | C |
| CH-2 | CH-1 + Keeper-backed lock | C |
| CH-3 | Migration proxy choke point | A- |
| CH-4 | Singleton executor | A* |

CH-0 and CH-1 are implemented in core. CH-2, CH-3, and CH-4 require external infrastructure and are documented as deployment options.

### CH-1 idempotency

Under CH-1, non-idempotent statements are refused:

- Idempotent: `CREATE TABLE IF NOT EXISTS`, `DROP TABLE IF EXISTS`, `ALTER TABLE ADD COLUMN IF NOT EXISTS`
- Non-idempotent: `RENAME`, `MODIFY COLUMN`, `UPDATE`, `DELETE`
- Unknown: treated as non-idempotent for safety

## Preventing concurrent migration in CI

In CI/CD, run migrations from a single job with no parallelism:

```yaml
# GitHub Actions: serialize via job dependency
jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - run: dbwarden migrate --database primary
  deploy:
    needs: migrate
    ...
```

If your pipeline can trigger multiple concurrent deploys, add a concurrency group:

```yaml
concurrency:
  group: migrate-${{ github.ref }}
  cancel-in-progress: false
```

`cancel-in-progress: false` queues the second run instead of cancelling it, which avoids orphaned locks from killed jobs.

## Configuration

| Key | Default | Description |
|-----|---------|-------------|
| `lock_namespace` | `"default"` | Lock scope (allows independent lock streams) |

## Distributed locking with Redis

For multi-instance deployments where multiple application replicas could
trigger migrations concurrently, the `dbwarden-redis` plugin provides
a Redis-backed distributed lock:

```bash
dbwarden plugin add dbwarden-redis
```

```python
from redis.asyncio import Redis
from dbwarden_redis import migration_lock

redis = Redis.from_url("redis://localhost:6379")

async with migration_lock(redis):
    await run_migration(
        sql_statements,
        version="0002",
        migration_operation="apply",
        filename="primary__0002_feature.sql",
    )
```

The Redis lock uses `SET NX EX` with a configurable TTL (default 60 seconds).
If the application crashes while holding the lock, Redis releases it
automatically after the TTL expires.

### Database-level vs Redis lock

| Aspect | Database lock | Redis lock |
|--------|---------------|------------|
| Scope | CLI commands (`migrate`, `rollback`, `downgrade`) | Any code path you wrap |
| Storage | `dbwarden_lock` table in the target database | Redis key |
| TTL | Heartbeat-based stale detection (default 45s); native locks auto-release on connection close | Configurable TTL (default 60s) |
| Failure mode | Detects stale locks via heartbeat; recovery state machine guides recovery | Auto-released after TTL |
| External dependency | None (uses the database itself) | Redis required |

Both locks can be used independently or together; they guard different
entry points. The database lock protects the CLI; the Redis lock
protects any entry point you wrap.

See the [dbwarden-redis](https://github.com/dbwarden-org/dbwarden-redis) plugin for full documentation.

## Lifespan integration

The `dbwarden_lifespan` context manager wraps migration logic and
engine disposal into a single FastAPI-compatible lifespan. When using
the Redis lock in a lifespan, acquire the lock before entering the
migration context:

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from dbwarden_fastapi import dbwarden_lifespan
from dbwarden_redis import migration_lock

@asynccontextmanager
async def lifespan(app: FastAPI):
    redis = Redis.from_url("redis://localhost:6379")
    async with migration_lock(redis):
        async with dbwarden_lifespan(mode="migrate", allow_in_production=True):
            yield
```

See also: [Safe Deployment](safe-deployment.md) | [CI/CD Patterns](ci-cd-patterns.md) | [`lock` commands](../commands/lock.md)

The FastAPI lifespan helper lives in the `dbwarden-fastapi` plugin: [dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi).
The Redis lock lives in the `dbwarden-redis` plugin: [dbwarden-redis](https://github.com/dbwarden-org/dbwarden-redis).

========================================================================
PAGE: https://docs.dbwarden.org/advanced/safe-deployment/
========================================================================

# Safe Deployment

How to deploy schema changes with minimal risk and a clear recovery path.

## Pre-flight checklist

Before running migrations in production:

- [ ] Confirm no other migration is running (`dbwarden lock-status`)
- [ ] Review pending migrations (`dbwarden status`)
- [ ] Confirm migrations have been tested in staging
- [ ] Take a backup if your database does not have point-in-time recovery

## Standard deploy sequence

```bash
# 1. Verify lock is free
$ dbwarden lock-status --database primary

# 2. Check what will run
$ dbwarden status --database primary

# 3. Apply with backup
$ dbwarden migrate --database primary --with-backup --backup-dir ./backups

# 4. Confirm clean state post-migration
$ dbwarden status --database primary
$ dbwarden history --database primary
```

For multi-database deployments:

```bash
$ dbwarden migrate --all --with-backup --backup-dir ./backups
```

## What happens when a migration fails mid-run

### PostgreSQL (transactional DDL)

PostgreSQL wraps DDL in transactions. If a migration file fails partway through, the entire file is rolled back. The migration remains in "pending" state. The lock is released. You can safely fix the SQL and retry.

```bash
# After a failed migration:
$ dbwarden status --database primary     # confirm migration is still pending
$ dbwarden lock-status --database primary # confirm lock was released

# Fix the migration file, then:
$ dbwarden migrate --database primary
```

### MySQL / databases without transactional DDL

DDL cannot be rolled back. A failed migration may have partially applied changes (e.g., a table was created but the index was not). Manual inspection is required before retrying.

```bash
# Check what the migration was supposed to do
cat migrations/primary/V__0012_add_payment_tables.sql

# Inspect current schema state via your database client
# Determine what was applied and what was not

# Either:
# a) Manually apply the remaining SQL
# b) Create a corrective migration
# c) Roll back manually and retry from scratch
```

## Recovery: stuck lock

If a migration process was killed and the lock was not released:

```bash
# 1. Confirm no migration process is running
$ dbwarden lock-status --database primary

# 2. Inspect history to see the last applied migration
$ dbwarden history --database primary

# 3. Inspect pending state
$ dbwarden status --database primary

# 4. Only if the process is confirmed dead:
$ dbwarden unlock --database primary

# 5. Retry
$ dbwarden migrate --database primary
```

See [Migration Locking](migration-locking.md) for full lock recovery guidance.

## Recovery: failed migration, data is wrong

If a migration applied successfully but produced incorrect data or schema:

**Option A: Rollback** (if the migration has a `-- rollback` section):

```bash
$ dbwarden rollback --database primary
```

This executes the rollback SQL defined in the migration file. Verify the rollback SQL was written when the migration was created; not all migrations include one.

**Option B: Forward fix** (preferred for data migrations):

```bash
# Create a corrective migration
$ dbwarden new "fix column type on payments" --database primary
# Edit the generated file with the corrective SQL
$ dbwarden migrate --database primary
```

Forward fixes are safer than rollbacks for data migrations, as rollback SQL is harder to write correctly after the fact.

## Baseline migrations

For databases that already have a schema (migrating from another tool or brownfield setup):

```bash
$ dbwarden migrate --database primary --baseline --to-version 0005
```

`--baseline` marks migrations as applied without executing them. Use this to tell dbwarden "this database already has schema up to version 0005."

## Smoke test after deploy

After migrations complete, run a quick connectivity and schema check:

```bash
$ dbwarden check-db --database primary
```

`check-db` inspects the live database schema and reports what tables and columns exist. Use this to confirm the schema matches what your application expects.

See also: [Migration Locking](migration-locking.md) | [CI/CD Patterns](ci-cd-patterns.md) | [`rollback` command](../commands/rollback.md) | [Cookbook: Safety & Impact](../cookbook/06-safety-impact.md)

========================================================================
PAGE: https://docs.dbwarden.org/architecture-deep-dive/
========================================================================

# Architecture

This page explains dbwarden internals for contributors and advanced debugging. dbwarden is structured as a multi-pass compiler.

## Compilation pipeline

```text
CLI (Typer)
  -> Commands layer (compilation driver)
    -> Engine layer (parsing, canonicalization, diff, optimization, code generation)
      -> Repository layer (migration + lock records)
        -> Database layer (SQLAlchemy connection + SQL execution)
```

## Responsibilities

- CLI: parse args, global flags (`--dev`, `--strict-translation`, `--help`)
- Commands: orchestrate workflows (`migrate`, `rollback`, `make-migrations`, `status`, `history`, `check`, `diff`, `generate-models`, `export-models`, `seed`, `lock-status`, `unlock`, `init`, `snapshot`, `settings`, `database`, `version`)
- Engine: parse files, resolve ordering, checksums, model discovery
- Repositories: read/write migration and lock metadata
- Database: execute SQL with backend-aware connections

## Configuration resolution pipeline

When runtime config is requested:

1. discover one config source (`dbwarden.py` or single callsite)
2. fallback to `DBWARDEN_CONFIG_MODULE` when configured
3. import source and register concrete `DbwardenDatabase` classes or execute `database_config(...)` calls
4. validate uniqueness/default/model-path rules
5. resolve selected database and apply `--dev` swap when enabled

Ambiguous sources fail fast.

## Migration execution lifecycle

For `migrate`:

1. ensure migrations metadata table exists
2. ensure lock table exists
3. acquire lock
4. build pending execution plan
5. execute SQL statements
6. record migration metadata/checksums
7. release lock

## Rollback lifecycle

Rollback uses the same lock discipline, selecting rollback SQL from applied files in reverse order.

## Model-to-SQL generation lifecycle

Online, offline, merge, and reconcile generation use the shared generation pipeline in `commands/make_migrations/generation.py`.

`make-migrations` pipeline:

1. discover model paths
2. import model modules
3. extract table/column metadata
4. load latest schema snapshot (`.dbwarden/schemas/*.schema.json`) if one exists
5. if snapshot exists: **snapshot-diff path**
   - diff model tables against snapshot tables
   - auto-detect table renames from dropped↔added table pairs (column overlap ≥ 0.6)
   - apply user `--rename-table` flags and/or interactive prompts to confirm table renames
   - emit `ALTER TABLE ... RENAME TO` (confirmed) or `DROP TABLE` + `CREATE TABLE` (not confirmed)
   - apply confirmed table renames to snapshot before column processing
   - auto-detect column renames from dropped↔added pairs of the same type
   - apply user `--rename` flags and/or interactive prompts to confirm renames
   - detect column-level changes: type, nullability, default (same-name columns)
   - emit `RENAME COLUMN` (confirmed) or `DROP` + `ADD` (not confirmed)
   - emit `ALTER COLUMN TYPE` / `SET NOT NULL` / `DROP NOT NULL` / `SET DEFAULT` / `DROP DEFAULT`
   - optionally use multi-step safe type change (`--safe-type-change`)
   - order all operations by `StatementOrder` (RENAME_TABLE first) and assemble upgrade/rollback
   - generate upgrade and rollback SQL from the ops
 6. if no snapshot: **live-DB fallback path**
    - take a full schema snapshot from the live database via `extract_full_schema_snapshot()`
    - run standard snapshot-diff pipeline against it (type, nullability, default, FK, index changes)
    - only rename detection is unavailable without a cached snapshot
7. deduplicate against existing migration statements
8. classify operations using canonical safety levels; expand-contract type changes can create ordered stages and a severity split also moves dependent operations
9. let plugins assign statements to base, deferred, or registered custom groups
10. write migration file; the base state contains only base operations, and deferred state remains separate until composed or applied
11. write companion `.plan.json` metadata file (with `resolved_from` on rename ops)

See [safety-scoped migrations](correctness/safety-scoped-migrations.md) for pending-state composition, execution ceilings, and merge integrity rules.

## PostgreSQL Handler Pipeline

PostgreSQL support is implemented through `dbwarden.engine.backends.postgresql.handlers`. Each handler exposes a small contract: `extract`, `model_spec_from_tables`, `canonicalize`, `diff`, and `emit`.

The `RegistryDriver` runs that contract in order:

1. extract snapshot state into a handler specific shape
2. derive model state from `ModelTable` objects, or config for preamble handlers
3. canonicalize both sides
4. diff into `Op` objects
5. emit backend SQL from the ops

The SQL assembly layer then sorts statements by `StatementOrder` and joins upgrade and rollback SQL into migration files.

### Handler Groups

| Handler | Purpose |
|---------|---------|
| `ColumnHandler` | Column add, drop, type, nullability, default, autoincrement, comment, and backend specific column metadata |
| `ConstraintHandler` | Unique, check, and foreign key constraints |
| `IndexHandler` | PostgreSQL and ClickHouse index operations |
| `TableHandler` | Table create, drop, and table comments |
| `RenameTableHandler` | Table rename operations |
| `SchemaHandler` | Schema create and drop |
| `PgTableHandler` | PostgreSQL table options, inheritance, and exclude constraints |
| `PartitionHandler` | Native PostgreSQL partitioning and partition attachment |
| `StorageParamsHandler` | PostgreSQL storage parameter changes |
| `EnumHandler` | Enum create, drop, and add value |
| `DomainHandler` | Domain create and drop |
| `SequenceHandler` | Sequence create and drop |
| `ViewHandler` | Regular and materialized view changes |
| `PoliciesHandler` | RLS enablement and policy lifecycle |
| `GrantsHandler` | Table and schema grants |
| `RoleHandler` | Role create and alter |
| `DefaultPrivilegesHandler` | Default privilege grants and revokes |
| `FunctionHandler` | Function create, replace, and drop |
| `TriggerHandler` | Trigger create and drop |
| `EventTriggerHandler` | Event trigger lifecycle |
| `ExtendedStatisticsHandler` | `CREATE STATISTICS` and drop |
| `StatisticsHandler` | Extended statistics variants and compatibility helpers |
| `MyTableHandler` | MySQL table metadata |
| `ChTableHandler` | ClickHouse table options and engine recreate |

### Online and Offline Paths

The online path uses `diff_models_against_snapshot` and the registry driver directly.

The offline path uses `diff_model_states`, which still keeps a few raw dict comparisons for state only fields such as column diffs, PostgreSQL table scalars, MySQL table metadata, and table comments. Those raw ops still flow through the same emit layer, so the SQL output stays aligned with the handler path.

The equivalence tests in `tests/test_pg_registry.py` and `tests/engine/snapshot/test_backend.py` lock that behavior in.

### Snapshot write lifecycle (in `migrate`)

After applying versioned migrations, `migrate` calls `_write_migration_snapshot()`:

1. connect to database (respecting sandbox override)
2. extract full schema: tables, columns, types, indexes, constraints, enums
3. compute SHA-256 checksum
4. write `<migration_id>.schema.json` to `.dbwarden/schemas/`
5. on failure: log warning (non-blocking)

Snapshots are not written during `--dry-run`, `--sandbox`, or for repeatable migrations.

## Repeatable migration model

Supported classes:

- versioned (`NNNN_`): run once in ordered sequence
- runs always (`RA__`): run each migrate execution
- runs on change (`ROC__`): run only when checksum changes

## Integrity model

Checksums are recorded for migration content and used for:

- repeatable migration change detection
- migration consistency checks
- audit/debug confidence

## Concurrency model

Migration-mutating commands are serialized by lock state stored in database tables.

Recovery commands:

- `dbwarden lock-status`
- `dbwarden unlock`

## Dev translation path

With `--dev` and SQLite target:

1. extract model types/defaults
2. translate backend-specific constructs
3. fallback behavior in non-strict mode
4. fail-fast behavior with `--strict-translation`

Translation happens during SQL generation, not by mutating existing migration files.

## Error propagation strategy

- config/load validation errors fail early
- execution errors abort current run with context
- lock release is guarded in cleanup paths

========================================================================
PAGE: https://docs.dbwarden.org/changelog/
========================================================================

# Changelog

All notable changes to dbwarden, newest first. Versions follow semantic versioning and are tagged in the repository.

## Unreleased

## \[0.20.1\] - 2026-10-10

### Fixed

- **ClickHouse lock status updates use valid mutation syntax.** `update_heartbeat` and `update_state` emitted `ALTER TABLE dbwarden_lock UPDATE SET ...`, which ClickHouse rejects with a syntax error (code 62), so every ClickHouse migration failed while releasing its lock. The statements now emit `ALTER TABLE ... UPDATE col = expr`, matching the fencing token bump.

## \[0.20.0\] - 2026-10-10

### Added

- **`DbwardenConfig` for project-wide migration policy.** A singleton class (`pre_migrate_safety`, `pre_migrate_impact`, `missing_plan`, `impact_paths`) that controls built-in safety and impact gates. Defaults preserve current behavior.
- **Six lifecycle hooks.** `pre_migration_run`, `pre_migration`, `migration_progress`, `post_migration`, `on_migration_failure`, `post_migration_run` — all multi-value, all command-aware (`migrate`, `rollback`, `downgrade`).
- **`register_migration_hooks()` API.** Register project-wide app hooks with signature validation and deduplication.
- **`validate_migration_hooks()` API.** Pre-command validation of all registered hooks (global, plugin, per-database).
- **Per-database `migration_hooks`.** `database_config()` accepts `migration_hooks` dict; per-database hooks run after global hooks.
- **Structured preflight runner.** `run_preflight()` validates files, plans, dependencies, and impact before lock acquisition.
- **`validate_dependencies()` for migration deps.** Checks missing targets, circular deps, unmet applied deps, and superseded deps. Always blocks (correctness, not policy).
- **`DependencyError` dataclass.** Structured error type for dependency validation results.
- **`--force` acknowledgement for safety warnings.** When `pre_migrate_safety="block"`, WARNING safety ops block unless `--force` is used.
- **`progress_callback` in `run_migration()`.** Per-statement progress hooks fire via callback parameter.
- **Operation severity classification.** Canonical SAFE/INFO/WARN/CRITICAL/UNKNOWN levels via `engine.safety.classifiers`, derived per operation and backend.
- **Plugin migration groups through `register_migration_category`.** Operation/statement `category` and `safety` fields, `plugin categories` inspection with `--load`. Custom groups retain dependency closure, state checksums, and existing ceilings.
- **Safety-scoped generation and execution.** Dependency-closed split files, recorded severity, strict-prefix ceilings, exit 3 deferrals, and static plan adoption with exit 4 for unresolved files.
- **Schema 1.1 plans with full typed operations**, normalized content hashes, state checksums, split pairing, and required acknowledgements.
- **Shared effective-state generation** for online, offline, merge, and reconciliation workflows; pending plans compose without SQL replay.
- **Static SQL parsing through SQLGlot and native PostgreSQL grammar validation through pglast.** Unmapped statements remain UNKNOWN; neither parser executes SQL.
- **Severity status, structured deferral/skip events, and optional deferral metrics.**
- **Declarative data migrations.** `dbwarden data` command group (`render`, `describe`, `docs`, `data transition` workflows), frozen `.data.py` bundles, snapshot registry, and guarded rollback of created targets.
- **Data-aware CLI surface.** `make-data-migration`, `diff --data`, `check --write-plan`/`--all`, `migrate --max-severity`/`--data`/`--reapply-data`, and `make-migrations --split-at-severity`/`--strict-pending`/`--dry-run`/`--param`/`--show-managed-values`.
- **Data path configuration.** `data_paths`, `data_snapshot_dir`, and `snapshot_registry` on `database_config()`.
- **DB-scoped settings in resolved state.** `recovery_policy`, `tcp_keepalive`, `rename_policy`, etc. now flow through `DatabaseConfig`.
- **Python 3.10+ support.** Lowered from 3.12.7+ with `tomli` and `typing-extensions` fallbacks.

### Changed

- **Python version requirement lowered from 3.12.7+ to 3.10+.**
- **`impact_paths` default changed from `[]` to `["."]`.**
- **`ProjectConfigEntry` enforces** `pre_migrate_impact="block"` requires `missing_plan="block"`.
- **`impact_paths` validates** absolute paths, traversal, and symlink escapes.
- **Merge dirty-environment check fails closed** on parsing failures.
- **`resolve_migration_order()` error messages** include per-migration dependency details.
- **Backup runs after lock acquisition** (Section 7.1 ordering).
- **Preflight checks run before lock acquisition** (Section 7.1 ordering).
- **Dry-run mode** runs preflight, merge checks, and pre-hooks for all migration types (including RA/ROC).
- **`post_migration_run` does not fire** in dry-run mode.
- **`migration_progress` fires per-statement** via `progress_callback`.

### Fixed

- **Plan provenance is anchored and keyed.** Trusted plans require their typed-operations anchor, and content hashes are HMAC-SHA256 keyed by a project-local `.dbwarden/plan.secret` — fabricated or recomputed plan files are rejected instead of authorizing severity ceilings. Legacy unkeyed plans fail closed with a regenerate hint.
- **Frozen data bundles seal their execution steps.** `data_execution` is integrity-bound into the frozen artifact; tampered batch rows with a recomputed manifest are refused before any write. Bundles generated before the seal need one regeneration.
- **UNKNOWN severity exceeds every ceiling, including the CRITICAL default.** Plan-less files keep their `missing_plan` policy behavior; explicitly-UNKNOWN files now stop with exit 3 under any ceiling.
- **Model discovery extracts `__table_args__` constraints.** Table-level `UniqueConstraint`/`CheckConstraint` (named or unnamed) now reach model state, diffs, and CREATE TABLE output on every backend; previously SQLite rebuilds silently dropped them.
- **SQLite rebuilds absorb companion column operations.** A type change combined with add/drop column no longer leaves a stray op that breaks the rollback with `duplicate column name`.
- **Data-losing rollbacks are announced.** Dropping a column (or a rebuild that drops one) warns at generation time, records `rollback_warnings` in the plan artifact, and prints pre-apply — rollback restores schema with NULL values, not row data.
- **The static classifier fails closed on parser recursion** and the docs checker returns exit 4 instead of crashing on deeply nested input.
- **Deferred-file races no longer crash.** Deleted deferred files are tolerated by stop reports and status age metrics; missing `data_execution` raises the designed guard instead of a bare `KeyError`; tamper reasons propagate with specifics instead of "malformed plan".
- **Offline safe type changes accept SQLAlchemy's non-key `autoincrement="auto"`.**
- **PostgreSQL diffs no longer emit ClickHouse SQL from cached column type metadata.**
- **Type-change `--plan` output remains valid JSON.**
- **Merge refreshes both state JSON paths.**
- **Windows repeatables use matching history keys and preserve legacy records.**
- **Unknown emitters and failed diffs are rejected** instead of silently omitting operations.
- **Colliding branch files are preserved during merge** and all reconciliation versions are recorded.
- **Persistent-environment convergence is verified** before marking repair complete.
- **Baseline records history without executing migration SQL.** PostgreSQL history lookup includes every non-null version, including baseline and reconciliation rows.
- **Severity levels are canonical.** Drop table/column is CRITICAL; SET NOT NULL and non-concurrent PostgreSQL indexes are WARN. Legacy plan values retain their INFO/WARNING/ERROR spellings.
- **A PostgreSQL identity column is no longer rendered as `SERIAL` too.** A model declaring `pg.field(identity=...)` on an integer primary key had the base type rewritten to `SERIAL` by the primary-key mapper and *also* rendered as `GENERATED ... AS IDENTITY`, and PostgreSQL rejected the column with *both default and identity specified*.
- **PostgreSQL index sort options are captured.** The snapshot extractor and `generate-models` called `pg_index_column_has_property` with the 0-based series index where the function is 1-based, so no index ever recorded its `ASC`/`DESC` or `NULLS FIRST`/`NULLS LAST`.
- **`NULLS NOT DISTINCT` is emitted in the right position.** It was appended after `WHERE`, which PostgreSQL cannot parse; it now precedes `INCLUDE` and `WHERE` in both the upgrade and the rollback.
- **PostgreSQL storage parameters, column `COLLATE`, and `COMPRESSION` are emitted.** `pg_storage_params` was dropped from the model spec, `CREATE TABLE` never rendered a `WITH (...)` storage clause, and neither collation nor compression reached the DDL.
- **Reverse-engineered PostgreSQL models keep their index list.** `generate-models` now writes `pg_indexes` entries as `PgIndexSpec` objects, emitting `columns=[]` for expression indexes.
- **MySQL and MariaDB snapshot types carry their length and precision.** A reverse-engineered `varchar` reached the column-definition builder as the bare base type and raised *Incomplete MySQL column type*, so `diff` never converged. Inherited charset and collation are also no longer treated as model differences.
- **The last migration of a MySQL or MariaDB run is recorded.** Each DDL statement implicitly commits, but the bookkeeping `INSERT` opened a transaction nothing committed; the next migration's DDL committed it implicitly, so the final migration was silently dropped from the history and re-applying it failed with *Duplicate column name*.
- **MariaDB uses the MySQL lock DDL instead of the SQLite fallback.** The v2 lock templates are keyed by database type and `mariadb` was not a key, so `CREATE TABLE` used `TEXT` for the namespace primary key and MariaDB rejected it with *BLOB/TEXT column 'namespace' used in key specification without a key length*.
- **The configured PostgreSQL schema is created before the bookkeeping tables.** Bookkeeping queries reference `<schema>._dbwarden_migrations`, but the schema may only be created by a user migration; on a fresh database the first `CREATE` failed with `InvalidSchemaName`.
- **`_parse_plan_safety` no longer crashes** on `operations: null` in plan files.
- **`ProjectConfig.impact_paths` default** now matches spec (`["."]`).
- **Plugin-registered lifecycle hooks** are now collected and executed (previously silently ignored).
- **Per-database hooks receive signature validation** at pre-command time.
- **Two-class error message** now names both conflicting classes.
- **`_collect_hooks()` ordering**: plugins first, then app hooks, then per-database hooks.
- **`check_dirty_environment()` fails closed** on merge-record parsing failures.
- **SQLite is a first-class backend.** Changes SQLite's `ALTER TABLE` cannot express - column type, nullability, default, table constraints, `WITHOUT ROWID`, `STRICT`, generated expressions and collation - are emitted as a table rebuild (create, copy, drop, rename, recreate indexes) with the reverse rebuild as the rollback, instead of a comment. All changes to one table collapse into a single rebuild.
- **SQLite table and column metadata.** `SqTableMeta` supports `sq_without_rowid`, `sq_strict` and `sq_indexes`; the new `SqColumnMeta` plus `sq.field(generated=..., generated_mode=..., collate=...)` cover generated columns and per-column collation. Both are emitted in `CREATE TABLE`, captured by schema snapshots, and written back by `generate-models`.
- **SQLite impact analysis.** `check-impact` reports table-option, generated-column and collation changes as warnings, because each rebuilds the table.
- **PostgreSQL reverse-engineering now works with mixed-case and quoted identifiers.** `generate-models` and snapshot queries use quoted `regclass` references, so tables and columns such as `"MyTable"` and `"weird-col"` no longer silently fail metadata extraction.
- **`generate-models` no longer crashes against PostgreSQL.** The command now opens a raw SQLAlchemy connection instead of a transactional one, avoiding the closed-transaction error during SQLAlchemy reflection.
- **Generated model imports are complete.** Models that only have column-level metadata now correctly import the required table-meta class (`PGTableMeta`, `MyTableMeta`, `SqTableMeta`).
- **Python identifier sanitization.** SQL column names containing hyphens or other non-identifier characters are converted into valid Python attribute and nested-class names while preserving the original SQL name in `Column(...)`.
- **PostgreSQL array columns render correctly.** Array types are emitted as `text[]`, `integer[]`, etc. instead of bare `array`.
- **ClickHouse codec extraction is parenthesis-aware.** Multi-codec chains like `CODEC(Delta(8), ZSTD(1))` are preserved in order, and the implicit default `LZ4` codec is omitted.
- **ClickHouse `LowCardinality(Nullable(T))` is preserved.** Reverse-engineering now correctly captures both `low_cardinality=True` and `nullable=True` for nested wrappers.
- **PostgreSQL `SERIAL`/`BIGSERIAL` and identity columns round-trip cleanly.** `generate-models` only marks columns as `autoincrement=True` when the live default is a `nextval(...)` sequence; identity columns are left as `autoincrement=False` and rely on `pg_meta`. `make-migrations` now treats both `SERIAL` types and identity columns as autoincrement-equivalent.
- **PostgreSQL identity column ALTER syntax is valid.** Identity changes now emit `ALTER TABLE ... ALTER COLUMN ... SET START WITH ...`, `SET INCREMENT BY ...`, `SET MINVALUE ...`, `SET MAXVALUE ...` instead of the invalid `SET (...)` form.
- **ClickHouse default changes generate valid DDL.** `_build_alter_default_sql` now emits `ALTER TABLE ... MODIFY COLUMN ... DEFAULT ...` and `... REMOVE DEFAULT` instead of falling through to PostgreSQL `ALTER COLUMN SET DEFAULT` syntax.
- **ClickHouse nullability changes are type-safe.** The shared nullable builder guards against generic SQLAlchemy type strings and requires a ClickHouse type such as `Int64` or `String`.
- **ClickHouse materialized view DDL is complete.** `CREATE MATERIALIZED VIEW` now includes `POPULATE`, `REFRESH ...`, and `SETTINGS ...` when declared, and omits engine/`ORDER BY`/`PARTITION BY` clauses when a `TO` target is set.
- **ClickHouse `ChEngineSpec` engines converge with snapshot engines.** Model-side `ChEngineSpec('MergeTree')` and similar specs are serialized to the same string/tuple representation stored in snapshots, so `make-migrations` no longer emits spurious `alter_ch_options` operations for engine-only differences.
- **ClickHouse implicit MV backing tables are excluded from diffs.** Internal `.inner_id.*` tables created by ClickHouse for materialized views with implicit storage are filtered out of model-to-snapshot diffs.
- **ClickHouse default `MergeTree` settings no longer cause drift.** Snapshot extraction records the server's default setting values; both sides strip never-declared defaults so hand-written models that omit settings converge with snapshots.
- **ClickHouse materialized view `SELECT` database qualifier is stripped during reverse-engineering.** `FROM dbwarden_test.events` becomes `FROM events` in the snapshot, preventing spurious `MODIFY QUERY` diffs when models are written without qualifiers.
- **ClickHouse materialized views reverse-engineer to `MaterializedView` models.** `generate-models` now emits `MaterializedView` base classes with `CHViewMeta` and `materialized_view(...)` specs for both implicit-storage and `TO`-target MVs, including correct imports for single-file and per-file output.
- **PostgreSQL enum types are created before tables in migrations.** `make-migrations` now emits `CREATE TYPE ... AS ENUM (...)` before any `CREATE TABLE` that references the enum, both when diffing against a snapshot and when generating the very first migration.
- **PostgreSQL foreign keys round-trip when the referenced table has no constraints.** The constraint handler now recognizes the referenced table from the full model set rather than only from tables that declare their own constraints.
- **PostgreSQL migration parsing no longer invents a `constraint` column.** `ALTER TABLE ... ADD CONSTRAINT ...` statements in pending migrations are ignored by snapshot merging; previously they were misread as `ADD COLUMN constraint`.
- **PostgreSQL identity and storage metadata no longer drift after reverse-engineering.** `make-migrations` only emits `ALTER TABLE ... SET STORAGE ...` or identity-sequence parameter changes when the model explicitly requests them, so models produced by `generate-models` converge cleanly with the database.
- **PostgreSQL identity values are normalized to lowercase.** `Identity(always=True)` and `pg.field(identity="ALWAYS")` both normalize to `pg_identity="always"`, eliminating spurious identity diffs.
- **Official plugin provenance recognizes renamed publisher workflows.** `dbwarden plugin add` now verifies the trusted-publishing attestations for the ClickHouse RBAC, PostgreSQL extensions, PostgreSQL RBAC, and PostgreSQL types plugins against their `publishing.yml` workflows.
- **A migration generated without a snapshot no longer drops every table constraint.** `make-migrations` falls back to generating from the models alone when there is no schema snapshot, the database is unreachable, or the snapshot diff raises. That path emitted columns, indexes and foreign keys but silently omitted `UNIQUE` and `CHECK` constraints, so the schema applied cleanly and then failed at runtime with *there is no unique or exclusion constraint matching the ON CONFLICT specification*.
- **`EXCLUDE` constraints survive model-only generation.** They are rendered by the PostgreSQL table handler, which the fallback path does not run.
- **A single-column foreign key is emitted once on MySQL and MariaDB.** It was rendered inline as `REFERENCES` in the column definition *and* as a named `ADD CONSTRAINT`, which MariaDB honours as two constraints.
- **An unnamed `CHECK` constraint keeps the name the database gave it.** PostgreSQL names an inline `CHECK (age >= 0)` itself while dbwarden's generated name is index-based, so every unnamed check was dropped and recreated on every run - and reordering a model's checks renamed them. Checks the model does not name are now matched by expression, with casts and parentheses normalized away; a changed expression is still detected.
- **An aggregating view's target table is emitted as ClickHouse DDL.** When the model declared its dimension columns, the target was built from the SQLAlchemy types, producing `VARCHAR(50)` and `INTEGER` instead of `String` and `Int32`, and giving the aggregate column its declared type instead of the `AggregateFunction(...)` signature it needs to accept `-State` rows.
- **PostgreSQL roles are no longer lost when a table has a row-level security policy.** The policy loop bound its list of policy roles to the same name as the snapshot's role accumulator, so every later role lookup raised `TypeError` and the snapshot recorded no roles at all.
- **An aggregating view keeps a declared ClickHouse aggregate type.** Only the default `Float64` that `agg.sum()` / `agg.avg()` fall back to is replaced by the source column's resolved type; an explicit type such as `agg.raw("quantile(0.95)", col, "UInt32")` was being rewritten to the source column's `Int32`, changing the `AggregateFunction` signature.
- **Falling back to model-only generation is announced.** The snapshot diff failing used to be swallowed, leaving no way to tell a degraded migration from a correct one; it now logs the cause and warns.
- **A column-level `unique=True` produces one constraint, not two.** The column was rendered `UNIQUE` inline *and* emitted again as a named table constraint, leaving two unique constraints - and two indexes - on the same column.
- **A unique constraint dbwarden named itself no longer renames the database's.** When a model declares a unique constraint without a name, an existing constraint on the same columns - `users_email_key`, say - is now left alone instead of being renamed to `uq_users_email` and, in some orderings, dropped.
- **Column defaults are compared without their cast.** PostgreSQL reports a default as `'queued'::character varying` while the model spells it `'queued'`, so every run re-emitted the default and the rollback re-quoted the cast into `'''queued''::character varying'`.
- **Default index sort options are no longer recorded.** `ASC NULLS LAST` and `DESC NULLS FIRST` are what PostgreSQL implies; recording them made plain indexes differ from the models and be dropped and recreated on every run.
- **A column-level `unique=True` is read from every model.** The constraint was only collected when the model also declared a `class Meta`, so a diff against a database that had the constraint emitted a `DROP CONSTRAINT` for it.
- **Column storage is compared against the type's own default.** `NUMERIC` defaults to `MAIN` and `TEXT` to `EXTENDED`; recording those made untouched columns differ from the models.
- **Foreign keys are matched regardless of how the action is spelled.** `ondelete="cascade"` in a model and `CASCADE` in the database no longer read as different constraints.
- **Constraint canonicalization no longer edits the snapshot it was given.** The diff rewrote the caller's snapshot in place, so anything reading the snapshot after a diff saw constraints it could no longer classify.
- **Unnamed `CHECK` constraints get a stable key.** The generated name used `hash()`, which is salted per process, so the same schema produced a different snapshot on every run.
- **Configuration-declared objects are no longer recreated on every migration.** Roles, domains, sequences, functions, triggers, composite types, extended statistics, event triggers and default privileges were diffed against a hardcoded empty snapshot, so each generation emitted `CREATE` for objects that already existed - the second `migrate` failed with *role "..." already exists*, and an attribute change never became an `ALTER`. The preamble now diffs against the same snapshot as the rest of the migration; offline generation, which cannot see the server, still creates.
- **A SQLite table rebuild no longer leaves its staging table in the snapshot.** Pending-migration merging read every `CREATE TABLE` out of the migration text but ignored the `DROP`/`RENAME` that followed, so `t__dbw_new` survived as a table nothing declared and every later run generated a migration to drop it, in SQL built from columns typed `unknown`.
- **MySQL and MariaDB integer primary keys are `AUTO_INCREMENT`.** The same model produced `SERIAL` on PostgreSQL and `AUTOINCREMENT` on SQLite but a plain `id INTEGER NOT NULL PRIMARY KEY` here, so the migration applied and the first insert that omitted the key failed with *Field 'id' doesn't have a default value*.
- **Reverse-engineered models import `func`.** A column defaulting to `now()` was written as `default=func.now()` while `func` was filtered out of the generated import line, so the artifact raised `NameError` the moment dbwarden loaded it.
- **`generate-models` no longer writes an empty class body for SQLite.** Column metadata with nothing renderable emitted `class id(SqColumnMeta):` followed by nothing, and the generated module failed to import with *expected an indented block after class definition*.
- **SQLite migrations no longer fail the rollback contract.** A column type change or foreign key change on SQLite previously raised `RollbackContractError` because the generated rollback was a placeholder.
- **SQLite primary keys no longer churn.** A primary key column is recorded as `NOT NULL` when reading a SQLite schema, which SQLite itself reports as nullable, and autoincrement changes are no longer generated for SQLite primary keys where `AUTOINCREMENT` is either implicit or illegal.
- **Generated SQLite models import cleanly.** `generate-models` writes `sq = sq.field(...)` specs instead of flat `sq_*` attributes, which the metadata reader rejects.
- **Rebuilt tables keep their column order.** Reconstructing a table from model state preserved sorted order rather than declaration order.
- **A SQLite rebuild preserves what reflection cannot see.** Declared types keep their length and case (`VARCHAR(255)`, not `varchar`), and `AUTOINCREMENT`, foreign key `ON DELETE` / `ON UPDATE`, unnamed `UNIQUE (...)` constraints, partial indexes, `DESC` / `COLLATE` inside an index, and expression indexes all survive the rebuild.
- **SQLite partial indexes are recorded.** The index predicate was read only under PostgreSQL's dialect key, so `WHERE` was dropped from every SQLite index.

## \[0.19.0\] - 2026-09-03

### Added

- **V2 migration locking with per-engine strategies.** PostgreSQL advisory locks, MySQL named locks, SQLite `BEGIN IMMEDIATE`, and ClickHouse lease-based locking with configurable TTL. A heartbeat background task updates `last_heartbeat_at` on native-lock engines, enabling stale (STUCK) and dead worker detection.
- **`dbwarden lock-status` and `dbwarden unlock` commands.** Enhanced with detailed holder diagnostics including host, PID, execution ID, migration version, and health status.
- **Lock exceptions for structured error handling.** `LockAcquireTimeout`, `LockStuck`, and `RecoveryRequired` exceptions.
- **ClickHouse cluster configuration and CH-1 idempotency.** `clickhouse_lock_ttl` and `lock_namespace` config keys.
- **Merge handling pipeline.** `dbwarden merge` detects schema conflicts between branches and generates merge plans. `dbwarden rebase` replays migrations onto a new base. `dbwarden reconcile` resolves dirty environments with unreconciled merge changes.
- **Merge detection integration.** `dbwarden make-migrations` and `dbwarden status` surface pending merge conflicts during normal workflows.
- **`--all-environments` flag** for cross-environment merge operations.
- **MariaDB merge documentation.** Semantic conflict detection and model state reconstruction documented.
- **Documentation overhaul.** All documentation comprehensively revised: commands, configuration, database setup, advanced topics, correctness guarantees, plugins, and getting-started guide. ClickHouse documentation receives dedicated setup guides for CH-0 through CH-4.

### Fixed

- **Logger bug in `dbwarden unlock`.** Audit log call referenced an undefined variable.
- **PostgreSQL advisory lock key size and type casting.**
- **Critical audit findings in lock implementation.** Heartbeat timestamp generation for SQLite.
- **SQLite lock handling.** Reuses the `BEGIN IMMEDIATE` connection for migration execution.
- **Heartbeat task correctly skipped on SQLite.** Staleness is inferred from `acquired_at` and process liveness.

## \[0.18.0\] - 2026-09-01

### Added

- **Exception hierarchy.** New `dbwarden.exceptions` package organizes all exceptions under `DBWardenError`: `core.py` (13 exceptions), `plugin.py` (5 plugin exceptions), and `engine.py` (`OrderingError` and `RollbackContractError`).
- **Typed snapshot structures.** `dbwarden.engine.snapshot.types` introduces 26 TypedDicts covering the full snapshot hierarchy: `Snapshot`, `SnapshotTable`, `SnapshotColumn`, indexes, constraints, PG/MySQL/SQLite metadata, roles, grants, policies, sequences, functions, and event triggers.
- **Per-component CLI log filtering.** `--log-level COMPONENT:LEVEL` replaces the all-or-nothing debug flag.
- **`-h` shorthand for `--help`** across all CLI commands.

### Changed

- **`extract.py` module split.** The 1,500-line `extract.py` God Function is split into focused modules: `extract_common`, `extract_pg`, `extract_mysql`, and `extract_sqlite`. `extract.py` is now a thin dispatch module.
- **Connection layer rename.** `dbwarden.database` is renamed to `dbwarden.connection`. A backward-compatibility shim with `DeprecationWarning` preserves existing imports.
- **142 parametrised CLI option combination tests** covering every flag permutation.

### Fixed

- **ClickHouse immutable option detection.**
- **PostgreSQL enum, identity, and foreign key reverse-engineering gaps.**
- **PostgreSQL column statistics, storage, and compression metadata** now captured during snapshot extraction.
- **SQLite foreign key action fixups** handle edge cases that previously produced incorrect DDL.

## \[0.17.1\] - 2026-08-15

### Added

- **Declarative configuration as the default scaffold.** `dbwarden init` now generates `DbwardenDatabase` subclasses, while discovery supports inherited configuration, aliases, indirect imports, and the existing function-based API.
- **Migration convergence and replay hardening.** Migration generation and execution now share stronger convergence checks, replay handling, connection cleanup, and PostgreSQL column behavior coverage.

### Changed

- **Declarative configuration parity.** `DbwardenDatabase` now supports the complete `database_config(...)` field and default surface, inherited and direct plugin configuration, equivalent handles and validation, and aliased or indirect class discovery.
- **Generated files use atomic replacement.** Migration files, model state, generated models, rollback files, plugin state, and exported configuration now avoid leaving truncated files after an interrupted write.
- **Database settings mask URLs through SQLAlchemy URL parsing.** Settings and config output now handle encoded credentials and IPv6 hosts without exposing passwords.

### Fixed

- **Unsafe SQL identifiers are quoted or rejected.** Backend-generated schema, table, column, statistics, and drop-object SQL now protects reserved words and embedded identifier quotes.
- **Live command disconnects no longer report success.** Explicit `check` and `snapshot` operations propagate connection failures after retries.
- **Configuration and impact scans reject symlink escapes.** Workspace discovery no longer imports or scans symlinked files and directories outside the intended root.
- **Plugin provenance is HTTPS-only and size-bounded.** Plugin lock and consent TOML serialization escapes structural characters, and installer/provenance inputs are validated.
- **Migration SQL splitting respects quoted strings and comments.** Semicolons inside SQL literals and comments no longer split statements incorrectly.

## \[0.17.0\] - 2026-08-13

### Added

- **Global `--json` flag.** `dbwarden --json <command>` switches display commands (`status`, `history`, `database list`, `settings show`, `config`, `version`, `lock-status`) to structured JSON output and also routes the command's log output through JSON formatting (the same effect as `DBWARDEN_LOG_JSON=true`). Commands that already support JSON output (`check`, `check-db`, `check-impact`, `diff`, `plugin list`, `plugin info`) honor the global flag too.
- **TRACE log level.** A `trace` level (numeric `5`) is available via `--debug-level trace` and logs per-statement SQL during `migrate`, `rollback`, and `downgrade`.
- **`--perf` flag for migration commands.** `migrate`, `rollback`, `downgrade`, and `make-migrations` accept `--perf` to add per-SQL-statement timing breakdowns on top of the always-on phase timings (lock acquisition, snapshot write, model state write, and SQL statement durations).
- **Optional database availability handling.** Database entries support `skip_if_missing=True` for optional databases in multi-database operations. Use the global `--disable-skip` flag to force configured optional databases to fail normally.
- **Declarative database configuration.** Concrete subclasses of `DbwardenDatabase` are automatically registered and support inherited configuration, while the existing `database_config(...)` API remains supported.
- **Partial-success results.** Multi-database migration, status, and seed commands report skipped databases and use exit code `3` when the operation completes with optional databases unavailable.

### Fixed

- **ClickHouse DDL generation for fresh migrations.** `make-migrations` now emits ClickHouse DDL that applies cleanly to a fresh ClickHouse 24.x database: column comments are emitted before `CODEC(...)`, `Nullable` is nested inside `LowCardinality(...)` rather than the invalid reverse, and columns used in `ORDER BY` / `PRIMARY KEY` / `PARTITION BY` / `SAMPLE BY` are rendered as non-nullable.
- **Plugin list table rendering.** The `plugin list` table no longer wraps or truncates plugin distribution names in narrow terminals.
- **Rollback warning test stability.** The irreversible-rollback warning test now asserts against the log record instead of Rich console wrapping.

## \[0.16.5\] - 2026-08-05

### Added

- **Plugin-declared configuration keys.** Plugins can now register their own config keys and have them surface through the standard configuration path, so an installed plugin ships its settings surface instead of relying on loose user-side keys.

### Changed

- **`settings` output masks secret values.** The `settings` command now redacts values it recognises as credentials, so dumping configuration for a support ticket no longer prints a database password or API token.

### Fixed

- **MySQL diff for newly created tables.** A newly added table in a MySQL database no longer produces an empty or broken diff; the new-table path now emits the full create statement.
- **Publish and CI workflows hardened.** Release publishing and the CI pipeline were tightened to fail fast on the conditions that previously produced partial artifacts.

## \[0.16.4\] - 2026-08-05

### Added

- **`--debug` and `--debug-level` CLI flags.** Both flags enable per-file scan logging, so a slow or misbehaving `make-migrations` run can be traced down to the individual model file that caused it.

### Fixed

- **Config cache miss no longer rescans the whole workspace.** A config cache miss previously triggered a repeated full-workspace rescan; the cache now reloads in place.
- **Offline rollback no longer double-reverses.** The offline rollback path was reversing the same operation twice; the reversal is now applied exactly once, and agg-target key types resolve through the cascade chain.

## \[0.16.3\] - 2026-08-03

### Fixed

- **Materialized view drops use `DROP_VIEW` ordering.** Drops for cascading materialized views are emitted in `DROP_VIEW` order, matching how the views depend on one another.
- **`ch_raw` group-by column types resolve through the cascade chain.** A `ch_raw` view whose group-by keys come from an upstream view now resolves the types transitively instead of falling back to an unresolved state.

## \[0.16.2\] - 2026-07-29

### Fixed

- **Cascade materialized views resolve string forward references.** Cascading MVs that reference an upstream view by a forward string now resolve through the registry rather than failing at snapshot time.
- **Group-by key types fixed in cascading views.**
- **`ch_meta.ch_type` populated regardless of backend.** The `ch_type` metadata is now filled in even when the diff is running under a non-ClickHouse backend.
- **Config fallback now warns.** Falling back to a default backend or schema when a config value is missing now emits a warning instead of silently proceeding.

## \[0.16.1\] - 2026-07-28

### Added

- **Container test for cascade combinator correctness.** A live-ClickHouse test now verifies that cascading aggregate views emit the correct combinator.

### Changed

- **Cascading aggregate views use the `MergeState` combinator.** Cascades no longer emit `Sum` for upstream aggregate state; they use `MergeState` so the aggregate survives the cascade correctly.
- **`schemap` and `@auto_schema` moved to the `dbwarden-fastapi` plugin.** The schema-map integration is no longer part of core; install `dbwarden-fastapi` to keep using `@auto_schema`.

### Fixed

- **Three agg-target DDL bugs.** Aggregation target handling in the DDL layer emitted incorrect SQL in three edge cases.

## \[0.16.0\] - 2026-07-24

### Added

- **Plugin system with trust tiers, consent, and provenance verification.** dbwarden now discovers plugins through the `dbwarden.plugins` entry point group and classifies each distribution into official, verified, or community tiers. Plugin code from unverified sources is not imported until the operator explicitly consents to that exact version. See the [Plugins](plugins/) section for the trust model.
- **`recover-model-state` command.** Model state is now stored in the database as well as on disk, and `dbwarden recover-model-state` restores it when the on-disk file is lost or corrupt.
- **GPG-signed commit requirement.** Contribution guidelines now require GPG-signed commits.

### Changed

- **Remaining sandbox and five PostgreSQL extension handlers extracted to plugins.** Core no longer ships the sandbox provider or the PostgreSQL extensions, event triggers, functions, triggers, and storage-parameter handlers; they live in `dbwarden-sandbox` and `dbwarden-pgsql-extensions`.
- **Plugin-duplicated code stripped from core.** Every handler that now lives in a plugin was removed from core so there is a single implementation.
- **Normalized live snapshot diff.** Live-database snapshots now diff against the same normalized representation as stored snapshots.
- **PostgreSQL round-trip diff stabilized.**

### Fixed

- **Generated model metadata preserved.** Reverse-engineered models keep their detected backend metadata instead of dropping it during regeneration.

## \[0.15.0\] - 2026-07-21

### Added

- **Strict rollback contracts.** Every generated migration now has a declared rollback contract: executable rollback when it is safe, placeholder refusal by default, and an explicit irreversible declaration when rollback cannot be produced. See [Rollback Generation](correctness/rollback-generation.md).
- **Irreversible rollback annotations.** An operator can annotate a migration as irreversible, which dbwarden records and respects.
- **Rollback round-trip integration harness.** A test harness applies upgrade and rollback in sequence and verifies the schema lands back where it started. See [Round Trip Verification](correctness/round-trip-verification.md).
- **ObjectHandler protocol documented.**

### Changed

- **Rollback metadata preserved across diff pipelines.** Rollback information survives the snapshot, diff, and emission stages instead of being regenerated per stage.
- **Rollback state restored for ClickHouse and PostgreSQL.** RBAC, collection, profile, and policy rollback are restored for ClickHouse; PostgreSQL rollback state is restored as well.

## \[0.14.3\] - 2026-07-21

### Added

- **`recover-model-state` command.** Restores model state from the database when the on-disk state is missing or stale.

## \[0.14.2\] - 2026-07-21

### Fixed

- **`clickhouse_options` normalized to `ch_options`.** The snapshot pipeline now uses a single dict key, `ch_options`, so diffs no longer churn on naming alone.
- **TTL expression query conditional for ClickHouse before 24.4.** The `ttl_expression` query only runs against versions that expose the column, preventing failures on older servers.
- **Dead `ChRbacHandler` removed.**
- **Mobile sidebar drawer fixed.** The Zensical modern theme now uses a class-based drawer instead of `:has()`, and the viewport meta tag is present.

## \[0.14.1\] - 2026-07-20

### Changed

- **Docs aligned to the API.** Reference pages now match the exported surface, including the missing aggregation methods and RBAC class exports.

## \[0.14.0\] - 2026-07-20

### Changed

- **Core refactored to a single registry-driven protocol.** The twin registries were collapsed into one, the five backends were extracted into `dbwarden/databases/`, and every handler now speaks one protocol with `**kwargs` propagation and a `cluster_ctx` on the driver.
- **Config, commands, and FastAPI extracted into packages.** `config/`, `commands/`, and the FastAPI extension now live as packages rather than modules, which is the groundwork for the plugin system.
- **ClickHouse view API rewritten.** `ChView`, `MaterializedView`, and `AggregatingView` are now registry-backed non-SQLAlchemy base classes, with builder signatures (`to` instead of `to_table`), `_resolve_source`, and `AggregatingViewSpec` for correctness.
- **ClickHouse RBAC, named collections, and data operations.** New handlers and spec classes cover roles, users, grants, policies, quotas, settings profiles, named collections, partition operations, mutations, and `OPTIMIZE` via a `DataOp` type.
- **`render_expr()` and compiled-expression acceptance.** Expression sites now accept SQLAlchemy `ColumnElement` objects and render them through `render_expr()`.

### Added

- **`p0` and `integration` pytest markers.** `p0` marks the two-cycle convergence gate that must never regress; `integration` marks tests that need a live ClickHouse container.
- **Convergence-audit and ClickHouse integration CI jobs.**
- **ClickHouse documentation split into a multi-page reference** with dedicated pages for RBAC, views, dictionaries, and data operations.

## \[0.13.0\] - 2026-07-06

### Added

- **PyPI publishing workflow.** A `v*` tag now publishes the package to PyPI automatically.
- **SEO integration via seoslug 2.0.1.** The docs site generates canonical URLs, OG images, and schema-aware frontmatter through the inline Zensical extension.
- **Favicon, OG image, and robots.txt content-signal directives.**

### Fixed

- **Nine SQL generation bugs resolved** across the PostgreSQL expansion.
- **Schema package re-exports removed** so imports resolve against the single canonical path.

## \[0.12.5\] - 2026-07-01

### Changed

- **README version badge updated** and the publishing workflow prepared for the PyPI release.

## \[0.12.4\] - 2026-06-24

### Changed

- **`_write_model_state` skipped when no migrations applied.** Startup no longer burns CPU rewriting model state that did not change; this removes a multi-minute stall on every startup with nothing pending.

### Fixed

- **MySQL metadata preserved in generated models.**
- **Config and model caches refreshed** when the underlying files change.
- **Docs lint issues fixed.**

## \[0.12.3\] - 2026-06-23

### Fixed

- **`make-migrations` output deduplicated.**
- **MySQL DDL savepoint destruction handled** so a failed statement does not invalidate the transaction.
- **Primary key inferred for tables missing one**, and MySQL DDL translation improved.
- **Generated SQLite databases ignored** by version control.

## \[0.12.1\] - 2026-06-17

### Added

- **`--base` flag documented.** `dbwarden generate-models --base app.database:Base` imports a project base instead of declaring a new one.
- **Graceful database disconnection.** Connection failures now trigger retry logic and a clear error instead of a bare traceback.

### Changed

- **Schema backends moved to `dbwarden/databases/`.**
- **Safe SQL quoting and config cache fixes.** Duplicate model loads are avoided and default serialization is stable.

### Fixed

- **Live snapshot taken when no cached snapshot exists** for `make-migrations`.
- **Unique module name computed per file** in the sandbox loader, so two files with the same basename no longer collide.

## \[0.12.0\] - 2026-06-13

### Added

- **MySQL round-trip support.** MySQL schema classes, model discovery, a round-trip engine, and comprehensive test coverage, alongside the MySQL dependency group. See [MySQL](databases/mysql/index.md).
- **Seed export command.** `dbwarden export` writes code seeds to ROC SQL files.

### Changed

- **ClickHouse `ForeignKey` prohibited.** A foreign key on a ClickHouse table is now an error rather than silently unsupported SQL.
- **Snapshots moved to `.dbwarden/`**, and model state is synced online.
- **Auto-generated SQL for previously manual operations.** ClickHouse rename, nullable, `LowCardinality`, and projection changes, plus the PostgreSQL `USING` clause, are now emitted automatically.
- **Generated models emit `pg.field()` and `ch.field()` spec objects** instead of flat backend attributes.

## \[0.11.2\] - 2026-06-12

### Added

- **In-code seed engine.** `@seed_data`, `SeedRow`, and `DBWardenSeed` define seeds in Python; a tracking table records what has been applied.
- **`auto_apply_seeds` config** wired into `migrate`.
- **Checksum drift detection and seed pruning.**

### Fixed

- **Offline migration first run generates SQL for all tables**, not an empty set.
- **Seed types moved to the `dbwarden.seed` module** so the CLI and engine share one definition.

## \[0.11.0\] - 2026-06-11

### Added

- **Typed Meta system.** A `_MetaValidator` metaclass validates every attribute on `class Meta` at import time; a typo now raises `DBWardenConfigError` instead of silently producing wrong DDL. Per-backend field factories produce spec objects.
- **Offline v2 state format.** ClickHouse engine recreate and column rename flags are captured in the offline state.

### Changed

- **`requires-python` relaxed to `>=3.12`** and the upper bound removed, so Python 3.14 is supported.

## \[0.10.2\] - 2026-06-11

### Added

- **`model_tables` per-database filter** with overlap validation, so one model file set can be partitioned across databases with ownership checks.

## \[0.10.1\] - 2026-06-11

### Fixed

- **Config sandbox classification for in-package and `src/` layout projects.** Model files discovered inside the package directory or under `src/` are now classified correctly.

## \[0.10.0\] - 2026-06-09

### Added

- **PostgreSQL auto-increment lifecycle support.** Serial, identity, and sequence lifecycle are handled end to end, including rename and rollback.
- **`--type` flag for repeatable migrations.**

### Changed

- **Real `diff` implemented**, replacing the placeholder, and obsolete CLI commands removed.
- **Lock system overhauled.** Migration locking, connection safety, and sandbox fixes landed as a mass bug-fix pass (38+ fixes).

### Fixed

- **`alter_column_type` rollback and `drop_table` placeholder fixed**, with the full rollback cycle verified.
- **Rollback SQL for ClickHouse indexes and foreign key constraints fixed**, and the `make-rollback` regex corrected.
- **System tables excluded from diffs** and rollback counts corrected.
- **Offline migration engine fixed** for compatible operations and a wrong import, with 35 comprehensive tests; 26 edge-case tests added, and the crash on corrupted state resolved.

## \[0.9.5\] - 2026-06-09

### Added

- **`--type` flag for repeatable migrations.**

### Changed

- **Documentation updated** for the new flag and related CLI changes.

## \[0.9.4\] - 2026-06-09

### Added

- **Cookbook docs, examples, and an integration test suite.**

### Fixed

- **Six bugs fixed.**
- **Missing import in `rollback.py` fixed** (`get_database`).
- **Comments added to every example.**

## \[0.9.0\] - 2026-06-09

### Fixed

- **Offline migration engine fixed** for compatible operations and the wrong import path, with 35 comprehensive tests.

### Changed

- **Docs restructured.** Index and introduction merged, models/modeling split into separate references, squashing folded into the squash page, and the navigation rebuilt.

## \[0.8.0\] - 2026-04-26 through 2026-06-09

This window aggregated the 0.8 and 0.9 development lines into core; the version was not bumped per feature.

### Added

- **Schema snapshots for rename detection.** A checksummed JSON snapshot is written after every migration and powers rename detection without querying the live database. See [Deterministic Diff](correctness/deterministic-diff.md).
- **Column-level diff with `StatementOrder` and `--safe-type-change`.** Type, nullability, default, and comment changes produce precise `ALTER COLUMN` statements, and type changes require the flag.
- **Table and column rename detection** with an interactive prompt and a rename flag.
- **Richer index metadata.** Partial and covering indexes, `USING` access methods, `WITH` storage parameters, `WHERE`, `TABLESPACE`, `NULLS NOT DISTINCT`, per-column sort order, and ClickHouse skip indexes, plus a `--concurrent` flag.
- **`DatabaseHandle`** returned by `database_config()`, with `database_url_sync` and `database_url_async` split into separate settings and a unified async engine dispatcher.
- **Sandbox providers and `--dry-run`.** `dbwarden migrate --dry-run --sandbox` replays migrations against SQLite or a testcontainers provider.
- **ClickHouse materialized views, projections, and a safety analyzer**, wired into the new `check` command.
- **ClickHouse replicated engines and external dictionaries.**
- **Versioned seed management** with CLI commands and a configurable tracking table.
- **Prometheus metrics and JSON logging**, plus FastAPI Redis lock, status, and migrate endpoints and a `dbwarden_lifespan` context manager. See [Observability](observability.md).
- **Migration impact analysis** with the `check-impact` command, using AST analysis with a grep fallback.
- **Offline migrations.** `export-models` dumps model state; `make-migrations --offline` generates SQL with no database connection. See [Offline Integrity](correctness/offline-integrity.md).
- **In-code seed definitions** (`@seed_data`, `SeedRow`, `DBWardenSeed`).
- **Class Meta metadata foundation**, with `IndexSpec`, `CheckSpec`, `UniqueSpec` dataclasses and factory functions.
- **PostgreSQL first-class support with zero-diff round-trip**, including unlogged tables, `no_inherit`, deferred uniques, `tsvector`, `enum ADD VALUE`, partitioning, and the typed `@auto_schema` wrapper.
- **ClickHouse first-class Meta** with `ChEngineSpec`, `ChIndexSpec`, and spec serialization.

## \[0.7\] - 2026-04-26

### Added

- **Auto-generated migration names** when no description is provided, with descriptive names derived from the change.
- **Migration plan output** and configurable migration tracking tables.
- **Health router hardening.** Authentication via the `DBWARDEN_HEALTH_AUTH` environment variable, sanitized error responses, and validation of identifiers and paths against strict patterns.

### Changed

- **Migrations now use savepoints.** Any statement failure rolls back all statements in the migration.
- **Atomic writes with backups.** Config and snapshot writes are atomic and validated before replace, with the backup directory checked first.

### Fixed

- **No spurious backup on initial `init`.**
- **`RestrictedFileLoader` added** so sandboxed config loads cannot escape the project root.

## \[0.6\] - 2026-04-24

### Added

- **FastAPI integration.** Session dependencies, health router, and startup helpers.
- **`settings` command group and mutators.**
- **Rich colorized CLI output** with command accents.

### Changed

- **Python-based `database_config` runtime** replaced the TOML-only config surface.
- **Docs restructured** into Getting Started, Tutorial, Advanced, and Reference tiers.

## \[0.5\] - 2026-04-23

### Changed

- **Documentation updates** across the site.

## \[0.4\] - 2026-04-23

### Added

- **SQLite translation layer with strict mode support.** PostgreSQL-flavored models run against SQLite locally, with a `--strict` flag enforcing translation correctness.
- **Global `--dev` mode** and database URL and target uniqueness validation.

## \[0.3.6\] - 2026-04-14

### Changed

- **Checksum-based migration tracking.** Migrations are checked by checksum instead of version, so deployments are idempotent.
- **Migrations table keyed by filename** (auto-increment id removed) with a UNIQUE constraint on the version column for PostgreSQL `ON CONFLICT` support.
- **ClickHouse-specific SQL queries** for migration tracking.

## \[0.3.2\] - 2026-04-10

### Fixed

- **Migration filename uses the config section name** instead of the URL.
- **Logging and type mapping improved** across the command layer.

## \[0.3.1\] - 2026-04-10

### Fixed

- **Logging and type mapping improved** across the command layer.

## \[0.3.0\] - 2026-04-10

### Added

- **ClickHouse support.** URL conversion and the `clickhouse-connect` dependency.
- **Multi-database support.** An explicit `database_type` field, a multi-database config structure, `--database` and `--all` CLI options, and a `database` management command. See [Multi-Database](configuration/multi-database.md).

### Changed

- **Backend-specific internal migration SQL** for PostgreSQL, MySQL, and SQLite.

### Fixed

- **Duplicate migrations fixed.**

## \[0.2.0\] - 2026-03-04

### Changed

- **Icon and banner added**, and tests and docs updated for the rebrand.

## \[0.1.5\] - 2026-02-27

### Added

- **ALTER TABLE support.**

### Changed

- **`.env` alternatives and async mode removed** in favour of a single sync path.
- **Diffs compare against the database schema** instead of the migrations table.

### Fixed

- **Async mode detection and duplicate connection logs** in the connection layer.

## \[0.1.3\] - 2026-02-16

### Added

- **`warden.toml` documented.**

## \[0.1.2\] - 2026-02-10

### Added

- **Issue templates and CONTRIBUTING.md.**

### Changed

- **Config moved from `.env` to `warden.toml`.**

### Fixed

- **`CallableColumnDefault` handled** so SQLite no longer emits syntax errors for callable defaults.

## \[0.1.1\] - 2026-02-08

### Added

- **Baseline migrations** and automatic backup before migration.
- **Migration dependency support.**
- **Colored output with SQL syntax highlighting.**

### Changed

- **Checksum-based deduplication** for `RA__` and `ROC__` migrations.
- **Auto-discovery scans all subdirectories** for model files.

### Fixed

- **"Table already exists" errors fixed** by adding `IF NOT EXISTS` to `CREATE TABLE` statements.
- **Migration deduplication and pending-migration detection fixed.**

========================================================================
PAGE: https://docs.dbwarden.org/cli-reference/
========================================================================

# CLI Reference

Pure command lookup for dbwarden CLI. The [CLI option inventory](reference/cli-options.md) lists every built-in argument, option, type, and default from the current CLI.

## Safety-Scoped Options

| Command | Option | Effect |
|---|---|---|
| `make-migrations` | `--split-at-severity LEVEL` | Defer operations at or above LEVEL plus dependent operations |
| `make-migrations` | `--strict-pending / --no-strict-pending` | Override refusal of pending files without composable plans |
| `make-migrations` | `--dry-run` | Preview artifacts without writing |
| `migrate` | `--max-severity LEVEL` | Apply a strict version prefix; severity stop exits 3 |
| `migrate` | `--force` | Acknowledge risks without changing the ceiling |
| `check` | `--write-plan [--all] [VERSION]` | Statically classify SQL; unresolved files exit 4 |
| `merge` | `--dry-run`, `--split-at-severity LEVEL` | Preview or split reconciliation |
| `reconcile` | `--force`, `--split-at-severity LEVEL` | Acknowledge and split environment repair |
| `rebase` | `--dry-run` | Alias of `--check`; no mutation |

LEVEL is `SAFE`, `INFO`, `WARN`, or `CRITICAL`. See [Safety-scoped migrations](correctness/safety-scoped-migrations.md) for plans, UNKNOWN handling, configuration, and exit-code compatibility.

## Syntax

```bash
$ dbwarden [GLOBAL_OPTIONS] COMMAND [ARGS] [COMMAND_OPTIONS]
```

## Global options

| Option | Description |
|---|---|
| `--dev` | Use `dev_database_url` and `dev_database_type` for selected database |
| `--strict-translation` | Fail on unsupported/lossy dev SQLite translation |
| `--disable-skip` | Do not skip databases configured with `skip_if_missing` |
| `--debug` | Enable DEBUG-level logging (shows per-file model scanning on `make-migrations`) |
| `--debug-level <LEVEL>` | Set an exact log level: `trace`, `debug`, `info`, `warning`, `error`, `critical`, or `5`/`10`/`20`/`30`/`40`/`50` |
| `--log-level <COMPONENT:LEVEL>` | Per-component log level (repeatable). Example: `--log-level snapshot:debug` |
| `--json`/`-j` | Emit structured JSON for display commands and JSON-formatted logs |
| `--help` | Show help |

`--debug`/`--debug-level` set the logging severity and compose with the
per-command `--verbose`/`-v` flag, which controls INFO-level verbosity. Use both
(`--debug` plus `-v`) for DEBUG diagnostics with verbose INFO output. If both
`--debug` and `--debug-level` are given, `--debug-level` wins. `trace` (or `5`)
is below `debug` and additionally logs per-statement SQL during migration
commands.

`--json` switches display commands (`status`, `history`, `database list`,
`settings show`, `config`, `version`, `lock-status`) to machine-readable JSON
and also formats the command's log output as newline-delimited JSON (the same
effect as `DBWARDEN_LOG_JSON=true`). Commands with their own format option
(`check`, `check-db`, `check-impact`, `diff`, `plugin list`, `plugin info`)
honor the global flag when no explicit format is given.

## Configuration

### `settings show`

```bash
$ dbwarden settings show
$ dbwarden settings show primary
$ dbwarden settings show --all
```

### `database list`

```bash
$ dbwarden database list
```

## Migration authoring

### `make-migrations`

```bash
$ dbwarden make-migrations "create users table" --database primary
$ dbwarden make-migrations --verbose --database primary
$ dbwarden --debug make-migrations --database primary
$ dbwarden --debug-level warning make-migrations --database primary
$ dbwarden make-migrations --plan --database primary
$ dbwarden make-migrations --rename users.username:email --database primary
$ dbwarden make-migrations --rename-table users:accounts --database primary
$ dbwarden make-migrations --safe-type-change --database primary
```

Options:

- `--database`/`-d`: Target database
- `--plan`: Print migration plan JSON without writing files
- `--sql`: Output raw migration SQL to stdout without writing files
- `--offline`: Use model state file instead of live database (run `export-models` first)
- `--verbose`/`-v`: Verbose output
- `--debug`/`--debug-level <LEVEL>`: Global options (see Global options table). DEBUG shows every model file scanned during discovery.
- `--rename`: Repeatable. Declare a column rename in format `table.old_name:new_name`.
- `--rename-table`: Repeatable. Declare a table rename in format `old_table:new_table`.
- `--safe-type-change`: Multi-step safe type change strategy.
- `--concurrent`/`--no-concurrent`: Use concurrent index creation (default: on).
- `--clickhouse-engine-recreate`: Allow automatic ClickHouse table rebuild on engine change.
- `--drop-preserved-clickhouse-table` / `--keep-preserved-clickhouse-table`: Drop or keep the preserved old ClickHouse table after engine-recreate swap.
- `--postgres-auto-using`: Emit active `USING` clause on PostgreSQL `ALTER COLUMN TYPE` (default: commented out).
- `--type`/`-t`: Output prefix: `versioned` (default), `ra`/`runs_always`, or `roc`/`runs_on_change`.
- `--perf`: Log SQL-generation phase timing.

See [make-migrations](commands/make-migrations.md) for full documentation including rename detection, column-level changes, schema snapshots, and plan format.

### `new`

```bash
$ dbwarden new "manual hotfix" --database primary
$ dbwarden new "backfill" --database primary --version 0042
$ dbwarden new "seed data" --database primary --type ra
```

Options: `--database`, `--version`, `--type`/`-t`

### `generate-models`

```bash
$ dbwarden generate-models --output ./models/ --database primary
$ dbwarden generate-models --database primary --single-file
$ dbwarden generate-models --database primary --tables users,posts
$ dbwarden generate-models --database primary --exclude-tables logs,audit
```

Options: `--output`/`-o` (default `models`), `--tables`, `--exclude-tables`, `--clickhouse-engines`, `--relationships`, `--dialect`, `--single-file`, `--base`, `--database`/`-d`

### `export-models`

```bash
$ dbwarden export-models --database primary
$ dbwarden export-models --database primary --output .dbwarden/model_state.json
```

Exports current model definitions to a JSON state file for offline migration diffs.

> **Important:** The model state file is used for offline migration generation. It is auto-generated and committed to version control. If accidentally deleted, restore it from git or regenerate it by running `dbwarden export-models --database <db>` against a live database. Without it, offline commands like `make-migrations --offline` will not work, but online operations are unaffected.

Options: `--output`/`-o` (default `.dbwarden/model_state.json`), `--database`/`-d`

### `diff`

```bash
$ dbwarden diff --database primary
$ dbwarden diff --database primary --out json
$ dbwarden diff --database primary --out sql
$ dbwarden diff --database primary --offline
```

Read-only model-vs-database comparison. No files are written.

Options: `--database`/`-d`, `--out`/`-o` (`table`, `json`, `sql`), `--offline`, `--verbose`/`-v`

### `check-impact`

```bash
$ dbwarden check-impact 0042 --database primary
$ dbwarden check-impact 0042 --database primary --out json
$ dbwarden check-impact 0042 --database primary --scan-path app/
$ dbwarden check-impact path/to/primary__0042_add_bio.plan.json
```

Scans your codebase for references to schema elements affected by a migration.

| Option | Description |
|--------|-------------|
| `migration` | Migration version (e.g. `0042`) or plan file path (required) |
| `--out`/`-o` | Output format: `text` (default) or `json` |
| `--scan-path` | Directory to scan for affected code (default: `.`) |
| `--deep` | Enable deep introspection (imports models live) |
| `--verbose`/`-v` | Include INFO-level operations in the scan |
| `--database`/`-d` | Target database name |

## Migration execution

### `migrate`

```bash
$ dbwarden migrate --database primary
$ dbwarden migrate --all
$ dbwarden migrate --database primary --to-version 0010
$ dbwarden migrate --database primary --count 2
$ dbwarden migrate --database primary --with-backup
$ dbwarden migrate --database primary --baseline --to-version 0005
```

Options:

- `--database`, `--all`
- `--to-version`, `--count`
- `--baseline`
- `--with-backup`, `--backup-dir`
- `--dry-run` (show what would be applied without executing)
- `--sandbox` (apply in a temporary sandbox database)
- `--apply-seeds` (apply pending seeds after migrations, overrides config)
- `--defer-snapshots` (write one final schema snapshot instead of one after every migration)
- `--verbose`
- `--perf` (log per-SQL-statement timing breakdowns)

### `rollback`

```bash
$ dbwarden rollback --database primary
$ dbwarden rollback --database primary --count 2
$ dbwarden rollback --database primary --to-version 0007
```

Options: `--database`, `--count`, `--to-version`, `--verbose`, `--perf`

### `downgrade`

```bash
$ dbwarden downgrade --to 0005 --database primary
```

Options: `--to` (required), `--database`, `--verbose`, `--perf`

### `make-rollback`

```bash
$ dbwarden make-rollback migrations/primary__0005_add_table.sql
```

Generates a `.rollback.sql` file for the given migration file.

### `snapshot`

```bash
$ dbwarden snapshot users --database primary
```

Outputs the DDL schema of the specified table.

## Seed management

### `seed create`

```bash
$ dbwarden seed create "seed initial data" --database primary
$ dbwarden seed create "populate lookup tables" --database primary --type python
```

Options: `--database`, `--type` (`sql` or `python`, default `sql`), `--verbose`

### `seed apply`

```bash
$ dbwarden seed apply --database primary
$ dbwarden seed apply --database primary --version 0003
$ dbwarden seed apply --database primary --dry-run
$ dbwarden seed apply --all
```

Options: `--database`, `--all` (`-a`), `--version`, `--dry-run`, `--verbose`

### `seed list`

```bash
$ dbwarden seed list --database primary
$ dbwarden seed list --all
$ dbwarden seed list --prune
```

Options: `--database`, `--all`, `--prune`, `--verbose`

### `seed rollback`

```bash
$ dbwarden seed rollback --database primary
$ dbwarden seed rollback --database primary --count 2
$ dbwarden seed rollback --database primary --to-version 0003
$ dbwarden seed rollback --all
```

Options: `--database`, `--count`, `--to-version`, `--all`, `--verbose`

### `seed export`

```bash
$ dbwarden seed export --database primary
$ dbwarden seed export --all
$ dbwarden seed export --database clickhouse --output-dir ./seeds
```

Export code seeds to ROC SQL files for stateless production application.

Options: `--database`/`-d`, `--all`/`-a`, `--output-dir`/`-o` (default `seeds/`)

## Inspection and diagnostics

### `status`

```bash
$ dbwarden status --database primary
$ dbwarden status --all
$ dbwarden status --all-environments
```

### `history`

```bash
$ dbwarden history --database primary
```

### `check-db`

```bash
$ dbwarden check-db --database primary
$ dbwarden check-db --database primary --out json
```

Output formats: `txt`, `json`, `yaml`, `sql`

### `check`

```bash
$ dbwarden check --database primary
$ dbwarden check --database primary --force
$ dbwarden check --database primary --data
$ dbwarden check --database primary --out json
$ dbwarden check --write-plan 0042 --database primary
$ dbwarden check --write-plan --all
```

Output formats: `txt`, `json`

The live command combines model-versus-database findings with the safety of SQL that can still execute: unapplied versioned files, all runs-always files, and runs-on-change files whose checksum changed since their recorded execution. Superseded files are excluded. A trusted checksum-bound plan supplies the file's classification when present; otherwise dbwarden classifies the SQL in memory without writing a sidecar. Pending `WARN` and `CRITICAL` operations require `--force`. An incompletely classified file is `UNKNOWN` and always blocks, even with `--force`. Declarative-data drift requested through `--data` also cannot be forced.

## Locking

### `lock-status`

```bash
$ dbwarden lock-status --database primary
```

### `unlock`

```bash
$ dbwarden unlock --database primary
```

## Merge handling

### `merge`

```bash
$ dbwarden merge --database primary
```

### `rebase`

```bash
$ dbwarden rebase --database primary
```

### `reconcile`

```bash
$ dbwarden reconcile staging
```

## Plugin management

See the [Plugins guide](plugins/index.md) for the trust model and development docs.

### `plugin list`

```bash
$ dbwarden plugin list
$ dbwarden plugin list --format json
```

Shows discovered plugins with tier, trust/load state, registered hooks, object handlers, and lock status. Use `--load` to import trusted plugins without prompting and inspect their registered handlers and migration categories. Default inspection does not import plugins.

Options: `--format`/`-f` (`table` or `json`, default `table`), `--load` (import trusted plugins before inspection)

### `plugin info`

```bash
$ dbwarden plugin info dbwarden-fastapi
$ dbwarden plugin info dbwarden-fastapi --format json
```

Shows entry point, tier, trust/load state, hooks, official repository, approved minimum version, and lockfile provenance. Exits `1` if the plugin is not found. Accepts `--load`; JSON includes `migration_categories`.

Options: `--format`/`-f` (`table` or `json`, default `table`)

### `plugin add`

```bash
$ dbwarden plugin add dbwarden-fastapi
$ dbwarden plugin add dbwarden-fastapi --version 0.2.0 --uv
$ dbwarden plugin add dbwarden-example --dry-run
```

Installs a plugin. Official plugins are provenance-verified and fail closed if verification is unavailable; community plugins are installed but not trusted (run `plugin trust` next).

Options: `--uv` (use `uv add` instead of pip), `--version` (pin an exact version), `--dry-run` (print the plan without installing)

### `plugin remove`

```bash
$ dbwarden plugin remove dbwarden-example
$ dbwarden plugin remove dbwarden-example --dry-run --uv
```

Uninstalls the distribution and removes its consent and lockfile entries.

Options: `--uv` (use `uv remove` instead of pip), `--dry-run` (print the plan without uninstalling)

### `plugin trust`

```bash
$ dbwarden plugin trust dbwarden-example
```

Records consent for the installed version of a community plugin in `.dbwarden/consent.toml`. Consent is version-specific.

### `plugin untrust`

```bash
$ dbwarden plugin untrust dbwarden-example
```

Revokes consent for a community plugin.

### `plugin categories`

```bash
$ dbwarden plugin categories --format json
```

Loads trusted plugins without prompting and lists built-in/custom migration groups with `name`, `order`, and owning `plugin`. Default output is a table. `status` also displays each generated file's category.

## Utility

### `recover-model-state`

```bash
$ dbwarden recover-model-state --database primary
```

Recovers a deleted model state file by replaying migrations in a sandbox.

Options: `--database`/`-d`

### `config`

```bash
$ dbwarden config
```

### `version`

```bash
$ dbwarden version
```

For worked command examples, see the [Cookbook & Examples](cookbook/index.md).

========================================================================
PAGE: https://docs.dbwarden.org/codebase/
========================================================================

# Codebase Organization

## Top-Level Layout

```
dbwarden/        # The package itself
tests/           # Test suite (~40 modules)
docs/            # Documentation site (MkDocs)
examples/        # Runnable example projects
scripts/         # Development and CI tooling
assets/          # Images, icons, branding
site/            # Built documentation output (gitignored)
```

## Package Layout (`dbwarden/`)

| Directory / Module | Responsibility |
|---|---|
| `cli/` | Typer CLI definitions, argument parsing |
| `commands/` | Command orchestration (migrate, generate-models, check, etc.) |
| `engine/` | Core logic: model discovery, snapshot extraction, diff, offline migration, safety checks |
| `database/` | Connection management, SQL queries by dialect |
| `databases/` | Concrete backend specs: ClickHouse, MySQL, PostgreSQL, MariaDB, SQLite |
| `schema/` | Dialect-agnostic metadata layer: table/column/field metadata classes |
| `repositories/` | Migration and lock metadata persistence |
| `config*.py` | Configuration loading and resolution |
| `constants.py` | Shared constants |
| `exceptions/` | Exception hierarchy |
| `seed.py` | Seed data infrastructure |
| `engine/sandbox.py` | Module loading sandbox for user model files |

## The `schema/` vs `databases/` Boundary

The `dbwarden/schema/` package is the abstract metadata layer. It defines dialect-agnostic constructs that make no assumptions about the target database:

- `TableMeta` and `*ColumnMeta` classes (e.g. `PGColumnMeta`, `CHColumnMeta`, `MyColumnMeta`)
- `DBWardenMeta`: the runtime metadata container attached to each model
- `_MetaValidator`: metaclass that validates `class Meta` attribute names at import time
- `IndexSpec`, `CheckSpec`, `UniqueSpec`: cross-database object specs
- `_meta_reader.py`: logic that reads `class Meta` from user models and populates `DBWardenMeta`

The `dbwarden/databases/` package is the concrete backend layer. It contains dialect-specific specs and helpers:

- `clickhouse/`: `ChEngineSpec`, `ProjectionSpec`, `ChIndexSpec`, `ChTableSpec`, merge-tree helpers, `ChFieldSpec`
- `mysql/`: `MyFieldSpec`, `MyTableSpec`
- `pgsql/`: `PgFieldSpec`, `PgIndexSpec`, `PgTableSpec`, exclude/partition helpers
- `mariadb/`: `MdbFieldSpec`, `MdbTableSpec`
- `sqlite/`: `SqFieldSpec`, `SqTableSpec`, re-exported `SqTableMeta` / `SqColumnMeta`

### The Import Contract

The single most important rule in the codebase is:

> **`schema/` must never import from `databases/`.**

This keeps the metadata layer database-agnostic. `databases/` may import from `schema/` (and does, for `TableMeta`, `DBWardenMeta`, `IndexSpec`, etc.), but the reverse dependency is forbidden.

Consequences of this boundary:

- **`ChEngineSpec` and `ProjectionSpec` live in `databases/clickhouse/`**, not `schema/`. They are ClickHouse-specific types, not abstract schema concepts.
- Backend specs (`ChTableSpec`, `MyTableSpec`, etc.) are defined per-database, not in `schema/`.
- The `schema/__init__.py` only re-exports classes from `schema/` submodules. It does not re-export backend-specific types from `databases/`.
- Users import backend types through `from dbwarden.databases.clickhouse import ChEngineSpec` or the top-level `from dbwarden import ChEngineSpec`.

### What Changed in the v0.13.0 Refactor

The refactor tightened this boundary. Previously, `ChEngineSpec`, `ProjectionSpec`, and the `*FieldMeta` hierarchy lived in `schema/`. They were moved to their correct locations:

- `ChEngineSpec`, `_split_engine_args`: now in `databases/clickhouse/engine.py`
- `ProjectionSpec`: now in `databases/clickhouse/projection.py`
- `*FieldMeta` classes (`PGFieldMeta`, `CHFieldMeta`, etc.): deleted; fields inlined directly into `*ColumnMeta` in `table_meta.py`

The orphan `__pycache__` directories under `schema/{clickhouse,mysql,pgsql,mariadb,sqlite}/` were removed.

## Contribution Guidelines

### Before submitting a PR

1. Ensure your changes respect the `schema/` vs `databases/` import boundary (see above).
2. Run the full test suite before pushing:
   ```
   python -m pytest tests/ -x -q
   ```
3. If you add or remove a public export, update the corresponding `__all__` list in the module's `__init__.py`.
4. If you introduce a new top-level directory, add it to the table in this document.
5. If you change public APIs or CLI options, regenerate the inventories and run the docs audit:
   ```bash
   python scripts/check-docs.py --api-reference > docs/reference/python-api.md
   python scripts/check-docs.py --cli-reference > docs/reference/cli-options.md
   python scripts/check-docs.py
   python scripts/generate_llms_full.py
   zensical build --strict
   ```
   The inventories come from local source and CLI registration. The audit checks exports, examples, command flags, links, and navigation. It does not execute external-service examples; run the relevant integration tests separately.

### Code style

- No comments in production code unless the logic is genuinely subtle.
- Mimic existing patterns: same typing style, same docstring conventions, same import organization.
- Prefer `from __future__ import annotations` at the top of every module.
- Use `metaclass=_MetaValidator` for any new `class Meta`-like user-facing configuration class.

### Adding a new database backend

1. Create a new subpackage under `databases/<name>/` with `__init__.py`, `field.py`, and any backend-specific specs.
2. Define a `*TableSpec` dataclass and a `*FieldSpec` dataclass matching the existing backends.
3. Register the backend in `databases/__init__.py` and add the shortcut import (`sq`, `my`, etc.).
4. If the backend needs no column-level `Meta` attributes, add no `*ColumnMeta` class.
5. Do not touch files in `schema/` unless you are adding cross-database metadata fields.

========================================================================
PAGE: https://docs.dbwarden.org/commands/check-db/
========================================================================

# `check-db`

Inspect live database schema.

## Usage

```bash
$ dbwarden check-db --database primary
$ dbwarden check-db --database primary --out json
$ dbwarden check-db --database primary --out yaml
```

## Options

- `--database`, `-d`
- `--out`, `-o` (`txt`, `json`, `yaml`)

## Notes

- useful for schema inspection and diagnostics
- complements `status` and `history`

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/check-impact/
========================================================================

# check-impact

Scan your codebase for references to schema elements affected by a migration.

## Usage

```bash
dbwarden check-impact 0042 --database primary
dbwarden check-impact 0042 --database primary --out json
dbwarden check-impact 0042 --database primary --scan-path app/
dbwarden check-impact path/to/primary__0042_add_bio.plan.json
```

## What it does

`check-impact` reports which code references would be affected by a migration before you apply it. It uses AST analysis with a grep fallback, so results reflect actual code structure rather than text matches.

Run this before any destructive deploy (column drops, type changes, table renames) to surface breaking changes before they reach production.

## Options

| Option | Description |
|--------|-------------|
| `migration` | Migration version (e.g. `0042`) or path to a plan JSON file (required) |
| `--database`/`-d` | Target database name |
| `--out`/`-o` | Output format: `text` (default) or `json` |
| `--scan-path` | Directory to scan for affected code (default: `.`) |
| `--deep` | Enable deep introspection (imports models live) |
| `--verbose`/`-v` | Include INFO-level operations in the scan |

## Output

```
drop_column on users.username
  References: 2
    app/routes/users.py:34  attribute_access
      .username
    app/templates/profile.jinja2:12  grep
      user.username
```

Each affected operation lists the schema element, the number of references found, and the specific files and line numbers where those references occur.

## See also

- [check](check.md) - Classify operations by danger level
- [Safety & Impact cookbook](../cookbook/06-safety-impact.md) - Practical examples

========================================================================
PAGE: https://docs.dbwarden.org/commands/check/
========================================================================

# `check`

Analyze schema differences between your SQLAlchemy models and the live database.

`check --write-plan --all` classifies existing SQL without a database connection. Scope with `--database primary`, or replace `--all` with a version. Trusted plans remain unchanged; generated plans are never overwritten. Any unmapped statement leaves its whole file UNKNOWN and makes the command exit **4**. The report lists severity totals, unresolved reasons, and CRITICAL files for review. Live check output also names matching generated migration files, including deferred siblings. See [Safety-scoped migrations](../correctness/safety-scoped-migrations.md).

The live check also evaluates SQL that is still eligible to run: unapplied versioned migrations, every runs-always migration, and runs-on-change migrations whose checksum differs from the recorded checksum. Superseded files are ignored.

For each pending file, dbwarden uses its trusted, checksum-bound safety plan when one exists. A handwritten file without a trusted plan is classified in memory by the read-only SQL classifier; `check` does not create or update a sidecar. If the classifier cannot map the file completely, the result is `UNKNOWN` and the check fails. `--force` cannot bypass an `UNKNOWN` result.

## Usage

```bash
$ dbwarden check --database primary
$ dbwarden check --database primary --force
$ dbwarden check --database primary --data
$ dbwarden check --database primary --out json
$ dbwarden check --write-plan 0042 --database primary
$ dbwarden check --write-plan --all
```

## Options

- `--database`, `-d` - Target database
- `--out`, `-o` - Output format: `txt` or `json`
- `--force` - Acknowledge warning- and critical-level findings; it cannot bypass `UNKNOWN` SQL or declarative-data drift
- `--data` - Add live declarative-data convergence checks
- `--write-plan` - Statically classify SQL and write checksum-bound plan sidecars without connecting to a database
- `VERSION` - With `--write-plan`, classify one version
- `--all` - With `--write-plan`, classify all migration files

See the generated [CLI option inventory](../reference/cli-options.md) for the authoritative option types and defaults.

## Severity model

- `INFO` - safe changes like adding a projection or adding a new object
- `WARNING` - risky changes, including pending `WARN` operations; requires `--force`
- `ERROR` - critical changes, including pending `CRITICAL` operations; requires `--force`
- `UNKNOWN` - SQL that could not be classified completely; always blocks

## Current behavior

dbwarden runs generic safety checks for all backends, covering column type changes, nullability changes, default changes, and table operations. For ClickHouse specifically, additional checks classify changes for:

- added or removed columns
- type changes
- engine changes
- TTL changes
- `ORDER BY` changes
- `PARTITION BY` changes
- materialized view query changes
- projection additions/removals

## Notes

- warning-level changes exit non-zero unless `--force` is provided
- critical/error-level findings require `--force`; data drift remains blocking
- unknown pending SQL remains blocking even with `--force`
- `check --data` succeeds only when the schema, pending SQL, and declared data all pass; `--force` does not convert data drift into success
- output combines live database inspection, current model metadata, and pending migration-file safety

========================================================================
PAGE: https://docs.dbwarden.org/commands/database/
========================================================================

# `database`

Display configured databases. Config is defined in Python code via declarative
`DbwardenDatabase` classes or `database_config()`, so
`database list` is a read-only command for viewing what's registered.

## Usage

```bash
$ dbwarden database list
```

## See also

- [`settings show`](./settings.md): detailed view of all configuration
- [Configuration docs](../configuration/index.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/diff/
========================================================================

# `diff`

Show structural differences between SQLAlchemy models and a live database.
Read-only: no files are written.

## Usage

```bash
$ dbwarden diff --database primary
$ dbwarden diff --database primary --out json
$ dbwarden diff --database primary --out sql
$ dbwarden diff --database primary --offline
```

## Options

| Option | Description |
|--------|-------------|
| `--database`, `-d` | Target database name |
| `--out`, `-o` | Output format: `table` (default), `json`, `sql` |
| `--offline` | Use exported model state file instead of live DB snapshot |
| `--verbose`, `-v` | Enable verbose logging |

## Output formats

### `table` (default)

Displays a Rich table with columns: Operation, Table, Target, Severity.

```text
          Schema Diff           
┏━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
┃ Operation    ┃ Table ┃ Target ┃ Severity┃
┡━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│ add_column   │ users │ email  │ INFO    │
│ drop_column  │ users │ name   │ WARNING │
└──────────────┴───────┴────────┴─────────┘
```

### `json`

```json
[
  {"operation": "add_column", "table": "users", "target": "email", "severity": "INFO"},
  {"operation": "drop_column", "table": "users", "target": "name", "severity": "WARNING"}
]
```

### `sql`

Prints the raw migration SQL that would be generated.

## Offline mode

Requires a model state file created by `dbwarden export-models`:

```bash
$ dbwarden export-models --database primary
# Switch to offline machine
$ dbwarden diff --database primary --offline
```

## See also

- [`make-migrations`](./make-migrations.md): generates migration files from diffs
- [`check`](./check.md): safety analyzer for schema changes
- [`check-db`](./check-db.md): inspect live database schema

========================================================================
PAGE: https://docs.dbwarden.org/commands/downgrade/
========================================================================

# `downgrade`

Revert applied migrations to reach a specific target version.

## Usage

```bash
$ dbwarden downgrade --to 0005 --database primary
```

## Options

- `--to`, `-t` (required) - Target version to downgrade to
- `--database`, `-d`
- `--perf`: log per-SQL-statement timing breakdowns
- `--verbose`, `-v`

## Notes

- reads `-- rollback` sections from migration files and applies them in reverse order
- only reverts versions after the target version; versions at or before the target are preserved
- same lock discipline as `migrate` and `rollback`
- fails if the target version has not been applied

See also: [rollback](rollback.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/generate-models/
========================================================================

# `generate-models`

Reverse-engineer SQLAlchemy model code from a live database.

## Usage

```bash
$ dbwarden generate-models --output ./models/ --database primary
$ dbwarden generate-models --output ./models/ --database primary --single-file
$ dbwarden generate-models --output ./models/ --database primary --base app.database:Base
$ dbwarden generate-models --database primary --tables users,posts
$ dbwarden generate-models --database primary --exclude-tables logs,audit
```

> **Note:** `generate-models` works for all supported databases: PostgreSQL, MySQL, MariaDB, ClickHouse, and SQLite. For ClickHouse, use `--clickhouse-engines` or rely on auto-detection from `database_type="clickhouse"`. SQLite produces basic table models without backend-specific metadata.
>
> For PostgreSQL, `generate-models` respects the config-level `pg_schema` setting and reverse-engineers tables from that schema. Mixed-case or quoted identifiers (e.g. `"MyTable"`, `"weird-col"`) are sanitized into valid Python attribute names while the original SQL name is preserved in `Column(...)`. For ClickHouse, full codec chains are preserved (e.g. `CODEC(Delta(8), ZSTD(1))`) and `LowCardinality(Nullable(T))` round-trips correctly.

## Options

| Option | Description |
|--------|-------------|
| `--output`, `-o` | Output directory (default: `models`) |
| `--tables` | Comma-separated list of tables to include |
| `--exclude-tables` | Comma-separated list of tables to exclude |
| `--clickhouse-engines` | Include ClickHouse engine metadata. Auto-detected when `database_type="clickhouse"` |
| `--relationships` | Generate `relationship()` attributes for foreign keys |
| `--dialect` | SQL dialect for type mapping (auto-detected from database type) |
| `--single-file` | Generate a single `models.py` instead of one file per table |
| `--base` | Custom Base class import path (e.g. `app.database:Base`). Default: generates `declarative_base()` in each file |
| `--database`, `-d` | Target database name |

## Output rules

- **Default**: one `.py` file per table (e.g., `users.py`, `posts.py`)
- **`--single-file`**: generates `models.py` with all models
- Each file imports `declarative_base()` and defines `Base` (or imports from the path given by `--base`)

## Type mapping

Database column types are mapped to SQLAlchemy types:

| Database Type | SQLAlchemy Type |
|---------------|----------------|
| `INTEGER` | `Integer` |
| `VARCHAR(N)` | `String(length=N)` |
| `TEXT` | `Text` |
| `BOOLEAN` / `TINYINT(1)` | `Boolean` |
| `DECIMAL(P,S)` | `Numeric(precision=P, scale=S)` |
| `DATETIME` / `TIMESTAMP` | `DateTime` |
| `BIGINT` | `BigInteger` |
| `FLOAT` / `DOUBLE` | `Float` |
| `Nullable(...)` (ClickHouse) | Inner type (nullable is explicit) |

## PostgreSQL First-Class Output

For PostgreSQL databases, `generate-models` reverse-engineers all supported metadata and emits it as `class Meta` inner classes with `PGTableMeta` and `PGColumnMeta`:

```python
from sqlalchemy.orm import DeclarativeBase
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id:    Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True)
    bio:   Mapped[str | None] = mapped_column(Text, nullable=True)

    class Meta(PGTableMeta):
        comment = "Core user accounts"
        pg_fillfactor = 80

        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100, identity_increment=1)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", collation="en_US.UTF-8")
```

The following metadata is reverse-engineered:

- **Identity columns**: `GENERATED ALWAYS/BY DEFAULT AS IDENTITY` with sequence options
- **Collation**: per-column `COLLATE` setting
- **Storage**: per-column `STORAGE` (PLAIN, MAIN, EXTERNAL, EXTENDED)
- **Generated columns**: `GENERATED ALWAYS AS (...) STORED`
- **Enum columns**: `Enum(..., name='type_name')` with the PostgreSQL enum type name
- **Array columns**: `ARRAY(Text)`, `ARRAY(Integer)`, etc.
- **JSONB / TSVECTOR columns**: native PostgreSQL types preserved
- **Table fillfactor**: `WITH (fillfactor = N)`
- **Tablespace**: `SET TABLESPACE`
- **Inheritance**: `INHERITS (parent)`
- **EXCLUDE constraints**: `EXCLUDE USING ...`
- **FK options**: `ondelete`, `onupdate`, `deferrable` on `ForeignKey()`
- **Index options**: `USING`, `WHERE`, `INCLUDE`, `WITH`, `TABLESPACE`, `NULLS NOT DISTINCT`
- **Table and column comments**

For the complete feature reference, see [PostgreSQL Deep Dive](../databases/postgresql/index.md).

## ClickHouse First-Class Output

For ClickHouse databases, `generate-models` reverse-engineers all supported metadata and emits it as `class Meta` inner classes with `CHTableMeta`, `CHColumnMeta`, `ChEngineSpec`, and `ProjectionSpec`. Engine metadata is included automatically when `database_type="clickhouse"` (no `--clickhouse-engines` flag required).

```python
from sqlalchemy.orm import DeclarativeBase
from dbwarden.databases.clickhouse import CHTableMeta, CHColumnMeta, ChEngineSpec, ProjectionSpec, ch

class Base(DeclarativeBase):
    pass

class Event(Base):
    __tablename__ = "events"

    id:    Mapped[int] = mapped_column(Int64, primary_key=True)
    event_date: Mapped[date] = mapped_column(Date)
    payload: Mapped[str] = mapped_column(String)

    class Meta(CHTableMeta):
        ch_engine = ChEngineSpec("MergeTree")
        ch_order_by = ["event_date", "id"]
        ch_partition_by = "toYYYYMM(event_date)"
        ch_ttl = ["event_date + toIntervalYear(1)"]
        ch_settings = {"index_granularity": "8192"}
        ch_projections = [
            ProjectionSpec("by_date", "SELECT event_date, sum(amount) GROUP BY event_date"),
        ]

        class payload(CHColumnMeta):
            ch = ch.field(codec="ZSTD(3)")
```

### Materialized views

Materialized views are reverse-engineered as `MaterializedView` subclasses with a `CHViewMeta` block. The current database qualifier is stripped from both the `SELECT` source tables and the `TO` target, so hand-written models without qualifiers converge with snapshots.

```python
from dbwarden.databases.clickhouse import MaterializedView, CHViewMeta, materialized_view, ChEngineSpec

class EventCountsMv(MaterializedView):
    __tablename__ = "event_counts_mv"

    date: Mapped[date] = mapped_column(primary_key=True)
    count: Mapped[int] = mapped_column()

    class Meta(CHViewMeta):
        ch = materialized_view(
            select="SELECT event_date AS date, count() AS count FROM events GROUP BY event_date",
            engine=ChEngineSpec("MergeTree"),
            order_by=["date"],
        )
```

Mode B (explicit `TO` target) omits `engine`, `order_by`, and `partition_by` in the generated model:

```python
class EventCountsMv(MaterializedView):
    __tablename__ = "event_counts_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select="SELECT event_date AS date, count() AS count FROM events GROUP BY event_date",
            to="event_counts",
        )
```

The following metadata is reverse-engineered:

- **Engine spec**: engine name, arguments, ZooKeeper path, replica name, settings via `ChEngineSpec`
- **Ordering and partitioning**: `ch_order_by`, `ch_primary_key`, `ch_partition_by`, `ch_sample_by`
- **TTL**: table-level TTL expressions
- **Projections**: named projections via `ProjectionSpec`
- **Materialized views**: `SELECT` statement, `TO` target, `REFRESH` schedule, `POPULATE`, and `SETTINGS`; the current database qualifier is stripped from identifiers
- **Dictionaries**: `ch_dictionary`, `ch_dict_layout`, `ch_dict_source`, `ch_dict_lifetime`, `ch_dict_primary_key`
- **Column metadata**: codec, default expression, LowCardinality/Nullable wrappers via `CHColumnMeta`
- **Skip indexes**: `ChIndexSpec` entries in `ch_indexes`
- **Table and column comments**

For the complete feature reference, see [ClickHouse Deep Dive](../databases/clickhouse/index.md).

## Use cases

- **Bootstrapping**: start a new project from an existing database
- **Documentation**: generate model stubs to document the schema
- **Recovery**: regenerate models when migration scripts are missing

## Warnings

- Generated code requires manual review and cleanup.
- ClickHouse engine metadata is auto-detected; review the generated `ChEngineSpec` to ensure correctness.
- PostgreSQL `SERIAL` columns are reverse-engineered as `Integer` with `autoincrement=True`; identity columns are reverse-engineered as `Integer` with `pg_identity` metadata. Both forms round-trip cleanly with `make-migrations`.
- ClickHouse materialized views are introspected but not currently emitted as discoverable model tables.

========================================================================
PAGE: https://docs.dbwarden.org/commands/history/
========================================================================

# `history`

Show migration execution history.

## Usage

```bash
$ dbwarden history --database primary
```

## Options

- `--database`, `-d`

## Notes

- shows applied migrations, order, and timestamps
- useful for audit and incident analysis

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/init/
========================================================================

# `init`

Initialize dbwarden project scaffolding.

## Usage

```bash
$ dbwarden init
$ dbwarden init --database primary
```

## What it does

- creates `migrations/` and `migrations/<database>/` if missing
- creates/updates a declarative config scaffold (`dbwarden.py`) if needed
- does not mutate your database schema

## Notes

- safe to run multiple times
- first command to run in a new project

See also: [Configuration](../configuration/index.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/lock/
========================================================================

# `lock-status` and `unlock`

Inspect and recover migration lock state.

## Usage

```bash
$ dbwarden lock-status --database primary
$ dbwarden unlock --database primary
$ dbwarden unlock --database primary --force
```

## `lock-status`

Shows detailed lock status including state, health, holder information, and heartbeat age.

### Output

When unlocked:

```
Migration lock: INACTIVE
```

When locked:

```
Migration lock status
  Namespace:   default
  State:       RUNNING
  Health:      HEALTHY
  Execution:   abc123def456ghi7
  Owner:       f47ac10b58cc
  Host:        deploy-runner-7
  PID:         1234
  DB Conn ID:  90210
  Migration:   V042
  Checksum:    sha2569ab1c2d3e4f5
  Acquired:    2026-08-22 12:03:14Z
  Heartbeat:   2026-08-22 12:04:01Z
  Fencing:     184
```

### Health verdicts

- **HEALTHY**: Lock held, heartbeat is fresh
- **STUCK**: Lock held, heartbeat is stale (process may be paused or dead)
- **DEAD**: Lock free, status row shows dead worker
- **AVAILABLE**: No lock held
- **COMPLETE**: Migration completed successfully
- **FAILED**: Migration failed
- **INSPECTING**: Recovery inspection in progress
- **NEEDS_REVIEW**: Human intervention required

### JSON output

```bash
$ dbwarden --json lock-status --database primary
```

```json
{
  "database": "primary",
  "locked": true,
  "state": "RUNNING",
  "execution_id": "abc123def456ghi7",
  "owner_id": "f47ac10b-58cc",
  "host": "deploy-runner-7",
  "pid": 1234,
  "migration_version": "V042",
  "acquired_at": "2026-08-22T12:03:14Z",
  "last_heartbeat_at": "2026-08-22T12:04:01Z"
}
```

## `unlock`

Releases the migration lock. Without `--force`, shows holder diagnostics and requires confirmation.

### Options

- `--database`, `-d` - Target database name
- `--force`, `-f` - Skip confirmation and force-release the lock

### Behavior

Without `--force`:

```
Lock holder information
  State:       RUNNING
  Host:        deploy-runner-7
  PID:         1234
  Execution:   abc123def456ghi7
  Migration:   V042
  Acquired:    2026-08-22 12:03:14Z
  Heartbeat:   2026-08-22 12:04:01Z
This will terminate the holder's server connection and release the lock.
Use 'dbwarden unlock --force' to skip this prompt in automation.
```

With `--force`:

```
Migration lock released successfully.
```

### When to use

- After confirming no migration process is running
- When the lock status shows STUCK or DEAD
- In automation scripts (always use `--force`)

### When NOT to use

- If a migration process might still be running
- If the lock status shows HEALTHY with a recent heartbeat
- Without first checking `lock-status` and `history`

## STUCK message format

When a lock is STUCK (lock held, heartbeat stale), the output provides diagnostic guidance:

```
STUCK: holder deploy-runner-7/pid=1234/execution=abc123def456
holds the lock but has not heartbeated since 2026-08-22 12:04:01Z.

The process may be paused (GC, SIGSTOP, overloaded host) or its heartbeat
connection may have failed while the migration continues.

Killing it mid-statement on a non-transactional engine can leave partial
schema changes. Inspect the host first:
  - is pid 1234 alive on deploy-runner-7?  (ps / container runtime)
  - is it making progress?         (dbwarden status --watch, history table)
If you confirm it is dead or wedged:  dbwarden unlock
```

## Notes

- `lock-status` reads the full status row including heartbeat age
- `unlock` uses `force_release_lock()` which sets state to AVAILABLE
- For native-lock engines (PostgreSQL, MySQL), the actual lock is released when the holder's connection closes
- The status row is updated to reflect the release

See also: [Migration Locking](../advanced/migration-locking.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/make-migrations/
========================================================================

# `make-migrations`

Generate SQL migration file(s) from SQLAlchemy models.

Use `--split-at-severity WARN` to write low-severity changes and dependent deferred work as separate versioned files. `--strict-pending` requires composable plans for pending files; `--no-strict-pending` overrides that database setting. `--dry-run` previews without writing SQL, plans, or model state.

Online and offline generation share the same typed pipeline. Its baseline is the latest applied snapshot plus pending trusted plans. Superseded files are audit-only. See [Safety-scoped migrations](../correctness/safety-scoped-migrations.md).

## Usage

```bash
# Auto-generated name from schema changes
$ dbwarden make-migrations

# User-provided description
$ dbwarden make-migrations "create users table"

# With database option
$ dbwarden make-migrations --database primary --verbose

# Output plan JSON only
$ dbwarden make-migrations --database primary --plan

# Explicitly declare column renames
$ dbwarden make-migrations --rename users.username:email --rename posts.title:headline

# Explicitly declare table renames
$ dbwarden make-migrations --rename-table users:accounts

# Safe multi-step type changes
$ dbwarden make-migrations --database primary --safe-type-change
```

## Options

- `description` (optional): Custom migration name. If not provided, automatically generated from schema changes.
- `--database`, `-d`: Target database.
- `--plan`: Print the migration plan JSON without writing files.
- `--sql`: Output raw migration SQL to stdout without writing files.
- `--perf`: Log SQL-generation phase timing.
- `--verbose`, `-v`: Verbose output.
- `--rename`: Repeatable. Declare a column rename in the format `table.old_name:new_name`. See [Rename Detection](#rename-detection) below.
- `--rename-table`: Repeatable. Declare a table rename in the format `old_table:new_table`. See [Table Rename Detection](#table-rename-detection) below.
- `--safe-type-change`: Use a multi-step strategy for type changes: add a temporary column, data migration comment, verification step, then drop-and-rename. Useful for databases where `ALTER COLUMN TYPE` would lock the table.
- `--concurrent` / `--no-concurrent`: Enable or disable `CREATE INDEX CONCURRENTLY` for PostgreSQL (default: `--concurrent`). Use `--no-concurrent` when the migration runs inside a transaction block.
- `--offline`: Use a model state file (`.dbwarden/model_state.json`) instead of a live database or schema snapshot. Run `dbwarden export-models` first to establish a baseline. Useful for CI pipelines without a database service.
- `--clickhouse-engine-recreate`: Allow automatic ClickHouse table rebuild when engine changes require recreation. Required to generate `recreate_ch_table` operations. See [ClickHouse Engine Recreate](#clickhouse-engine-recreate) below.
- `--drop-preserved-clickhouse-table` / `--keep-preserved-clickhouse-table`: Control whether the preserved old ClickHouse table is dropped after the engine-recreate swap. If omitted, interactive terminals are prompted; non-TTY preserves by default.
- `--postgres-auto-using`: Emit an active `USING col::newtype` clause on PostgreSQL `ALTER COLUMN TYPE` statements. Default is a commented-out line for manual review. See the PostgreSQL docs section on Column Type Changes for details.
- `--type`, `-t`: Output prefix for the generated migration file: `versioned` (default), `ra` / `runs_always`, or `roc` / `runs_on_change`. Use `ra` for SQL that should run every migration cycle (e.g. grants, materialized view refreshes) and `roc` for SQL that should re-run when the file changes (e.g. stored procedures, triggers).

## Schema Snapshots

After each migration is applied, dbwarden writes a **schema snapshot** to `.dbwarden/schemas/<migration_id>.schema.json`. These snapshots capture the full DDL state (tables, columns, types, indexes, constraints, enums) at that point in time.

For long migration histories, `dbwarden migrate --defer-snapshots` writes only
the final schema snapshot for the batch. This avoids repeated full-schema
reflection while preserving the final state needed by convergence and offline
diff workflows. Omit the flag when per-migration audit snapshots are required.

`make-migrations` diffs your SQLAlchemy models against the **latest** snapshot instead of the live database. This means:

- You don't need a running database to generate migrations (after the first `migrate`).
- Rename detection works by comparing dropped and added columns between the snapshot and your models.
- If no snapshot exists, `make-migrations` falls back to diffing against the live database.

See [Schema Snapshots](schema-snapshots.md) for details.

## Rollback contract

`make-migrations` enforces rollback correctness for generated migrations. Each emitted rollback statement is classified before the file is accepted:

| Kind | Behavior |
|------|----------|
| `real` | Generated normally. |
| `conditional` | Generated only when prior state exists and the operation is statically proven safe. Verbose mode prints a warning. |
| `irreversible` | Generated only for known irreversible operations or explicit acknowledgement. Normal mode prints a warning. |
| `placeholder` | Refused by default. |

A rollback SQL comment is not treated as a successful rollback. If rollback cannot be generated safely, dbwarden fails generation instead of writing a migration that appears reversible.

Explicit irreversible declaration for committed migrations:

```sql
-- dbwarden: irreversible
```

See [Rollback Coverage](../correctness/rollback-coverage-matrix.md) for PostgreSQL and ClickHouse operation coverage.

## Rename Detection

When a column is dropped from the snapshot and a new column of the same type is added to the model, dbwarden auto-detects it as a potential **rename** and emits `ALTER TABLE ... RENAME COLUMN` instead of `DROP` + `ADD`.

### Auto-detection rules

| Condition | Outcome |
|-----------|---------|
| Same table, 1 dropped + 1 added, same normalized type | Auto-detected as rename |
| Different types | Never auto-detected (emits drop + add) |
| Same name kept | Skipped (no op) |
| 2+ dropped + 2+ added of same type | Paired sequentially (positional) |

### Rename detection edge cases

- **Ambiguous multi-rename**: When 2+ dropped columns and 2+ added columns share the same normalized type, all are treated as renames (paired in insertion order). This is intentionally permissive; false positives can be declined interactively or overridden with `--rename`.
- **Drop-add conversion with `resolved_from`**: A confirmed drop+add pair is converted to a `rename_column` op with `resolved_from` tracking the confirmation source (`"rename_flag"` or `"prompt"`).
- **Non-matching confirmed set**: If a confirmed rename tuple does not match any op (e.g., table name mismatch), it is silently ignored.
- **Table + column rename interaction**: Table renames are processed first (statement order 0). After the snapshot is updated with the new table name, column renames are detected against the renamed table. The column rename's `resolved_from` is independent of the table rename's `resolved_from`.

### Interactive prompt (TTY)

When auto-detected renames are found and you're in an interactive terminal, `make-migrations` prompts you to confirm each one:

**Single rename:**
```
Detected rename: users.username → email. Confirm rename? [Y/n]:
```

**Multiple renames:**
```
Detected column renames:
  [1] users.username → email
  [2] posts.title → headline
  [s] Skip all
  [a] Accept all
Select renames to confirm (e.g. 1,3 or a or s):
```

### CI / non-TTY behavior

When not in an interactive terminal, auto-detected renames are **not applied**. Instead, a warning is printed suggesting the `--rename` flag:

```
The following auto-detected column renames were not confirmed:
  users.username → email (use --rename users.username:email to confirm)
  posts.title → headline (use --rename posts.title:headline to confirm)
These will be emitted as DROP + ADD instead of RENAME.
```

To apply them in CI, pass the corresponding `--rename` flags.

### `--rename` flag

The `--rename` flag explicitly tells `make-migrations` to treat a drop+add pair as a rename. It is required in non-TTY environments (CI) and can also be used to force renames that auto-detection would miss (e.g., type changes).

Format: `--rename <table>.<old_name>:<new_name>`

Examples:

```bash
# Single rename
$ dbwarden make-migrations --rename users.username:email

# Multiple renames
$ dbwarden make-migrations --rename users.username:email --rename posts.title:headline

# Force rename even when types differ
$ dbwarden make-migrations --rename users.phone:mobile_phone
```

### Resolution order

1. **`--rename` flags** are always applied as `RENAME COLUMN` with `resolved_from: "rename_flag"`.
2. **Auto-detected renames** confirmed via interactive prompt get `resolved_from: "prompt"`.
3. **Auto-detected renames** not in `--rename` flags and not confirmed (or in CI) are emitted as `DROP` + `ADD` instead.

## Column-Level Diff

When a schema snapshot exists, `make-migrations` does more than detect new and dropped columns. It also compares columns that exist in both the snapshot and the model for three kinds of change:

| Change | Snapshot vs Model | Generated SQL |
|--------|-------------------|---------------|
| **Type change** | `snapshot.type != model.type` (normalized) | `ALTER COLUMN ... TYPE ...` |
| **Nullability change** | `snapshot.nullable != model.nullable` | `ALTER COLUMN ... SET NOT NULL` / `DROP NOT NULL` |
| **Default change** | `snapshot.default != model.default` | `ALTER COLUMN ... SET DEFAULT` / `DROP DEFAULT` |

### Type change detection

Types are normalized before comparison (see [Schema Snapshots](schema-snapshots.md)). If the normalized type differs between the snapshot and the model, an `alter_column_type` operation is emitted.

Example: model changes `VARCHAR` to `TEXT`:

```sql
-- upgrade

ALTER TABLE users ALTER COLUMN bio TYPE TEXT

-- rollback

ALTER TABLE users ALTER COLUMN bio TYPE VARCHAR
```

### Nullability change

When a model column changes nullable, the corresponding `SET NOT NULL` or `DROP NOT NULL` is generated:

```sql
-- upgrade

ALTER TABLE users ALTER COLUMN email SET NOT NULL

-- rollback

ALTER TABLE users ALTER COLUMN email DROP NOT NULL
```

### Default change

```sql
-- upgrade

ALTER TABLE users ALTER COLUMN role SET DEFAULT 'user'

-- rollback

ALTER TABLE users ALTER COLUMN role DROP DEFAULT
```

### Safe type change (`--safe-type-change`)

For databases that don't support in-place `ALTER COLUMN TYPE` (or when you want to avoid table locks), pass `--safe-type-change` to generate a multi-step strategy:

1. Add a temporary column with the new type
2. Comment indicating a data migration (`UPDATE ... SET temp = CAST(...)`)
3. Verification step comment
4. After manual verification, drop the old column and rename the temporary column

**Limitations:**

| Backend | Supported | Notes |
|---------|-----------|-------|
| PostgreSQL | Yes | Multi-step temp column strategy |
| MySQL / MariaDB | Yes | Multi-step temp column strategy |
| SQLite | No | Comment emitted (SQLite cannot drop columns before 3.35.0 and has limited ALTER TABLE) |
| ClickHouse | No | Comment emitted |

## ClickHouse Engine Recreate

When a ClickHouse table's engine changes, for example `MergeTree` to `ReplicatedMergeTree`, it cannot be altered in-place. dbwarden supports two strategies depending on the table type.

### Table strategy (CREATE + INSERT + RENAME)

For regular `MergeTree`-family tables, dbwarden generates a multi-step operation:

1. Create the new table with the new engine as `<table>__dbw_new`
2. Copy data: `INSERT INTO __dbw_new SELECT ... FROM <table>`
3. Swap: `RENAME TABLE <table> TO <table>__dbw_old, <table>__dbw_new TO <table>`
4. (optional) Drop the preserved old table

**Materialized view targets:** If a materialized view targets the table being recreated (via `TO <table>`), the MV is automatically detached before and reattached after the swap:

```sql
DETACH TABLE events_mv;
CREATE TABLE events__dbw_new (...);
INSERT INTO events__dbw_new SELECT ... FROM events;
RENAME TABLE events TO events__dbw_old, events__dbw_new TO events;
ATTACH TABLE events_mv;
```

**Column renames:** If the table also has column renames, these should be performed in a separate migration (engine recreate and column rename in the same migration is not supported).

### Dictionary strategy (DROP + CREATE)

Dictionaries are recreated with a simple DROP + CREATE since ClickHouse does not support `RENAME DICTIONARY`:

```sql
-- upgrade
DROP DICTIONARY my_dict;
CREATE DICTIONARY my_dict (... ReplicatedMergeTree() ...);

-- rollback
DROP DICTIONARY my_dict;
CREATE DICTIONARY my_dict (... MergeTree() ...);
```

> ⚠️ Dictionaries lose their cached data on recreation. The data will be re-fetched from the source.

### Materialized views and unsupported objects

Engine recreation is **blocked** for tables that are themselves materialized views (`ch_select_statement` or `ch_object_type = materialized_view`). Handle these manually with a DROP/CREATE migration.

Projections (`ch_projections`) are automatically preserved through the table rebuild and do not block it.

### Safety

Rollback for ClickHouse table recreation is deliberately conservative:

| Transition class | Rollback behavior |
|------------------|-------------------|
| Row-preserving engines | Reverse recreate rollback is generated as conditional. |
| Lossy engines | Rollback is irreversible because row-level detail can be lost. |
| Unknown engines | Rollback is irreversible because dbwarden cannot prove safety. |

Lossy engines include `ReplacingMergeTree`, `SummingMergeTree`, `AggregatingMergeTree`, `CollapsingMergeTree`, `VersionedCollapsingMergeTree`, and replicated variants.

| Object type | Safety | Notes |
|------------|--------|-------|
| Regular table with engine change | INFO | Preserved old table by default |
| Table with dependent MVs | INFO | MVs detached before, reattached after |
| Dictionary | CRITICAL | Cached data lost on DROP/CREATE |

### Flags

#### `--clickhouse-engine-recreate`

**Required** to generate engine recreation operations. Without this flag, any detected engine change raises an error:

```
ClickHouse table 'events' cannot be automatically recreated:
is a dictionary (current). This operation requires manual DROP/CREATE,
or use --force to skip this check.
```

#### `--drop-preserved-clickhouse-table` / `--keep-preserved-clickhouse-table`

Controls whether the preserved old table (renamed to `<table>__dbw_old`) is dropped immediately after the swap:

- `--drop-preserved-clickhouse-table`: Drop the old table after successful swap
- `--keep-preserved-clickhouse-table`: Keep the old table (default in non-TTY)

Interactive terminals are prompted to confirm. The preserved table name always ends with `__dbw_old` for easy identification.

### DROP COLUMN warning

All `DROP COLUMN` statements are prefixed with a warning comment:

```sql
-- WARNING: DROPPING COLUMN users.legacy_field

ALTER TABLE users DROP COLUMN legacy_field
```

## Table Rename Detection

When a table is dropped from the snapshot and a new table with similar columns is added to the model, dbwarden auto-detects it as a potential **table rename** using a column-overlap heuristic.

### Auto-detection

| Condition | Outcome |
|-----------|---------|
| A table present in the snapshot but absent from models AND a table absent from the snapshot but present in models | Overlap computed by matching column names and normalized types |
| Overlap ratio ≥ 0.6 | Prompted as a rename candidate |
| Overlap ratio < 0.6 | Emitted as drop+add with a warning comment |

The overlap ratio is `matching_columns / max(len(snapshot_cols), len(model_cols))`. A 0.6 threshold is intentionally conservative: a 10-column table with 6 matching columns is a plausible rename, while a 2-column table with 1 match is not.

### Interactive prompt (TTY)

**Single candidate:**
```
Possible table rename detected:
  users → accounts  (78% columns match)

Treat as rename? [Y/n]:
```

**Multiple candidates:**
```
Possible table renames detected:
  [1] users → accounts     (78% columns match)
  [2] posts → articles     (100% columns match)

Treat as renames? (default: all yes)
  - Press Enter to rename all
  - Type numbers to drop+add instead (e.g. "1" or "1 2"):
```

Table rename prompts appear **before** column rename prompts to ensure table names are resolved before column-level changes.

### CI / non-interactive path

```
Warning: table rename candidates detected but running non-interactive. Emitting drop+add.
  users → accounts  (78% columns match)
Rerun with --rename-table users:accounts to resolve.
```

### `--rename-table` flag

Format: `--rename-table <old_table>:<new_table>`

```bash
# Single table rename
$ dbwarden make-migrations --rename-table users:accounts

# Multiple renames
$ dbwarden make-migrations --rename-table users:accounts --rename-table posts:articles

# Combined with column rename
$ dbwarden make-migrations --rename-table users:accounts --rename accounts.username:email
```

Note: when combining table and column renames, the column rename references the **new** table name. Table renames are applied to the snapshot before column-level processing.

### Table rename edge cases

- **Empty tables**: If either the snapshot table or the model table has zero columns, the overlap ratio is `0.0` and the pair is not a rename candidate.
- **Zero overlap**: If no columns match by name and normalized type, the ratio is `0.0`: emitted as drop+add.
- **Exact match**: If all columns match, the ratio is `1.0`: always a rename candidate.
- **Table rename + column changes in the same table**: After the table rename is applied to the snapshot, column diffs are computed against the new table name. Column renames, type changes, nullable changes, and default changes are all detected on the renamed table.
- **ClickHouse**: `ALTER TABLE RENAME` emits a comment-only placeholder since ClickHouse does not support it.

### SQL generation

All four supported backends (SQLite, PostgreSQL, MySQL, MariaDB) use the same syntax:

```sql
-- upgrade
ALTER TABLE users RENAME TO accounts;

-- rollback
ALTER TABLE accounts RENAME TO users;
```

ClickHouse emits `RENAME TABLE old TO new;` (ClickHouse supports this as a standalone statement).

## Foreign Key and Index Diff

When a schema snapshot exists, `make-migrations` also detects changes to foreign keys and indexes by comparing the snapshot's stored constraints and indexes against the model's declared relationships and indexes.

### Foreign Key vs Index limitations and edge cases

- **Silent skip on missing ref**: If an FK references a table that does not exist in the snapshot, the FK is silently skipped (no error, no SQL). This prevents generating broken SQL but can be surprising. To ensure the FK is emitted, make sure the referenced table exists in the snapshot before running `make-migrations`.
- **Content-based comparison (not name-based)**: Both FKs and indexes are compared by their structural properties, not their names. Renaming a constraint or index does not produce a drop+add.
- **ClickHouse**: FK and index operations emit comment-only placeholders (not supported).
- **SQLite FKs**: Not directly alterable. A comment suggesting table recreation is emitted.

### Foreign Key Detection

| Change | Detection | Generated SQL |
|--------|-----------|---------------|
| **FK added** | FK present in model columns but absent from snapshot constraints | `ALTER TABLE ... ADD CONSTRAINT ... FOREIGN KEY ...` |
| **FK dropped** | FK present in snapshot constraints but absent from model columns | `ALTER TABLE ... DROP CONSTRAINT ...` (or `DROP FOREIGN KEY` on MySQL/MariaDB) |

FKs are compared by content (columns, referenced table, referenced columns), not by name. This means renaming an FK constraint is not treated as a drop+add.

**Validation:** Before emitting an `ADD FOREIGN KEY`, the diff engine verifies that the referenced table and columns exist in the snapshot. If they don't, the FK is silently skipped to avoid generating broken SQL.

**Deferrable constraints (Postgres only):** When detected, `DEFERRABLE INITIALLY DEFERRED` is appended to the constraint SQL.

**SQLite:** FK constraints are not directly alterable. A comment is emitted suggesting table recreation.

### Index Detection

| Change | Detection | Generated SQL |
|--------|-----------|---------------|
| **Index added** | Index present in model but absent from snapshot indexes | `CREATE [UNIQUE] INDEX ... ON table (columns)` |
| **Index dropped** | Index present in snapshot but absent from model indexes | `DROP INDEX ...` |

**Full-content comparison:** Indexes are compared by **all** of these attributes, not just columns + unique. Any difference triggers a drop+add:

| Attribute | SQL Clause | Backend | Example |
|-----------|-----------|---------|---------|
| `using` | `USING <method>` | PostgreSQL, SQLite (partial) | `USING gin`, `USING gist`, `USING hash` |
| `unique` | `UNIQUE` | All | `CREATE UNIQUE INDEX` |
| `where` | `WHERE <predicate>` | PostgreSQL | `WHERE status = 'active'` |
| `include` | `INCLUDE (<cols>)` | PostgreSQL | `INCLUDE (email, name)` |
| `with_params` | `WITH (<params>)` | PostgreSQL | `WITH (fillfactor = 70)` |
| `tablespace` | `TABLESPACE <name>` | PostgreSQL | `TABLESPACE fast_space` |
| `nulls_not_distinct` | `NULLS NOT DISTINCT` | PostgreSQL 15+ | On unique indexes |
| `column_sorting` | Per-column `ASC/DESC NULLS FIRST/LAST` | PostgreSQL | `col1 DESC NULLS LAST, col2 ASC` |
| `type` | `TYPE <type>` | ClickHouse | `TYPE minmax`, `TYPE bloom_filter` |
| `granularity` | `GRANULARITY <n>` | ClickHouse | `GRANULARITY 3` |
| `concurrently` | `CONCURRENTLY` | PostgreSQL | `--concurrent` / `--no-concurrent` |

Omitted attributes or `None`-valued attributes are treated as defaults (btree, no partial, no INCLUDE, etc.), so a plain `Index("ix", "col")` produces the same signature across versions.

**Name generation:** Auto-generated index names follow the pattern:
- `idx_{table}_{col1}_{col2}` for non-unique indexes
- `uq_{table}_{col1}_{col2}` for unique indexes
- Non-btree `USING` methods append a suffix: `idx_{table}_{col}_{method}`

**Backend specifics:**
- PostgreSQL uses `CREATE INDEX CONCURRENTLY` by default (`--concurrent`). Use `--no-concurrent` inside transaction blocks.
- SQLite and MySQL use standard `CREATE INDEX`.
- ClickHouse generates `ALTER TABLE ... ADD INDEX ... TYPE <type> GRANULARITY <n>` for `ChIndexSpec` entries in `ch_indexes`; standard SQL indexes still emit a comment.

## Statement ordering

Operations in the generated migration are ordered consistently:

```
RENAME TABLE        (0)  : table renames first (all subsequent ops use new name)
RENAME COLUMN       (1)
ALTER COLUMN TYPE   (2)
ALTER COLUMN NULLABLE (3)
ALTER COLUMN DEFAULT  (4)
CREATE TABLE        (5)
ADD COLUMN          (6)
ALTER FOREIGN KEY   (7)  : FK adds and drops
ALTER INDEX         (8)  : index adds and drops
DROP COLUMN         (9)
DROP TABLE          (10)
```

Table renames are ordered first so that all subsequent statements reference the new table name.

## Generated artifacts

When a migration is generated, dbwarden writes two files side by side:

- `{database_name}__{version}_{description}.sql`
- `{database_name}__{version}_{description}.plan.json`

The companion plan file contains machine-readable metadata about the generated migration:

- `migration_id`
- `operations`: each operation includes `type`, `table`, `severity` and optionally `resolved_from` (for rename operations)
- `required_flags`
- `checksum`

Example with rename:

```json
{
  "migration_id": "primary__0003_rename_column_users_username",
  "operations": [
    {
      "type": "rename_column",
      "table": "users",
      "new_name": "email",
      "severity": "INFO",
      "resolved_from": "rename_flag"
    },
    {
      "type": "add_column",
      "table": "users",
      "column": "phone",
      "severity": "INFO"
    }
  ],
  "required_flags": [],
  "checksum": "sha256..."
}
```

Possible `resolved_from` values:

| Value | Meaning |
|-------|---------|
| `"rename_flag"` | Explicitly declared via `--rename` or `--rename-table` CLI flag |
| `"prompt"` | Confirmed interactively by the user |
| (absent) | Auto-detected rename kept without prompt (currently unused, reserved) |

`--plan` switches the command into JSON-output mode. In that mode dbwarden prints the plan to stdout and does not write the `.sql` or `.plan.json` files.

## Auto-Generated Names

When no description is provided, dbwarden automatically generates a descriptive name from the schema changes:

| Change | Generated Name |
|--------|----------------|
| Single CREATE TABLE | `create_table_tablename` |
| Multiple CREATE TABLE | `create_tables_users_posts` |
| Single ADD COLUMN | `add_column_tablename_columnname` |
| Multiple ADD COLUMN (same table) | `add_columns_tablename_col1_col2` |
| Single RENAME COLUMN | `rename_column_tablename_new_name` |
| Multiple RENAME COLUMN (same table) | `rename_columns_tablename_col1_col2` |
| Single RENAME TABLE | `rename_table_oldname_newname` |
| Multiple RENAME TABLE | `rename_tables_old1_old2` |
| Single ALTER COLUMN TYPE | `alter_column_type_tablename_col` |
| Single ALTER COLUMN NULLABLE | `alter_column_nullable_tablename_col` |
| Single ALTER COLUMN DEFAULT | `alter_column_default_tablename_col` |
| Single ADD FOREIGN KEY | `add_foreign_key_tablename_ref_table` |
| Single DROP FOREIGN KEY | `drop_foreign_key_tablename` |
| Single ADD INDEX | `add_index_tablename_col` |
| Single DROP INDEX | `drop_index_tablename` |
| Single RECREATE CH TABLE | `recreate_ch_table_tablename` |
| ADD + DROP (same table) | `alter_tablename_col1_col2` |
| Changes across tables | `add_column_users_email_and_1_more_tables` |
| Many targets | `add_columns_tablename_col1_col2_and_3_more` |

### Name Rules

- Snake case throughout.
- Operation words pluralized for multiple targets (e.g., `add_column` → `add_columns`).
- Mixed operations use `alter`.
- Max 72 characters (table/target names truncated as needed).

## Examples

```bash
# Creates primary__0001_create_table_users.sql + .plan.json
$ dbwarden make-migrations --database primary

# Creates primary__0002_add_column_users_email.sql + .plan.json
$ dbwarden make-migrations --database primary

# Creates primary__0003_rename_column_users_email.sql with a confirmed rename
$ dbwarden make-migrations --database primary --rename users.username:email

# Creates primary__0003_alter_column_type_users_bio.sql with type change
$ dbwarden make-migrations --database primary

# Creates primary__0004_add_columns_users_email_name.sql + .plan.json
$ dbwarden make-migrations --database primary

# Creates primary__0003_rename_table_users_accounts.sql with a confirmed table rename
$ dbwarden make-migrations --database primary --rename-table users:accounts

# Uses safe multi-step type change for PostgreSQL
$ dbwarden make-migrations --database primary --safe-type-change

# Uses custom name
$ dbwarden make-migrations "initial_schema" --database primary

# Preview plan JSON without writing files
$ dbwarden make-migrations --database primary --plan
```

## Notes

- Generated file includes both `-- upgrade` and `-- rollback`.
- Generated `.plan.json` files are useful for CI checks and debugging.
- If no models are discovered, configure `model_paths` explicitly.
- With `--dev`, translation can target dev SQLite behavior.
- Schema snapshots are written to `.dbwarden/schemas/` after each successful `migrate`: see [Schema Snapshots](schema-snapshots.md).
- Column-level diff (type/null/default changes) works with a cached schema snapshot, or via a live snapshot taken automatically by `make-migrations` when no cached snapshot exists.
- Without a cached snapshot, `make-migrations` takes a full schema snapshot from the live database internally and detects column-level changes. Only rename detection requires a cached snapshot.
- For authoring guidelines and the review checklist, see [Migration File Format](../migration-files.md).

See also: [Migration File Format](../migration-files.md), [Schema Snapshots](schema-snapshots.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/make-rollback/
========================================================================

# `make-rollback`

Generate a rollback SQL file for a given migration file.

## Usage

```bash
$ dbwarden make-rollback migrations/primary__0005_add_table.sql
```

## Arguments

- `MIGRATION_FILE` (required) - Path to the migration SQL file

## Output

Creates a `.rollback.sql` file next to the given migration file with auto-generated rollback statements.

## Supported reverse transformations

| Upgrade Pattern | Generated Rollback |
|----------------|-------------------|
| `CREATE TABLE t (...)` | `DROP TABLE IF EXISTS t;` |
| `CREATE MATERIALIZED VIEW v AS ...` | `DROP VIEW IF EXISTS v;` |
| `CREATE DICTIONARY d (...)` | `DROP DICTIONARY IF EXISTS d;` |
| `ALTER TABLE t ADD COLUMN c ...` | `ALTER TABLE t DROP COLUMN c;` |
| `CREATE INDEX i ON t (...)` | `DROP INDEX IF EXISTS i;` |
| `CREATE UNIQUE INDEX i ON t (...)` | `DROP INDEX IF EXISTS i;` |
| Other patterns | Refused unless the migration is explicitly irreversible |

## Irreversible annotation

If the command cannot derive executable rollback SQL, it refuses to create a placeholder rollback file. To acknowledge that a migration cannot be rolled back automatically, add this comment to the migration file:

```sql
-- dbwarden: irreversible
```

With that annotation, `make-rollback` may create a rollback file that contains a clear comment instead of executable SQL. This is an intentional declaration, not a successful rollback.

## Notes

- Generated rollback is conservative and may not handle all edge cases.
- Always review the generated rollback before using it.
- For best results, write executable rollback SQL in the `-- rollback` section of the original migration.
- Do not commit placeholder rollback unless the migration is explicitly declared irreversible.

========================================================================
PAGE: https://docs.dbwarden.org/commands/merge-commands/
========================================================================

# `merge`, `rebase`, and `reconcile`

Manage branch merge reconciliation for migration histories.

`merge --dry-run` previews artifacts and superseded filenames without writes. `merge --split-at-severity WARN` uses the same typed writer and dependency split as `make-migrations`. Both reconciliation files, their plans, and complete version metadata are recorded. Untrusted branch files require review and `--force`.

`reconcile --dry-run` computes real environment repair SQL without applying it. `--force` acknowledges repair risks, and `--split-at-severity` splits its artifacts. Successful repair verifies the target schema and records normal reconciliation versions as satisfied. `rebase --dry-run` aliases `--check`; non-interactive rebase requires `--yes`. See [Safety-scoped migrations](../correctness/safety-scoped-migrations.md).

## `merge`

Reconciles divergent migration histories after a branch merge.

### Usage

```bash
$ dbwarden merge --database primary
$ dbwarden merge --database primary --force
$ dbwarden merge --database primary --rename-column users.username=users.handle
```

### Options

- `--database`, `-d`: Target database name
- `--rename-column`: Column rename to confirm (format: `table.old=new`)
- `--rename-table`: Table rename to confirm (format: `old=new`)
- `--force`, `-f`: Force marking hand-edited migrations
- `--commit`: Create a git commit with the changes
- `--json`: Output results as JSON
- `--verbose`, `-v`: Enable verbose logging

### What it does

1. Checks preconditions (clean working tree, no conflict markers)
2. Resolves merge-base state from git
3. Rebuilds current model state from merged models
4. Computes reconciliation diff using the snapshot diff pipeline
5. Probes persistent environments (works with all database types)
6. Generates reconciliation migration with actual SQL
7. Marks branch migrations as superseded
8. Reports results
9. Optionally creates a git commit with `--commit`

### Example

```bash
$ dbwarden merge --database primary
Merge reconciliation summary
  Merge base:        0004 (state checksum 9f2c...)
  Superseded:        0005_add_profile.sql, 0005_extend_billing.sql
  Reconciliation:    0006_merge_feature_a_feature_b.sql
  Environments:      staging: clean, production: clean
  Next steps:        commit; developers on feature branches: dbwarden rebase
Merge reconciliation complete.
```

## `rebase`

Recovers a disposable environment after a merge.

### Usage

```bash
$ dbwarden rebase --database local
$ dbwarden rebase --database local --yes
$ dbwarden rebase --database local --check
```

### Options

- `--database`, `-d`: Target database name
- `--yes`, `-y`: Skip confirmation prompts
- `--force`, `-f`: Force operation even against persistent environments
- `--check`: Only check what would happen, don't make changes
- `--verbose`, `-v`: Enable verbose logging

### What it does

1. Reads local applied migrations
2. Identifies superseded versions
3. Rolls back to merge-base using superseded files (preferred) or resets (fallback)
4. Re-applies runnable chain
5. Verifies convergence

**Note:** The rebase command can read rollback SQL from superseded files, which are normally excluded from the rollback chain.

### Example

```bash
$ dbwarden rebase --database local
Found 2 superseded migration(s) applied: 0005, 0006
Rolling back to merge-base version 0004...
Rolled back to version 0004
Re-applying migrations...
Migrations re-applied successfully.
Verifying convergence...
Convergence verified.
Rebase complete.
```

## `reconcile`

Recovers a persistent environment after a dirty merge.

### Usage

```bash
$ dbwarden reconcile staging
$ dbwarden reconcile staging --dry-run
```

### Options

- `environment`: Environment name to reconcile (required)
- `--database`, `-d`: Target database name
- `--rename-column`: Column rename to confirm
- `--dry-run`: Only show what would happen
- `--verbose`, `-v`: Enable verbose logging

### What it does

1. Snapshots live environment
2. Diffs against merged models
3. Generates environment-specific reconciliation
4. Applies with lock discipline
5. Updates merge record

### Example

```bash
$ dbwarden reconcile staging
Reconciling environment: staging
Environment: staging (persistent: true)
Snapshotting live environment...
Computing diff against merged models...
Generating environment-specific reconciliation...
Applying reconciliation...
Updating merge record...
Reconciliation for environment 'staging' complete.
```

## See also

- [Merge Handling](../advanced/merge-handling.md): Complete merge handling guide
- [Migration Locking](../advanced/migration-locking.md): How migrations are locked
- [Status](status.md): Check migration status

========================================================================
PAGE: https://docs.dbwarden.org/commands/migrate/
========================================================================

# `migrate`

Apply pending migrations.

`--max-severity INFO` applies a strict prefix and exits **3** before the first higher-severity or UNKNOWN file. Later files and repeatables do not execute. UNKNOWN files stop under every ceiling, including the default CRITICAL; only a file with no plan at all passes the default ceiling, governed by the missing_plan preflight policy. `--force` acknowledges risks independently of this ceiling. Dry runs report the stop and return 0; baseline records metadata without SQL. See [Safety-scoped migrations](../correctness/safety-scoped-migrations.md).

## Usage

```bash
$ dbwarden migrate --database primary
$ dbwarden migrate --all
$ dbwarden migrate --database primary --to-version 0010
$ dbwarden migrate --database primary --count 2
$ dbwarden migrate --database primary --with-backup --backup-dir ./backups
$ dbwarden migrate --database primary --baseline --to-version 0005
```

## Options

- `--database`, `-d`
- `--all`, `-a`
- `--count`, `-c`
- `--to-version`, `-t`
- `--baseline`
- `--reapply-data`: explicitly start a new data execution epoch after a completed rollback
- `--with-backup`, `-b`
- `--backup-dir`
- `--dry-run`: preview changes without applying
- `--data`: include frozen data plans and available read-only probes with `--dry-run`
- `--sandbox`: apply in a temporary sandbox database
- `--apply-seeds`: apply pending seeds after migrations
- `--defer-snapshots`: write one final schema snapshot instead of one after every migration
- `--perf`: log per-SQL-statement timing breakdowns
- `--verbose`, `-v`

## Notes

- creates metadata/lock tables if needed
- executes versioned + repeatable migrations
- uses lock protection to prevent concurrent migration mutation
- refuses to run against dirty unreconciled environments (directs to `dbwarden reconcile`)
- verifies paired SQL, plan, and frozen artifacts before applying declarative data
- records explicit data baseline acknowledgement and skipped checks; baseline is not a convergence proof

Rolled-back declarative data requires `--reapply-data` before execution can start a new epoch. `--force` does not authorize data reapply. Successful migrations remain skipped even with `--reapply-data`. See [declarative data migrations](../declarative-data-migrations.md) for journals, ownership, backend limits, and recovery after uncertain effects.

## Dirty environment detection

If the target environment has unreconciled merge changes (dirty merge), `dbwarden migrate` refuses to run and directs you to run `dbwarden reconcile` first. This prevents applying the wrong migration chain to a persistent environment.

```bash
$ dbwarden migrate --database primary
Error: Environment 'primary' has unreconciled merge changes. Run 'dbwarden reconcile' first, or use --dry-run to preview.
```

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/new/
========================================================================

# `new`

Create a manual migration file.

## Usage

```bash
$ dbwarden new "manual hotfix" --database primary
$ dbwarden new "backfill users" --database primary --version 0042
$ dbwarden new "seed data" --database primary --type ra
$ dbwarden new "update view" --database primary --type roc
```

## Options

- positional `description`
- `--database`, `-d`
- `--version`
- `--type`, `-t`: Migration type: `versioned` (default), `ra` / `runs_always`, or `roc` / `runs_on_change`

## Notes

- use when change is not model-driven
- file is scaffolded with `-- upgrade` and `-- rollback` sections

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/rollback/
========================================================================

# `rollback`

Rollback applied migrations using `-- rollback` SQL sections.

## Usage

```bash
$ dbwarden rollback --database primary
$ dbwarden rollback --database primary --count 2
$ dbwarden rollback --database primary --to-version 0007
```

## Options

- `--database`, `-d`
- `--count`, `-c`
- `--to-version`, `-t`
- `--perf`: log per-SQL-statement timing breakdowns
- `--verbose`, `-v`

## Notes

- rollback runs in reverse order
- same lock discipline as migrate

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/schema-snapshots/
========================================================================

# Schema Snapshots

Schema snapshots are JSON files that record the full DDL state of a database at the point a migration was applied. They enable offline migration generation, intelligent rename detection, and column-level change detection (type, nullability, default).

## How they work

After each versioned migration is successfully applied by `migrate`, dbwarden extracts the complete schema from the live database and writes it as a `<migration_id>.schema.json` file:

```
.dbwarden/schemas/
  primary__0001_init.schema.json
  primary__0002_add_email.schema.json
  primary__0003_create_posts.schema.json
```

### Snapshot contents

Each snapshot captures:

```json
{
  "format_version": 2,
  "migration_id": "primary__0003_create_posts",
  "database_name": "primary",
  "database_type": "postgresql",
  "applied_at": "2026-06-07T10:30:00Z",
  "checksum": "sha256...",
  "tables": {
    "users": {
      "object_type": "table",
      "columns": {
        "id": {
          "name": "id",
          "type": "integer",
          "nullable": false,
          "primary_key": true,
          "default": null,
          "comment": null,
          "pg_column": {}
        },
        "email": {
          "name": "email",
          "type": "varchar",
          "nullable": false,
          "primary_key": false,
          "default": null,
          "comment": null,
          "pg_column": {}
        }
      },
      "backend_table_spec": {"backend": "postgresql"},
      "comment": null
    }
  },
  "enums": {},
  "indexes": {},
  "constraints": {}
}
```

Fields added in format v2:

| Field | Level | Purpose |
|-------|-------|---------|
| `backend_table_spec` | Per-table | Backend-specific table options (e.g., `ch_engine`, `pg_fillfactor`, `my_engine`) |
| `pg_column` / `ch_column` / `my_column` | Per-column | Backend-specific column metadata (e.g., `ch_type`, `ch_codec`, `pg_type`, `my_unsigned`) |
| `object_type` | Per-table | `"table"`, `"materialized_view"`, or `"dictionary"` |
| `name` | Per-column | Column name (v1 stored names as keys; v2 stores them in both key and value) |

**Backward compatibility:** Snapshots with `format_version: 1` are automatically normalized to v2 when read. V1 keys like `clickhouse_options`, `pg_table`, `my_table`, and per-table `indexes`/`primary_key` are mapped to their v2 equivalents.

### Integrity

Snapshots are self-integrity-checked with a SHA-256 checksum. If a snapshot file is tampered with (manually edited, corrupted), `read_snapshot()` returns `None` and `make-migrations` falls back to the live database.

## Why snapshots?

### Offline diffing

Once the first snapshot exists, `make-migrations` can generate new migrations **without a live database connection**. This is useful in:

- CI pipelines where only model files are available
- Air-gapped environments
- Development setups without a full database

### Rename detection (columns)

Rename detection relies on comparing the snapshot's columns against the model's columns:

1. A column present in the snapshot but absent from the model → **dropped**.
2. A column present in the model but absent from the snapshot → **added**.
3. If a dropped column and an added column share the same normalized type, they are candidates for a **rename**.

Without snapshots (the legacy live-DB fallback path), rename detection is not possible because the live DB already reflects the current state and has no record of dropped columns.

### Rename detection (tables)

Table renames are detected by comparing tables present in the snapshot but absent from models (dropped) against tables absent from the snapshot but present in models (added). A **column overlap heuristic** computes the ratio of matching column names+types between the two tables. If the ratio is ≥ 0.6, the pair is a rename candidate.

Detected table renames are prompted interactively (TTY) or suggested via the `--rename-table` flag (CI). Confirmed renames emit `ALTER TABLE ... RENAME TO` and are applied to the snapshot before column-level processing, so subsequent column renames reference the new table name.

### Column-level change detection

Snapshots also enable `make-migrations` to detect when a column's **type**, **nullability**, or **default** has changed. For columns present in both the snapshot and the model with the same name, the diff engine compares:

- **Normalized type**: If `varchar` in the snapshot but `text` in the model, an `ALTER COLUMN TYPE` operation is emitted.
- **Nullable flag**: If nullable differs, `SET NOT NULL` or `DROP NOT NULL` is generated.
- **Default value**: If the default differs, `SET DEFAULT` or `DROP DEFAULT` is generated.

When a cached snapshot exists, these operations are compared against the historical schema. Without a cached snapshot, `make-migrations` takes a full schema snapshot from the live database internally and detects column-level changes (type, nullability, default) from that live snapshot. Only rename detection requires a cached snapshot.

### Foreign key and index change detection

Snapshots also store the database's foreign keys (in `constraints`) and indexes (in `indexes`). The diff engine compares these against the model's declared FK relationships (parsed from `ModelColumn.foreign_key`) and indexes (extracted from `__table__.indexes`). Comparisons are content-based (columns, referenced table, referenced columns for FKs; columns + unique flag for indexes), so constraint/index name changes are not treated as drop+add.

### Audit trail

Every schema snapshot is an immutable record of the database schema at a specific migration version. You can inspect any historical snapshot to see exactly what the schema looked like.

## How they are created

Snapshots are created automatically by the `migrate` command after applying a versioned migration:

```
$ dbwarden migrate --database primary
```

The snapshot is written **after** all pending migrations have been applied. If the write fails (permission issue, disk full, etc.), a warning is logged but the migration itself is not rolled back. Failure to write a snapshot is non-fatal.

### Snapshots are NOT created for

- `--dry-run` runs (no real schema change)
- Rollback operations (snapshot remains as-is for audit)
- Repeatable migrations (`RA__`, `ROC__`)

## Rollback and re-apply

- **Rollback does not delete the snapshot.** The snapshot stays as an audit record of what was applied.
- **Re-applying a migration** overwrites the snapshot with the current schema state.

## Finding the latest snapshot

`make-migrations` uses `find_latest_snapshot()` which scans `.dbwarden/schemas/` for snapshot files matching the current database name and picks the one with the highest version prefix (e.g., `0003` > `0002`).

If no snapshot exists for the database, `make-migrations` takes a full schema snapshot from the live database internally and runs the standard diff pipeline against it. This enables column-level change detection (type, nullability, default). Only rename detection requires a cached snapshot.

## Snapshot lifecycle summary

| Event | Snapshot |
|-------|----------|
| First `migrate` | Created after apply |
| Subsequent `migrate` | Overwritten with latest schema |
| `rollback` | Unchanged (kept as audit) |
| Re-apply same version | Overwritten |
| `--dry-run` / `--sandbox` | Not written |
| `make-migrations` (snapshot exists) | Read for diff + rename detection |
| `make-migrations` (no snapshot) | Fallback to live DB diff |

## DB-agnostic type normalization

Column types in the snapshot are normalized to a canonical set so that equivalent types across databases are treated the same:

| Canonical type | Matches |
|----------------|---------|
| `integer` | INT, INTEGER, INT4, TINYINT, SMALLINT |
| `biginteger` | BIGINT, INT8 |
| `varchar` | VARCHAR, CHARACTER VARYING |
| `text` | TEXT, LONGTEXT, CLOB |
| `boolean` | BOOLEAN, BOOL |
| `timestamp` | TIMESTAMP, DATETIME |
| `numeric` | NUMERIC, DECIMAL (with precision/scale) |
| `float` | FLOAT, REAL, DOUBLE |
| `bytes` | BYTEA, BLOB, BINARY |
| `uuid` | UUID |
| `enum` | ENUM |
| (unknown) | Stored as-is with `"raw": true` |

This normalization is what powers the rename detection: two columns with the same normalized type are candidates for rename, even if their raw SQL type strings differ.

## Edge Cases and Restrictions

### Rename detection
- **Ambiguous renames**: When multiple columns of the same type are dropped and added, all possible pairs are treated as renames (not just one). This maximizes detection but may produce false positives that must be confirmed interactively or via `--rename`.
- **Type change prevents rename**: If a dropped column and an added column have different normalized types, they are never auto-detected as renames. Use `--rename` to force the rename anyway.
- **Same name**: If a column with the same name exists in both the snapshot and the model, no rename is detected even if its type changes (that is handled by type-change detection).

### Table rename detection
- **Column-overlap heuristic**: The ratio is `matching_columns / max(len(snapshot_cols), len(model_cols))`. The 0.6 threshold is intentionally conservative.
- **Empty tables**: Either table having zero columns results in a ratio of `0.0` (no candidate).
- **Table rename + column diff interaction**: Confirmed table renames are applied to the snapshot before column diffs are computed. This ensures column renames and other column-level changes reference the new table name.

### Foreign key and index detection
- **Silent skip on missing ref**: If an FK references a table that does not exist in the snapshot, the FK is silently skipped (no error, no SQL emitted). This prevents broken SQL but can be surprising. Ensure the referenced table exists in the snapshot first.
- **Content-based comparison**: FKs are compared by `(columns, referenced_table, referenced_columns)`. Indexes are compared by `(frozenset(columns), unique)`. Renaming a constraint or index does not produce a drop+add.
- **ClickHouse**: FK and index operations emit comment-only placeholders (not supported by ClickHouse).
- **SQLite FKs**: A comment is emitted suggesting table recreation (not directly alterable).

### Column-level change detection
- **Cached snapshot not required**: Column-level diff works with a live snapshot taken automatically by `make-migrations`. A cached snapshot enables rename detection in addition to column-level diff.
- **Backend limits**: Type changes emit different SQL per backend. SQLite emits comment-only placeholders for type and nullable changes. ClickHouse auto-generates `MODIFY COLUMN` for type, nullable, and LowCardinality changes. Default changes work uniformly across all backends.

### Integrity
- **Tampered snapshots**: If the checksum does not match, `read_snapshot()` returns `None`, and `make-migrations` falls back to the live database. A warning is logged.
- **Checksum-excluded fields**: The `checksum` field itself is excluded from the hash computation, so checksum updates do not cascade.

========================================================================
PAGE: https://docs.dbwarden.org/commands/seed/
========================================================================

# `seed`

Manage seed data for a database.

## Subcommands

- `seed create`: create a new file seed (legacy)
- `seed apply`: apply pending seeds (file + code seeds)
- `seed list`: list seeds and their status
- `seed rollback`: roll back applied seeds
- `seed export`: export code seeds to ROC SQL files for stateless production application

---

## `seed create`

Create a new file-based seed file (SQL or Python). For new projects, prefer [code seeds](../seeds.md#code-seeds-recommended) instead.

### Usage

```bash
$ dbwarden seed create "seed initial data" --database primary
$ dbwarden seed create "populate lookup tables" --database primary --type python
```

### Options

- `--database`, `-d`: target database handle
- `--type`: `sql` (default) or `python`
- `--verbose`, `-v`

---

## `seed apply`

Apply pending seeds. Both file seeds and [code seeds](../seeds.md#code-seeds-recommended) are discovered and applied.

### Usage

```bash
$ dbwarden seed apply --database primary
$ dbwarden seed apply --database primary --version 0003
$ dbwarden seed apply --database primary --dry-run
$ dbwarden seed apply --all
```

### Options

- `--database`, `-d`
- `--all`, `-a`: apply across all configured databases
- `--version`: apply up to this seed version
- `--dry-run`: preview without executing
- `--verbose`, `-v`

---

## `seed list`

List seeds and their applied status. Includes both file seeds and code seeds.

### Usage

```bash
$ dbwarden seed list --database primary
$ dbwarden seed list --all
$ dbwarden seed list --prune              # clean up orphaned tracking records
```

### Options

- `--database`, `-d`
- `--all`, `-a`
- `--prune`: remove tracking records for seed files that no longer exist on disk
- `--verbose`, `-v`

---

## `seed rollback`

Reverse applied seeds and then remove their tracking records. SQL file seeds require a `-- rollback` section; Python file and procedural code seeds require `reverse(connection, session)`. Seeds without reverse logic are irreversible and rollback fails without changing tracking.

### Usage

```bash
$ dbwarden seed rollback --database primary
$ dbwarden seed rollback --database primary --count 2
$ dbwarden seed rollback --database primary --to-version 0003
```

### Options

- `--database`, `-d`
- `--all`, `-a`: rollback on all databases
- `--count`, `-c`: number of seeds to roll back
- `--to-version`, `-t`: roll back to this seed version
- `--verbose`, `-v`

See also: [Seed Management](../seeds.md)

---

## `seed export`

Export code seeds to ROC (runs-on-change) SQL files for stateless application. The generated file contains `INSERT ... ON CONFLICT` statements rendered in the target database dialect. ROC files are re-applied when their content checksum changes.

### Usage

```bash
$ dbwarden seed export --database primary
$ dbwarden seed export --all
$ dbwarden seed export --database clickhouse --output-dir ./seeds
```

### Options

- `--database`, `-d`: target database handle
- `--all`, `-a`: export seeds for all configured databases
- `--output-dir`, `-o`: output directory (default: `seeds/`)

### Behavior

- **Row-based seeds** (`rows = [...]`): each row is rendered as an `INSERT` statement with `ON CONFLICT` matching the seed's `__seed_on_conflict__`
- **Procedural seeds** (`forward(connection, session)`): cannot be exported because executing arbitrary code under a different database backend could produce incorrect SQL
- Seeds are ordered by FK dependency (topological sort) so foreign-key-safe insert order is preserved

### Dialect requirement

Exporting requires the same dialect packages as connecting to that database. For ClickHouse, install `clickhouse-sqlalchemy`. Missing packages produce a clear error at export time.

### Non-handled problems

- Removed rows are not deleted (no purge on re-export)
- Procedural seeds must be applied directly to their configured database

========================================================================
PAGE: https://docs.dbwarden.org/commands/settings/
========================================================================

# `settings`

View dbwarden configuration. All database settings are defined in Python code via
declarative database classes or `database_config()`, so `settings show` is a read-only command for inspecting
the current configuration.

## `settings show`

### Usage

```bash
$ dbwarden settings show
$ dbwarden settings show primary
$ dbwarden settings show --all
```

### Options

- `--all`: show all configured databases

### Example output

```
Database: PRIMARY (default)
  Default: True
  Type: SQLite
  URL: sqlite:///./app.db
  Migrations Directory: migrations/primary
  Migration Table: _dbwarden_migrations
  Seed Table: _dbwarden_seeds
  Model Paths: app
  Dev Database Type: 
  Dev Database URL: 
  Overlap Models: False
  Skip If Missing: False
```

## See also

- [`database list`](./database.md)
- [Configuration docs](../configuration/index.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/snapshot/
========================================================================

# `snapshot`

Output the DDL schema of a specific database table.

## Usage

```bash
$ dbwarden snapshot users --database primary
```

## Options

- `TABLE` (required) - Name of the table to snapshot
- `--database`, `-d`

## Output

For standard SQL databases (SQLite, PostgreSQL, MySQL, MariaDB):

- `CREATE TABLE` statement with column types, nullability, and defaults
- `CREATE INDEX` statements
- Foreign key constraints

For ClickHouse:

- The raw `CREATE TABLE` query from `system.tables`

## Notes

- output is printed to stdout
- useful for debugging schema differences or documenting table structure
- internally uses `sqlalchemy.inspect()` for generic databases

========================================================================
PAGE: https://docs.dbwarden.org/commands/status/
========================================================================

# `status`

Show migration status (applied vs pending).

Pending rows include recorded severity and their state under configured `max_severity`: pending, deferred, or blocked by an earlier version. JSON includes trust reasons, blocking versions, and deferred age. Status reads plans without reparsing SQL; applied historical files need no severity plan.

## Usage

```bash
$ dbwarden status --database primary
$ dbwarden status --all
```

## Options

- `--database`, `-d`
- `--all`, `-a`
- `--all-environments`: show status for all registered environments

## Notes

- run before and after migration execution
- supports multi-database status with `--all`

See also: [Your First Migration](../getting-started/first-migration.md)

========================================================================
PAGE: https://docs.dbwarden.org/commands/version/
========================================================================

# `version`

Display dbwarden version and compatibility information.

## Usage

```bash
$ dbwarden version
```

## Notes

- useful for support/debug and release verification

========================================================================
PAGE: https://docs.dbwarden.org/configuration/concepts/
========================================================================

# Configuration Concepts

Understand how dbwarden configuration works under the hood.

## Declarative Data Paths

Use `data_paths` for live declarative data modules, separately from `model_paths`. dbwarden imports these paths only to compile current declarations; it excludes generated `*.data.py` artifacts. `data_snapshot_dir` defaults to `.dbwarden/data` and `snapshot_registry` defaults to `.dbwarden/snapshots/registry.json`. Both must stay inside the project. Two databases cannot share a live data path unless `overlap_models=True` explicitly allows the overlap. See [Declarative data migrations](../declarative-data-migrations.md).

## What is Configuration?

Configuration tells dbwarden:
- **Where** your databases are (connection URLs)
- **What** kind of databases they are (PostgreSQL, SQLite, etc.)
- **Where** your SQLAlchemy models live (for migration generation)
- **Where** to store migrations (directories)

## Why Python Configuration?

### Type Safety

Your IDE can help you:

```python
primary = database_config(
    database_name="primary",  #  IDE suggests parameter names
    default=True,             #  IDE knows this is boolean
    database_type="sqlite",   #  IDE can validate enum values
    database_url_sync="...",
)
```

### Dynamic Configuration

You can use Python logic:

```python
import os

# Different config per environment
environment = os.getenv("ENV", "dev")

if environment == "production":
    database_url = "postgresql://prod-host/myapp"
else:
    database_url = "sqlite:///./dev.db"

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql" if environment == "production" else "sqlite",
    database_url_sync=database_url,
)
```

### Multiple Databases

Easy to configure multiple databases:

```python
DATABASES = {
    "primary": "postgresql://localhost/main",
    "analytics": "postgresql://localhost/analytics",
    "logging": "postgresql://localhost/logs",
}

for name, url in DATABASES.items():
    db = database_config(
        database_name=name,
        default=(name == "primary"),
        database_type="postgresql",
        database_url_sync=url,
        model_paths=[f"app.models.{name}"],
    )
```

## Two Equivalent APIs

Use `database_config(...)` for direct, one-off declarations. Use
`DbwardenDatabase` subclasses when several databases share settings or plugin
configuration through inheritance:

```python
from dbwarden import DbwardenDatabase


class Production(DbwardenDatabase):
    __abstract__ = True
    database_type = "postgresql"
    model_paths = ["app.models"]


class Primary(Production):
    database_name = "primary"
    database_url_sync = "postgresql://localhost/main"
    default = True
```

Every explicit `database_config(...)` field is available as a class attribute.
Concrete classes register on import, and their `.handle` is the same handle
returned by the function API. Mark shared bases with `__abstract__ = True` so
they are not registered independently.

## Configuration Loading

### Discovery Order

dbwarden searches for configuration in this order:

```
1. Walk project tree for dbwarden.py files
      not found
2. DBWARDEN_CONFIG_MODULE environment variable
      not found
3. Full scan for files with database_config() calls
      not found
Error: No configuration found
```

### When Configuration Loads

Configuration loads when you run **any** dbwarden command:

```bash
$ dbwarden migrate    #  Config loads here
$ dbwarden status     #  Config loads here
$ dbwarden history    #  Config loads here
```

**Load process:**
1. Python imports your config module
2. Concrete `DbwardenDatabase` classes register, or `database_config()` calls execute
3. Databases register in internal registry
4. Validation runs
5. Command executes with loaded config

### Validation Rules

dbwarden validates configuration at load time:

| Rule | Why It Matters |
|------|----------------|
| Exactly one `default=True` | CLI needs to know which DB to use when `--database` is omitted |
| Unique `database_name` | Commands target databases by name |
| Unique `database_url` | Prevents accidental duplicate configurations |
| Unique physical targets | Prevents two configs pointing to same DB with different credentials |
| Required `model_paths` in multi-DB | Keeps model discovery boundaries clear |
| No overlapping `model_paths` | Prevents ambiguous model ownership (unless `overlap_models=True`) |

### Validation Timing

```python
# dbwarden.py
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
)

primary = database_config(
    database_name="primary",  #  Duplicate!
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/other",
)
```

When you run `dbwarden migrate`:

```
Error: Duplicate database_name 'primary'
```

Validation happens **before** any commands execute.

### Config Source Precedence

When looking for config, dbwarden uses this precedence:

1. **Top-level `dbwarden.py`**: the conventional standalone config file at your project root. This is the default scaffold created by `dbwarden init`, not the only valid location. Always sandboxed (only `dbwarden` imports allowed).

2. **`DBWARDEN_CONFIG_MODULE`**: an explicit environment variable override. Always imported normally as a Python module (no sandbox). This is the escape hatch for projects with ambiguous full-scan results or non-standard layouts.

3. **Full-scan discovery**: if neither of the above produces a config source, dbwarden walks your project tree looking for any `database_config(...)` call. This means `database_config(...)` can live in any discovered Python file inside your project. Files directly at the project root are sandboxed; files inside subdirectories are imported normally.

### Config Loading Security (Sandbox)

dbwarden applies import restrictions only to config files that are **isolated** (sandboxed) vs **in-package** (normal import):

| Mode | Import behavior | Applies to |
|------|----------------|------------|
| `isolated` | Sandboxed: only `dbwarden.*` imports allowed | Top-level `dbwarden.py`; any full-scan-discovered file at the project root |
| `in-package` | Normal Python import | Full-scan-discovered files inside subdirectories; `DBWARDEN_CONFIG_MODULE` modules |

An isolated config file runs in a sandbox that prevents accidental escalation of file-read access to arbitrary code execution. Only `dbwarden` and its submodules can be imported.

An in-package config file is imported as a normal Python module, with full access to `app.*` and any other project imports. This is the correct path when your `database_config(...)` call lives in an application package that imports other project modules.

**Import root detection.** For full-scan-discovered files, dbwarden tries to resolve the dotted module path. It checks two common import roots in order:

- `src/` (PEP 517/518, setuptools, poetry)
- The project root itself

For example, `src/myapp/databases.py` resolves as `myapp.databases` with import root `src/`. If neither root produces an importable path, the file falls back to `isolated` (sandboxed). Projects with other layouts should set `DBWARDEN_CONFIG_MODULE` explicitly.

**Path validation** (path traversal blocking) applies to all file-based sources regardless of mode.

For debugging, set `DBWARDEN_DISABLE_SANDBOX=1` to disable the sandbox for isolated files:

```bash
DBWARDEN_DISABLE_SANDBOX=1 dbwarden status  # Skip sandbox (debug only)
```

Disabling the sandbox also removes import restrictions for isolated config files, which can be useful in development.  Keep it enabled in production.

## The `default` Database

### Why `default=True` Exists

Consider these commands:

```bash
# Explicit database
$ dbwarden migrate --database primary

# Implicit database (uses default)
$ dbwarden migrate
```

Without `default=True`, dbwarden wouldn't know which database to use for the second command.

### Only One Default

```python
#  Good
analytics = database_config(
    database_name="analytics",
    default=False,  # or omit default
)

#  Bad - two defaults
analytics = database_config(
    database_name="analytics",
    default=True,  # Error!
)
```

### Default Affects CLI Behavior

```bash
# These are equivalent when primary is default:
$ dbwarden migrate
$ dbwarden migrate --database primary

# These are NOT equivalent:
$ dbwarden migrate
$ dbwarden migrate --database analytics  # Targets analytics, not primary
```

## Model Discovery

### What Are `model_paths`?

`model_paths` tells dbwarden where your SQLAlchemy models live:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    model_paths=["app.models"],  #  Look here for models
)
```

### How Discovery Works

```
1. Import each module in model_paths
     
2. Find all classes inheriting from DeclarativeBase
     
3. Extract table metadata (__tablename__, columns, etc.)
     
4. Build internal representation for migration generation
```

### Filtering by Table Name

When two databases share the same `model_paths` but should own different
subsets of tables, use `model_tables`:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models"],
    model_tables=["users", "posts", "comments"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://clickhouse-host:8123/analytics",
    model_paths=["app.models"],
    model_tables=["analytics_events", "analytics_sessions"],
)
```

This is useful when all models live under one shared package but each
database only owns a subset.  dbwarden validates every name in
`model_tables` exists among the discovered tables and prevents overlap
between databases (unless `overlap_models=True`).

### When Is It Required?

**Single database:** Optional (dbwarden scans entire codebase)

```python
# This works
primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    # No model_paths needed
)
```

**Multiple databases:** Required for each database

```python
# This is required
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],  #  Required
)

analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    model_paths=["app.models.analytics"],  #  Required
)
```

**Why?** To prevent ambiguity about which models belong to which database.

## Dev Mode

### What Is Dev Mode?

Dev mode lets you use a different database for local development:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",              # Production
    database_url_sync="postgresql://prod/myapp",
    dev_database_type="sqlite",              # Development
    dev_database_url="sqlite:///./dev.db",
)
```

Run commands with `--dev`:

```bash
$ dbwarden --dev migrate  # Uses SQLite
$ dbwarden migrate        # Uses PostgreSQL
```

### How It Works

```
Command: dbwarden --dev migrate
    
Check for --dev flag
    
Swap database_type → dev_database_type
Swap database_url → dev_database_url
    
Connect to dev database
    
Execute command
```

### Why Use It?

**Speed:**
- SQLite is faster than PostgreSQL for local iteration
- No network latency
- No server setup

**Safety:**
- Can't accidentally affect production
- Each developer has their own isolated database
- Easy to reset (just delete the file)

**Simplicity:**
- No Docker containers needed
- No database server installation
- Works on all platforms

## Multi-Database Configuration

### Why Multiple Databases?

Common scenarios:
- **Separation of concerns** - Transactions vs analytics
- **Performance** - Offload reporting to separate database
- **Compliance** - Audit logs in separate database
- **Legacy systems** - New and old databases coexist

### How It Works

Each concrete `DbwardenDatabase` class registers an independent database. The
equivalent `database_config()` calls behave the same way:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],
)

analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    model_paths=["app.models.analytics"],
)
```

They're completely independent:
- Separate migration histories
- Separate migration directories
- Separate model sets
- Can use different database types

### Model Path Boundaries

```
app/
  models/
    primary/
      user.py         #  Goes to primary database
      order.py
    analytics/
      event.py        #  Goes to analytics database
      metric.py
```

Configuration:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],  #  Only primary models
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],  #  Only analytics models
)
```

## Secure Values

### What Is `secure_values`?

Prevents credentials from appearing in terminal output:

```python
import os

DATABASE_URL = os.getenv("DATABASE_URL")

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=DATABASE_URL,
    secure_values=True,  #  Hide credentials
)
```

### Without `secure_values`:

```bash
$ dbwarden settings show
Database: primary
  URL: postgresql://user:SECRET_PASSWORD@prod-host/myapp
```

### With `secure_values`:

```bash
$ dbwarden settings show --all
Database: primary
  URL: DATABASE_URL (expression)
```

Shows the variable name instead of resolved value.

## Configuration vs Runtime

### Configuration Time

When config loads:
- Declarative classes register, or `database_config()` calls execute
- Validation runs
- Internal registry populates
- **No database connections made**

### Runtime

When commands run:
- dbwarden reads from registry
- **Connects to database**
- Executes command logic

**Key point:** Configuration errors are caught early, before any database operations.

## What's Next?

- **[Connection URLs](connection-urls.md)** - URL format reference
- **[Model Discovery](model-discovery.md)** - Deep dive into model paths
- **[Dev Mode](dev-mode.md)** - Local development workflows
- **[Multi-Database](multi-database.md)** - Multi-database patterns

========================================================================
PAGE: https://docs.dbwarden.org/configuration/connection-urls/
========================================================================

# Connection URLs

Complete reference for database connection URL formats.

New projects should declare fields on a `DbwardenDatabase` subclass; the
`database_config(...)` function alternative remains supported.

## URL Format

All database URLs follow this general structure:

```
[dialect[+driver]]://[username[:password]@][host][:port][/database][?option=value&...]
```

## PostgreSQL

### Basic Format

```
postgresql://[user[:password]@][host][:port]/database[?options]
```

### Examples

**Local default:**
```python
database_url_sync="postgresql://localhost/myapp"
```

**With credentials:**
```python
database_url_sync="postgresql://user:password@localhost:5432/myapp"
```

**Remote host:**
```python
database_url_sync="postgresql://user:password@db.example.com:5432/myapp"
```

**With SSL:**
```python
database_url_sync="postgresql://user:password@localhost/myapp?sslmode=require"
```

**With connection pool:**
```python
database_url_sync="postgresql://user:password@localhost/myapp?pool_size=20&max_overflow=10"
```

### SSL Modes

| Mode | Description |
|------|-------------|
| `disable` | No SSL |
| `allow` | Try SSL, fall back to non-SSL |
| `prefer` | Try SSL first (default) |
| `require` | Require SSL, fail if unavailable |
| `verify-ca` | Require SSL + verify CA |
| `verify-full` | Require SSL + verify CA + hostname |

**Example:**
```python
database_url_sync="postgresql://user:pass@host/db?sslmode=verify-full&sslrootcert=/path/to/ca.pem"
```

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `sslmode` | SSL connection mode | `sslmode=require` |
| `sslcert` | Client certificate | `sslcert=/path/to/cert.pem` |
| `sslkey` | Client key | `sslkey=/path/to/key.pem` |
| `sslrootcert` | CA certificate | `sslrootcert=/path/to/ca.pem` |
| `connect_timeout` | Connection timeout (seconds) | `connect_timeout=10` |
| `application_name` | App name in pg_stat_activity | `application_name=myapp` |

### Cloud Providers

**AWS RDS:**
```python
database_url_sync="postgresql://user:pass@mydb.abc123.us-east-1.rds.amazonaws.com:5432/myapp?sslmode=require"
```

**Google Cloud SQL:**
```python
database_url_sync="postgresql://user:pass@/myapp?host=/cloudsql/project:region:instance"
```

**Azure Database:**
```python
database_url_sync="postgresql://user@server:pass@server.postgres.database.azure.com:5432/myapp?sslmode=require"
```

**Heroku:**
```python
import os
database_url_sync=os.getenv("DATABASE_URL")  # Provided by Heroku
```

## SQLite

### Basic Format

```
sqlite:///[path]
```

### Examples

**Relative path:**
```python
database_url_sync="sqlite:///./app.db"
database_url_sync="sqlite:///./data/app.db"
```

**Absolute path:**
```python
database_url_sync="sqlite:////absolute/path/to/app.db"
```

**In-memory (testing only):**
```python
database_url_sync="sqlite:///:memory:"
```

In-memory databases are lost when the connection closes. Only use for testing.

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `timeout` | Lock timeout (seconds) | `?timeout=20` |
| `check_same_thread` | Thread safety check | `?check_same_thread=false` |

**Example:**
```python
database_url_sync="sqlite:///./app.db?timeout=20"
```

## MySQL / MariaDB

### Basic Format

```
mysql://[user[:password]@][host][:port]/database[?options]
```

### Examples

**Local:**
```python
database_url_sync="mysql://root:password@localhost:3306/myapp"
```

**With charset:**
```python
database_url_sync="mysql://user:pass@localhost/myapp?charset=utf8mb4"
```

**With SSL:**
```python
database_url_sync="mysql://user:pass@localhost/myapp?ssl_ca=/path/to/ca.pem"
```

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `charset` | Character set | `charset=utf8mb4` |
| `ssl_ca` | CA certificate | `ssl_ca=/path/to/ca.pem` |
| `ssl_cert` | Client certificate | `ssl_cert=/path/to/cert.pem` |
| `ssl_key` | Client key | `ssl_key=/path/to/key.pem` |

### MariaDB

MariaDB uses the same URL format as MySQL:

```python
database_url_sync="mysql://user:pass@localhost:3306/myapp"
```

Configure with `database_type="mariadb"`:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    database_type = "mariadb"
    database_url_sync = "mysql://localhost/myapp"
```

## ClickHouse

### Basic Format

```
http://[user[:password]@]host[:port]/database[?options]
```

### Examples

**Local:**
```python
database_url_sync="http://default:@localhost:8123/myapp"
```

**With authentication:**
```python
database_url_sync="http://user:password@localhost:8123/myapp"
```

**With HTTPS:**
```python
database_url_sync="https://user:password@clickhouse.example.com:8443/myapp"
```

### Common Options

| Option | Description | Example |
|--------|-------------|---------|
| `compression` | Enable compression | `compression=1` |
| `connect_timeout` | Connection timeout | `connect_timeout=10` |
| `send_timeout` | Send timeout | `send_timeout=300` |
| `receive_timeout` | Receive timeout | `receive_timeout=300` |

**Example:**
```python
database_url_sync="http://user:pass@localhost:8123/myapp?compression=1&connect_timeout=10"
```

## Environment Variables

### Basic Pattern

```python
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("DATABASE_URL")
```

### With Fallback

```python
import os
from dbwarden import DbwardenDatabase

database_url = os.getenv("DATABASE_URL", "sqlite:///./dev.db")

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql" if "postgresql" in database_url else "sqlite"
    database_url_sync = database_url
```

### Required Environment Variables

```python
import os
from dbwarden import DbwardenDatabase

DATABASE_URL = os.getenv("DATABASE_URL")
if not DATABASE_URL:
    raise ValueError("DATABASE_URL environment variable is required")

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = DATABASE_URL
```

### Multiple Databases

```python
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("PRIMARY_DATABASE_URL")

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "postgresql"
    database_url_sync = os.getenv("ANALYTICS_DATABASE_URL")
```

## URL Encoding

### Special Characters

If your password contains special characters, URL-encode them:

| Character | Encoded |
|-----------|---------|
| `@` | `%40` |
| `:` | `%3A` |
| `/` | `%2F` |
| `?` | `%3F` |
| `#` | `%23` |
| `&` | `%26` |
| `%` | `%25` |

**Example:**

Password: `p@ss:word`

```python
database_url_sync="postgresql://user:p%40ss%3Aword@localhost/myapp"
```

### Python URL Encoding

```python
from urllib.parse import quote_plus

username = "user"
password = "p@ss:word"
host = "localhost"
database = "myapp"

database_url = f"postgresql://{username}:{quote_plus(password)}@{host}/{database}"
# Result: postgresql://user:p%40ss%3Aword@localhost/myapp
```

## Connection Pools

### PostgreSQL Pool Options

```python
database_url_sync="postgresql://user:pass@localhost/myapp?pool_size=20&max_overflow=10&pool_timeout=30"
```

| Option | Description | Default |
|--------|-------------|---------|
| `pool_size` | Max connections in pool | 5 |
| `max_overflow` | Extra connections if pool full | 10 |
| `pool_timeout` | Wait time for connection (seconds) | 30 |
| `pool_recycle` | Recycle connections after (seconds) | -1 (never) |

### Connection Lifetime

Recycle connections after 1 hour:

```python
database_url_sync="postgresql://user:pass@localhost/myapp?pool_recycle=3600"
```

## Testing Connections

### Verify URL Format

```python
from sqlalchemy import create_engine

try:
    engine = create_engine("postgresql://user:pass@localhost/myapp")
    with engine.connect() as conn:
        result = conn.execute("SELECT 1")
        print("Connection successful!")
except Exception as e:
    print(f"Connection failed: {e}")
```

### Test with dbwarden

```bash
# Check configuration
$ dbwarden settings show

# Test connection
$ dbwarden check-db
```

## Common Mistakes

### Forgetting Port

**Wrong:**
```python
database_url_sync="postgresql://user:pass@localhost/myapp"  # Uses default port 5432
```

**If you need a different port:**
```python
database_url_sync="postgresql://user:pass@localhost:5433/myapp"
```

### Missing Slashes

**Wrong:**
```python
database_url_sync="sqlite://./app.db"  # Only 2 slashes
```

**Correct:**
```python
database_url_sync="sqlite:///./app.db"  # 3 slashes for relative path
database_url_sync="sqlite:////absolute/path/app.db"  # 4 slashes for absolute path
```

### Special Characters Not Encoded

**Wrong:**
```python
database_url_sync="postgresql://user:p@ss@localhost/myapp"  # @ not encoded
```

**Correct:**
```python
database_url_sync="postgresql://user:p%40ss@localhost/myapp"  # @ encoded as %40
```

## What's Next?

- **[Model Discovery](model-discovery.md)** - Configure model paths
- **[Dev Mode](dev-mode.md)** - Local development URLs
- **[Production Patterns](production-patterns.md)** - Real-world examples
- **[Troubleshooting](troubleshooting.md)** - Connection issues

========================================================================
PAGE: https://docs.dbwarden.org/configuration/credentials/
========================================================================

# Credentials and Secrets

Never hardcode database credentials in `dbwarden.py`. This page covers how to inject secrets safely.

New projects should declare fields on a `DbwardenDatabase` subclass. The
`database_config(...)` function alternative is shown where it keeps the
credential-loading pattern concise; both APIs accept the same URL values.

## The problem

The quick-start examples show inline connection strings:

```python
# Do not ship this
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_url_sync = "postgresql://admin:s3cr3t@localhost:5432/myapp"
    ...
```

`dbwarden.py` is Python source. It ends up in version control. Credentials in source are a liability.

## Environment variables

The standard pattern: read from the environment at config load time.

```python
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("DATABASE_URL")
    model_paths = ["app.models"]
    secure_values = True
```

`secure_values=True` tells dbwarden to redact the URL in CLI output and logs.

### Fail fast on missing env var

```python
import os
from dbwarden import DbwardenDatabase

DATABASE_URL = os.getenv("DATABASE_URL")
if not DATABASE_URL:
    raise ValueError("DATABASE_URL is required")

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = DATABASE_URL
    model_paths = ["app.models"]
    secure_values = True
```

Without the guard, a missing env var silently passes `None` to `database_url_sync`, which fails later with a confusing error.

## `.env` files with python-dotenv

For local development, use a `.env` file to avoid setting vars manually each session.

Install:

```bash
uv add python-dotenv
```

Create `.env` in your project root:

```
DATABASE_URL=postgresql://dev_user:dev_pass@localhost:5432/myapp_dev
```

Load it at the top of `dbwarden.py`:

```python
import os
from dotenv import load_dotenv
from dbwarden import DbwardenDatabase

load_dotenv()

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("DATABASE_URL")
    model_paths = ["app.models"]
    secure_values = True
```

Add `.env` to `.gitignore`. Commit a `.env.example` with placeholder values:

```
DATABASE_URL=postgresql://user:password@localhost:5432/myapp
```

## Multi-database with separate secrets

Each database gets its own env var:

```python
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("PRIMARY_DATABASE_URL")
    model_paths = ["app.models"]
    secure_values = True

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = os.getenv("ANALYTICS_DATABASE_URL")
    model_paths = ["app.models.analytics"]
    secure_values = True
```

## Dev mode with secrets

Dev mode can also use env vars, keeping SQLite paths out of source:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("DATABASE_URL")
    dev_database_type = "sqlite"
    dev_database_url = os.getenv("DEV_DATABASE_URL", "sqlite:///./dev.db")
    model_paths = ["app.models"]
    secure_values = True
```

`DEV_DATABASE_URL` defaults to a local SQLite path if not set, which is reasonable for development.

## CI/CD environments

In GitHub Actions, set secrets in the repository settings and reference them in the workflow:

```yaml
- name: Run migrations
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
  run: dbwarden migrate --database primary
```

In GitLab CI, use masked CI/CD variables:

```yaml
migrate:
  script:
    - dbwarden migrate --database primary
  variables:
    DATABASE_URL: $DATABASE_URL  # set in GitLab CI/CD settings
```

## Third-party secret managers

For production systems using Vault, AWS Secrets Manager, or Infisical, fetch the secret before passing it to `DbwardenDatabase`:

```python
import os
import boto3
import json
from dbwarden import DbwardenDatabase

def get_secret(secret_name: str) -> str:
    client = boto3.client("secretsmanager")
    response = client.get_secret_value(SecretId=secret_name)
    secret = json.loads(response["SecretString"])
    return secret["database_url"]

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = get_secret("myapp/production/database")
    model_paths = ["app.models"]
    secure_values = True
```

`dbwarden.py` is plain Python, so any secret retrieval logic is valid here.

## `secure_values=True`

When set, dbwarden redacts the connection URL in:

- `dbwarden settings show` output
- Log lines

The URL is still used internally for connections. This prevents accidental credential exposure in terminal output shared in screenshots or logs.

See also: [Production Patterns](production-patterns.md) | [CI/CD Patterns](../advanced/ci-cd-patterns.md)

========================================================================
PAGE: https://docs.dbwarden.org/configuration/dev-mode/
========================================================================

# Dev Mode

Use SQLite for local development and PostgreSQL in production with the same codebase.

New projects should declare fields on a `DbwardenDatabase` subclass. The
`database_config(...)` function alternative is shown where it keeps the
example concise.

## What Is Dev Mode?

Dev mode lets you configure **two database URLs**:
- **Production URL** - Used by default
- **Dev URL** - Used when you pass `--dev` flag

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"    # Production
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./dev.db"           # Development
```

Run commands with `--dev`:

```bash
$ dbwarden --dev migrate        # Uses SQLite
$ dbwarden --dev status         # Uses SQLite
$ dbwarden migrate              # Uses PostgreSQL
```

## Why Use Dev Mode?

### Speed

SQLite is **much faster** for local development:
- No network latency
- No authentication overhead
- File-based, not server-based
- Instant startup

### Simplicity

No PostgreSQL server required:
- No Docker setup
- No installation
- No configuration
- Works on all platforms

### Safety

Can't accidentally affect production:
- Dev database is a local file
- Each developer has their own database
- Easy to reset (`rm dev.db`)
- No shared state

### Portability

Easy to share between developers:
- One config file works everywhere
- No server setup instructions
- Fresh developers can start immediately

## Basic Setup

### Step 1: Configure Dev Database

```python
# dbwarden.py
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    dev_database_type = "sqlite"                    #  Add this
    dev_database_url = "sqlite:///./dev.db"         #  Add this
    model_paths = ["app.models"]
```

### Step 2: Use Dev Mode

```bash
# Development workflow
$ dbwarden --dev make-migrations "create users"
$ dbwarden --dev migrate
$ dbwarden --dev status

# Production workflow
$ dbwarden make-migrations "create users"
$ dbwarden migrate
$ dbwarden status
```

## Dev Mode Workflow

### Daily Development

```bash
# Morning: pull latest code
git pull

# Run migrations against dev database
$ dbwarden --dev migrate

# Work on features...
# Add new models

# Generate migration
$ dbwarden --dev make-migrations "add orders table"

# Test migration
$ dbwarden --dev migrate

# Verify
$ dbwarden --dev status

# Commit
git add migrations/primary/0002_add_orders_table.sql
git commit -m "Add orders table"
```

### Testing Rollbacks

```bash
# Apply migration
$ dbwarden --dev migrate

# Test rollback
$ dbwarden --dev rollback

# Re-apply
$ dbwarden --dev migrate
```

### Fresh Start

Reset your dev database anytime:

```bash
# Delete dev database
rm dev.db

# Re-run migrations
$ dbwarden --dev migrate
```

## Production Workflow

Dev mode only affects **local development**. Production uses the main URL:

### CI/CD Pipeline

```yaml
# .github/workflows/deploy.yml
- name: Run migrations
  run: dbwarden migrate  # No --dev flag
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
```

### Production Deployment

```bash
# Staging
$ dbwarden migrate --database primary

# Production
$ dbwarden migrate --database primary
```

Dev mode is **never used** in CI/CD or production.

## SQLite Limitations

### What Works

Most features work in SQLite:
-  Tables, columns, indexes
-  Primary keys, foreign keys
-  Unique constraints
-  Basic data types
-  Transactions

### What Doesn't Work

Some PostgreSQL features aren't available in SQLite:
-  Advanced types (JSONB, arrays, enums)
-  Partial indexes
-  Generated columns (in older SQLite)
-  Multiple schemas
-  Concurrent writes

### Translation

dbwarden **translates types and defaults** for SQLite compatibility when generating SQL from models (see [SQL Translation](../sql-translation.md)), but does not translate arbitrary hand-written SQL in manual migration files. Your manual migrations should work on both SQLite and PostgreSQL.

**Approach 1:** Write portable SQL

```sql
--  Works on both
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    email VARCHAR(255) NOT NULL
);

--  PostgreSQL-specific
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    metadata JSONB
);
```

**Approach 2:** Use PostgreSQL for dev too

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    dev_database_type = "postgresql"                         # Same as prod
    dev_database_url = "postgresql://localhost/myapp_dev"    # Different database
```

## Environment-Based Configuration

### Automatic Dev Mode

Use environment variables to automatically detect dev:

```python
import os
from dbwarden import DbwardenDatabase

is_dev = os.getenv("ENV", "dev") in ["dev", "development", "local"]

if is_dev:
    database_url = "sqlite:///./dev.db"
    database_type = "sqlite"
else:
    database_url = os.getenv("DATABASE_URL")
    database_type = "postgresql"

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = database_type
    database_url_sync = database_url
```

Run commands:

```bash
# Dev
ENV=dev dbwarden migrate

# Production
ENV=production dbwarden migrate
```

## Multiple Dev Databases

If you have multiple databases, configure dev mode for each:

```python
from dbwarden import DbwardenDatabase

# Primary database
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/main"
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./dev_primary.db"
    model_paths = ["app.models.primary"]

# Analytics database
class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/analytics"
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./dev_analytics.db"
    model_paths = ["app.models.analytics"]
```

Run against all dev databases:

```bash
$ dbwarden --dev migrate --all
$ dbwarden --dev status --all
```

## Common Patterns

### Pattern 1: SQLite for Dev, PostgreSQL for Prod (Recommended)

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./dev.db"
```

**Pros:**
- Fast local iteration
- No server setup
- Easy to reset

**Cons:**
- SQL must be portable
- Some features unavailable in dev

### Pattern 2: PostgreSQL for Both

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://prod-host/myapp"
    dev_database_type = "postgresql"
    dev_database_url = "postgresql://localhost/myapp_dev"
```

**Pros:**
- Identical environments
- Use all PostgreSQL features
- Catches more bugs in dev

**Cons:**
- Requires PostgreSQL server locally
- Slower than SQLite

### Pattern 3: Dynamic Based on Environment

```python
import os
from dbwarden import DbwardenDatabase

environment = os.getenv("ENV", "dev")

if environment == "production":
    database_url = os.getenv("DATABASE_URL")
    database_type = "postgresql"
elif environment == "staging":
    database_url = os.getenv("STAGING_DATABASE_URL")
    database_type = "postgresql"
else:
    database_url = "sqlite:///./dev.db"
    database_type = "sqlite"

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = database_type
    database_url_sync = database_url
```

## Testing with Dev Mode

### Unit Tests

Use SQLite for fast unit tests:

```python
# tests/conftest.py
import pytest
from sqlalchemy import create_engine
from app.models import Base

@pytest.fixture(scope="function")
def db():
    engine = create_engine("sqlite:///:memory:")
    Base.metadata.create_all(engine)
    yield engine
    Base.metadata.drop_all(engine)
```

### Integration Tests

Use `--dev` for integration tests:

```bash
# Run integration tests
$ dbwarden --dev migrate
pytest tests/integration/
```

## Troubleshooting

### "SQL syntax error in SQLite"

**Cause:** Using PostgreSQL-specific SQL.

**Solution:** Make SQL portable or use PostgreSQL for dev:

```python
dev_database_type="postgresql"
dev_database_url="postgresql://localhost/myapp_dev"
```

### "dev_database_url is required"

**Cause:** Set `dev_database_type` without `dev_database_url`.

**Solution:** Add both:

```python
dev_database_type="sqlite"
dev_database_url="sqlite:///./dev.db"  #  Add this
```

### Dev database not updating

**Cause:** Forgot `--dev` flag.

**Solution:** Use `--dev`:

```bash
$ dbwarden --dev migrate  #  Add --dev
```

## What's Next?

- **[Multi-Database](multi-database.md)** - Multiple databases with dev mode
- **[Production Patterns](production-patterns.md)** - Deploy to production
- **[Troubleshooting](troubleshooting.md)** - Common issues

========================================================================
PAGE: https://docs.dbwarden.org/configuration/
========================================================================

# Configuration

dbwarden uses Python-based configuration with declarative `DbwardenDatabase`
classes as the default method. The equivalent `database_config()` function
alternative remains supported, and some plugins use it in their examples.

**One configuration source** for migrations, CLI tools, and runtime: no split configs.

## Quick Start

The simplest configuration possible:

```python
# dbwarden.py
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "sqlite"
    database_url_sync = "sqlite:///./app.db"
```

That's it! **4 parameters** to get started.

Run your first migration:

```bash
$ dbwarden init
$ dbwarden make-migrations "initial schema"
$ dbwarden migrate
```

## Learning Path

### New to dbwarden?
Start here to understand configuration basics:

1. **[Quick Start](quick-start.md)** - Your first configuration in 2 minutes
2. **[Concepts](concepts.md)** - How configuration works
3. **[Connection URLs](connection-urls.md)** - Database connection formats

### Building Your Configuration
Learn specific features:

- **[Model Discovery](model-discovery.md)** - How dbwarden finds your SQLAlchemy models
- **[Dev Mode](dev-mode.md)** - Local development with SQLite
- **[Multi-Database](multi-database.md)** - Configure multiple databases

### Production Ready
Deploy with confidence:

- **[Production Patterns](production-patterns.md)** - Real-world examples
- **[Troubleshooting](troubleshooting.md)** - Common issues and solutions

### Complete Reference
- **[Configuration API](../reference/configuration-api.md)** - Complete function signature and parameters

## Key Features

###  Simple Configuration

Define once, use everywhere:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    model_paths=["app.models"],
)
```

###  Dev Mode

Use SQLite locally, PostgreSQL in production:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/myapp",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",
)
```

Run commands with `--dev`:

```bash
$ dbwarden --dev migrate
$ dbwarden --dev status
```

###  Multi-Database

Configure as many databases as you need:

```python
# Primary database
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    model_paths=["app.models.primary"],
)

# Analytics database
analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://localhost:8123/analytics",
    model_paths=["app.models.analytics"],
)
```

###  Security First

Keep credentials out of code:

```python
import os

primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=os.getenv("DATABASE_URL"),
    secure_values=True,  # Hide credentials in output
)
```

###  Validation

dbwarden validates your configuration:

-  Exactly one `default=True`
-  Unique database names
-  No duplicate URLs
-  Required `model_paths` for multi-database
-  Consistent dev mode configuration

## Configuration Loading

dbwarden discovers your configuration automatically:

1. **Walk project tree** for `dbwarden.py` files
2. **Check `DBWARDEN_CONFIG_MODULE`** environment variable
3. **Scan for `database_config()` calls** in your codebase (full project tree walk)

`dbwarden.py` is the default convention and the file created by `dbwarden init`. The default class API or the supported `database_config(...)` function alternative can live in any discovered Python file inside your project.

## Common Patterns

### Single Database (Minimal)

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
```

### With Dev Mode (Recommended)

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./dev.db"
    model_paths = ["app.models"]
```

### Multiple Databases

```python
from dbwarden import DbwardenDatabase

# Primary
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/main"
    model_paths = ["app.models.primary"]

# Analytics
class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/analytics"
    model_paths = ["app.models.analytics"]
```

### Production with Environment Variables

```python
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("DATABASE_URL")
    model_paths = ["app.models"]
    secure_values = True
```

## Why Python Configuration?

**vs TOML/YAML/INI:**
-  Type checking with your IDE
-  Dynamic configuration (loops, conditionals)
-  Environment variable integration
-  No schema mismatches
-  Can compute values

**vs Environment Variables Only:**
-  Version controlled
-  Self-documenting
-  Validation at load time
-  Multiple databases easy
-  Can reference code structures

## What's Next?

Ready to configure your first database? Start here:

- **[Quick Start](quick-start.md)** - Build your first configuration
- **[Concepts](concepts.md)** - Understand how it works
- **[Production Patterns](production-patterns.md)** - Real-world examples

Already familiar with configuration? Jump to:

- **[Connection URLs](connection-urls.md)** - URL format reference
- **[Troubleshooting](troubleshooting.md)** - Common issues
- **[Configuration API](../reference/configuration-api.md)** - Complete reference

========================================================================
PAGE: https://docs.dbwarden.org/configuration/model-discovery/
========================================================================

# Model Discovery

Learn how dbwarden discovers your SQLAlchemy models for migration generation.

Examples use the `database_config(...)` function alternative for compactness.
The same `model_paths` and `model_tables` fields can be declared on the
default `DbwardenDatabase` class API.

## What Is Model Discovery?

Model discovery is the process where dbwarden:
1. Imports Python modules
2. Finds SQLAlchemy model classes
3. Extracts table metadata
4. Uses metadata to generate migrations

Generated `*.data.py` files are not model modules. dbwarden excludes them from recursive model discovery so frozen migration artifacts cannot become live declarations. Put live transition modules in `data_paths` instead.

## The `model_paths` Parameter

### Basic Usage

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    model_paths = ["app.models"]  #  Discover models here
```

### What Gets Discovered

dbwarden looks for classes that inherit from:
- `DeclarativeBase` (SQLAlchemy 2.0+)
- `declarative_base()` return value (SQLAlchemy 1.4)

**Example models:**

```python
# app/models.py
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from sqlalchemy import String, Integer

class Base(DeclarativeBase):
    pass

class User(Base):  #  Discovered
    __tablename__ = "users"
    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))

class Order(Base):  #  Discovered
    __tablename__ = "orders"
    id: Mapped[int] = mapped_column(Integer, primary_key=True)
```

## When Is It Required?

### Single Database (Optional)

For single-database projects, `model_paths` is optional:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "sqlite"
    database_url_sync = "sqlite:///./app.db"
    # No model_paths - dbwarden scans entire codebase
```

dbwarden will scan your entire codebase for models.

Even for single-database projects, specifying `model_paths` makes discovery faster and more predictable.

### Multiple Databases (Required)

For multi-database projects, `model_paths` is **required** for each database:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/main"
    model_paths = ["app.models.primary"]  #  Required

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/analytics"
    model_paths = ["app.models.analytics"]  #  Required
```

**Why required?** To prevent ambiguity about which models belong to which database.

## Discovery Algorithm

### Step 1: Import Modules

dbwarden imports each module in `model_paths`:

```python
model_paths=["app.models", "app.legacy.models"]
```

Becomes:

```python
import app.models
import app.legacy.models
```

### Step 2: Recursive Discovery

For each imported module, dbwarden recursively imports submodules:

```
app/
  models/
    __init__.py       #  Imported
    user.py           #  Imported
    order.py          #  Imported
    admin/
      __init__.py     #  Imported
      admin_user.py   #  Imported
```

### Step 3: Find Model Classes

For each module, dbwarden inspects all classes and finds those inheriting from `DeclarativeBase`.

### Step 4: Extract Metadata

For each model class, dbwarden extracts:
- Table name
- Columns (name, type, constraints)
- Indexes
- Foreign keys
- Check constraints
- Unique constraints

## Module Path Examples

### Single Module

```python
model_paths=["app.models"]
```

Discovers models from:
- `app/models.py` (if it's a file)
- `app/models/__init__.py` (if it's a package)
- `app/models/*.py` (all submodules)

### Multiple Modules

```python
model_paths=["app.models", "app.legacy"]
```

Discovers models from both `app.models` and `app.legacy`.

### Nested Modules

```python
model_paths=["app.models.api", "app.models.admin"]
```

Discovers models from:
- `app/models/api.py` or `app/models/api/*.py`
- `app/models/admin.py` or `app/models/admin/*.py`

### Absolute vs Relative

**Absolute (recommended):**
```python
model_paths=["app.models"]  # From project root
```

**Not supported:**
```python
model_paths=["./models"]  # Relative paths don't work
model_paths=["models"]     # May work if on PYTHONPATH
```

## Common Patterns

### Pattern 1: Single Module

```
app/
  models.py    # All models in one file
```

```python
model_paths=["app.models"]
```

### Pattern 2: Module Package

```
app/
  models/
    __init__.py
    user.py
    order.py
    product.py
```

```python
model_paths=["app.models"]
```

### Pattern 3: Multi-Database

```
app/
  models/
    primary/
      __init__.py
      user.py
      order.py
    analytics/
      __init__.py
      event.py
      metric.py
```

```python
# Primary database
model_paths=["app.models.primary"]

# Analytics database  
model_paths=["app.models.analytics"]
```

### Pattern 4: Legacy + New

```
app/
  models/      # New models
    __init__.py
    user.py
  legacy/      # Legacy models
    models.py
```

```python
model_paths=["app.models", "app.legacy.models"]
```

## Model Path Validation

### No Overlap (Default)

By default, model paths cannot overlap between databases:

```python
from dbwarden import DbwardenDatabase

#  Error: overlap detected
class Primary(DbwardenDatabase):
    database_name = "primary"
    model_paths = ["app.models"]

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    model_paths = ["app.models"]  # Same path!
```

### Allow Overlap

If models genuinely belong to multiple databases:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    model_paths = ["app.shared"]
    overlap_models = True  #  Allow overlap

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    model_paths = ["app.shared"]
    overlap_models = True  #  Allow overlap
```

Both databases will include the same tables. Make sure this is intentional.

## Troubleshooting

### "No SQLAlchemy models found"

**Symptom:** dbwarden can't find your models.

**Causes:**

1. **Models not imported**

```python
# app/models/__init__.py
#  Wrong - models not imported
from sqlalchemy.orm import DeclarativeBase

class Base(DeclarativeBase):
    pass

#  Correct - import models
from app.models.user import User
from app.models.order import Order
```

2. **Wrong module path**

```python
#  Wrong
model_paths=["models"]  # Not on PYTHONPATH

#  Correct
model_paths=["app.models"]
```

3. **Circular imports**

```python
# app/models/user.py
from app.models.order import Order  #  Circular import

# app/models/order.py
from app.models.user import User  #  Circular import
```

**Solution:** Use forward references:

```python
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from app.models.order import Order
```

### "model_paths is required"

**Symptom:** Error when running commands with multiple databases.

**Cause:** Multiple databases configured without `model_paths`.

**Solution:** Add `model_paths` to each database:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],  #  Add this
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],  #  Add this
)
```

### "model_paths overlap detected"

**Symptom:** Two databases have overlapping model paths.

**Cause:** Same path used for multiple databases.

**Solution 1:** Use separate paths:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],  # Different path
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],  # Different path
)
```

**Solution 2:** Allow overlap (if intentional):

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.shared"],
    overlap_models=True,  #  Allow overlap
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.shared"],
    overlap_models=True,  #  Allow overlap
)
```

### Import Errors

**Symptom:** `ModuleNotFoundError` or `ImportError` when running commands.

**Cause:** dbwarden tries to import module but it doesn't exist.

**Solution:** Verify the module path:

```bash
python -c "import app.models"  # Test import
```

If import fails, fix your module structure or PYTHONPATH.

## Performance Considerations

### Slow Discovery

If discovery is slow, reduce the search space:

**Before (slow):**
```python
model_paths=["app"]  # Scans entire app
```

**After (fast):**
```python
model_paths=["app.models"]  # Only scans models
```

### Import Side Effects

Models should be pure:

```python
#  Bad - side effects on import
class User(Base):
    __tablename__ = "users"
    ...

print("User model loaded!")  # Side effect

#  Good - no side effects
class User(Base):
    __tablename__ = "users"
    ...
```

## Advanced: Dynamic Model Paths

You can compute `model_paths` dynamically:

```python
import os
from dbwarden import DbwardenDatabase

environment = os.getenv("ENV", "dev")

if environment == "production":
    model_paths = ["app.models.production"]
else:
    model_paths = ["app.models.dev"]

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "..."
    model_paths = model_paths
```

## What's Next?

- **[Dev Mode](dev-mode.md)** - Local development workflows
- **[Multi-Database](multi-database.md)** - Organize multi-database models
- **[Troubleshooting](troubleshooting.md)** - Common configuration issues

========================================================================
PAGE: https://docs.dbwarden.org/configuration/multi-database/
========================================================================

# Multi-Database

Configure and manage multiple databases in a single project.

## When to Use Multiple Databases

Common scenarios:
- **Microservices** - Each service has its own database
- **Read/Write Split** - Primary for writes, replica for reads
- **Domain Separation** - Transactions, analytics, logs in separate databases
- **Legacy Integration** - New and old databases coexist
- **Multi-Tenancy** - One database per tenant

## Basic Setup

Define each database as a concrete `DbwardenDatabase` subclass:

```python
# dbwarden.py
from dbwarden import DbwardenDatabase

# Primary database
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/main"
    model_paths = ["app.models.primary"]

# Analytics database
class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/analytics"
    model_paths = ["app.models.analytics"]

# Logging database
class Logging(DbwardenDatabase):
    database_name = "logging"
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/logs"
    model_paths = ["app.models.logging"]
```

The equivalent `database_config(...)` function API remains supported, including
for plugins and integration code that use function-style declarations.

The same setup can use inherited declarative classes when databases share
backend or model settings:

```python
from dbwarden import DbwardenDatabase


class ServiceDatabase(DbwardenDatabase):
    __abstract__ = True
    database_type = "postgresql"
    model_paths = ["app.models"]


class Primary(ServiceDatabase):
    database_name = "primary"
    database_url_sync = "postgresql://localhost/main"
    default = True


class Analytics(ServiceDatabase):
    database_name = "analytics"
    database_url_sync = "postgresql://localhost/analytics"
```

Both APIs can also be mixed in one source, but database names must remain
unique and exactly one entry must set `default=True`.

## Model Organization

### Pattern 1: Separate Modules

```
app/
  models/
    primary/
      __init__.py
      user.py
      order.py
    analytics/
      __init__.py
      event.py
      metric.py
    logging/
      __init__.py
      audit_log.py
```

Configuration:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],
)

analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],
)

logging = database_config(
    database_name="logging",
    model_paths=["app.models.logging"],
)
```

### Pattern 2: Shared Base Classes

```python
# app/models/base.py
from sqlalchemy.orm import DeclarativeBase

class PrimaryBase(DeclarativeBase):
    pass

class AnalyticsBase(DeclarativeBase):
    pass

# app/models/primary/user.py
from app.models.base import PrimaryBase

class User(PrimaryBase):
    __tablename__ = "users"
    ...

# app/models/analytics/event.py
from app.models.base import AnalyticsBase

class Event(AnalyticsBase):
    __tablename__ = "events"
    ...
```

## CLI Usage

### Target Specific Database

```bash
# Migrate primary
$ dbwarden migrate --database primary

# Migrate analytics
$ dbwarden migrate --database analytics

# Status for logging
$ dbwarden status --database logging
```

### Target All Databases

```bash
# Migrate all
$ dbwarden migrate --all

# Status for all
$ dbwarden status --all

# Rollback all
$ dbwarden rollback -d logging
```

### Default Database

The database with `default=True` is used when `--database` is omitted:

```bash
# These are equivalent when primary is default:
$ dbwarden migrate
$ dbwarden migrate --database primary
```

## Migration Directories

Each database has its own migration directory:

```
migrations/
  primary/
    0001_create_users.sql
    0002_create_orders.sql
  analytics/
    0001_create_events.sql
    0002_create_metrics.sql
  logging/
    0001_create_audit_logs.sql
```

Configure custom directories:

```python
primary = database_config(
    database_name="primary",
    migrations_dir="migrations/primary",  # Custom path
)
```

## Independent Migration Histories

Each database maintains its own migration history:

```bash
# Check primary history
$ dbwarden history --database primary
Applied Migrations (primary)
  0001_create_users (2024-01-15 10:30:00)
  0002_create_orders (2024-01-16 11:00:00)

# Check analytics history
$ dbwarden history --database analytics
Applied Migrations (analytics)
  0001_create_events (2024-01-15 10:35:00)
```

Migrations are **completely independent** - you can migrate one database without affecting others.

## Dev Mode with Multiple Databases

Configure dev mode for each database:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_primary.db",
    model_paths=["app.models.primary"],
)

analytics = database_config(
    database_name="analytics",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/analytics",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_analytics.db",
    model_paths=["app.models.analytics"],
)
```

Use dev mode:

```bash
# Dev mode for all databases
$ dbwarden --dev migrate --all

# Dev mode for specific database
$ dbwarden --dev migrate --database analytics
```

## Common Patterns

### Pattern 1: Read/Write Split

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://primary-host/myapp",
    model_paths=["app.models"],
)

replica = database_config(
    database_name="replica",
    database_type="postgresql",
    database_url_sync="postgresql://replica-host/myapp",
    model_paths=["app.models"],  # Same models
    overlap_models=True,          # Allow overlap
)
```

**Note:** Run migrations only against primary; replica replicates automatically.

### Pattern 2: Domain Separation

```python
# Transactions
transactions = database_config(
    database_name="transactions",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/transactions",
    model_paths=["app.models.transactions"],
)

# Analytics
analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://localhost:8123/analytics",
    model_paths=["app.models.analytics"],
)

# Audit logs
audit = database_config(
    database_name="audit",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/audit",
    model_paths=["app.models.audit"],
)
```

### Pattern 3: Multi-Tenant

```python
tenants = ["tenant_a", "tenant_b", "tenant_c"]

for tenant in tenants:
    db = database_config(
        database_name=tenant,
        default=(tenant == "tenant_a"),
        database_type="postgresql",
        database_url_sync=f"postgresql://localhost/{tenant}",
        model_paths=["app.models"],  # Same models for all tenants
    )
```

## Validation Rules

### Required: `model_paths`

When you have multiple databases, each **must** specify `model_paths`:

```python
#  Error: model_paths required
analytics = database_config(
    database_name="analytics",  # Missing model_paths
)

#  Correct
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],
)
```

### No Overlap (Default)

Model paths cannot overlap:

```python
#  Error: overlap detected
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models"],  # Same path
)
```

### Allow Overlap

For read replicas or shared models:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    overlap_models=True,  #  Allow overlap
)
replica = database_config(
    database_name="replica",
    model_paths=["app.models"],
    overlap_models=True,  #  Allow overlap
)
```

## Troubleshooting

### "model_paths is required"

**Solution:** Add `model_paths` to all databases:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],  #  Add this
)
```

### "model_paths overlap detected"

**Solution 1:** Use separate paths:
```python
model_paths=["app.models.primary"]
model_paths=["app.models.analytics"]
```

**Solution 2:** Allow overlap:
```python
overlap_models=True
```

### Wrong database targeted

**Check default:**
```bash
$ dbwarden settings show  # Shows which is default
```

**Be explicit:**
```bash
$ dbwarden migrate --database analytics  # Specify database
```

## What's Next?

- **[Production Patterns](production-patterns.md)** - Deploy multi-database apps
- **[Troubleshooting](troubleshooting.md)** - Common issues

========================================================================
PAGE: https://docs.dbwarden.org/configuration/production-patterns/
========================================================================

# Production Patterns

Real-world configuration patterns for production deployments.

New projects should declare fields on a `DbwardenDatabase` subclass. The
`database_config(...)` function alternative is shown where it keeps
environment-variable patterns concise.

## Environment Variables

### Basic Pattern

```python
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("DATABASE_URL")
    model_paths = ["app.models"]
    secure_values = True
```

### With Validation

```python
import os
from dbwarden import DbwardenDatabase

DATABASE_URL = os.getenv("DATABASE_URL")
if not DATABASE_URL:
    raise ValueError("DATABASE_URL environment variable is required")

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = DATABASE_URL
    model_paths = ["app.models"]
    secure_values = True
```

### With Defaults

```python
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv(
        "DATABASE_URL",
        "postgresql://localhost/myapp"  # Fallback
    )
    model_paths = ["app.models"]
```

## Docker

### Docker Compose

```yaml
# docker-compose.yml
services:
  app:
    build: .
    environment:
      DATABASE_URL: postgresql://user:password@db:5432/myapp
    depends_on:
      - db
  
  db:
    image: postgres:15
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: myapp
```

```python
# dbwarden.py
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("DATABASE_URL")
    model_paths = ["app.models"]
```

### Dockerfile

```dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY pyproject.toml uv.lock .
RUN uv sync

COPY . .

# Run migrations on container start
CMD ["sh", "-c", "dbwarden migrate && python app/main.py"]
```

## Kubernetes

### Secrets

```yaml
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: database-secret
type: Opaque
stringData:
  url: postgresql://user:password@postgres-service:5432/myapp
```

### Deployment with Init Container

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      initContainers:
      # Run migrations before app starts
      - name: migrate
        image: myapp:latest
        command: ["dbwarden", "migrate"]
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-secret
              key: url
      
      containers:
      - name: app
        image: myapp:latest
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-secret
              key: url
```

### ConfigMap for Model Paths

```yaml
# config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  model_paths: "app.models"
```

## AWS

### RDS with Secrets Manager

```python
import os
import json
import boto3
from dbwarden import DbwardenDatabase

def get_database_url():
    secret_name = os.getenv("DB_SECRET_NAME")
    region = os.getenv("AWS_REGION", "us-east-1")
    
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    secret = json.loads(response["SecretString"])
    
    return (
        f"postgresql://{secret['username']}:{secret['password']}"
        f"@{secret['host']}:{secret['port']}/{secret['dbname']}"
    )

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = get_database_url()
    model_paths = ["app.models"]
    secure_values = True
```

### RDS Connection via IAM

```python
import os
import boto3
from dbwarden import DbwardenDatabase

def get_rds_auth_token():
    rds_client = boto3.client("rds")
    return rds_client.generate_db_auth_token(
        DBHostname=os.getenv("DB_HOST"),
        Port=5432,
        DBUsername=os.getenv("DB_USER"),
    )

database_url = (
    f"postgresql://{os.getenv('DB_USER')}:{get_rds_auth_token()}"
    f"@{os.getenv('DB_HOST')}:5432/{os.getenv('DB_NAME')}"
)

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = database_url
    model_paths = ["app.models"]
```

## Multi-Environment

### Environment-Based Configuration

```python
import os
from dbwarden import DbwardenDatabase

ENVIRONMENT = os.getenv("ENVIRONMENT", "dev")

if ENVIRONMENT == "production":
    database_url = os.getenv("PROD_DATABASE_URL")
    database_type = "postgresql"
elif ENVIRONMENT == "staging":
    database_url = os.getenv("STAGING_DATABASE_URL")
    database_type = "postgresql"
else:
    database_url = "sqlite:///./dev.db"
    database_type = "sqlite"

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = database_type
    database_url_sync = database_url
    model_paths = ["app.models"]
    secure_values = (ENVIRONMENT != "dev")
```

### Separate Config Files

```python
# dbwarden.py
import os
from importlib import import_module

environment = os.getenv("ENVIRONMENT", "dev")
config_module = import_module(f"config.{environment}")
config_module.setup_databases()
```

```python
# config/production.py
import os
from dbwarden import DbwardenDatabase

def setup_databases():
    class Primary(DbwardenDatabase):
        database_name = "primary"
        default = True
        database_type = "postgresql"
        database_url_sync = os.getenv("DATABASE_URL")
        model_paths = ["app.models"]
        secure_values = True
```

## Connection Pools

### PostgreSQL with Pooling

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = (
        "postgresql://user:pass@localhost/myapp"
        "?pool_size=20"
        "&max_overflow=10"
        "&pool_timeout=30"
        "&pool_recycle=3600"
    )
    model_paths = ["app.models"]
```

### External Pooler (PgBouncer)

```python
from dbwarden import DbwardenDatabase

# Connection through PgBouncer
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:pass@pgbouncer:6432/myapp"
    model_paths = ["app.models"]
```

## SSL/TLS

### PostgreSQL with SSL

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = (
        "postgresql://user:pass@host/myapp"
        "?sslmode=require"
        "&sslrootcert=/path/to/ca.pem"
        "&sslcert=/path/to/client-cert.pem"
        "&sslkey=/path/to/client-key.pem"
    )
    model_paths = ["app.models"]
```

### Environment-Based SSL

```python
import os
from dbwarden import DbwardenDatabase

ssl_mode = os.getenv("DB_SSL_MODE", "prefer")
ca_cert = os.getenv("DB_CA_CERT_PATH", "")

ssl_params = f"?sslmode={ssl_mode}"
if ca_cert:
    ssl_params += f"&sslrootcert={ca_cert}"

database_url = f"postgresql://user:pass@host/myapp{ssl_params}"

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = database_url
    model_paths = ["app.models"]
```

## High Availability

### Multiple Replicas

```python
import os
from dbwarden import DbwardenDatabase

# Primary (writes)
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("PRIMARY_DATABASE_URL")
    model_paths = ["app.models"]

# Replica (reads)
class Replica(DbwardenDatabase):
    database_name = "replica"
    database_type = "postgresql"
    database_url_sync = os.getenv("REPLICA_DATABASE_URL")
    model_paths = ["app.models"]
    overlap_models = True
```

### Automatic Failover

```python
import os
from dbwarden import DbwardenDatabase

# Try primary, fallback to replica
primary_url = os.getenv("PRIMARY_DATABASE_URL")
replica_url = os.getenv("REPLICA_DATABASE_URL")

# Application logic handles failover
database_url = primary_url  # Start with primary

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = database_url
    model_paths = ["app.models"]
```

## Monitoring

### Application Name

```python
import os
from dbwarden import DbwardenDatabase

app_name = os.getenv("APP_NAME", "myapp")
hostname = os.getenv("HOSTNAME", "unknown")

database_url = (
    f"postgresql://user:pass@host/myapp"
    f"?application_name={app_name}-{hostname}"
)

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = database_url
    model_paths = ["app.models"]
```

Check active connections:

```sql
SELECT application_name, count(*)
FROM pg_stat_activity
GROUP BY application_name;
```

## Security Best Practices

### Never Commit Credentials

```python
#  Bad
database_url_sync="postgresql://user:password@localhost/myapp"

#  Good
database_url_sync=os.getenv("DATABASE_URL")
```

### Use Least Privilege

Create application user with minimal permissions:

```sql
CREATE USER myapp_user WITH PASSWORD 'secret';
GRANT CONNECT ON DATABASE myapp TO myapp_user;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO myapp_user;
-- Don't grant DROP, TRUNCATE, CREATE, etc.
```

### Rotate Credentials

```python
from dbwarden import DbwardenDatabase

# Use short-lived tokens
database_url = get_temporary_database_credentials()

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = database_url
    model_paths = ["app.models"]
```

## CI/CD Integration

### GitHub Actions

```yaml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Install dependencies
        run: uv add dbwarden
      
      - name: Run migrations
        run: dbwarden migrate --database primary
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
```

### GitLab CI

```yaml
migrate:
  stage: deploy
  script:
    - uv add dbwarden
    - dbwarden migrate --database primary
  environment:
    name: production
  only:
    - main
  variables:
    DATABASE_URL: $DATABASE_URL
```

## What's Next?

- **[Troubleshooting](troubleshooting.md)** - Common production issues
- **[Configuration API Reference](../reference/configuration-api.md)** - Complete parameter docs

========================================================================
PAGE: https://docs.dbwarden.org/configuration/quick-start/
========================================================================

# Quick Start

Configure your first database in **2 minutes**.

## Prerequisites

You should have:
- Python 3.12.7+ installed
- dbwarden installed (`uv add dbwarden`)
- A database to connect to (or use SQLite)

## Step 1: Initialize

Create project structure:

```bash
$ dbwarden init
```

This creates:
- `migrations/` directory
- `dbwarden.py` configuration file

## Step 2: Your First Configuration

Open `dbwarden.py` and add the default class-based configuration:

```python
from dbwarden import DbwardenDatabase


class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "sqlite"
    database_url_sync = "sqlite:///./app.db"
```

That's it! **4 required parameters**:
- `database_name` - What to call this database
- `default` - Is this the default?
- `database_type` - What kind of database?
- `database_url_sync` - How to connect? (sync URL for CLI/migrations)

Start with SQLite for the simplest setup. Switch to PostgreSQL later.

The function alternative, `database_config(...)`, is also fully supported. Some
plugins use that form in their examples or integration code, so you may see
both configuration styles in dbwarden documentation.

## Step 3: Test the Configuration

Verify dbwarden can read your config:

```bash
$ dbwarden settings show
```

You'll see:

```
Database: PRIMARY (default)
  Default: True
  Type: SQLite
  URL: sqlite:///./app.db
  Migrations Directory: migrations/primary
  Migration Table: _dbwarden_migrations
  Seed Table: _dbwarden_seeds
  Model Paths: app
  Dev Database Type: 
  Dev Database URL: 
  Overlap Models: False
  Skip If Missing: False
```

## Step 4: Add Model Paths (Optional)

If you have SQLAlchemy models, tell dbwarden where they are:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    model_paths=["app.models"],  #  Add this
)
```

dbwarden will discover models from `app.models` and its submodules.

## Step 5: Upgrade to PostgreSQL

When you're ready for PostgreSQL:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/myapp",
    model_paths=["app.models"],
)
```

## Step 6: Add Dev Mode (Recommended)

Keep SQLite for local dev, use PostgreSQL in production:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/myapp",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",
    model_paths=["app.models"],
)
```

Now you can run commands against SQLite locally:

```bash
$ dbwarden --dev migrate
$ dbwarden --dev status
```

And against PostgreSQL in production:

```bash
$ dbwarden migrate
$ dbwarden status
```

## What Just Happened?

### `database_config` registered your database

When Python loads `dbwarden.py`, it registers the concrete `DbwardenDatabase`
class, which:
1. Validates your parameters
2. Registers the database in dbwarden's internal registry
3. Sets up migration directories

The supported `database_config(...)` function alternative performs the same
registration and validation steps.

### Declarative configuration

For shared settings, the equivalent class-based API supports inheritance:

```python
from dbwarden import DbwardenDatabase


class Shared(DbwardenDatabase):
    __abstract__ = True
    database_type = "sqlite"
    model_paths = ["app.models"]


class Primary(Shared):
    database_name = "primary"
    database_url_sync = "sqlite:///./app.db"
    default = True
```

Concrete subclasses register automatically, and `Primary.handle` provides the
same `DatabaseHandle` returned by `database_config(...)`.

### dbwarden Can Now Find Your Database

All CLI commands now know about your database:

```bash
$ dbwarden make-migrations "create users"
$ dbwarden migrate
$ dbwarden status
$ dbwarden history
```

## Common First-Time Issues

### "No configuration found"

**Cause:** dbwarden can't find `dbwarden.py`

**Solution:** Ensure you're in the project directory and `dbwarden.py` exists.

### "No SQLAlchemy models found"

**Cause:** dbwarden can't discover your models

**Solution:** Add `model_paths` to your config:

```python
model_paths=["app.models"]
```

### "Exactly one default=True required"

**Cause:** Multiple databases without one marked as default

**Solution:** Set one database to `default=True`

## Complete Minimal Example

```python
# dbwarden.py
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "sqlite"
    database_url_sync = "sqlite:///./app.db"
    model_paths = ["app.models"]
```

## Complete Production Example

```python
# dbwarden.py
import os
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = os.getenv("DATABASE_URL")
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./dev.db"
    model_paths = ["app.models"]
    secure_values = True
```

The `database_config(...)` function alternative remains supported and may be
used by plugins or integration code.

## What's Next?

- **[Concepts](concepts.md)** - Understand how configuration works
- **[Connection URLs](connection-urls.md)** - Learn URL formats for different databases
- **[Dev Mode](dev-mode.md)** - Deep dive into dev workflows
- **[Production Patterns](production-patterns.md)** - Real-world examples

========================================================================
PAGE: https://docs.dbwarden.org/configuration/troubleshooting/
========================================================================

# Troubleshooting

Solutions to common configuration issues.

New projects should declare fields on a `DbwardenDatabase` subclass. The
`database_config(...)` function alternative is also supported.

## "No configuration found"

### Symptom

```
ConfigurationError: No configuration found
```

### Causes & Solutions

**Cause 1: No `dbwarden.py` file**

```bash
# Check if file exists
ls dbwarden.py
```

**Solution:** Create `dbwarden.py`:

```bash
$ dbwarden init
```

**Cause 2: Wrong directory**

dbwarden looks in current directory and parents.

**Solution:** Navigate to project root:

```bash
cd /path/to/project
$ dbwarden migrate
```

**Cause 3: No database configuration**

**Solution:** Add configuration:

```python
# dbwarden.py
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "sqlite"
    database_url_sync = "sqlite:///./app.db"
```

**Cause 4: Import error in config file**

```python
# dbwarden.py
from app.models import Base  #  Import fails
```

**Solution:** Fix imports or use lazy loading:

```python
# Don't import models in config file
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],  #  Use model_paths instead
)
```

## "Exactly one default=True required"

### Symptom

```
ConfigurationError: Exactly one default=True required
```

### Causes & Solutions

**Cause 1: No default database**

```python
#  Wrong
analytics = database_config(
    database_name="analytics",
    default=False,
)
```

**Solution:** Set one database as default:

```python
#  Correct
analytics = database_config(
    database_name="analytics",
    default=True,
)
```

**Cause 2: Multiple defaults**

```python
#  Wrong
analytics = database_config(
    database_name="analytics",
    default=True,
)
```

**Solution:** Only one default:

```python
#  Correct
analytics = database_config(
    database_name="analytics",  # default=False implied
)
```

## "Duplicate database_name"

### Symptom

```
ConfigurationError: Duplicate database_name 'primary'
```

### Cause

Same `database_name` used twice:

```python
primary = database_config(
    database_name="primary",  #  Duplicate
)
```

### Solution

Use unique names:

```python
analytics = database_config(
    database_name="analytics",  #  Different name
)
```

## "No SQLAlchemy models found"

### Symptom

```
Warning: No SQLAlchemy models found
```

### Causes & Solutions

**Cause 1: Wrong `model_paths`**

```python
#  Wrong
model_paths=["models"]  # Not on PYTHONPATH
```

**Solution:** Use correct Python path:

```python
#  Correct
model_paths=["app.models"]
```

**Cause 2: Models not imported**

```python
# app/models/__init__.py
#  Wrong - models not imported
from sqlalchemy.orm import DeclarativeBase

class Base(DeclarativeBase):
    pass
```

**Solution:** Import models:

```python
# app/models/__init__.py
#  Correct
from sqlalchemy.orm import DeclarativeBase
from app.models.user import User
from app.models.order import Order

class Base(DeclarativeBase):
    pass
```

**Cause 3: Missing `model_paths`**

**Solution:** Add `model_paths`:

```python
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],  #  Add this
)
```

**Cause 4: Circular imports**

```python
# app/models/user.py
from app.models.order import Order  #  Circular

# app/models/order.py
from app.models.user import User  #  Circular
```

**Solution:** Use TYPE_CHECKING:

```python
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from app.models.order import Order
```

## "model_paths is required"

### Symptom

```
ConfigurationError: model_paths is required when more than one database is configured
```

### Cause

Multiple databases without `model_paths`:

```python
#  Wrong
analytics = database_config(
    database_name="analytics",  # No model_paths
)
```

### Solution

Add `model_paths` to all databases:

```python
#  Correct
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],
)
```

## "model_paths overlap detected"

### Symptom

```
ConfigurationError: model_paths overlap detected: path 'app.models' from 'analytics' is also defined in 'primary'; set overlap_models=True to allow
```

### Cause

Same model paths for different databases:

```python
#  Wrong
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models"],  #  Same path
)
```

### Solutions

**Solution 1: Use separate paths**

```python
#  Correct
primary = database_config(
    database_name="primary",
    model_paths=["app.models.primary"],
)
analytics = database_config(
    database_name="analytics",
    model_paths=["app.models.analytics"],
)
```

**Solution 2: Allow overlap (if intentional)**

```python
#  Correct for read replicas
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    overlap_models=True,
)
replica = database_config(
    database_name="replica",
    model_paths=["app.models"],
    overlap_models=True,
)
```

## "model_tables overlap detected"

### Symptom

```
ConfigurationError: model_tables overlap detected: table 'users' in 'primary' is also in 'analytics'
```

### Cause

Two databases have `model_tables` lists that share table names:

```python
#  Wrong
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
    model_tables=["users", "posts"],
)
analytics = database_config(
    database_name="analytics",
    model_paths=["other_models"],
    model_tables=["users"],  # 'users' already owned by primary
)
```

### Solutions

**Solution 1: Remove duplicate table name**

```python
#  Correct
analytics = database_config(
    database_name="analytics",
    model_paths=["other_models"],
    model_tables=["analytics_events"],  # No overlap with primary
)
```

**Solution 2: Allow overlap (if intentional)**

```python
#  Correct for shared tables
analytics = database_config(
    database_name="analytics",
    model_paths=["other_models"],
    model_tables=["users", "analytics_events"],
    overlap_models=True,  # Allow overlap
)
```

## "dev_database_url is required"

### Symptom

```
ConfigurationError: dev_database_url is required when dev_database_type is set
```

### Cause

Set `dev_database_type` without `dev_database_url`:

```python
#  Wrong
primary = database_config(
    database_name="primary",
    dev_database_type="sqlite",
    # Missing dev_database_url
)
```

### Solution

Add both dev parameters:

```python
#  Correct
primary = database_config(
    database_name="primary",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev.db",  #  Add this
)
```

## Connection Errors

### "could not connect to server"

**Cause:** Database server not running or unreachable.

**Solutions:**

1. **Check database is running:**

```bash
# PostgreSQL
sudo systemctl status postgresql

# Docker
docker ps | grep postgres
```

2. **Check connection URL:**

```python
# Verify host, port, credentials
database_url_sync="postgresql://user:pass@localhost:5432/myapp"
```

3. **Test connection:**

```bash
# PostgreSQL
psql -h localhost -U user -d myapp

# MySQL
mysql -h localhost -u user -p myapp
```

### "authentication failed"

**Cause:** Wrong credentials.

**Solutions:**

1. **Check credentials:**

```bash
# PostgreSQL
psql -h localhost -U user -d myapp
```

2. **Verify environment variable:**

```bash
echo $DATABASE_URL
```

3. **URL encode special characters:**

```python
from urllib.parse import quote_plus

password = "p@ss:word"
encoded = quote_plus(password)  # "p%40ss%3Aword"
```

### "database does not exist"

**Cause:** Database not created.

**Solution:** Create database:

```sql
-- PostgreSQL
CREATE DATABASE myapp;

-- MySQL
CREATE DATABASE myapp CHARACTER SET utf8mb4;
```

## Import Errors

### "ModuleNotFoundError"

**Cause:** Python can't find module.

**Solutions:**

1. **Check PYTHONPATH:**

```bash
export PYTHONPATH=/path/to/project:$PYTHONPATH
```

2. **Install package:**

```bash
uv add -e .  # Editable install
```

3. **Verify import:**

```bash
python -c "import app.models"
```

## Performance Issues

### Slow configuration loading

**Cause:** Large codebase scan.

**Solution:** Specify `model_paths`:

```python
#  Slow - scans everything
primary = database_config(
    database_name="primary",
)

#  Fast - targeted scan
primary = database_config(
    database_name="primary",
    model_paths=["app.models"],
)
```

### Slow imports

**Cause:** Heavy imports in config file.

**Solution:** Avoid imports in `dbwarden.py`:

```python
#  Slow
from app.models import Base
from app.services import setup

#  Fast
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    ...
```

## Debugging Tips

### Enable verbose output

```bash
$ dbwarden migrate --verbose
```

### Check configuration

```bash
# Show all configuration
$ dbwarden settings show

# Show specific database
$ dbwarden settings show primary
```

### Test imports

```bash
python -c "import dbwarden; print('OK')"
python -c "from dbwarden import database_config; print('OK')"
```

### Verify database connection

```bash
$ dbwarden check-db
$ dbwarden check-db --database primary
```

## What's Next?

- **[Configuration API Reference](../reference/configuration-api.md)** - Complete parameter docs
- **[Quick Start](quick-start.md)** - Start fresh with correct setup
- **[CLI option inventory](../reference/cli-options.md)** - Every built-in flag, including `--disable-skip`

## Exit code 3

Exit code `3` can report a skipped database or a safety ceiling stop; inspect the
accompanying result before treating it as success. See
[safety-scoped migrations](../correctness/safety-scoped-migrations.md) for the
severity ceiling semantics.

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/01-project-setup/
========================================================================

# 1. Project Setup

## What You'll Learn

- How to initialize a dbwarden project with `dbwarden init`
- How configuration is structured via `DbwardenDatabase` classes
- How to inspect your loaded configuration

## Prerequisites

- Python 3.12+ with `uv add dbwarden sqlalchemy`
- The `examples/core/` directory (see [Cookbook Index](index.md))

## Step 1: Initialize the Project

```bash
cd examples/core/
bash scripts/01-setup.sh
```

The `dbwarden init` command creates the directory structure dbwarden expects:

```
migrations/
  primary/          # Migration files for the 'primary' database
```

It also writes a starter `dbwarden.py` if one doesn't exist. In our case, we already have one with our configuration.

## Step 2: Understanding the Configuration

Our `examples/core/dbwarden.py`:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "sqlite"
    database_url_sync = "sqlite:///./app.db"
    model_paths = ["app"]
    model_tables = ["users", "posts"]
```

Each parameter has a specific role:

| Parameter | Value | Purpose |
|-----------|-------|---------|
| `database_name` | `"primary"` | Logical name used in `--database primary` CLI flags |
| `default` | `True` | Used when no `--database` flag is given |
| `database_type` | `"sqlite"` | Dialect for SQL generation and connection |
| `database_url_sync` | `"sqlite:///./app.db"` | Synchronous connection URL |
| `model_paths` | `["app"]` | Python module paths to scan for SQLAlchemy models |
| `model_tables` | `["users", "posts"]` | Optional table-name filter for this database |

`Primary.handle` is a `DatabaseHandle` object. It's also used later for FastAPI dependency injection: the same object provides `Primary.handle.async_session` and `Primary.handle.sync_session`. The equivalent `database_config(...)` function API remains supported.

## Step 3: Viewing the Configuration

```text
$ dbwarden config
Configuration:
  Databases:
    primary (default):
      Type: sqlite
      Sync URL: sqlite:///./app.db
      Model Paths: app
      Migrations Dir: migrations/primary
```

This confirms dbwarden has discovered and loaded your configuration. The `(default)` marker means `--database` can be omitted when targeting this database.

## What Happens Under the Hood

When you import `dbwarden` and define a concrete `DbwardenDatabase` subclass:

1. The concrete class is registered in dbwarden's internal registry
2. On first CLI command, dbwarden discovers `dbwarden.py` via AST scanning
3. It imports the module and registers each concrete database class
4. It validates uniqueness, default rules, and model path resolution
5. The resolved configuration is cached for the session

## Key Takeaways

- `dbwarden init` creates the directory skeleton: run it once per project
- `dbwarden config` shows what dbwarden actually resolved (useful for debugging)
- `DbwardenDatabase` is the default configuration API; `database_config()` remains a supported function alternative
- `model_paths` controls which Python modules are scanned for models
- We chose SQLite here so the example runs with zero external services

## Next

[Section 2: Models & Migrations](02-models-and-migrations.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/02-models-and-migrations/
========================================================================

# 2. Defining Models and Generating Migrations

## What You'll Learn

- How to define SQLAlchemy models with `class Meta` annotations
- How `make-migrations` generates SQL from model changes
- How the generated SQL maps to database DDL
- How to create manual migrations with `dbwarden new`
- How to extract rollback SQL from an existing migration

## Prerequisites

- Completed [Section 1: Project Setup](01-project-setup.md)
- `examples/core/` with `app/models.py`

## Step 1: The Models

Our example project defines four models in `examples/core/app/models.py`. Here they are with explanations:

### User

```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    username: Mapped[str] = mapped_column(String(100), unique=True, nullable=False)
    full_name: Mapped[str | None] = mapped_column(String(200), nullable=True)
    is_active: Mapped[bool] = mapped_column(Boolean, default=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("CURRENT_TIMESTAMP"))

    class Meta(TableMeta):
        comment = "Core user accounts"
        indexes = [
            IndexSpec(name="ix_users_created_at", columns=["created_at"]),
        ]
```

Key points:

- `unique=True` on `email` and `username` generates `UNIQUE` constraints
- `nullable=True` (the default) allows `NULL`; `nullable=False` adds `NOT NULL`
- `server_default=text(...)` becomes a database-level `DEFAULT` clause in the DDL; `default=` is a Python-level default and is not rendered in SQL
- `class Meta(TableMeta)` is how we attach table-level metadata
- `IndexSpec` generates a named `CREATE INDEX` statement

### Post

```python
class Post(Base):
    __tablename__ = "posts"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String(255), nullable=False)
    body: Mapped[str] = mapped_column(Text, nullable=False)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("CURRENT_TIMESTAMP"))

    class Meta(TableMeta):
        comment = "User blog posts"
        indexes = [
            IndexSpec(name="ix_posts_user_id", columns=["user_id"]),
            IndexSpec(name="ix_posts_created_at", columns=["created_at"]),
        ]
```

Key points:

- `ForeignKey("users.id")` generates a `REFERENCES` clause
- Foreign key targets are rendered inline in `CREATE TABLE`

### Product (with CHECK constraint)

```python
class Product(Base):
    __tablename__ = "products"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    name: Mapped[str] = mapped_column(String(200), nullable=False)
    price: Mapped[float] = mapped_column(Float, nullable=False)
    description: Mapped[str | None] = mapped_column(Text, nullable=True)
    in_stock: Mapped[bool] = mapped_column(Boolean, default=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("CURRENT_TIMESTAMP"))

    class Meta(TableMeta):
        comment = "Product catalog"
        checks = [
            {"name": "ck_products_price_positive", "sql": "price > 0"},
        ]
```

Key points:

- `checks` in `class Meta` generates `CHECK` constraints
- Each check needs a `name` (constraint name) and `sql` (the expression)
- This prevents negative prices at the database level

### Tag

```python
class Tag(Base):
    __tablename__ = "tags"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    name: Mapped[str] = mapped_column(String(50), unique=True, nullable=False)

    class Meta(TableMeta):
        comment = "Taxonomy tags for products"
```

The simplest model: just an ID and a unique name.

## Step 2: Generating the Migration

```bash
cd examples/core
bash scripts/02-models-migrations.sh
```

The first script step runs:

```bash
$ dbwarden make-migrations "create core tables" --database primary
```

This compares the current model state against the database (or a stored snapshot). Since this is a fresh project, it detects four new tables and generates:

```sql
-- upgrade

CREATE TABLE IF NOT EXISTS posts (
    id INTEGER NOT NULL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    user_id INTEGER NOT NULL REFERENCES users(id),
    created_at DATETIME
)

CREATE TABLE IF NOT EXISTS products (
    id INTEGER NOT NULL PRIMARY KEY,
    name VARCHAR(200) NOT NULL,
    price FLOAT NOT NULL,
    description TEXT,
    in_stock BOOLEAN DEFAULT TRUE,
    created_at DATETIME
)

CREATE TABLE IF NOT EXISTS tags (
    id INTEGER NOT NULL PRIMARY KEY,
    name VARCHAR(50) NOT NULL UNIQUE
)

CREATE TABLE IF NOT EXISTS users (
    id INTEGER NOT NULL PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    username VARCHAR(100) NOT NULL UNIQUE,
    full_name VARCHAR(200),
    is_active BOOLEAN DEFAULT TRUE,
    created_at DATETIME
)

-- rollback

DROP TABLE users
DROP TABLE tags
DROP TABLE products
DROP TABLE posts
```

> **Note:** The example uses SQLite, which has limited DDL support. With PostgreSQL, dbwarden generates additional features:
> - **`CREATE INDEX IF NOT EXISTS ...`**: from `IndexSpec` entries in `class Meta`
> - **`COMMENT ON TABLE ...`**: from `Meta.comment` attributes
> - **`CONSTRAINT ... CHECK (...)`**: from `Meta.checks`
> - **`server_default`** expressions rendered as native SQL defaults
> - Inline `REFERENCES` become table-level `FOREIGN KEY` constraints
>
> The generated SQL is always backend-specific. dbwarden adapts to the `database_type` configured in `dbwarden.py`.

### Reading the Generated SQL

Let's walk through what each section does:

**`-- upgrade`**: Applied when you run `dbwarden migrate`

1. **`CREATE TABLE IF NOT EXISTS posts (...)`**: Creates posts with a foreign key reference to `users(id)` (inline `REFERENCES` style for SQLite). The foreign key originates from `ForeignKey("users.id")` on the `user_id` column.

2. **`CREATE TABLE IF NOT EXISTS products (...)`**: Creates products with a `CHECK` constraint defined in `class Meta`. In SQLite, CHECK constraints must be inline; with PostgreSQL they become `CONSTRAINT ... CHECK (...)`.

3. **`CREATE TABLE IF NOT EXISTS tags (...)`**: Simple table with a unique constraint on `name`.

4. **`CREATE TABLE IF NOT EXISTS users (...)`**: Creates the users table with all columns, primary key, and unique constraints inline.

Note that with this SQLite backend the table order differs from the order in our Python models, and some features are omitted:
- **IndexSpec entries** generate `CREATE INDEX` only on PostgreSQL and ClickHouse
- **`COMMENT ON TABLE`** is only generated for PostgreSQL
- **`server_default`** expressions render as native SQL defaults on PostgreSQL

**`-- rollback`**: Applied when you run `dbwarden rollback`

1. Drops tables. Order may vary by backend; dbwarden handles dependency ordering automatically.

### Auto-generated Migration Name

The migration file is named automatically:

```
primary__0001_create_core_tables.sql
```

The naming pattern is:

```
{database_name}__{4-digit-version}_{auto-generated-description}.sql
```

### PostgreSQL-Specific Model Metadata

When your `database_type` is `"postgresql"`, dbwarden supports PostgreSQL-specific table and column metadata. The following model shows tablespace, fillfactor, identity columns, and column compression:

```python
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class Order(Base):
    __tablename__ = "orders"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    total: Mapped[float] = mapped_column(Float)
    created_at: Mapped[datetime] = mapped_column(TIMESTAMP)

    class Meta(PGTableMeta):
        pg_tablespace = "fast_space"
        pg_fillfactor = 90
        comment = "Customer orders"

        class id(PGColumnMeta):
            comment = "Order ID"
            pg = pg.field(identity="ALWAYS")

        class created_at(PGColumnMeta):
            pg = pg.field(compression="pglz")
```

The generated PostgreSQL DDL includes tablespace, fillfactor, identity columns, and column-level options:

```sql
CREATE TABLE IF NOT EXISTS orders (
    id INTEGER GENERATED ALWAYS AS IDENTITY NOT NULL,
    total FLOAT NOT NULL,
    created_at TIMESTAMP NOT NULL COMPRESSION pglz
) TABLESPACE fast_space WITH (fillfactor=90);

COMMENT ON TABLE orders IS 'Customer orders';
COMMENT ON COLUMN orders.id IS 'Order ID';
```

For PostgreSQL views, use `PGViewMeta` instead of `PGTableMeta`:

```python
from dbwarden.databases.pgsql import PGViewMeta

class ActiveUser(Base):
    __tablename__ = "active_users"

    id: Mapped[int] = mapped_column(Integer)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(PGViewMeta):
        pg_view_query = "SELECT id, email FROM users WHERE active = true"
        pg_view_materialized = False
```

The generated DDL creates a regular view:

```sql
CREATE OR REPLACE VIEW active_users AS SELECT id, email FROM users WHERE active = true;
```

For a materialized view with auto-refresh:

```python
class OrderSummary(Base):
    __tablename__ = "order_summary"

    user_id: Mapped[int] = mapped_column(Integer)
    total: Mapped[float] = mapped_column(Integer)

    class Meta(PGViewMeta):
        pg_view_query = "SELECT user_id, count(*) AS total FROM orders GROUP BY user_id"
        pg_view_materialized = True
        pg_view_auto_refresh = True
```

The first migration generates `CREATE MATERIALIZED VIEW order_summary AS ...`. Subsequent migrations include `REFRESH MATERIALIZED VIEW order_summary;`.

To scope a table or view to a PostgreSQL schema, set `pg_schema`:

```python
class Meta(PGTableMeta):
    pg_schema = "app"
```

The generated DDL uses the fully qualified name (e.g. `app.users`). Set `pg_schema` at the config level in `database_config(...)` to set the connection `search_path` for all unqualified references.

## Step 3: Creating a Manual Migration

Sometimes you need a migration that isn't model-driven: a data backfill, a stored procedure, or a complex SQL operation.

```bash
$ dbwarden new add_custom_table --database primary
```

This creates a blank migration:

```sql
-- upgrade

-- TODO: write your upgrade SQL here

-- rollback

-- TODO: write your rollback SQL here
```

You fill in both sections manually. Manual migrations follow the same file naming convention and are tracked alongside auto-generated ones.

## Step 4: Extracting Rollback SQL

If you have a migration file and need to extract just its rollback section:

```bash
$ dbwarden make-rollback migrations/primary/primary__0001_create_core_tables.sql
```

This prints the rollback SQL to stdout. Useful for quickly verifying what a rollback will do before running it.

## Key Takeaways

- dbwarden generates explicit, reviewable SQL: no hidden runtime behavior
- Every migration has both `-- upgrade` and `-- rollback` sections
- `class Meta(TableMeta)` is where table-level metadata (comments, indexes, checks) lives
- `IndexSpec` produces named `CREATE INDEX` statements; always prefer named indexes
- `dbwarden new` creates blank migrations for non-model-driven changes
- `dbwarden make-rollback` extracts rollback SQL for review

## Related Documentation

- [SQLAlchemy Models Reference](../models.md)
- [Modeling Guide](../getting-started/modeling.md)
- [Migration File Format](../migration-files.md)
- [`make-migrations` command](../commands/make-migrations.md)

## Next

[Section 3: Apply & Inspect](03-apply-and-inspect.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/03-apply-and-inspect/
========================================================================

# 3. Applying and Inspecting Migrations

## What You'll Learn

- How `dbwarden migrate` applies pending SQL
- How to roll back and downgrade to specific versions
- How to inspect migration history and status
- How to validate schema integrity and database connectivity

## Prerequisites

- Completed [Section 2](02-models-and-migrations.md) (migration file exists)
- `examples/core/` project

## Step 1: Apply Migrations

```bash
cd examples/core
bash scripts/03-apply-inspect.sh
```

### The Migrate Command

```bash
$ dbwarden migrate --database primary
```

When you run `migrate`, dbwarden:

1. Creates the metadata table (`_dbwarden_migrations`) if it doesn't exist
2. Creates the lock table (`dbwarden_lock`) if it doesn't exist
3. Acquires a native lock (advisory lock for PostgreSQL, named lock for MySQL, BEGIN IMMEDIATE for SQLite)
4. Writes a status row for observability and starts a heartbeat
5. Reads all migration files and filters to pending (unapplied) ones
6. Executes the `-- upgrade` SQL of each pending migration
7. Records each migration's version, checksum, and timestamp
8. Writes a schema snapshot file for future diffs
9. Stops the heartbeat and releases the lock

```
[dbwarden] Applying primary__0001_create_core_tables...
[dbwarden] Migration applied successfully (42ms)
[dbwarden] All migrations applied. Pending: 0
```

### Verify Status

```bash
$ dbwarden status --database primary
```

Output:

```
Database: primary
  Applied:  1
  Pending:  0
  Status:   up-to-date
```

### View History

```bash
$ dbwarden history --database primary
```

Output:

```
 Migration History (primary)
  V0001  create_core_tables  2025-01-15 10:30:00  a1b2c3d4...
```

The checksum (`a1b2c3d4...`) is a SHA-256 hash of the migration file content. This detects tampering or accidental edits after apply.

## Step 2: Rollback

```bash
$ dbwarden rollback --database primary --count 1
```

Rollback executes the `-- rollback` section of the most recently applied migration. After rollback:

```
[dbwarden] Rolling back primary__0001_create_core_tables...
[dbwarden] Rollback complete
```

```bash
$ dbwarden status --database primary
```

```
Database: primary
  Applied:  0
  Pending:  1
  Status:   pending
```

### Rollback Mechanics

- Rollback always executes the `-- rollback` section of the file; never auto-generates reverse SQL
- `--count` controls how many migrations to roll back (default: 1)
- Rollbacks are also lock-protected
- After rollback, the migration is considered "pending" again and can be re-applied

## Step 3: Re-apply

```bash
$ dbwarden migrate --database primary
```

Re-applies the migration. Since rollback removed the tracking record, the migration runs again.

## Step 4: Downgrade to a Version

```bash
$ dbwarden downgrade --to 0000 --database primary
```

`downgrade` is a bulk rollback: it rolls back all migrations down to (but not including) the target version. `--to 0000` rolls back everything.

```
[dbwarden] Rolling back primary__0001_create_core_tables...
[dbwarden] Downgrade complete. At version: 0000
```

### migrate vs rollback vs downgrade

| Command | What it does | Safe to run twice? |
|---------|-------------|-------------------|
| `migrate` | Applies pending migrations | Yes (idempotent) |
| `rollback` | Reverses the last N applied migrations | Yes (tracks what's applied) |
| `downgrade` | Rolls back to a specific target version | Yes |

## Step 5: Re-apply All

```bash
$ dbwarden migrate --database primary
$ dbwarden status --database primary
```

After the final apply, status should show:

```
Database: primary
  Applied:  1
  Pending:  0
  Status:   up-to-date
```

## Step 6: Schema Validation

```bash
$ dbwarden check --database primary
```

`check` scans each migration file and classifies operations by safety level:

- **SAFE**: Adding a nullable column, creating an index
- **INFO**: Table comment changes
- **WARN**: Dropping a default, changing column type
- **CRITICAL**: Dropping a table or column, removing a NOT NULL

```
Checking migrations for 'primary'...
  primary__0001_create_core_tables:
    CREATE TABLE users           SAFE
    CREATE TABLE posts           SAFE
    CREATE TABLE products        SAFE
    CREATE TABLE tags            SAFE
    CREATE INDEX                 SAFE
    COMMENT ON TABLE             INFO
  Result: 5 SAFE, 1 INFO, 0 WARN, 0 CRITICAL
```

## Step 7: Database Connectivity Check

```bash
$ dbwarden check-db --database primary
```

`check-db` connects to the live database and reports its schema:

```
Database: primary
  Connection: OK
  Tables:
    users (6 columns)
    posts (5 columns)
    products (6 columns)
    tags (2 columns)
  Migration table: _dbwarden_migrations (present)
  Lock table: dbwarden_lock (present)
```

This confirms the database is reachable and has the expected schema.

## Key Takeaways

- `migrate` applies pending SQL with lock protection and checksum recording
- `rollback` and `downgrade` give you precise control over reversal
- `status` and `history` are your windows into migration state
- `check` classifies each operation by safety before it runs
- `check-db` validates database connectivity and schema existence

## Related Documentation

- [`migrate` command](../commands/migrate.md)
- [`rollback` command](../commands/rollback.md)
- [`downgrade` command](../commands/downgrade.md)
- [`status` command](../commands/status.md)
- [`history` command](../commands/history.md)
- [`check` command](../commands/check.md)
- [`check-db` command](../commands/check-db.md)

## Next

[Section 4: Offline & CI Workflows](04-offline-ci.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/04-offline-ci/
========================================================================

# 4. Offline & CI Workflows

## What You'll Learn

- How to export model state to JSON for offline use
- How to generate migrations without a live database
- How to integrate this into CI/CD pipelines

## Prerequisites

- Completed [Section 3](03-apply-and-inspect.md) (migrations applied, models in sync)
- `examples/core/` project

## The Problem

In CI/CD pipelines, you often need to generate migrations as part of your build, but your CI runner may not have a database connection. dbwarden's offline mode solves this by serializing model state to a JSON file.

## Step 1: Export Model State

```bash
cd examples/core
bash scripts/04-offline-ci.sh
```

The key command:

```bash
$ dbwarden export-models --database primary
```

This connects to the live database, introspects the current schema, and writes a JSON file to `.dbwarden/model_state.json`:

```json
{
  "version": "1.0",
  "exported_at": "2025-01-15T10:30:00",
  "database": "primary",
  "tables": {
    "users": {
      "columns": {
        "id": {"type": "INTEGER", "nullable": false, "primary_key": true},
        "email": {"type": "VARCHAR(255)", "nullable": false, "unique": true},
        "username": {"type": "VARCHAR(100)", "nullable": false, "unique": true},
        "full_name": {"type": "VARCHAR(200)", "nullable": true},
        "is_active": {"type": "BOOLEAN", "nullable": true, "default": "1"},
        "created_at": {"type": "DATETIME", "nullable": true, "default": "CURRENT_TIMESTAMP"}
      },
      "indexes": [
        {"name": "ix_users_created_at", "columns": ["created_at"]}
      ],
      "checks": [],
      "comment": "Core user accounts"
    }
  }
}
```

This file becomes your source of truth for future diffs; no database required.

## Step 2: Commit the State File

```bash
git add .dbwarden/model_state.json
git commit -m "Update model state snapshot"
```

## Step 3: Generate Migrations Offline

On any machine (including CI without a database):

```bash
$ dbwarden make-migrations "offline schema change" --offline --database primary
```

The `--offline` flag tells dbwarden to:

1. Read the model state from `.dbwarden/model_state.json` instead of querying a live database
2. Introspect the current model definitions in your Python code
3. Diff the two and generate migration SQL
4. Write the migration file AND update the snapshot file

This means the snapshot is always in sync after each generation.

## CI/CD Integration

In a GitHub Actions workflow:

```yaml
jobs:
  generate-migrations:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: uv add dbwarden sqlalchemy

      # Generate migrations using the committed state file
      - name: Check for new migrations
        run: dbwarden make-migrations "ci change" --offline --database primary

      # Commit any newly generated migrations
      - uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: "Auto-generate migration"
```

The full CI pipeline can then `dbwarden migrate` against staging/production using the generated SQL files.

## Key Takeaways

- `export-models` serializes the current database schema to JSON
- `make-migrations --offline` generates migrations using the snapshot instead of a live database
- Offline mode enables migration generation in CI without database access
- The snapshot file should be committed and kept in sync

## Related Documentation

- [CI/CD Patterns](../advanced/ci-cd-patterns.md)
- [`export-models` command](../cli-reference.md) (see CLI reference)

## Next

[Section 5: Schema Inspection](05-schema-inspection.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/05-schema-inspection/
========================================================================

# 5. Schema Inspection

Schema inspection allows you to compare your SQLAlchemy model definitions against the live database, capture DDL snapshots of individual tables, and reverse-engineer models from an existing database.

For complete documentation see the [`diff`](../commands/diff.md), [`snapshot`](../commands/snapshot.md), and [`generate-models`](../commands/generate-models.md) command references.

## What You'll Learn

- How to diff models against the live database
- How to capture DDL snapshots of individual tables
- How to reverse-engineer models from a live database

## Prerequisites

- Completed [Section 3](03-apply-and-inspect.md) (migrations applied)
- `examples/core/` project

## Step 1: Diff Models vs Database

```bash
cd examples/core
bash scripts/05-schema-inspection.sh
```

The key command:

```bash
$ dbwarden diff --database primary
```

`diff` compares your SQLAlchemy model definitions against the current database schema and reports any discrepancies:

```
No differences found between models and database.
```

If you add a column to a model without running `make-migrations`, `diff` would report a schema diff table:

```
Schema Diff
┌───────────┬───────┬────────┬──────────┐
│ Operation │ Table │ Target │ Severity │
├───────────┼───────┼────────┼──────────┤
│ add_column│ users │  bio   │ INFO     │
└───────────┴───────┴────────┴──────────┘
Total changes: 1
```

This is useful for catching drift before deployments.

## Step 2: Capture a DDL Snapshot

```bash
$ dbwarden snapshot users --database primary
```

The `snapshot` command captures the DDL for a specific table:

```sql
CREATE TABLE users (
    id INTEGER NOT NULL,
    email VARCHAR(255) NOT NULL,
    username VARCHAR(100) NOT NULL,
    full_name VARCHAR(200),
    is_active BOOLEAN DEFAULT true,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (id),
    UNIQUE (email),
    UNIQUE (username)
);

-- Indexes:
CREATE INDEX ix_users_created_at ON users (created_at);
```

Useful for:
- Documenting schema for code reviews
- Comparing schemas across environments
- Debugging migration issues

## Step 3: Reverse-Engineer Models

```bash
$ dbwarden generate-models -d primary --tables users,posts
```

This connects to the live database and generates SQLAlchemy model code:

```python
from sqlalchemy import Integer, String, Boolean, DateTime, Text, ForeignKey
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), nullable=False, unique=True)
    username: Mapped[str] = mapped_column(String(100), nullable=False, unique=True)
    full_name: Mapped[str | None] = mapped_column(String(200), nullable=True)
    is_active: Mapped[bool] = mapped_column(Boolean, default=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.now(UTC))


class Post(Base):
    __tablename__ = "posts"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String(255), nullable=False)
    body: Mapped[str] = mapped_column(Text, nullable=False)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False)
```

This is the fastest way to bootstrap models from an existing database. You can review and annotate the output with `class Meta` afterward.

Options:
- `--tables users,posts`: limit to specific tables
- `--exclude-tables`: exclude tables by pattern
- `--single-file`: output all models in one file
- `--output ./models/`: write to a directory instead of stdout

## Key Takeaways

- `diff` detects drift between models and the live database
- `snapshot` captures table DDL for documentation or debugging
- `generate-models` reverse-engineers live tables into SQLAlchemy model code
- These three commands form your schema inspection toolkit

## Related Documentation

- [`diff` command](../commands/diff.md)
- [`snapshot` command](../commands/snapshot.md)
- [`generate-models` command](../commands/generate-models.md)
- [SQLAlchemy Models Reference](../models.md)

## Next

[Section 6: Safety & Impact Analysis](06-safety-impact.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/06-safety-impact/
========================================================================

# 6. Safety & Impact Analysis

Schema changes are the highest-risk operation in most deployments. Dropping a column that application code still references causes runtime errors. Changing a column type can break queries. dbwarden provides two tools to detect these issues before deploy: `check` classifies every migration operation by danger level, and `check-impact` finds affected code references.

For complete documentation see the [`check`](../commands/check.md) and [`check-impact`](../cli-reference.md) command references.

## What You'll Learn

- How `dbwarden check` classifies operations by danger level
- How `dbwarden check-impact` finds code references affected by a migration
- How to detect destructive changes before they reach production

## Prerequisites

- Completed [Section 3](03-apply-and-inspect.md) (migrations applied)
- `examples/core/` project

## Step 1: Safety Check

```bash
cd examples/core
bash scripts/06-safety-impact.sh
```

The key command:

```bash
$ dbwarden check --database primary
```

This scans every migration file and classifies each SQL operation by safety level:

| Level | Meaning |
|-------|---------|
| **SAFE** | No data loss risk (add table, add nullable column, create index) |
| **INFO** | Metadata changes (comments, renames) |
| **WARN** | Potential impact (change column type, drop default) |
| **CRITICAL** | Destructive (drop table, drop column, remove NOT NULL) |

Output for our baseline migrations:

```
Safety Check - primary
┌──────────┬──────────────┬──────────┬────────┬─────────┬───────────────┐
│ Severity │ Change       │ Table    │ Column │ Message │ Required Flag │
├──────────┼──────────────┼──────────┼────────┼─────────┼───────────────┤
│ SAFE     │ create_table │ users    │        │         │               │
│ SAFE     │ create_table │ posts    │        │         │               │
│ SAFE     │ create_table │ products │        │         │               │
│ SAFE     │ create_table │ tags     │        │         │               │
│ SAFE     │ create_index │          │        │         │               │
│ INFO     │ comment_on   │ users    │        │         │               │
└──────────┴──────────────┴──────────┴────────┴─────────┴───────────────┘
```

A migration with a destructive change would show:

```
┌──────────┬──────────────┬───────┬──────────┬─────────┬───────────────┐
│ Severity │ Change       │ Table │ Column   │ Message │ Required Flag │
├──────────┼──────────────┼───────┼──────────┼─────────┼───────────────┤
│ CRITICAL │ drop_column  │ users │ username │         │               │
└──────────┴──────────────┴───────┴──────────┴─────────┴───────────────┘
```

This gives you a quick visual signal during code review: if a migration contains CRITICAL operations, it needs extra scrutiny.

## Step 2: Code Impact Analysis

```bash
$ dbwarden check-impact 0001 --database primary
```

`check-impact` scans your application code (not just migration files) for references that would be affected by a migration. It uses AST analysis with a grep fallback:

```
No impact detected
Scanned: .
```

A more realistic scenario with a destructive change:

```
Migration: 0002_drop_username
Impact detected: 1 operation(s) affect code

drop_column on users.username
  References: 2
    app/routes/users.py:34  attribute_access
      .username
    app/templates/profile.jinja2:12  grep
      user.username
```

The scan finds each reference, identifies the access pattern (attribute access in Python, grep match in templates), and reports the file and line number.

### How It Works

1. Reads the migration's plan file and parses the schema changes
2. Identifies schema changes (DROP COLUMN, ALTER COLUMN TYPE, etc.)
3. Scans `.py` files using Python's `ast` module for attribute access patterns
4. Falls back to grep for non-Python files (templates, configs, etc.)
5. Reports all references grouped by change type

### Flags

- `--scan-path app/`: limit scanning to a specific directory (default: project root)
- `--deep`: also scan dependencies (imported packages)
- `--out json`: output as JSON for CI processing
- `--verbose`: show scan progress

## Pre-Deploy Workflow

Combine both tools for a safe deploy sequence:

```bash
# 1. Check migration safety
$ dbwarden check --database primary

# 2. Check code impact
$ dbwarden check-impact 0042 --database primary

# 3. Only proceed if no unexpected CRITICAL or WARN items
$ dbwarden migrate --database primary
```

In CI:

```yaml
- name: Safety check
  run: dbwarden check --database primary
- name: Impact analysis
  run: dbwarden check-impact 0042 --database primary
- name: Apply (only if previous steps succeeded)
  run: dbwarden migrate --database primary
```

## Key Takeaways

- `check` classifies every migration operation by safety level using a severity table
- `check-impact` finds code references affected by a migration using AST + grep
- Together they catch breaking changes before deploy
- CRITICAL operations aren't blocked; they're flagged for human review
- Use `--out json` for CI integration

## Related Documentation

- [`check` command](../commands/check.md)
- [`check-impact` command](../cli-reference.md) (see CLI reference)
- [Safe Deployment](../advanced/safe-deployment.md)

## Next

[Section 7: Seeds](07-seeds.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/07-seeds/
========================================================================

# 7. Seeds

## What You'll Learn

- How to define code seeds using the `Seed` base class
- How to create and apply file-based SQL/Python seeds (legacy)
- How to list, apply, and roll back seeds
- How to auto-apply seeds after migrations

## Prerequisites

- Completed [Section 3](03-apply-and-inspect.md) (migrations applied, tables exist)
- `examples/core/` project
- Seed plugin installed: `dbwarden plugin add dbwarden-seeds`

## Step 1: Define a Code Seed

Code seeds are the recommended way to seed data. They live alongside your models and keep seed logic close to the schema it populates.

Create a seed file in your models directory (e.g. `models/seeds.py`):

```python
from dbwarden.seed import Seed

class CountrySeed(Seed):
    __seed_version__ = "C0001"
    __seed_database__ = "primary"
    __seed_description__ = "initial countries"
    __seed_on_conflict__ = "update"
    __seed_conflict_columns__ = ["code"]

    model = Country
    rows = [
        Country(code="UY", name="Uruguay"),
        Country(code="AR", name="Argentina"),
    ]
```

Notice:

- **Version**: declare a permanent `__seed_version__` such as `C0001`; never change or reuse a released version
- **Model instances** in `rows`: your IDE gives full autocompletion on column names
- **Keyword arguments required**: SQLAlchemy 2.0's `DeclarativeBase` does not accept positional args; always use `Model(col=val)` syntax
- **`__seed_database__`**: route the seed to the correct database

### Procedural Seeds

Define `forward(connection, session)` and `reverse(connection, session)` for programmatic data:

```python
class PermissionSeed(Seed):
    __seed_version__ = "C0002"
    __seed_database__ = "primary"
    __seed_description__ = "load permissions"

    model = Permission

    @staticmethod
    def forward(connection, session):
        for resource in ["users", "orders"]:
            for action in ["read", "write", "delete"]:
                session.add(Permission(name=f"{resource}:{action}"))
```

## Step 2: Apply Seeds

```bash
$ dbwarden seed apply --database primary
```

Output:

```
Applying code seed C0001: initial countries
```

Code seeds and file seeds are both discovered and applied together. Each seed version is tracked in `_dbwarden_seeds` and can only be applied once until rolled back.

## Step 3: List Applied Seeds

```bash
$ dbwarden seed list --database primary
```

Output:

```
Seeds for database 'primary':
  C0001  initial countries                 applied  2025-01-15 10:30:00  (code seed)
```

## Step 4: Auto-Apply Seeds After Migrations

Configure seeds to be applied automatically after `dbwarden migrate`:

```python
database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    model_paths=["models"],
    auto_apply_seeds=True,
)
```

Now running `dbwarden migrate` will also apply any pending seeds.

Or apply seeds once without changing config:

```bash
$ dbwarden migrate --apply-seeds
```

## Step 5: Traditional File Seeds (Legacy)

For complex multi-statement SQL, you can still use file-based seeds.

### Create a SQL Seed

```bash
$ dbwarden seed create "initial admin users" --database primary
```

This creates `seeds/V0001__initial_admin_users.sql`. Fill it with data:

```sql
INSERT INTO users (email, username, full_name, is_active, created_at)
VALUES ('admin@example.com', 'admin', 'Admin User', 1, CURRENT_TIMESTAMP);

INSERT INTO users (email, username, full_name, is_active, created_at)
VALUES ('moderator@example.com', 'moderator', 'Moderator User', 1, CURRENT_TIMESTAMP);
```

### Apply and List

```bash
$ dbwarden seed apply --database primary
$ dbwarden seed list --database primary
```

Output:

```
Seeds for database 'primary':
  C0001  initial countries                 applied  2025-01-15 10:30:00  (code seed)
  V0001  initial_admin_users              applied  2025-01-15 10:31:00
```

### Python File Seeds

```bash
$ dbwarden seed create "generate sample data" --database primary --type python
```

Creates `seeds/V0002__generate_sample_data.py` with a `seed(connection, session)` function.

## Step 6: Roll Back a Seed

```bash
$ dbwarden seed rollback --database primary --count 1
```

Seed rollback executes the configured inverse operation and removes the tracking record only after it succeeds. SQL file seeds require a `-- rollback` section; Python and procedural code seeds require `reverse(connection, session)`.

After rollback:

```
Seeds for database 'primary':
  C0001  initial countries                 applied  2025-01-15 10:30:00  (code seed)
   V0001  initial_admin_users              pending
```

## Step 7: Prune Orphaned Records

Remove tracking records for seed files that no longer exist on disk:

```bash
$ dbwarden seed list --prune
```

## Key Takeaways

- **Code seeds (`Seed` base class) are the recommended approach**: immutable explicit versions, full IDE support, stays in sync with models
- `auto_apply_seeds: True` or `dbwarden migrate --apply-seeds` applies seeds automatically after migrations
- File seeds (`.sql` / `.py`) are still available for complex multi-statement SQL
- `seed list --prune` cleans up orphaned tracking records
- Seed rollback executes the seed's explicit reverse operation

## Related Documentation

- [Seeds Reference](../seeds.md)
- [`seed` command](../commands/seed.md)
- [CLI Reference: Seed Management](../cli-reference.md#seed-management)

## Next

[Section 8: Multi-Database](08-multi-database.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/08-multi-database/
========================================================================

# 8. Multi-Database & Configuration

dbwarden supports managing multiple databases in a single project; each with its own migration directory, lock, tracking table, and model paths. You can mix PostgreSQL, MySQL, and ClickHouse backends in the same codebase.

For complete documentation see the [Multi-Database Configuration](../configuration/multi-database.md) reference.

## What You'll Learn

- How to configure multiple databases in one project
- How to target specific databases with CLI flags
- How to manage PostgreSQL + MySQL + ClickHouse in the same codebase
- How to use `dbwarden settings` for runtime configuration changes

## Prerequisites

- Docker (for PostgreSQL, MySQL, and ClickHouse containers)
- `examples/multi-database/` directory

## Scenario

A project with three databases:

- **primary** (PostgreSQL): transactional user data
- **legacy** (MySQL): legacy CRM and reporting data
- **analytics** (ClickHouse): page view events for analysis

## Step 1: The Configuration

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/primary"
    database_url_async = "postgresql+asyncpg://user:password@localhost:5432/primary"
    model_paths = ["app.models.primary"]

class Legacy(DbwardenDatabase):
    database_name = "legacy"
    database_type = "mysql"
    database_url_sync = "mysql+pymysql://user:password@localhost:3306/legacy"
    model_paths = ["app.models.legacy"]

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "http://localhost:8123/analytics"
    model_paths = ["app.models.analytics"]
```

The equivalent `database_config(...)` function API remains supported for
function-style integrations and plugin examples.

Key rules:
- Exactly one database must have `default=True` (used when `--database` is omitted)
- Each database must have separate `model_paths` (no overlap by default)
- Each database gets its own migration directory under `migrations/`
- MySQL models use `MyTableMeta` / `MyColumnMeta` for engine, charset, and column metadata

## Step 2: Start the Databases

```bash
cd examples/multi-database
docker compose up -d
```

## Step 3: Initialize and Migrate

```bash
$ dbwarden init
$ dbwarden migrate --all
```

This applies migrations to both databases in sequence. Each has its own lock, its own tracking table, and its own migration history.

## Step 4: Target a Specific Database

```bash
# Generate migrations for primary only
$ dbwarden make-migrations "add user table" --database primary

# Apply to analytics only
$ dbwarden migrate --database analytics

# Check status of one database
$ dbwarden status --database primary
```

## Step 5: Check Status of All Databases

```bash
$ dbwarden status --all
```

Output:

```
Database: primary
  Applied:  1
  Pending:  0
  Status:   up-to-date

Database: analytics
  Applied:  1
  Pending:  0
  Status:   up-to-date
```

## Step 6: Using `dbwarden settings`

The settings commands allow runtime configuration changes without editing `dbwarden.py` directly:

```bash
# View current configuration
$ dbwarden settings show --all

# To modify configuration, edit dbwarden.py directly.
# Settings commands are read-only; changes are made by editing the Python config file.
```

Settings commands modify the `dbwarden.py` file directly using AST-based mutation. The changes are permanent and committed to version control.

## Step 7: Dev Mode with Multiple Databases

Each database can independently configure dev mode:

```python
primary = database_config(
    database_name="primary",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/primary",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_primary.db",
    model_paths=["app.models.primary"],
)

legacy = database_config(
    database_name="legacy",
    database_type="mysql",
    database_url_sync="mysql+pymysql://localhost:3306/legacy",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_legacy.db",
    model_paths=["app.models.legacy"],
)

analytics = database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="http://localhost:8123/analytics",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./dev_analytics.db",
    model_paths=["app.models.analytics"],
)
```

```bash
# Dev mode for all databases
$ dbwarden --dev migrate --all

# Dev mode for a specific database
$ dbwarden --dev migrate --database analytics
```

## Step 8: Legacy Database with MySQL Metadata

The legacy MySQL database uses `MyTableMeta` and `MyColumnMeta` for MySQL-specific features. Here is a sample model from `app/models/legacy/customer.py`:

```python
from sqlalchemy import Integer, String, TIMESTAMP, Text
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta, MyColumnMeta, my

class Customer(Base):
    __tablename__ = "customers"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    name: Mapped[str] = mapped_column(String(200), nullable=False)
    notes: Mapped[str | None] = mapped_column(Text)
    created_at: Mapped[str] = mapped_column(TIMESTAMP)

    class Meta(MyTableMeta):
        my_engine = "InnoDB"
        my_charset = "utf8mb4"
        my_collate = "utf8mb4_unicode_ci"
        comment = "Legacy CRM customers"

        class id(MyColumnMeta):
            my = my.field(unsigned=True)

        class created_at(MyColumnMeta):
            my = my.field(on_update="CURRENT_TIMESTAMP")
```

Migrations for the legacy database work identically to other databases:

```bash
# Generate migration for MySQL legacy database
$ dbwarden make-migrations "add customer table" --database legacy

# Apply to legacy only
$ dbwarden migrate --database legacy
```

The generated DDL will target MySQL-native syntax:

```sql
CREATE TABLE IF NOT EXISTS customers (
    id INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
    name VARCHAR(200) NOT NULL,
    notes TEXT,
    created_at TIMESTAMP NOT NULL ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT='Legacy CRM customers';
```

## Key Takeaways

- Multiple declarative classes, or `database_config()` calls, create independent database targets
- Each database has its own migration directory, lock, and history
- `--database` targets a specific database; `--all` targets every database
- `default=True` controls which database is used when `--database` is omitted
- `settings` commands modify `dbwarden.py` at runtime without manual editing
- Dev mode can be configured independently per database

## Related Documentation

- [Multi-Database Configuration](../configuration/multi-database.md)
- [Dev Mode](../configuration/dev-mode.md)
- [Settings Command](../commands/settings.md)

## Next

[Section 9: FastAPI Integration](09-fastapi-integration.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/09-fastapi-integration/
========================================================================

# 9. FastAPI Integration

## What You'll Learn

- How to wire dbwarden into a FastAPI application lifecycle
- How to use `primary.async_session` as a dependency injection
- How to expose health check and migration endpoints
- How to validate schema on startup

## Prerequisites

- Docker (for PostgreSQL)
- `examples/fastapi-app/` directory

## Step 1: Configuration with Session Handles

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/myapp"
    database_url_async = "postgresql+asyncpg://user:password@localhost:5432/myapp"
    model_paths = ["app.models"]
```

`Primary.handle` is a `DatabaseHandle`. It exposes `Primary.handle.async_session` and `Primary.handle.sync_session` as FastAPI-compatible dependency annotations; no separate dependency module needed. The equivalent `database_config(...)` function API returns the same handle.

## Step 2: Lifespan Hook

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from dbwarden_fastapi import dbwarden_lifespan


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with dbwarden_lifespan(app, mode="check"):
        yield


app = FastAPI(lifespan=lifespan)
```

`dbwarden_lifespan` runs on every startup:

1. **Schema validation** (mode `"check"`): verifies all pending migrations exist and the database is in a known state
2. **Readiness gate**: the app won't accept traffic until validation passes
3. **Connection pool warmup**: pre-connects to the database
4. **On shutdown**: disposes all engine pools and ClickHouse clients

Available modes:
- `"check"`: validate schema, fail on pending migrations (recommended for production)
- `"migrate"`: apply pending migrations automatically on startup
- `"skip"`: no startup checks

## Step 3: Session Dependency in Routes

```python
from config import primary
from app.models import User
from app.schemas import UserResponse


@router.get("/{user_id}", response_model=UserResponse)
async def get_user(user_id: int, session: primary.async_session):
    result = await session.execute(
        select(User).where(User.id == user_id)
    )
    user = result.scalar_one_or_none()
    if not user:
        raise HTTPException(status_code=404, detail="User not found")
    return user
```

`primary.async_session` is a type alias for `Annotated[AsyncSession, Depends(...)]`. FastAPI resolves it to an actual database session using the engine configured by `Primary` (or the equivalent `database_config()` function).

The session is automatically:
- Opened when the route handler starts
- Committed (or rolled back on exception) when the handler finishes
- Closed and returned to the pool

## Step 4: Health Endpoints

```python
from dbwarden_fastapi import DBWardenHealthRouter

app.include_router(DBWardenHealthRouter(), prefix="/health")
```

This adds:

| Endpoint | Description |
|----------|-------------|
| `GET /health/` | Overall health status across all databases |
| `GET /health/liveness` | Is the app alive? (lightweight) |
| `GET /health/readiness` | Is the app ready for traffic? (checks DB connectivity) |
| `GET /health/{database_name}` | Per-database health status |

Sample response:

```json
{
  "status": "ok",
  "databases": {
    "primary": {
      "status": "ok",
      "connected": true,
      "pending_migrations": 0,
      "applied_migrations": 5,
      "lock_active": false
    }
  }
}
```

## Step 5: Migration Endpoints

```python
from dbwarden_fastapi import DBWardenRouter

app.include_router(DBWardenRouter(), prefix="/db")
```

| Endpoint | Description |
|----------|-------------|
| `GET /db/status` | JSON representation of `dbwarden status` |
| `POST /db/migrate` | Trigger migration execution at runtime |

These endpoints are useful for management UIs or automated deployment tooling.

## Step 6: The Complete App

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from dbwarden_fastapi import (
    DBWardenHealthRouter,
    DBWardenRouter,
    dbwarden_lifespan,
)
from app.routes import users


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with dbwarden_lifespan(app, mode="check"):
        yield


app = FastAPI(
    title="dbwarden FastAPI Example",
    lifespan=lifespan,
)

app.include_router(users.router, prefix="/api/v1")
app.include_router(DBWardenHealthRouter(), prefix="/health")
app.include_router(DBWardenRouter(), prefix="/db")
```

## Step 7: Run and Test

```bash
# Install dependencies
uv add dbwarden sqlalchemy fastapi uvicorn asyncpg
dbwarden plugin add dbwarden-fastapi

# Start PostgreSQL
docker run -d --name pg -e POSTGRES_USER=user \
    -e POSTGRES_PASSWORD=password -e POSTGRES_DB=myapp \
    -p 5432:5432 postgres:16

# Initialize and migrate
$ dbwarden init
$ dbwarden make-migrations "create users table"
$ dbwarden migrate

# Start the app
uvicorn app.main:app --reload
```

```bash
# Health check
curl http://localhost:8000/health/

# Create a user
curl -X POST http://localhost:8000/api/v1/users/ \
    -H "Content-Type: application/json" \
    -d '{"email": "alice@example.com", "username": "alice"}'

# Migration status
curl http://localhost:8000/db/status
```

## Key Takeaways

- `Primary.handle` is a `DatabaseHandle` with built-in FastAPI dependencies; `database_config()` returns the equivalent handle
- `dbwarden_lifespan` integrates schema validation into the app lifecycle
- `primary.async_session` works directly as a route parameter type annotation
- `DBWardenHealthRouter` exposes liveness, readiness, and per-database health
- `DBWardenRouter` exposes migration status and execution as HTTP endpoints

## Related Documentation

FastAPI support is not part of core. It ships as the `dbwarden-fastapi` plugin, which owns the session dependencies, health endpoints, migration routes, and metrics middleware shown above. Install it with `dbwarden plugin add dbwarden-fastapi`.

Full reference and tutorials live in the plugin's repository: [dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi).

## Next

[Section 10: Auto Schemas](10-auto-schemas.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/10-auto-schemas/
========================================================================

# 10. Auto-Generated Pydantic Schemas

## What You'll Learn

- How `@auto_schema` generates Pydantic schemas from model annotations
- How `public = False` controls field visibility
- How `CreateSchema`, `UpdateSchema`, and `PublicSchema` differ

## Prerequisites

- `dbwarden-fastapi` plugin installed: `dbwarden plugin add dbwarden-fastapi`
- `uv add dbwarden sqlalchemy`

## The Problem

In FastAPI applications, you typically define SQLAlchemy models for the database and Pydantic schemas for the API. This means maintaining two parallel definitions for every entity: the ORM layer and the API layer. They drift apart over time.

dbwarden's `@auto_schema` eliminates this duplication by deriving Pydantic schemas directly from model annotations.

## Step 1: Define a Model with @auto_schema

```python
from dbwarden.databases import TableMeta
from dbwarden_fastapi import auto_schema


@auto_schema
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    username: Mapped[str] = mapped_column(String(100), unique=True, nullable=False)
    full_name: Mapped[str | None] = mapped_column(String(200), nullable=True)
    password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
    is_active: Mapped[bool] = mapped_column(Boolean, default=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("CURRENT_TIMESTAMP"))

    class Meta(TableMeta):
        comment = "User accounts with auto-generated Pydantic schemas"

        class email:
            comment = "Login email"

        class password_hash:
            public = False  # Excluded from PublicSchema
```

## Step 2: What Gets Generated

The decorator creates four schema classes on the model:

### `User.CreateSchema`

Used for POST requests: includes all fields that the client should provide. Server-defaulted fields (like auto-increment `id`) are excluded.

```python
create = User.CreateSchema(
    email="alice@example.com",
    username="alice",
    password_hash="secret",
    full_name="Alice Smith",
    is_active=True,
)
```

### `User.UpdateSchema`

All fields optional: used for PATCH requests.

```python
update = User.UpdateSchema(full_name="Alice Johnson")
```

### `User.PublicSchema`

Excludes fields marked `public = False`. Perfect for API responses where you never want to leak `password_hash`.

```python
user = User(
    id=1,
    email="alice@example.com",
    username="alice",
    password_hash="secret",
    is_active=True,
)
public = User.PublicSchema.model_validate(user)
print(public.model_dump())
# {
#     "id": 1,
#     "email": "alice@example.com",
#     "username": "alice",
#     "full_name": None,
#     "is_active": True,
#     "created_at": ...,
# }
# password_hash is NOT included
```

### `User.Schema`

All mapped columns, including those marked `public = False`.

## Step 3: Controlling Visibility

| Technique | Effect |
|-----------|--------|
| `class Meta: class field: public = False` | Excluded from PublicSchema |
| Field name starting with `_` | Implicitly `public = False` |
| `SchemaConfig(exclude_public=["field"])` | Excluded from PublicSchema |
| `SchemaConfig(exclude_create=["field"])` | Excluded from CreateSchema |

## Step 4: Customizing Schema Generation

```python
from dbwarden_fastapi import auto_schema, SchemaConfig


@auto_schema(config=SchemaConfig(
    exclude_public=["internal_note"],
    exclude_create=["created_at"],
    field_overrides={
        "email": EmailStr,
    },
))
class User(Base):
    ...
```

`SchemaConfig` supports:

| Option | Description |
|--------|-------------|
| `exclude_always` | Excluded from all schemas |
| `exclude_create` | Excluded from CreateSchema only |
| `exclude_update` | Excluded from UpdateSchema only |
| `exclude_public` | Excluded from PublicSchema only |
| `field_overrides` | Override Pydantic field types |
| `required_always` | Fields always required |
| `optional_always` | Fields always optional |

## Key Takeaways

- `@auto_schema` generates CreateSchema, UpdateSchema, PublicSchema, and Schema
- `public = False` in `class Meta` controls API visibility; no manual filtering in routes
- Fields starting with `_` are implicitly non-public
- Use `User.PublicSchema.model_validate(instance)` to convert model instances to API responses
- Customize with `SchemaConfig` for advanced use cases

## Related Documentation

- [Modeling Guide: Auto-Generated Schemas](../getting-started/modeling.md#auto-generated-pydantic-schemas-with-auto_schema)
- [SQLAlchemy Models Reference](../models.md)

## Next

[Section 11: Observability](11-observability.md)

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/11-observability/
========================================================================

# 11. Observability

## What You'll Learn

- How to enable Prometheus metrics for dbwarden
- How to use structured JSON logging
- How to add query tracing middleware to FastAPI
- How to monitor connection pool health

## Prerequisites

- `examples/observability/` directory
- Docker (for optional Prometheus + Grafana)

## Step 1: Enable Prometheus Metrics

Install with metrics support:

```bash
pip install prometheus-client
dbwarden plugin add dbwarden-fastapi
```

dbwarden exposes Prometheus metric families:

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `dbwarden_migrations_total` | Counter | `database`, `version`, `success` | Migration count |
| `dbwarden_migration_duration_seconds` | Histogram | `database`, `version` | Execution time |
| `dbwarden_schema_version` | Gauge | `database` | Current version |
| `dbwarden_migrations_pending` | Gauge | `database` | Pending count |
| `dbwarden_migration_errors_total` | Counter | `database` | Error count |
| `dbwarden_seed_version` | Gauge | `database` | Current seed version |

## Step 2: Add Metrics to FastAPI

```python
from dbwarden_fastapi import MetricsMiddleware, MetricsRouter

# Middleware captures request duration and counts
app.add_middleware(MetricsMiddleware)

# Router exposes /metrics endpoint
app.include_router(MetricsRouter(), prefix="/metrics")
```

```bash
curl http://localhost:8000/metrics
```

Output:

```
# HELP dbwarden_migrations_total Total number of migrations
# TYPE dbwarden_migrations_total counter
dbwarden_migrations_total{database="primary",status="applied"} 5

# HELP dbwarden_pending_migrations Number of pending migrations
# TYPE dbwarden_pending_migrations gauge
dbwarden_pending_migrations{database="primary"} 0
```

## Step 3: Structured Logging

```python
import os
os.environ["DBWARDEN_LOG_JSON"] = "1"
```

Or via environment variable:

```bash
DBWARDEN_LOG_JSON=1 uvicorn app.main:app
```

This switches from colored human-readable output to JSON:

```json
{"timestamp": "2025-01-15T10:30:00Z", "level": "INFO", "event": "migration_applied", "database": "primary", "duration_ms": 42, "version": "0005"}
```

JSON logs are easier to ingest into ELK, Datadog, or other log aggregators.

For CLI runs, the global `--json` flag produces the same JSON-formatted logs and
also switches display commands to structured output:

```bash
dbwarden --json status --database primary
dbwarden --json migrate --database primary
```

### TRACE logging

`--debug-level trace` logs every SQL statement executed by `migrate`,
`rollback`, and `downgrade`:

```bash
dbwarden --debug-level trace migrate --database primary
```

Each statement is logged as `SQL (2.1ms): CREATE TABLE ...`.

### Migration phase timing

Phase timings (lock acquisition, snapshot write, model state write) are logged
at INFO by default. Add `--perf` for a per-SQL-statement breakdown:

```bash
dbwarden migrate --perf --database primary
```

## Step 4: Query Tracing

```python
from dbwarden_fastapi import QueryTracingMiddleware

app.add_middleware(QueryTracingMiddleware)
```

This logs every SQL query with its duration:

```json
{"event": "query", "duration_ms": 3, "database": "primary", "statement": "SELECT ..."}
```

Useful for:
- Identifying slow queries in development
- Building a query performance baseline
- Debugging N+1 query patterns

## Step 5: Pool Metrics Collector

```python
from dbwarden_fastapi import PoolMetricsCollector
```

This monitors SQLAlchemy connection pool health and exposes:

- Pool size (current/total)
- Connections in use
- Connections overflow
- Pool timeouts

## Step 6: Full Setup

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from dbwarden_fastapi import (
    DBWardenHealthRouter,
    dbwarden_lifespan,
    MetricsMiddleware,
    MetricsRouter,
    QueryTracingMiddleware,
)


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with dbwarden_lifespan(app, mode="check"):
        yield


app = FastAPI(
    title="dbwarden Observability Example",
    lifespan=lifespan,
)

app.add_middleware(QueryTracingMiddleware)
app.add_middleware(MetricsMiddleware)
app.include_router(MetricsRouter(), prefix="/metrics")
app.include_router(DBWardenHealthRouter(), prefix="/health")
```

## Step 7: Prometheus + Grafana (Optional)

```yaml
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
```

```bash
docker compose up -d
```

- Prometheus: http://localhost:9090
- Grafana: http://localhost:3000

In Grafana, add Prometheus data source (`http://prometheus:9090`) and create dashboards using the `dbwarden_*` metrics.

## Key Takeaways

- Metrics are opt-in via `uv add "dbwarden[metrics]"`
- Six metric families cover migration, schema, and error tracking
- `DBWARDEN_LOG_JSON=1` (or the global `--json` flag) switches to structured JSON logging
- `--debug-level trace` logs every SQL statement during migration commands
- `--perf` adds per-statement timing breakdowns to migration commands
- `QueryTracingMiddleware` logs every SQL query with duration
- `PoolMetricsCollector` monitors connection pool health
- Metrics are compatible with standard Prometheus + Grafana setup

## Related Documentation

- [Observability Guide](../observability.md)

The FastAPI metrics middleware and routers are part of the `dbwarden-fastapi` plugin, documented in its own repository: [dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi).

========================================================================
PAGE: https://docs.dbwarden.org/cookbook/
========================================================================

# Cookbook & Examples

Practical, runnable examples that walk through the entire dbwarden workflow: from project setup through advanced observability patterns.

## How to Use

Each cookbook section links to code under the [`examples/`](https://github.com/dbwarden-org/dbwarden/tree/main/examples) directory. The **core examples** (sections 1-7) use SQLite and require only `uv add dbwarden`. Advanced examples may need Docker for PostgreSQL, ClickHouse, or Prometheus.

```
examples/
├── core/                 # Sections 1-7: progressive SQL workflow
├── multi-database/       # Section 8
├── fastapi-app/          # Sections 9-10 (FastAPI + auto-schema examples)
└── observability/        # Section 11
```

## Sections

| # | Section | What You'll Learn | Example Dir |
|---|---------|-------------------|-------------|
| 1 | [Project Setup](01-project-setup.md) | `init`, `config`, understanding declarative configuration | `examples/core/` |
| 2 | [Models & Migrations](02-models-and-migrations.md) | Model definitions, `make-migrations`, `new`, `make-rollback` | `examples/core/` |
| 3 | [Apply & Inspect](03-apply-and-inspect.md) | `migrate`, `rollback`, `downgrade`, `history`, `status`, `check`, `check-db` | `examples/core/` |
| 4 | [Offline & CI](04-offline-ci.md) | `export-models`, `make-migrations --offline` | `examples/core/` |
| 5 | [Schema Inspection](05-schema-inspection.md) | `diff`, `snapshot`, `generate-models` | `examples/core/` |
| 6 | [Safety & Impact](06-safety-impact.md) | `check`, `check-impact`, destructive change detection | `examples/core/` |
| 7 | [Seeds](07-seeds.md) | `seed create/apply/rollback/list`, SQL seeds, `@seed_data` | `examples/core/` |
| 8 | [Multi-Database](08-multi-database.md) | Multiple declarative databases, PG + ClickHouse, `--all` flag | `examples/multi-database/` |
| 9 | [FastAPI Integration](09-fastapi-integration.md) | Lifespan hooks, health endpoints, session DI, migration endpoints | `examples/fastapi-app/` |
| 10 | [Auto Schemas](10-auto-schemas.md) | `@auto_schema`, `CreateSchema`, `UpdateSchema`, `PublicSchema` (requires `dbwarden-fastapi` plugin) | `examples/fastapi-app/` |
| 11 | [Observability](11-observability.md) | Prometheus metrics, structured logging, query tracing | `examples/observability/` |

## Quick Start (Core)

```bash
cd examples/core
uv add dbwarden sqlalchemy
bash scripts/01-setup.sh
bash scripts/02-models-migrations.sh
bash scripts/03-apply-inspect.sh
```

Each section in the cookbook explains what these commands do, what SQL they produce, and why it matters.

## Database-Specific Examples

The core examples use SQLite for zero-dependency setup. For production, dbwarden fully supports PostgreSQL, MySQL, and ClickHouse; each with its own deep-dive guide and dedicated example patterns.

### PostgreSQL

PostgreSQL is a first-class backend with full round-trip support (read and write schema). The FastAPI integration example in [Section 9](09-fastapi-integration.md) uses PostgreSQL, and [Section 8](08-multi-database.md) shows PostgreSQL + ClickHouse together.

For the complete reference on PostgreSQL-specific metadata (identity columns, collation, compression, generated columns, tablespace, inheritance, exclusion constraints, deferrable FKs, advanced index options), see the [PostgreSQL Deep Dive](../databases/postgresql/index.md).

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/myapp"
    database_url_async = "postgresql+asyncpg://user:password@localhost:5432/myapp"
    model_paths = ["app.models"]
```

### MySQL / MariaDB

MySQL and MariaDB are first-class backends with full round-trip support. All MySQL-specific metadata (engine, charset, collation, row format, auto_increment, unsigned columns, ON UPDATE, column comments) is captured by the snapshot, diffed correctly, and emitted as valid DDL.

See the [MySQL Deep Dive](../databases/mysql/index.md) for the complete reference, including MySQL-specific model metadata via `class Meta(MyTableMeta)`.

```python
from dbwarden import DbwardenDatabase

class Legacy(DbwardenDatabase):
    database_name = "legacy"
    database_type = "mysql"
    database_url_sync = "mysql+pymysql://user:password@localhost:3306/legacy"
    model_paths = ["app.legacy_models"]
```

### ClickHouse

ClickHouse is supported with partial round-trip (read schema and auto-generate most DDL). dbwarden uses the ClickHouse HTTP client directly for DDL execution and supports full engine metadata via `class Meta(CHTableMeta)` with `ChEngineSpec`, `ProjectionSpec`, and `CHColumnMeta`.

See the [ClickHouse Deep Dive](../databases/clickhouse/index.md) for full details on materialized views, projections, dictionaries, replicated engines, and ClickHouse-specific metadata.

ClickHouse is typically configured alongside a transactional database (see [Section 8](08-multi-database.md) for a PostgreSQL + ClickHouse example).

========================================================================
PAGE: https://docs.dbwarden.org/correctness/convergence-gate/
========================================================================

# Convergence Gate

The convergence gate is the strongest correctness check in dbwarden. It proves that the complete migration history can reproduce the schema declared by the current models.

The gate answers one question:

> If a new environment starts with an empty database and applies every migration, does the resulting schema exactly match the models in the repository?

If the answer is no, CI should fail.

## Definition

Convergence means:

```text
Empty database
  |
  v
Apply every migration
  |
  v
Extract live schema
  |
  v
Compare live schema with model spec
  |
  v
Pass only if there is zero drift
```

This is an end-to-end, model-history-consistency check. It does not only test the newest migration. It tests the entire path from an empty database to the current model state.

## Pipeline

The typical CI pipeline has four phases.

### 1. Spin Up an Ephemeral Database

CI starts a disposable database service. For PostgreSQL this is usually a container. For ClickHouse this can be a ClickHouse service container. The database should be empty at the start of the job.

The important property is isolation. The gate should never run against a shared developer database, because manual changes in that database would hide or create drift that is unrelated to the repository.

### 2. Apply the Full Migration History

Run the same command used in deployment:

```bash
dbwarden migrate --database primary
```

This applies versioned migrations and any repeatable migration behavior supported by the project configuration. The gate tests the migration files that will run in production, not a simplified test fixture.

### 3. Check the Resulting Schema

Run dbwarden's diff command:

```bash
dbwarden diff --database primary
```

The diff compares the database state with the model state. If the command reports differences, CI fails. `dbwarden check` is a separate safety classifier, not the schema-convergence comparison. Some teams also run `dbwarden diff` in table or JSON mode for diagnostic output:

```bash
dbwarden diff --database primary --out table
dbwarden diff --database primary --out json
```

Use the diff output to see which object drifted. The gate should not ignore drift. If the models are correct, generate or write a migration. If the migration is correct, update the model.

### 4. Fail Fast on Drift

The final rule is simple:

```text
No drift -> merge allowed
Any drift -> merge blocked
```

The migration history and the models must agree.

## GitHub Actions Example

This example uses PostgreSQL. The same pattern applies to any backend that can run in CI.

```yaml
name: dbwarden-convergence

on:
  pull_request:
  push:
    branches: [main]

jobs:
  convergence:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: dbwarden
          POSTGRES_PASSWORD: dbwarden
          POSTGRES_DB: app
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    env:
      DATABASE_URL: postgresql://dbwarden:dbwarden@localhost:5432/app

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install project
        run: pip install -e .

      - name: Apply migrations
        run: dbwarden migrate --database primary

       - name: Check convergence
         run: dbwarden diff --database primary

      - name: Print drift diagnostics on failure
        if: failure()
        run: dbwarden diff --database primary --out table
```

The `--health-cmd` option belongs to Docker service configuration. The dbwarden commands are the two important convergence commands: `migrate` and `diff`.

## Measured Cost

The cost is dominated by replaying the migration history, not by the final
schema diff. On 2026-08-14, the harness measured a synthetic 500-migration
SQLite project with one table per migration:

| dbwarden executable | Preparation | Full replay | Final diff | Total |
|---|---:|---:|---:|---:|
| Current core checkout | 2.2s | 274.1s | 6.0s | 282.3s |
| PyPI `0.16.5` | 1.8s | 291.8s | 6.7s | 300.3s |

The current core checkout also supports bulk replay with one final snapshot:

```bash
dbwarden migrate --defer-snapshots
```

The same 500-migration SQLite workload measured `10.4s` for replay and
`18.3s` total with deferred snapshots. This preserves the final convergence
artifact while avoiding a full schema reflection after every migration. The
default remains per-migration snapshots for compatibility with audit workflows;
long-history CI gates should use the deferred mode and retain the final
snapshot as their correctness artifact.

These are local reference measurements, not an SLA. Database startup,
filesystem speed, migration complexity, indexes, plugins, and backend network
latency can materially change the result. The benchmark is reproducible from
the standalone harness with:

```bash
DBWARDEN_HARNESS_RUN_500_MIGRATION=1 uv run pytest \
  suites/performance/test_convergence_500.py -s
```

The current `convergence-audit` GitHub Actions job runs `pytest -m p0`. At
present only a handler-level ClickHouse test carries the `p0` marker, so that
job is not equivalent to replaying a project's full migration history. A
project that needs the stronger guarantee should run the harness's
provider-backed convergence suite as a separate, explicitly budgeted job.

## Example Error Caught by the Gate

Assume a model declares an index:

```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(255), nullable=False)

    class Meta:
        indexes = [IndexSpec(name="idx_users_email", columns=["email"])]
```

But the migration only creates the table:

```sql
-- upgrade
CREATE TABLE users (
    id INTEGER NOT NULL PRIMARY KEY,
    email VARCHAR(255) NOT NULL
);

-- rollback
DROP TABLE IF EXISTS users CASCADE;
```

The migration applies successfully. A unit test that only checks syntax might pass. Production would have a table without the declared index.

The convergence gate catches this:

```text
Model state:
  users has idx_users_email

Extracted database state after migrations:
  users has no idx_users_email

Result:
  drift detected
```

The fix is to generate or add a migration that creates the missing index.

## Other Errors It Catches

### Leftover Column

A column is removed from the model, but no migration drops it. The gate applies all migrations, extracts the schema, and sees that the database still contains the old column.

### Reversed Migration

A migration accidentally renames `customer_id` to `client_id` while the model still declares `customer_id`. The migration applies, but the final schema disagrees with the model.

### Missing Backend Metadata

PostgreSQL table storage parameters, identity options, RLS policies, or ClickHouse engine settings are easy to miss in hand-written SQL. The gate compares the extracted backend-specific shape, not only the visible column list.

## Why This Is the Gold Standard

The convergence gate tests exactly the behavior that matters in production:

- It uses the real migration files.
- It uses the real backend SQL emitters.
- It uses a real database engine.
- It verifies the final schema against the models.
- It catches drift introduced anywhere in the history.

This is stronger than checking that a single generated migration looks plausible. A migration can be syntactically valid and still fail to converge.

## Relationship to Offline Integrity

Offline workflows are useful when CI cannot reach a database service. They let dbwarden generate migrations from a checked-in model state instead of a live database. That is deterministic, but it does not prove the SQL applies to a real engine.

A strong pipeline uses both:

```text
Offline integrity check
  |
  v
Generate or verify migration files deterministically
  |
  v
Convergence gate on a live database
  |
  v
Prove the SQL reproduces the model state
```

See [Offline Integrity](offline-integrity.md) for the first half of that workflow.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/deterministic-diff/
========================================================================

# Deterministic Diff

The deterministic diff is the reason dbwarden can generate stable, reviewable migrations. Given the same model state and the same database snapshot, dbwarden should produce the same operation list every time.

Without deterministic diffing, automated migrations become noisy. A pull request might alternate between two equivalent SQL forms. Reviewers would waste time checking fake changes instead of real schema changes.

## The Problem

Databases and model code often describe the same schema in different ways.

Examples:

- A model says `String(255)` while PostgreSQL reports `character varying(255)`.
- A model says `Boolean` while a database default is rendered as `false` or `FALSE`.
- A PostgreSQL identifier may appear quoted in SQL and unquoted in model metadata.
- ClickHouse engine parameters may be returned in a different order than they were declared.
- Whitespace in view definitions may differ while the query is logically unchanged.

If dbwarden compared raw strings directly, it would produce false diffs.

## Canonicalization

dbwarden handlers canonicalize specs before diffing. Each handler owns the rules for its object type.

The shape is:

```text
Model spec
  |
  v
ObjectHandler.canonicalize(model_spec)
  |
  v
Canonical model spec

Database snapshot
  |
  v
ObjectHandler.canonicalize(snapshot_spec)
  |
  v
Canonical snapshot spec

Canonical model spec + canonical snapshot spec
  |
  v
ObjectHandler.diff(...)
  |
  v
Typed operations
```

The important property is that equivalent states normalize to the same representation.

## Handler Contract

Backend handlers expose a small contract:

```text
extract(snapshot) -> raw backend state
model_spec_from_tables(models) -> raw model state
canonicalize(spec) -> stable comparable state
diff(snapshot_spec, model_spec) -> operations
emit(operation) -> SQL statements
```

The `canonicalize` step sits between raw state and diffing. It is the boundary where noisy representation differences are removed.

## PostgreSQL Normalization Examples

### Type Aliases

PostgreSQL may report type names differently than SQLAlchemy models declare them.

```text
Model declaration:
  String(255)

Database extraction:
  character varying(255)

Canonical form:
  varchar(255)
```

After canonicalization, dbwarden does not emit a type-change migration for this column.

### Boolean Defaults

Different sources may use different boolean spelling.

```text
Model default:
  true

Database default:
  TRUE

Canonical meaning:
  true
```

The diff should compare the normalized meaning, not the original spelling.

### Identifier Quoting

PostgreSQL accepts both quoted and unquoted identifiers, but quoted identifiers preserve case. dbwarden normalizes safe names while preserving names that require quotes.

```text
users.email
"users"."email"

Canonical object identity:
  users.email
```

The goal is not to remove all quotes from emitted SQL. The goal is to compare object identity consistently before SQL generation.

## ClickHouse Normalization Examples

### Engine Metadata

ClickHouse table metadata is not just the engine name. A `MergeTree` table can include partition keys, primary keys, order keys, sample keys, TTL expressions, and settings.

```text
Model:
  engine = MergeTree()
  order_by = ["event_date", "id"]
  settings = {"index_granularity": "8192"}

Snapshot:
  ENGINE = MergeTree()
  ORDER BY (event_date, id)
  SETTINGS index_granularity = 8192

Canonical form:
  engine family: MergeTree
  order_by: [event_date, id]
  settings.index_granularity: 8192
```

The canonical form lets dbwarden decide whether a real engine change occurred.

### Projection and Skip Index Specs

ClickHouse projections and skip indexes contain expressions. dbwarden normalizes the supported metadata fields so the diff sees stable names, expressions, types, and granularities.

```text
Projection model:
  name: by_date
  select: SELECT event_date, count() GROUP BY event_date

Snapshot projection:
  name: by_date
  query returned by system tables

Canonical comparison:
  same projection identity and same normalized query body
```

## Tricky Case Walkthrough

Consider a model column:

```python
name: Mapped[str] = mapped_column(String(255), nullable=False, default="hello")
```

PostgreSQL may report:

```sql
name character varying(255) NOT NULL DEFAULT 'hello'::character varying
```

The raw strings differ:

```text
String(255)
character varying(255)

"hello"
'hello'::character varying
```

The canonical comparison should reduce them to the same meaning:

```text
type: varchar(255)
nullable: false
default: 'hello'
```

Result:

```text
No operation emitted
```

If the model changes to `String(500)`, canonicalization no longer hides the difference:

```text
Canonical model type: varchar(500)
Canonical snapshot type: varchar(255)
Result: alter column type operation
```

## Why Determinism Matters

Determinism gives dbwarden four practical properties.

### Reviewability

The same input produces the same migration. Reviewers can trust that a migration is caused by a real schema change, not by formatter noise.

### Auditability

Generated SQL can be compared across CI runs. If the output changes, the inputs changed or the generator changed.

### No Flapping

Flapping happens when a tool alternates between two equivalent forms. For example:

```text
Run 1 emits: ALTER COLUMN name TYPE VARCHAR(255)
Run 2 emits: ALTER COLUMN name TYPE character varying(255)
Run 3 emits: ALTER COLUMN name TYPE VARCHAR(255)
```

Canonicalization prevents this class of migration noise.

### Safe Automation

CI can block drift because the diff is stable. A nondeterministic diff would make CI untrustworthy.

## Link to SQL Generation

The canonical spec is not the final SQL. It is the stable state that the diff engine compares. Once the diff produces typed operations, backend emitters render SQL.

See [SQL Generation](sql-generation.md) for the next stage in the pipeline.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/
========================================================================

# Compilation Correctness

dbwarden compiles models to SQL. This section explains the mechanical checks that make that compilation trustworthy: how dbwarden proves the compiled SQL matches the declared model state, how it prevents noisy diffs, how it warns before risky changes, and how rollback SQL is produced under a strict contract.

The claim is not "trust the tool because it is convenient". The claim is stronger: dbwarden exposes a chain of independent checks. Each check catches a different class of failure. Together they make generated migrations reviewable, reproducible, and suitable for CI enforcement.

Start with the [convergence gate](convergence-gate.md). It is the single most powerful correctness check because it tests the full migration history against an empty database and verifies that the resulting schema matches the models.

## Chain of Trust

```text
Models
  |
  v
Canonical model state
  |
  v
Deterministic diff
  |
  v
Typed operations in a migration plan
  |
  +--> Safety classifier
  |
  v
Backend-native SQL emitters
  |
  v
Plain SQL migration file
  |
  +--> Strict rollback contract
  |
  v
Apply to database
  |
  v
Extract schema snapshot
  |
  v
Convergence and round-trip verification
```

Each stage has a different responsibility. Canonicalization removes representation noise. Diffing decides what changed. SQL emitters turn typed operations into backend-native SQL. Safety classification highlights risky operations before execution. Rollback generation checks whether the reverse path is executable or explicitly irreversible. Convergence checks prove that the final database state matches the model state.

## Defense in Depth

### Convergence Gate

The convergence gate applies the complete migration history to an empty database, extracts the resulting schema, and compares it with the current model specification. The gate passes only when there is zero schema drift.

This catches errors that unit tests can miss, such as a migration file that forgot an index, a model rename that was emitted as a drop and add, or a rollback edit that accidentally changed the upgrade section. See [Convergence Gate](convergence-gate.md).

### Round-Trip Verification

Round-trip verification checks extract and emit consistency. dbwarden reads a real database with `generate-models`, emits models, generates migrations from those models, and verifies that extraction after apply produces the same schema shape.

This is especially important for backend-specific features such as PostgreSQL identity columns, partitioning, exclusion constraints, ClickHouse engines, projections, skip indexes, and RBAC objects. See [Round-Trip Verification](round-trip-verification.md).

### Deterministic Diff

dbwarden canonicalizes model specs and database snapshots before comparing them. The same inputs produce the same diff every time.

Canonicalization prevents representation noise from turning into fake migrations. For example, PostgreSQL may report `character varying(255)` while a SQLAlchemy model says `String(255)`. ClickHouse may report engine metadata in an order that differs from the declaration. dbwarden normalizes those forms before diffing. See [Deterministic Diff](deterministic-diff.md).

### Safety Classifier

The safety classifier scans migration plans and identifies risky changes before execution. It distinguishes safe changes from warnings and blocking changes, then requires explicit acknowledgement for risky operations.

The goal is simple: dbwarden should never silently drop data. See [Safety Classifier](safety-classifier.md).

### Offline Integrity

Offline mode lets CI generate migrations without a live database by using a checked-in model state file. Schema snapshots and model state files make migration generation deterministic and protect the pipeline from accidental drift in a developer database.

Offline integrity is not a replacement for a live convergence gate. It is the deterministic first half of the workflow. See [Offline Integrity](offline-integrity.md).

### SQL Generation

dbwarden does not hide runtime logic inside migration files. It emits plain SQL. The diff becomes typed operations, operations are ordered, and backend-specific handlers render native SQL for PostgreSQL, MySQL, SQLite, MariaDB, and ClickHouse.

The output is designed for review. See [SQL Generation](sql-generation.md).

### Rollback Generation

Rollback SQL is generated with the upgrade SQL under a strict contract. Executable rollback SQL is accepted. Placeholder rollback is refused by default. Operations that cannot be rolled back automatically must be explicitly declared irreversible.

This means rollback correctness is not an afterthought and not a manually maintained parallel file. See [Rollback Generation](rollback-generation.md) and [Rollback Coverage Matrix](rollback-coverage-matrix.md).

## Recommended Reading Order

1. [Convergence Gate](convergence-gate.md)
2. [Deterministic Diff](deterministic-diff.md)
3. [SQL Generation](sql-generation.md)
4. [Rollback Generation](rollback-generation.md)
5. [Safety Classifier](safety-classifier.md)
6. [Offline Integrity](offline-integrity.md)
7. [Round-Trip Verification](round-trip-verification.md)

The pages are independent, but the trust model is cumulative. The convergence gate proves the final database shape. Deterministic diffing explains why generated changes are stable. SQL generation explains why the emitted file is backend-native and reviewable. Rollback generation explains the reverse path. Safety classification and offline integrity make the workflow suitable for CI.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/object-handler-protocol/
========================================================================

# ObjectHandler Protocol

The `ObjectHandler` protocol is the mechanical contract that makes deterministic diffs, backend-native SQL generation, and symmetric rollbacks possible. Every backend object - tables, columns, indexes, views, roles, policies, grants, ClickHouse projections, materialized views, and more - has a dedicated handler that owns its full lifecycle: extraction, canonicalization, diffing, and SQL emission.

Without understanding the handler contract, the other correctness guarantees feel like magic. With it, they become inspectable engineering.

## The Problem It Solves

Every database backend has a different DDL dialect, different catalog tables, and different type systems. PostgreSQL uses `pg_catalog`, MySQL uses `information_schema`, ClickHouse uses `system.tables`, and SQLite uses `sqlite_master`. Column types that look similar (`VARCHAR(255)`, `String`, `TEXT`) have different internal representations. Engine metadata - partitioning, sorting keys, TTL expressions - exists only in ClickHouse. Roles and policies are structured differently across PostgreSQL and ClickHouse.

Core must treat all of these uniformly without hard-coding backend knowledge into the diff engine. Adding a new object type or a new backend must not require rewriting the diff pipeline.

The handler protocol solves this by defining a narrow interface that each backend implements per object type. Core calls the interface. Backend code provides the implementation.

## The Handler Contract

Every handler implements the `ObjectHandler` protocol defined in `dbwarden/engine/core/protocol.py`:

```python
class ObjectHandler(Protocol):
    object_type: str
    run_phase: RunPhase
    statement_order: StatementOrder
    ordering: OrderingConstraint

    def extract(self, snapshot: dict[str, Any]) -> dict[str, Any]: ...

    def model_spec_from_config(self, config: Any) -> dict[str, Any]: ...

    def model_spec_from_tables(
        self, model_tables: list[Any]
    ) -> dict[str, Any]: ...

    def canonicalize(self, spec: dict[str, Any]) -> dict[str, Any]: ...

    def diff(
        self,
        snap_spec: dict[str, Any],
        model_spec: dict[str, Any],
    ) -> Tuple[List[Op], List[Op]]: ...

    def emit(
        self, op: Op, db_name: Optional[str] = None, **kwargs: Any,
    ) -> List[MigrationStatement]: ...
```

### Method Roles

| Method | Input | Output | Responsibility |
|---|---|---|---|
| `extract` | Full database snapshot dict | Raw backend state for this object type | Pulls the handler's relevant objects out of a schema snapshot |
| `model_spec_from_config` | Backend config | Raw config-derived spec | Builds spec from configuration (used in PREAMBLE phase for objects not in model tables) |
| `model_spec_from_tables` | List of model tables | Raw model-derived spec | Builds spec from Python model definitions (used in DIFF phase) |
| `canonicalize` | Raw spec | Normalized comparable spec | Removes representation noise so that equivalent states produce identical dicts |
| `diff` | Canonical snapshot spec, canonical model spec | Upgrade ops + rollback ops | Detects differences and produces paired typed operations |
| `emit` | A single Op | List of MigrationStatement objects | Renders backend-native SQL for upgrade and rollback |

### Supporting Types

**`Op`** is the typed operation that `diff` produces:

```python
@dataclass
class Op:
    object_type: str
    upgrade_attrs: dict[str, Any] = field(default_factory=dict)
    rollback_attrs: dict[str, Any] = field(default_factory=dict)
    irreversible: bool = False
```

`upgrade_attrs` stores the data needed to emit the upgrade SQL. `rollback_attrs` stores the data needed to reverse it. Both are carried together so that rollback never needs to re-inspect the database or guess prior state.

**`MigrationStatement`** is the emit output:

```python
@dataclass
class MigrationStatement:
    order: StatementOrder
    upgrade_sql: str
    rollback_sql: str
    rollback_kind: str = "real"
    rollback_reason: str | None = None
```

`StatementOrder` is an `IntEnum` that controls the sequence of SQL emission. Upgrade statements are sorted by this value. Rollback statements are emitted in reverse order.

**`RunPhase`** controls when the handler runs:

```python
class RunPhase(IntEnum):
    PREAMBLE = 0   # runs before table-diff phase; uses model_spec_from_config
    DIFF = 1       # runs during table-diff phase; uses model_spec_from_tables
```

PREAMBLE handlers cover objects like roles and schemas that exist outside the table model tree. DIFF handlers cover objects that are derived from table metadata.

## How This Enables Correctness

### Canonicalize → Deterministic Diffs

`canonicalize` is the noise-removal step. Two specs that represent the same logical state must produce identical dictionaries after canonicalization. This guarantees that the diff only fires on real changes, not on formatting differences.

For example, PostgreSQL may report `character varying(255)` while a model says `String(255)`. The column handler's canonicalization reduces both to `varchar(255)`. A model default of `true` and a database default of `TRUE` both normalize to `true`.

If the canonicalized specs match, no diff is produced. If they differ, the diff is a real schema change.

### Diff Returns Paired Ops → Symmetric Rollbacks

`diff` returns two lists: upgrade ops and rollback ops. They are produced at the same time from the same inputs. This means rollback is not a separate best-effort pass.

Each op carries `upgrade_attrs` (what `emit` needs for the forward SQL) and `rollback_attrs` (what `emit` needs for the reverse SQL). For a column type change:

```python
Op(
    object_type="alter_column_type",
    upgrade_attrs={"table": "users", "column": "name", "model_type": "varchar(500)"},
    rollback_attrs={"table": "users", "column": "name", "model_type": "varchar(255)"},
)
```

The same `emit` method uses `upgrade_attrs` to render the upgrade statement and `rollback_attrs` to render the rollback. The reverse is structurally paired.

### Emit → Backend-Native SQL

`emit` produces raw SQL strings wrapped in `MigrationStatement` objects. Different backends implement `emit` differently for the same object type. A column type change in PostgreSQL produces:

```sql
ALTER TABLE users ALTER COLUMN name TYPE VARCHAR(500);
```

The same logical change in ClickHouse produces:

```sql
ALTER TABLE users MODIFY COLUMN name String;
```

The diff pipeline does not know or care which dialect is being generated. Core calls `emit`, and the backend handler owns the SQL.

## How Handlers Are Registered

Handlers are registered with the `RegistryDriver` in `dbwarden/engine/core/registry.py`:

```python
class RegistryDriver:
    def __init__(self, *, include_plugins: bool = True):
        self._handlers: dict[str, ObjectHandler] = {}
        self._plugin_claims: dict[str, ObjectHandlerRegistration] = {}

    def register(self, handler: ObjectHandler) -> None:
        self._handlers[handler.object_type] = handler
```

Registration happens during backend initialization. Each backend module (PostgreSQL, ClickHouse, MySQL) creates its set of handlers and registers them:

```python
driver.register(ColumnHandler())
driver.register(IndexHandler())
driver.register(TableHandler())
driver.register(ConstraintHandler())
driver.register(EnumHandler())
driver.register(DomainHandler())
driver.register(RoleHandler())
driver.register(PoliciesHandler())
driver.register(GrantsHandler())
driver.register(ViewHandler())
driver.register(TriggerHandler())
driver.register(SequenceHandler())
driver.register(PartitionHandler())
driver.register(SchemaHandler())
driver.register(FunctionHandler())
driver.register(CompositeTypeHandler())
driver.register(DefaultPrivilegesHandler())
driver.register(EventTriggerHandler())
driver.register(ExtendedStatisticsHandler())
driver.register(StorageParamsHandler())
driver.register(RenameTableHandler())
driver.register(PgTableHandler())
# ClickHouse handlers
driver.register(ChTableHandler())
driver.register(ChColumnHandler())
driver.register(ChProjectionHandler())
driver.register(ChSkipIndexHandler())
driver.register(ChMaterializedViewHandler())
driver.register(ChCommentHandler())
driver.register(ChDictionaryHandler())
driver.register(ChRoleHandler())
driver.register(ChUserHandler())
driver.register(ChRowPolicyHandler())
driver.register(ChSettingsProfileHandler())
driver.register(ChQuotaHandler())
driver.register(ChNamedCollectionHandler())
driver.register(ChGrantHandler())
driver.register(ChAggTargetHandler())
driver.register(ChDataOpHandler())
```

Core's `RegistryDriver.run()` iterates over all registered handlers, extracts their specs, canonicalizes, diffs, and collects the paired ops. The caller then orders them by `statement_order` and emits SQL.

Because registration is just a method call, third-party plugins can register handlers for custom object types using the same protocol.

## Example Walk-Through: ALTER TABLE ADD COLUMN

This trace follows a single `ALTER TABLE ADD COLUMN` through the full pipeline.

### 1. Extract

The database snapshot is a large dict of all extracted schema objects. The `ColumnHandler.extract` pulls only the column data:

```python
# ColumnHandler.extract
def extract(self, snapshot):
    result = {}
    for tname, tdata in snapshot.get("tables", {}).items():
        result[tname] = dict(tdata.get("columns", {}))
    return result
```

Input snapshot (abbreviated):
```json
{
  "tables": {
    "users": {
      "columns": {
        "id": {"type": "integer", "nullable": false, "primary_key": true},
        "name": {"type": "character varying(255)", "nullable": true}
      }
    }
  }
}
```

Output:
```json
{
  "users": {
    "id": {"type": "integer", "nullable": false, "primary_key": true},
    "name": {"type": "character varying(255)", "nullable": true}
  }
}
```

### 2. Model Spec

`model_spec_from_tables` reads the same columns from the Python model definitions:

```json
{
  "users": {
    "id": {"type": "INTEGER", "nullable": false, "primary_key": true},
    "name": {"type": "VARCHAR(255)", "nullable": true},
    "display_name": {"type": "VARCHAR(255)", "nullable": true}
  }
}
```

The model has a new column `display_name` that does not exist in the snapshot.

### 3. Canonicalize

Both specs are run through `canonicalize`. The column handler currently passes through directly, but in general this step normalizes type names, default expressions, and metadata formats so that the diff is stable.

After canonicalization, `character varying(255)` and `VARCHAR(255)` both become `varchar(255)`.

### 4. Diff

`diff` compares the canonical specs. It detects:

- `display_name` exists in the model spec but not in the snapshot spec → **add column**
- `display_name` does not exist in the snapshot spec → no column to drop → the rollback of add is **drop column**

It produces paired ops:

```python
# Upgrade op
Op(
    object_type="add_column",
    upgrade_attrs={
        "table": "users",
        "column": "display_name",
        "model_column": model_col,  # ModelColumn("display_name", "VARCHAR(255)", ...)
    },
    rollback_attrs={"table": "users", "column": "display_name"},
)

# Rollback op
Op(
    object_type="drop_column",
    upgrade_attrs={"table": "users", "column": "display_name"},
    rollback_attrs={"table": "users", "column": "display_name"},
)
```

### 5. Emit

During SQL generation, `RegistryDriver.emit_all` dispatches each op to the correct handler's `emit` method. For `add_column`, the `ColumnHandler.emit` renders:

```python
sql = generate_add_column_sql("users", model_col, db_name)
# → "ALTER TABLE users ADD COLUMN display_name VARCHAR(255);"

stmt = MigrationStatement(
    order=StatementOrder.ADD_COLUMN,
    upgrade_sql=sql,
    rollback_sql="ALTER TABLE users DROP COLUMN display_name",
)
```

The `MigrationStatement` is then included in the final migration file:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

The same protocol handles column type changes, nullable changes, default changes, comment changes, indexes, constraints, roles, policies, and every other supported object type across all backends.

## The Ordering Constraint

Each handler declares a `statement_order` that controls when its emitted SQL appears relative to other statements. The `StatementOrder` enum defines anchor points such as `ADD_COLUMN`, `DROP_TABLE`, `ALTER_CONSTRAINT`, and `CREATE_TYPE`.

Core sorts all `MigrationStatement` objects by their `order` value before writing the migration file. Rollback statements are emitted in reverse order so that dependencies are satisfied in both directions.

This ordering mechanism is intentionally simple. Future work may introduce a topological sort where handlers declare edges against named anchors instead of numeric enum values. For now, the integer enum provides enough granularity for correct migration ordering across all supported backends.

## How to Add a New Handler

### 1. Implement the Protocol

Create a new class that implements `ObjectHandler`:

```python
from dbwarden.engine.core.protocol import ObjectHandler, Op, RunPhase
from dbwarden.engine.core.statement_order import MigrationStatement, StatementOrder

class MyCustomHandler(ObjectHandler):
    object_type: str = "my_custom_object"
    op_types: tuple[str, ...] = ("my_custom_op",)
    run_phase: RunPhase = RunPhase.DIFF
    statement_order: StatementOrder = StatementOrder.ALTER_TABLE_OPTIONS

    def extract(self, snapshot):
        ...

    def model_spec_from_config(self, config):
        return {}

    def model_spec_from_tables(self, model_tables):
        ...

    def canonicalize(self, spec):
        ...

    def diff(self, snap_spec, model_spec):
        upgrade_ops = []
        rollback_ops = []
        ...
        return upgrade_ops, rollback_ops

    def emit(self, op, db_name=None, **kwargs):
        ...
```

### 2. Register It

```python
from dbwarden.engine.core.registry import RegistryDriver

driver = RegistryDriver()
driver.register(MyCustomHandler())
```

### 3. Write Round-Trip Tests

- Verify that `extract` + `canonicalize` applied to a known snapshot produces the expected spec.
- Verify that `diff` with identical canonical specs produces no ops.
- Verify that `diff` with differing specs produces the expected upgrade/rollback pair.
- Verify that `emit` produces the expected SQL for each operation variant.
- Verify that the full round trip (extract → canonicalize → diff → emit → apply → extract) returns to the original state.

### 4. Add Extraction Support

If the new handler reads from the database, add extraction logic in the appropriate backend extractor (e.g., `extract.py` for PostgreSQL, `extract_ch.py` for ClickHouse). The extractor populates the snapshot dict that `extract` reads.

### 5. Add Safety Classification

If the handler produces operations that could drop data or require user acknowledgement, add entries in the safety classifier in `dbwarden/engine/safety/snapshot.py`.

## Relation to Other Correctness Pages

The handler protocol is the bridge between the abstract guarantees and the concrete pipeline:

| Correctness property | Role of the handler protocol |
|---|---|
| **Deterministic diff** | `canonicalize` removes representation noise before comparison. Two equivalent states produce identical canonical forms, so diffs only fire on real changes. |
| **SQL generation** | `emit` produces backend-native SQL. The diff pipeline generates typed operations; the handler renders them for one specific backend and object family. |
| **Symmetric rollback** | `diff` returns paired upgrade and rollback ops. Each Op carries both forward and reverse attributes, so rollback is structurally paired with upgrade from the start. |

- [Deterministic Diff](deterministic-diff.md) - the handler's `canonicalize` method is what makes diffs stable
- [SQL Generation](sql-generation.md) - the handler's `emit` method is what produces backend-native SQL
- [Rollback Generation](rollback-generation.md) - the handler's `diff` method returns paired ops that make symmetric rollbacks possible

========================================================================
PAGE: https://docs.dbwarden.org/correctness/offline-integrity/
========================================================================

# Offline Integrity

Offline integrity lets dbwarden generate migrations without connecting to a live database. It does this by comparing current models to a checked-in model state file or schema snapshot instead of querying the database at generation time.

This is a correctness feature because it removes accidental dependence on a developer's local database. CI can generate or verify migrations from repository state alone.

## Snapshot and Model State Concepts

dbwarden uses two related state files.

### Schema Snapshots

After migrations are applied, dbwarden can write checksummed schema snapshots under `.dbwarden/schemas/`. These snapshots represent the database schema after a migration point.

They support:

- Rename detection
- Offline comparisons
- Column-level diffing without querying a live database
- Auditability of schema history

### Model State

`export-models` writes a model state JSON file, usually under `.dbwarden/model_state.json` or a database-specific variant. This file records the model-derived schema state that offline migration generation uses as a baseline.

Create it with:

```bash
dbwarden export-models --database primary
```

Commit it with the repository:

```bash
git add .dbwarden/model_state.json
git commit -m "Update dbwarden model state"
```

The model state file records the schema that dbwarden expects the database to be in after the last migration. It is the baseline for offline diffing and can be regenerated at any time from a live database. `export-models` reads current model declarations without connecting to a database. Export the intended baseline before changing models; exporting after an unrecorded model edit can hide that edit from the next diff.

## Offline Migration Generation

Offline migration generation compares the checked-in state file with the current models:

```text
Checked-in model state
  |
  v
Normalize baseline

Current SQLAlchemy models
  |
  v
Extract model state

Baseline + current model state
  |
  v
Offline diff
  |
  v
Generated migration SQL and plan
```

Run it with:

```bash
dbwarden make-migrations "add profile fields" --offline --database primary
```

If the state file is missing, dbwarden tells you to run `export-models` first. If the state file is invalid, dbwarden refuses to use it.

> **If accidentally deleted:** restore it from git (`git checkout .dbwarden/model_state.json`), use `dbwarden recover-model-state --database <db>` to recover from available migration state, or regenerate it by running `dbwarden export-models --database <db>` against a live database. Run `export-models` only when current models represent the intended baseline, and preserve the database-specific and legacy files together. Offline commands will work again immediately.

## Integrity Check

Model state and schema snapshots are checksummed. Before dbwarden uses a state file, it validates that the file content matches the stored checksum.

Generated migration plans record the SQL content hash and the checksums of the schema before and after their operations. Pending-plan composition checks these values and rejects stale or inconsistent plans. `--strict-pending` also rejects pending files without a composable plan.

The reason is straightforward:

```text
State file on disk
  |
  v
Recompute SHA-256 checksum
  |
  v
Compare with stored checksum
  |
  +--> match: use the state file
  |
  +--> mismatch: refuse and ask for regeneration
```

This protects against accidental edits, merge corruption, and stale generated files. A modified JSON file should not silently become the baseline for migration generation.

Exported model-state JSON has no independent signature or embedded integrity checksum. Version-control review or an external file manifest is required to detect arbitrary edits to that baseline. Missing or malformed offline state exits with an error and writes no migration.

Generation and merge restore previous SQL, plan, marker, and state-file bytes if a write raises an exception. Both database-specific and legacy state files participate. Restoration attempts every file; if a file cannot be restored, the error lists its path and retains the original error as its cause. This rollback does not guarantee recovery from persistent filesystem errors, process termination or power loss during a multi-file update.

## CI Workflow Example

A deterministic CI workflow can use offline generation first, then live convergence when a database service is available.

```yaml
name: dbwarden-offline-integrity

on:
  pull_request:

jobs:
  offline-integrity:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install project
        run: pip install -e .

      - name: Generate migrations from checked-in state
        run: dbwarden make-migrations "ci offline check" --offline --database primary

      - name: Show changed files
        run: git status --short
```

In a strict repository, CI should fail if offline generation creates unexpected changes. That means a developer changed models without committing the corresponding migration or model state update.

The live convergence job can run after this step:

```text
Offline integrity
  |
  v
Generated files are stable
  |
  v
Live convergence gate
  |
  v
Generated SQL applies and matches models
```

See [Convergence Gate](convergence-gate.md).

## Example Failure

Assume the committed model state says `users` has two columns:

```json
{
  "tables": {
    "users": {
      "columns": {
        "id": {"type": "INTEGER", "primary_key": true},
        "email": {"type": "VARCHAR(255)", "nullable": false}
      }
    }
  }
}
```

A developer adds a model field:

```python
display_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
```

Then CI runs:

```bash
dbwarden make-migrations "ci offline check" --offline --database primary
```

dbwarden detects the difference between the committed state and current models:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

If this migration was expected, commit it. If it was not expected, the model change is accidental and should be reverted or corrected.

## Why Offline Integrity Improves Correctness

Offline integrity decouples migration generation from live database accidents.

Without offline state, a developer's local database could contain manual changes. dbwarden might diff against that local drift and produce a migration that looks correct on one machine but fails in CI or production.

With offline state:

- The baseline is versioned in git.
- The same inputs produce the same diff.
- CI can detect missing migrations without a database service.
- Tampered state files are rejected by checksum validation.
- Migration plans validate SQL and state-transition checksums. External manifests or version control detect arbitrary edits to exported baselines.

Offline integrity is not the final proof. The final proof is still a live database convergence gate. Offline integrity ensures the inputs to that gate are deterministic.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/rollback-coverage-matrix/
========================================================================

# Rollback Coverage Matrix

This matrix describes rollback behavior implemented by the PostgreSQL and ClickHouse handlers. It distinguishes real inverse SQL from conditional rollback, explicit irreversible changes, and placeholder or manual rollback comments.

## Classification

- `real`: Emits executable reverse SQL for the normal operation.
- `conditional`: Emits executable reverse SQL only when dbwarden can prove the operation is structurally reversible from static metadata and all required prior state is present. Conditional must be explicit and limited. If rollback safety is not secured, the operation must not be classified as conditional.
- `irreversible`: The operation is intentionally marked or treated as not safely reversible.
- `placeholder`: Rollback is a comment, manual instruction, or incomplete placeholder.
- `no-op`: No reverse action is expected for the operation.

## Warning Policy

- `irreversible` rollback emits a warning during standard migration generation.
- `conditional` rollback emits a warning only in verbose mode.
- `placeholder` rollback is a hard generation error by default.
- Placeholder rollback comments are allowed only for explicitly irreversible or manual cases, not as a successful rollback claim.
- The committed-migration escape hatch is an explicit annotation: `-- dbwarden: irreversible`.
- `--allow-placeholder-rollback` is intentionally not the primary escape hatch and should be treated only as a future local-development convenience.

## ClickHouse Recreate Policy

ClickHouse table recreation uses a detect-and-refuse rollback policy. Unknown means unsafe.

| Case | Classification | Behavior |
| --- | --- | --- |
| Structurally lossy transitions | irreversible | Engines such as `ReplacingMergeTree`, `SummingMergeTree`, `AggregatingMergeTree`, `CollapsingMergeTree`, and `VersionedCollapsingMergeTree` may collapse, deduplicate, aggregate, or otherwise lose row-level detail. Reverse recreate cannot restore data that no longer exists. |
| Structurally reversible transitions | conditional | Both engine families must be known row-preserving, such as `MergeTree`, `ReplicatedMergeTree`, `Memory`, `Log`, `TinyLog`, and `StripeLog`. Rollback emits reverse recreate SQL, and should be promoted only when covered by live round-trip tests. |
| Cannot-tell transitions | irreversible | Custom engines, unknown engines, unparsed engine behavior, or transitions whose safety depends on data are treated as irreversible. |

## PostgreSQL

| Operation | Classification | Notes |
| --- | --- | --- |
| `create_schema` | real | Rollback drops the schema. |
| `drop_schema` | real | Rollback recreates the schema. |
| `create_table` | real | Rollback drops the table. |
| `drop_table` | conditional | Recreates from captured `state_table`; strict generation fails if the prior table definition is unavailable. |
| `alter_table_comment` | real | Restores prior comment or `NULL`. |
| `add_column` | real | Rollback drops the column. |
| `drop_column` | conditional | Re-adds the column when definition exists; data is not restored. |
| `rename_column` | real | Renames back to the original column name. |
| `alter_column_type` | conditional | Emits inverse type change; safety depends on database castability and helper behavior. |
| `alter_column_nullable` | real | Restores previous nullability. |
| `alter_column_autoincrement` | real | Toggles sequence/default behavior back. |
| `alter_column_default` | real | Restores previous default from `rollback_attrs`. |
| `alter_column_comment` | real | Restores prior column comment or `NULL`. |
| `alter_pg_column_meta` | conditional | Depends on PostgreSQL column metadata helper support. |
| `alter_enum_add_value` | irreversible | PostgreSQL enum values cannot be removed directly. |
| `create_type` | real | Rollback drops enum type. |
| `drop_type` | real | Rollback recreates enum from saved values. |
| `create_domain` | real | Rollback drops domain. |
| `drop_domain` | conditional | Recreates domain when definition is present; strict generation fails if the prior definition is unavailable. |
| `create_composite_type` | real | Rollback drops composite type. |
| `drop_composite_type` | conditional | Recreates type when definition is present; strict generation fails if the prior definition is unavailable. |
| `create_function` | real | Rollback drops the created function. |
| `drop_function` | conditional | Recreates only when saved definition exists. |
| `create_event_trigger` | real | Rollback drops event trigger. |
| `drop_event_trigger` | conditional | Recreates only when full trigger info exists. |
| `create_extended_statistics` | real | Rollback drops statistics. |
| `drop_extended_statistics` | conditional | Recreates only when statistic info exists. |
| `add_index` | conditional | Diff carries inverse state; executable SQL depends on `_build_index_sql`. |
| `drop_index` | conditional | Recreates when index attributes are present. |
| `add_unique_constraint` | real | Rollback drops unique constraint. |
| `drop_unique_constraint` | real | Rollback recreates unique constraint. |
| `rename_unique_constraint` | real | Rollback renames back. |
| `add_check_constraint` | real | Rollback drops check constraint. |
| `drop_check_constraint` | real | Rollback recreates check constraint. |
| `add_foreign_key` | real | Rollback drops foreign key. |
| `drop_foreign_key` | real | Rollback recreates foreign key. |
| `add_grant` | real | Rollback revokes grant. |
| `revoke_grant` | real | Rollback grants privilege back. |
| `add_schema_grant` | real | Rollback revokes schema grant. |
| `revoke_schema_grant` | real | Rollback grants schema privilege back. |
| `alter_default_privileges` | conditional | Privilege grant/revoke is reversible when the previous privilege state is represented in the plan. |
| `create_role` | real | Rollback drops role. |
| `drop_role` | conditional | Recreates the role from attributes captured in `rollback_attrs` from the live schema or baseline snapshot. |
| `alter_role` | conditional | Restores prior role attributes captured in `rollback_attrs` from the live schema or baseline snapshot. |
| `create_sequence` | real | Rollback drops sequence. |
| `drop_sequence` | conditional | Recreates only when sequence info exists. |
| `alter_pg_rls` | real | Restores previous RLS/force setting. |
| `add_policy` | real | Rollback drops policy. |
| `drop_policy` | conditional | Recreates policy from `rollback_attrs` when available. |
| `alter_policy` | conditional | Restores prior policy from `rollback_attrs` when available. |
| `alter_pg_storage_param` | real | Restores previous table storage parameter or resets it. |
| `alter_pg_table` | conditional | Fillfactor and logged/unlogged are reversible; unsupported physical rewrites fail strict generation unless handled manually. |
| `add_exclude_constraint` | real | Rollback drops exclusion constraint. |
| `drop_exclude_constraint` | real | Rollback recreates exclusion constraint. |
| `alter_column_statistics` | real | Restores prior statistics target or resets to the PostgreSQL default target with `SET STATISTICS -1`. |
| `alter_pg_partition` | irreversible | Partition strategy changes require a table rebuild or a hand-authored migration. dbwarden refuses to claim automatic rollback. |
| `attach_partition` | real | Rollback detaches partition. |
| `detach_partition` | real | Rollback attaches partition with saved bound. |
| `rename_table` | conditional | Delegates to rename SQL helper. |
| `create_trigger` | conditional | Rollback drops trigger; create is placeholder if definition is missing. |
| `drop_trigger` | conditional | Recreates only when trigger definition exists. |
| `alter_trigger` | placeholder | Emit path is not implemented and uses comments. |
| `alter_view` | real | Drops and recreates view or materialized view with previous definition. |
| `refresh_matview` | no-op | Refresh has no stateful rollback. |

## ClickHouse

| Operation | Classification | Notes |
| --- | --- | --- |
| `alter_ch_options` | conditional | TTL/settings/order-by can emit inverse ALTERs; immutable/recreate-required keys can require manual handling. |
| `recreate_ch_table` | conditional or irreversible | Reverse recreate is emitted only when both engines are known row-preserving. Lossy or unknown engine transitions emit an irreversible rollback comment. |
| `alter_ch_column` | conditional | Emits inverse ALTERs for supported type/default/codec/TTL/materialized/alias/nullability/LowCardinality changes. |
| `modify_mv_query` | real | Restores previous materialized view SELECT. |
| `modify_mv_refresh` | conditional | Restores previous refresh when present; absent refresh is no-op, refresh removal is manual. |
| `alter_ch_dict` | conditional | Create/drop reversible when options exist; unsupported alter keys emit manual comments. |
| `create_ch_named_collection` | real | Rollback drops collection. |
| `drop_ch_named_collection` | conditional | Recreates the named collection from prior entries and overridable flags captured in `rollback_attrs`. |
| `alter_ch_named_collection` | conditional | Drops and recreates from full target/prior state so entry and overridable changes are reversible when prior state is captured. |
| `alter_ch_projection` | conditional | Add/drop/replace are reversible when prior definitions are in rollback attrs. |
| `alter_ch_skip_index` | conditional | Add/drop/replace are reversible when prior definitions are in rollback attrs. |
| `apply_data_op` | irreversible | Uses authored rollback when provided; otherwise remains explicitly irreversible because arbitrary data mutations cannot be inferred. |
| `create_ch_agg_target` | real | Rollback drops aggregate target table. |
| `drop_ch_agg_target` | conditional | Recreates from captured options; incomplete options can produce incomplete SQL. |
| `alter_ch_comment` | conditional | Restores table and column comments when prior values exist; otherwise may no-op. |
| `grant_ch_privilege` | real | Rollback revokes privilege. |
| `revoke_ch_privilege` | real | Rollback grants privilege back. |
| `create_ch_role` | real | Rollback drops role. |
| `drop_ch_role` | conditional | Recreates the role and captured settings from `rollback_attrs`. |
| `alter_ch_role` | conditional | Restores captured role settings with drop and recreate from `rollback_attrs`. |
| `create_ch_user` | real | Rollback drops user. |
| `drop_ch_user` | conditional | Recreates the user from captured auth, host, roles, default roles, and settings profile in `rollback_attrs`. |
| `alter_ch_user` | conditional | Restores captured user state with drop and recreate from `rollback_attrs`. |
| `create_ch_quota` | real | Rollback drops quota. |
| `drop_ch_quota` | conditional | Recreates quota from captured interval, limits, and role assignments in `rollback_attrs`. |
| `alter_ch_quota` | conditional | Restores captured quota state with drop and recreate from `rollback_attrs`. |
| `create_ch_row_policy` | real | Rollback drops row policy. |
| `drop_ch_row_policy` | conditional | Recreates the row policy from captured table, predicate, roles, and mode in `rollback_attrs`. |
| `alter_ch_row_policy` | conditional | Restores the row policy from prior state using drop and recreate from `rollback_attrs`. |
| `create_ch_settings_profile` | real | Rollback drops settings profile. |
| `drop_ch_settings_profile` | conditional | Recreates settings profile from captured settings and role assignments in `rollback_attrs`. |
| `alter_ch_settings_profile` | conditional | Restores prior settings and role assignments from `rollback_attrs`. |

## Manual and Irreversible Boundaries

These cases are not open rollback gaps. They are explicit policy boundaries where dbwarden either emits no rollback because no schema state changes, requires authored SQL, or marks the operation irreversible.

| Case | Policy |
| --- | --- |
| PostgreSQL partition strategy changes | Require a table rebuild or hand-authored migration. dbwarden refuses automatic rollback because changing partition strategy is not a metadata-only inverse. |
| PostgreSQL `REFRESH MATERIALIZED VIEW` | Rollback is intentionally no-op because refresh does not change schema definition. |
| PostgreSQL enum value additions | Irreversible in PostgreSQL because enum values cannot be removed directly. |
| ClickHouse data operations | Require authored rollback SQL when a reverse data operation exists; otherwise they remain explicitly irreversible. |
| ClickHouse engine transitions | Conditional reverse recreate is allowed only for known row-preserving engine families. Lossy and unknown transitions are irreversible by policy. |

For ordinary generated drop and alter operations, prior state is captured at generation time from the live schema snapshot or from the baseline snapshot used by offline and squash workflows. The captured definition is serialized through `rollback_attrs`, preserved in the migration plan, and consumed by the backend handler that emits rollback SQL. If the required prior state is missing, strict generation fails instead of accepting placeholder rollback.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/rollback-generation/
========================================================================

# Rollback Generation

dbwarden generates rollback SQL at the same time it generates upgrade SQL. Rollback is not a separate best-effort file that developers maintain by hand after the fact.

The current rollback system uses a strict contract:

- Executable rollback SQL is accepted.
- Conditional rollback is accepted only when dbwarden has captured enough prior state.
- Irreversible operations must be explicitly acknowledged.
- Placeholder rollback is refused by default.

See the [Rollback Coverage Matrix](rollback-coverage-matrix.md) for backend-specific operation coverage.

## Core Principle

For every upgrade operation, the diff engine computes the reverse operation at the same time.

```text
Diff detects model change
  |
  v
Upgrade operation + rollback operation
  |
  v
SQL generation
  |
  v
Migration file with upgrade and rollback sections
```

This keeps upgrade and rollback logic synchronized. The same handler that knows how to create an object also knows what information is needed to remove or restore it.

## Symmetric Pairs

Many operations have direct structural inverses.

| Upgrade | Rollback |
|---------|----------|
| `CREATE TABLE` | `DROP TABLE` |
| `ADD COLUMN` | `DROP COLUMN` |
| `CREATE INDEX` | `DROP INDEX` |
| `ADD CONSTRAINT` | `DROP CONSTRAINT` |
| `CREATE POLICY` | `DROP POLICY` |
| `CREATE ROLE` | `DROP ROLE` |

Example:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

This rollback removes the column introduced by the upgrade. If the upgrade is applied and then rolled back immediately, the table shape returns to its previous form.

## Conditional Rollback

Some operations are reversible only if dbwarden captured previous state.

Example: changing a PostgreSQL column default.

```sql
-- upgrade
ALTER TABLE users ALTER COLUMN status SET DEFAULT 'active';

-- rollback
ALTER TABLE users ALTER COLUMN status SET DEFAULT 'pending';
```

The rollback is correct only if dbwarden knows the previous default was `'pending'`. If the previous default was unknown, dbwarden must not invent one.

Conditional rollback appears in operations such as:

- Restoring prior PostgreSQL role attributes.
- Restoring prior PostgreSQL policy definitions.
- Restoring prior ClickHouse RBAC object state.
- Reversing a ClickHouse table recreate only when the engine transition is classified as row-preserving and the prior definition is available.

## Irreversible Operations

Some operations cannot be automatically reversed in a way that restores data.

Example:

```sql
-- upgrade
ALTER TABLE users DROP COLUMN legacy_code;
```

dbwarden can add `legacy_code` back if it knows the column definition, but it cannot reconstruct the deleted values. That means a structural rollback is not the same as data recovery.

For generated migrations, dbwarden refuses placeholder rollback by default. If a migration is intentionally irreversible, it must be explicit.

Use the committed migration annotation:

```sql
-- dbwarden: irreversible
```

That annotation tells reviewers and automation that the migration is intentionally not automatically reversible.

## Rollback Ordering

Rollback statements run in the reverse of upgrade order.

Upgrade order:

```text
1. Add column
2. Create index on the new column
3. Add constraint that uses the new column
```

Rollback order:

```text
1. Drop constraint
2. Drop index
3. Drop column
```

This matters because dependencies point in the opposite direction during rollback. You cannot drop a column while an index or constraint still depends on it.

## Complex Example

Model change:

- Add `display_name` to `users`.
- Drop an old index.
- Add a new uniqueness constraint.

Generated migration:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

DROP INDEX IF EXISTS idx_users_name;

ALTER TABLE users ADD CONSTRAINT uq_users_display_name UNIQUE (display_name);

-- rollback
ALTER TABLE users DROP CONSTRAINT IF EXISTS uq_users_display_name;

CREATE INDEX idx_users_name ON users (name);

ALTER TABLE users DROP COLUMN display_name;
```

Why this rollback order is correct:

1. The constraint must be removed before the column can be dropped.
2. The old index is recreated before the rollback finishes because it existed before the upgrade.
3. The added column is removed last after dependencies are gone.

## Manual SQL and Data Changes

dbwarden can generate rollback for schema operations it understands. It cannot infer the inverse of arbitrary manual data changes.

Example:

```sql
-- upgrade
UPDATE users SET status = 'active' WHERE status IS NULL;
```

The inverse is not generally knowable. Which rows were originally `NULL`? Which rows were already `active`? Without captured data, rollback cannot answer that.

For manual data changes, write a manual rollback section or explicitly mark the migration irreversible if rollback cannot be made correct.

## Rollback Convergence Pattern

Teams that want stronger rollback assurance can add a rollback convergence job:

```text
Empty database
  |
  v
Apply all migrations
  |
  v
Rollback all versioned migrations
  |
  v
Verify expected empty or baseline schema
```

This does not prove that dropped data can be recovered. It proves that rollback SQL is syntactically valid and structurally consistent for the schema objects under test.

## Why Rollbacks Should Not Be Hand-Maintained

Hand-written rollback sections drift. A developer changes an upgrade statement and forgets to update the rollback. A reviewer checks the upgrade and misses the reverse path.

dbwarden avoids that by computing both directions from the same operation model. When rollback cannot be computed safely, the generator refuses placeholder rollback rather than producing a misleading comment.

The result is stricter than convenience-oriented migration tools: either dbwarden emits executable rollback SQL, or the migration explicitly declares why automatic rollback is not available.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/round-trip-verification/
========================================================================

# Round-Trip Verification

Round-trip verification checks whether dbwarden can extract a database schema, represent it as models, generate SQL from those models, and arrive back at the same schema.

It is a consistency check between extractors, model metadata, diffing, and emitters.

## What Round-Tripping Means

The loop is:

```text
Existing database
  |
  v
dbwarden generate-models
  |
  v
SQLAlchemy models with backend-specific Meta
  |
  v
dbwarden make-migrations
  |
  v
Generated SQL migration
  |
  v
Apply migration to another empty database
  |
  v
Extract schema again
  |
  v
Compare extracted schema with original shape
```

If the second extraction matches the first, the handler round trip is stable for those object types.

## Why It Matters

Every supported database has features that are not visible in simple `CREATE TABLE` syntax.

PostgreSQL examples:

- Identity column options
- Generated columns
- Collation
- Per-column storage and compression
- Table storage parameters
- Exclusion constraints
- Row-level security policies
- Materialized view settings

ClickHouse examples:

- `MergeTree` engine families
- `ORDER BY`, `PRIMARY KEY`, `PARTITION BY`, and `SAMPLE BY`
- Engine settings
- Projections
- Skip indexes
- Materialized views
- Dictionaries
- RBAC objects

Round-trip verification proves that dbwarden does not lose those details when moving between database state, model state, and SQL.

## It Is Not a Full Mathematical Proof

Round-tripping is strong evidence, not a complete proof of all possible database behavior.

It proves internal consistency for supported schema objects. It does not prove:

- Application data is preserved.
- Manually written SQL has an inverse.
- Runtime behavior such as locks, query plans, replication lag, or trigger side effects is harmless.
- A database extension behaves identically across all versions.

That is why round-trip verification complements the [Convergence Gate](convergence-gate.md). Round-trip verification checks extractor and emitter consistency. The convergence gate checks that the repository's migration history reproduces the current model state.

## PostgreSQL Example

Start with a PostgreSQL schema that uses backend-specific features:

```sql
CREATE TABLE accounts (
    id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    tenant_id integer NOT NULL,
    email character varying(255) NOT NULL,
    created_at timestamp with time zone DEFAULT now() NOT NULL
) WITH (fillfactor = 80);

CREATE INDEX idx_accounts_tenant_email
    ON accounts USING btree (tenant_id, email);
```

dbwarden extracts the schema:

```bash
dbwarden generate-models --database primary --output generated_models.py
```

The generated model should preserve the meaningful PostgreSQL metadata:

```python
class Account(Base):
    __tablename__ = "accounts"

    id: Mapped[int] = mapped_column(BigInteger, primary_key=True)
    tenant_id: Mapped[int] = mapped_column(Integer, nullable=False)
    email: Mapped[str] = mapped_column(String(255), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)

    class Meta(PGTableMeta):
        pg_storage_params = {"fillfactor": 80}
        indexes = [
            IndexSpec(
                name="idx_accounts_tenant_email",
                columns=["tenant_id", "email"],
            ),
        ]
```

The exact generated Python may differ by import style, but the important part is that schema-relevant metadata is represented. The identity column, storage parameters, and index are not discarded.

Then generate migrations from that model against an empty database:

```bash
dbwarden make-migrations "recreate accounts" --database primary
dbwarden migrate --database primary
dbwarden diff --database primary --out table
```

The expected diff is empty. If dbwarden emits a migration every time from the generated model, some part of extraction, canonicalization, or emission is losing information.

Known PostgreSQL gaps: `SERIAL` / identity column sequence metadata may not fully round-trip through generated SQLAlchemy models, producing spurious `DROP DEFAULT` / `DROP SEQUENCE` operations. Mixed-case or quoted identifiers are now handled, but some index sort-option canonicalization (e.g. default `NULLS LAST`) can still produce noise.

## ClickHouse Example

Start with a ClickHouse table that uses a `MergeTree` engine, partitioning, a projection, and a skip index:

```sql
CREATE TABLE events (
    id UInt64,
    event_date Date,
    user_id UInt64,
    path String,
    INDEX idx_path path TYPE bloom_filter(0.01) GRANULARITY 64,
    PROJECTION by_date (
        SELECT event_date, count()
        GROUP BY event_date
    )
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_date, id)
SETTINGS index_granularity = 8192;
```

The generated model must preserve engine and table metadata:

```python
class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    user_id: Mapped[int] = mapped_column()
    path: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            partition_by="toYYYYMM(event_date)",
            order_by=["event_date", "id"],
            settings={"index_granularity": "8192"},
            projections=[
                ProjectionSpec(
                    name="by_date",
                    query="SELECT event_date, count() GROUP BY event_date",
                ),
            ],
            indexes=[
                ChIndexSpec(
                    name="idx_path",
                    columns=["path"],
                    type="bloom_filter(0.01)",
                    granularity=64,
                ),
            ],
        )
```

The verification flow is the same:

```bash
dbwarden generate-models --database analytics --output generated_analytics.py
dbwarden make-migrations "recreate clickhouse schema" --database analytics
dbwarden migrate --database analytics
dbwarden diff --database analytics --out table
```

The diff should be empty. If the engine settings, projection, or skip index are missing after extraction, the round trip fails and the handler needs a fix.

Known ClickHouse gaps: materialized views are not emitted as discoverable model tables; some native types (`Enum8`/`Enum16`, `DateTime64`, `FixedString`, `UUID`) may map to generic SQLAlchemy types; and Replicated engine string arguments / MV `SETTINGS` require manual review.

## Manual Round-Trip Checklist

Use this checklist when validating a backend feature:

1. Create the object directly in a disposable database.
2. Run `dbwarden generate-models` for that database.
3. Inspect the generated `class Meta` block.
4. Apply the generated model to an empty disposable database with `make-migrations` and `migrate`.
5. Run `dbwarden diff` and confirm no schema drift remains.
6. Add a test for the feature so future changes cannot regress it.

## What Round-Trip Verification Does Not Cover

Round-trip verification does not cover data migration semantics. A table can round-trip perfectly while a manual `UPDATE` statement still needs human review.

It also does not replace backend integration tests. For example, ClickHouse may accept syntax but behave differently depending on engine family. PostgreSQL may require a lock level that matters in production. Those concerns belong in safety review and integration testing.

Use round-trip verification to prove schema representation fidelity. Use the [Convergence Gate](convergence-gate.md) to prove the repository history lands on the declared model state.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/safety-classifier/
========================================================================

# Safety Classifier

The safety classifier detects risky schema changes before migration execution. It reads the migration plan, classifies each operation, and forces the operator to acknowledge changes that may affect data or availability.

The principle is simple: dbwarden should not silently drop data.

The shared classifier now feeds generated plans, static SQL adoption, and deployment scope. Canonical levels are SAFE, INFO, WARN, CRITICAL, and UNKNOWN. Legacy plan fields keep INFO/WARNING/ERROR. A missing mapping remains UNKNOWN. See [Safety-scoped migrations](safety-scoped-migrations.md) for split closure, ceiling behavior, parser coverage, and plan trust rules.

## What It Scans

`make-migrations` writes a companion `.plan.json` file next to generated SQL migrations. The plan records the typed operations that produced the SQL. A safety check can inspect that plan before the SQL is applied.

The flow is:

```text
Models or manual migration request
  |
  v
Diff engine
  |
  v
Typed operations
  |
  v
.plan.json
  |
  v
Safety classifier
  |
  v
Safe, warning, or blocking result
```

Run the check command before applying migrations:

```bash
dbwarden check --database primary
```

If the command reports warnings that require acknowledgement, use the documented command option only after reviewing the plan:

```bash
dbwarden check --database primary --force
```

`--force` is an acknowledgement. It should not be used as a default in CI.

## Severity Levels

dbwarden safety classifications map to three operational meanings.

### Info

Info-level changes are expected to be safe from a schema perspective.

Examples:

- Create a table.
- Create an index.
- Add a nullable column.
- Add a column with a safe default on an empty table.

Example operation:

```sql
ALTER TABLE users ADD COLUMN nickname VARCHAR(255);
```

Why it is low risk:

- Existing rows remain valid because the column is nullable.
- No data is removed.
- The operation is visible in the migration file and plan.

### Warning

Warning-level changes may be valid, but they need human review.

Examples:

- Add `NOT NULL` to a column when existing data may violate it.
- Change a column type where conversion may fail.
- Change storage parameters or backend-specific physical settings.
- Recreate a ClickHouse table for an engine transition that is classified as row-preserving but still operationally sensitive.

Example operation:

```sql
ALTER TABLE users ALTER COLUMN status SET NOT NULL;
```

Questions to answer before allowing it:

- Are there existing rows with `NULL` in `status`?
- Does the migration include a backfill if needed?
- Does the backend require a lock that could affect production traffic?

### Blocking

Blocking changes are destructive or ambiguous enough that they must not be hidden inside an ordinary migration review.

These changes are classified as `ERROR` severity in the safety check output.

Examples:

- Drop a table.
- Drop a column.
- Rename a table without explicit rename intent.
- Change a ClickHouse engine in a way that can collapse, aggregate, or otherwise lose row-level data.

Example operation:

```sql
DROP TABLE audit_log;
```

Why it blocks:

- The table data is removed.
- Rollback may recreate the table shape, but it cannot recover deleted rows.
- The operator must explicitly confirm this is intended.

## Examples by Severity

### Info: Add a Nullable Column

Model change:

```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(primary_key=True)
    display_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
```

Generated SQL:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

The upgrade does not invalidate existing rows. The rollback removes the newly added column.

### Warning: Add NOT NULL

Model change:

```python
status: Mapped[str] = mapped_column(String(32), nullable=False)
```

Generated SQL may include:

```sql
ALTER TABLE users ALTER COLUMN status SET NOT NULL;
```

This is structurally valid but operationally risky if existing rows contain `NULL`. The safe approach is to backfill first, verify, then enforce `NOT NULL`.

### Blocking: Drop a Column

Model change:

```python
# legacy_code was removed from the model
```

Generated SQL may include:

```sql
ALTER TABLE users DROP COLUMN legacy_code;
```

This removes stored data. A rollback can add the column back, but it cannot reconstruct the dropped values. The plan should be reviewed before applying.

## How It Fits in CI

A practical CI sequence is:

```text
Generate migration
  |
  v
Inspect plan and safety classification
  |
  v
Reject unexpected warnings or blocking changes
  |
  v
Apply to ephemeral database
  |
  v
Run convergence gate
```

The safety check runs before the convergence gate. Safety asks whether the operation is acceptable. Convergence asks whether the resulting schema is correct.

See [Convergence Gate](convergence-gate.md).

## Philosophy

dbwarden does not try to guess business intent. It can tell that a column drop destroys stored values, but it cannot know whether those values are obsolete. It can identify that a ClickHouse engine transition is lossy, but it cannot know whether the application already copied the data elsewhere.

For that reason, the safety classifier makes risk visible and requires acknowledgement. It is a correctness mechanism because it prevents accidental data loss from being treated as ordinary schema drift.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/safety-scoped-migrations/
========================================================================

# Safety-scoped migrations

Generate every intended change, then choose which files may run during a deployment.
Severity controls scheduling. It does not prove that SQL is safe for production data,
replace backups, or remove acknowledgement requirements.

## Generate and deploy

```bash
dbwarden make-migrations --database primary --split-at-severity WARN
dbwarden migrate --database primary --max-severity INFO
# Later, after reviewing deferred SQL:
dbwarden migrate --database primary --max-severity CRITICAL --force
```

The first command produces consecutive versioned files when both groups contain
operations:

```text
primary__0007_update_users.sql
primary__0008_update_users__deferred.sql
```

The base contains operations below the split threshold. The deferred file contains
operations at or above it and every operation that depends on them. Each file has
its own upgrade, rollback, and plan. If either group is empty, only the other file
is written. A threshold of SAFE therefore puts every operation into one deferred
file. Repeatables cannot be split; combining a split threshold with `--type ra`
or `--type roc` is a usage error.

The partition preserves statement ordering, checks explicit operation dependencies,
and adds dependencies for table/column changes, indexes, constraints, foreign keys,
schemas, and view references. Cycles, unknown dependency IDs, unsupported emitters,
and dependencies incompatible with statement ordering fail before writing files.
Plugins must declare dependencies that cannot be inferred from these attributes.

## One severity model

Generation and static classification call
`dbwarden.engine.safety.classifiers.classify_operation`. Execution and status read
the resulting plan; they do not classify SQL at deployment time.

| Level | Examples |
|---|---|
| SAFE | Read-only SELECT; PostgreSQL type changes classified SAFE by its type table |
| INFO | Create table; nullable column without volatile default; concurrent PostgreSQL index |
| WARN | Rename; SET NOT NULL; non-concurrent PostgreSQL index; unbounded DML |
| CRITICAL | Drop table/column; TRUNCATE; lossy ClickHouse rebuild; safe-type contract |
| UNKNOWN | Missing, malformed, stale, unsupported, or inconsistent plan |

PostgreSQL narrowing and cross-family conversions use its existing type classifier.
ClickHouse column/option changes use its backend tables. Bounded DML classification
recognizes a column equality or range predicate structurally. It cannot prove that
the column is indexed, unique, or selective.

Existing `operations[].severity` values remain INFO, WARNING, and ERROR for legacy
readers. The new `severity` block uses SAFE, INFO, WARN, CRITICAL, and UNKNOWN.
Unsupported operation kinds never default to INFO.

## Plans and effective state

Plans retain legacy fields and add:

```json
{
  "schema_version": "1.1",
  "backend": "postgresql",
  "content_hash": "sha256:...",
  "base_checksum": "...",
  "target_checksum": "...",
  "upgrade_ops": [],
  "severity": {
    "file": "WARN",
    "provenance": "generated",
    "split": {"threshold": "WARN", "role": "deferred", "paired_with": "0007"},
    "ops": [{"id": "op:1", "kind": "alter_column_nullable", "severity": "WARN"}]
  },
  "required_flags": ["--force"]
}
```

The example abbreviates typed operations. Generated operations contain full emitter
attributes, rollback attributes where required, dependencies, and checked state
changes. IDs remain stable across both files. The content hash covers the complete
SQL file with line endings normalized to LF. Editing headers, upgrade, or rollback
invalidates trust. State checksums exclude timestamps and generation bookkeeping.

Generation starts from the latest applied snapshot, then composes pending typed
plans in version order. It excludes applied and superseded files. Config-defined
plugin objects participate through their registered handlers. A reconciliation
plan participates once as the replacement for the superseded branch files.
The legacy SQL preview helper retains a warned fallback for old planless files;
normal generation never uses SQL replay to compose pending state.

A pending deferred change is not emitted again. Deleting a pending deferred file
and its plan lets generation emit the missing change again. Deleting its prerequisite
while keeping the deferred plan causes a stale-base error. A model edit that
contradicts pending deferred state requires applying or superseding that file first.

Planless pending files warn that effective state is incomplete. `--strict-pending`
refuses them. A trusted plan whose base state no longer matches fails closed in
both modes. Generated model state stores the generation baseline separately from
the desired target; generation does not claim an unapplied target is an applied
database snapshot.

## Execution and exit codes

Versioned migrations run as a strict prefix. The first file above the ceiling
stops execution before its SQL. Later versions and repeatables do not run.
Applied prefix files retain their history entries. The existing lock remains held
through the partial run and releases through normal cleanup.

| Exit | Meaning |
|---|---|
| 0 | Completed, no pending work, or successful dry run |
| 3 | Successful versioned prefix followed by severity deferral |
| 4 | `check --write-plan` left unresolved files |
| Other existing nonzero codes | SQL, lock, checksum, configuration, or policy failures |

The existing optional-database skip result also uses exit 3. With `--all`, JSON
distinguishes skipped databases and severity deferrals. Other databases continue
after a deferral; real failures take precedence.

`--count` and `--to-version` bound the candidate set before the ceiling.
`--force` acknowledges operation risks but never raises the ceiling.
`--dry-run` reports the stop without DDL or a deferred exit code.
`--baseline` records history only, without executing SQL or applying a ceiling.
Backup and sandbox options retain their normal execution paths.

Repeatables above the ceiling warn on every skipped run. Eligible repeatables
run normally if no versioned file caused a stop. Changed-only repeatables that
have not changed remain outside the candidate set.

UNKNOWN blocks under every ceiling, including the default CRITICAL ceiling:
a file whose plan resolves to explicit UNKNOWN (parse failure, tamper
evidence, tagged UNKNOWN) stops with exit 3. The only exception is a file
with no plan at all, which is governed by the missing_plan preflight policy
and passes at the default ceiling. Applied historical files are not
reclassified. There is no per-file severity override.

## Adopt existing SQL

```bash
dbwarden check --write-plan --all --database primary
dbwarden check --write-plan 0008 --database primary
```

Classification is static and whole-file. Every upgrade statement must map before
a trusted static plan is written. Generated plans are never overwritten. Matching
trusted plans are unchanged; stale static plans are reclassified. Review the
CRITICAL list and resolve the UNKNOWN list before enabling a capped deployment.

PostgreSQL syntax is validated by pglast/libpg_query (PostgreSQL 18 grammar).
SQLGlot supplies dialect ASTs for statement mapping and for MySQL, MariaDB,
ClickHouse, and SQLite. Both dependencies are pinned to reviewed version ranges.
They replace regex-based risk guesses; neither executes SQL or connects to a database.

| Engine | Grammar coverage | Classification boundary |
|---|---|---|
| PostgreSQL | Native full statement grammar | Conservative mapped subset; unmapped ASTs remain UNKNOWN |
| MySQL / MariaDB | Core DDL/DML, grade B target | Exotic ALTER options and executable comments remain UNKNOWN |
| ClickHouse | DDL/DML dialect, grade B target | Unsupported engine/ALTER forms remain UNKNOWN |
| SQLite | Minimal ALTER surface, grade C target | Unsupported rebuild scripts and statements remain UNKNOWN |

Parser grammar coverage does not claim complete classification coverage. DO blocks,
dynamic SQL, function/procedure bodies, opaque calls, and nested data modification
remain unresolved. Unsupported syntax and SQLGlot fallback Command nodes fail closed.
Static type alterations without previous type metadata also remain UNKNOWN.

## Safe type changes

`--safe-type-change --split-at-severity WARN` expands with a nullable temporary
column in the base and defers the contract. The base includes a backfill instruction.
Backfill and application dual writes remain operator work.

The contract locks the table, verifies populated temporary values and lossless
round trips, then drops/renames and restores defaults, nullability, and comments.
Its rollback recreates and repopulates the old column. Columns with keys, indexes,
constraints, identity/generated metadata, or unsupported backend semantics are
refused before writing; those require an explicit migration preserving dependencies.
This workflow does not itself guarantee zero downtime.

## Merge and reconciliation

`merge --dry-run` computes the same artifacts without writing or superseding files.
Merge checks hand edits before writing, preserves colliding branch filenames for
audit, and routes reconciliation through the shared typed writer and splitter.
Records include the merge-base commit, version and checksum, superseded filenames
and versions, all reconciliation versions, and per-environment status.

Dirty persistent environments use `reconcile`, which generates real SQL from live
state to the merged target. It uses the migration runner, verifies convergence,
records the normal reconciliation versions as satisfied, and updates the environment
record while the existing runner lock is held. `rebase --dry-run` inspects disposable
recovery; non-interactive mutation requires `--yes`.

## Status and observability

Plugins can register named migration groups alongside base/deferred and assign
operations or emitted statements to them. Group order controls file allocation;
dependencies and severity thresholds can move operations later. Custom groups do
not add severity levels or change execution ceilings. Plans record category
metadata; `status` exposes the category in table and JSON output. See
[plugin API](../plugins/developing/object-plugins.md#safety-and-migration-categories).

`status` displays plan severity, deferred files, and later blocked files. JSON
includes the blocking version, trust reason, and deferred age in seconds. Age is
based on SQL file mtime; copying files or checking out a repository can reset it.

Structured logs emit `migration_deferred` with database, ceiling, stopping file,
file severity, and remaining pending count. Repeatable skips emit
`repeatable_skipped_severity`. Optional Prometheus metrics record deferral counts
and per-version deferred age. A severity stop does not invoke failure hooks.

## Configuration

```python
database_config(
    database_name="primary",
    database_type="postgresql",
    database_url_sync="postgresql://localhost/app",
    split_at_severity=None,
    max_severity="CRITICAL",
    strict_pending=False,
)
```

These exact defaults keep splitting off and execution uncapped. CLI values override
database settings; `--no-strict-pending` disables a configured strict-pending policy.
UNKNOWN is not a selectable threshold or ceiling.

========================================================================
PAGE: https://docs.dbwarden.org/correctness/sql-generation/
========================================================================

# SQL Generation

SQL generation is the stage where dbwarden turns a typed diff into a plain SQL migration file. The generated file contains backend-native SQL and is meant to be reviewed by humans.

There is no hidden runtime library inside the migration. Once generated, the migration is SQL.

## Pipeline

```text
Canonical model state + canonical snapshot state
  |
  v
Diff
  |
  v
Typed operations
  |
  v
Statement ordering
  |
  v
Backend ObjectHandler.emit()
  |
  v
MigrationStatement objects
  |
  v
Plain .sql migration file
```

Each stage has a narrow job.

## 1. Diff to Typed Operations

The diff engine does not start by writing SQL strings. It creates typed operations such as:

```text
create_table
drop_table
add_column
drop_column
alter_column_type
alter_column_nullable
add_index
drop_index
alter_ch_options
recreate_ch_table
```

Typed operations make safety checks, ordering, rollback generation, and backend dispatch possible.

Example operation shape:

```json
{
  "type": "add_column",
  "table": "users",
  "column": "display_name",
  "definition": {
    "type": "VARCHAR(255)",
    "nullable": true
  }
}
```

The exact internal representation may include backend-specific metadata, but the key idea is that the operation is structured before it becomes SQL.

## 2. Operation Ordering

Not every valid operation order is safe. dbwarden uses statement ordering to make generated migrations executable.

Examples:

- Create schemas before creating tables inside them.
- Rename tables before applying column changes to the renamed table.
- Create tables before creating indexes on those tables.
- Drop dependent objects before dropping objects they depend on.
- Reverse upgrade order for rollback statements.

The ordering layer uses `StatementOrder` and backend handler constraints so SQL is emitted in a stable sequence.

## 3. Backend Dispatch

Each backend owns SQL rendering for its supported object types. The dispatch step sends operations to the relevant `ObjectHandler.emit()` implementation.

```text
Operation: add_column
  |
  +--> PostgreSQL ColumnHandler.emit()
  |
  +--> MySQL column renderer
  |
  +--> SQLite-compatible renderer
  |
  +--> ClickHouse ChColumnHandler.emit()
```

This is why the same model can produce different SQL for different database configurations.

## PostgreSQL Example: Identity Column

A PostgreSQL model can declare identity behavior through backend metadata:

```python
class Account(Base):
    __tablename__ = "accounts"

    id: Mapped[int] = mapped_column(BigInteger, primary_key=True)

    class Meta(PGTableMeta):
        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=1000)
```

The PostgreSQL emitter understands that this is not a generic integer column. It must render PostgreSQL-native identity syntax:

```sql
CREATE TABLE accounts (
    id BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1000) PRIMARY KEY
);
```

Depending on the model metadata and backend version, PostgreSQL auto-increment behavior may use identity syntax or sequence-backed behavior. The important correctness property is that the PostgreSQL handler owns that decision. It is not guessed by a generic string formatter.

## PostgreSQL Example: Add a Column

Model change:

```python
display_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
```

Typed operation:

```text
add_column users.display_name varchar(255) nullable
```

Generated SQL:

```sql
-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

-- rollback
ALTER TABLE users DROP COLUMN display_name;
```

The output is intentionally simple. Reviewers can see exactly what will run.

## ClickHouse Example: MergeTree Table

A ClickHouse model can declare engine metadata:

```python
class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    user_id: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["event_date", "id"],
            partition_by="toYYYYMM(event_date)",
            settings={"index_granularity": "8192"},
        )
```

The ClickHouse emitter writes native engine syntax:

```sql
CREATE TABLE IF NOT EXISTS events (
    id UInt64,
    event_date Date,
    user_id UInt64
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_date, id)
SETTINGS index_granularity = 8192;
```

This is not a generic SQL dialect. ClickHouse engine clauses, settings, projections, skip indexes, and materialized view syntax are rendered by ClickHouse-specific handlers.

## Example Migration File

A generated migration contains both upgrade and rollback sections:

```sql
-- migration: primary__0007_add_user_display_name.sql

-- upgrade
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

CREATE INDEX idx_users_display_name ON users (display_name);

-- rollback
DROP INDEX IF EXISTS idx_users_display_name;

ALTER TABLE users DROP COLUMN display_name;
```

This file is self-contained. Applying it does not require importing dbwarden code inside the database. dbwarden is the generator and executor. The migration artifact is plain SQL.

## Why Emitters Are Trustworthy

The trust comes from separation and tests:

- Canonicalization decides whether a change is real.
- Diffing creates typed operations.
- Safety checks inspect the plan before execution.
- Backend handlers render native SQL for one backend and object family.
- Tests cover handler output, ordering, rollback metadata, and round-trip behavior.

A generic SQL generator would need to understand every backend rule at once. dbwarden instead gives each backend handler a focused responsibility.

## Link to Deterministic Diff

SQL generation depends on deterministic diffing. If canonicalization is stable, the operation list is stable. If the operation list is stable, SQL output is reviewable.

See [Deterministic Diff](deterministic-diff.md).

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/aggregating-views/
========================================================================

# Aggregating views (`AggregatingMergeTree`)

## Overview

An aggregating view is a coherent triad:

1. An **AggregatingMergeTree target table** whose columns have
   `AggregateFunction(...)` types derived from aggregate expressions.
2. A **materialized view** that uses `<func>State(...)` combinators in its
   SELECT, `TO` the target.
3. The **source table** (referenced, not created; it must already exist).

Because the target column types and the MV combinators both derive from the
same single list of `AggExpr`, they are guaranteed consistent; the
correspondence that is manual and drift-prone in the string-SELECT form is
here derived and safe.

## Declaring aggregating views

Use `AggregatingView` as the base class, `aggregating_view()` in `Meta`:

```python
from sqlalchemy import func
from dbwarden.databases.clickhouse import AggregatingView, CHViewMeta, aggregating_view, agg

class EventsHourly(AggregatingView):
    __tablename__ = "events_hourly"

    class Meta(CHViewMeta):
        ch = aggregating_view(
            source=Event,
            group_by=[func.toStartOfHour(Event.event_time).label("hour")],
            aggregates=[
                agg.sum(Event.amount).as_("total_amount"),
                agg.count().as_("event_count"),
            ],
            order_by=["hour"],
            partition_by="toYYYYMM(hour)",
        )
```

This generates:

1. The target table `events_hourly` with `AggregatingMergeTree` engine and
   columns `hour DateTime`, `total_amount AggregateFunction(sum, Float64)`,
   `event_count AggregateFunction(count)`.
2. A materialized view `events_hourly_mv TO events_hourly` whose SELECT uses
   `sumState`, `countState`; automatically derived from the `AggExpr` list.
3. The MV reads FROM `Event`.

### The source parameter

`source` may be:

- A model class (preferred); its `__tablename__` is resolved at spec
  construction time.
- A string class name (forward reference); resolved at discovery time when
  all models are loaded.
- A bare table name string; used as-is.

When you pass a model class, rename safety is automatic: if the source table
is renamed, regeneration picks up the new name.

### Aggregate expressions

Every aggregate must have an alias (`.as_(name)`):

```python
aggregates=[
    agg.count().as_("event_count"),
    agg.sum("amount", "Float64").as_("total_amount"),
    agg.uniq("user_id", "String").as_("unique_users"),
]
```

Supported aggregate functions include `sum`, `count`, `min`, `max`, `avg`,
`uniq`, `uniq_exact`, `any`, `any_last`, `groupArray`, `groupUniqArray`,
`quantile`, and the `raw` escape hatch for combinators not enumerated above.

## Configuring the target table

The `AggregatingViewSpec` is a frozen dataclass. Configure it via
`aggregating_view()` keyword arguments:

| Parameter | Description |
|-----------|-------------|
| `source` | Source model class or table name |
| `group_by` | GROUP BY keys: ColumnElement or string |
| `aggregates` | AggExpr list (each with `.as_()`) |
| `order_by` | ORDER BY for the target |
| `partition_by` | Optional PARTITION BY |
| `ttl` | Optional TTL expression(s) |
| `settings` | Optional engine SETTINGS |

## Full example

```python
from sqlalchemy import func, String
from dbwarden.databases.clickhouse import (
    AggregatingView, CHViewMeta, aggregating_view, agg,
)

class EventStats(AggregatingView):
    __tablename__ = "event_stats"

    class Meta(CHViewMeta):
        ch = aggregating_view(
            source=PageView,
            group_by=[
                PageView.url.label("url"),
                func.toDate(PageView.viewed_at).label("day"),
            ],
            aggregates=[
                agg.count().as_("views"),
                agg.uniq(PageView.session_id, "String").as_("unique_sessions"),
                agg.sum(PageView.duration).as_("total_duration"),
            ],
            order_by=["url", "day"],
            partition_by="toYYYYMM(day)",
            ttl="day + INTERVAL 90 DAY DELETE",
        )
```

## Populating an aggregating view

Use the `populate()` helper from `data_ops` to generate an `INSERT ... SELECT`
DataOp that backfills the target table:

```python
from dbwarden.databases.clickhouse import data_ops, AggregatingView

pop = data_ops.populate(EventStats.Meta.ch)
```

This produces a `DataOp` whose forward SQL is equivalent to:

```sql
INSERT INTO event_stats
SELECT
    url,
    toDate(viewed_at) AS day,
    countState() AS views,
    uniqState(session_id) AS unique_sessions,
    sumState(duration) AS total_duration
FROM page_view
GROUP BY url, toDate(viewed_at)
```

## Discoverability

`AggregatingView` subclasses are automatically registered in
`ChView._ch_view_registry` and discovered by `ch_view_tables_from_models()`.
They contribute both the aggregating target model and the materialized view to
the model list.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/alter-semantics/
========================================================================

# ALTER semantics

ClickHouse ALTERs behave differently from every other database dbwarden supports. Understanding these differences prevents production failures that are hard to diagnose.

## Why ClickHouse ALTERs are special

Unlike PostgreSQL or MySQL, ClickHouse ALTERs have three properties that affect how migrations must be structured:

1. **Non-transactional DDL.** DDL statements (CREATE, ALTER, DROP) cannot be rolled back. If a migration fails midway, there is no automatic undo. dbwarden's recreate pipeline handles this for critical changes, but metadata-only ALTERs are still individual commits.

2. **ZooKeeper-based replication.** On replicated tables, ALTER statements add instructions to ZooKeeper. Replicas pick them up asynchronously. The ALTER statement returns before replicas have applied the change.

3. **Async mutations.** Data-mutating ALTERs (UPDATE, DELETE, MATERIALIZE INDEX) are background processes that rewrite entire data parts. They are not instant and do not block inserts.

These properties mean that the *shape* of your ALTER statements matters. Two separate `ALTER TABLE` statements are not equivalent to one combined statement, even if the SQL looks equivalent.

## CANNOT_ASSIGN_ALTER (Code 517)

### What it is

On ReplicatedMergeTree tables, ClickHouse rejects concurrent ALTER mutations on the same table with:

```
Code: 517. DB::Exception: CANNOT_ASSIGN_ALTER
```

The error message says: "Metadata on replica is not up to date with common metadata in Zookeeper. Cannot alter."

### When it fires

The race window is between two separate `ALTER TABLE` statements against the same table:

```sql
-- Statement 1: reaches ClickHouse, begins processing
ALTER TABLE events ADD INDEX IF NOT EXISTS ix_a col_a TYPE minmax GRANULARITY 100;

-- Statement 2: arrives before Statement 1's metadata has been applied to all replicas
ALTER TABLE events ADD INDEX IF NOT EXISTS ix_b col_b TYPE minmax GRANULARITY 100;
-- ^ Code: 517 CANNOT_ASSIGN_ALTER
```

Even with `mutations_sync=2` or `alter_sync=2`, the next HTTP request can arrive before the previous ALTER has cleared the mutation slot. The ClickHouse docs state:

> On replicated tables, submitting several separate `ALTER` statements against the same table in quick succession can fail with `CANNOT_ASSIGN_ALTER` (code 517). The replicated path raises this when the replica has not yet applied some previous `ALTER`s.

### How to avoid it

**Combine compatible subcommands into a single ALTER TABLE statement:**

```sql
-- Wrong: two separate statements, race-prone
ALTER TABLE events ADD INDEX IF NOT EXISTS ix_a col_a TYPE minmax GRANULARITY 100;
ALTER TABLE events ADD INDEX IF NOT EXISTS ix_b col_b TYPE minmax GRANULARITY 100;

-- Right: one combined statement, no race
ALTER TABLE events
    ADD INDEX IF NOT EXISTS ix_a col_a TYPE minmax GRANULARITY 100,
    ADD INDEX IF NOT EXISTS ix_b col_b TYPE minmax GRANULARITY 100;
```

For operations that cannot be combined (see [ALTER batching](#alter-batching) below), serialize them and retry on Code 517 until the previous ALTER has been applied.

### How dbwarden handles it

dbwarden's handlers batch most ALTER subcommands into a single `MigrationStatement`. However, there is a known limitation: **`MODIFY SETTING` is emitted as one statement per setting change** (see [MODIFY SETTING](#modify-setting)). On replicated tables with multiple setting changes, this can trigger Code 517.

See the [operation catalog](#operation-catalog) for per-handler batching behavior.

## mutations_sync vs alter_sync

ClickHouse has **two distinct settings** that control ALTER synchronicity. They apply to different categories of operations.

### Which setting applies to which operations

| Category | Setting | Operations |
|----------|---------|------------|
| **Metadata-only** | `alter_sync` | ADD/DROP/MODIFY COLUMN, ADD/DROP INDEX, ADD/DROP PROJECTION, MODIFY SETTING, MODIFY TTL, REMOVE TTL, MODIFY ORDER BY, COMMENT, RENAME TABLE |
| **Data mutations** | `mutations_sync` | UPDATE, DELETE, MATERIALIZE INDEX, MATERIALIZE PROJECTION, MATERIALIZE COLUMN, APPLY DELETED MASK, APPLY PATCHES, CLEAR STATISTIC, MATERIALIZE STATISTIC |

The distinction matters because:
- Metadata ALTERs propagate via ZooKeeper metadata entries (fast, small)
- Mutations rewrite data parts (slow, heavy I/O)

### Setting values

| Value | alter_sync | mutations_sync |
|-------|-----------|----------------|
| 0 | Do not wait | Do not wait |
| 1 | Wait for own execution | Wait for own execution |
| 2 | Wait for all replicas | Wait for all replicas |
| 3 | Wait for active replicas | **Bug: see below** |

Both settings apply only to Replicated and SharedMergeTree tables. Non-replicated tables always execute synchronously.

### Default behavior

- `alter_sync` defaults to `0` (do not wait) in ClickHouse Cloud, `2` in self-hosted
- `mutations_sync` defaults to `0` (do not wait)

For migrations, the safe default is `alter_sync=2` and `mutations_sync=2` when you need confirmation that changes have propagated.

### Setting them in dbwarden

dbwarden does not set these automatically. You control them through `data_op()`:

```python
from dbwarden.databases.clickhouse import data_op

# Metadata ALTER with wait
data_op(
    name="add_index_with_sync",
    forward="ALTER TABLE events ADD INDEX ix_a col_a TYPE minmax GRANULARITY 100 SETTINGS alter_sync = 2",
)

# Mutation with wait
data_op(
    name="delete_old_events_with_sync",
    forward="ALTER TABLE events DELETE WHERE event_date < '2020-01-01' SETTINGS mutations_sync = 2",
)
```

Or at the session level before running `dbwarden migrate`:

```sql
SET mutations_sync = 2;
SET alter_sync = 2;
```

## The mutations_sync=3 bug

!!! warning "Do not use mutations_sync=3 on ReplicatedMergeTree"

    On ReplicatedMergeTree, `mutations_sync=3` and `alter_sync=3` **silently do nothing**; the statement returns immediately without waiting, behaving like value `0`.

### What happens

The documented contract for value `3` is:

> The query waits only for active replicas. Supported only for SharedMergeTree. For ReplicatedMergeTree it behaves the same as mutations_sync = 2.

But in the C++ implementation, `StorageReplicatedMergeTree::waitMutation` only branches on `== 1` and `== 2`. Value `3` falls through both branches, leaving the replica list empty, and `waitMutationToFinishOnReplicas` returns immediately on an empty list.

### Impact

A user who sets `mutations_sync=3` gets no error and no wait; the statement returns as if it succeeded synchronously, and scripts proceed on the assumption that the mutation has been applied. This is a silent loss of a synchronization guarantee.

### Safe values

On ReplicatedMergeTree, use only `0` or `2`:

```sql
-- Safe
ALTER TABLE t DELETE WHERE ... SETTINGS mutations_sync = 2;

-- Unsafe (silently does nothing on ReplicatedMergeTree)
ALTER TABLE t DELETE WHERE ... SETTINGS mutations_sync = 3;
```

Value `3` is only safe on SharedMergeTree.

See [ClickHouse issue #117395](https://github.com/ClickHouse/clickHouse/issues/117395) for tracking.

## MODIFY SETTING

### Batching requirement

Multiple settings should be combined into a **single** `ALTER TABLE ... MODIFY SETTING` statement:

```sql
-- Wrong: two separate statements, race-prone on ReplicatedMergeTree
ALTER TABLE events MODIFY SETTING index_granularity = 4096;
ALTER TABLE events MODIFY SETTING max_bytes_to_merge_at_max_space_in_pool = 10000000000;

-- Right: one combined statement
ALTER TABLE events MODIFY SETTING
    index_granularity = 4096,
    max_bytes_to_merge_at_max_space_in_pool = 10000000000;
```

### dbwarden limitation

dbwarden currently emits **one `MODIFY SETTING` per changed setting** in `ch_table_handler.py`:

```python
for setting_key, setting_value in to_val.items():
    suffix = f"MODIFY SETTING {setting_key} = {setting_value}"
    target.append(ClusterableStatement(prefix, suffix))
```

On non-replicated tables this works fine. On ReplicatedMergeTree, multiple setting changes in the same migration can trigger Code 517.

### Workaround

Until dbwarden batches MODIFY SETTING into a single statement, you can manually combine them in a `data_op()`:

```python
data_op(
    name="batch_modify_settings",
    forward="ALTER TABLE events MODIFY SETTING index_granularity = 4096, max_bytes_to_merge_at_max_space_in_pool = 10000000000",
)
```

## ALTER batching

### What can be combined

Compatible metadata-only ALTER subcommands can appear in a single `ALTER TABLE` statement. ClickHouse processes them atomically (single ZooKeeper coordination round, no Code 517 race):

| Subcommands | Example |
|-------------|---------|
| ADD INDEX + ADD INDEX | `ALTER TABLE t ADD INDEX ix_a ..., ADD INDEX ix_b ...` |
| ADD/DROP INDEX mix | `ALTER TABLE t ADD INDEX ix_a ..., DROP INDEX ix_old` |
| ADD PROJECTION + ADD PROJECTION | `ALTER TABLE t ADD PROJECTION p1 ..., ADD PROJECTION p2 ...` |
| ADD COLUMN + ADD INDEX | `ALTER TABLE t ADD COLUMN col Int64, ADD INDEX ix col TYPE minmax GRANULARITY 1` |
| MODIFY SETTING (multiple) | `ALTER TABLE t MODIFY SETTING k1 = v1, k2 = v2` |
| COMMENT + MODIFY COLUMN COMMENT | `ALTER TABLE t COMMENT 'table comment', MODIFY COLUMN c COMMENT 'col comment'` |

### What cannot be combined

Some operations must be in separate statements:

| Restriction | Reason |
|-------------|--------|
| MATERIALIZE INDEX + ADD INDEX | DatabaseReplicated rejects mixed AlterCommand/MutationCommand segments with `QUERY_IS_PROHIBITED` |
| UPDATE/DELETE + any metadata ALTER | Mutations are async background processes; metadata ALTERs are synchronous ZooKeeper operations |
| MODIFY SETTING + MATERIALIZE INDEX | Mixes metadata and mutation in one statement |
| DROP TABLE + CREATE TABLE (same name) | Requires atomic swap, not combinable in one ALTER |

When operations cannot be combined, serialize them and ensure each completes before the next begins (via `mutations_sync=2`, `alter_sync=2`, or polling `system.mutations`).

### dbwarden's batching behavior

| Handler | Batches? | Separator | Notes |
|---------|----------|-----------|-------|
| ChColumnHandler | Yes | `";\n"` | Multiple MODIFY COLUMN for same column joined |
| ChTableHandler | Partially | `"\n"` | MODIFY SETTING emitted per-setting (see limitation above) |
| ChSkipIndexHandler | Yes | `"\n"` | ADD/DROP INDEX batched |
| ChProjectionHandler | Yes | `"\n"` | ADD/DROP PROJECTION batched |
| ChCommentHandler | Yes | `"\n"` | Table + column comments batched |
| ChDictionaryHandler | Yes | `"\n"` | MODIFY LAYOUT/SOURCE/LIFETIME/PRIMARY KEY batched |
| ChMVHandler | No | N/A | One op per statement (MODIFY QUERY, MODIFY REFRESH) |
| ChDataOpHandler | No | N/A | User provides raw SQL |

## Operation catalog

Complete reference of every ALTER subcommand dbwarden generates, its sync category, batching behavior, and safety classification.

### Table-level operations

| Operation | Handler | Sync setting | Batched? | Safety | Example |
|-----------|---------|-------------|----------|--------|---------|
| MODIFY SETTING | ChTableHandler | alter_sync | **No** (per-setting) | INFO | `ALTER TABLE t MODIFY SETTING k = v` |
| MODIFY TTL | ChTableHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY TTL expr` |
| REMOVE TTL | ChTableHandler | alter_sync | Yes | INFO | `ALTER TABLE t REMOVE TTL` |
| MODIFY ORDER BY | ChTableHandler | alter_sync | Yes | CRITICAL | `ALTER TABLE t MODIFY ORDER BY (col)` |
| COMMENT | ChCommentHandler | alter_sync | Yes | INFO | `ALTER TABLE t COMMENT 'text'` |
| Recreate (swap) | ChTableHandler | N/A (DDL) | N/A | CRITICAL | CREATE new → INSERT → RENAME → DROP old |

### Column operations

| Operation | Handler | Sync setting | Batched? | Safety | Example |
|-----------|---------|-------------|----------|--------|---------|
| MODIFY COLUMN (type) | ChColumnHandler | alter_sync | Yes | WARN | `ALTER TABLE t MODIFY COLUMN c Int64` |
| MODIFY COLUMN (codec) | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 CODEC(ZSTD)` |
| MODIFY COLUMN (TTL) | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 TTL expr` |
| MODIFY COLUMN (default) | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 DEFAULT 0` |
| MODIFY COLUMN (materialized) | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 MATERIALIZED expr` |
| MODIFY COLUMN (alias) | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 ALIAS expr` |
| MODIFY COLUMN (ephemeral) | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 EPHEMERAL` |
| REMOVE CODEC | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 REMOVE CODEC` |
| REMOVE TTL | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 REMOVE TTL` |
| REMOVE DEFAULT | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 REMOVE DEFAULT` |
| REMOVE MATERIALIZED | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 REMOVE MATERIALIZED` |
| REMOVE ALIAS | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 REMOVE ALIAS` |
| REMOVE EPHEMERAL | ChColumnHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c Int64 REMOVE EPHEMERAL` |
| Column comment | ChCommentHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c COMMENT 'text'` |
| Column REMOVE COMMENT | ChCommentHandler | alter_sync | Yes | INFO | `ALTER TABLE t MODIFY COLUMN c REMOVE COMMENT` |

### Index operations

| Operation | Handler | Sync setting | Batched? | Safety | Example |
|-----------|---------|-------------|----------|--------|---------|
| ADD INDEX | ChSkipIndexHandler | alter_sync | Yes | INFO | `ALTER TABLE t ADD INDEX ix col TYPE minmax GRANULARITY 1` |
| DROP INDEX | ChSkipIndexHandler | alter_sync | Yes | WARN | `ALTER TABLE t DROP INDEX ix` |

### Projection operations

| Operation | Handler | Sync setting | Batched? | Safety | Example |
|-----------|---------|-------------|----------|--------|---------|
| ADD PROJECTION | ChProjectionHandler | alter_sync | Yes | INFO | `ALTER TABLE t ADD PROJECTION p (SELECT ...)` |
| DROP PROJECTION | ChProjectionHandler | alter_sync | Yes | WARN | `ALTER TABLE t DROP INDEX p` |

### MV operations

| Operation | Handler | Sync setting | Batched? | Safety | Example |
|-----------|---------|-------------|----------|--------|---------|
| MODIFY QUERY | ChMVHandler | alter_sync | No | CRITICAL | `ALTER TABLE mv MODIFY QUERY SELECT ...` |
| MODIFY REFRESH | ChMVHandler | alter_sync | No | CRITICAL | `ALTER TABLE mv MODIFY REFRESH EVERY 1 HOUR` |

### Dictionary operations

| Operation | Handler | Sync setting | Batched? | Safety | Example |
|-----------|---------|-------------|----------|--------|---------|
| MODIFY LAYOUT | ChDictionaryHandler | alter_sync | Yes | INFO | `ALTER DICTIONARY d MODIFY LAYOUT(...)` |
| MODIFY SOURCE | ChDictionaryHandler | alter_sync | Yes | INFO | `ALTER DICTIONARY d MODIFY SOURCE(...)` |
| MODIFY LIFETIME | ChDictionaryHandler | alter_sync | Yes | INFO | `ALTER DICTIONARY d MODIFY LIFETIME(...)` |
| MODIFY PRIMARY KEY | ChDictionaryHandler | alter_sync | Yes | INFO | `ALTER DICTIONARY d MODIFY PRIMARY KEY(...)` |

### Data operations (user-authored)

| Operation | Handler | Sync setting | Batched? | Safety | Example |
|-----------|---------|-------------|----------|--------|---------|
| DELETE | ChDataOpHandler | mutations_sync | N/A (user SQL) | WARN | `ALTER TABLE t DELETE WHERE ...` |
| UPDATE | ChDataOpHandler | mutations_sync | N/A (user SQL) | WARN | `ALTER TABLE t UPDATE col = val WHERE ...` |
| MATERIALIZE INDEX | ChDataOpHandler | mutations_sync | N/A (user SQL) | INFO | `ALTER TABLE t MATERIALIZE INDEX ix` |
| MATERIALIZE PROJECTION | ChDataOpHandler | mutations_sync | N/A (user SQL) | INFO | `ALTER TABLE t MATERIALIZE PROJECTION p` |
| OPTIMIZE | ChDataOpHandler | N/A | N/A (user SQL) | INFO | `OPTIMIZE TABLE t FINAL` |
| Partition operations | ChDataOpHandler | N/A | N/A (user SQL) | varies | `ALTER TABLE t DROP PARTITION '2024-01'` |

## Related

- [Safety classification](safety.md): INFO/WARN/CRITICAL levels and the recreate pipeline
- [Migration locking](../../advanced/clickhouse-locking.md): CH-0 through CH-4 coordination profiles
- [Data operations](data-operations.md): non-DDL mutations and ad-hoc operations
- [ON Cluster](on-cluster.md): DDL propagation in clustered deployments

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/columns-types/
========================================================================

# Columns and types

## Column-level Meta

Use `CHColumnMeta` inner classes on the model, named after the column:

```python
from dbwarden.databases.clickhouse import CHTableMeta, CHColumnMeta, ch

class Event(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    payload: Mapped[str] = mapped_column()
    event_time: Mapped[datetime] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="event_time")

        class payload(CHColumnMeta):
            ch = ch.field(codec="ZSTD(3)")

        class event_time(CHColumnMeta):
            ch = ch.field(default_expression="now()")
```

Generated DDL:

```sql
CREATE TABLE events (
    id Int64,
    payload String CODEC(ZSTD(3)),
    event_time DateTime DEFAULT now()
) ENGINE = MergeTree() ORDER BY event_time
```

## Additional model examples

### Column with codec and LowCardinality

```python
class UserSession(Base):
    __tablename__ = "user_sessions"

    user_id: Mapped[int] = mapped_column(primary_key=True)
    browser: Mapped[str] = mapped_column()
    ip: Mapped[str] = mapped_column()
    duration: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="user_id")

        class browser(CHColumnMeta):
            ch = ch.field(
                codec="ZSTD(3)",
                low_cardinality=True,
            )

        class ip(CHColumnMeta):
            ch = ch.field(
                codec="LZ4HC(9)",
                nullable=True,
            )

        class duration(CHColumnMeta):
            ch = ch.field(
                default_expression="0",
                ttl="now() - toIntervalDay(30)",
            )
```

Generated DDL:

```sql
CREATE TABLE user_sessions (
    user_id Int64,
    browser LowCardinality(String) CODEC(ZSTD(3)),
    ip Nullable(String) CODEC(LZ4HC(9)),
    duration Int64 DEFAULT 0 TTL now() - toIntervalDay(30)
) ENGINE = MergeTree() ORDER BY user_id
```

Codec chains are preserved in order:

```python
class Event(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    value: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="id")

        class value(CHColumnMeta):
            ch = ch.field(codec="Delta(8), ZSTD(1)")
```

```sql
CREATE TABLE events (
    id Int64,
    value Float64 CODEC(Delta(8), ZSTD(1))
) ENGINE = MergeTree() ORDER BY id
```

The default `LZ4` codec is treated as implicit and is omitted from generated models. `LowCardinality(Nullable(T))` round-trips as `low_cardinality=True, nullable=True`.

### Column with alias and comment

```python
class Metrics(Base):
    __tablename__ = "metrics"

    width: Mapped[float] = mapped_column()
    height: Mapped[float] = mapped_column()
    area: Mapped[float] = mapped_column(comment="Calculated area in pixels")

    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="width")

        class area(CHColumnMeta):
            ch = ch.field(alias="width * height")
```

### REMOVE clause example

```python
# Removing a codec from a column
class Meta(CHTableMeta):
    ch = ch_table(engine=merge_tree(), order_by="id")

    class payload(CHColumnMeta):
        ch = ch.field(codec=None)  # emits MODIFY COLUMN payload REMOVE CODEC
```

## `ch.field()` options

| Parameter | Type | SQL | REMOVE form |
|-----------|------|-----|-------------|
| `codec` | `str` | `CODEC(ZSTD(3))` | `MODIFY COLUMN c REMOVE CODEC` |
| `default_expression` | `str` | `DEFAULT now()` | `MODIFY COLUMN c REMOVE DEFAULT` |
| `materialized` | `str` | `MATERIALIZED expr` | `MODIFY COLUMN c REMOVE MATERIALIZED` |
| `alias` | `str` | `ALIAS expr` | `MODIFY COLUMN c REMOVE ALIAS` |
| `ephemeral` | `str` | `EPHEMERAL expr` | `MODIFY COLUMN c REMOVE EPHEMERAL` |
| `ttl` | `str` | `TTL expr` | `MODIFY COLUMN c REMOVE TTL` |
| `low_cardinality` | `bool` | `LowCardinality(String)` | Wrapped into type |
| `nullable` | `bool` | `Nullable(String)` | Wrapped into type |

Column comments are not a `ch.field()` option; they ride on the SQLAlchemy
column itself (`mapped_column(comment=...)`) and are diffed by the comment
handler into `MODIFY COLUMN c COMMENT '...'` / `MODIFY COLUMN c REMOVE COMMENT`.

Setting a property to `None` (or omitting it) and then setting it to a value emits `MODIFY COLUMN ... <property>`. The reverse (removing a property) emits the `REMOVE` form. This is a write-only asymmetry in ClickHouse that dbwarden handles for you.

## Type normalization

SQLAlchemy types are normalized to ClickHouse native types:

| SQLAlchemy type | ClickHouse type |
|----------------|-----------------|
| `Integer`, `BIGINT` | `Int32`, `Int64` |
| `VARCHAR`, `String` | `String` |
| `FLOAT(53)`, `REAL` | `Float64`, `Float32` |
| `NUMERIC(p,s)` | `Decimal(p,s)` |
| `BOOLEAN` | `Bool` |
| `ARRAY(Integer)` | `Array(Int32)` |
| `Enum` | `Enum8` / `Enum16` |
| `UUID` | `UUID` |
| `JSON` | `JSON` |
| `DATETIME`, `DATETIME64` | `DateTime`, `DateTime64` |

## What changes are allowed

| Change | Safety | Notes |
|--------|--------|-------|
| Add column | INFO | Standard `ALTER TABLE ADD COLUMN` |
| Drop column | WARN | Requires `--force` |
| Type change (compatible) | INFO | e.g. `Int32` → `Int64` |
| Type change (incompatible) | CRITICAL | e.g. `String` → `Int64`, requires `--force` + recreate |
| Codec change | WARN | Relatively cheap |
| Default / Materialized / Alias change | WARN | |
| TTL change | WARN | |
| LowCardinality / Nullable toggle | CRITICAL | Requires `--force` |
| Comment change | INFO | |
| REMOVE any property | INFO | Emitted as `MODIFY COLUMN c REMOVE ...` |

## Rollback behavior

Column changes emit inverse operations. `ADD COLUMN` rolls back as `DROP COLUMN`. A `MODIFY COLUMN` rolls back as `MODIFY COLUMN` with the previous state.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/conventions/
========================================================================

# Conventions

## Canonicalization rules

The canonicalizer normalizes DDL before diffing. These normalizations apply:

| Rule | Example |
|------|---------|
| Whitespace normalization | `ORDER BY (a,b)` → `ORDER BY (a, b)` |
| Trailing comma removal | `(a, b,)` → `(a, b)` |
| Engine parameter normalization | ReplicatedMergeTree arguments |
| Settings key normalization | Underscore-vs-hyphen normalization |
| Type alias expansion | `INT` → `Int32`, `VARCHAR` → `String` |
| Nullable/LowCardinality wrapper normalization | Wrapping order |
| `ENGINE = Distributed(cluster, db, table)` | Shard/key delimiters |
| Primary key matched to sorting key | `ch_primary_key` omitted when it equals `ch_order_by` |
| TTL normalization | `INTERVAL` → `toInterval`, absence vs `[]` |
| Settings matching captured server defaults removed | Compared against defaults snapshotted at extract time |
| Materialized-view table reference qualification | Bare MV table names qualified with the database |

## Defaults-as-absence

If a property matches the ClickHouse default, it is omitted from the emitted DDL. For example:

- `index_granularity = 8192` is the default and is not emitted unless explicitly set to a non-default value
- `SETTINGS` block is omitted entirely when all settings are at their defaults

Default settings come from the captured server metadata: unknown defaults are not guessed, and offline state must contain the relevant metadata before a comparison can remove those differences. An explicit `index_granularity=8192` does not imply identical output on every server or in every generation path.

This means the diff is clean: only non-default values appear in the DDL and the field is absent from declarations until overridden.

## Secrets-declare-only

Credentials are never diffed. See [Named collections](named-collections.md).

- Values of `password`, `sasl_password`, `secret_access_key` etc. are not compared between model and server
- Named collection declarations list the **keys** that should exist, not the secret values
- RBAC `identified_by` values are not stored in the model at all

## Additional model examples

### Builder vs raw dict comparison

```python
# Builder path (preferred): typed, validated, autocompletable
ch = ch_table(
    engine=replicated_merge_tree("/zk/path", "{replica}"),
    order_by=["ts", "id"],
    partition_by="toYYYYMM(ts)",
    settings=MergeTreeSettings({
        "index_granularity": 4096,
        "min_bytes_for_wide_part": "10485760",
    }),
)

# Raw dict path (escape hatch): same output, no validation
ch = {
    "engine": {
        "type": "ReplicatedMergeTree",
        "params": ["/zk/path", "{replica}"],
    },
    "order_by": ["ts", "id"],
    "partition_by": "toYYYYMM(ts)",
    "settings": {
        "index_granularity": "4096",
        "min_bytes_for_wide_part": "10485760",
    },
}
```

### Defaults-as-absence example

```python
# These two models emit identical DDL:
class Explicit(Base):
    __tablename__ = "t"
    x: Mapped[int] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="x",
                       settings=MergeTreeSettings(index_granularity=8192))

class Implicit(Base):
    __tablename__ = "t"
    x: Mapped[int] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by="x")
        # index_granularity=8192 is the default, omitted from DDL
```

Both produce:

```sql
CREATE TABLE t (x Int64) ENGINE = MergeTree() ORDER BY x
```

## Two-door API

Every operation has two paths:

| Path | Purpose |
|------|---------|
| Builder (e.g., `ch_table()`, `ChRoleSpec()`) | Primary. Provides type-checking, autocomplete, and validation. Covers 95% of use cases. |
| Raw dict (e.g., `{"type": "MergeTree"}`) | Escape hatch. For infrequent or edge-case settings the builder doesn't expose. |

Raw dicts are passed through without validation:

```python
# Builder path
ch = ch_table(
    engine=merge_tree(),
    settings={"allow_experimental_inverted_index": 1},
)

# Raw path
ch = {"engine": {"type": "MergeTree"}, "settings": {"allow_experimental_inverted_index": "1"}}
```

Raw dicts are validated for structure (must match a known schema) but not for content correctness. The builder path is always preferred.

## Version support evidence

Every assertion about version support is backed by a test case in the audit harness:

```
tests/databases/clickhouse/audit/
├── 24.3/       # 31 cases, zero drift
└── 26.6/       # same 31 cases, zero drift
```

New ClickHouse versions are added by running the existing test suite against the new version. Zero branching in the canonicalizer means there is no per-version code path to update.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/data-operations/
========================================================================

# Data operations

Data operations (`data_op()`) are non-DDL statements that mutate data or trigger maintenance. They are not declared in the model: they are ad-hoc operations.

## Additional model examples

### Batch partition cleanup

```python
# Freeze, back up, then drop old partitions
from dbwarden.databases.clickhouse import data_op

# Freeze for backup
for m in ["2023-01", "2023-02", "2023-03"]:
    data_op(name=f"freeze_{m}", forward=f"ALTER TABLE events FREEZE PARTITION '{m}'")

# After backup verified, drop
for m in ["2023-01", "2023-02", "2023-03"]:
    data_op(name=f"drop_{m}", forward=f"ALTER TABLE events DROP PARTITION '{m}'")
```

### Conditional mutation with setting override

```python
# Large mutation with timeout
data_op(
    name="archive_old_events",
    forward="""
    ALTER TABLE events
        UPDATE status = 'archived'
        WHERE event_date < '2020-01-01'
        SETTINGS mutations_sync = 2
""",
)
```

### OPTIMIZE with deduplicate

```python
# Force full merge and deduplication on all parts
data_op(name="optimize_deduplicate", forward="OPTIMIZE TABLE events FINAL DEDUPLICATE")

# With column-specific deduplication
data_op(name="optimize_deduplicate_by", forward="OPTIMIZE TABLE events FINAL DEDUPLICATE BY id, event_date")
```

### Multi-step migration with data ops

```python
def migrate_events():
    # 1. Create new table via migrate
    # 2. Backfill from old partition
    data_op(
        name="replace_partition_from_v1",
        forward="""
        ALTER TABLE events_v2
            REPLACE PARTITION '2024-01'
            FROM events_v1
    """,
    )
    # 3. Drop old partition
    data_op(name="drop_v1_partition", forward="ALTER TABLE events_v1 DROP PARTITION '2024-01'")
    # 4. Verify
    data_op(name="optimize_v2", forward="OPTIMIZE TABLE events_v2 FINAL")
```

## Partition operations

```python
from dbwarden.databases.clickhouse import data_op

# Attach a detached partition
data_op(name="attach_partition", forward="ALTER TABLE events ATTACH PARTITION '2024-01'")

# Replace one partition with another
data_op(name="replace_partition", forward="ALTER TABLE events REPLACE PARTITION '2024-02' FROM staging_events")

# Drop a partition
data_op(name="drop_partition", forward="ALTER TABLE events DROP PARTITION '2024-01'")

# Clear column in partition
data_op(name="clear_column", forward="ALTER TABLE events CLEAR COLUMN payload IN PARTITION '2024-01'")

# Freeze partition for backup
data_op(name="freeze_partition", forward="ALTER TABLE events FREEZE PARTITION '2024-01'")

# Unfreeze
data_op(name="unfreeze_partition", forward="ALTER TABLE events UNFREEZE PARTITION '2024-01'")
```

## Mutations

```python
# DELETE
data_op(name="delete_old_events", forward="ALTER TABLE events DELETE WHERE event_date < '2023-01-01'")

# UPDATE
data_op(name="redact_payload", forward="ALTER TABLE events UPDATE payload = 'redacted' WHERE id = 123")
```

!!! tip "Synchronicity"
    Mutations are async by default. Use `SETTINGS mutations_sync = 2` to wait for all replicas, or `mutations_sync = 1` to wait for the current server only. See [ALTER semantics](alter-semantics.md#mutations_sync-vs-alter_sync) for the full distinction between `mutations_sync` and `alter_sync`. **Do not use value `3` on ReplicatedMergeTree**; it silently does nothing ([known bug](alter-semantics.md#the-mutations_sync3-bug)).

## OPTIMIZE

```python
# Merge parts
data_op(name="optimize_final", forward="OPTIMIZE TABLE events FINAL")

# With partition
data_op(name="optimize_partition", forward="OPTIMIZE TABLE events PARTITION '2024-01' FINAL")

# Deduplicate
data_op(name="optimize_deduplicate", forward="OPTIMIZE TABLE events FINAL DEDUPLICATE")
```

## POPULATE

```python
# Populate a materialized view
data_op(name="populate_mv", forward="ALTER TABLE mv_name POPULATE")
```

This is a data-op rather than a DDL property because it is a write concern, not structural. See [Materialized views](materialized-views.md).

The `data_ops` module also provides a `populate(materialized_view_spec)` helper (exported as `data_ops_populate`) that derives the populate statement from a materialized or aggregating view spec and returns a `DataOp` whose forward SQL is `INSERT INTO <target> <select>`. It requires a target table and a select query; pass `rollback=` to make it reversible. A plugin or authoring tool can pass the descriptor in an explicit `apply_data_op` operation to `ChDataOpHandler.emit()`. When `requires_confirmation` is set, the emitter comments out the forward SQL for manual review instead of opening an interactive prompt. Deduplication is by migration history, not by the descriptor's name.

## Secret rotation

Named collection secrets are rotated through ClickHouse's secret store:

```python
# Refresh credentials from secret store
data_op(
    name="rotate_kafka_secret",
    forward="ALTER NAMED COLLECTION kafka_prod UPDATE sasl_password = SECRET 'new_secret_id'",
)
```

## Safety

| Operation | Safety | Notes |
|-----------|--------|-------|
| ATTACH PARTITION | INFO | Cheap metadata operation |
| REPLACE PARTITION | WARN | Overwrites target |
| DROP PARTITION | WARN | Data loss within a partition |
| CLEAR COLUMN | WARN | Data cleared for partition |
| DELETE mutation | WARN | Async, causes part rewrites |
| UPDATE mutation | WARN | Async, causes part rewrites |
| OPTIMIZE FINAL | INFO | Heavy IO |
| POPULATE | INFO | Inserts current data |
| Secret rotation | INFO | |

## Rollback behavior

Data operations are **not reversible** by dbwarden: they are ad-hoc mutations. Plan accordingly: test on staging, back up partitions before DROP or REPLACE.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/declaring-tables/
========================================================================

# Declaring tables

## The `ch_table()` builder (preferred path)

Use `ch_table()` inside a `class Meta(CHTableMeta)` block. This is the primary, recommended API:

```python
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.clickhouse import (
    CHTableMeta, merge_tree, ch_table,
)

class Base(DeclarativeBase):
    pass

class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    payload: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["event_date", "id"],
            partition_by="toYYYYMM(event_date)",
            ttl=["event_date + toIntervalYear(1)"],
            settings={"index_granularity": "8192"},
        )
```

`ch_table()` returns a `ChTableSpec` dataclass. Full signature:

| Parameter | Type | SQL |
|-----------|------|-----|
| `engine` | `ChEngineSpec` | `ENGINE = MergeTree()` |
| `order_by` | `str` or `list[str]` | `ORDER BY (col1, col2)` |
| `primary_key` | `str` or `list[str]` | `PRIMARY KEY (col1)` |
| `partition_by` | `str` | `PARTITION BY toYYYYMM(col)` |
| `sample_by` | `str` | `SAMPLE BY intHash64(col)` |
| `ttl` | `str` or `list[str]` | `TTL expr1, expr2` |
| `settings` | `MergeTreeSettings` dict | `SETTINGS key=value` |
| `projections` | `list[ProjectionSpec]` | `PROJECTION name (SELECT ...)` |
| `indexes` | `list[ChIndexSpec]` | `ALTER TABLE ... ADD INDEX ...` |

## Generated DDL

```sql
CREATE TABLE IF NOT EXISTS events (
    id Int64,
    event_date Date,
    payload String
) ENGINE = MergeTree()
ORDER BY (event_date, id)
PARTITION BY toYYYYMM(event_date)
TTL event_date + toIntervalYear(1)
SETTINGS index_granularity = 8192;
```

## Loose `ch_*` attrs (legacy)

Individual `ch_engine`, `ch_order_by`, etc. class attributes on `Meta` still work but are **deprecated**:

```python
class Meta(CHTableMeta):
    ch_engine = ChEngineSpec("MergeTree")       # deprecated
    ch_order_by = ["event_date", "id"]           # deprecated
    ch_partition_by = "toYYYYMM(event_date)"     # deprecated
```

**Migration path:** Replace all loose `ch_*` attrs with a single `ch = ch_table(...)` assignment. Both forms coexist in the same codebase during migration; the loose path emits a `DeprecationWarning`.

## Additional model examples

### Replicated table with custom settings

```python
class ReplicatedEvents(Base):
    __tablename__ = "replicated_events"

    id: Mapped[int] = mapped_column(primary_key=True)
    ts: Mapped[datetime] = mapped_column()
    value: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=replicated_merge_tree(
                "/clickhouse/tables/{shard}/replicated_events",
                "{replica}",
            ),
            order_by=["ts", "id"],
            partition_by="toYYYYMM(ts)",
            ttl=["ts + toIntervalMonth(3)"],
            settings={
                "index_granularity": 4096,
                "min_bytes_for_wide_part": 10485760,
            },
        )
```

### Table with multiple projections and indexes

```python
class AnalyticsEvents(Base):
    __tablename__ = "analytics_events"

    dt: Mapped[date] = mapped_column()
    user_id: Mapped[int] = mapped_column()
    event_type: Mapped[str] = mapped_column()
    amount: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=summing_merge_tree("amount"),
            order_by=["dt", "user_id", "event_type"],
            partition_by="toYYYYMM(dt)",
            settings={"allow_experimental_full_text_index": 1},
            projections=[
                ch_projection(
                    name="user_summary",
                    select=["user_id", "sum(amount)", "count()"],
                    group_by=["user_id"],
                ),
            ],
            indexes=[
                ch_index(
                    name="type_bloom",
                    expression="event_type",
                    type="bloom_filter(0.02)",
                ),
            ],
        )
```

### Distributed table

```python
class DistributedEvents(Base):
    __tablename__ = "distributed_events"

    id: Mapped[int] = mapped_column(primary_key=True)

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=distributed_engine(
                cluster="analytics_cluster",
                database="analytics",
                table="events",
                sharding_key="rand()",
            ),
        )
```

## What changes are allowed

See [Immutability](immutability.md) for the full rules.

| Property | Allowed changes |
|----------|----------------|
| `order_by` | Append-only extension |
| `settings` | Any key-value change |
| `ttl` | Any expression change |
| `projections` | Add/drop by name |
| `indexes` | Add/drop by name |
| `engine` | Only with `--force` (recreate) |
| `partition_by` | Never |
| `primary_key` | Only with `--force` (recreate) |
| `sample_by` | Never |

## Rollback behavior

Every `ALTER` and `CREATE` statement has a rollback. The rollback for a CREATE is DROP. The rollback for an ALTER is the inverse ALTER. For recreates, the rollback restores the original table.

See [Safety](safety.md) for the full rebuild pipeline.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/dictionaries/
========================================================================

# Dictionaries

## Declaration

Dictionaries are declared with flat `ch_dict_*` attributes on `class Meta`, not with builder functions. Set `ch_dictionary = True` to mark the model as a dictionary:

```python
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.clickhouse import CHTableMeta

class CountryLookup(Base):
    __tablename__ = "country_lookup"

    code: Mapped[str] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_primary_key = "code"
        ch_dict_layout = "flat"
        ch_dict_source = {
            "clickhouse": {"query": "SELECT code, name FROM source_countries"},
        }
        ch_dict_lifetime = 300
```

The five recognised attributes are `ch_dictionary`, `ch_dict_primary_key`, `ch_dict_layout`, `ch_dict_source`, and `ch_dict_lifetime`. Because `CHTableMeta` is validated at import time, a misspelled attribute raises `DBWardenConfigError` when the module loads.

There is also a `dictionary()` helper that builds a `DictSpec` directly, for code that constructs specs programmatically rather than declaring them on a model:

```python
from dbwarden.databases.clickhouse import dictionary

spec = dictionary(
    layout="hashed",
    source={"clickhouse": {"table": "dim_users"}},
    lifetime=300,
    primary_key="id",
)
```

## Additional model examples

### MySQL-sourced dictionary

```python
class MySQLCountry(Base):
    __tablename__ = "mysql_country"

    code: Mapped[str] = mapped_column()
    name: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_primary_key = "code"
        ch_dict_layout = "hashed"
        ch_dict_source = {
            "mysql": {
                "named_collection": "mysql_dict",
                "query": "SELECT iso_code, full_name FROM ref.countries",
            },
        }
        ch_dict_lifetime = "MIN 60 MAX 300"
```

### HTTP-sourced dictionary with complex key

```python
class CurrencyRate(Base):
    __tablename__ = "currency_rate"

    currency: Mapped[str] = mapped_column()
    rate: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_primary_key = "currency"
        ch_dict_layout = "cache"
        ch_dict_source = {
            "http": {
                "url": "https://api.example.com/rates",
                "format": "JSONEachRow",
            },
        }
        ch_dict_lifetime = 3600
```

### Range-hashed dictionary for time-based lookup

```python
class TaxRate(Base):
    __tablename__ = "tax_rate"

    region: Mapped[str] = mapped_column()
    rate: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_primary_key = ["region", "valid_from"]
        ch_dict_layout = "range_hashed"
        ch_dict_source = {
            "clickhouse": {
                "query": "SELECT region, valid_from, valid_to, rate FROM ref.tax_rates",
            },
        }
        ch_dict_lifetime = 86400
```

Usage in queries:

```sql
SELECT dictGet('tax_rate', 'rate', ('CA', today()))
```

## Source types

`ch_dict_source` is a dict keyed by source type, whose value carries that source's settings:

| Source type | `ch_dict_source` |
|-------------|------------------|
| ClickHouse | `{"clickhouse": {"query": "..."}}` |
| MySQL | `{"mysql": {...}}` |
| PostgreSQL | `{"postgresql": {...}}` |
| MongoDB | `{"mongodb": {...}}` |
| HTTP(S) | `{"http": {...}}` |
| Local file | `{"file": {...}}` |
| Executable | `{"executable": {...}}` |

For connection secrets, reference a named collection rather than inlining credentials:

```python
ch_dict_source = {"clickhouse": {"named_collection": "clickhouse_dict_source"}}
```

## Layout types

`ch_dict_layout` is the layout name as a string:

```python
ch_dict_layout = "flat"                # One key, single value
ch_dict_layout = "hashed"              # Hash table, all in memory
ch_dict_layout = "sparse_hashed"       # Like hashed but sparse
ch_dict_layout = "cache"               # LRU cache
ch_dict_layout = "complex_key_hashed"  # Composite keys
ch_dict_layout = "ip_trie"             # IP prefix matching
ch_dict_layout = "direct"              # No caching
ch_dict_layout = "range_hashed"        # Time ranges
```

## Lifetime

`ch_dict_lifetime` accepts an integer for a fixed interval, or a string for ClickHouse's ranged form:

```python
ch_dict_lifetime = 300              # Fixed interval, in seconds
ch_dict_lifetime = "MIN 300 MAX 600"  # Ranged
```

## What changes are allowed

| Change | Safety |
|--------|--------|
| Lifetime adjustment | INFO |
| Layout change | CRITICAL: requires recreate |
| Source connection change | INFO (named collection swap) |
| Query/SELECT change | WARN |
| Primary key change | CRITICAL: requires recreate |

## Rollback behavior

Dictionary changes that require a recreate follow the full pipeline. See [Safety](safety.md).

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/engines-integration/
========================================================================

# Integration engines

dbwarden supports 13 integration engines. Credentials use the [named-collections](named-collections.md) declare-only pattern: no secret values are diffed.

## Kafka

```python
from dbwarden.databases.clickhouse import kafka

class Meta(CHTableMeta):
    ch = ch_table(
        engine=kafka(
            named_collection="kafka_prod",
            topic_list="events",
            format="JSONEachRow",
            group_name="dbwarden_consumer",
        ),
    )
```

Generated DDL:

```sql
CREATE TABLE kafka_events (
    payload String
) ENGINE = Kafka
SETTINGS kafka_named_collection = 'kafka_prod',
         kafka_topic_list = 'events',
         kafka_format = 'JSONEachRow',
         kafka_group_name = 'dbwarden_consumer';
```

Parameters that can be set directly (overriding the named collection):

| Factory parameter | DDL setting |
|------------------|-------------|
| `named_collection` | collection name argument |
| `broker_list` | `kafka_broker_list` |
| `topic_list` | `kafka_topic_list` |
| `group_name` | `kafka_group_name` |
| `format` | `kafka_format` |
| `num_consumers` | `kafka_num_consumers` |

Other Kafka settings (`kafka_thread_per_consumer`, `kafka_handle_error_mode`,
`kafka_commit_every_batch`, ...) are set through the `KafkaSettings` TypedDict
below, not as `kafka()` parameters.

`KafkaSettings` is a fully-typed TypedDict for arbitrary Kafka engine settings:

```python
from dbwarden.databases.clickhouse import KafkaSettings

settings: KafkaSettings = {
    "kafka_max_block_size": 524288,
}
```

## S3

```python
from dbwarden.databases.clickhouse import s3

class Meta(CHTableMeta):
    ch = ch_table(
        engine=s3(
            named_collection="s3_prod",
            path="events/*.parquet",
            format="Parquet",
        ),
    )
```

Parameters:

| Parameter | DDL setting |
|-----------|-------------|
| `named_collection` | collection name argument |
| `path` | `url` |
| `format` | `format` |
| `compression` | `compression` |

`S3Settings` for arbitrary settings. Key naming variants (`s3_*`, `s3queue_*`) are all typed.

## S3Queue

```python
from dbwarden.databases.clickhouse import s3_queue

class Meta(CHTableMeta):
    ch = ch_table(
        engine=s3_queue(
            named_collection="s3_prod",
            path="incoming/*.json",
            format="JSONEachRow",
        ),
    )
```

`S3QueueSettings` covers all s3queue-specific settings.

## RabbitMQ

```python
from dbwarden.databases.clickhouse import rabbitmq

class Meta(CHTableMeta):
    ch = ch_table(
        engine=rabbitmq(
            named_collection="rabbit_prod",
            format="JSONEachRow",
        ),
    )
```

`RabbitMQSettings`, typed TypedDict.

## NATS

```python
from dbwarden.databases.clickhouse import nats

class Meta(CHTableMeta):
    ch = ch_table(
        engine=nats(
            named_collection="nats_prod",
            format="JSONEachRow",
        ),
    )
```

`NatsSettings`, typed TypedDict.

## MySQL, PostgreSQL, MongoDB, Redis

```python
from dbwarden.databases.clickhouse import mysql_engine, postgresql_engine, mongodb, redis

# MySQL engine
engine = mysql_engine("mysql-host.example.com", 3306, "source_db", "source_table", "reader", "secret")

# PostgreSQL engine
engine = postgresql_engine("pg-host.example.com", 5432, "source_db", "source_table", "reader", "secret")

# MongoDB engine
engine = mongodb("mongo-host.example.com", 27017, "source_db", "source_collection", "reader", "secret")

# Redis engine (storage is the database index)
engine = redis("redis-host.example.com", 6379, "secret", 0)
```

Each has an associated `*Settings` TypedDict for engine-specific settings.

## Additional model examples

### Named collection for multi-engine reuse

```python
# Single named collection reused by Kafka and S3 engines
named_collection(
    name="aws_prod",
    keys={
        "region": "us-east-1",
        "access_key_id": "AKIA...",
        # secret_access_key from secret store
    },
)

engine = kafka(
    named_collection="aws_prod",
    topic_list="events",
    format="JSONEachRow",
    group_name="ch_consumer",
)

engine2 = s3(
    named_collection="aws_prod",
    path="data/*.parquet",
    format="Parquet",
)
```

### S3Queue with complex settings

```python
engine = s3_queue(
    named_collection="aws_prod",
    path="incoming/*.json",
    format="JSONEachRow",
)

# With custom settings
settings: S3QueueSettings = {
    "s3queue_processing_threads": 8,
    "s3queue_polling_min_timeout_ms": 1000,
    "s3queue_polling_max_timeout_ms": 30000,
    "s3queue_tracked_files_limit": 100000,
}
```

### PostgreSQL engine

```python
engine = postgresql_engine("pg-host.example.com", 5432, "source_db", "users", "reader", "secret")
```

### URL engine with multiple formats

```python
# CSV
engine = url_engine("https://data.example.com/export.csv", "CSV")

# JSONEachRow
engine = url_engine("https://data.example.com/events.json", "JSONEachRow")
```

## URL

```python
from dbwarden.databases.clickhouse import url_engine

class Meta(CHTableMeta):
    ch = ch_table(
        engine=url_engine("https://data.example.com/export.csv", "CSV"),
    )
```

`URLSettings`.

## File

```python
from dbwarden.databases.clickhouse import file_engine

class Meta(CHTableMeta):
    ch = ch_table(
        engine=file_engine(
            path="/var/lib/clickhouse/user_files/export.csv",
            format="CSV",
        ),
    )
```

## HDFS

```python
from dbwarden.databases.clickhouse import hdfs

class Meta(CHTableMeta):
    ch = ch_table(
        engine=hdfs("hdfs://namenode:9000/data/events.parquet", "Parquet"),
    )
```

`HDFSSettings`.

## Per-engine settings TypedDicts

Every integration engine has its own `*Settings` TypedDict for arbitrary settings. These are all defined in `dbwarden.databases.clickhouse`:

| Engine | TypedDict |
|--------|-----------|
| Kafka | `KafkaSettings` |
| S3 | `S3Settings` |
| S3Queue | `S3QueueSettings` |
| RabbitMQ | `RabbitMQSettings` |
| NATS | `NatsSettings` |
| MySQL | `MySQLSettings` |
| PostgreSQL | `PostgreSQLSettings` |
| MongoDB | `MongoDBSettings` |
| Redis | `RedisSettings` |
| URL | `URLSettings` |
| HDFS | `HDFSSettings` |

## What changes are allowed

| Change | Safety |
|--------|--------|
| Named collection swap | CRITICAL (metadata only, not data) |
| Any setting | INFO: `ALTER TABLE MODIFY SETTING` |
| Format change | WARN: requires data re-ingestion |
| Pattern / query / key change | WARN |

## Rollback behavior

Settings changes revert via `RESET SETTING`. Named collection swaps require reversing the collection reference.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/engines-mergetree/
========================================================================

# MergeTree engine family

## Factories

All MergeTree variants have typed factory functions in `dbwarden.databases.clickhouse`:

```python
from dbwarden.databases.clickhouse import (
    merge_tree, replacing_merge_tree, replicated_merge_tree,
    summing_merge_tree, aggregating_merge_tree,
    collapsing_merge_tree, versioned_collapsing_merge_tree,
    graphite_merge_tree,
)
```

| Factory | Engine name | Signature |
|---------|-------------|-----------|
| `merge_tree()` | `MergeTree` | `()` |
| `replacing_merge_tree(ver?)` | `ReplacingMergeTree` | `(version_col: str \| None = None)` |
| `replicated_merge_tree(zk, replica, ...)` | `ReplicatedMergeTree` | `(zookeeper_path, replica_name, *args)` |
| `summing_merge_tree(...)` | `SummingMergeTree` | `(*columns: str)` |
| `aggregating_merge_tree()` | `AggregatingMergeTree` | `()` |
| `collapsing_merge_tree(sign)` | `CollapsingMergeTree` | `(sign_col: str)` |
| `versioned_collapsing_merge_tree(sign, ver)` | `VersionedCollapsingMergeTree` | `(sign_col, version_col)` |
| `graphite_merge_tree(section?)` | `GraphiteMergeTree` | `(config_section: str = "default")` |

Example:

```python
class Meta(CHTableMeta):
    ch = ch_table(
        engine=replicated_merge_tree(
            "/clickhouse/tables/events",
            "{replica}",
            "ver",
        ),
        order_by=["event_date", "id"],
    )
```

Generated DDL:

```sql
CREATE TABLE events (
    event_date Date,
    id Int64
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/events', '{replica}', 'ver')
ORDER BY (event_date, id)
```

## MergeTreeSettings

`ch_table(settings=MergeTreeSettings(...))` type-checks known MergeTree settings:

```python
from dbwarden.databases.clickhouse import MergeTreeSettings

settings: MergeTreeSettings = {
    "index_granularity": 8192,
    "ttl_only_drop_parts": True,
    "min_bytes_for_wide_part": 10485760,
}
```

Boolean values are automatically converted to `"0"` / `"1"`. Integer values are stringified. All values are rendered as strings before reaching the server.

## What changes are allowed

| Property | Allowed |
|----------|---------|
| Engine variant | Only with `--force` (full recreate) |
| ZK path / replica name | Only with `--force` |
| Settings | Any key-value via `MODIFY SETTING` (where supported by server) |
| ORDER BY | Append-only |
| PARTITION BY | Never |

## Additional model examples

### ReplacingMergeTree with version column

```python
class Product(Base):
    __tablename__ = "products"

    sku: Mapped[str] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column()
    price: Mapped[float] = mapped_column()
    updated_at: Mapped[datetime] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=replacing_merge_tree(version_col="updated_at"),
            order_by="sku",
        )
```

Deduplicates by `sku`, keeping the row with the latest `updated_at`.

### CollapsingMergeTree for mutable state

```python
class OrderState(Base):
    __tablename__ = "order_state"

    order_id: Mapped[str] = mapped_column(primary_key=True)
    status: Mapped[str] = mapped_column()
    amount: Mapped[float] = mapped_column()
    sign: Mapped[int8] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=collapsing_merge_tree(sign_col="sign"),
            order_by="order_id",
        )
```

Cancellations emit a row with `sign = -1` that collapses with the original `sign = 1`.

### SummingMergeTree

```python
class DailySummary(Base):
    __tablename__ = "daily_summary"

    dt: Mapped[date] = mapped_column()
    product: Mapped[str] = mapped_column()
    revenue: Mapped[float] = mapped_column()
    units: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=summing_merge_tree("revenue"),
            order_by=["dt", "product"],
        )
```

`revenue` and `units` are summed automatically during merge.

### GraphiteMergeTree

```python
class GraphiteMetrics(Base):
    __tablename__ = "graphite_metrics"

    path: Mapped[str] = mapped_column()
    value: Mapped[float] = mapped_column()
    timestamp: Mapped[datetime] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=graphite_merge_tree(config_section="rollup_default"),
            order_by=["path", "timestamp"],
            partition_by="toYYYYMM(timestamp)",
        )
```

## Rollback behavior

Engine changes with `--force` trigger the full recreate pipeline. See [Safety](safety.md).

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/engines-special/
========================================================================

# Special engines

These engines do not participate in `ORDER BY`: they are storage-format-specific serverside readers.

## Factories

```python
from dbwarden.databases.clickhouse import (
    null, memory, merge,
    set_engine, join_engine, dictionary_engine,
    log, tiny_log, stripe_log,
)
```

## Additional model examples

### Null engine as MV sink

```python
from dbwarden.databases.clickhouse import MaterializedView, CHViewMeta, materialized_view

class NullSink(Base):
    __tablename__ = "null_sink"

    payload: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=null(),
        )

class ViewFromSink(MaterializedView):
    __tablename__ = "view_from_sink"

    class Meta(CHViewMeta):
        ch = materialized_view(
            to="sink_dest",
            select="SELECT count(*) AS value FROM null_sink",
        )
```

### Merge engine for partitioned read

```python
class AllEvents(Base):
    __tablename__ = "all_events"

    date: Mapped[date] = mapped_column()
    payload: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge("analytics", "events_202[0-9]_*"),
        )
```

### Dictionary engine with explicit dictionary

```python
class CountryDict(Base):
    __tablename__ = "country_dict"

    code: Mapped[str] = mapped_column()
    name: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(engine=dictionary_engine("countries"))
```

The dictionary is declared separately via `dictionary()`. See [Dictionaries](dictionaries.md).

## Null

```python
engine = null()
```

DDL: `ENGINE = Null`. Accepts any data and discards it. Used as the target of a materialized view that does its own aggregation.

## Memory

```python
engine = memory()
```

DDL: `ENGINE = Memory`. In-memory storage, lost on restart. Schema management only.

## Merge

```python
engine = merge("analytics", "events_.*")
```

DDL: `ENGINE = Merge('analytics', 'events_.*')`. A virtual table that reads from multiple tables whose names match the regex.

## Set

```python
engine = set_engine()
```

DDL: `ENGINE = Set`. Always in-memory. Use for IN-query acceleration.

## Join

```python
engine = join_engine("ALL", "LEFT")
```

DDL: `ENGINE = Join(ALL, LEFT)`. Specialized for JOIN queries.

## Dictionary

```python
engine = dictionary_engine("countries")
```

DDL: `ENGINE = Dictionary(<dict_name>)`. References a [Dictionary](dictionaries.md) object by name.

## Log, TinyLog, StripeLog

```python
engine = log()
engine = tiny_log()
engine = stripe_log()
```

DDLs: `ENGINE = Log`, `TinyLog`, `StripeLog`. Append-only file-based storage. No ORDER BY, no parts merging. StripeLog is multithreaded on read; TinyLog is the simplest.

## What changes are allowed

These engines have no ORDER BY, so immutability rules don't apply in the same way. An engine change (e.g., Memory → MergeTree) requires `--force` and a recreate.

| Change | Safety |
|--------|--------|
| Engine variant | CRITICAL with `--force` |
| Merge source/target | INFO |
| Join type/strictness | WARN |

## Rollback behavior

Engine changes trigger recreate. See [Safety](safety.md) for the pipeline.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/immutability/
========================================================================

# Immutability: what can never change

This page is the single most important thing to read before writing your first ClickHouse model. PG users expect most properties to be mutable via `ALTER`. ClickHouse is different: many properties are design-time commitments that can never be changed, and others require a full table rebuild.

## Model example: table with immutable properties

```python
class Orders(Base):
    __tablename__ = "orders"

    id: Mapped[int] = mapped_column(primary_key=True)
    created_at: Mapped[datetime] = mapped_column()
    amount: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["created_at", "id"],
            partition_by="toYYYYMM(created_at)",
            primary_key=["id"],
            sample_by="intHash64(id)",
        )
```

Once applied, `partition_by`, `primary_key`, `sample_by` and `engine` can never be altered. Only `order_by` can be extended by appending new columns: `["created_at", "id"]` → `["created_at", "id", "status"]`.

## What can never change

| Property | Constraint | Mechanism |
|----------|------------|-----------|
| `PARTITION BY` | Cannot be altered after creation | ClickHouse does not support `ALTER TABLE MODIFY PARTITION BY`. The only fix is a full table rebuild. |
| `PRIMARY KEY` | Cannot be altered | Unlike PG where you can `ALTER TABLE ... SET WITHOUT CLUSTER`, ClickHouse's primary key is baked into the storage order at creation. |
| `SAMPLE BY` | Cannot be altered | Same as partition by: set at CREATE time only. |
| `ENGINE` | Cannot be `ALTER`ed | Changing `MergeTree` to `ReplicatedMergeTree` (or vice versa) requires a full data copy. See below. |

## What extends only

| Property | Behavior | Example |
|----------|----------|---------|
| `ORDER BY` | New columns can be appended to the end. Existing columns cannot be removed or reordered. | `ORDER BY (a, b)` → `ORDER BY (a, b, c)` is valid. `ORDER BY (b, a)` is not. |

This is enforced by dbwarden: the differ refuses an `ORDER BY` change that is not an append-only extension. If you try, you get a CRITICAL-safety classification and must use `--force` to trigger a recreate.

## Model example: change requiring recreate

This ORDER BY change is refused by dbwarden:

```python
# Current model
class Meta(CHTableMeta):
    ch = ch_table(
        engine=replicated_merge_tree("/zk/orders", "{replica}"),
        order_by=["created_at", "id"],
    )

# Attempted change (reordered, not append-only)
class Meta(CHTableMeta):
    ch = ch_table(
        engine=replicated_merge_tree("/zk/orders", "{replica}"),
        order_by=["id", "created_at"],  # REORDERED: not append-only
    )
```

dbwarden emits: `CRITICAL: Changing ORDER BY from (created_at, id) to (id, created_at) requires --force`. The correct extension:

```python
class Meta(CHTableMeta):
    ch = ch_table(
        order_by=["created_at", "id", "status"],  # append-only: OK
    )
```

## What requires `--force` (recreate)

These properties trigger the recreate pipeline (DETACH source → CREATE new → INSERT INTO ... SELECT → RENAME → ATTACH) when changed:

| Change | Safety |
|--------|--------|
| Engine change (including ZK path / replica name) | CRITICAL |
| ORDER BY non-extension change (remove/reorder columns) | CRITICAL |
| PRIMARY KEY change | CRITICAL |
| PARTITION BY change | CRITICAL |
| SAMPLE BY change | INFO |
| Object type change (`table` ↔ `materialized_view`) | CRITICAL |
| MV target (`to`) change | CRITICAL |
| Column type change (incompatible) | CRITICAL |
| LowCardinality / Nullable wrapper change | CRITICAL |

### Model example: full recreate with AggregateFunction

```python
# Source column type change triggers MV recreate
# Current: amount is Float64
class Meta(CHViewMeta):
    ch = materialized_view(
        engine=merge_tree(),
        order_by="date",
        select="SELECT date, sumState(amount) AS state FROM events GROUP BY date",
    )

# If amount changes to Float32, the AggregateFunction signature changes:
#   AggregateFunction(sum, Float64) -> AggregateFunction(sum, Float32)
# This is CRITICAL and requires --force + recreate
```

**The 40GB-vs-500GB rebuild argument.** A 40GB table recreates in minutes. A 500GB table needs planning: provision a second table, backfill, verify, swap. dbwarden's recreate pipeline handles the orchestration but the cost is real. Test on a staging environment first.

## AggregateFunction signatures are incompatible state

`AggregateFunction(sum, Float64)` and `AggregateFunction(sum, Float32)` are different types. An MV that selects `sum(value)` where `value` is `Float64` produces `AggregateFunction(sum, Float64)`. If the source column type changes, the MV's aggregate type is locked: it cannot be `ALTER`ed to `Float32`. The only fix is to drop and recreate the MV.

This is not a dbwarden limitation: it is ClickHouse's columnar storage model. dbwarden will flag it as a CRITICAL change requiring a recreate.

## What dbwarden refuses entirely

dbwarden will not emit `ALTER TABLE ... MODIFY ORDER BY` for a non-extension change. It will refuse to emit `ALTER TABLE ... MODIFY PARTITION BY` (ClickHouse doesn't support it). It will refuse to emit an engine change without the recreate flag.

The error messages name the constraint and the flag required to override:
```
CRITICAL: Changing ORDER BY from (a, b) to (c, d) requires --force and
triggers a full table recreate (DETACH -> CREATE -> INSERT -> ATTACH).
```

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/
========================================================================

# ClickHouse

dbwarden treats ClickHouse as a first-class backend. Every natively supported feature is reverse-engineered, diffed, and emitted as correct DDL.

Database declarations use `DbwardenDatabase` subclasses by default. The
equivalent `database_config(...)` function API remains supported for existing
projects and integration examples.

**Before reading further:** ClickHouse's object model is fundamentally different from PostgreSQL. What is a `SET` in PG is often a `CREATE` commitment in CH. Read [Immutability](immutability.md) first.

## Documentation Sections

- [Immutability](immutability.md) : What can never change, and what forces a table recreate
- [Conventions](conventions.md) : Canonicalization, defaults-as-absence, declare-only secrets, the two-door API
- [Declaring Tables](declaring-tables.md) : The `ch_table()` builder, generated DDL, and legacy `ch_*` attrs
- [Columns & Types](columns-types.md) : Column-level Meta, `ch.field()` options, type normalization
- [MergeTree Engines](engines-mergetree.md) : Engine factories, `MergeTreeSettings`, allowed changes, rollback behavior
- [Integration Engines](engines-integration.md) : Kafka, S3, S3Queue, RabbitMQ, NATS, and the other external sources
- [Special Engines](engines-special.md) : Distributed, Buffer, Join, Set, Memory, Null, Merge, and the Log family
- [Materialized Views](materialized-views.md) : The two MV shapes, refreshable MVs, `MODIFY QUERY` vs recreate
- [Aggregating Views](aggregating-views.md) : `AggregatingMergeTree` views, target tables, populating
- [Projections & Indexes](projections-indexes.md) : Projections, skip indexes, `MATERIALIZE` as a data operation
- [Dictionaries](dictionaries.md) : Declaration, source types, layout types, lifetime
- [Named Collections](named-collections.md) : Credential-bearing collections, declare-only by design (needs `dbwarden-ch-rbac`)
- [RBAC](rbac.md) : Roles, users, row policies, quotas, settings profiles, grants (needs `dbwarden-ch-rbac`)
- [Data Operations](data-operations.md) : Partition operations, mutations, `OPTIMIZE`, `POPULATE`
- [Safety Classification](safety.md) : Classification levels, `--force`, and the recreate pipeline
- [ALTER Semantics](alter-semantics.md) : Code 517, mutations_sync vs alter_sync, statement batching
- [ON Cluster](on-cluster.md) : Cluster modes, DDL propagation, and replicated databases
- [Migration Locking](../../advanced/clickhouse-locking.md) : Lock strategies, ON CLUSTER, idempotency, and production setup

## Quick-start

Set up a table, a materialized view, and an aggregated table:

```python
from datetime import date
from sqlalchemy import func, Date
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.clickhouse import (
    CHTableMeta, CHViewMeta, ch_table, ch,
    merge_tree, materialized_view, kafka, aggregating_view, agg,
    AggregatingView, MaterializedView,
)

class Base(DeclarativeBase):
    pass

# 1. Source table
class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    amount: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["event_date", "id"],
            partition_by=func.toYYYYMM(Event.event_date),
        )

# 2. Materialized view: Mode A (class IS the target, MV is auto-generated)
class EventDaily(MaterializedView):
    __tablename__ = "event_daily"

    date: Mapped[date] = mapped_column(primary_key=True)
    total: Mapped[float] = mapped_column()
    cnt: Mapped[int] = mapped_column()

    class Meta(CHViewMeta):
        ch = materialized_view(
            select="SELECT event_date AS date, sum(amount) AS total, "
                   "count(*) AS cnt FROM events GROUP BY event_date",
            engine=merge_tree(),
            order_by=["date"],
        )

# 3. Aggregating view: sources from EventDaily model
class EventAggregated(AggregatingView):
    __tablename__ = "event_aggregated"

    class Meta(CHViewMeta):
        ch = aggregating_view(
            source=EventDaily,
            group_by=[EventDaily.date],
            aggregates=[
                agg.sum(EventDaily.total, "Float64").as_("state"),
            ],
            order_by=[EventDaily.date],
        )
```

Generate DDL and apply:

```bash
dbwarden make-migrations -d analytics
dbwarden migrate -d analytics
```

This produces: `events` (source MergeTree), `event_daily` (target MergeTree), `event_daily_mv` (MV TO event_daily), `event_aggregated` (AggregatingMergeTree target), and `event_aggregated_mv` (MV TO event_aggregated). Query the final table:

```sql
SELECT date, sumMerge(state) FROM event_aggregated GROUP BY date
```

## Version support

| Version | Status | Evidence |
|---------|--------|----------|
| 24.3 | Verified | 39 audit cases, zero drift |
| 26.6 (latest) | Verified | Same 39 cases, zero drift |

The canonicalizer has **zero version branching**: a single code path covers 24.3–26.6. This is measured fact from the multi-version audit harness, not an assumption.

## Capability matrix

| Category | Feature | Status |
|----------|---------|--------|
| Engines | MergeTree family (8 variants) | Done |
| | Distributed, Buffer | Done |
| | Kafka, S3, S3Queue, RabbitMQ, NATS | Done |
| | MySQL, PostgreSQL, MongoDB, Redis | Done |
| | URL, File, HDFS | Done |
| | Null, Memory, Merge, Set, Join, Dictionary | Done |
| | Log, TinyLog, StripeLog | Done |
| Column features | Codecs, TTL, DEFAULT/MATERIALIZED/ALIAS | Done |
| | LowCardinality, Nullable wrappers | Done |
| | Type normalization | Done |
| | REMOVE clauses (CODEC, TTL, DEFAULT, MATERIALIZED, ALIAS, COMMENT) | Done |
| Compiled expressions | `render_expr()` accepts SQLAlchemy `ColumnElement`/`ChRaw`/`str` | Done |
| | Expression fields in all specs accept `ColumnElement` | Done |
| Table features | ORDER BY, PRIMARY KEY, PARTITION BY, SAMPLE BY | Done |
| | TTL (table + column) | Done |
| | Settings | Done |
| | Comments (table + column) | Done |
| | Projections, skip indexes | Done |
| Materialized views | Class-based API (`materialized_view()` + `CHViewMeta`) | Done |
| | Forward DDL: TO target, implicit `.inner`, refreshable, POPULATE | Done |
| | Reverse engineering (`generate-models`) | Done |
| | MODIFY QUERY vs recreate | Done |
| | POPULATE (data-op) | Done |
| Dictionaries | CREATE DICTIONARY via ch_dict_* | Done |
| Aggregating views | Class-based API (`aggregating_view()` + `CHViewMeta`) | Done |
| | `agg()` namespace, `-State`/`-Merge` correspondence | Done |
| | Auto-expansion: single declaration → target + MV | Done |
| | `derive_agg_target_columns()` utility | Done |
| RBAC | Roles, users, settings profiles, quotas, row policies, grants | Done |
| | `storage != 'users.xml'` filter | Done |
| | Drop gating (plugin configuration) | Done |
| Named collections | Key-set diffed, values declare-only | Done |
| Class-based views | `ChView`, `MaterializedView`, `AggregatingView` mixin bases | Done |
| | `CHViewMeta`: Meta class for view models | Done |
| | `get_all_ch_views()`: view discovery | Done |
| | `MaterializedViewSpec`: typed spec with expression compilation | Done |
| Safety | Classify options, column, and object changes | Done |
| | --force gating for destructive changes | Done |
| | Recreate pipeline (DETACH → CREATE → INSERT → ATTACH) | Done |
| Data ops | Partition ops, mutations, OPTIMIZE, POPULATE, secret rotation | Done |

## Deliberate exclusions

These are not gaps: they are deliberate boundaries, documented with reasoning so nobody wonders "why doesn't this work" or invents syntax to fill the vacuum.

| Feature | Reason |
|---------|--------|
| **Window View** | Requires `allow_experimental_window_view`. Experimental: DDL surface can change. Building a handler introduces the first version branch into the canonicalizer. Cost of waiting is near zero (MV-handler variant when it stabilizes). |
| **LIVE VIEW** | Experimental (`allow_experimental_live_view`), effectively abandoned, superseded by refreshable MVs (already supported). |
| **ANN (vector similarity) index** | Experimental (`allow_experimental_vector_similarity_index`). When stable it is a skip-index type addition: one Literal member, one audit case. Deferred, not excluded. |
| **Full-text index** | Experimental: the flag was renamed (`allow_experimental_inverted_index` → `allow_experimental_full_text_index`). The rename alone is the argument. Deferred. |
| **Replicated database engine** | dbwarden operates *within* a database, it does not provision databases. That is orchestration, same category as `config.xml`. |
| **SYSTEM commands** | Operational concern, not schema management. Not declarable in a model. |
| **Server config (`config.xml`)** | Infrastructure. Same boundary as PostgreSQL's `postgresql.conf`. |
| **Secret values** | Declare-only by design (see [named-collections](named-collections.md)). Values are not diffed. |

## Migration locking and coordination

ClickHouse has no session-scoped user locks, no synchronous compare-and-swap on table rows, and non-transactional DDL. dbwarden implements a coordination profile system to handle this:

### Coordination profiles

| Profile | Description | Grade | Use case |
|---------|-------------|-------|----------|
| **CH-0** | Lease row with fencing token | C | Non-production, human-gated |
| **CH-1** | CH-0 + strict idempotency enforcement | C | Production with idempotent migrations |
| **CH-2** | CH-1 + Keeper-backed lock | C | Replicated deployments with clickhouse-keeper |
| **CH-3** | Migration proxy choke point | A- | Production with dedicated migration proxy |
| **CH-4** | Singleton executor | A* | Kubernetes with dedicated executor pod |

### CH-0: Lease with fencing token

The baseline profile uses a lease row in `dbwarden_lock` with a fencing token:

```sql
INSERT INTO dbwarden_lock (namespace, execution_id, fencing_token, expires_at, ...)
SELECT ...
WHERE NOT EXISTS (
    SELECT 1 FROM dbwarden_lock FINAL
    WHERE namespace = :ns AND expires_at > now()
)
```

- **Acquisition**: Atomic conditional upsert; fencing token incremented from max
- **Heartbeat**: Separate connection updates `last_heartbeat_at` every 15s
- **Expiry**: Lease expires after `ttl_seconds` (default 120s)
- **Residual risk**: Fence check and DDL statement are not atomic; a paused worker can mutate after lease loss

### CH-1: Idempotency enforcement

CH-1 adds strict idempotency checking. Non-idempotent statements are refused:

**Idempotent** (allowed):

- `CREATE TABLE IF NOT EXISTS`
- `DROP TABLE IF EXISTS`
- `ALTER TABLE ADD COLUMN IF NOT EXISTS`
- `ALTER TABLE DROP COLUMN IF EXISTS`
- Metadata-only alters (COMMENT, TTL)

**Non-idempotent** (refused):

- `RENAME` (no idempotent form)
- `MODIFY COLUMN` (rewrites data)
- `UPDATE` / `DELETE` (data mutations)

**Unknown** (treated as non-idempotent for safety)

### CH-2: Keeper-backed lock

For replicated deployments with clickhouse-keeper (ZooKeeper-compatible):

- Lock via ephemeral-sequential znodes
- Crash → session expiry → znode vanishes
- Fencing token = znode sequence number (monotonic by construction)
- Status row maintained for observability

### CH-3: Migration proxy

All migration DDL flows through a single migration proxy process:

- Proxy holds the lock (CH-2 or CH-0)
- Workers connect only to the proxy, not directly to ClickHouse
- Network-restricted migration user (proxy-only DDL access)
- Takeover: `KILL QUERY WHERE initial_user = 'dbwarden_migration'` during grace window

### CH-4: Singleton executor

Workers submit migration jobs; a single executor process executes them:

- Workers have no DDL credentials
- Executor is the only mutation channel
- Enforced by deployment platform (Kubernetes Job with `parallelism: 1`)
- Executor holds CH-2 Keeper lock as defense in depth

### Configuration

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://..."
```

### Production recommendations

1. **Start with CH-1**: Enforce idempotency for all migrations
2. **Upgrade to CH-2** if you have clickhouse-keeper (better liveness)
3. **Upgrade to CH-3** for zero-downtime production (migration proxy)
4. **Use CH-4** in Kubernetes with dedicated executor pods

### Idempotency checklist

Before enabling CH-1 in production, verify all migrations are idempotent:

```bash
# Check migration files for non-idempotent patterns
grep -E "RENAME|MODIFY COLUMN|UPDATE|DELETE" migrations/primary/*.sql

# Test migration idempotency
dbwarden migrate --database analytics
dbwarden migrate --database analytics  # Should be no-op
```

## Config keys

These are the `database_config()` / `DbwardenDatabase` parameters for ClickHouse. Exact key shapes, documented because undocumented config keys are what assistants hallucinate.

**Core keys** (no plugin required):

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://..."
    ch_cluster = "production_cluster"          # str | None (ON CLUSTER name)
    ch_replicated_database = False             # bool (Replicated database engine)
```

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `ch_cluster` | `str \| None` | `None` | Cluster name. Appends `ON CLUSTER '<name>'` to every DDL statement. Mutually exclusive with `ch_replicated_database`. |
| `ch_replicated_database` | `bool` | `False` | Use `Replicated` database engine. DDL propagates automatically via ZooKeeper; `ON CLUSTER` must be omitted. Mutually exclusive with `ch_cluster`. |

See [ON Cluster](on-cluster.md) for full details on cluster modes, DDL injection, and replicated databases.

**RBAC keys** (require `dbwarden-ch-rbac` plugin): `dbwarden plugin add dbwarden-ch-rbac`. The plugin owns both these config keys and the handlers that emit their DDL, so declaring them without it installed raises `DBWardenConfigError` at config load.

```python
from dbwarden import DbwardenDatabase
from dbwarden.databases.clickhouse import (
    NamedCollectionSpec, named_collection,
    ChRoleSpec, ChUserSpec, ChRowPolicySpec,
    ChQuotaSpec, ChSettingsProfileSpec, ChGrantSpec,
)

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://..."
    ch_named_collections = [...]       # list[NamedCollectionSpec | dict]
    ch_roles = [...]                   # list[ChRoleSpec | dict]
    ch_users = [...]                   # list[ChUserSpec | dict]
    ch_row_policies = [...]            # list[ChRowPolicySpec | dict]
    ch_quotas = [...]                  # list[ChQuotaSpec | dict]
    ch_settings_profiles = [...]       # list[ChSettingsProfileSpec | dict]
    ch_grants = [...]                  # list[ChGrantSpec | dict]
```

## Model examples

### Partitioned time-series table with TTL

```python
class PageView(Base):
    __tablename__ = "page_views"

    id: Mapped[int] = mapped_column(primary_key=True)
    url: Mapped[str] = mapped_column()
    user_id: Mapped[int] = mapped_column()
    event_time: Mapped[datetime] = mapped_column()
    duration_ms: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=replicated_merge_tree("/zk/pv", "{replica}"),
            order_by=["event_time", "user_id"],
            partition_by="toYYYYMM(event_time)",
            ttl=["event_time + toIntervalMonth(6)"],
            settings={"index_granularity": 4096},
        )
```

### Table with codecs and column TTL

```python
from datetime import datetime
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.clickhouse import (
    CHColumnMeta, CHTableMeta, ch_table, ch,
    merge_tree,
)

class SensorReading(Base):
    __tablename__ = "sensor_readings"

    sensor_id: Mapped[str] = mapped_column()
    ts: Mapped[datetime] = mapped_column()
    temp: Mapped[float] = mapped_column()
    humidity: Mapped[float] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["sensor_id", "ts"],
        )

        class temp(CHColumnMeta):
            ch = ch.field(codec="ZSTD(5)")

        class ts(CHColumnMeta):
            ch = ch.field(codec="DoubleDelta", ttl="ts + toIntervalDay(90)")

        class humidity(CHColumnMeta):
            ch = ch.field(ttl="ts + toIntervalDay(30)")
```

### Kafka ingestion with MV

```python
class KafkaEvents(Base):
    __tablename__ = "kafka_events"

    payload: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=kafka(
                named_collection="kafka_prod",
                topic_list="raw_events",
                format="JSONEachRow",
                group_name="dbwarden",
            ),
        )

class ParsedEvents(MaterializedView):
    __tablename__ = "parsed_events"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select="""
                SELECT
                    JSONExtractString(payload, 'type') AS event_type,
                    JSONExtractFloat(payload, 'value') AS value,
                    JSONExtractDateTime(payload, 'ts') AS ts
                FROM kafka_events
            """,
            to="parsed_events_dest",
        )
```

### RBAC config

```python
from dbwarden import DbwardenDatabase
from dbwarden.databases.clickhouse import (
    named_collection, ChRoleSpec, ChUserSpec, ChGrantSpec,
)

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://localhost:9000"
    ch_named_collections = [
        named_collection("ldap_auth", ldap_server="ldap.example.com"),
    ]
    ch_roles = [ChRoleSpec("analyst"), ChRoleSpec("engineer")]
    ch_users = [
        ChUserSpec(
            name="bob",
            auth="ldap BY 'ldap.example.com'",
            default_roles=("analyst",),
        ),
    ]
    ch_grants = [
        ChGrantSpec(privileges=["SELECT"], on="analytics.*", to="analyst"),
    ]
```

### S3-backed table with projection

```python
from datetime import datetime
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.clickhouse import (
    CHTableMeta, ch_table, s3, projection,
)

class S3Logs(Base):
    __tablename__ = "s3_logs"

    ts: Mapped[datetime] = mapped_column()
    level: Mapped[str] = mapped_column()
    message: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=s3(
                named_collection="s3_logs",
                path="logs/*.parquet",
                format="Parquet",
            ),
            projections=[
                projection(
                    name="by_level",
                    query="SELECT level, count() GROUP BY level",
                ),
            ],
        )
```

## Workflow examples

### Generate models from existing ClickHouse

```bash
# Point dbwarden at an existing ClickHouse instance
$ dbwarden generate-models -d analytics

# Models are written to models/analytics/*.py with ch_table() / materialized_view()
# declarations that match the live schema exactly
# (verified: 39 audit cases, zero drift)
```

### Diff and preview a migration

```bash
# Change a model (e.g., add a column), then preview
$ cat >> models/analytics.py << 'EOF'
class Event(Base):
    __tablename__ = "events"
    # ... existing columns ...
    user_agent: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["event_date", "id"],
        )
EOF

$ dbwarden make-migrations --plan -d analytics

# Output:
# ALTER TABLE events ADD COLUMN user_agent String   (INFO)
```

### Handle a destructive change

```bash
# Model changes ORDER BY to a non-extension
$ dbwarden make-migrations --plan -d analytics

# Output:
# CRITICAL: Changing ORDER BY from (a, b) to (c) requires --force

# Review the plan, then apply
$ dbwarden make-migrations --plan -d analytics
# Shows the full recreate pipeline:
#   DETACH TABLE events
#   CREATE TABLE events_new ...
#   INSERT INTO events_new SELECT * FROM events
#   RENAME TABLE ...

$ dbwarden migrate --force -d analytics
```

### Deploy RBAC changes

```bash
# Add a new role and user, drop an old one
$ dbwarden make-migrations --plan -d analytics

# Output:
# CREATE ROLE IF NOT EXISTS engineer        (INFO)
# CREATE USER IF NOT EXISTS alice ...       (INFO)
# DROP USER bob                             (WARN: gated)

$ dbwarden migrate -d analytics
```

### Materialize a projection on existing data

```python
from dbwarden.databases.clickhouse import data_op

# After adding a projection to a table with existing data:
data_op(
    name="materialize_daily_agg",
    forward="ALTER TABLE events MATERIALIZE PROJECTION daily_agg",
)
```

## Verification workflow

```bash
# Reverse-engineer a live database
$ dbwarden generate-models -d analytics

# Preview ops without writing files
$ dbwarden make-migrations --plan -d analytics

# Write and apply
$ dbwarden make-migrations -d analytics
$ dbwarden migrate -d analytics
```

Always review auto-generated migrations before applying, especially for destructive changes flagged as CRITICAL.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/materialized-views/
========================================================================

# Materialized views

## Plain MVs: two shapes

### Mode A: Class IS the target table (preferred)

The view class IS the target table; `__tablename__` is the table name.
The MV is auto-generated as `f"{__tablename__}_mv"`.  Columns, engine, and
`order_by` are required for Mode A (engine is enforced; columns and
`order_by` are strongly recommended but not strictly validated).

Columns are declared via `mapped_column` on the class.  Although no
SQLAlchemy `Base` is needed; the class is NOT a SQLAlchemy model, and
`session.query(ClassName)` will *not* work; the column descriptors are
read from `cls.__dict__` by the discovery pipeline.

```python
from datetime import date
from sqlalchemy import func
from sqlalchemy.orm import Mapped, mapped_column
from dbwarden.databases.clickhouse import MaterializedView, CHViewMeta, materialized_view, merge_tree

class EventCount(MaterializedView):
    __tablename__ = "event_counts"

    date: Mapped[date] = mapped_column(primary_key=True)
    count: Mapped[int] = mapped_column()

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            engine=merge_tree(),
            order_by=["date"],
        )
```

Generated DDL:

```sql
CREATE TABLE event_counts (date Date, count Int64)
ENGINE = MergeTree() ORDER BY date

CREATE MATERIALIZED VIEW event_counts_mv TO event_counts
AS SELECT sum(amount) AS total FROM events
```

### Mode B: Explicit `to=` target

The class IS the MV; it writes to a pre-existing target table.  No columns,
no engine, no `order_by`; the target owns its own storage.

```python
from sqlalchemy import func
from dbwarden.databases.clickhouse import MaterializedView, CHViewMeta, materialized_view

class EventCountMV(MaterializedView):
    __tablename__ = "event_counts_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            to="events_dest",
        )
```

Generated DDL:

```sql
CREATE MATERIALIZED VIEW event_counts_mv TO events_dest
AS SELECT sum(amount) AS total FROM events
```

## Refreshable MVs (24.3+)

```python
class DailyRollupMV(MaterializedView):
    __tablename__ = "daily_rollup_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            to="rollup_dest",
            refresh="EVERY 3600 SECONDS",
        )
```

Generated DDL:

```sql
CREATE MATERIALIZED VIEW daily_rollup_mv
REFRESH EVERY 3600 SECONDS
TO rollup_dest
AS SELECT sum(amount) AS total FROM events
```

Refreshable MVs (introduced in CH 24.3) replace LIVE VIEW and support:

- Periodic refresh (`EVERY n SECONDS`)
- Refresh dependencies (`DEPENDS ON`) embedded in the `refresh=` string
- Empty vs populating initial state

These options are combined in a single `refresh` string:

```python
refresh="EVERY 3600 SECONDS DEPENDS ON my_other_mv"
```

Refresh on cluster is configured at the deployment level via `--cluster-mode`.

### Refresh settings

Refreshable MVs accept a `settings` dict for refresh settings such as
`refresh_retries`:

```python
class DailyRollupMV(MaterializedView):
    __tablename__ = "daily_rollup_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            to="rollup_dest",
            refresh="EVERY 3600 SECONDS",
            settings={"refresh_retries": "5"},
        )
```

Generated DDL:

```sql
CREATE MATERIALIZED VIEW daily_rollup_mv
REFRESH EVERY 3600 SECONDS
SETTINGS refresh_retries=5
TO rollup_dest
AS SELECT sum(amount) AS total FROM events
```

## Additional model examples

### Refreshable MV with DEPENDS ON

```python
class HourlyRollupMV(MaterializedView):
    __tablename__ = "hourly_rollup_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            to="hourly_rollup_dest",
            refresh="EVERY 300 SECONDS DEPENDS ON daily_rollup_mv",
        )
```

Generated DDL:

```sql
CREATE MATERIALIZED VIEW hourly_rollup_mv
REFRESH EVERY 300 SECONDS DEPENDS ON daily_rollup_mv
TO hourly_rollup_dest
AS SELECT sum(amount) AS total FROM events
```

### MV on clustered setup

```python
class ClusterMV(MaterializedView):
    __tablename__ = "cluster_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select="SELECT hostName() AS node, count(*) AS cnt FROM events",
            to="cluster_mv_dest",
        )
```

### MV chaining (MV reading from MV)

```python
# First MV: raw -> hourly
class RawToHourlyMV(MaterializedView):
    __tablename__ = "raw_to_hourly_mv"
    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Raw.value).label("total"),
            to="hourly_dest",
        )

# Second MV: hourly -> daily
class HourlyToDailyMV(MaterializedView):
    __tablename__ = "hourly_to_daily_mv"
    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(HourlyDest.total).label("total"),
            to="daily_dest",
        )
```

## What the `.inner` table is

When an MV has no `TO target` (deprecated module-level form only), ClickHouse
creates a hidden table named ``.inner.<view_name>`` with the MV's result
schema.  dbwarden reverse-engineers this table during ``generate-models``.

The class API **does not support** implicit ``.inner.`` storage; every
MV must either be the target (Mode A) or name a target (Mode B).  The
``.inner.`` form appears only when reverse-engineering legacy MVs that were
created outside dbwarden.

## `MODIFY QUERY` vs recreate

ClickHouse supports `ALTER TABLE ... MODIFY QUERY` for refreshable MVs. For
plain (non-refreshable) MVs, the query is immutable: any change requires a
DROP + CREATE.

dbwarden classifies this as:

| MV type | `MODIFY QUERY` | Safety |
|---------|----------------|--------|
| Refreshable | Supported | INFO |
| Plain (non-refreshable) | Not supported by CH | CRITICAL: requires `--force` |

## `POPULATE` as DDL or data operation

`POPULATE` is a one-time statement that inserts existing source data into the
MV on creation. You can request it inline in the MV declaration:

```python
class EventCountMV(MaterializedView):
    __tablename__ = "event_counts_mv"

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=func.sum(Event.amount).label("total"),
            to="events_dest",
            populate=True,
        )
```

Generated DDL:

```sql
CREATE MATERIALIZED VIEW event_counts_mv TO events_dest
POPULATE
AS SELECT sum(amount) AS total FROM events
```

`POPULATE` cannot be combined with `refresh`. For a separate, reversible data
migration that runs `INSERT INTO <target> <select>` after the MV is created,
use [`data_ops.populate`](data-operations.md) instead:

```python
from dbwarden.databases.clickhouse import data_ops
from dbwarden.databases.clickhouse.materialized_view import materialized_view

spec = materialized_view(
    name="event_counts_mv",
    select="SELECT sum(amount) AS total FROM events",
    to="events_dest",
)
pop = data_ops.populate(spec)
```

## What changes are allowed

| Change | Safety |
|--------|--------|
| Add/drop MV | WARN |
| Change `TO` target | CRITICAL: requires recreate |
| Change SELECT (non-refreshable) | CRITICAL: requires recreate |
| Change SELECT (refreshable) | INFO |
| Change refresh interval | INFO |
| Toggle POPULATE | Data-op, not structural |

## Rollback behavior

MV DROP is a rollback of MV CREATE. Data created by the MV is not restored by
rollback of the DDL: you must re-POPULATE.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/named-collections/
========================================================================

# Named collections

**Requires the `dbwarden-ch-rbac` plugin:** `dbwarden plugin add dbwarden-ch-rbac`. The `ch_named_collection` object handler ships in that plugin, not in core.

Named collections are the mechanism for declaring credentials without exposing secret values. They are declared in the config layer, referenced by name from engine and RBAC specs. The examples use the supported `database_config(...)` function form for plugin-owned settings; new projects may declare the same plugin configuration on a `DbwardenDatabase` subclass.

## Declaration

```python
from dbwarden import database_config
from dbwarden.databases.clickhouse import named_collection

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://...",
    ch_named_collections=[
        named_collection(
            name="kafka_prod",
            keys={
                "sasl_username": "kafka_user",
# sasl_password is NOT declared here: it comes from the
        # secret store. See "declare-only" below.
            },
        ),
        named_collection(
            name="s3_prod",
            keys={
                "region": "us-east-1",
                "access_key_id": "AKIA...",
            },
        ),
    ],
)
```

## Key-set diffed, values declare-only

Named collections are diffed on their **key set**: what keys are declared and what values are referenced. The **values themselves are never diffed**. This is the "declare-only" principle:

- `named_collection("kafka_prod", keys={"sasl_username": "kafka_user"})` declares that a collection named `kafka_prod` should have the key `sasl_username`.
- The value `"kafka_user"` is metadata for dbwarden's diff output but is **never compared** to the server state. Secret values (`password`, `sasl_password`, `secret_access_key`) are not declared at all: they come from ClickHouse's secret store.

This means dbwarden will detect that a key exists in a model but is missing from the server, and emit `CREATE NAMED COLLECTION ...`. But it will never emit `ALTER NAMED COLLECTION` with a changed password value: it can't know the real value. The descriptor stores entries verbatim in `.entries`, and `.to_dict()` includes them, so do not put credentials in it if they must stay out of serialized artifacts.

## Additional model examples

### Named collection for Postgres engine

```python
named_collection(
    name="pg_source",
    keys={
        "connection_string": "postgresql://user:pass@pg-host:5432/db",
        "port": "5432",
    },
)
```

### Named collection with cluster and secret reference

```python
named_collection(
    name="kafka_secure",
    keys={
        "bootstrap_servers": "kafka-broker:9092",
        "sasl_mechanism": "SCRAM-SHA-256",
        "sasl_username": "ch_user",
        # sasl_password = SECRET(...) stored in ClickHouse secret store
        "security_protocol": "sasl_ssl",
    },
)
```

Engine usage:

```python
engine = kafka(
    named_collection="kafka_secure",
    topic_list="events",
    format="Avro",
    group_name="dbwarden",
)
```

### Dict config (raw path)

```python
database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://localhost:9000",
    ch_named_collections=[
        {
            "name": "s3_data",
            "keys": {
                "region": "us-east-1",
                "url": "https://s3.amazonaws.com/mybucket",
            },
        },
    ],
)
```

## Reference from engines

```python
engine = kafka(
    named_collection="kafka_prod",
    topic_list="events",
)
```

The engine gets credentials from the named collection. The named collection is referenced by name only in the engine settings (`kafka_named_collection`, `s3_named_collection`, etc.).

## Reference from RBAC

Users reference server-side authentication configuration (e.g. an LDAP
server) by name through the `auth` string; the value is declare-only and
never diffed:

```python
ChUserSpec(
    name="alice",
    auth="ldap BY 'ldap_prod'",
)
```

## What changes are allowed

| Change | Safety |
|--------|--------|
| Add named collection | INFO |
| Drop named collection | WARN (may break references) |
| Add key to declaration | INFO |
| Remove key from declaration | WARN |
| Change a non-secret value | INFO |
| Secret values | Not tracked |

## Rollback behavior

`DROP NAMED COLLECTION` rolls back as `CREATE NAMED COLLECTION`. Key changes roll back as inverse key changes.
# Named Collections

The examples use the supported `database_config(...)` function form for
plugin-owned settings. New projects may declare the same plugin configuration
on a `DbwardenDatabase` subclass.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/on-cluster/
========================================================================

# ON CLUSTER

ClickHouse supports distributed DDL via the `ON CLUSTER` clause. When set, every DDL statement is automatically propagated to all nodes in the cluster. dbwarden injects this clause for you; no manual SQL editing required.

## Configuration

### Class-based (DbwardenDatabase)

Set `ch_cluster` on your database class:

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    default = True
    database_type = "clickhouse"
    database_url_sync = "clickhouse://default:@clickhouse1:8123/analytics"
    model_paths = ["app/analytics_models"]
    ch_cluster = "production_cluster"
```

### Function-based (database_config)

```python
from dbwarden import database_config

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://default:@clickhouse1:8123/analytics",
    model_paths=["app/analytics_models"],
    ch_cluster="production_cluster",
)
```

### Config keys

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `ch_cluster` | `str \| None` | `None` | Cluster name. When set, `ON CLUSTER '<name>'` is appended to every DDL statement. |
| `ch_replicated_database` | `bool` | `False` | Use `Replicated` database engine. DDL propagates automatically via ZooKeeper; `ON CLUSTER` must be omitted. |

These are mutually exclusive. Setting both raises `ConfigurationError`.

## Three cluster modes

dbwarden supports three mutually exclusive cluster modes, resolved from the config via `ClusterContext.from_config()`:

| Mode | Config | DDL behavior | Use case |
|------|--------|--------------|----------|
| **NONE** | (default) | Bare DDL, no cluster clause | Single-node development |
| **ON_CLUSTER** | `ch_cluster = "name"` | Appends `ON CLUSTER '<name>'` to every DDL statement | Multi-node with explicit DDL distribution |
| **REPLICATED** | `ch_replicated_database = True` | DDL propagates automatically via ZooKeeper; `ON CLUSTER` must be omitted | Full replication with automatic DDL sync |

The modes are defined by the `ClusterMode` enum:

```python
class ClusterMode(Enum):
    NONE = "none"
    ON_CLUSTER = "on_cluster"
    REPLICATED = "replicated"
```

### Mutual exclusivity

`ch_cluster` and `ch_replicated_database` are mutually exclusive. Setting both raises a `ConfigurationError`:

```python
# WRONG (raises ConfigurationError)
ch_cluster = "my_cluster"
ch_replicated_database = True

# CORRECT: ON CLUSTER mode
ch_cluster = "my_cluster"
ch_replicated_database = False

# CORRECT: Replicated database mode
ch_cluster = None
ch_replicated_database = True
```

### ClusterContext

dbwarden resolves the cluster mode once per migration run into an immutable `ClusterContext`:

```python
@dataclass(frozen=True)
class ClusterContext:
    mode: ClusterMode
    cluster_name: str | None = None
```

Validation rules:

- `ON_CLUSTER` mode requires a non-empty `cluster_name`. Raises `ValueError` otherwise.
- Non-`ON_CLUSTER` modes reject any `cluster_name`. Raises `ValueError` if one is provided.

Handlers never construct or mutate this context. It is created once, passed read-only into the emit layer, and used to decorate every DDL statement.

## How DDL injection works

### The ClusterableStatement mechanism

dbwarden wraps each DDL statement in a `ClusterableStatement`, a dataclass that splits the SQL at the object name boundary:

```python
@dataclass
class ClusterableStatement:
    prefix: str      # SQL up to the ON CLUSTER insertion point
    suffix: str      # SQL after the insertion point
    supports_cluster: bool = True
```

When `render()` is called with a `ClusterContext`, it inserts `ON CLUSTER '<name>'` between prefix and suffix:

```sql
-- prefix:  CREATE TABLE db.events
-- suffix:  (id Int32, ...) ENGINE = MergeTree() ORDER BY id
-- result:  CREATE TABLE db.events ON CLUSTER 'production_cluster' (id Int32, ...) ENGINE = MergeTree() ORDER BY id
```

The `supports_cluster` flag allows specific statements to opt out of ON CLUSTER injection. When `False`, the statement passes through untouched regardless of cluster mode.

### The parsing regex

`ClusterableStatement.from_sql()` parses raw DDL using a regex that identifies the DDL verb and object name, then splits at the boundary:

```
\b(
  CREATE\s+(TABLE|MATERIALIZED\s+VIEW|DICTIONARY)
  (?:\s+IF\s+NOT\s+EXISTS)?
  |
  ALTER\s+TABLE
  |
  RENAME\s+TABLE
  |
  (?:DETACH|ATTACH)\s+TABLE
  |
  DROP\s+(TABLE|DICTIONARY)(?:\s+IF\s+EXISTS)?
)\s+(<qualified_identifier>)
```

The `<qualified_identifier>` pattern matches bare identifiers, backtick-quoted identifiers (with escape handling), and dot-qualified names like `database.table`:

```
identifier = `(?:`(?:``|[^`])+`|[a-zA-Z_][a-zA-Z0-9_]*)`
qualified_identifier = `identifier(?:\.identifier)*`
```

### Supported DDL types

| DDL type | Insertion point | Example |
|----------|----------------|---------|
| CREATE TABLE | After object name | `CREATE TABLE events ON CLUSTER 'c' (...)` |
| CREATE VIEW | After object name | `CREATE VIEW v ON CLUSTER 'c' AS ...` |
| CREATE MATERIALIZED VIEW | After object name | `CREATE MATERIALIZED VIEW mv ON CLUSTER 'c' TO ...` |
| CREATE DICTIONARY | After object name | `CREATE DICTIONARY d ON CLUSTER 'c' (...)` |
| ALTER TABLE | After object name | `ALTER TABLE events ON CLUSTER 'c' ADD COLUMN ...` |
| RENAME TABLE | After first object name | `RENAME TABLE events ON CLUSTER 'c' TO events_v2` |
| DETACH TABLE | After object name | `DETACH TABLE events ON CLUSTER 'c'` |
| ATTACH TABLE | After object name | `ATTACH TABLE events ON CLUSTER 'c'` |
| DROP TABLE | After object name | `DROP TABLE IF EXISTS events ON CLUSTER 'c'` |
| DROP DICTIONARY | After object name | `DROP DICTIONARY IF EXISTS d ON CLUSTER 'c'` |

### Fallback behavior

If the regex does not match (e.g., non-DDL statements like `INSERT INTO`), the entire string becomes the prefix with an empty suffix, and `ON CLUSTER` is appended at the end. In practice, the recreate pipeline filters non-DDL through a separate check before calling `from_sql`, so this fallback is not exercised for `INSERT` statements.

### Cluster name quoting

dbwarden quotes cluster names as ClickHouse string literals. Single quotes in the name are escaped:

```python
ch_cluster = "my-cluster-v2"  # → ON CLUSTER 'my-cluster-v2'
```

### No double spaces

The `render()` method checks whether the suffix already starts with whitespace before inserting a space separator. The output is `.strip()`'d to prevent trailing whitespace.

## Which handlers use ON CLUSTER

All ClickHouse-specific handler classes are decorated with `@emit_with_cluster`, which extracts the cluster context from the migration pipeline and stores it on the handler instance:

| Handler | Object type | Op types |
|---------|-------------|----------|
| `ChTableHandler` | Tables | `alter_ch_options`, `recreate_ch_table` |
| `ChMaterializedViewHandler` | Materialized views | `modify_mv_query`, `modify_mv_refresh` |
| `ChDictionaryHandler` | Dictionaries | `alter_ch_dict` |
| `ChProjectionHandler` | Projections | `alter_ch_projection` |
| `ChSkipIndexHandler` | Skip indexes | `alter_ch_skip_index` |
| `ChColumnHandler` | Column changes | `alter_ch_column` |
| `ChCommentHandler` | Comment changes | `alter_ch_comment` |
| `ChAggTargetHandler` | Aggregating view targets | `create_ch_agg_target`, `drop_ch_agg_target` |

Data operations (`ChDataOpHandler`) are **not** cluster-decorated. They are user-authored raw SQL and pass through unchanged.

## How cluster context flows through the pipeline

The complete flow from config to rendered SQL:

1. **Config**: `ch_cluster` / `ch_replicated_database` are set on `DatabaseEntry` or `DbwardenDatabase`
2. **Context creation**: `ClusterContext.from_config(entry)` is called once per migration run
3. **RegistryDriver.emit_all**: Passes `cluster_ctx` to every handler's `emit()` call
4. **emit_with_cluster decorator**: Pops `cluster_ctx` from kwargs, stores as `self._cluster_ctx` on the handler. Falls back to `NONE` mode if not provided.
5. **Handler emit methods**: Build `ClusterableStatement` objects and call `.render(self._cluster_ctx)` to produce the final SQL
6. **Migration output**: Both upgrade and rollback SQL contain the ON CLUSTER clause

## Rollback behavior

Every handler constructs rollback SQL the same way as upgrade SQL. The `ClusterableStatement.to_migration()` method renders both:

```python
def to_migration(self, order, ctx, rollback=None):
    forward_sql = self.render(ctx)
    if isinstance(rollback, ClusterableStatement):
        rollback_sql = rollback.render(ctx)   # cluster applied to rollback too
    elif isinstance(rollback, str):
        rollback_sql = rollback               # raw string, no cluster injection
    else:
        rollback_sql = f"-- no-op ..."
```

When rollback is a `ClusterableStatement`, it receives the same ON CLUSTER treatment. When it is a raw string, it is used verbatim.

### Example migration

With `ch_cluster = "production_cluster"`:

```sql
-- upgrade

CREATE TABLE IF NOT EXISTS events ON CLUSTER 'production_cluster' (
    id Int32,
    event_date Date,
    amount Float64
) ENGINE = MergeTree()
ORDER BY (event_date, id)
PARTITION BY toYYYYMM(event_date);

-- rollback

DROP TABLE IF EXISTS events ON CLUSTER 'production_cluster';
```

Without `ch_cluster`:

```sql
-- upgrade

CREATE TABLE IF NOT EXISTS events (
    id Int32,
    event_date Date,
    amount Float64
) ENGINE = MergeTree()
ORDER BY (event_date, id)
PARTITION BY toYYYYMM(event_date);

-- rollback

DROP TABLE IF EXISTS events;
```

## ON CLUSTER and the recreate pipeline

When a table change requires a full recreate (engine change, ORDER BY change, etc.), the recreate pipeline uses ON CLUSTER on every step. The sequence:

### Upgrade path

```sql
-- 1. Detach dependent MVs
DETACH TABLE event_daily_mv ON CLUSTER 'production_cluster';

-- 2. Create new table
CREATE TABLE events_new ON CLUSTER 'production_cluster' (...) ENGINE = MergeTree() ORDER BY ...;

-- 3. Insert data (NOT cluster-decorated; this is DML, not DDL)
INSERT INTO events_new SELECT * FROM events;

-- 4. Swap
RENAME TABLE events ON CLUSTER 'production_cluster' TO events__dbw_old, events_new ON CLUSTER 'production_cluster' TO events;

-- 5. Drop old
DROP TABLE IF EXISTS events__dbw_old ON CLUSTER 'production_cluster';

-- 6. Reattach MVs
ATTACH TABLE event_daily_mv ON CLUSTER 'production_cluster';
```

### Key detail: INSERT is not decorated

The recreate pipeline uses `_cluster_sql()` which checks if the SQL starts with a DDL verb (`CREATE|ALTER|DROP|RENAME|DETACH|ATTACH`). `INSERT INTO` does not match, so it passes through without ON CLUSTER. This is correct; `INSERT` is DML, not DDL, and ClickHouse does not support `ON CLUSTER` for DML.

### RENAME TABLE handling

For multi-pair renames like `RENAME TABLE a TO b, c TO d`, `from_sql` splits after the first qualified identifier:

```
Input:  RENAME TABLE events TO events__dbw_new, events__dbw_old TO events
Prefix: RENAME TABLE events
Suffix:  TO events__dbw_new, events__dbw_old TO events
Result: RENAME TABLE events ON CLUSTER 'c' TO events__dbw_new, events__dbw_old TO events
```

## Replicated databases

When `ch_replicated_database = True`, ClickHouse uses the `Replicated` database engine. DDL propagates automatically through ZooKeeper, so `ON CLUSTER` is omitted.

```python
class Analytics(DbwardenDatabase):
    database_type = "clickhouse"
    database_url_sync = "clickhouse://default:@clickhouse1:8123/analytics"
    ch_replicated_database = True
```

With replicated databases, you should use `Replicated*` engine variants in your models:

```python
from dbwarden.databases.clickhouse import replicated_merge_tree

class Event(Base):
    __tablename__ = "events"

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=replicated_merge_tree("/zk/pv", "{replica}"),
            order_by=["event_date", "id"],
        )
```

### Why ON CLUSTER is omitted in replicated mode

In a replicated database, ClickHouse handles DDL distribution through ZooKeeper. Adding `ON CLUSTER` would be redundant and can cause conflicts. The `ClusterContext` in `REPLICATED` mode never injects the clause; `render()` only adds `ON CLUSTER` when `mode is ClusterMode.ON_CLUSTER`.

## ON CLUSTER vs Replicated: when to use which

| Consideration | ON CLUSTER | Replicated database |
|---------------|------------|---------------------|
| DDL propagation | Explicit `ON CLUSTER` clause | Automatic via ZooKeeper |
| Engine choice | Any engine | Must use `Replicated*` variants |
| ZooKeeper dependency | Not required for DDL | Required for DDL propagation |
| Setup complexity | Lower | Higher |
| Rollback support | Full (every step is cluster-decorated) | Full (ZooKeeper propagates rollback DDL too) |
| Best for | Mixed deployments, selective clustering, existing non-replicated tables | New deployments with full replication from day one |

## Cluster name quoting

dbwarden quotes cluster names as ClickHouse string literals. Single quotes are escaped:

```python
ch_cluster = "my-cluster-v2"     # → ON CLUSTER 'my-cluster-v2'
ch_cluster = "cluster's_name"    # → ON CLUSTER 'cluster\'s_name'
```

## Diagnostics

### Checking your cluster config

```python
# Class-based
print(Analytics.ch_cluster)           # None or cluster name
print(Analytics.ch_replicated_database)  # True or False

# From a DatabaseEntry
from dbwarden.config_registry import registered_entries
for entry in registered_entries():
    print(f"{entry.database_name}: ch_cluster={entry.ch_cluster}, replicated={entry.ch_replicated_database}")
```

### Verifying ON CLUSTER appears in migrations

```bash
# Generate a migration and check for ON CLUSTER
dbwarden make-migrations --plan -d analytics | grep "ON CLUSTER"
```

### Inspecting the ClusterContext

The `ClusterContext` is resolved once per migration run. To see what mode was resolved:

```python
from dbwarden.databases.clickhouse.cluster import ClusterContext, ClusterMode

ctx = ClusterContext.from_config(entry)
print(ctx.mode)         # ClusterMode.ON_CLUSTER
print(ctx.cluster_name) # "production_cluster"
```

## Troubleshooting

### ON CLUSTER not appearing in migrations

1. Verify `ch_cluster` is set on your database class or config:
   ```python
   print(Analytics.ch_cluster)  # Should print the cluster name
   ```

2. Check that `ch_replicated_database` is not `True` (they are mutually exclusive)

3. Ensure the handler for your object type supports ON CLUSTER (see Which handlers use ON CLUSTER above)

### ConfigurationError: mutually exclusive

You've set both `ch_cluster` and `ch_replicated_database`. Pick one:

```python
# ON CLUSTER mode
ch_cluster = "production_cluster"
ch_replicated_database = False

# Replicated database mode
ch_cluster = None
ch_replicated_database = True
```

### Replication lag on DDL

ON CLUSTER DDL is fire-and-forget from ClickHouse's perspective; it sends the statement to all nodes but does not wait for completion. If you see DDL applied on some nodes but not others, check the ClickHouse server logs for errors on the lagging nodes.

### Cluster name not found

ClickHouse will error with `Cluster ... not found` if the cluster name in `ch_cluster` does not match a cluster configured in the ClickHouse server's `config.xml` or `remote_servers.xml`. Verify the cluster name:

```sql
SELECT * FROM system.clusters WHERE cluster = 'production_cluster';
```

### RENAME TABLE fails with ON CLUSTER

Multi-pair `RENAME TABLE` with ON CLUSTER requires all tables to exist on the same cluster. If one of the tables in the rename pair does not exist on all nodes, the operation will fail on some nodes. Ensure the table exists on all cluster nodes before renaming.

### Non-idempotent statements with ON CLUSTER

Under coordination profile CH-1, non-idempotent statements like `RENAME` are refused. This is independent of ON CLUSTER; the idempotency check runs before DDL injection. Restructure the migration to use idempotent forms instead.

## See also

- [Migration Locking](../../advanced/clickhouse-locking.md): Coordination profiles (CH-0 through CH-4), idempotency enforcement, and production lock strategies
- [Immutability](immutability.md): What can never change, and what forces a table recreate
- [Safety Classification](safety.md): Change classification levels and `--force`
- [Config Reference](../../reference/configuration-api.md): All `DbwardenDatabase` / `database_config()` parameters

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/projections-indexes/
========================================================================

# Projections & indexes

Both projections and secondary indexes are acceleration structures, declared inside `ch_table()`.

## Projections

```python
from dbwarden.databases.clickhouse import projection

class Meta(CHTableMeta):
    ch = ch_table(
        engine=merge_tree(),
        order_by=["event_date", "id"],
        projections=[
            projection(
                name="daily_agg",
                query="SELECT event_date, count(), sum(amount) GROUP BY event_date",
            ),
        ],
    )
```

Generated DDL (the projection is part of CREATE TABLE or added via ALTER):

```sql
CREATE TABLE events (
    event_date Date,
    id Int64,
    amount Float64,
    PROJECTION daily_agg
    (
        SELECT event_date, count(), sum(amount)
        GROUP BY event_date
        ORDER BY event_date
    )
) ENGINE = MergeTree()
ORDER BY (event_date, id)
```

`ch_projection()` parameters:

| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | `str` | Projection name |
| `select` | `list[str]` | SELECT expressions |
| `group_by` | `list[str]` | GROUP BY columns |
| `order_by` | `list[str]` | ORDER BY within projection |

## Skip indexes

```python
from dbwarden.databases.clickhouse import skip_index

class Meta(CHTableMeta):
    ch = ch_table(
        engine=merge_tree(),
        order_by=["event_date", "id"],
        indexes=[
            skip_index(
                name="payload_idx",
                columns=["payload"],
                type="tokenbf_v1(1024, 3, 0)",
                granularity=1,
            ),
            ch_index(
                name="date_bloom",
                expression="event_date",
                type="bloom_filter(0.05)",
                granularity=4,
            ),
        ],
    )
```

Generated DDL:

```sql
CREATE TABLE events (
    payload String,
    INDEX payload_idx payload TYPE tokenbf_v1(1024, 3, 0) GRANULARITY 1,
    INDEX date_bloom event_date TYPE bloom_filter(0.05) GRANULARITY 4
) ENGINE = MergeTree()
ORDER BY (event_date, id)
```

`ch_index()` parameters:

| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | `str` | Index name |
| `expression` | `str` | Column or expression to index |
| `type` | `str` | Index type + parameters |
| `granularity` | `int` | Number of granules (default 1) |

Supported index types: `minmax`, `set(max_rows)`, `bloom_filter(false_positive)`, `ngrambf_v1(n, size, hashes, seed)`, `tokenbf_v1(size, hashes, seed)`, `hypothesis`, `inverted` (experimental), `vector_similarity` (experimental).

## Additional model examples

### Two projections on one table

```python
class Orders(Base):
    __tablename__ = "orders"

    dt: Mapped[date] = mapped_column()
    product: Mapped[str] = mapped_column()
    category: Mapped[str] = mapped_column()
    revenue: Mapped[float] = mapped_column()
    qty: Mapped[int] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["dt", "product"],
            partition_by="toYYYYMM(dt)",
            projections=[
                ch_projection(
                    name="category_summary",
                    select=["category", "sum(revenue)", "sum(qty)"],
                    group_by=["category"],
                ),
                ch_projection(
                    name="product_top",
                    select=["product", "sum(revenue)"],
                    group_by=["product"],
                    order_by=["sum(revenue) DESC"],
                ),
            ],
        )
```

### Multiple index types

```python
class LogSearch(Base):
    __tablename__ = "log_search"

    ts: Mapped[datetime] = mapped_column()
    level: Mapped[str] = mapped_column()
    message: Mapped[str] = mapped_column()
    ip: Mapped[str] = mapped_column()

    class Meta(CHTableMeta):
        ch = ch_table(
            engine=merge_tree(),
            order_by=["ts", "ip"],
            indexes=[
                ch_index(
                    name="level_idx",
                    expression="level",
                    type="set(10)",
                    granularity=4,
                ),
                ch_index(
                    name="msg_bloom",
                    expression="message",
                    type="bloom_filter(0.01)",
                    granularity=1,
                ),
                ch_index(
                    name="ip_minmax",
                    expression="ip",
                    type="minmax",
                    granularity=8,
                ),
            ],
        )
```

### MATERIALIZE workflow

```python
# 1. Add projection to model
# 2. Generate migration (ADD PROJECTION is INFO)
# 3. Apply migration: dbwarden migrate -d analytics
# 4. Materialize on existing data
from dbwarden.databases.clickhouse import data_op

data_op(
    name="materialize_category_summary",
    forward="ALTER TABLE orders MATERIALIZE PROJECTION category_summary",
)
```

## MATERIALIZE as data operation

Indexes and projections written to new parts automatically, but existing parts need a `MATERIALIZE` operation:

```sql
ALTER TABLE events MATERIALIZE INDEX payload_idx
ALTER TABLE events MATERIALIZE PROJECTION daily_agg
```

dbwarden treats `MATERIALIZE` as a [data operation](data-operations.md), not DDL:

```python
with data_ops() as ops:
    ops.materialize_index("events", "payload_idx")
    ops.materialize_projection("events", "daily_agg")
```

## What changes are allowed

| Change | Safety |
|--------|--------|
| Add projection | INFO (new parts only; existing need MATERIALIZE) |
| Drop projection | WARN |
| Add index | INFO (new parts only; existing need MATERIALIZE) |
| Drop index | WARN |
| Change projection definition | CRITICAL: requires drop + recreate |
| Change index definition | CRITICAL: requires drop + recreate |

## Rollback behavior

Projections and indexes follow ALTER semantics: ADD rolls back as DROP and vice versa. MATERIALIZE is a data-op that is not structurally reversible: it is idempotent in practice.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/rbac/
========================================================================

# RBAC (roles, users, policies, quotas, profiles, grants)

**Requires the `dbwarden-ch-rbac` plugin:** `dbwarden plugin add dbwarden-ch-rbac`. The plugin owns both the `ch_*` RBAC config keys and the object handlers that emit their DDL. Core ships the spec dataclasses only, so declaring these keys without the plugin installed raises `DBWardenConfigError` when your `dbwarden.py` loads.

All RBAC objects are declared in the config layer. The examples use the
supported `database_config()` function form for plugin-owned settings; new
projects may declare the same plugin configuration on a `DbwardenDatabase`
subclass.

## Config keys

```python
from dbwarden import database_config
from dbwarden.databases.clickhouse import (
    ChRoleSpec, ChUserSpec, ChRowPolicySpec,
    ChQuotaSpec, ChSettingsProfileSpec, ChGrantSpec,
)

database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://...",
    ch_roles=[...],
    ch_users=[...],
    ch_row_policies=[...],
    ch_quotas=[...],
    ch_settings_profiles=[...],
    ch_grants=[...],
)
```

## Roles

```python
ch_roles=[
    ChRoleSpec(name="analyst"),
    ChRoleSpec(name="admin"),
]
```

Generated DDL:

```sql
CREATE ROLE IF NOT EXISTS analyst;
CREATE ROLE IF NOT EXISTS admin;
```

No diff beyond name: roles are identifiers.

## Users

```python
ch_users=[
    ChUserSpec(
        name="alice",
        # Authentication is declare-only: written at creation, never diffed.
        auth="sha256_password BY 'secret'",
        default_roles=("analyst",),
    ),
]
```

Generated DDL:

```sql
CREATE USER IF NOT EXISTS alice
IDENTIFIED WITH sha256_password BY 'secret'
HOST ANY
DEFAULT ROLE analyst;
```

## Row policies

```python
ch_row_policies=[
    ChRowPolicySpec(
        name="analyst_filter",
        table="events",
        using="event_date >= '2024-01-01'",
        to_roles=("analyst",),
    ),
]
```

Generated DDL:

```sql
CREATE ROW POLICY IF NOT EXISTS analyst_filter
ON events
AS PERMISSIVE
FOR SELECT USING event_date >= '2024-01-01'
TO analyst;
```

## Quotas

```python
ch_quotas=[
    ChQuotaSpec(
        name="monthly_reads",
        interval="1 MONTH",
        limits={"queries": 1000000, "errors": 0, "result rows": 0},
        to_roles=("analyst",),
    ),
]
```

Generated DDL:

```sql
CREATE QUOTA IF NOT EXISTS monthly_reads
FOR INTERVAL 1 MONTH
MAX QUERIES 1000000, ERRORS 0, RESULT ROWS 0
TO analyst;
```

## Settings profiles

```python
ch_settings_profiles=[
    ChSettingsProfileSpec(
        name="strict",
        settings={"max_memory_usage": "10000000000"},
    ),
]
```

Generated DDL:

```sql
CREATE SETTINGS PROFILE IF NOT EXISTS strict
SETTINGS max_memory_usage = 10000000000;
```

## Grants

```python
ch_grants=[
    ChGrantSpec(
        privileges=["SELECT", "INSERT"],
        on="analytics.events",
        to="analyst",
    ),
]
```

Generated DDL:

```sql
GRANT SELECT, INSERT ON analytics.events TO analyst;
```

## Additional model examples

### Complete RBAC config with multiple objects

```python
database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://localhost:9000",
    ch_named_collections=[
        named_collection("ldap_corp", ldap_server="ldap.corp.example.com"),
    ],
    ch_roles=[
        ChRoleSpec("readonly"),
        ChRoleSpec("analyst"),
        ChRoleSpec("admin"),
    ],
    ch_settings_profiles=[
        ChSettingsProfileSpec(
            name="strict_read",
            settings={
                "max_memory_usage": "5000000000",
                "max_result_rows": "10000",
            },
        ),
    ],
    ch_users=[
        ChUserSpec(
            name="alice",
            auth="ldap BY 'ldap.corp.example.com'",
            default_roles=("analyst",),
            settings_profile="strict_read",
        ),
        ChUserSpec(
            name="bob",
            auth="sha256_password BY 'changeme'",  # declare-only: not diffed after creation
            default_roles=("readonly",),
        ),
    ],
    ch_row_policies=[
        ChRowPolicySpec(
            name="analyst_filter",
            table="analytics.events",
            using="event_date >= '2024-01-01'",
        ),
    ],
    ch_quotas=[
        ChQuotaSpec(
            name="monthly_cap",
            interval="1 MONTH",
            limits={"queries": 100000, "errors": 0, "result rows": 0},
        ),
    ],
    ch_grants=[
        ChGrantSpec(privileges=["SELECT"], on="analytics.*", to="readonly"),
        ChGrantSpec(privileges=["SELECT", "INSERT"], on="analytics.*", to="analyst"),
        ChGrantSpec(privileges=["ALL"], on="analytics.*", to="admin"),
    ],
)
```

### Dict config (raw path)

```python
database_config(
    database_name="analytics",
    database_type="clickhouse",
    database_url_sync="clickhouse://localhost:9000",
    ch_roles=[{"name": "analyst"}, {"name": "engineer"}],
    ch_users=[{
        "name": "carol",
        "auth": "sha256_password BY 's3cret'",
        "default_roles": ("engineer",),
    }],
)
```

## `storage != 'users.xml'` filter

dbwarden refuses to manage roles, users, or any RBAC object stored in `users.xml`:

```
ERROR: Cannot manage RBAC objects stored in users.xml.
Set storage = 'replicated' or use ClickHouse-native RBAC.
```

This is checked at config load time. If the server reports that RBAC storage is `users.xml`, all RBAC operations are skipped with a clear error.

## Drop gating

`DROP USER`, `DROP ROLE`, etc. are gated by the `dbwarden-ch-rbac` plugin. The plugin must be installed and configured to allow RBAC drops.

Without the plugin or proper configuration, RBAC drop statements are skipped:

```
INFO: RBAC drop skipped: plugin not configured for drops
```

This prevents accidental deactivation of users during migration runs.

## What changes are allowed

| Change | Safety |
|--------|--------|
| Add RBAC object | INFO |
| Drop RBAC object | WARN (gated by plugin configuration) |
| Modify user settings | INFO |
| Modify grant set | INFO |
| Change row policy expression | WARN |
| Change quota interval | INFO |
| Change settings profile | INFO |
| Named collection swap | INFO |

## Rollback behavior

Every RBAC CREATE has a DROP rollback and vice versa. Settings changes revert via `ALTER USER ... SETTINGS ...`.
# ClickHouse RBAC

The examples use the supported `database_config(...)` function form for
plugin-owned settings. New projects may declare the same plugin configuration
on a `DbwardenDatabase` subclass.

========================================================================
PAGE: https://docs.dbwarden.org/databases/clickhouse/safety/
========================================================================

# Safety classification

Every change dbwarden detects is classified. Destructive operations require `--force` and emit warnings with reasons.

## Classification levels

| Level | Color | Behavior |
|-------|-------|----------|
| `INFO` | Green | Applied automatically. Safe metadata changes: ADD COLUMN, ADD INDEX, ADD PROJECTION, SETTING changes, TTL changes. |
| `WARN` | Yellow | Applied automatically but logged as a warning. Attention recommended: DROP COLUMN, DROP TABLE, DROP INDEX, mutations, partition DROP/REPLACE. |
| `CRITICAL` | Red | Skipped unless `--force` is passed. Requires explicit acknowledgement: engine changes, ORDER BY non-extension, PRIMARY KEY changes, MV TO target change, type incompatibility, LowCardinality/Nullable toggle. |

## The `--force` flag

```bash
# Preview what would run
dbwarden make-migrations --plan -d analytics

# Apply with force
dbwarden migrate --force -d analytics
```

`--force` is all-or-nothing for CRITICAL items in the plan. There is no per-item override.

## The recreate pipeline

When a CRITICAL change requires a full table rebuild, dbwarden executes:

```
DETACH TABLE source
CREATE TABLE source_new (...new definition...)
INSERT INTO source_new SELECT * FROM source
RENAME TABLE source TO source_old,
             source_new TO source
ATTACH TABLE source_old
```

Steps:
1. **DETACH**: unmounts the table metadata from the database (data remains on disk)
2. **CREATE new**: creates a table with the new definition
3. **INSERT INTO ... SELECT**: copies all data from old to new (blocking if `source` is still receiving writes)
4. **RENAME**: atomically swaps source and source_old under an exclusive lock
5. **ATTACH**: remounts the old table as a backup under its new name

After the pipeline, `source` has the new definition and `source_old` holds the original data. Verification steps:

```sql
SELECT count(*) FROM source
SELECT engine, create_table_query FROM system.tables WHERE name = 'source'
```

If something went wrong, swap back:

```sql
RENAME TABLE source TO source_broken,
             source_old TO source
```

!!! note "Non-recreate ALTERs"
    For metadata-only ALTERs (ADD COLUMN, ADD INDEX, MODIFY SETTING, etc.), the recreate pipeline does not apply. These are single-statement operations subject to the [CANNOT_ASSIGN_ALTER race](alter-semantics.md#cannot_assign_alter-code-517) on ReplicatedMergeTree. See [ALTER semantics](alter-semantics.md) for batching requirements and `mutations_sync` / `alter_sync` guidance.

## Rollback of a recreate

The rollback is the reverse pipeline:

```
DETACH TABLE source_old
CREATE TABLE source (...original definition...)
INSERT INTO source SELECT * FROM source_old
RENAME TABLE source TO source_old2,
             source_old TO source
ATTACH TABLE source_old2
```

This restores the original table and keeps the failed new table as `source_old2`.

## When to use `--force`

- **Staging/testing**: always use `--force` to verify the pipeline works
- **Production small tables** (< 40 GB): `--force` with a maintenance window
- **Production large tables** (> 40 GB): avoid `--force`. Instead, manually plan a zero-downtime migration using `clickhouse-copier` or double-write during backfill

## Additional model examples

### Model that triggers CRITICAL classification

```python
# Current table for reference:
class OldEvents(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by=["event_date", "id"])

# Change: remove a column from ORDER BY (non-extension)
class UpdatedEvents(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by=["event_date"])  # 'id' removed
```

Plan output:

```
CRITICAL: Changing ORDER BY from (event_date, id) to (event_date) requires --force
Apply with: dbwarden migrate --force -d analytics
```

### Safe change that passes without --force

```python
# Add a column and extend ORDER BY
class SafeEvents(Base):
    __tablename__ = "events"
    id: Mapped[int] = mapped_column(primary_key=True)
    event_date: Mapped[date] = mapped_column()
    status: Mapped[str] = mapped_column()
    class Meta(CHTableMeta):
        ch = ch_table(engine=merge_tree(), order_by=["event_date", "id", "status"])
```

Plan output:

```
ALTER TABLE events ADD COLUMN status String   (INFO)
ALTER TABLE events MODIFY ORDER BY (event_date, id, status)   (INFO)
```

### Recreate pipeline example with verification

```bash
# 1. Preview the recreate
$ dbwarden make-migrations --plan -d analytics

# 2. Apply
$ dbwarden migrate --force -d analytics

# 3. Verify the new table
$ clickhouse-client -q "SELECT count(*) FROM events"
$ clickhouse-client -q "SELECT engine, create_table_query FROM system.tables WHERE name = 'events'"

# 4. If rollback needed:
$ clickhouse-client -q "RENAME TABLE events TO events_broken, events_old TO events"
```

## ClickHouse-native safety features

dbwarden also respects ClickHouse's server-side `allow_ddl` and `readonly` settings. If the server refuses a DDL statement, dbwarden logs the error and continues with the remaining plan items.

========================================================================
PAGE: https://docs.dbwarden.org/databases/
========================================================================

# Supported Databases

dbwarden supports PostgreSQL (the default backend), MySQL, MariaDB, SQLite, and ClickHouse.

A **round-trip** backend is one where dbwarden can both read schema (via `generate-models`) and write schema (via `make-migrations` / `migrate`).

## Backend Matrix

| Backend | `database_type` | Typical URL | Round-Trip |
|---------|------------------|-------------|------------|
| PostgreSQL | `postgresql` | `postgresql://user:pass@host:5432/db` | Yes |
| MySQL | `mysql` | `mysql://user:pass@host:3306/db` | Yes |
| MariaDB | `mariadb` | `mariadb://user:pass@host:3306/db` | Partial |
| ClickHouse | `clickhouse` | `clickhouse://user:pass@host:8123/db` | Yes |
| SQLite | `sqlite` | `sqlite:///./app.db` | Yes |

## Optional Dependency Groups

None of the database drivers are installed by default. Install the extra for your target backend:

| Extra | Command | Driver |
|-------|---------|--------|
| `[postgres]` | `uv add "dbwarden[postgres]"` | `psycopg2-binary` |
| `[mysql]` | `uv add "dbwarden[mysql]"` | `pymysql` |
| `[clickhouse]` | `uv add "dbwarden[clickhouse]"` | `clickhouse-connect` |

See [Installation](../installation.md) for full details.

## Locking

dbwarden uses per-engine native locking to prevent concurrent schema mutation. See [Locking Architecture](locking.md) for the general model, per-backend details, and recovery procedures.

## Config Examples

The default configuration style uses `DbwardenDatabase` subclasses. The
equivalent `database_config(...)` function API remains supported for plugins
and integrations.

PostgreSQL:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/main"
```

MySQL:

```python
class Legacy(DbwardenDatabase):
    database_name = "legacy"
    database_type = "mysql"
    database_url_sync = "mysql://user:password@localhost:3306/legacy"
```

SQLite:

```python
class Dev(DbwardenDatabase):
    database_name = "dev"
    database_type = "sqlite"
    database_url_sync = "sqlite:///./development.db"
```

ClickHouse:

```python
class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://user:password@localhost:8123/analytics"
```

## Internal Connection Handling

dbwarden uses SQLAlchemy engines, with backend-specific URL normalization where needed.

Conceptual flow:

```python
def get_engine(config):
    url = config.database_url_sync
    if config.database_type == "clickhouse":
        url = normalize_clickhouse_dialect(url)
    return create_engine(url)
```

Connections include retry logic: `get_db_connection()` wraps engine connections with up to 5 attempts and linear backoff when the database is temporarily unavailable (e.g. during a restart or network hiccup). Engines are cached and reused across calls.

For PostgreSQL schema support, set `pg_schema` in `database_config(...)`. dbwarden sets `search_path` on connection so all unqualified references use that schema:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/main",
    pg_schema="app",
)
```

At the model level, set `pg_schema` on `PGTableMeta` or `PGViewMeta` to scope a specific table or view to a schema. This takes precedence over the config-level `search_path`. See [PostgreSQL Deep Dive](postgresql/index.md) for full details.

## Development Database Strategy

Recommended pattern:

- Production-like primary DB (for example PostgreSQL)
- SQLite for dev DB via `dev_database_url`
- Run local commands with `--dev`

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

```bash
$ dbwarden --dev make-migrations "sync models" -d primary
$ dbwarden --dev migrate -d primary
```

## Translation Note

When targeting SQLite in dev mode, dbwarden translates unsupported backend-specific types/defaults.

- Unknown/unsupported types fallback to `TEXT` with warnings
- `--strict-translation` turns those warnings into errors

Details: [SQL Translation](../sql-translation.md)

## Backend-Specific Notes

Each backend has deep-dive documentation:

| Backend | Guide |
|---------|-------|
| PostgreSQL | [PostgreSQL Deep Dive](postgresql/index.md) |
| MySQL / MariaDB | [MySQL Deep Dive](mysql/index.md) |
| SQLite | [SQL Databases](sql-databases.md) |
| ClickHouse | [ClickHouse Deep Dive](clickhouse/index.md) |

### PostgreSQL

PostgreSQL is a **first-class backend** with full round-trip support. All metadata: identity columns, collation, storage, compression, generated columns, fillfactor, tablespace, inheritance, exclude constraints, deferrable FKs, and advanced index options, is captured by the snapshot, diffed correctly, and emitted as valid DDL.

See [PostgreSQL Deep Dive](postgresql/index.md) for the complete reference.

### MySQL

MySQL is a **first-class backend** with full round-trip support. All metadata: engine, charset, collation, row format, auto_increment, unsigned columns, ON UPDATE, and column comments, is captured by the snapshot, diffed correctly, and emitted as valid DDL.

Key MySQL DDL behavior:

- **DDL is NOT transactional**: each statement auto-commits; partial failure possible
- Column type/nullable changes use `MODIFY COLUMN` (requires full column definition)
- Table comments use `ALTER TABLE t COMMENT = '...'` (not `COMMENT ON`)
- Column comments use `MODIFY COLUMN ... COMMENT '...'` (full column definition preserved)
- Auto-increment toggle uses `MODIFY COLUMN ... AUTO_INCREMENT`
- FK drop uses `DROP FOREIGN KEY` (not `DROP CONSTRAINT`)

See [MySQL Deep Dive](mysql/index.md) for the complete reference.

### MariaDB

MariaDB is supported as a separate `database_type` (`mariadb`). Schema layer is complete with `MdbTableMeta` / `MdbColumnMeta` and `mdb.field()` spec objects including MariaDB-specific features. Snapshot capture and reverse-engineering of MariaDB-specific features are not yet complete, so full round-trip is not yet available.

See [MySQL Deep Dive](mysql/index.md) for MariaDB-specific notes.

### SQLite

- Great for local tests and dev loops via `--dev` mode
- Limited DDL: no `ALTER COLUMN TYPE`, no `SET/DROP NOT NULL`, no FK alterations
- `--safe-type-change` emits a comment (not supported)
- Type affinity differs from server databases
- See [SQL Translation](../sql-translation.md) for dev-mode type mapping

### ClickHouse

ClickHouse has full round-trip support: `generate-models` reads schema from a live ClickHouse server, and `make-migrations` / `migrate` auto-generates DDL for table operations.

- HTTP-based wire protocol; dbwarden uses ClickHouse client, not SQLAlchemy session
- DDL operations are mostly auto-generated: table rename, column type change, nullable and LowCardinality changes, projections, RBAC objects, and engine recreation. Placeholder rollback is refused by the strict rollback contract unless the migration is explicitly irreversible.
- Full engine metadata support via `class Meta(CHTableMeta)` with `ChEngineSpec`, `ProjectionSpec`, `CHColumnMeta`
- Supports materialized views, projections, dictionaries, replicated engines
- See [ClickHouse Deep Dive](clickhouse/index.md) for full details

## Recommended Verification Workflow

```bash
# local loop on dev DB
$ dbwarden --dev migrate -d primary

# pre-release validation on production-like DB
$ dbwarden migrate -d primary
$ dbwarden status -d primary
```

========================================================================
PAGE: https://docs.dbwarden.org/databases/locking/
========================================================================

# Locking Architecture

dbwarden uses per-engine native locking to prevent concurrent schema mutation. This page explains the general architecture, why each engine needs a different strategy, and links to per-backend details.

## Why locking matters

A schema migration is a read-modify-write cycle on the database catalog. If two processes run migrations concurrently, they can:

- Apply the same migration twice (double execution)
- Interleave DDL statements in unpredictable order
- Corrupt schema state (e.g., one process creates a table, another drops it mid-transaction)

dbwarden prevents this by acquiring an exclusive lock before any DDL executes and holding it until the migration completes (or fails).

## Lock strategy by engine

Each database engine provides different native locking primitives. dbwarden uses whichever primitive the engine offers:

| Engine | Native Lock | Grade | Auto-release on crash | How it works |
|--------|-------------|-------|----------------------|--------------|
| **PostgreSQL** | Advisory lock | A | Yes (connection teardown) | `pg_advisory_lock()` on a fixed key; session-scoped, released when the connection closes |
| **MySQL/MariaDB** | Named user lock | A | Yes (connection teardown) | `GET_LOCK('dbwarden')`; released when the connection closes or `RELEASE_LOCK()` is called |
| **SQLite** | BEGIN IMMEDIATE | B | Yes (journal cleanup) | `BEGIN IMMEDIATE` acquires a write lock on the database file; held for the entire migration run |
| **ClickHouse** | Lease row + fencing token | C | After TTL expiry | INSERT + verification to `dbwarden_lock`; no native lock primitive, so dbwarden emulates one |

### Why the grades differ

- **Grade A** (PostgreSQL, MySQL): The database engine provides a session-scoped lock that auto-releases on crash. The lock is authoritative; no stale-lock recovery needed.
- **Grade B** (SQLite): The database is a single file. `BEGIN IMMEDIATE` prevents concurrent writes, but the lock is tied to the process, not the session. A crash releases it via journal cleanup, but the window is less predictable.
- **Grade C** (ClickHouse): ClickHouse has no session-scoped user locks, no synchronous compare-and-swap on table rows, and non-transactional DDL. dbwarden must emulate locking using a lease table with TTL-based expiry, which introduces a stale-lock window.

## General lock lifecycle

```
migrate
  -> acquire lock (engine-specific)
  -> write status row to dbwarden_lock
  -> start heartbeat (background thread, every 15s)
  -> execute pending migrations
  -> stop heartbeat
  -> release lock
```

If the process crashes mid-migration, the lock is released according to the engine's native behavior (connection teardown for PostgreSQL/MySQL, journal cleanup for SQLite, TTL expiry for ClickHouse).

## Observability

All engines write a status row to the `dbwarden_lock` table. This row records:

- Execution ID (unique per migration run)
- Owner identity (host, PID)
- Fencing token (monotonically increasing, prevents stale workers)
- Heartbeat timestamp (updated every 15s)
- Lease expiry (ClickHouse only)

Check lock state with:

```bash
dbwarden lock-status --database <name>
```

Health verdicts:

| Verdict | Meaning |
|---------|---------|
| HEALTHY | Lock held, heartbeat is fresh |
| STUCK | Lock held, heartbeat is stale (process may be paused or dead) |
| DEAD | Lock free, status row shows dead worker |
| AVAILABLE | No lock held |
| COMPLETE | Migration completed successfully |
| FAILED | Migration failed |
| INSPECTING | Recovery inspection in progress |
| NEEDS_REVIEW | Human intervention required |

## Recovery state machine

When a new worker acquires the lock and detects a dead predecessor, it enters the recovery state machine:

```
AVAILABLE -> RUNNING -> COMPLETE -> AVAILABLE (loop back)
                    -> FAILED -> AVAILABLE (loop back)
                    -> DEAD (detected by new acquirer)
                         -> INSPECTING
                              -> COMPLETE (all steps applied)
                              -> RUNNING (resume safely)
                              -> NEEDS_REVIEW (human decision)
                                   -> AVAILABLE (loop back)
                                   -> RUNNING (retry)
```

The INSPECTING state compares the recorded migration checksum with the candidate migration. If they match, the migration can be resumed. If they do not match, the worker transitions to NEEDS_REVIEW and waits for human intervention.

## Distributed locking (Redis)

For multi-instance deployments where multiple application replicas could trigger migrations concurrently, the `dbwarden-redis` plugin provides a Redis-backed distributed lock:

```bash
dbwarden plugin add dbwarden-redis
```

```python
from redis.asyncio import Redis
from dbwarden_redis import migration_lock

redis = Redis.from_url("redis://localhost:6379")

async with migration_lock(redis):
    await run_migration(
        sql_statements,
        version="0002",
        migration_operation="apply",
        filename="primary__0002_feature.sql",
    )
```

The Redis lock uses `SET NX EX` with a configurable TTL (default 60 seconds). The database lock and Redis lock guard different entry points (CLI vs any wrapped code path) and can be used independently or together. See [dbwarden-redis](https://github.com/dbwarden-org/dbwarden-redis) for full documentation.

## Per-backend details

- [PostgreSQL, MySQL, SQLite](../advanced/migration-locking.md): General locking guide covering lock acquisition, heartbeat, stuck lock recovery, and CI/CD patterns
- [ClickHouse](../advanced/clickhouse-locking.md): Coordination profiles (CH-0 through CH-4), why ClickHouse needs custom locking, idempotency enforcement, and production setup

## Configuration

| Key | Default | Description |
|-----|---------|-------------|
| `lock_namespace` | `"default"` | Lock scope (allows independent lock streams) |

## See also

- [ON Cluster](clickhouse/on-cluster.md): Cluster modes and DDL propagation
- [Safe Deployment](../advanced/safe-deployment.md): Deployment patterns that interact with locking
- [CI/CD Patterns](../advanced/ci-cd-patterns.md): Serialize migrations in pipelines

========================================================================
PAGE: https://docs.dbwarden.org/databases/mysql/alter-semantics/
========================================================================

# ALTER semantics

MySQL and MariaDB ALTERs behave differently from PostgreSQL or SQLite. DDL is non-transactional, `MODIFY COLUMN` silently drops any attribute you do not restate, and every statement goes through an algorithm/locking machinery that can turn a simple-looking change into a full table copy. Understanding these properties prevents the classic MySQL migration failures: half-applied migrations, lost column attributes, and metadata-lock pileups.

## Why MySQL ALTERs are special

1. **Non-transactional DDL.** Every DDL statement implicitly commits the current transaction — before *and* after it runs — and cannot be rolled back. A migration file that fails on statement 4 leaves statements 1–3 applied.

2. **`MODIFY COLUMN` requires the full column definition.** Attributes present in the original definition but not restated are dropped. A bare `MODIFY COLUMN c BIGINT` on an `INT UNSIGNED AUTO_INCREMENT COMMENT '...'` column silently removes `UNSIGNED`, `AUTO_INCREMENT`, and the comment.

3. **The algorithm/lock machinery decides the cost.** Every `ALTER TABLE` runs as `INSTANT` (metadata only), `INPLACE` (may rebuild in place), or `COPY` (full table copy), with a lock level from `NONE` to `EXCLUSIVE`. The same statement is free on one version and a multi-hour rewrite on another — and even "instant" operations must wait for a metadata lock.

## Non-transactional DDL

From [Statements That Cause an Implicit Commit](https://dev.mysql.com/doc/refman/8.0/en/implicit-commit.html):

> The statements listed in this section implicitly end any transaction active in the current session, as if you had done a `COMMIT` before executing the statement. Most of these statements also cause an implicit commit after executing.

The list covers all DDL: `ALTER TABLE`, `CREATE`/`DROP` of tables, indexes, views, triggers, `RENAME TABLE`, `TRUNCATE TABLE`, and more.

### What "atomic DDL" is not

MySQL 8.0 introduced *atomic DDL*, which is frequently misread as transactional DDL. [The manual is explicit](https://dev.mysql.com/doc/refman/8.0/en/atomic-ddl.html):

> **Atomic DDL is not transactional DDL.** DDL statements, atomic or otherwise, implicitly end any transaction that is active in the current session... DDL statements cannot be performed within another transaction, within transaction control statements such as `START TRANSACTION ... COMMIT`, or combined with other statements within the same transaction.

Atomic DDL means crash safety: data dictionary updates, storage engine operations, and binlog writes are committed or rolled back as one unit (via the hidden `mysql.innodb_ddl_log` table), even if the server halts mid-operation. It says nothing about rolling back a *successful* statement. Only InnoDB supports it. MariaDB reached the equivalent in **10.6.1** ("most \[DDL operations\] atomic, and the rest crash-safe", via `ddl_recovery.log`), with the same implicit-commit semantics.

### How dbwarden handles it

In `run_migration` (`migrations_repo.py`), the MySQL/MariaDB branch disables savepoints and executes statements individually, because DDL would implicitly commit and destroy them anyway:

> MySQL/MariaDB DDL statements implicitly commit the current transaction and destroy any active savepoints. For MySQL/MariaDB, savepoints are disabled and DDL statements are executed individually (they auto-commit).

Consequences:

- **A failed migration leaves earlier statements applied.** dbwarden records only completed migration files; reconciling a file that died halfway is a manual step — inspect the database, then either fix forward or run the file's `-- rollback` section by hand.
- **Rollback SQL is still generated for every operation** (inverse `MODIFY COLUMN` with the snapshot's full definition, inverse table-option ALTER, `DROP INDEX`, re-`ADD CONSTRAINT`), and `dbwarden rollback` / `downgrade` execute it through the same runner.
- Where an inverse cannot be produced, dbwarden writes a placeholder instead of guessing: `-- Cannot unset engine for t`, `-- MANUAL ACTION REQUIRED: ...`, or `/* REQUIRES MANUAL COLUMN DEFINITION - type info unavailable */`. With the strict rollback contract enabled, a placeholder fails generation unless the migration is annotated irreversible.

## MODIFY COLUMN drops unspecified attributes

From the [ALTER TABLE documentation](https://dev.mysql.com/doc/refman/8.0/en/alter-table.html):

> For column definition changes using `CHANGE` or `MODIFY`, the definition must include the data type and **all attributes** that should apply to the new column, other than index attributes such as `PRIMARY KEY` or `UNIQUE`. **Attributes present in the original definition but not specified for the new definition are not carried forward.**

The manual's own example: after `MODIFY col1 BIGINT` on a column defined `INT UNSIGNED DEFAULT 1 COMMENT 'my column'`, "it also drops the `UNSIGNED`, `DEFAULT`, and `COMMENT` attributes."

### How dbwarden handles it

The comment, MySQL-meta, and default change paths rebuild the complete definition with `mysql_column_definition_for_meta` (`dbwarden/engine/backends/mysql/extract.py`), always in the same clause order:

```sql
-- type -> UNSIGNED -> AUTO_INCREMENT -> NULL/NOT NULL -> DEFAULT -> COMMENT -> CHARACTER SET -> COLLATE -> ON UPDATE
ALTER TABLE users MODIFY COLUMN id INT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary key';
```

Every type used in a `MODIFY COLUMN` passes `assert_complete_mysql_type`, so a bare `VARCHAR` in a model fails fast at generation time instead of producing a syntax error mid-migration.

!!! warning "Limitation: type, nullability, and autoincrement changes emit a partial definition"

    The type-change, nullability-change, and autoincrement-toggle paths emit `MODIFY COLUMN c <type> [NULL|NOT NULL] [AUTO_INCREMENT]` without re-emitting `DEFAULT`, `COMMENT`, `ON UPDATE`, charset, or collation. MySQL drops those attributes. A subsequent `make-migrations` run re-detects the missing attributes and emits a separate `MODIFY COLUMN` to restore them — but only as a new migration. Review the generated SQL when changing the type or nullability of a column that carries attributes, and prefer splitting such changes into their own migration.

## ALGORITHM and LOCK: what the server picks

dbwarden never emits `ALGORITHM=` or `LOCK=` clauses — every statement runs with the server default, which [selects](https://dev.mysql.com/doc/refman/8.0/en/alter-table.html):

> If the `ALGORITHM` clause is omitted, MySQL uses `ALGORITHM=INSTANT` for storage engines and `ALTER TABLE` clauses that support it. Otherwise, `ALGORITHM=INPLACE` is used. If `ALGORITHM=INPLACE` is not supported, `ALGORITHM=COPY` is used.

| Algorithm | Cost | Concurrent DML |
|-----------|------|----------------|
| `INSTANT` (8.0.12+) | Metadata in the data dictionary only | Yes |
| `INPLACE` | No data copy, but may rebuild the table in place | Typically yes |
| `COPY` | Row-by-row copy into a new table | **No** |

Specifying either clause is **fail-not-fallback**: an unsupported algorithm or lock level aborts the statement with `ERROR 1845/1846 (0A000): ALGORITHM=INSTANT is not supported for this operation...` rather than silently downgrading. Only `LOCK=DEFAULT` is permitted with `ALGORITHM=INSTANT`, and `LOCK=NONE` is not permitted at all on tables with `ON ... CASCADE` or `ON ... SET NULL` foreign key constraints.

### Version milestones for INSTANT

| Capability | MySQL | MariaDB |
|------------|-------|---------|
| Instant `ADD COLUMN` (last position) | 8.0.12 | 10.3.2 |
| Instant `RENAME COLUMN` | 8.0.28 | 10.4 (definition unchanged) |
| Instant `ADD COLUMN` at any position, instant `DROP COLUMN` | 8.0.29 | 10.4 |
| Instant `VARCHAR` lengthening | never (INPLACE, same byte-width) | 10.4.3 |

Exclusions differ too: MySQL excludes `ROW_FORMAT=COMPRESSED`, FULLTEXT-indexed, temporary, and data-dictionary tables; MariaDB excludes `ROW_FORMAT=COMPRESSED` and leaves tables in a non-canonical format that older versions cannot import until rebuilt. MariaDB has no row-version limit; MySQL has one:

!!! warning "MySQL's 64 row-version limit and the 8.0.29–8.0.31 corruption bug"

    Every instant `ADD`/`DROP COLUMN` bumps the table's row version, tracked in `INFORMATION_SCHEMA.INNODB_TABLES.TOTAL_ROW_VERSIONS`. At **64**, instant column changes are rejected: `ERROR 4092 (HY000): Maximum row versions reached... Please use COPY/INPLACE.` Remedy: `OPTIMIZE TABLE` or any rebuilding `ALTER`, which resets the counter (rows also carry a per-read version-conversion cost until then). Additionally, the instant-column redo format in **8.0.29–8.0.31 has a design flaw that can corrupt tables on crash recovery** — Percona XtraBackup refuses to back up such servers. Use **8.0.32 or later** for instant add/drop column, as the [official blog](https://dev.mysql.com/blog-archive/mysql-8-0-instant-add-and-drop-columns/) recommends.

### Which dbwarden operations rebuild the table

| Operation | Server algorithm | Rebuilds? |
|-----------|------------------|-----------|
| `ADD COLUMN` | INSTANT (default) | No |
| `DROP COLUMN` | INSTANT (8.0.29+) | No |
| `RENAME COLUMN` / `RENAME TO` | INSTANT (8.0.28+) | No |
| Type change (`MODIFY COLUMN c <type>`) | **COPY only** | **Yes, full copy** |
| Nullability change | INPLACE | Yes (in place) |
| `DEFAULT` change | INSTANT (metadata) | No |
| Comment change | Metadata only | No |
| Column charset/collation change | COPY (data conversion) | **Yes, full copy** |
| `ENGINE=` | INPLACE | **Yes — even if the engine is unchanged** |
| `ROW_FORMAT=` | INPLACE | Yes (in place) |
| `AUTO_INCREMENT=N` | INPLACE | No (in-memory value) |
| `DEFAULT CHARACTER SET` (table default) | Metadata only | No (existing columns untouched) |
| Add / drop secondary index | INPLACE | No |
| `ADD FOREIGN KEY` | **COPY** unless `foreign_key_checks=0` | Yes, with checks on |
| `DROP FOREIGN KEY` | INPLACE | No |

## Metadata locks: the real production risk

[Metadata locking](https://dev.mysql.com/doc/refman/8.0/en/metadata-locking.html) holds a table's MDL until the *transaction* ends, and DDL must wait for all holders:

> A table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends.

The pileup that takes down production: write-lock requests have **higher priority** than read requests, so an `ALTER TABLE` waiting behind one long transaction blocks *every new query* on the table behind it — and this applies to `ALGORITHM=INSTANT` statements too, because every online DDL ["always requires an exclusive metadata lock in the final phase"](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-limitations.html). A failed statement inside an explicit transaction even keeps its MDLs until the transaction ends.

Mitigations:

- `lock_wait_timeout` defaults to **31536000 seconds (one year)**. Set it to seconds on the migration session (`SET SESSION lock_wait_timeout = 10`) so a stuck migration fails fast instead of queueing half the application behind it.
- Keep transactions short during deploys; kill the blocker, not the migration.
- MySQL has no `ALTER TABLE ... NOWAIT`. **MariaDB does** (10.3+): `ALTER TABLE t WAIT 10 ...` / `NOWAIT`, plus `ALTER ONLINE TABLE` as an alias for `LOCK=NONE`.

dbwarden's MySQL lock strategy uses `GET_LOCK()`/`RELEASE_LOCK()` to coordinate concurrent migration runs and refuses to run on a server in `read_only`/`super_read_only`; it does not set `lock_wait_timeout` for you.

## Online DDL failure modes

Worth knowing before running a large `INPLACE` change ([failure conditions](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-failure-conditions.html), [limitations](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-limitations.html)):

- **Concurrent DML wins.** If rows written during the ALTER violate the new definition, the ALTER fails *at the very end* of the operation — possibly with `ERROR 1062 (23000): Duplicate entry` even for a duplicate that only existed transiently in the online log.
- **`innodb_online_alter_log_max_size`** (default 128MB) caps the concurrent-DML log; exceeding it aborts the ALTER with `DB_ONLINE_LOG_TOO_BIG`.
- There is no way to pause or throttle an online DDL, and rolling back a failed one can be expensive. Kill behavior differs on MariaDB: killing the connection rolls the ALTER back "in a controlled manner", slowly.

## Replication

> Long running online DDL operations can cause replication lag. An online DDL operation must finish running on the source before it is run on the replica. — [Online DDL Limitations](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-limitations.html)

DDL is binlogged as statement events, and a single large DDL is one transaction the multithreaded applier cannot split — a one-hour `ALTER TABLE` on the source is (at least) a one-hour replica stall. **MariaDB 10.8+** can start the ALTER on replicas when it *starts* on the primary (`binlog_alter_two_phase`), eliminating that lag.

One 8.0 behavior change for hand-written SQL: partially-completed multi-object `DROP TABLE`/`DROP VIEW` statements replicated from a 5.7 source **fail** on an 8.0 replica. Always write `DROP TABLE IF EXISTS`.

## Charset conversion subtleties

dbwarden emits two different charset changes with very different costs:

- **Table default** (`my_charset` change → `ALTER TABLE t DEFAULT CHARACTER SET utf8mb4`): metadata only; existing columns keep their charset.
- **Column charset** (`my.field(charset=...)` change → `MODIFY COLUMN ... CHARACTER SET ...`): a data conversion — COPY algorithm, full table rewrite.

What dbwarden never emits, and you should avoid in hand-written SQL: `CONVERT TO CHARACTER SET`, which [silently widens types](https://dev.mysql.com/doc/refman/8.0/en/alter-table.html) ("a `VARCHAR` column might be converted to `MEDIUMTEXT`") and is **forbidden on tables with a character column used in a foreign key** while `foreign_key_checks=1` — converting only one side can corrupt data via `ON DELETE/UPDATE CASCADE` implicit conversion (Bug #45290, Bug #74816). The safe path for `latin1` columns that actually store `utf8mb4` bytes is the two-step conversion through `BLOB`. Note `utf8` is a deprecated alias for 3-byte `utf8mb3` since 8.0.29; new schemas should use `utf8mb4`. Since 8.0.14, changing a column `utf8mb3`→`utf8mb4` is metadata-only when the column is unindexed.

## Foreign keys and indexes

- **InnoDB auto-creates the FK index.** "Such an index is created on the referencing table automatically if it does not exist. This index might be silently dropped later if you create another index that can be used to enforce the foreign key constraint" ([FOREIGN KEY Constraints](https://dev.mysql.com/doc/refman/8.0/en/create-table-foreign-keys.html)).
- **`ADD FOREIGN KEY` validates existing rows** when `foreign_key_checks=1` — which forces the COPY algorithm. With `foreign_key_checks=0` it runs INPLACE, but rows written while checks were off are *never* re-validated: re-enabling does not scan the table.
- **`DROP FOREIGN KEY` vs `DROP CONSTRAINT`.** Foreign keys are dropped with `ALTER TABLE t DROP FOREIGN KEY name`; `DROP CONSTRAINT` (8.0.19+) only resolves to CHECK constraints on MySQL. MariaDB's `DROP CONSTRAINT` covers `UNIQUE`, `FOREIGN KEY`, and `CHECK`.
- **CHECK constraints are parsed and ignored before 8.0.16** — a `CHECK` "existing" on an older server enforces nothing.
- Adding and dropping a foreign key in the same `ALTER TABLE` statement requires `ALGORITHM=INPLACE` (not COPY).

!!! warning "dbwarden limitation: add-FK rollback uses DROP CONSTRAINT"

    The rollback for an added foreign key is emitted as `ALTER TABLE t DROP CONSTRAINT fk_name;` for all backends. That is valid on MariaDB, but MySQL rejects `DROP CONSTRAINT` for foreign keys — the rollback must be edited to `DROP FOREIGN KEY fk_name` before it can run on MySQL. (The forward direction of a *dropped* FK correctly uses `DROP FOREIGN KEY`.)

## AUTO_INCREMENT

- **Counter persistence.** On 5.7 the InnoDB counter lived in memory and was reinitialized to `SELECT MAX(col)+1` on restart — delete the newest rows, restart, and their ids are reissued. Since 8.0 the counter is written to the redo log and persisted in the data dictionary across restarts.
- **`ALTER TABLE t AUTO_INCREMENT = N`** (what dbwarden emits for `my_auto_increment`) cannot set the counter below the current maximum: the value is silently reset to `MAX+1`. Changing it is an in-memory, non-rebuilding operation.
- dbwarden never emits an *unset*: a model-side `my_auto_increment = None` is ignored, and the op is skipped entirely for newly created tables.

## Batching: one statement per change

MySQL's own grammar accepts comma-joined actions (`ALTER TABLE t ENGINE=InnoDB, ROW_FORMAT=DYNAMIC, ADD COLUMN ...`) — one statement, one table rebuild. **dbwarden never combines them**: every handler emits one `ALTER TABLE` per change, joined by blank lines in the migration file.

| Handler | Emits | Batching |
|---------|-------|----------|
| MyTableHandler | `ENGINE=` / `DEFAULT CHARACTER SET` / `COLLATE=` / `ROW_FORMAT=` / `AUTO_INCREMENT=` | One statement **per changed option** |
| ColumnHandler | `MODIFY COLUMN`, `ADD COLUMN`, `DROP COLUMN`, `RENAME COLUMN` | One statement per op |
| TableHandler | `ALTER TABLE t COMMENT = '...'` | One statement per op |
| ConstraintHandler | `ADD CONSTRAINT ... FOREIGN KEY`, `DROP FOREIGN KEY` | One statement per op |
| IndexHandler | `CREATE [UNIQUE] INDEX`, `DROP INDEX` | One statement per op |
| RenameTableHandler | `ALTER TABLE old RENAME TO new` | One statement per op |

The practical consequence: changing engine, charset, and row format in one migration produces **three separate `ALTER TABLE` statements, two of which rebuild the table**. For large tables, write the combined statement yourself in a `dbwarden new` migration. (Within `CREATE TABLE`, options *are* combined into one statement.)

## MariaDB divergences

- **`ALGORITHM=NOCOPY`** exists only on MariaDB, and a specified algorithm means "the least efficient algorithm you accept": `INPLACE` allows NOCOPY or INSTANT to be chosen instead. Default chain: INSTANT → NOCOPY → INPLACE → COPY. Since 11.2, even COPY can run `LOCK=NONE`.
- **Syntax extras MySQL lacks**: `IF [NOT] EXISTS` on subclauses (`ADD COLUMN IF NOT EXISTS`, `DROP INDEX IF EXISTS`, ...), `WAIT n`/`NOWAIT`, `ALTER ONLINE TABLE` ≡ `LOCK=NONE`, `DROP CONSTRAINT` for UNIQUE/FK/CHECK, and client progress reporting (`Stage: 1 of 2 'copy to tmp table'`).
- **Invisible columns** arrived in MariaDB 10.3.3 (column attribute `INVISIBLE`) vs MySQL 8.0.23 (`ALTER COLUMN c SET INVISIBLE`) — different syntax, same name.
- **dbwarden note**: the MariaDB model-side features (`mdb.field(invisible=..., sequence=...)`, `mdb_page_compressed`) are captured in models and snapshots, but no DDL emitter consumes them yet — changing them generates no `ALTER` statement.

## Operation catalog

Every ALTER shape dbwarden generates for MySQL/MariaDB, with its classification. Two systems apply: the **`--force` gate** (blocks migration generation/apply on dangerous changes) and the **impact-plan severity** (reported by `dbwarden check-impact`; unlisted operations default to INFO).

The `--force` gate flags only: drop table, drop column, and column type change (`WARNING`, requires `--force`). MySQL table-option and column-meta changes are **not gated** — review them in the generated SQL instead.

The canonical classifier uses SAFE, INFO, WARN, CRITICAL, and UNKNOWN; legacy impact labels map onto these levels. See the [safety scope contract](../../correctness/safety-scoped-migrations.md).

### Column operations

| Operation | Handler | Server algorithm | Impact severity | Example |
|-----------|---------|------------------|-----------------|---------|
| Type change | ColumnHandler | COPY (full rebuild) | WARNING | `ALTER TABLE t MODIFY COLUMN c BIGINT` |
| Nullability change | ColumnHandler | INPLACE, rebuilds | WARNING | `ALTER TABLE t MODIFY COLUMN c INT NOT NULL` |
| Default change | ColumnHandler | INSTANT (metadata) | INFO | `ALTER TABLE t MODIFY COLUMN c INT NOT NULL DEFAULT 0` |
| Comment change | ColumnHandler | Metadata only | INFO | `ALTER TABLE t MODIFY COLUMN c VARCHAR(255) COMMENT '...'` |
| my-meta change (unsigned/charset/collate/on_update) | ColumnHandler | charset: COPY; others vary | WARNING | `ALTER TABLE t MODIFY COLUMN c INT UNSIGNED` |
| Autoincrement toggle | ColumnHandler | INPLACE, `LOCK=SHARED` min. | WARNING | `ALTER TABLE t MODIFY COLUMN c INT NOT NULL AUTO_INCREMENT` |
| Add column | ColumnHandler | INSTANT | INFO | `ALTER TABLE t ADD COLUMN c INT` |
| Drop column | ColumnHandler | INSTANT (8.0.29+) | **ERROR** | `ALTER TABLE t DROP COLUMN c` |
| Rename column | ColumnHandler | INSTANT (8.0.28+) | INFO | `ALTER TABLE t RENAME COLUMN a TO b` |

### Table options (one statement per changed option)

| Option | Server algorithm | Impact severity | Example |
|--------|------------------|-----------------|---------|
| Engine | INPLACE, **always rebuilds** | INFO | `ALTER TABLE t ENGINE=InnoDB;` |
| Charset (table default) | Metadata only | INFO | `ALTER TABLE t DEFAULT CHARACTER SET utf8mb4;` |
| Collation | Metadata only | INFO | `ALTER TABLE t COLLATE=utf8mb4_unicode_ci;` |
| Row format | INPLACE, rebuilds | INFO | `ALTER TABLE t ROW_FORMAT=DYNAMIC;` |
| Auto increment | INPLACE, no rebuild | INFO | `ALTER TABLE t AUTO_INCREMENT=1000;` |
| Table comment | Metadata only | INFO | `ALTER TABLE t COMMENT = '...';` |

### Keys and indexes

| Operation | Handler | Server algorithm | Impact severity | Example |
|-----------|---------|------------------|-----------------|---------|
| Add foreign key | ConstraintHandler | COPY with checks on | INFO | `ALTER TABLE t ADD CONSTRAINT fk_t_a FOREIGN KEY (a) REFERENCES r(id);` |
| Drop foreign key | ConstraintHandler | INPLACE | WARNING | `ALTER TABLE t DROP FOREIGN KEY fk_t_a;` |
| Add index | IndexHandler | INPLACE, no rebuild | INFO | `CREATE INDEX ix_t_a ON t (a);` |
| Drop index | IndexHandler | Metadata only | WARNING | `DROP INDEX ix_t_a;` |
| Rename table | RenameTableHandler | INSTANT (metadata) | INFO | `ALTER TABLE old RENAME TO new;` |

## Related

- [MySQL & MariaDB overview](index.md): metadata declaration, snapshot format, reverse engineering
- [Migration locking](../../advanced/migration-locking.md): the `GET_LOCK()` coordination strategy
- [Rollback generation](../../correctness/rollback-generation.md): the rollback contract and placeholders
- [Common SQL databases](../sql-databases.md): cross-backend DDL behavior comparison

========================================================================
PAGE: https://docs.dbwarden.org/databases/mysql/
========================================================================

# MySQL & MariaDB

dbwarden treats MySQL (and its fork MariaDB) as **first-class backends**: every natively supported feature is reverse-engineered, diffed, and emitted as correct DDL.

Database declarations use `DbwardenDatabase` subclasses by default. The
equivalent `database_config(...)` function API remains supported for existing
projects and integration examples.

## First-Class Features

"First-class" means the round-trip is verified: reverse-engineer a live database with `generate-models`, feed the output back into `make-migrations`, and get **zero diff**.

```bash
# Step 1: reverse-engineer your live MySQL/MariaDB database
$ dbwarden generate-models -d primary

# Step 2: feed the generated models back in, zero diff
$ dbwarden make-migrations -d primary
# -> "No new migrations to generate"  (output is empty; your models match the DB exactly)
```

The following MySQL/MariaDB features are fully supported in this round-trip:

| Category | Features |
|----------|----------|
| Engine | `ENGINE=InnoDB`, `MyISAM`, etc. via `my_engine` |
| Charset & Collation | Table: `DEFAULT CHARACTER SET` / `COLLATE` via `my_charset`, `my_collate`. Column: per-column `CHARACTER SET` / `COLLATE` via `my.field(charset=..., collate=...)` |
| Row Format | `ROW_FORMAT=DYNAMIC`, `COMPACT`, `COMPRESSED`, `REDUNDANT` via `my_row_format` |
| Auto Increment | Table-level `AUTO_INCREMENT=N` via `my_auto_increment`. Column-level toggle via `autoincrement` field |
| Unsigned | `UNSIGNED` on integer columns via `my.field(unsigned=True)` |
| ON UPDATE | `ON UPDATE CURRENT_TIMESTAMP` via `my.field(on_update="CURRENT_TIMESTAMP")` |
| Comments | Table: `ALTER TABLE t COMMENT = '...'`. Column: `MODIFY COLUMN ... COMMENT '...'` (full column definition preserved) |
| Foreign Keys | `ON DELETE` / `ON UPDATE` options; DROP uses `DROP FOREIGN KEY` (MySQL syntax) |
| Indexes | Full index support; `USING BTREE / HASH` preserved |
| Auto-increment Lifecycle | Toggle autoincrement on integer PKs via `autoincrement` field: generates `MODIFY COLUMN ... AUTO_INCREMENT` |
| Type Normalization | `TINYINT(1)` -> `BOOLEAN`, `INT`, `BIGINT`, `VARCHAR(n)`, `TEXT`, `DATETIME`, `TIMESTAMP`, `YEAR`, `DECIMAL(p,s)`, `FLOAT`, `DOUBLE`, `BLOB`, `JSON`, `ENUM`, `SET` |

### MariaDB-Specific Features

| Category | Features |
|----------|----------|
| Page Compression | `PAGE_COMPRESSED=1` / `PAGE_COMPRESSION_LEVEL=N` via `mdb_page_compressed`, `mdb_page_compression_level` on `MdbTableMeta` |
| Invisible Columns | Column invisibility via `mdb.field(invisible=True)` on `MdbColumnMeta` |
| Sequences | `CREATE SEQUENCE` support via `mdb.field(sequence=...)` |

## Installation

Install with the MySQL driver:

```bash
uv add "dbwarden[mysql]"
```

Or with uv:

```bash
uv add "dbwarden[mysql]"
```

## Configuration

The MySQL backend is enabled by setting `database_type="mysql"` (or `database_type="mariadb"`) in your dbwarden config:

```python
from dbwarden import database_config

database_config(
    database_name="primary",
    default=True,
    database_type="mysql",
    database_url_sync="mysql+pymysql://user:password@localhost:3306/mydb",
)
```

The connection URL uses the `mysql+pymysql://` scheme (the `pymysql` driver is included via the `[mysql]` extra). You can also use any SQLAlchemy-compatible MySQL driver such as `mysql+mysqlconnector://`.

## Declaring Metadata

MySQL/MariaDB metadata is declared in a `class Meta` inner class on the model. This is the **only** supported surface: `mapped_column(info=...)` raises `DBWardenConfigError`.

### Table-Level Meta

Inherit from `MyTableMeta` on your `class Meta`:

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(MyTableMeta):
        my_engine = "InnoDB"
        my_charset = "utf8mb4"
        my_collate = "utf8mb4_unicode_ci"
        my_row_format = "DYNAMIC"
        my_auto_increment = 1000
        comment = "Core user accounts"
```

`MyTableMeta` inherits from `TableMeta`, which provides common attributes shared across all backends:

| Attribute | Type | SQL |
|-----------|------|-----|
| `comment` | `str` | `ALTER TABLE t COMMENT = '...'` |
| `indexes` | `list[dict]` | `CREATE INDEX ...` |
| `checks` | `list[dict]` | `ALTER TABLE t ADD CONSTRAINT ... CHECK (...)` |
| `uniques` | `list[dict]` | `ALTER TABLE t ADD CONSTRAINT ... UNIQUE (...)` |

MySQL-specific `MyTableMeta` attributes:

| Attribute | Type | SQL |
|-----------|------|-----|
| `my_engine` | `str` | `ALTER TABLE t ENGINE = name` |
| `my_charset` | `str` | `ALTER TABLE t DEFAULT CHARACTER SET name` |
| `my_collate` | `str` | `ALTER TABLE t COLLATE = name` |
| `my_row_format` | `str` | `ALTER TABLE t ROW_FORMAT = name` |
| `my_auto_increment` | `int` | `ALTER TABLE t AUTO_INCREMENT = N` |

For MariaDB, use `MdbTableMeta`:

```python
from dbwarden.databases.mariadb import MdbTableMeta

class Meta(MdbTableMeta):
    my_engine = "InnoDB"
    mdb_page_compressed = True
    mdb_page_compression_level = 3
```

MariaDB-specific `MdbTableMeta` attributes (in addition to all `MyTableMeta` attributes):

| Attribute | Type | SQL |
|-----------|------|-----|
| `mdb_page_compressed` | `bool` | `PAGE_COMPRESSED=1` |
| `mdb_page_compression_level` | `int` | `PAGE_COMPRESSION_LEVEL=N` |

### Column-Level Meta

Use `MyColumnMeta` inner classes for per-column metadata. The inner class must be named after the column. Use `my = my.field(...)` to set column-level options:

```python
from sqlalchemy import Integer, String, Text, TIMESTAMP
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta, MyColumnMeta, my

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    bio: Mapped[str] = mapped_column(Text)
    updated_at: Mapped[str] = mapped_column(TIMESTAMP)

    class Meta(MyTableMeta):
        class id(MyColumnMeta):
            comment = "Primary key"
            my = my.field(unsigned=True)

        class email(MyColumnMeta):
            my = my.field(charset="utf8mb4", collate="utf8mb4_unicode_ci")

        class updated_at(MyColumnMeta):
            my = my.field(on_update="CURRENT_TIMESTAMP")
```

`MyColumnMeta` includes common column attributes shared across all backends:

| Attribute | Type | SQL |
|-----------|------|-----|
| `comment` | `str` | `MODIFY COLUMN ... COMMENT '...'` |
| `public` | `bool` | Controls field visibility in `@auto_schema` (requires `dbwarden-fastapi` plugin) |
| `my` | `MyFieldSpec` | MySQL-specific column options (see table below) |

MySQL-specific `MyFieldSpec` fields (set via `my.field(...)`):

| Keyword | Type | SQL |
|---------|------|-----|
| `unsigned` | `bool` | `UNSIGNED` on integer columns |
| `charset` | `str` | `CHARACTER SET name` (per-column charset) |
| `collate` | `str` | `COLLATE name` (per-column collation) |
| `on_update` | `str` | `ON UPDATE CURRENT_TIMESTAMP` (typically on TIMESTAMP columns) |

For MariaDB, use `MdbColumnMeta` and `mdb.field(...)`:

```python
from dbwarden.databases.mariadb import MdbColumnMeta
from dbwarden.databases.mariadb import mdb

class Meta(MdbTableMeta):
    class id(MdbColumnMeta):
        mdb = mdb.field(invisible=True)
```

MariaDB-specific `MdbFieldSpec` fields (set via `mdb.field(...)`):

| Keyword | Type | SQL |
|---------|------|-----|
| `invisible` | `bool` | `ALTER TABLE ... ALTER COLUMN c SET INVISIBLE` |
| `sequence` | `str` | Sequence name for MariaDB sequence support |
| `unsigned` | `bool` | `UNSIGNED` on integer columns |
| `charset` | `str` | `CHARACTER SET name` |
| `collate` | `str` | `COLLATE name` |
| `on_update` | `str` | `ON UPDATE CURRENT_TIMESTAMP` |

### Foreign Key Options

Foreign key options (`ondelete`, `onupdate`) are captured from the database by `generate-models` and emitted in the `ForeignKey` constructor:

```python
from sqlalchemy import ForeignKey
from sqlalchemy.orm import Mapped, mapped_column

class OrderItem(Base):
    __tablename__ = "order_items"

    order_id: Mapped[int] = mapped_column(ForeignKey("orders.id", ondelete="CASCADE"), nullable=False)
```

### Model Example (Generated)

Here is the complete generated model output for a MySQL table with engine, charset, unsigned PK, ON UPDATE, and per-column charset:

```python
from sqlalchemy import BigInteger, Column, Integer, String, TIMESTAMP, Text, text
from sqlalchemy.orm import DeclarativeBase

Base = declarative_base()

from dbwarden.databases.mysql import MyColumnMeta, MyTableMeta, my

class User(Base):
    __tablename__ = 'users'

    id = Column('id', Integer, primary_key=True, nullable=False)
    email = Column('email', String(255), nullable=False)
    bio = Column('bio', Text)
    updated_at = Column('updated_at', TIMESTAMP, nullable=False, server_default=text('CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP'))

    class Meta(MyTableMeta):
        my_engine = 'InnoDB'
        my_charset = 'utf8mb4'
        my_collate = 'utf8mb4_unicode_ci'
        my_row_format = 'DYNAMIC'
        comment = 'Core user accounts'

        class id(MyColumnMeta):
            comment = 'Primary key'
            my = my.field(unsigned=True)

        class email(MyColumnMeta):
            my = my.field(charset='utf8mb4', collate='utf8mb4_unicode_ci')

        class updated_at(MyColumnMeta):
            my = my.field(on_update='CURRENT_TIMESTAMP')
```

## DDL Behavior

For the semantics behind these statements — implicit commits, algorithm and lock selection, metadata locking, replication lag, and version-specific gotchas — see [ALTER Semantics](alter-semantics.md).

### DDL Is NOT Transactional

MySQL and MariaDB DDL is **non-transactional**: each DDL statement implicitly commits the current transaction. If a migration file contains multiple statements and one fails, the prior DDL cannot be rolled back. This makes MySQL/MariaDB more fragile than PostgreSQL for automated migration runs.

### Column Type Changes

Emits `ALTER TABLE t MODIFY COLUMN c newtype`:

```sql
ALTER TABLE users MODIFY COLUMN email VARCHAR(255);
```

Unlike PostgreSQL, MySQL drops any column attribute not restated in the `MODIFY COLUMN` definition. The comment, MySQL-meta, and default change paths re-emit the complete definition (type, unsigned, nullable, default, comment, charset, collate, auto_increment); the type-change and nullable-change paths emit only the type and nullability — see [ALTER Semantics](alter-semantics.md#modify-column-drops-unspecified-attributes) for the consequences.

### Column Nullable Changes

Emits `ALTER TABLE t MODIFY COLUMN c type [NULL | NOT NULL]`, again with the full column type.

### Column Meta Changes

When MySQL-specific column metadata changes (unsigned, charset, collate, on_update), dbwarden generates a full `MODIFY COLUMN` that preserves the column's type, nullable, default, comment, and autoincrement state:

```sql
ALTER TABLE users MODIFY COLUMN id INT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary key';
```

### Table Option Changes

MySQL table-level option changes generate individual `ALTER TABLE` statements:

| Change | Generated SQL |
|--------|---------------|
| Engine | `ALTER TABLE t ENGINE = InnoDB` |
| Charset | `ALTER TABLE t DEFAULT CHARACTER SET utf8mb4` |
| Collation | `ALTER TABLE t COLLATE = utf8mb4_unicode_ci` |
| Row Format | `ALTER TABLE t ROW_FORMAT = DYNAMIC` |
| Auto Increment | `ALTER TABLE t AUTO_INCREMENT = 1000` |

### Auto-increment Lifecycle

dbwarden supports toggling auto-increment on integer primary key columns. The `autoincrement` field in your model controls whether a column uses auto-increment:

```python
class User(Base):
    __tablename__ = "users"

    # Autoincrement enabled: same as default behavior
    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)

    class Meta(MyTableMeta):
        class id(MyColumnMeta):
            pass  # uses default autoincrement from model
```

To explicitly disable auto-increment on a PK column:

```python
class User(Base):
    __tablename__ = "users"

    # Plain integer PK: no auto-increment
    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=False)
```

**What happens when autoincrement changes:**

| Change | Generated SQL |
|--------|---------------|
| Add autoincrement | `ALTER TABLE t MODIFY COLUMN c INT NOT NULL AUTO_INCREMENT` |
| Remove autoincrement | `ALTER TABLE t MODIFY COLUMN c INT NOT NULL` |

### Comments

Unlike PostgreSQL, MySQL has no `COMMENT ON` syntax. dbwarden generates the correct MySQL syntax:

```sql
-- Table comment
ALTER TABLE users COMMENT = 'Core user accounts';

-- Column comment (full MODIFY COLUMN preserving all attributes)
ALTER TABLE users MODIFY COLUMN email VARCHAR(255) NOT NULL COMMENT 'User email address';
```

When a comment is cleared, MySQL syntax is used:

```sql
ALTER TABLE users COMMENT = '';
ALTER TABLE users MODIFY COLUMN email VARCHAR(255) NOT NULL COMMENT '';
```

## Snapshot Format

When `database_type` is `"mysql"` or `"mariadb"`, the snapshot captures MySQL-specific metadata in `my_column` and `my_table` blocks.

### Column Extras

```json
{
  "columns": {
    "id": {
      "name": "id",
      "type": "int",
      "nullable": false,
      "default": null,
      "autoincrement": true,
      "primary_key": true,
      "comment": "Primary key",
      "my_column": {
        "my_unsigned": true,
        "my_charset": null,
        "my_collate": null,
        "my_on_update": null
      }
    },
    "updated_at": {
      "name": "updated_at",
      "type": "timestamp",
      "nullable": false,
      "default": "CURRENT_TIMESTAMP",
      "autoincrement": false,
      "primary_key": false,
      "my_column": {
        "my_unsigned": false,
        "my_on_update": "CURRENT_TIMESTAMP"
      }
    }
  }
}
```

### Table Extras

```json
{
  "my_table": {
    "my_engine": "InnoDB",
    "my_charset": "utf8mb4",
    "my_collate": "utf8mb4_unicode_ci",
    "my_row_format": "Dynamic",
    "my_auto_increment": 1000
  }
}
```

For MariaDB, additional fields appear:

```json
{
  "my_table": {
    "mdb_page_compressed": false,
    "mdb_page_compression_level": null
  }
}
```

## Reverse Engineering

`generate-models` queries `information_schema.TABLES` and `information_schema.COLUMNS` to reverse-engineer all MySQL/MariaDB metadata. The emitted model uses `class Meta` with `MyTableMeta` and `MyColumnMeta` inner classes.

```bash
$ dbwarden generate-models -d primary
```

Generated output includes automatic detection of:
- Engine, charset, collation, row format from `information_schema.TABLES`
- Column unsigned, charset, collation, on_update from `information_schema.COLUMNS`
- Foreign key options (`ON DELETE`, `ON UPDATE`)
- Auto-increment columns
- Column comments

## Safety Classification

dbwarden classifies migration changes using the `Safety` enum:

```python
from dbwarden.engine.safety import Safety

assert Safety.SAFE == "SAFE"
assert Safety.INFO == "INFO"
assert Safety.WARN == "WARN"
assert Safety.CRITICAL == "CRITICAL"
```

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add column | `INFO` | None |
| Drop column | `WARN` | `--force` |
| Change column type | `WARN` | `--force` |
| Change column nullable | `WARN` | `--force` |
| Change column comment | `INFO` | None |
| Change MySQL column meta | `WARN` | `--force` |
| Change engine | `INFO` | None |
| Change charset | `INFO` | None |
| Change collation | `INFO` | None |
| Change row format | `INFO` | None |
| Change auto_increment | `INFO` | None |
| Change table comment | `INFO` | None |
| Add / drop index | `INFO` / `WARNING` | `--force` |
| Add / drop FK | `INFO` / `WARNING` | `--force` |

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/config-keys/
========================================================================

# Config Keys

PostgreSQL features are configured through keys on your `DbwardenDatabase`
class or in the equivalent `database_config(...)` call. These keys live
alongside the connection URL and model paths.

**Most of these keys are contributed by plugins.** The plugin owns both the config key and the object handler that emits its DDL. Declaring a key whose plugin is not installed raises `DBWardenConfigError` when your `dbwarden.py` loads, naming the plugin to install. Keys that no plugin owns are rejected as unknown arguments, so typos fail immediately instead of being silently ignored.

| Config key | Required plugin |
|---|---|
| `pg_roles`, `pg_default_privileges` | `dbwarden-pgsql-rbac` |
| `pg_domains`, `pg_sequences`, `pg_composite_types` | `dbwarden-pgsql-types` |
| `pg_extensions`, `pg_functions`, `pg_triggers`, `pg_event_triggers`, `pg_extended_statistics` | `dbwarden-pgsql-extensions` |
| `pg_schema`, `pg_migration_lock_timeout` | None, core |

Install with `dbwarden plugin add <name>`.

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/mydb",
    # Config keys below
    pg_schema="app",
    pg_extensions=["uuid-ossp", "pgcrypto"],
    pg_roles=[...],
    pg_domains=[...],
    pg_sequences=[...],
    pg_functions=[...],
    pg_triggers=[...],
    pg_default_privileges=[...],
    pg_composite_types=[...],
    pg_extended_statistics=[...],
    pg_event_triggers=[...],
    pg_migration_lock_timeout=30,
)
```

## `pg_schema`

Default schema for unqualified table references. Sets the connection's `search_path`.

| Type | Default |
|------|---------|
| `str \| None` | `None` |

```python
pg_schema="app"
```

## `pg_extensions`

SQL extensions to create (equivalent to `CREATE EXTENSION IF NOT EXISTS`).

| Type | Default |
|------|---------|
| `list[str]` | `[]` |

```python
pg_extensions=["uuid-ossp", "pgcrypto", "postgis"]
```

Generated DDL: `CREATE EXTENSION IF NOT EXISTS "uuid-ossp";`

## `pg_roles`

Roles to create or alter. Each entry supports PostgreSQL role options.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_roles=[
    {"name": "app_user", "login": True, "password": "encrypted"},
    {"name": "readonly", "login": True, "connection_limit": 5},
]
```

Generated DDL: `CREATE ROLE app_user WITH LOGIN PASSWORD 'encrypted';`

### Role Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Role name |
| `login` | `bool` | `LOGIN` / `NOLOGIN` |
| `password` | `str` | `PASSWORD` (plain or `encrypted`) |
| `superuser` | `bool` | `SUPERUSER` / `NOSUPERUSER` |
| `createdb` | `bool` | `CREATEDB` / `NOCREATEDB` |
| `createrole` | `bool` | `CREATEROLE` / `NOCREATEROLE` |
| `inherit` | `bool` | `INHERIT` / `NOINHERIT` |
| `replication` | `bool` | `REPLICATION` / `NOREPLICATION` |
| `bypassrls` | `bool` | `BYPASSRLS` / `NOBYPASSRLS` |
| `connection_limit` | `int` | `CONNECTION LIMIT n` |
| `valid_until` | `str` | `VALID UNTIL 'timestamp'` |
| `in_role` | `str` | `IN ROLE parent_role` |
| `membership` | `list[str]` | `IN GROUP members` |

## `pg_domains`

Domain type declarations.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_domains=[
    {
        "name": "us_postal_code",
        "type": "text",
        "not_null": True,
        "check": r"VALUE ~ '^\d{5}(-\d{4})?$'",
    },
]
```

Generated DDL:
```sql
CREATE DOMAIN us_postal_code AS text NOT NULL CHECK (VALUE ~ '^\d{5}(-\d{4})?$');
```

### Domain Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Domain name |
| `type` | `str` | Base type |
| `schema` | `str` | Schema (optional) |
| `default` | `str` | Default expression |
| `not_null` | `bool` | `NOT NULL` constraint |
| `check` | `str` | `CHECK` expression |

## `pg_sequences`

Sequence declarations.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_sequences=[
    {
        "name": "order_number_seq",
        "start": 1000,
        "increment": 1,
        "minvalue": 1,
        "maxvalue": 999999,
        "cycle": True,
        "owned_by": None,
    },
]
```

Generated DDL:
```sql
CREATE SEQUENCE order_number_seq START WITH 1000 INCREMENT BY 1 MINVALUE 1 MAXVALUE 999999 CYCLE;
```

### Sequence Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Sequence name |
| `schema` | `str` | Schema (optional) |
| `start` | `int` | `START WITH` |
| `increment` | `int` | `INCREMENT BY` |
| `minvalue` | `int` | `MINVALUE` |
| `maxvalue` | `int` | `MAXVALUE` |
| `cycle` | `bool` | `CYCLE` / `NO CYCLE` |
| `owned_by` | `str \| None` | `OWNED BY table.column` |

## `pg_functions`

Function declarations. Supports SQL, PL/pgSQL, and other languages.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_functions=[
    {
        "name": "update_timestamp",
        "language": "plpgsql",
        "body": """
            BEGIN
                NEW.updated_at = NOW();
                RETURN NEW;
            END;
        """,
        "returns": "trigger",
        "args": [],
    },
]
```

### Function Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Function name |
| `schema` | `str` | Schema (optional) |
| `language` | `str` | Language (`sql`, `plpgsql`, `c`, etc.) |
| `body` | `str` | Function body |
| `returns` | `str` | Return type |
| `args` | `list[dict]` | Arguments: `[{"name": "x", "type": "int"}]` |
| `volatility` | `str` | `VOLATILE`, `STABLE`, or `IMMUTABLE` |
| `security_definer` | `bool` | `SECURITY DEFINER` |
| `leakproof` | `bool` | `LEAKPROOF` |
| `parallel` | `str` | `PARALLEL UNSAFE`, `RESTRICTED`, or `SAFE` |
| `cost` | `int` | `COST` |
| `rows` | `int` | `ROWS` (for `RETURNS SETOF`) |

## `pg_triggers`

Trigger declarations. Each trigger references a table and an existing function.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_triggers=[
    {
        "name": "trg_users_updated_at",
        "table": "users",
        "function": "update_timestamp",
        "timing": "BEFORE",
        "events": ["UPDATE"],
        "for_each": "ROW",
    },
]
```

Generated DDL:
```sql
CREATE TRIGGER trg_users_updated_at BEFORE UPDATE ON users FOR EACH ROW EXECUTE FUNCTION update_timestamp();
```

### Trigger Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Trigger name |
| `table` | `str` | Table name |
| `schema` | `str` | Schema (optional) |
| `function` | `str` | Function to execute |
| `func_schema` | `str` | Function schema (optional) |
| `timing` | `str` | `BEFORE`, `AFTER`, or `INSTEAD OF` |
| `events` | `list[str]` | `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE` |
| `for_each` | `str` | `ROW` or `STATEMENT` |
| `condition` | `str` | `WHEN` clause (optional) |
| `args` | `list[str]` | Arguments passed to function |

## `pg_default_privileges`

Default privileges applied per schema, role, or object type.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_default_privileges=[
    {
        "schema": "public",
        "role": "app_user",
        "kind": "TABLES",
        "privileges": "SELECT, INSERT, UPDATE, DELETE",
    },
]
```

Generated DDL:
```sql
ALTER DEFAULT PRIVILEGES FOR ROLE app_user IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_user;
```

### Default Privilege Keys

| Key | Type | Description |
|-----|------|-------------|
| `schema` | `str` | Schema name |
| `role` | `str` | Target role |
| `kind` | `str` | Object type: `TABLES`, `SEQUENCES`, `FUNCTIONS`, `TYPES`, `SCHEMAS` |
| `privileges` | `str` | Comma-separated privileges |

## `pg_composite_types`

Composite type declarations.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_composite_types=[
    {
        "name": "address",
        "columns": [
            {"name": "street", "type": "text"},
            {"name": "city", "type": "text"},
            {"name": "zip", "type": "text"},
        ],
    },
]
```

Generated DDL:
```sql
CREATE TYPE address AS (street text, city text, zip text);
```

### Composite Type Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Type name |
| `schema` | `str` | Schema (optional) |
| `columns` | `list[dict]` | List of `{"name": ..., "type": ...}` |

## `pg_extended_statistics`

Extended statistics objects for the query planner (PG 14+).

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_extended_statistics=[
    {
        "name": "stats_users_email_city",
        "table": "users",
        "kinds": ["d", "f"],
        "columns": "email, city",
    },
]
```

Generated DDL:
```sql
CREATE STATISTICS stats_users_email_city (ndistinct, dependencies) ON email, city FROM users;
```

### Extended Statistics Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Statistics name |
| `table` | `str` | Table name |
| `schema` | `str` | Schema (optional) |
| `kinds` | `list[str]` | Kind codes: `d` (ndistinct), `f` (dependencies), `m` (MCV), `e` (expressions, PG 14+) |
| `columns` | `str` | Comma-separated column names |
| `expressions` | `list[str]` | Expression columns (PG 14+) |

## `pg_event_triggers`

Event triggers fired on DDL events at the database level.

| Type | Default |
|------|---------|
| `list[dict]` | `[]` |

```python
pg_event_triggers=[
    {
        "name": "trg_ddl_audit",
        "event": "ddl_command_start",
        "function": "audit_ddl",
        "tags": ["CREATE TABLE", "ALTER TABLE"],
    },
]
```

Generated DDL:
```sql
CREATE EVENT TRIGGER trg_ddl_audit ON ddl_command_start WHEN TAG IN ('CREATE TABLE', 'ALTER TABLE') EXECUTE FUNCTION audit_ddl();
```

### Event Trigger Keys

| Key | Type | Description |
|-----|------|-------------|
| `name` | `str` | Trigger name |
| `event` | `str` | Event: `ddl_command_start`, `ddl_command_end`, `sql_drop`, `table_rewrite` |
| `function` | `str` | Function to execute |
| `func_schema` | `str` | Function schema (optional) |
| `tags` | `list[str]` | DDL command tags to filter (optional) |
| `enabled` | `str` | `O` (enabled), `D` (disabled), `R` (replica), `A` (always) |

## `pg_migration_lock_timeout`

Timeout (milliseconds) for `LOCK TABLE` statements during migration DDL to prevent indefinite blocking.

| Type | Default |
|------|---------|
| `int \| None` | `None` |

```python
pg_migration_lock_timeout=5000
```

When set, emits `SET lock_timeout = '5000ms'` before each migration statement.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/constraints/
========================================================================

# Constraints

Constraints are handled by `ConstraintHandler` during the DIFF phase.

## Foreign Keys

FK comparison uses a signature tuple: `(columns, ref_table, ref_columns, on_delete, on_update, deferrable, match)`.

### MATCH Behaviour

PostgreSQL supports three FK match modes: `MATCH FULL`, `MATCH PARTIAL`, and `MATCH SIMPLE` (default).

- `MATCH FULL` is explicitly emitted in DDL
- `MATCH PARTIAL` is explicitly emitted
- `MATCH SIMPLE` and absent match are canonicalized to the same representation (no match clause) to avoid churn on existing FKs

```sql
ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (ref_id) REFERENCES ref (id) MATCH FULL;
ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (ref_id) REFERENCES ref (id) ON DELETE CASCADE;
```

`MATCH PARTIAL` is defined by the SQL standard but PostgreSQL implements it identically to `MATCH SIMPLE`. Both allow some columns in a multi-column FK to be NULL; `MATCH FULL` requires all columns to be NULL or all to be non-NULL.

### ON DELETE / ON UPDATE

Supported actions: `CASCADE`, `SET NULL`, `SET DEFAULT`, `RESTRICT`, `NO ACTION`.

`NO ACTION` is the default. Unlike `RESTRICT`, `NO ACTION` allows deferrable constraints to defer checking to transaction end.

### DEFERRABLE

Support for `DEFERRABLE` / `NOT DEFERRABLE` and `INITIALLY DEFERRED` / `INITIALLY IMMEDIATE`.

A deferrable FK does not check referential integrity at statement end; checking is deferred to transaction end. Use `SET CONSTRAINTS ALL DEFERRED` at the start of a transaction to restore deferred checking.

```sql
BEGIN;
SET CONSTRAINTS all_fks DEFERRED;
-- Can now delete referenced rows before updating referencing rows
DELETE FROM orders WHERE id = 1;
UPDATE order_items SET order_id = 2 WHERE order_id = 1;
COMMIT; -- Constraints checked here
```

### NOT VALID / VALIDATE

FKs can be created `NOT VALID` and validated later with `VALIDATE CONSTRAINT`. This allows adding an FK to a live table without locking it for a full table scan during creation.

```sql
ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (ref_id) REFERENCES ref (id) NOT VALID;
-- later, during low-traffic window:
ALTER TABLE t VALIDATE CONSTRAINT fk;
```

!!! tip "Zero-downtime NOT NULL"
    The same two-step pattern applies to NOT NULL: add a `CHECK (col IS NOT NULL) NOT VALID`, validate it under `SHARE UPDATE EXCLUSIVE`, then `SET NOT NULL` (instant, scan skipped on PG12+). See [DDL Semantics](ddl-semantics.md#not-null-via-check-validate) for the full four-step recipe.

### ALTER ALTER CONSTRAINT

PostgreSQL 9.4+ supports modifying constraint deferrability without drop/create:

```sql
ALTER TABLE t ALTER CONSTRAINT fk DEFERRABLE INITIALLY DEFERRED;
```

dbwarden detects when only the deferrability changed and emits `ALTER CONSTRAINT` instead of a drop+add cycle.

## Unique Constraints

Support for:

- `DEFERRABLE` / `INITIALLY DEFERRED`
- `INCLUDE` columns
- `NULLS NOT DISTINCT` (PG 15+)

A unique constraint with `NULLS NOT DISTINCT` treats NULLs as equal, so only one row can contain NULL.
If only the constraint name changes, dbwarden emits `ALTER TABLE ... RENAME CONSTRAINT ...` instead of dropping and recreating the unique constraint.

## Primary Key Constraints

Declared via SQLAlchemy's `primary_key=True` on `mapped_column`. The PK constraint is managed automatically:

| Operation | DDL |
|-----------|-----|
| Create | `PRIMARY KEY (col1, col2)` inline in `CREATE TABLE` |
| Add | `ALTER TABLE t ADD PRIMARY KEY (col1)` |
| Drop | `ALTER TABLE t DROP CONSTRAINT t_pkey` |

PK constraint options (set via the column/s in SQLAlchemy):

| Option | SQL | Notes |
|--------|-----|-------|
| Tablespace | `USING INDEX TABLESPACE name` | Separates PK index from table storage |
| Deferrable | `DEFERRABLE` / `INITIALLY DEFERRED` | PK can be deferred (rare) |

## Check Constraints

Support for:

- Arbitrary CHECK expressions
- `NO INHERIT` (constraint not applied to child tables)
- `NOT VALID` + `VALIDATE`

### NO INHERIT Example

```python
class Meta(PGTableMeta):
    pg_checks = [
        {"name": "ck_users_type", "sql": "user_type IN ('admin', 'user')", "no_inherit": True},
    ]
```

```sql
ALTER TABLE users ADD CONSTRAINT ck_users_type CHECK (user_type IN ('admin', 'user')) NO INHERIT;
```

`NO INHERIT` is useful when a parent table has a constraint that should not apply to child tables.

## Exclude Constraints

Declared via `pg_excludes` on `PGTableMeta`. Uses `PgTableHandler`.

```python
class Meta(PGTableMeta):
    pg_excludes = [
        {"name": "excl_room_booking", "expression": "USING gist (room_id WITH =, during WITH &&)"},
    ]
```

Exclusion constraints require a GiST or SP-GiST index. They prevent any two rows from having overlapping values in the specified columns. Common use cases: time-range booking, geo-spatial exclusion.

Diffing compares the full expression; any change in operator, access method, or columns produces a drop+add cycle.

## UniqueSpec vs unique=True on PgIndexSpec

PostgreSQL implements unique constraints as unique indexes under the hood. dbwarden supports both approaches, and the choice is a semantic one.

| Aspect | `UniqueSpec` in `uniques` / `pg_uniques` | `unique=True` on `PgIndexSpec` in `pg_indexes` |
|--------|------------------------------------------|------------------------------------------------|
| SQL generated | `ALTER TABLE ... ADD CONSTRAINT ... UNIQUE (...)` | `CREATE UNIQUE INDEX ... ON ... (...)` |
| Shows in `information_schema.table_constraints` | Yes | No |
| Targetable by FK references | Yes | No (needs the unique index to also be a constraint) |
| Business meaning | A constraint: a rule the data must satisfy | An index: a data structure for performance |

### When to use each

**Use `UniqueSpec`** when the uniqueness is a business rule:

```python
from dbwarden.databases import UniqueSpec

class Meta(PGTableMeta):
    uniques = [
        UniqueSpec(name="uq_users_email", columns=["email"]),
    ]
```

This generates an `ALTER TABLE` constraint. It shows in `information_schema.table_constraints` and can be referenced by foreign keys.

**Use `unique=True` on `PgIndexSpec`** when the uniqueness is a performance concern:

```python
from dbwarden.databases.pgsql import PgIndexSpec

class Meta(PGTableMeta):
    pg_indexes = [
        PgIndexSpec("ix_users_email", ["email"], unique=True),
    ]
```

This generates a `CREATE UNIQUE INDEX`. It enforces uniqueness identically but does not appear in the SQL standard constraint views and cannot be targeted by FK `REFERENCES`.

### Practical guidance

For a business rule like "no two users can share the same username within a branch":

```python
class Meta(PGTableMeta):
    uniques = [
        UniqueSpec(name="uq_user_branch_username",
                   columns=["branch_id", "username"]),
    ]
```

A constraint is semantically correct here: it declares a rule the data must satisfy. It also enables future FK references and works with `ON CONFLICT` in the same way as a unique index.

See [Indexes](indexes.md) for `PgIndexSpec` options.

## Constraint Diffing

Constraints are compared by full attribute content. Any difference in signature (columns, expression, options) produces `DROP` + `ADD`. Constraint name changes are detected as a new constraint.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/ddl-behavior/
========================================================================

# DDL Behavior

!!! note "Lock modes and runtime behavior"
    This page covers transactional DDL, index creation strategies, and type change behavior. For the complete lock mode reference, table rewrite conditions, and the CREATE INDEX CONCURRENTLY lifecycle, see [DDL Semantics](ddl-semantics.md).

## Transactional DDL

PostgreSQL DDL is transactional. If a migration file contains multiple statements and one fails, all prior DDL in that file is rolled back. This makes PostgreSQL the safest backend for automated migration runs.

Operations that cannot run inside a transaction block:

- `CREATE INDEX CONCURRENTLY`
- `REFRESH MATERIALIZED VIEW CONCURRENTLY`
- `REINDEX DATABASE` / `REINDEX INDEX CONCURRENTLY`
- `ALTER TABLE DETACH PARTITION CONCURRENTLY` (PG 16+)
- `VACUUM` / `ANALYZE` (maintenance commands)

## Index Creation

dbwarden defaults to `CREATE INDEX CONCURRENTLY` to avoid table locking. Pass `--no-concurrent` when the migration must run inside a transaction block (PostgreSQL requires `CONCURRENTLY` outside a transaction).

### CONCURRENTLY Behaviour

- Requires more total time (two table scans)
- Allows concurrent reads and writes during index build
- If it fails, an invalid index is left behind (clean up with `DROP INDEX CONCURRENTLY`)
- Not supported for `CREATE INDEX ON ...` with partitioned tables (PG 14+ added partial support)

## Column Type Changes

Emits `ALTER TABLE t ALTER COLUMN c TYPE newtype` with a commented-out `-- USING col::newtype` line. Pass `--postgres-auto-using` to emit an active `USING` clause. Without the flag, uncomment and verify the USING expression before running the migration against production.

### USING Clause Examples

```sql
-- String to enum conversion
ALTER TABLE users ALTER COLUMN status TYPE user_status USING status::user_status;

-- Text to boolean
ALTER TABLE users ALTER COLUMN active TYPE boolean USING active::boolean;

-- JSONB extraction
ALTER TABLE users ALTER COLUMN metadata TYPE text USING metadata->>'name';
```

### Table Rewrite Behaviour

`ALTER COLUMN TYPE` rewrites the entire table (ACCESS EXCLUSIVE lock). Other operations that rewrite the table:

| Operation | Rewrite? | Notes |
|-----------|----------|-------|
| `ALTER COLUMN TYPE` | Yes | Full table rewrite |
| `SET STORAGE` | Yes | Rewrites column data |
| `SET COMPRESSION` | Yes | Rewrites column data (PG 14+) |
| `SET TABLESPACE` | Yes | Moves entire table |
| `ALTER COLUMN SET/DROP NOT NULL` | No | Metadata only |
| `ALTER COLUMN SET/DROP DEFAULT` | No | Metadata only |
| `ADD COLUMN` (no default) | No | Metadata only |
| `ADD COLUMN` (volatile default) | Yes | Rewrites table |
| `ALTER TABLE SET (fillfactor)` | No | Metadata only |
| `DROP COLUMN` | No | Metadata only (mark as dropped) |
| `SET UNLOGGED` | Yes | Writes all data to WAL |

### Safe Type Change

The `--safe-type-change` flag generates a multi-step strategy:
1. Add a temporary column with the new type
2. Emit a `--` comment with an `UPDATE` statement template
3. Emit a verification comment
4. After manual verification, drop the old column and rename the temporary column

### Lock Levels

| Operation | Lock Mode | Concurrent Access |
|-----------|-----------|-------------------|
| `ALTER TABLE t ALTER COLUMN c TYPE` | `ACCESS EXCLUSIVE` | Blocks all reads and writes |
| `ALTER TABLE t ADD COLUMN c` | `ACCESS EXCLUSIVE` (brief) | Very short lock |
| `ALTER TABLE t DROP COLUMN c` | `ACCESS EXCLUSIVE` | Table rewrite avoided, metadata only |
| `CREATE INDEX` (non-concurrent) | `ACCESS EXCLUSIVE` | Blocks all access |
| `CREATE INDEX CONCURRENTLY` | `SHARE UPDATE EXCLUSIVE` | Allows reads and writes |
| `DROP INDEX CONCURRENTLY` | `SHARE UPDATE EXCLUSIVE` | Allows reads and writes |
| `ALTER TABLE t SET (fillfactor)` | `ACCESS EXCLUSIVE` | Brief metadata change |
| `VALIDATE CONSTRAINT` | `SHARE UPDATE EXCLUSIVE` | Allows reads and writes |
| `REFRESH MATERIALIZED VIEW CONCURRENTLY` | `SHARE UPDATE EXCLUSIVE` | Allows reads and writes |

## Generated Columns

Adding a generated column via `ALTER TABLE` is not supported by PostgreSQL. dbwarden emits a comment placeholder noting this limitation. Dropping the generation expression (`ALTER COLUMN c DROP EXPRESSION`) produces real DDL.

PostgreSQL supports `GENERATED ALWAYS AS (expr) STORED` only. Virtual generated columns are not supported.

## Auto-increment Lifecycle

dbwarden supports toggling auto-increment on integer primary key columns:

```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
```

To explicitly disable auto-increment:

```python
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=False)
```

| Change | Generated SQL |
|--------|---------------|
| Adding autoincrement | `CREATE SEQUENCE users_id_seq` + `ALTER COLUMN id SET DEFAULT nextval(...)` + `ALTER SEQUENCE ... OWNED BY` |
| Removing autoincrement | `ALTER COLUMN id DROP DEFAULT` + `DROP SEQUENCE IF EXISTS users_id_seq` |

### GENERATED ALWAYS vs GENERATED BY DEFAULT

| Mode | Behaviour |
|------|-----------|
| `GENERATED ALWAYS AS IDENTITY` | User cannot provide a value; `INSERT` with explicit value fails |
| `GENERATED BY DEFAULT AS IDENTITY` | User can provide a value; auto-generation is skipped when value is supplied |

Override for `ALWAYS`: `INSERT OVERRIDING SYSTEM VALUE`.

### Detection from live databases

dbwarden detects autoincrement by:
1. SERIAL/BIGSERIAL column types
2. SQLAlchemy's `.autoincrement` attribute
3. `nextval(...)` default patterns

### Type mapping

| Condition | Resulting Type |
|-----------|---------------|
| `autoincrement=True` (default) | `SERIAL` / `BIGSERIAL` |
| `autoincrement=False` | `INTEGER` / `BIGINT` |
| `autoincrement=None` (unspecified) | `SERIAL` / `BIGSERIAL` (backward compatible) |

## DDL Limitations

- **Adding generated column**: Not supported via ALTER TABLE; requires table recreation
- **`CONCURRENTLY` in transaction**: `CREATE INDEX CONCURRENTLY`, `REFRESH MATERIALIZED VIEW CONCURRENTLY`, and similar operations cannot run inside a transaction block
- **Volatile default on ADD COLUMN**: `ALTER TABLE t ADD COLUMN c type DEFAULT random()` rewrites the table because each row needs a distinct default value
- **Dropping column with dependencies**: Requires `CASCADE` if views, FKs, or other objects reference the column
- **Removing enum values**: PostgreSQL does not support `ALTER TYPE ... DROP VALUE`; enum values can only be added, renamed, or the type recreated with CASCADE
- **Composite type modification**: No `ALTER TYPE` for composite types; must drop and recreate with CASCADE
- **Constraint rename**: Must use `ALTER TABLE t RENAME CONSTRAINT old TO new` (no `ALTER CONSTRAINT` rename)
- **SET STORAGE after table creation**: Rewrites the column data; cannot be done without a table lock

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/ddl-semantics/
========================================================================

# DDL semantics

PostgreSQL DDL statements have runtime behaviors that affect migration safety: lock modes, table rewrites, transaction boundaries, and version-gated optimizations. Understanding these prevents production outages caused by migrations that are syntactically correct but operationally dangerous.

## DDL locking fundamentals

### The three lock modes that matter

PostgreSQL has eight table-level lock modes. For schema migrations, three dominate:

| Lock mode | Blocks reads? | Blocks writes? | Used by |
|-----------|--------------|----------------|---------|
| **ACCESS EXCLUSIVE** | Yes | Yes | Most `ALTER TABLE`, `DROP TABLE`, `TRUNCATE`, `REINDEX`, `CLUSTER`, `VACUUM FULL` |
| **SHARE UPDATE EXCLUSIVE** | No | Yes (and other DDL) | `VACUUM`, `ANALYZE`, `CREATE INDEX CONCURRENTLY`, `VALIDATE CONSTRAINT` |
| **ACCESS SHARE** | No | No | `SELECT` (held by every read query) |

`ACCESS EXCLUSIVE` conflicts with *every other lock mode*, including `ACCESS SHARE`. This means a single `ALTER TABLE` blocks all `SELECT` statements on that table.

### The lock queue pile-up

A blocked `ALTER TABLE` does not wait politely to the side. It takes the head of the lock queue, and everything else lines up behind it:

```
1. Reporting query starts. Holds ACCESS SHARE on users. Runs for 40 seconds.
2. Migration: ALTER TABLE users ADD COLUMN age int. Wants ACCESS EXCLUSIVE.
   Conflicts with the reporting query. Waits.
3. Normal SELECT from the app arrives. Wants ACCESS SHARE.
   Does NOT conflict with the reporting query, but DOES conflict with
   the queued ACCESS EXCLUSIVE ahead of it. Waits.
4. Every subsequent SELECT piles up behind step 3.
```

The table is dark for the duration of the slowest query that was running when the migration started. The `ALTER TABLE` itself was never slow.

### lock_timeout

`lock_timeout` bounds the *acquisition* phase. If the lock cannot be acquired within the timeout, the statement errors with `55P03 lock_not_available`, releases its queue position, and the backlog drains:

```sql
SET lock_timeout = '2s';
ALTER TABLE users ADD COLUMN age int;
-- If lock not acquired in 2s: ERROR, backlog drains, SELECTs resume
```

**Without lock_timeout**, a migration waiting behind a 40-minute reporting query holds the queue for 40 minutes. **With lock_timeout = 2s**, it fails in 2 seconds and can be retried.

Recommended values:

| Environment | lock_timeout | Rationale |
|-------------|-------------|-----------|
| Production | 500ms - 2s | Fail fast, retry on deploy |
| Staging | 5s | More tolerance for background queries |
| CI/development | 0 (default) | No concurrent work to conflict with |

!!! warning "lock_timeout vs statement_timeout"
    `lock_timeout` bounds *waiting for the lock*. `statement_timeout` bounds *executing the statement after the lock is acquired*. They serve different purposes:
    - `lock_timeout` prevents the queue pile-up
    - `statement_timeout` prevents a long-running DDL (e.g., `ALTER COLUMN TYPE` on a large table) from running indefinitely

    Set `lock_timeout` smaller than any non-zero `statement_timeout`. A unitless value is milliseconds.

### Hold-time: locks held until transaction end

PostgreSQL holds table-level locks until the **end of the transaction**, not until the statement finishes. A 6ms `ALTER TABLE` inside a 5-minute transaction holds `ACCESS EXCLUSIVE` for the full 5 minutes:

```sql
BEGIN;
ALTER TABLE orders ADD COLUMN note3 text;  -- 6ms work, but...
UPDATE orders SET note3 = compute(note3);  -- 5 minutes of backfill
COMMIT;  -- ACCESS EXCLUSIVE released here, 5 minutes later
```

The fix: separate DDL from backfill. Run DDL in its own short transaction, then backfill in batches:

```sql
-- Migration: DDL only. Brief lock.
BEGIN;
SET LOCAL lock_timeout = '2s';
ALTER TABLE orders ADD COLUMN note3 text;
COMMIT;

-- Separate: backfill in batches, each its own short transaction
UPDATE orders SET note3 = compute(note3) WHERE id BETWEEN 1 AND 5000;
UPDATE orders SET note3 = compute(note3) WHERE id BETWEEN 5001 AND 10000;
-- ...
```

## Operation lock reference

Complete reference of every `ALTER TABLE` subform dbwarden generates, its lock mode, rewrite behavior, and zero-downtime safety.

### Column operations

| Operation | Lock mode | Rewrite? | Duration (large table) | Safe? |
|-----------|-----------|----------|----------------------|-------|
| ADD COLUMN (no default) | ACCESS EXCLUSIVE | No | Instant | Yes |
| ADD COLUMN (constant default, PG11+) | ACCESS EXCLUSIVE | No | Instant | Yes |
| ADD COLUMN (volatile default) | ACCESS EXCLUSIVE | **Yes** | Full rewrite | **No** |
| ADD COLUMN (stored generated) | ACCESS EXCLUSIVE | **Yes** | Full rewrite | **No** |
| ADD COLUMN (identity) | ACCESS EXCLUSIVE | **Yes** | Full rewrite | **No** |
| ADD COLUMN (virtual generated) | ACCESS EXCLUSIVE | No | Instant | Yes |
| DROP COLUMN | ACCESS EXCLUSIVE | No | Instant | Yes |
| ALTER COLUMN TYPE | ACCESS EXCLUSIVE | **Usually** | Full rewrite | **No** |
| ALTER COLUMN TYPE (binary coercible) | ACCESS EXCLUSIVE | No | Index rebuild only | Partial |
| SET DEFAULT | ACCESS EXCLUSIVE | No | Instant | Yes |
| DROP DEFAULT | ACCESS EXCLUSIVE | No | Instant | Yes |
| SET NOT NULL | ACCESS EXCLUSIVE | No (scan) | Full table scan | **No** |
| SET NOT NULL (after validated CHECK, PG12+) | ACCESS EXCLUSIVE | No | Instant | Yes |
| SET NOT NULL NOT VALID (PG18+) | ACCESS EXCLUSIVE | No | Instant | Yes |
| SET DATA TYPE | ACCESS EXCLUSIVE | Usually | Full rewrite | **No** |
| SET STORAGE | ACCESS EXCLUSIVE | No | Instant | Yes |
| SET COMPRESSION | ACCESS EXCLUSIVE | No | Instant | Yes |
| SET STATISTICS | ACCESS EXCLUSIVE | No | Instant | Yes |
| ALTER COLUMN SET (attribute_option) | ACCESS EXCLUSIVE | No | Instant | Yes |
| SET COLUMN ENCRYPTED/DECRYPTED | ACCESS EXCLUSIVE | No | Instant | Yes |

### Table operations

| Operation | Lock mode | Rewrite? | Duration | Safe? |
|-----------|-----------|----------|----------|-------|
| SET TABLESPACE | ACCESS EXCLUSIVE | **Yes** | Full rewrite + copy | **No** |
| SET LOGGED | ACCESS EXCLUSIVE | **Yes** | Full rewrite | **No** |
| SET UNLOGGED | ACCESS EXCLUSIVE | **Yes** | Full rewrite | **No** |
| SET SCHEMA | ACCESS EXCLUSIVE | No | Metadata only | Yes |
| CLUSTER | ACCESS EXCLUSIVE | **Yes** | Full rewrite | **No** |
| ENABLE/DISABLE TRIGGER | ACCESS EXCLUSIVE | No | Instant | Yes |
| ENABLE/DISABLE RULE | ACCESS EXCLUSIVE | No | Instant | Yes |
| DISABLE ROW LEVEL SECURITY | ACCESS EXCLUSIVE | No | Instant | Yes |
| FORCE ROW LEVEL SECURITY | ACCESS EXCLUSIVE | No | Instant | Yes |
| SET ACCESS METHOD | ACCESS EXCLUSIVE | No | Metadata only | Yes |
| INHERIT/NO INHERIT | ACCESS EXCLUSIVE | No | Metadata only | Yes |
| OF type_name/NOT OF | ACCESS EXCLUSIVE | No | Metadata only | Yes |
| REPLICA IDENTITY | ACCESS EXCLUSIVE | No | Metadata only | Yes |
| ENABLE/DISABLE ROW LEVEL SECURITY | ACCESS EXCLUSIVE | No | Instant | Yes |
| ATTACH PARTITION | ACCESS EXCLUSIVE | No | Metadata only | Yes |
| DETACH PARTITION | ACCESS EXCLUSIVE | No | Metadata only | Yes |
| DETACH PARTITION CONCURRENTLY (PG16+) | SHARE UPDATE EXCLUSIVE | No | Metadata only | Yes |

### Index operations

| Operation | Lock mode | Duration | Safe? | Notes |
|-----------|-----------|----------|-------|-------|
| CREATE INDEX | ACCESS EXCLUSIVE | Full scan | **No** | Blocks all reads and writes |
| CREATE INDEX CONCURRENTLY | SHARE UPDATE EXCLUSIVE | Full scan (slower) | Yes | Cannot run in transaction block |
| DROP INDEX | ACCESS EXCLUSIVE | Instant | Yes | |
| DROP INDEX CONCURRENTLY | SHARE UPDATE EXCLUSIVE | Instant | Yes | Cannot run in transaction block |
| REINDEX | ACCESS EXCLUSIVE | Full rebuild | **No** | Cannot run in transaction block |
| REINDEX CONCURRENTLY (PG12+) | SHARE UPDATE EXCLUSIVE | Full rebuild | Yes | Cannot run in transaction block |
| ALTER INDEX | ACCESS EXCLUSIVE | No | Yes | Metadata only |

### Constraint operations

| Operation | Lock mode | Duration | Safe? |
|-----------|-----------|----------|-------|
| ADD CHECK ... NOT VALID | ACCESS EXCLUSIVE | Instant (catalog only) | Yes |
| VALIDATE CONSTRAINT (CHECK) | SHARE UPDATE EXCLUSIVE | Full scan, reads+writes OK | Yes |
| ADD FOREIGN KEY | SHARE ROW EXCLUSIVE | Validates rows | Partial |
| ADD FOREIGN KEY ... NOT VALID | SHARE ROW EXCLUSIVE | Instant (catalog only) | Yes |
| VALIDATE CONSTRAINT (FK) | SHARE UPDATE EXCLUSIVE | Full scan, reads+writes OK | Yes |
| ADD UNIQUE CONSTRAINT | ACCESS EXCLUSIVE | Index build | Partial |
| ADD PRIMARY KEY | ACCESS EXCLUSIVE | Index build | Partial |
| ADD EXCLUDE | ACCESS EXCLUSIVE | GiST build | Partial |
| DROP CONSTRAINT | ACCESS EXCLUSIVE | Instant | Yes |
| ALTER CONSTRAINT ... DEFERRABLE | ACCESS EXCLUSIVE | No | Yes |

### Comment operations

| Operation | Lock mode | Duration | Safe? |
|-----------|-----------|----------|-------|
| COMMENT ON TABLE | ACCESS EXCLUSIVE | Instant | Yes |
| COMMENT ON COLUMN | ACCESS EXCLUSIVE | Instant | Yes |
| COMMENT ON INDEX | ACCESS EXCLUSIVE | Instant | Yes |
| COMMENT ON CONSTRAINT | ACCESS EXCLUSIVE | Instant | Yes |
| COMMENT ON TYPE | ACCESS EXCLUSIVE | Instant | Yes |

## CREATE INDEX CONCURRENTLY

### Why it cannot run inside a transaction

`CREATE INDEX CONCURRENTLY` splits its work across multiple internal transactions: build against a snapshot, wait for older transactions to drain, scan again to catch changes, mark valid. Waiting for other transactions to finish is impossible from inside a transaction that is itself still open. PostgreSQL enforces this:

```
ERROR: CREATE INDEX CONCURRENTLY cannot run inside a transaction block
```

This is a permanent server-side constraint. No tool can work around it.

### Three-transaction lifecycle

1. **Transaction 1**: Index entered as `INVALID` in system catalogs
2. **Transaction 2**: First table scan (builds index against snapshot)
3. **Transaction 3**: Second table scan (catches changes since first scan)
4. **Mark valid**: Index becomes usable for queries

### On failure: invalid index left behind

If the build fails (deadlock, uniqueness violation, OOM), the index remains as `INVALID`:

```
Indexes:
    "idx" btree (col) INVALID
```

Invalid indexes are ignored for queries but still consume update overhead. The recovery path:

```sql
DROP INDEX CONCURRENTLY IF EXISTS idx;
CREATE INDEX CONCURRENTLY idx ON t (col);
```

!!! warning "IF NOT EXISTS trap"
    `CREATE INDEX CONCURRENTLY IF NOT EXISTS` will silently "succeed" over a broken `INVALID` index. Always `DROP INDEX CONCURRENTLY IF EXISTS` before retrying.

### dbwarden handles this automatically

dbwarden prefixes concurrent index creation with `-- @dbwarden:autocommit`, which executes the statement outside the migration transaction on a separate autocommit connection. The `--no-concurrent` flag disables this for development environments.

See [DDL Behavior](ddl-behavior.md) for the autocommit mechanism details.

## ADD COLUMN fast path (PG11+)

### Constant defaults: instant on any table size

PostgreSQL 11 changed `ADD COLUMN` with a constant default from a full table rewrite to a catalog-only operation. The default value is evaluated at `ALTER TABLE` time and stored in `pg_attribute.atthasmissing`. Existing rows receive the value at read time:

```sql
-- Instant, even on 100M rows (PG11+)
ALTER TABLE orders ADD COLUMN status TEXT NOT NULL DEFAULT 'pending';
```

**Constant defaults** (fast path): string literals, integers, booleans, `CURRENT_TIMESTAMP`, `CURRENT_DATE`, `CURRENT_TIME`, `NULL`, `ARRAY[]`, etc.

**Volatile defaults** (full rewrite): `random()`, `gen_random_uuid()`, `clock_timestamp()`, `nextval()`, any function marked `VOLATILE`.

### When the old advice is still correct

Before PG11, the standard pattern was:

```sql
-- Old pattern (PG10 and below): add nullable, backfill, set NOT NULL
ALTER TABLE orders ADD COLUMN status TEXT;
UPDATE orders SET status = 'pending';
ALTER TABLE orders ALTER COLUMN status SET NOT NULL;
```

This is **still necessary** when:

- The default is volatile (e.g., `DEFAULT now()` or `DEFAULT gen_random_uuid()`)
- The column uses a stored generated expression
- The column uses an identity column
- You are on PG 10 or below

For PG11+ with constant defaults, the direct approach is safe and fast:

```sql
-- PG11+: direct is fine for constant defaults
ALTER TABLE orders ADD COLUMN status TEXT NOT NULL DEFAULT 'pending';
```

## NOT NULL via CHECK + VALIDATE

### The problem

`SET NOT NULL` scans the entire table under `ACCESS EXCLUSIVE` to verify no NULLs exist. On a large table, this holds the lock for the duration of the scan:

```sql
-- Dangerous on large tables: full scan under ACCESS EXCLUSIVE
ALTER TABLE orders ALTER COLUMN customer_ref SET NOT NULL;
```

### The four-step recipe

PostgreSQL 12+ can skip the scan if a validated `CHECK` constraint already proves the column contains no NULLs:

```sql
-- Step 1: Add CHECK NOT VALID (ACCESS EXCLUSIVE, instant, catalog only)
ALTER TABLE orders
    ADD CONSTRAINT orders_ref_notnull CHECK (customer_ref IS NOT NULL) NOT VALID;

-- Step 2: Validate (SHARE UPDATE EXCLUSIVE, full scan, reads+writes continue)
ALTER TABLE orders VALIDATE CONSTRAINT orders_ref_notnull;

-- Step 3: SET NOT NULL (ACCESS EXCLUSIVE, instant, scan skipped)
ALTER TABLE orders ALTER COLUMN customer_ref SET NOT NULL;

-- Step 4: Cleanup (optional, removes redundant CHECK)
ALTER TABLE orders DROP CONSTRAINT orders_ref_notnull;
```

Step 2 takes `SHARE UPDATE EXCLUSIVE` which allows concurrent reads and writes. Step 3 is instant because PG12+ recognizes the validated CHECK and skips the scan.

### dbwarden support

dbwarden supports `NOT VALID` / `VALIDATE CONSTRAINT` via the constraint handler. The `VALIDATE` step is marked with `-- @dbwarden:autocommit` to execute outside the migration transaction.

See [Constraints](constraints.md) for the full NOT VALID / VALIDATE documentation.

## Advisory lock operational details

dbwarden uses PostgreSQL advisory locks (`pg_try_advisory_lock`) for migration coordination. These have several operational subtleties.

### Session-scoped

Advisory locks are session-scoped: they release when the connection closes. This means:

- If the migration process crashes, the lock is released automatically
- If the connection is closed by a pooler, the lock is released
- There is no "dead lock" scenario where a stale lock persists

### PgBouncer detection

PgBouncer in transaction-pooling mode assigns different backend connections per transaction, making session-level advisory locks meaningless. dbwarden detects this by calling `pg_backend_pid()` twice and comparing results:

```python
# If the PID changes between calls, we're behind a pooler
pid1 = conn.execute(text("SELECT pg_backend_pid()")).scalar()
pid2 = conn.execute(text("SELECT pg_backend_pid()")).scalar()
if pid1 != pid2:
    raise LockError("Transaction-pooling proxy detected")
```

### Replica detection

dbwarden refuses to run migrations on read replicas (detected via `pg_is_in_recovery()`). Migrations must target the primary.

### Prepared transactions prohibition

Prepared transactions (2PC) interact badly with advisory locks. If a prepared transaction is open, dbwarden refuses to acquire the lock.

### Re-entrancy protection

If the same namespace is already locked on the same connection, dbwarden refuses to acquire a nested lock. This prevents deadlocks from nested migration calls.

## Canonicalization behaviors

dbwarden normalizes several PostgreSQL constructs during diffing to avoid spurious changes. Understanding these prevents confusion when the generated DDL differs from what you wrote.

### CHECK expression cast stripping

PostgreSQL stores CHECK constraints with type casts added by the parser:

```sql
-- You wrote:
CHECK (email <> '')
-- PostgreSQL stores:
CHECK (email <> ''::text)
```

Without normalization, every unnamed CHECK constraint would appear changed on every migration run. dbwarden strips `::type` suffixes and normalizes whitespace before comparison.

### FK action normalization

PostgreSQL reports `NO ACTION` for an unspecified FK action. dbwarden normalizes absent values to `NO ACTION` and uppercases the result:

```python
# Model omits on_delete → stored as None → normalized to "NO ACTION"
# Snapshot has "No Action" → normalized to "NO ACTION"
# Both match, no spurious change
```

### Index null ordering

PostgreSQL's default null ordering follows sort direction: `NULLS LAST` is implied by `ASC`, `NULLS FIRST` by `DESC`. Recording the implied value would cause every index to differ from a model that says nothing about sorting, producing spurious drop+recreate cycles.

dbwarden only emits null ordering when it **contradicts** the PG default.

### nextval() suppression for autoincrement

When a column is marked as autoincrement, the `nextval(...)` default in the snapshot is set to `None` during diffing. Without this, every autoincrementing column would appear to have a default change.

### Physical storage tuning preservation

If the model does not mention `pg_storage` or `pg_compression` for a column, dbwarden will **not** emit `SET STORAGE` or `SET COMPRESSION` to reset the column to its type default. This preserves deliberate DBA tuning (e.g., a column set to `MAIN` storage for compression).

To reset storage to the type default, explicitly set `pg_storage` or `pg_compression` in the model.

## Version-gated features

| Feature | PG version | What changes |
|---------|-----------|-------------|
| Fast ADD COLUMN with constant default | 11+ | No table rewrite for non-volatile defaults |
| SET NOT NULL skip via validated CHECK | 12+ | Instant when CHECK proves no NULLs |
| REINDEX CONCURRENTLY | 12+ | Online index rebuild |
| Column compression metadata (`attcompression`) | 14+ | Snapshot extraction includes compression info |
| Extended statistics expressions | 14+ | `pg_get_statisticsobjdef_expressions()` available |
| NULLS NOT DISTINCT on unique constraints | 15+ | Index and constraint null handling |
| DETACH PARTITION CONCURRENTLY | 16+ | Non-blocking partition detach |
| NOT NULL NOT VALID | 18+ | Attach NOT NULL without table scan |
| NOT ENFORCED constraints | 18+ | CHECK/FK defined but not enforced |

## Known limitations

dbwarden does not currently support:

| Limitation | Impact | Workaround |
|-----------|--------|------------|
| **No statement_timeout** | Long-running DDL (e.g., ALTER TYPE) can run indefinitely | Set `statement_timeout` at session level before migration |
| **ALTER TYPE USING is commented** | Generated as `-- USING col::newtype`, requires manual uncomment | Use `--postgres-auto-using` flag or uncomment manually |
| **No automatic NOT VALID/VALIDATE two-step** | User must opt in to the safe pattern | Declare constraints with `not_valid=True` in model, add separate VALIDATE |
| **Generated columns: ALTER TABLE not supported** | Cannot add/modify generated columns after creation | Recreate table |
| **No automatic lock_timeout** | Must be configured via `pg_migration_lock_timeout` | Set it in config |

## Related

- [DDL Behavior](ddl-behavior.md) : Transactional DDL, index creation, column type changes, lock levels
- [Indexes](indexes.md) : Index types, PgIndexSpec, CONCURRENTLY field
- [Constraints](constraints.md) : NOT VALID / VALIDATE, deferrable, NO INHERIT
- [Tables and Columns](tables-and-columns.md) : Column lifecycle, table properties
- [Migration Locking](../../advanced/migration-locking.md) : Advisory lock implementation, PgBouncer detection, heartbeats
- [Migration Safety](migration-safety.md) : SAFE/WARN/CRITICAL classification per object type

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/declaring-metadata/
========================================================================

# Declaring Metadata

PostgreSQL metadata is declared in a `class Meta` inner class on the model. This is the **only** supported surface: `mapped_column(info=...)` raises `DBWardenConfigError`.

## Table-Level Meta

Inherit from `PGTableMeta` on your `class Meta`:

```python
from sqlalchemy import Integer
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.pgsql import PGTableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)

    class Meta(PGTableMeta):
        pg_fillfactor = 80
        pg_tablespace = "fastspace"
        pg_storage_params = {
            "fillfactor": 80,
            "autovacuum_enabled": "false",
        }
        pg_inherits = "base_entity"
        pg_excludes = [
            {"name": "excl_room_booking", "expression": "USING gist (room_id WITH =, during WITH &&)"},
        ]
```

`PGTableMeta` inherits from `TableMeta`, which provides common attributes shared across all backends:

| Attribute | Type | SQL |
|-----------|------|-----|
| `comment` | `str` | `COMMENT ON TABLE t IS '...'` |
| `indexes` | `list[IndexSpec]` | `CREATE INDEX ...` |
| `checks` | `list[CheckSpec]` | `ALTER TABLE t ADD CONSTRAINT ... CHECK (...)` |
| `uniques` | `list[UniqueSpec]` | `ALTER TABLE t ADD CONSTRAINT ... UNIQUE (...)` |

PostgreSQL-specific `PGTableMeta` attributes:

| Attribute | Type | SQL |
|-----------|------|-----|
| `pg_fillfactor` | `int` | `ALTER TABLE t SET (fillfactor = N)` |
| `pg_tablespace` | `str` | `ALTER TABLE t SET TABLESPACE name` |
| `pg_storage_params` | `dict[str, Any]` | `ALTER TABLE t SET (param = value)` |
| `pg_unlogged` | `bool` | `CREATE UNLOGGED TABLE ...` / `ALTER TABLE t SET UNLOGGED` |
| `pg_partition` | `dict` | `PARTITION BY RANGE / LIST / HASH (columns)` |
| `pg_inherits` | `str \| list[str]` | `ALTER TABLE t INHERIT parent` |
| `pg_excludes` | `list[ExcludeSpec]` | `ALTER TABLE t ADD CONSTRAINT ... EXCLUDE USING ...` |
| `pg_indexes` | `list[PgIndexSpec]` | `CREATE INDEX ...` (with `USING`, `WHERE`, `INCLUDE`, `NULLS NOT DISTINCT`, column sorting) |
| `pg_checks` | `list[CheckSpec]` | `ALTER TABLE t ADD CONSTRAINT ... CHECK (...)` (with `NO INHERIT`) |
| `pg_uniques` | `list[UniqueSpec]` | `ALTER TABLE t ADD CONSTRAINT ... UNIQUE (...)` (with `DEFERRABLE`, `NULLS NOT DISTINCT`, `INCLUDE`) |

Pythonic model definitions use the typed spec classes with full IDE autocomplete:

```python
from dbwarden.databases import IndexSpec, CheckSpec, UniqueSpec
from dbwarden.databases.pgsql import PgIndexSpec, ExcludeSpec

class Meta(PGTableMeta):
    indexes = [
        IndexSpec(name="ix_users_email", columns=["email"], unique=True),
    ]
    checks = [
        CheckSpec(name="ck_users_age", expression="age >= 0"),
    ]
    uniques = [
        UniqueSpec(name="uq_users_email", columns=["email"],
                   deferrable=True, initially_deferred=True),
    ]
    pg_excludes = [
        ExcludeSpec(name="excl_room_booking",
                    elements=[{"column": "room_id", "with": "="},
                              {"column": "during", "with": "&&"}]),
    ]
```

Plain dicts are also accepted for backwards compatibility, but typed specs are the recommended and idiomatic form.

### INHERITS

`pg_inherits` accepts a single parent or a list for multiple inheritance:

```python
# Single inheritance
pg_inherits = "base_entity"

# Multiple inheritance
pg_inherits = ["base_entity", "audit_mixin"]
```

### ON COMMIT (Temporary Tables)

For `pg_unlogged` tables or temporary tables, use `ON COMMIT` options via `pg_on_commit`:

| Value | Behaviour |
|-------|-----------|
| `PRESERVE ROWS` | Default: rows persist across transaction boundaries |
| `DELETE ROWS` | All rows deleted at transaction end |
| `DROP` | Table dropped at transaction end |

### WITH OPTIONS (Storage Parameters)

Beyond `pg_fillfactor`, additional storage parameters can be set:

```sql
ALTER TABLE users SET (autovacuum_vacuum_threshold = 100, toast_tuple_target = 1024);
```

See [Storage Parameters](storage-params.md) for the complete list.

## JSONB Columns

JSONB columns use `from sqlalchemy.dialects.postgresql import JSONB`:

```python
from sqlalchemy.dialects.postgresql import JSONB

class User(Base):
    __tablename__ = "users"
    metadata = Column(JSONB)

    class Meta(PGTableMeta):
        pg_indexes = [
            PgIndexSpec("ix_users_metadata", ["metadata"], using="gin"),
        ]
```

For smaller indexes on path-based queries, use the `jsonb_path_ops` operator class:

```python
PgIndexSpec("ix_users_metadata", ["metadata"],
    using="gin",
    postgresql_ops={"metadata": "jsonb_path_ops"})
```

JSONB column type changes (e.g., `json` -> `jsonb`) are classified as **SAFE**.

### JSONB Operators for Indexing

GIN indexes on JSONB columns support these default operators:

| Operator | Description |
|----------|-------------|
| `?` | Does the string exist as a top-level key? |
| `?\|` | Do any of these strings exist as keys? |
| `?&` | Do all of these strings exist as keys? |
| `@>` | Does the left JSON contain the right JSON path/value? |
| `<@` | Is the left JSON contained by the right JSON? |

With `jsonb_path_ops`, only `@>` and `<@` are supported, but the index is smaller and faster for path queries.

## Column-Level Meta

Use `PGColumnMeta` inner classes for per-column metadata:

```python
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    bio: Mapped[str] = mapped_column(Text)

    class Meta(PGTableMeta):
        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100, identity_increment=1)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", compression="pglz", collation="en_US.UTF-8")
```

`PGColumnMeta` common attributes:

| Attribute | Type | SQL |
|-----------|------|-----|
| `comment` | `str` | `COMMENT ON COLUMN t.c IS '...'` |
| `public` | `bool` | Controls field visibility |
| `pg` | `PgFieldSpec` | PostgreSQL-specific column options |

PostgreSQL-specific `PgFieldSpec` fields (set via `pg.field(...)`):

| Keyword | Type | SQL |
|---------|------|-----|
| `collation` | `str` | `ALTER COLUMN c TYPE t COLLATE "name"` |
| `storage` | `str` | `ALTER COLUMN c SET STORAGE {PLAIN\|MAIN\|EXTERNAL\|EXTENDED}` |
| `compression` | `str` | `ALTER COLUMN c SET COMPRESSION {pglz\|zstd}` (PG 14+) |
| `generated` | `str` | `GENERATED ALWAYS AS (expr) STORED` |
| `identity` | `str` | `ADD GENERATED {ALWAYS\|BY DEFAULT} AS IDENTITY` |
| `identity_start` | `int` | Sequence `START WITH` |
| `identity_increment` | `int` | Sequence `INCREMENT BY` |
| `identity_min` | `int` | Sequence `MINVALUE` |
| `identity_max` | `int` | Sequence `MAXVALUE` |

### Column Defaults

Column default values are set via SQLAlchemy's `server_default` parameter, not through `PGColumnMeta`. The default is emitted as `ALTER COLUMN c SET DEFAULT expr` in DDL:

```python
from sqlalchemy import text

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    created_at: Mapped[datetime] = mapped_column(DateTime, server_default=text("NOW()"))
    slug: Mapped[str] = mapped_column(String, server_default=text("gen_random_uuid()"))
```

`server_default=text("...")` renders the SQL expression directly. Use `server_default="constant"` (string) for literal defaults.

## Foreign Key Options

FK options are captured from the database by `generate-models` and emitted in the `ForeignKey` constructor:

```python
from sqlalchemy import ForeignKey

class OrderItem(Base):
    __tablename__ = "order_items"
    order_id: Mapped[int] = mapped_column(
        ForeignKey("orders.id", ondelete="CASCADE", onupdate="CASCADE", deferrable=True),
        nullable=False,
    )
```

Supported FK options: `ondelete`, `onupdate`, `deferrable`, `initially`, `match` (`FULL`/`PARTIAL`/`SIMPLE`).

See [Constraints](constraints.md) for FK lifecycle and deferrable behaviour.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/event-triggers/
========================================================================

# Event Triggers

**Requires the `dbwarden-pgsql-extensions` plugin:** `dbwarden plugin add dbwarden-pgsql-extensions`. The `event_trigger` object handler ships in that plugin, not in core.

**Handler**: `EventTriggerHandler` (PREAMBLE phase, config-driven)

Event triggers fire on database-level DDL events. They are scoped to the entire database cluster (not per-schema).

```python
pg_event_triggers=[
    {
        "name": "trg_ddl_audit",
        "event": "ddl_command_start",
        "function": "audit_ddl",
        "tags": ["CREATE TABLE", "ALTER TABLE"],
    },
]
```

## Events

| Event | Fires On |
|-------|----------|
| `ddl_command_start` | Before any DDL statement |
| `ddl_command_end` | After any DDL statement |
| `sql_drop` | When objects are dropped |
| `table_rewrite` | When `ALTER TABLE` rewrites a table |

## DDL Command Tags

Available tags for `WHEN TAG IN` filtering (selected common tags):

| Tag Category | Tags |
|--------------|------|
| DDL | `CREATE TABLE`, `ALTER TABLE`, `DROP TABLE`, `CREATE INDEX`, `ALTER INDEX`, `DROP INDEX` |
| Schema | `CREATE SCHEMA`, `ALTER SCHEMA`, `DROP SCHEMA` |
| Type | `CREATE TYPE`, `ALTER TYPE`, `DROP TYPE` |
| Function | `CREATE FUNCTION`, `ALTER FUNCTION`, `DROP FUNCTION` |
| Trigger | `CREATE TRIGGER`, `ALTER TRIGGER`, `DROP TRIGGER` |
| View | `CREATE VIEW`, `ALTER VIEW`, `DROP VIEW` |
| Sequence | `CREATE SEQUENCE`, `ALTER SEQUENCE`, `DROP SEQUENCE` |
| Extension | `CREATE EXTENSION`, `ALTER EXTENSION`, `DROP EXTENSION` |

The full list of supported tags is available in the PostgreSQL documentation under "Server Event Trigger Command Tags".

## Function Context Variables

Event trigger functions access DDL context through special session variables:

| Variable | Type | Description |
|----------|------|-------------|
| `TG_EVENT` | `text` | Event name: `ddl_command_start`, `ddl_command_end`, `sql_drop`, `table_rewrite` |
| `TG_TAG` | `text` | Command tag: `CREATE TABLE`, `ALTER TABLE`, etc. |
| `TG_TABLE_SCHEMA` | `text` | Schema of the target object (when applicable) |
| `TG_TABLE_NAME` | `text` | Name of the target object (when applicable) |

Example function using context variables:

```sql
CREATE FUNCTION audit_ddl()
RETURNS event_trigger
LANGUAGE plpgsql
AS $$
BEGIN
    INSERT INTO ddl_audit_log (event, tag, schema, object, occurred_at)
    VALUES (TG_EVENT, TG_TAG, TG_TABLE_SCHEMA, TG_TABLE_NAME, NOW());
END;
$$;
```

## Function Signature Requirements

Event trigger functions must:
- Take **no arguments**
- Return type `event_trigger`
- Be created before the event trigger that references them

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE EVENT TRIGGER name ON event WHEN TAG IN ('tag1', 'tag2') EXECUTE FUNCTION func();` |
| Alter | `ALTER EVENT TRIGGER name DISABLE;` / `ALTER EVENT TRIGGER name ENABLE;` / `ALTER EVENT TRIGGER name RENAME TO new_name;` |
| Drop | `DROP EVENT TRIGGER IF EXISTS name;` |

## Enabled State

| Value | Meaning |
|-------|---------|
| `O` | Enabled (default) |
| `D` | Disabled |
| `R` | Enabled in replica mode |
| `A` | Always enabled |

## Notes

- Event triggers require a superuser to create
- The backing function must be created first (see [Functions & Triggers](functions-and-triggers.md))
- `DROP EVENT TRIGGER` does not auto-drop the backing function
- Tags filter which DDL commands fire the trigger; absent tags means all DDL commands
- If an event trigger function raises an exception, the DDL command is aborted and rolled back
- Use `sql_drop` with care: objects have already been removed from catalogs, so `TG_TABLE_SCHEMA` and `TG_TABLE_NAME` may be NULL for dropped objects; use `pg_event_trigger_dropped_objects()` to get the list

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/extended-statistics/
========================================================================

# Extended Statistics

**Requires the `dbwarden-pgsql-extensions` plugin:** `dbwarden plugin add dbwarden-pgsql-extensions`. The `extended_statistics` object handler ships in that plugin, not in core.

**Handler**: `ExtendedStatisticsHandler` (PREAMBLE phase, config-driven, PG 14+)

Extended statistics give the query planner better estimates for correlated columns.

```python
pg_extended_statistics=[
    {
        "name": "stats_users_email_city",
        "table": "users",
        "kinds": ["d", "f"],
        "columns": "email, city",
    },
]
```

## Kind Codes

| Code | Kind | Description | PG Version |
|------|------|-------------|------------|
| `d` | ndistinct | Distinct-value counts for column groups | 10+ |
| `f` | dependencies | Functional dependency statistics | 10+ |
| `m` | MCV | Most-common-values lists | 10+ |
| `e` | expressions | Expression statistics (PG 14+) | 14+ |

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE STATISTICS name (kinds) ON columns FROM table;` |
| Alter | `ALTER STATISTICS name SET STATISTICS target;` |
| Drop | `DROP STATISTICS IF EXISTS name;` |
| Analyze | `ANALYZE table;` (required to populate statistics) |

### ALTER STATISTICS

The statistics target controls sample size. Higher values produce better estimates at the cost of longer `ANALYZE` time:

```sql
ALTER STATISTICS stats_users_email_city SET STATISTICS 1000;
```

Values range from `-1` (use system default, typically 100) to `10000`.

### ANALYZE Requirements

`CREATE STATISTICS` defines the statistics object but does not populate it. Run `ANALYZE` on the table against the database directly to collect the first statistics sample:

```sql
ANALYZE users;
```

## Examples

```python
# ndistinct + dependencies on correlated columns
{"name": "stats_order_date_status", "table": "orders",
 "kinds": ["d", "f"], "columns": "order_date, status"}

# MCV on a high-cardinality group
{"name": "stats_product_category", "table": "products",
 "kinds": ["m"], "columns": "category_id, price"}

# Expression statistics (PG 14+)
{"name": "stats_user_email_domain", "table": "users",
 "kinds": ["d", "m"], "expressions": ["lower(email)"]}
```

Generated DDL:
```sql
CREATE STATISTICS stats_users_email_city (ndistinct, dependencies) ON email, city FROM users;
```

## Schema Support

Extended statistics can be scoped to a schema:

```python
{"name": "stats_order_date_status", "table": "orders",
 "schema": "app", "kinds": ["d", "f"], "columns": "order_date, status"}
```

## Migration Safety

| Change | Severity |
|--------|----------|
| Add extended statistic | `INFO` |
| Drop extended statistic | `WARNING` |
| Change kinds / columns | `WARNING` |
| Change statistics target | `INFO` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/functions-and-triggers/
========================================================================

# Functions & Triggers

**Requires the `dbwarden-pgsql-extensions` plugin:** `dbwarden plugin add dbwarden-pgsql-extensions`. The plugin owns the `pg_functions` and `pg_triggers` config keys and their object handlers, so declaring them without it installed raises `DBWardenConfigError` at config load.

Functions and triggers are **config-driven** objects processed during the PREAMBLE phase (before table diffing). They are declared in `database_config(...)`.

## Functions

Declared via `pg_functions`. Each function definition includes the body, language, and options.

```python
pg_functions=[
    {
        "name": "update_timestamp",
        "language": "plpgsql",
        "body": """
            BEGIN
                NEW.updated_at = NOW();
                RETURN NEW;
            END;
        """,
        "returns": "trigger",
    },
    {
        "name": "count_users",
        "language": "sql",
        "body": "SELECT count(*) FROM users",
        "returns": "bigint",
        "volatility": "STABLE",
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE FUNCTION name (...args...) RETURNS type AS $$ body $$ LANGUAGE lang;` |
| Drop | `DROP FUNCTION IF EXISTS name (...args...) CASCADE;` |

Changes are detected as drop-then-create. There is no `CREATE OR REPLACE` for changed bodies.

### Options

| Config Key | SQL |
|------------|-----|
| `volatility` | `VOLATILE` / `STABLE` / `IMMUTABLE` |
| `security_definer` | `SECURITY DEFINER` |
| `leakproof` | `LEAKPROOF` |
| `parallel` | `PARALLEL UNSAFE` / `RESTRICTED` / `SAFE` |
| `cost` | `COST n` |
| `rows` | `ROWS n` |

Volatility semantics:

| Volatility | Behaviour | Optimizer Assumptions |
|------------|-----------|-----------------------|
| `VOLATILE` | Can return different results on each call (default) | No optimizations |
| `STABLE` | Same results within same statement | Can be used in index scan conditions |
| `IMMUTABLE` | Same results for same arguments always | Can be pre-evaluated, used in expression indexes, partition pruning |

### Return Types

| Returns | Description |
|---------|-------------|
| `trigger` | Trigger function (returns `TRIGGER`) |
| `void` | No return value |
| `table (...)` | Set-returning function (use `rows` for estimate) |
| `setof type` | Set-returning function (alternative syntax) |
| Any PG type | Scalar return |

### Arguments

```python
{
    "name": "add_user",
    "args": [
        {"name": "p_name", "type": "text"},
        {"name": "p_email", "type": "text"},
    ],
    "returns": "int",
    "language": "sql",
    "body": "INSERT INTO users (name, email) VALUES (p_name, p_email) RETURNING id",
}
```

### Parameter Modes

| Mode | Description |
|------|-------------|
| `IN` | Default: input parameter |
| `OUT` | Output parameter, acts as return value |
| `INOUT` | Input-output parameter (both accepts and returns) |
| `VARIADIC` | Variable number of arguments (last parameter only) |

```python
{
    "name": "get_user_stats",
    "args": [
        {"name": "p_user_id", "type": "int", "mode": "IN"},
        {"name": "out_count", "type": "bigint", "mode": "OUT"},
    ],
    "returns": "bigint",
    "language": "sql",
    "body": "SELECT count(*) FROM orders WHERE user_id = p_user_id",
}
```

### Function Overloading

PostgreSQL supports multiple functions with the same name but different argument types. dbwarden tracks the full argument signature. Two functions with the same name but different `args` are treated as distinct objects.

### WINDOW Functions

```python
{
    "name": "rank_per_category",
    "returns": "int",
    "language": "sql",
    "body": "SELECT rank() OVER (PARTITION BY category ORDER BY score DESC)",
    "args": [{"name": "category", "type": "text"}],
}
```

Set `RETURNS TABLE (...) ` or `RETURNS SETOF` with `rows` estimate.

## Procedures

PostgreSQL 11+ supports `CREATE PROCEDURE`, distinct from functions in that procedures can use transaction control (`COMMIT` / `ROLLBACK`).

```python
pg_functions=[
    {
        "name": "transfer_funds",
        "language": "plpgsql",
        "body": """
            BEGIN
                UPDATE accounts SET balance = balance - amount WHERE id = from_id;
                UPDATE accounts SET balance = balance + amount WHERE id = to_id;
                COMMIT;
            END;
        """,
        "returns": "void",
        "kind": "procedure",
    },
]
```

Key differences from functions:

| Aspect | Function | Procedure |
|--------|----------|-----------|
| Transaction control | No | Yes (`COMMIT`/`ROLLBACK`) |
| Called via | `SELECT func()` | `CALL proc()` |
| Return value | Required (can be void) | No return value |
| `kind` field | Omit or `"function"` | `"procedure"` |

## Triggers

Declared via `pg_triggers`. Each trigger binds a function to a table event.

```python
pg_triggers=[
    {
        "name": "trg_users_updated_at",
        "table": "users",
        "function": "update_timestamp",
        "timing": "BEFORE",
        "events": ["UPDATE"],
        "for_each": "ROW",
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE TRIGGER name timing event ON table FOR EACH ROW EXECUTE FUNCTION func();` |
| Alter | `ALTER TRIGGER name ON table RENAME TO new_name;` |
| Drop | `DROP TRIGGER IF EXISTS name ON table;` |

### Events

One or more of: `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE`.

### Timing

| Timing | Description |
|--------|-------------|
| `BEFORE` | Fires before the event |
| `AFTER` | Fires after the event |
| `INSTEAD OF` | Replaces the event (views only) |

### UPDATE OF Columns

Fire the trigger only when specific columns are updated:

```python
pg_triggers=[{
    "name": "trg_user_email",
    "table": "users",
    "function": "send_email_verification",
    "timing": "AFTER",
    "events": ["UPDATE OF email"],
    "for_each": "ROW",
}]
```

### Condition

Use `condition` to add a `WHEN` clause:

```python
pg_triggers=[{
    "name": "trg_prevent_delete",
    "table": "users",
    "function": "prevent_delete",
    "timing": "BEFORE",
    "events": ["DELETE"],
    "for_each": "ROW",
    "condition": "OLD.is_protected",
}]
```

### Trigger Function Context

Trigger functions access row data through special variables:

| Variable | Type | Description |
|----------|------|-------------|
| `NEW` | `RECORD` | New row for INSERT/UPDATE (NULL for DELETE) |
| `OLD` | `RECORD` | Old row for UPDATE/DELETE (NULL for INSERT) |
| `TG_OP` | `text` | Operation: `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE` |
| `TG_TABLE_NAME` | `text` | Table that fired the trigger |
| `TG_TABLE_SCHEMA` | `text` | Schema of the table |
| `TG_NAME` | `text` | Trigger name |
| `TG_WHEN` | `text` | Timing: `BEFORE`, `AFTER`, or `INSTEAD OF` |
| `TG_LEVEL` | `text` | `ROW` or `STATEMENT` |
| `TG_NARGS` | `int` | Number of arguments passed to the trigger |
| `TG_ARGV` | `text[]` | Arguments passed to the trigger |

### Constraint Triggers

Constraint triggers are deferred triggers that fire at transaction end. Declared separately from regular triggers:

```python
pg_triggers=[{
    "name": "trg_check_balance",
    "table": "accounts",
    "function": "verify_balance",
    "timing": "AFTER",
    "events": ["UPDATE"],
    "for_each": "ROW",
    "constraint": True,
    "deferrable": True,
    "initially": "DEFERRED",
}]
```

Generated DDL:
```sql
CREATE CONSTRAINT TRIGGER trg_check_balance
AFTER UPDATE ON accounts
DEFERRABLE INITIALLY DEFERRED
FOR EACH ROW
EXECUTE FUNCTION verify_balance();
```

### Trigger Arguments

```python
pg_triggers=[{
    "name": "trg_log_changes",
    "table": "users",
    "function": "log_changes",
    "timing": "AFTER",
    "events": ["UPDATE"],
    "for_each": "ROW",
    "args": ["user_audit", "ignore_columns:password_hash"],
}]
```

Arguments are passed as `TG_ARGV[0]`, `TG_ARGV[1]`, etc. in the trigger function.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/grants-and-roles/
========================================================================

# Grants & Roles

**Requires the `dbwarden-pgsql-rbac` plugin:** `dbwarden plugin add dbwarden-pgsql-rbac`. The plugin owns the `pg_roles` and `pg_default_privileges` config keys along with the handlers described below, so declaring them without it installed raises `DBWardenConfigError` at config load.

Grants are handled by `GrantsHandler` (DIFF phase). Roles are handled by `RoleHandler` (PREAMBLE phase). Default privileges are handled by `DefaultPrivilegesHandler` (PREAMBLE phase).

## Table Grants

Grants are model-derived, declared per-table on the model.

### Grant Types

| Operation | DDL |
|-----------|-----|
| Grant | `GRANT privileges ON TABLE table TO role;` |
| Revoke | `REVOKE privileges ON TABLE table FROM role;` |

Supported privileges: `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE`, `REFERENCES`, `TRIGGER`, `ALL`.

### Column-Level Privileges

Grant access to specific columns only:

```sql
GRANT SELECT (email, name) ON TABLE users TO read_only_role;
GRANT UPDATE (email) ON TABLE users TO app_user;
```

### GRANT OPTION

Allow the grantee to grant the same privilege to others:

```sql
GRANT SELECT ON TABLE users TO admin_user WITH GRANT OPTION;
```

### All Tables in Schema

Bulk grant using `ALL TABLES IN SCHEMA`:

```sql
GRANT SELECT ON ALL TABLES IN SCHEMA public TO read_only_role;
```

## Schema Grants

Schema grants use the same `GrantsHandler` with `object_type="schema"`:

| Operation | DDL |
|-----------|-----|
| Grant | `GRANT USAGE ON SCHEMA schema TO role;` |
| Revoke | `REVOKE USAGE ON SCHEMA schema FROM role;` |

Additional schema privileges:

| Privilege | Description |
|-----------|-------------|
| `USAGE` | Allows access to objects in the schema |
| `CREATE` | Allows creating objects in the schema |
| `ALL` | All schema privileges |

## Database-Level Grants

| Operation | DDL |
|-----------|-----|
| Grant | `GRANT privilege ON DATABASE db TO role;` |
| Revoke | `REVOKE privilege ON DATABASE db FROM role;` |

Supported database privileges: `CREATE`, `CONNECT`, `TEMPORARY` (or `TEMP`), `ALL`.

## Type-Level Grants

```sql
GRANT USAGE ON TYPE my_enum TO app_user;
```

## Roles

**Handler**: `RoleHandler` (PREAMBLE phase, config-driven)

```python
pg_roles=[
    {"name": "app_user", "login": True, "password": "encrypted"},
    {"name": "readonly", "login": True, "connection_limit": 5},
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE ROLE name WITH options` |
| Alter | `ALTER ROLE name WITH options` |
| Drop | `DROP ROLE IF EXISTS name` |

Roles are filtered to non-bootstrap roles (excludes `pg_*` roles and default cluster roles).

### ADMIN OPTION

A role with `ADMIN OPTION` can grant its role membership to others:

```python
{"name": "app_admin", "login": True, "membership": ["app_user"], "admin_option": True}
```

### Role Membership and INHERIT

Role membership grants privileges of the parent role. With `INHERIT` (default), member roles automatically inherit privileges. Without `INHERIT`, use `SET ROLE parent_role` to activate them.

```sql
-- app_user inherits privileges of base_user
CREATE ROLE app_user INHERIT IN ROLE base_user;
```

### SET ROLE / SET SESSION AUTHORIZATION

```sql
SET ROLE app_user;              -- Switch to role within session
SET SESSION AUTHORIZATION app_user;  -- Switch session user
```

## Default Privileges

**Handler**: `DefaultPrivilegesHandler` (PREAMBLE phase, config-driven)

```python
pg_default_privileges=[
    {
        "schema": "public",
        "role": "app_user",
        "kind": "TABLES",
        "privileges": "SELECT, INSERT, UPDATE, DELETE",
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Grant | `ALTER DEFAULT PRIVILEGES FOR ROLE role IN SCHEMA schema GRANT privileges ON kind TO role;` |
| Revoke | `ALTER DEFAULT PRIVILEGES FOR ROLE role IN SCHEMA schema REVOKE privileges ON kind FROM role;` |

### Object Kinds

| Kind | Objects Covered |
|------|----------------|
| `TABLES` | Tables, views, materialized views |
| `SEQUENCES` | Sequences |
| `FUNCTIONS` | Functions, procedures |
| `TYPES` | Types, domains |
| `SCHEMAS` | Schemas |

### REVOKE Behaviour

`REVOKE` supports `CASCADE` and `RESTRICT`:

```sql
REVOKE ALL ON ALL TABLES IN SCHEMA public FROM read_only_role CASCADE;
```

`CASCADE` revokes the privilege from all users who received it through the target role. `RESTRICT` (default) fails if dependent privileges exist.

## BYPASSRLS and RLS Interaction

The `BYPASSRLS` role attribute lets a role bypass Row-Level Security entirely:

```python
{"name": "admin_user", "login": True, "bypassrls": True}
```

Roles without `BYPASSRLS` are subject to all RLS policies on accessed tables. See [RLS & Policies](rls-and-policies.md) for details.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/
========================================================================

# PostgreSQL

dbwarden treats PostgreSQL as a **first-class backend**: every natively supported feature is reverse-engineered, diffed, and emitted as correct DDL.

"First-class" means the round-trip is verified: reverse-engineer a live database with `generate-models`, feed the output back into `make-migrations`, and get **zero diff**.

Implementation note: PostgreSQL diffs and SQL emission flow through the `dbwarden.engine.backends.postgresql.handlers` handler package. The handler pipeline is described in the [Architecture Deep Dive](../../architecture-deep-dive.md#postgresql-handler-pipeline).

```bash
$ dbwarden generate-models -d primary --tables users,orders,items
$ dbwarden make-migrations
# -> "No changes detected"
```

## Feature Matrix

| Category | Features |
|----------|----------|
| Identity Columns | `GENERATED ALWAYS AS IDENTITY`, `GENERATED BY DEFAULT AS IDENTITY`, sequence options |
| Collation | Per-column `COLLATE` via `pg.field(collation=...)` |
| Storage | Per-column `STORAGE` (`PLAIN`, `MAIN`, `EXTERNAL`, `EXTENDED`) |
| Compression | Per-column `COMPRESSION` (`pglz`, `zstd`) via `pg.field(compression=...)` (PG 14+) |
| Generated Columns | `GENERATED ALWAYS AS (...) STORED` |
| Table Properties | Fillfactor, storage params, tablespace, unlogged, partitioning, inheritance |
| Renames | Table rename, column rename |
| Constraints | FK (`MATCH FULL/PARTIAL/SIMPLE`, `ON DELETE/UPDATE`, `DEFERRABLE`), unique (`NULLS NOT DISTINCT`, `INCLUDE`, `DEFERRABLE`), check (`NO INHERIT`, `NOT VALID`), exclude |
| Indexes | B-tree, hash, GiST, GIN, BRIN, SP-GiST; partial, expression, `INCLUDE`, `WHERE`, opclasses, `NULLS NOT DISTINCT`, column sorting, `CONCURRENTLY` |
| RLS & Policies | `ENABLE`/`DISABLE`/`FORCE`/`NO FORCE` row-level security; permissive/restrictive, role-scoped policies |
| Enums | `CREATE TYPE ... AS ENUM`, `ALTER TYPE ... ADD VALUE ... AFTER` |
| Domains | `CREATE DOMAIN` with base type, default, NOT NULL, CHECK |
| Composite Types | `CREATE TYPE ... AS (col1 type1, col2 type2, ...)` |
| Sequences | `CREATE SEQUENCE` with all options |
| Functions | `CREATE FUNCTION` with language, arguments, body |
| Triggers | `CREATE TRIGGER` with timing, events, FOR EACH ROW/STATEMENT |
| Roles | `CREATE ROLE` with login, password, privileges |
| Default Privileges | `ALTER DEFAULT PRIVILEGES` per schema/role/object-type |
| Extended Statistics | `CREATE STATISTICS` with ndistinct, dependencies, MCV, expressions (PG 14+) |
| Event Triggers | `CREATE EVENT TRIGGER` for DDL events |
| Views | Regular `CREATE OR REPLACE VIEW`, materialized views with auto-refresh |
| Schema-level Grants | `GRANT USAGE ON SCHEMA`, `GRANT ALL ON SCHEMA` |
| Table Grants | `GRANT SELECT/INSERT/UPDATE/DELETE` |
| Type Mapping | SQLAlchemy type → PostgreSQL native type normalization |
| Storage Parameters | Table-level and index-level `WITH` options, autovacuum tuning |

## Documentation Sections

- [Config Keys](config-keys.md) : All 12 `pg_*` configuration keys
- [Declaring Metadata](declaring-metadata.md) : Table-level, column-level, JSONB, FK options
- [Tables & Columns](tables-and-columns.md) : Column handler, type changes, auto-increment lifecycle
- [Registry Architecture](../../architecture-deep-dive.md#postgresql-handler-pipeline) : Handler map, phases, online and offline diff flow
- [Constraints](constraints.md) : FK (MATCH FULL, CASCADE), unique, check, exclude
- [Indexes](indexes.md) : B-tree, partial, expression indexes, operator classes, NULLS NOT DISTINCT
- [Types](types.md) : Enums, domains, composite types
- [Functions & Triggers](functions-and-triggers.md) : Function and trigger lifecycle
- [RLS & Policies](rls-and-policies.md) : Row-level security, policy lifecycle, FORCE
- [Grants & Roles](grants-and-roles.md) : Table grants, schema grants, roles, default privileges
- [Partitioning](partitioning.md) : RANGE/LIST/HASH partition strategies, attach/detach
- [Views](views.md) : Regular and materialized views, auto-refresh
- [Sequences](sequences.md) : Sequence creation and ownership
- [Extended Statistics](extended-statistics.md) : ndistinct, dependencies, MCV, expressions
- [Event Triggers](event-triggers.md) : DDL event trigger lifecycle
- [Schemas](schemas.md) : Config-level and model-level schemas, search path
- [DDL Behavior](ddl-behavior.md) : Transactional DDL, CONCURRENTLY, type change strategies
- [DDL Semantics](ddl-semantics.md) : Lock modes, CONCURRENTLY lifecycle, fast ADD COLUMN, canonicalization
- [Type Mapping](type-mapping.md) : SQLAlchemy type → PostgreSQL type normalization
- [Storage Parameters](storage-params.md) : Table and index storage parameters, autovacuum tuning
- [Migration Safety](migration-safety.md) : Safety classification table

## Migration locking

dbwarden uses **session-level advisory locks** for PostgreSQL migrations. This provides strong mutual exclusion without requiring external services.

### How it works

When `dbwarden migrate` runs:

1. Opens a dedicated migration connection
2. Verifies the target is a primary (not a read replica) via `pg_is_in_recovery()`
3. Detects transaction-pooling proxies (PgBouncer) and refuses if detected
4. Acquires a session-level advisory lock via `pg_try_advisory_lock()`
5. Holds the connection and lock for the entire migration run
6. Releases the lock when the connection closes

### Lock properties

- **Scope**: Session-level (tied to the connection, not the transaction)
- **Release**: Automatic on connection close (crash-safe)
- **Fencing**: Stale workers cannot mutate; their next statement fails at the protocol layer
- **Re-entrancy**: Refuses nested runs on the same namespace/connection

### Primary vs replica

dbwarden refuses to acquire locks on read replicas:

```sql
SELECT pg_is_in_recovery()  -- Must return false
```

This prevents the common mistake of running migrations against a replica.

### Transaction pooling detection

dbwarden detects transaction-pooling proxies like PgBouncer by calling `pg_backend_pid()` twice across a statement boundary. If the backend PIDs differ, the lock is refused because session-level advisory locks are meaningless under transaction pooling.

### Configuration

```python
database_config(
    database_name="primary",
    database_type="postgresql",
    database_url_sync="postgresql://...",
    pg_migration_lock_timeout=5000,  # ms, for DDL lock_timeout
)
```

The `pg_migration_lock_timeout` sets PostgreSQL's `lock_timeout` for DDL operations, preventing indefinite waits on table locks.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/indexes/
========================================================================

# Indexes

Indexes are handled by `IndexHandler` during the DIFF phase.

!!! note "CONCURRENTLY lifecycle and invalid indexes"
    For the CREATE INDEX CONCURRENTLY three-transaction lifecycle, invalid index cleanup, and the IF NOT EXISTS trap, see [DDL Semantics](ddl-semantics.md#create-index-concurrently).

## Supported Index Types

| Method | Description |
|--------|-------------|
| B-tree | Default, for equality and range queries |
| Hash | Equality-only, smaller than btree |
| GiST | Geometric, full-text, and custom data types |
| GIN | JSONB, array, full-text, and tsvector |
| BRIN | Block range indexes for large tables |
| SP-GiST | Space-partitioned GiST |

## Declaring Indexes

Model-level indexes use `PgIndexSpec`:

```python
from dbwarden.databases.pgsql import PgIndexSpec

class Meta(PGTableMeta):
    pg_indexes = [
        PgIndexSpec("ix_users_email", ["email"], unique=True, using="btree"),
        PgIndexSpec("ix_users_data", ["data"], using="gin",
            postgresql_ops={"data": "jsonb_path_ops"}),
    ]
```

`PgIndexSpec` supports:

| Field | Description |
|-------|-------------|
| `name` | Index name (auto-generated if omitted) |
| `columns` | Indexed columns |
| `unique` | `CREATE UNIQUE INDEX` |
| `using` | Access method (`btree`, `gin`, `gist`, `hash`, `brin`, `spgist`) |
| `where` | Partial index predicate |
| `include` | `INCLUDE` columns (covering indexes) |
| `expression` | Expression index (e.g., `lower(email)`) |
| `with_params` | Storage parameters (`fillfactor`, `autovacuum_*`) |
| `tablespace` | `TABLESPACE name` |
| `nulls_not_distinct` | `NULLS NOT DISTINCT` (PG 15+) |
| `column_sorting` | Per-column `ASC` / `DESC` / `NULLS FIRST` / `NULLS LAST` |
| `postgresql_ops` | Per-column operator classes |
| `concurrently` | `CREATE INDEX CONCURRENTLY` (default `True`) |

## Partial Indexes

```python
PgIndexSpec("ix_users_active", ["email"], where="active = true")
```

```sql
CREATE INDEX CONCURRENTLY ix_users_active ON users (email) WHERE active = true;
```

## Expression Indexes

```python
PgIndexSpec("ix_users_lower_email", columns=[], expression="lower(email)")
```

```sql
CREATE INDEX CONCURRENTLY ix_users_lower_email ON users (lower(email));
```

Expression indexes are stored with empty `columns` and the expression text in the `expression` field. Changes to the expression are detected and produce a `DROP INDEX` + `CREATE INDEX` cycle.

## Covering Indexes (INCLUDE)

Non-key columns stored in the index to enable index-only scans:

```python
PgIndexSpec("ix_users_email_covering", ["email"],
    include=["name", "avatar_url"])
```

```sql
CREATE INDEX CONCURRENTLY ix_users_email_covering ON users (email) INCLUDE (name, avatar_url);
```

`INCLUDE` columns are not used for index scans but are available as output columns without a table lookup. This makes index-only scans possible for queries that select only `email`, `name`, and `avatar_url`.

## Multi-Column Indexes

Column order matters for query planning:

```python
PgIndexSpec("ix_orders_user_date", ["user_id", "created_at DESC"])
```

B-tree multi-column indexes support queries on the leftmost columns. This index supports:
- `WHERE user_id = 1` (leftmost prefix)
- `WHERE user_id = 1 AND created_at > '2024-01-01'` (range on second column)
- Does NOT support: `WHERE created_at > '2024-01-01'` (no leftmost column)

## Operator Classes

```python
PgIndexSpec("ix_users_data", ["data"],
    using="gin",
    postgresql_ops={"data": "jsonb_path_ops"})
```

```sql
CREATE INDEX CONCURRENTLY ix_users_data ON users USING GIN (data jsonb_path_ops);
```

### Common Operator Classes

| Access Method | Operator Class | Use Case |
|---------------|---------------|----------|
| GIN | `jsonb_path_ops` | Smaller, faster JSONB path queries |
| GIN | `array_ops` (default) | Array containment queries |
| GiST | `inet_ops` | IP address range queries |
| GiST | `tsvector_ops` | Full-text search |
| BRIN | `bloom_ops` | Bloom filter indexes |

## Column Sorting

```python
PgIndexSpec("ix_orders_date", ["created_at DESC NULLS LAST", "id ASC"])
```

```sql
CREATE INDEX CONCURRENTLY ix_orders_date ON users (created_at DESC NULLS LAST, id ASC);
```

## BRIN Parameters

BRIN indexes accept access-method-specific storage parameters:

```python
PgIndexSpec("ix_logs_created_at", ["created_at"],
    using="brin",
    with_params={"pages_per_range": 64, "autosummarize": True})
```

| Parameter | Default | Description |
|-----------|---------|-------------|
| `pages_per_range` | `128` | Number of pages per block range. Lower values = finer granularity, larger index |
| `autosummarize` | `off` | Automatically summarize new pages on insert |

## REINDEX

dbwarden does not auto-generate `REINDEX` statements. When an index becomes corrupted or bloated, recreate it manually:

```sql
REINDEX INDEX CONCURRENTLY ix_users_email;
REINDEX TABLE CONCURRENTLY users;
REINDEX DATABASE CONCURRENTLY mydb;
```

`CONCURRENTLY` avoids locking, but requires extra resources and can fail if the index is a unique index with duplicates.

## NULLS NOT DISTINCT

```python
PgIndexSpec("ix_users_email_unique", ["email"],
    unique=True, nulls_not_distinct=True)
```

Without `NULLS NOT DISTINCT`, a unique index allows multiple NULL values (PostgreSQL treats NULLs as distinct by default). With `NULLS NOT DISTINCT` (PG 15+), only one NULL is permitted.

## Unique Constraint vs Unique Index

For declaring uniqueness, dbwarden offers two paths:

| Path | API | Use case |
|------|-----|----------|
| **Constraint** | `UniqueSpec` in `uniques` / `pg_uniques` | Business rule, FK-targetable, appears in `information_schema` |
| **Index** | `unique=True` on `PgIndexSpec` | Performance-focused, simpler configuration |

See [Constraints](constraints.md#uniquespec-vs-uniquetrue-on-pgindexspec) for the full comparison and guidance.

## Migration Safety

| Change | Severity |
|--------|----------|
| Add index | `INFO` |
| Drop index | `WARNING` |
| Change index expression / columns | `WARNING` |
| Add/drop INCLUDE column | `INFO` |
| Change storage parameters | `INFO` |
| Change tablespace | `WARNING` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/migration-safety/
========================================================================

# Migration Safety

dbwarden classifies migration changes using the `Safety` enum:

```python
from dbwarden.engine.safety import Safety

assert Safety.SAFE == "SAFE"
assert Safety.INFO == "INFO"
assert Safety.WARN == "WARN"
assert Safety.CRITICAL == "CRITICAL"
```

## Safety Classification by Object Type

### Tables & Columns

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add column | `INFO` | None |
| Drop column | `WARN` | `--force` |
| Change column type (safe) | `INFO` | None |
| Change column type (warn) | `WARN` | `--force` |
| Change column type (critical) | `WARN` | `--force` |
| Change column comment | `INFO` | None |
| Change nullable (SET/DROP NOT NULL) | `INFO` | None |
| Change default (SET/DROP DEFAULT) | `INFO` | None |
| Add autoincrement | `INFO` | None |
| Remove autoincrement | `WARN` | `--force` |
| Change PG column meta (storage, compression, collation) | `WARN` | `--force` |
| Rename column | `WARNING` | `--force` |

### Table Properties

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Change fillfactor | `INFO` | None |
| Change tablespace | `WARNING` | `--force` |
| Change inheritance | `WARNING` | `--force` |
| Change exclude constraints | `WARNING` | `--force` |
| Change table comment | `INFO` | None |
| Change object type (logged/unlogged) | `WARNING` | `--force` |
| Change ON COMMIT | `WARNING` | `--force` |

### Indexes

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add index | `INFO` | None |
| Drop index | `WARNING` | `--force` |
| Change index expression / columns | `WARNING` | `--force` |
| Add/drop INCLUDE column | `INFO` | None |
| Change index storage parameters | `INFO` | None |
| Change index tablespace | `WARNING` | `--force` |

### Constraints

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add FK | `INFO` | None |
| Drop FK | `WARNING` | `--force` |
| Change FK options | `WARNING` | `--force` |
| Add unique constraint | `INFO` | None |
| Drop unique constraint | `WARNING` | `--force` |
| Add check constraint | `INFO` | None |
| Drop check constraint | `WARNING` | `--force` |
| Add exclude constraint | `INFO` | None |
| Drop exclude constraint | `WARNING` | `--force` |
| Change constraint deferrability | `INFO` | None |

### Sequences

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add sequence | `INFO` | None |
| Drop sequence | `WARNING` | `--force` |
| Change sequence options (start, increment, etc.) | `WARNING` | `--force` |

### Types

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add enum value | `INFO` | None |
| Add enum type | `INFO` | None |
| Drop enum type | `WARNING` | `--force` |
| Add domain | `INFO` | None |
| Drop domain | `WARNING` | `--force` |
| Change domain definition | `WARNING` | `--force` |
| Add composite type | `INFO` | None |
| Drop composite type | `WARNING` | `--force` |
| Change composite type columns | `WARNING` | `--force` |

### Functions & Procedures

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add function | `INFO` | None |
| Drop function | `WARNING` | `--force` |
| Change function body | `WARNING` | `--force` |
| Add procedure | `INFO` | None |
| Drop procedure | `WARNING` | `--force` |

### Triggers

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add trigger | `INFO` | None |
| Drop trigger | `WARNING` | `--force` |
| Change trigger timing/events | `WARNING` | `--force` |
| Add constraint trigger | `INFO` | None |
| Drop constraint trigger | `WARNING` | `--force` |

### Roles & Grants

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add role | `INFO` | None |
| Drop role | `WARNING` | `--force` |
| Change role options | `WARNING` | `--force` |
| Add grant | `INFO` | None |
| Revoke grant | `WARNING` | `--force` |
| Add default privilege | `INFO` | None |
| Revoke default privilege | `WARNING` | `--force` |

### Event Triggers

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add event trigger | `INFO` | None |
| Drop event trigger | `WARNING` | `--force` |
| Change event trigger tags | `INFO` | None |

### Extended Statistics

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add extended statistic | `INFO` | None |
| Drop extended statistic | `WARNING` | `--force` |
| Change kinds / columns | `WARNING` | `--force` |
| Change statistics target | `INFO` | None |

### Views

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add view | `INFO` | None |
| Drop view | `WARNING` | `--force` |
| Change view query | `WARNING` | `--force` |
| Refresh materialized view | `INFO` | None |
| Change view schema | `WARNING` | `--force` |

### Schemas

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add schema | `INFO` | None |
| Drop schema | `WARNING` | `--force` |

### RLS & Policies

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Enable RLS | `INFO` | None |
| Disable RLS | `WARNING` | `--force` |
| Force RLS | `INFO` | None |
| Add policy | `INFO` | None |
| Drop policy | `INFO` | None |
| Change policy expression | `INFO` | None |

### Partitions

| Change Type | Severity | Flag Required |
|-------------|----------|---------------|
| Add partition | `INFO` | None |
| Detach partition | `WARNING` | `--force` |
| Change partition strategy | `CRITICAL` | `--force` |

## Safe / Warn / Critical Type Changes

### Safe Type Changes (INFO)

| Conversion | Example |
|------------|---------|
| `VARCHAR(n)` → `VARCHAR(m)` (widening) | `VARCHAR(50)` → `VARCHAR(100)` |
| `VARCHAR` → `TEXT` | `VARCHAR` → `TEXT` |
| Storing to same type family | No change |
| `TIMESTAMP` → `TIMESTAMPTZ` | Adding timezone info |
| `json` → `jsonb` | JSON to binary JSON |
| Adding `NOT NULL` when column has no NULLs | Safe when data validates |

### Warn Type Changes (WARNING)

| Conversion | Example |
|------------|---------|
| `VARCHAR(n)` → `VARCHAR(m)` (narrowing) | `VARCHAR(100)` → `VARCHAR(50)` |
| `INTEGER` → `BIGINT` | Widening (safe in PG, uses USING) |
| `BIGINT` → `INTEGER` | Narrowing (potential data loss) |
| `TEXT` → `VARCHAR(n)` | Truncation risk |
| `NUMERIC(p,s)` → `NUMERIC(p',s')` with precision loss | May truncate |
| Any type change that requires USING | Needs manual verification |

### Critical Type Changes (CRITICAL)

| Conversion | Example |
|------------|---------|
| Dropping the only column of a table | Always critical |
| Removing enum values | Not supported by PostgreSQL |
| Changing partition key columns | Requires full table rewrite + data migration |
| Converting from `SETOF`/`TABLE` return type | Schema breakage |

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/partitioning/
========================================================================

# Partitioning

**Handler**: `PartitionHandler` (DIFF phase)

Partitioning is declared on `PGTableMeta` and requires the table to be a true PostgreSQL partitioned table (not traditional inheritance).

## Declaring a Partitioned Table

```python
class Meta(PGTableMeta):
    pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
```

### Partition Strategies

| Strategy | Description |
|----------|-------------|
| `RANGE` | Partition by range of values on one or more columns |
| `LIST` | Partition by discrete values |
| `HASH` | Partition by hash of values |

### Partition Columns

```python
# Range partition by date
pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}

# List partition by category
pg_partition = {"strategy": "LIST", "columns": ["category"]}

# Hash partition by id
pg_partition = {"strategy": "HASH", "columns": ["id"]}
```

## Attaching Partitions

Child partitions are declared via `pg_partitions`:

```python
class Meta(PGTableMeta):
    pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
    pg_partitions = [
        {"name": "events_2024_q1", "bound": "FOR VALUES FROM ('2024-01-01') TO ('2024-04-01')"},
        {"name": "events_2024_q2", "bound": "FOR VALUES FROM ('2024-04-01') TO ('2024-07-01')"},
    ]
```

Partitions are automatically attached/detached during migration.

### Partition Bounds Syntax

| Strategy | Bound Syntax | Example |
|----------|-------------|---------|
| RANGE | `FROM (expr) TO (expr)` | `FROM ('2024-01-01') TO ('2024-04-01')` |
| LIST | `IN (value, ...)` | `IN ('active', 'pending')` |
| HASH | `MODULUS m, REMAINDER r` | `MODULUS 4, REMAINDER 0` |

RANGE bounds are half-open: the lower bound is inclusive, the upper bound is exclusive. Use `UNBOUNDED` for open-ended ranges:

```python
{"name": "events_archived", "bound": "FOR VALUES FROM ('2023-01-01') TO (MAXVALUE)"}
```

HASH partitions use modulus/remainder for round-robin distribution:

```python
{"name": "events_hash_0", "bound": "FOR VALUES WITH (MODULUS 4, REMAINDER 0)"},
{"name": "events_hash_1", "bound": "FOR VALUES WITH (MODULUS 4, REMAINDER 1)"},
{"name": "events_hash_2", "bound": "FOR VALUES WITH (MODULUS 4, REMAINDER 2)"},
{"name": "events_hash_3", "bound": "FOR VALUES WITH (MODULUS 4, REMAINDER 3)"},
```

### DEFAULT Partition

```python
{"name": "events_default", "bound": "DEFAULT"}
```

A DEFAULT partition catches all rows that do not match any other partition bound. Only one DEFAULT partition is allowed per partitioned table.

## Sub-Partitioning

Create partitions of partitions by declaring a partition strategy on a partition table:

```python
class Event(Base):
    __tablename__ = "events"
    ...

    class Meta(PGTableMeta):
        pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
        pg_partitions = [
            {"name": "events_2024", "bound": "FOR VALUES FROM ('2024-01-01') TO ('2025-01-01')",
             "sub_partition": {"strategy": "LIST", "columns": ["category"]}},
        ]
```

Sub-partitions can use a different strategy than the parent. Each sub-partition level has its own `pg_partitions` entries.

## Indexes on Partitioned Tables

Creating an index on the parent table automatically creates matching indexes on all partitions:

```python
class Meta(PGTableMeta):
    pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
    pg_indexes = [
        PgIndexSpec("ix_events_user_id", ["user_id"]),
    ]
```

The index is propagated to all existing and future partitions. Unique indexes (including PKs) must include the partition key.

## FK Constraints on Partitioned Tables

Foreign keys can reference a partitioned table (referencing the parent, referencing is routed to partitions). Foreign keys from a partitioned table must include the partition key:

```python
class OrderItem(Base):
    __tablename__ = "order_items"
    ...

    class Meta(PGTableMeta):
        pg_partition = {"strategy": "HASH", "columns": ["order_id"]}
```

FK from `order_items` to `orders` must include `order_id` in the FK columns.

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Declare partition | `ALTER TABLE parent ATTACH PARTITION child FOR VALUES ...;` |
| Detach partition | `ALTER TABLE parent DETACH PARTITION child;` |
| Detach concurrently (PG 16+) | `ALTER TABLE parent DETACH PARTITION child CONCURRENTLY;` |
| Change strategy | Manual rewrite required (commented in DDL) |

### DETACH CONCURRENTLY (PG 16+)

PostgreSQL 16+ supports `DETACH CONCURRENTLY`, which avoids the `ACCESS EXCLUSIVE` lock normally required:

```sql
ALTER TABLE events DETACH PARTITION events_2023 CONCURRENTLY;
```

The partition becomes an independent table. A background worker finalizes the detach. You can monitor with `pg_detach_pending` in `pg_partitioned_table`.

## Maintenance Operations

Individual partitions support the same maintenance as regular tables:

- `ANALYZE partition_name;` on each partition
- `VACUUM partition_name;` on each partition
- Index creation directly on a partition (not propagated to parent)
- Tablespace management per partition

## Inheritance vs Partitioning

PartitionHandler only tracks tables with `pg_partition` declared (true PG partitioning). Traditional inheritance (`pg_inherits`) is handled separately via `PgTableHandler`.

Partitioning vs Inheritance:

| Aspect | Partitioning | Inheritance |
|--------|-------------|-------------|
| Constraint exclusion | Automatic | Manual (CHECK constraints) |
| Unique constraints | Requires partition key | Supported across any columns |
| FK references | To parent only | To each child separately |
| Row movement | Possible (PG 17+) | Not supported |
| Sub-partitioning | Supported | Not supported |

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/rls-and-policies/
========================================================================

# RLS & Policies

**Requires the `dbwarden-pgsql-rbac` plugin:** `dbwarden plugin add dbwarden-pgsql-rbac`. The policies handler ships in that plugin, not in core.

**Handler**: `PoliciesHandler` (DIFF phase)

Row-Level Security and policies are model-derived, declared on `PGTableMeta`.

## Enabling RLS

```python
class Meta(PGTableMeta):
    pg_rls = True
```

Generated DDL:
```sql
ALTER TABLE users ENABLE ROW LEVEL SECURITY;
```

### Default Deny Behaviour

When RLS is enabled but no policy applies to the current user, the default behaviour is to **deny all access**. Every row is invisible for read operations, and all write operations are blocked. At least one permissive policy must exist for the user to access data.

### FORCE / NO FORCE

```python
class Meta(PGTableMeta):
    pg_rls = True
    pg_rls_force = True
```

Generated DDL:
```sql
ALTER TABLE users FORCE ROW LEVEL SECURITY;
```

To disable force (without disabling RLS):

```python
class Meta(PGTableMeta):
    pg_rls = True
    pg_rls_force = False
```

```sql
ALTER TABLE users NO FORCE ROW LEVEL SECURITY;
```

The force flag only emits when it explicitly changes. If `pg_rls_force` is absent or `False` on both the snapshot and model sides, no `FORCE`/`NO FORCE` DDL is generated. This avoids churn on indexes that only toggle RLS without changing force.

`FORCE ROW LEVEL SECURITY` applies RLS to the table owner, who would normally bypass it. `NO FORCE` (default) exempts the owner.

## Policies

Policies are declared as a list on `PGTableMeta`:

```python
class Meta(PGTableMeta):
    pg_rls = True
    pg_policies = [
        {
            "name": "tenant_isolation",
            "using": "tenant_id = current_setting('app.tenant_id')::int",
            "roles": ["app_user"],
            "permissive": True,
        },
    ]
```

### Policy Keys

| Key | Description |
|-----|-------------|
| `name` | Policy name |
| `using` | `USING` expression (row visibility) |
| `with_check` | `WITH CHECK` expression (row modification) |
| `roles` | Roles the policy applies to (absent = all roles) |
| `permissive` | `PERMISSIVE` (default) or `RESTRICTIVE` |
| `command` | `ALL` (default), `SELECT`, `INSERT`, `UPDATE`, `DELETE` |

### Permissive vs Restrictive

Multiple policies interact differently based on their type:

| Policy Type | Combination Logic |
|-------------|-------------------|
| `PERMISSIVE` (default) | **OR**: access is granted if ANY permissive policy allows it |

| `RESTRICTIVE` | **AND**: access is denied if ANY restrictive policy blocks it |
Restrictive policies filter results after permissive policies. Use restrictive policies to implement mandatory access controls that override permissive policies:

```python
pg_policies = [
    # Permissive: tenant-level access
    {"name": "tenant_access", "using": "tenant_id = current_setting('app.tenant_id')::int",
     "permissive": True},
    # Restrictive: block access to sensitive rows regardless of tenant
    {"name": "block_sensitive", "using": "NOT is_sensitive",
     "permissive": False, "command": "SELECT"},
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Add policy | `CREATE POLICY name ON table FOR command USING (expr)` |
| Alter policy | `ALTER POLICY name ON table USING (new_expr)` |
| Drop policy | `DROP POLICY IF EXISTS name ON table` |
| Enable RLS | `ALTER TABLE table ENABLE ROW LEVEL SECURITY` |
| Disable RLS | `ALTER TABLE table DISABLE ROW LEVEL SECURITY` |
| Force RLS | `ALTER TABLE table FORCE ROW LEVEL SECURITY` |
| Remove force | `ALTER TABLE table NO FORCE ROW LEVEL SECURITY` |

## BYPASSRLS Role Attribute

The `BYPASSRLS` role attribute lets a role bypass all RLS policies:

```python
pg_roles=[
    {"name": "admin_user", "login": True, "bypassrls": True},
]
```

This is equivalent to running without RLS for that role. See [Grants & Roles](grants-and-roles.md) for role configuration.

## RLS and COPY

`COPY TO` on a table with RLS respects policies; only rows visible through the user's policies are exported. `COPY FROM` on a table with RLS checks `WITH CHECK` policies for each inserted row.

## RLS and Unique Constraints

Unique constraints (including PKs) are **not** RLS-aware. A user may see a unique constraint violation caused by a row they cannot see. This is a PostgreSQL limitation: RLS filters query results but does not filter constraint enforcement.

## RLS and FK Constraints

Foreign key constraints are enforced server-side regardless of RLS. A user may not be able to `SELECT` a referenced row but can still insert a referencing row if the FK validates.

## Migration Safety

| Change | Severity |
|--------|----------|
| Enable RLS | `INFO` |
| Disable RLS | `WARNING` |
| Force RLS | `INFO` |
| Add policy | `INFO` |
| Drop policy | `INFO` |
| Change policy expression | `INFO` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/schemas/
========================================================================

# Schemas

**Handler**: `SchemaHandler` (PREAMBLE phase)

## Config-Level Schema

Set `pg_schema` in `database_config(...)` to set the connection's `search_path`:

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:pass@localhost:5432/main",
    pg_schema="app",
)
```

All unqualified table references use this schema. The `_dbwarden_seeds` tracking table is created in the schema specified by `search_path`.

`generate-models` also uses this schema: it sets `search_path` before introspection and lists tables from the configured schema. Snapshot queries against `pg_catalog` use quoted schema/table identifiers, so mixed-case or quoted table names are resolved correctly.

## Model-Level Schema

Set `pg_schema` on `PGTableMeta` or `PGViewMeta`:

```python
class Meta(PGTableMeta):
    pg_schema = "app"
```

When a model has `pg_schema`, all DDL references the fully qualified name (`app.users`). This takes precedence over the config-level `search_path`.

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE SCHEMA IF NOT EXISTS name;` |
| Drop | `DROP SCHEMA IF EXISTS name CASCADE;` |

## Schema Ownership

When a schema is created, it is owned by the role that created it. Set a different owner:

```sql
ALTER SCHEMA app OWNER TO app_admin;
```

## Schema Privileges

Schemas require privileges for access:

| Privilege | Effect |
|-----------|--------|
| `USAGE` | Allows access to objects in the schema |
| `CREATE` | Allows creating new objects in the schema |

```sql
GRANT USAGE ON SCHEMA app TO app_user;
GRANT CREATE ON SCHEMA app TO app_admin;
```

Without `USAGE` on a schema, a user cannot see or access any objects within it, even if they have table-level privileges.

## Search Path Resolution

PostgreSQL resolves unqualified names by searching schemas in `search_path` order:

```
current_schema (first match wins) -> pg_catalog -> public
```

Use the `current_schema` function to check the effective search path:

```sql
SELECT current_schema; -- Returns the first schema in the path
SHOW search_path;      -- Returns the full search path string
```

The config-level `pg_schema` becomes the first entry in `search_path`, which means it takes precedence for all unqualified references.

## pg_catalog vs public

- `pg_catalog` is always searched unless explicitly excluded
- `public` schema is accessible by default to all roles
- Custom schemas require explicit `USAGE` grants

## Temporary Schema

PostgreSQL creates a `pg_temp_*` schema for temporary tables. Temporary tables take precedence over permanent tables when their schema is first in `search_path`. You can reference `pg_temp.schema_name` explicitly.

## Extensions

Extensions are created during the PREAMBLE phase via `pg_extensions`, which requires the `dbwarden-pgsql-extensions` plugin (`dbwarden plugin add dbwarden-pgsql-extensions`). Without it, the key raises `DBWardenConfigError` at config load.

```python
pg_extensions=["uuid-ossp", "pgcrypto"]
```

Generated DDL: `CREATE EXTENSION IF NOT EXISTS "uuid-ossp";`

## Code Seeds and Schema

Code seeds automatically qualify the table name with the model's `pg_schema`. If `User` has `pg_schema = "app"`, the seed INSERT becomes `INSERT INTO app.users (...) VALUES (...)`.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/sequences/
========================================================================

# Sequences

**Requires the `dbwarden-pgsql-types` plugin:** `dbwarden plugin add dbwarden-pgsql-types`. The `sequence` object handler ships in that plugin, not in core.

**Handler**: `SequenceHandler` (PREAMBLE phase, config-driven)

```python
pg_sequences=[
    {
        "name": "order_number_seq",
        "start": 1000,
        "increment": 1,
        "minvalue": 1,
        "maxvalue": 999999,
        "cycle": True,
    },
]
```

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE SEQUENCE name START WITH 1000 INCREMENT BY 1 MINVALUE 1 MAXVALUE 999999 CYCLE;` |
| Alter | `ALTER SEQUENCE name INCREMENT BY 10 MAXVALUE 9999999;` |
| Drop | `DROP SEQUENCE IF EXISTS name;` |

### ALTER SEQUENCE

Modify sequence parameters after creation:

```sql
ALTER SEQUENCE order_number_seq INCREMENT BY 10;
ALTER SEQUENCE order_number_seq RESTART WITH 5000;
ALTER SEQUENCE order_number_seq MAXVALUE 9999999;
ALTER SEQUENCE order_number_seq NO CYCLE;
```

dbwarden detects changes to any sequence option and emits `ALTER SEQUENCE` instead of drop+create when feasible.

## Options

| Key | SQL |
|-----|-----|
| `start` | `START WITH n` |
| `increment` | `INCREMENT BY n` |
| `minvalue` | `MINVALUE n` |
| `maxvalue` | `MAXVALUE n` |
| `cycle` | `CYCLE` / `NO CYCLE` |
| `cache` | `CACHE n` |
| `owned_by` | `OWNED BY table.column` |

### CACHE Option

`CACHE n` pre-allocates sequence values in memory for better performance:

```python
{
    "name": "order_number_seq",
    "start": 1000,
    "increment": 1,
    "cache": 100,
}
```

```sql
CREATE SEQUENCE order_number_seq START WITH 1000 INCREMENT BY 1 CACHE 100;
```

Higher cache values improve multi-session throughput but increase gaps on crashes (cached values are lost).

## Ownership

When `owned_by` is set, dropping the owning table/column automatically drops the sequence. If `owned_by` is `None`, the sequence is standalone and persists after table drops.

Transfer sequence ownership:

```sql
ALTER SEQUENCE order_number_seq OWNED BY orders.order_number;
```

## Schema-Qualified Sequences

Reference a sequence in a specific schema:

```python
{
    "name": "app.order_number_seq",
    "start": 1000,
}
```

## Sequence Functions

| Function | Description |
|----------|-------------|
| `nextval('name')` | Advance and return next value |
| `currval('name')` | Return last value obtained in session |
| `lastval()` | Return last value from any sequence in session |
| `setval('name', n)` | Set current value |
| `setval('name', n, is_called)` | Set value with `is_called` flag |

## Sequence Privileges

| Privilege | Description |
|-----------|-------------|
| `USAGE` | Allows `nextval` and `currval` |
| `SELECT` | Allows `currval` only |
| `UPDATE` | Allows `setval` |
| `ALL` | All sequence privileges |

```sql
GRANT USAGE ON SEQUENCE order_number_seq TO app_user;
```

## Sequence vs Auto-increment

Auto-increment on PK columns (SERIAL/IDENTITY) automatically creates and manages sequences. Independent sequences declared via `pg_sequences` are useful for manual sequence management, custom numbering, or shared sequences across tables.

### Sequence Gaps

Sequence values are not intended to be gap-free. Gaps can occur from:

| Cause | Description |
|-------|-------------|
| `CACHE` loss | Cached values lost on crash |
| Rollback | `nextval` is not rolled back on transaction abort |
| `setval` | Manual value adjustment |
| Concurrent deletes | Rows deleted do not free sequence values |

## Migration Safety

| Change | Severity |
|--------|----------|
| Add sequence | `INFO` |
| Drop sequence | `WARNING` |
| Change sequence options (start, increment, etc.) | `WARNING` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/storage-params/
========================================================================

# Storage Parameters

**Table storage parameters require the `dbwarden-pgsql-extensions` plugin:** `dbwarden plugin add dbwarden-pgsql-extensions`. The `storage_params` object handler ships in that plugin. Index-level storage parameters are part of core index handling and need no plugin.

PostgreSQL supports table-level and index-level storage parameters that control physical storage behaviour, autovacuum tuning, and performance characteristics.

dbwarden tracks storage parameters through the `with_params` field on `PgIndexSpec` and table-level handlers.

## Table Storage Parameters

Set via `ALTER TABLE t SET (param = value, ...)` using the handler infrastructure.

### Autovacuum Parameters

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `autovacuum_enabled` | `bool` | Enables/disables autovacuum for this table | `True` |
| `autovacuum_vacuum_threshold` | `int` | Minimum number of dead tuples before vacuum | `50` |
| `autovacuum_vacuum_scale_factor` | `float` | Fraction of table dead tuples before vacuum | `0.2` |
| `autovacuum_vacuum_ins_threshold` | `int` | Minimum INSERT dead tuples before vacuum (PG 17+) | `1000` |
| `autovacuum_vacuum_ins_scale_factor` | `float` | Fraction of INSERT dead tuples before vacuum (PG 17+) | `0.2` |
| `autovacuum_analyze_threshold` | `int` | Minimum modified tuples before analyze | `50` |
| `autovacuum_analyze_scale_factor` | `float` | Fraction of modified tuples before analyze | `0.1` |
| `autovacuum_vacuum_cost_delay` | `int` | Millisecond delay between vacuum cost units | `2` |
| `autovacuum_vacuum_cost_limit` | `int` | Cost limit before vacuum pauses | `-1` (use global) |
| `autovacuum_freeze_min_age` | `int` | Minimum age before FREEZE | `50000000` |
| `autovacuum_freeze_max_age` | `int` | Maximum age before forced FREEZE | `200000000` |
| `autovacuum_freeze_table_age` | `int` | Age at which whole-table freeze is considered | `150000000` |
| `autovacuum_multixact_freeze_min_age` | `int` | Minimum multixact age before freeze | `5000000` |
| `autovacuum_multixact_freeze_max_age` | `int` | Maximum multixact age before forced freeze | `200000000` |
| `autovacuum_multixact_freeze_table_age` | `int` | Table-level multixact freeze age | `150000000` |

### Tuning Parameters

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `fillfactor` | `int` | Percentage of page space to fill (1-100) | `100` |
| `toast_tuple_target` | `int` | Minimum tuple size before TOAST (PG 11+) | `2048` |
| `parallel_workers` | `int` | Number of parallel workers for scans (PG 11+) | `0` (use global) |
| `vacuum_truncate` | `bool` | Allow vacuum to truncate empty pages (PG 14+) | `True` |
| `log_autovacuum_min_duration` | `int` | Log autovacuum actions exceeding this (ms) (PG 14+) | `-1` (disabled) |

### Table-Level Fillfactor

The `pg_fillfactor` attribute on `PGTableMeta` sets the fillfactor storage parameter:

```python
class Meta(PGTableMeta):
    pg_fillfactor = 80
```

```sql
ALTER TABLE users SET (fillfactor = 80);
```

Fillfactor applies per-table for heap storage and per-index for B-tree indexes. A lower fillfactor leaves more free space for future updates, reducing page splits at the cost of denser storage.

### Toast Tuple Target

Controls when inline values are moved to TOAST storage. Lower values move larger tuples to TOAST earlier:

```sql
ALTER TABLE users SET (toast_tuple_target = 1024);
```

## Index Storage Parameters

Set via the `with_params` field on `PgIndexSpec`:

```python
PgIndexSpec("ix_users_email", ["email"],
    with_params={"fillfactor": 70})
```

```sql
CREATE INDEX CONCURRENTLY ix_users_email ON users (email) WITH (fillfactor = 70);
```

### Access Method-Specific Parameters

| Access Method | Parameter | Type | Default | Description |
|---------------|-----------|------|---------|-------------|
| B-tree | `fillfactor` | `int` | `90` | Page fill percentage |
| GiST | `buffering` | `str \| bool` | `auto` | `on`, `off`, or `auto` |
| BRIN | `pages_per_range` | `int` | `128` | Pages per block range |
| BRIN | `autosummarize` | `bool` | `off` | Auto-summarize on insert |
| Hash | `fillfactor` | `int` | `100` | Page fill percentage |

Example: BRIN parameters:

```python
PgIndexSpec("ix_orders_created_at", ["created_at"],
    using="brin",
    with_params={"pages_per_range": 32, "autosummarize": True})
```

```sql
CREATE INDEX CONCURRENTLY ix_orders_created_at ON orders USING BRIN (created_at) WITH (pages_per_range = 32, autosummarize = on);
```

## Using `with_params`

The `with_params` field on `PgIndexSpec` accepts a dict of parameter name-value pairs:

| Field | Value Type | Description |
|-------|------------|-------------|
| Key | `str` | PostgreSQL storage parameter name (e.g. `fillfactor`, `autosummarize`) |
| Value | `str \| int \| bool` | Parameter value. Booleans render as `on`/`off` |

Parameters are rendered as `WITH (key1 = val1, key2 = val2, ...)` in the `CREATE INDEX` statement.

## Changing Storage Parameters

Storage parameter changes are detected during the DIFF phase. Parameter changes for tables and indexes produce `ALTER TABLE ... SET (...)` or `ALTER INDEX ... SET (...)` DDL. Safety classification varies:

| Change | Severity |
|--------|----------|
| fillfactor change | `INFO` |
| autovacuum setting change | `INFO` |
| BRIN parameter change | `INFO` |
| Index `with_params` change | `INFO` |

See [Migration Safety](migration-safety.md) for the full classification table.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/tables-and-columns/
========================================================================

# Tables & Columns

**Handlers**: `TableHandler`, `RenameTableHandler`, `ColumnHandler`, `PgTableHandler`, `StorageParamsHandler` (DIFF phase)

## Table Lifecycle

| Operation | DDL |
|-----------|-----|
| Create table | `CREATE TABLE name (...)` |
| Drop table | `DROP TABLE IF EXISTS name CASCADE;` |
| Rename table | `ALTER TABLE old_name RENAME TO new_name;` |
| Alter table comment | `COMMENT ON TABLE name IS 'comment';` |

## Column Lifecycle

!!! note "ADD COLUMN lock behavior and PG11+ fast path"
    For the lock modes, table rewrite conditions, and the PG11+ fast ADD COLUMN optimization (constant defaults are instant on any table size), see [DDL Semantics](ddl-semantics.md#add-column-fast-path-pg11).

| Operation | DDL |
|-----------|-----|
| Add column | `ALTER TABLE t ADD COLUMN c type;` |
| Drop column | `ALTER TABLE t DROP COLUMN c;` |
| Change type | `ALTER TABLE t ALTER COLUMN c TYPE newtype;` |
| Change nullable | `ALTER TABLE t ALTER COLUMN c SET/DROP NOT NULL;` |
| Change default | `ALTER TABLE t ALTER COLUMN c SET/DROP DEFAULT;` |
| Change comment | `COMMENT ON COLUMN t.c IS 'comment';` |
| Change autoincrement | `CREATE/DROP SEQUENCE` + `SET/DROP DEFAULT nextval` |
| Change PG meta | `SET STORAGE`, `SET COMPRESSION`, `SET COLLATION`, identity options |
| Rename column | `ALTER TABLE t RENAME COLUMN c TO new_name;` |

## Table Properties

| Property | Meta Attribute | DDL |
|----------|---------------|-----|
| Fillfactor | `pg_fillfactor` | `ALTER TABLE t SET (fillfactor = N);` |
| Tablespace | `pg_tablespace` | `ALTER TABLE t SET TABLESPACE name;` |
| Unlogged | `pg_unlogged` | `CREATE UNLOGGED TABLE` / `ALTER TABLE t SET UNLOGGED;` |
| Inheritance | `pg_inherits` | `ALTER TABLE t INHERIT parent;` |
| Storage params | (via handler) | `ALTER TABLE t SET (param = value);` |
| ON COMMIT | `pg_on_commit` | `ON COMMIT DELETE ROWS` / `DROP` / `PRESERVE ROWS` |

### Storage Params

`pg_storage_params` stores raw PostgreSQL table storage options. `pg_fillfactor` is kept as a shorthand and is folded into `pg_storage_params["fillfactor"]` during discovery.

```python
class Meta(PGTableMeta):
    pg_storage_params = {
        "fillfactor": 80,
        "autovacuum_enabled": "false",
    }
```

Generated DDL:

```sql
ALTER TABLE users SET (fillfactor = 80, autovacuum_enabled = false);
```

## ALTER TABLE Operations

### SET SCHEMA

Move a table between schemas:

```sql
ALTER TABLE users SET SCHEMA app;
```

### SET LOGGED / SET UNLOGGED

Toggle between logged and unlogged modes:

```sql
ALTER TABLE users SET LOGGED;
ALTER TABLE users SET UNLOGGED;
```

`SET LOGGED` converts an unlogged table back to logged mode (all data is written to WAL).

### ALTER COLUMN SET STATISTICS

Set column-level statistics target:

```sql
ALTER TABLE users ALTER COLUMN email SET STATISTICS 500;
```

Higher values improve query planner estimates for columns with non-uniform distributions. Values range from `-1` (use default) to `10000`.

### ALTER COLUMN SET (attribute_option)

Set column-level attribute options:

```sql
ALTER TABLE users ALTER COLUMN email SET (n_distinct = 0.01);
ALTER TABLE users ALTER COLUMN email RESET (n_distinct);
```

Common attribute options: `n_distinct`, `n_distinct_inherited`.

### CLUSTER

Cluster a table based on an index:

```sql
ALTER TABLE users CLUSTER ON ix_users_email;
```

### ENABLE / DISABLE TRIGGER

Control trigger execution:

```sql
ALTER TABLE users DISABLE TRIGGER trg_users_updated_at;
ALTER TABLE users ENABLE TRIGGER trg_users_updated_at;
ALTER TABLE users ENABLE REPLICA TRIGGER trg_users_updated_at;
ALTER TABLE users ENABLE ALWAYS TRIGGER trg_users_updated_at;
```

### TRUNCATE

```sql
TRUNCATE TABLE users;
TRUNCATE TABLE users, posts CASCADE;
```

`TRUNCATE` is not generated by `make-migrations` but can be run manually for bulk data removal.

### Temporary Tables

Temporary tables use `ON COMMIT` for cleanup behaviour:

```sql
CREATE TEMPORARY TABLE temp_data (id int) ON COMMIT DELETE ROWS;
```

Supported `ON COMMIT` values:

| Value | Behaviour |
|-------|-----------|
| `PRESERVE ROWS` | Default: rows persist across transaction boundaries |
| `DELETE ROWS` | All rows deleted at transaction end |
| `DROP` | Table dropped at transaction end |

### LIKE Clause

Create a table with the same structure as an existing one:

```sql
CREATE TABLE users_archive (LIKE users INCLUDING ALL);
```

`INCLUDING ALL` copies defaults, constraints, indexes, and storage. This is a DDL operation (no data copied).
### Storage Params

`pg_storage_params` stores raw PostgreSQL table storage options. `pg_fillfactor` is kept as a shorthand and is folded into `pg_storage_params["fillfactor"]` during discovery.

```python
class Meta(PGTableMeta):
    pg_storage_params = {
        "fillfactor": 80,
        "autovacuum_enabled": "false",
    }
```

Generated DDL:

```sql
ALTER TABLE users SET (fillfactor = 80, autovacuum_enabled = false);
```
## Snapshot Format

### Column Extras

```json
{
  "name": "bio",
  "type": "text",
  "pg_column": {
    "collation": "en_US.UTF-8",
    "storage": "EXTENDED",
    "compression": "pglz",
    "generated": null,
    "identity": "always",
    "identity_start": 1,
    "identity_increment": 1
  }
}
```

### Table Extras

```json
{
  "pg_table": {
    "pg_fillfactor": 80,
    "pg_tablespace": "fastspace",
    "pg_unlogged": false,
    "pg_inherits": "base_entity",
    "pg_partition": {
      "strategy": "RANGE",
      "columns": ["created_at"]
    },
    "pg_excludes": [
      {"name": "excl_room_booking", "expression": "EXCLUDE USING gist (room_id WITH =, during WITH &&)"}
    ]
  }
}
```

## Reverse Engineering

`generate-models` queries `pg_class`, `pg_attribute`, `pg_constraint`, `pg_inherits`, `pg_tablespace`, `pg_partitioned_table`, and `pg_collation` to reverse-engineer all metadata. Array columns are rendered with their element type and dimensions (e.g. `text[]`, `integer[][]`).

```bash
$ dbwarden generate-models -d primary
```

Generated output for a table with identity, storage, compression, collation, and fillfactor:

```python
class User(Base):
    __tablename__ = "users"

    id:    Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True)
    bio:   Mapped[str | None] = mapped_column(Text, nullable=True)

    class Meta(PGTableMeta):
        comment = "Core user accounts"
        pg_fillfactor = 80

        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100, identity_increment=1)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", compression="pglz", collation="en_US.UTF-8")
```

For a partitioned table:

```python
class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(primary_key=True)
    created_at: Mapped[datetime] = mapped_column(DateTime)

    class Meta(PGTableMeta):
        pg_partition = {"strategy": "RANGE", "columns": ["created_at"]}
```

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/type-mapping/
========================================================================

# Type Mapping

dbwarden normalizes SQLAlchemy column types to PostgreSQL native types during snapshot extraction and DDL generation.

## Standard SQLAlchemy Types

| SQLAlchemy Type | PostgreSQL Type | Notes |
|-----------------|-----------------|-------|
| `Integer` | `INTEGER` | |
| `BigInteger` | `BIGINT` | |
| `SmallInteger` | `SMALLINT` | |
| `String(n)` | `VARCHAR(n)` | |
| `Text` | `TEXT` | |
| `Unicode(n)` | `VARCHAR(n)` | |
| `UnicodeText` | `TEXT` | |
| `Boolean` | `BOOLEAN` | |
| `DateTime` | `TIMESTAMP WITHOUT TIME ZONE` | |
| `Date` | `DATE` | |
| `Time` | `TIME WITHOUT TIME ZONE` | |
| `Float` | `FLOAT` | |
| `Double` / `DOUBLE_PRECISION` | `DOUBLE PRECISION` | |
| `Numeric(p, s)` | `NUMERIC(p, s)` | |
| `LargeBinary` | `BYTEA` | |
| `PickleType` | `BYTEA` | |
| `JSON` | `JSON` | |
| `ARRAY(Type)` | `type[]` | e.g. `ARRAY(String)` → `text[]` |
| `Enum(*members)` | `CREATE TYPE ... AS ENUM` | Auto-creates enum type |

## PostgreSQL Dialect-Specific Types

These types from `sqlalchemy.dialects.postgresql` map directly to their PostgreSQL equivalents:

| SQLAlchemy Type | PostgreSQL Type | Notes |
|-----------------|-----------------|-------|
| `JSONB` | `JSONB` | Binary JSON, supports GIN indexes |
| `TIMESTAMP` | `TIMESTAMP WITHOUT TIME ZONE` | |
| `TIMESTAMPTZ` | `TIMESTAMP WITH TIME ZONE` | |
| `TIME` | `TIME WITHOUT TIME ZONE` | |
| `TIMETZ` | `TIME WITH TIME ZONE` | |
| `INTERVAL` | `INTERVAL` | |
| `UUID` | `UUID` | |
| `BYTEA` | `BYTEA` | |
| `OID` | `OID` | |
| `REGCLASS` | `REGCLASS` | |
| `TEXT` | `TEXT` | |
| `BOOLEAN` | `BOOLEAN` | |
| `CIDR` | `CIDR` | IPv4/IPv6 network |
| `INET` | `INET` | IPv4/IPv6 host address |
| `MACADDR` | `MACADDR` | MAC address |
| `MACADDR8` | `MACADDR8` | MAC address (EUI-64) |
| `MONEY` | `MONEY` | Currency amount |
| `TSVECTOR` | `TSVECTOR` | Full-text search document |
| `TSQUERY` | `TSQUERY` | Full-text search query |
| `INT4RANGE` | `INT4RANGE` | Range of integer |
| `INT8RANGE` | `INT8RANGE` | Range of bigint |
| `NUMRANGE` | `NUMRANGE` | Range of numeric |
| `DATERANGE` | `DATERANGE` | Range of date |
| `TSTZRANGE` | `TSTZRANGE` | Range of timestamptz |
| `TSRANGE` | `TSRANGE` | Range of timestamp |
| `BIT(n)` | `BIT(n)` | Fixed-length bit string |
| `VARBIT(n)` | `VARBIT(n)` | Variable-length bit string |
| `XML` | `XML` | XML data |
| `ARRAY(type, dimensions)` | `type[]` | Multi-dimensional array |
| `ENUM(*members)` | `CREATE TYPE ... AS ENUM` | Named enum (creates persistent type) |

## Auto-increment Normalization

| Condition | Resulting Type | Sequence Behavior |
|-----------|---------------|-------------------|
| `Integer` + `autoincrement=True` | `SERIAL` | Auto-creates `tablename_colname_seq` |
| `BigInteger` + `autoincrement=True` | `BIGSERIAL` | Auto-creates sequence |
| `Integer` + `autoincrement=False` | `INTEGER` | No sequence |
| `BigInteger` + `autoincrement=False` | `BIGINT` | No sequence |
| `Integer` (unspecified autoincrement) | `SERIAL` | Backward compatible |
| `GENERATED ALWAYS AS IDENTITY` | `INTEGER` | Creates implicit sequence |
| `GENERATED BY DEFAULT AS IDENTITY` | `INTEGER` | Creates implicit sequence |

See [DDL Behavior](ddl-behavior.md#auto-increment-lifecycle) for the full lifecycle.

## Type Normalization Details

### SERIAL / BIGSERIAL

`SERIAL` and `BIGSERIAL` are syntactic sugar for `INTEGER` / `BIGINT` with an auto-created sequence and a `DEFAULT nextval(...)` expression. dbwarden normalizes them during reverse-engineering:

- On input (`generate-models`): a column typed `INTEGER` with `nextval('seq'::regclass)` default is normalized to `Integer(autoincrement=True)`
- On output (`make-migrations`): a column with `autoincrement=True` emits `SERIAL` / `BIGSERIAL` in `CREATE TABLE`

### TIMESTAMP / TIMESTAMPTZ

| SQLAlchemy Type | Normalized DDL |
|----------------|----------------|
| `DateTime` | `TIMESTAMP WITHOUT TIME ZONE` |
| `TIMESTAMP` | `TIMESTAMP WITHOUT TIME ZONE` |
| `TIMESTAMPTZ` | `TIMESTAMP WITH TIME ZONE` |

### NUMERIC Precision

`Numeric(10, 2)` emits `NUMERIC(10, 2)`. Without precision: `NUMERIC`.

### JSONB vs JSON

- `JSON` → `JSON` (stores exact copy of input text)
- `JSONB` → `JSONB` (stores decomposed binary, supports indexing)

### ARRAY Handling

`ARRAY(String)` emits `text[]`. `ARRAY(Integer)` emits `integer[]`. Multi-dimensional arrays preserve dimensions:

| SQLAlchemy | PostgreSQL |
|------------|------------|
| `ARRAY(String)` | `text[]` |
| `ARRAY(Integer, dimensions=2)` | `integer[][]` |
| `ARRAY(JSONB)` | `jsonb[]` |

### Range Types

Range types accept `Range` objects in Python. dbwarden preserves the range type variant:

| SQLAlchemy | PostgreSQL | Example Value |
|------------|------------|---------------|
| `INT4RANGE` | `INT4RANGE` | `[1, 10)` |
| `TSTZRANGE` | `TSTZRANGE` | `["2024-01-01", "2024-12-31")` |
| `DATERANGE` | `DATERANGE` | `[2024-01-01, 2024-12-31)` |

### Enum Normalization

SQLAlchemy `Enum` types with `create_constraint=True` are extracted as `CREATE TYPE` statements. Enum members are tracked positionally so new values are added with `AFTER` to preserve ordering.

### Domain-Based Columns

When a column uses a domain type (e.g., `us_postal_code`), dbwarden preserves the domain type name in the snapshot rather than expanding to the base type. See [Types](types.md#domains) for domain lifecycle.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/types/
========================================================================

# Types

**Requires the `dbwarden-pgsql-types` plugin:** `dbwarden plugin add dbwarden-pgsql-types`. The `enum`, `domain`, and `composite_type` object handlers ship in that plugin, not in core.

dbwarden supports three object-level type families: enums, domains, and composite types. Enums are model-derived (auto-discovered from table columns). Domains and composite types are config-driven.

## Enums

**Handler**: `EnumHandler` (DIFF phase)

Enums are auto-discovered from column types during snapshot extraction. Enum values are tracked with position data, so `ALTER TYPE ... ADD VALUE ... AFTER ...` preserves ordering.

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE TYPE name AS ENUM ('val1', 'val2', ...)` |
| Add value | `ALTER TYPE name ADD VALUE 'newval' AFTER 'existing'` |
| Rename value | `ALTER TYPE name RENAME VALUE 'old' TO 'new'` (PG 10+) |
| Drop | `DROP TYPE IF EXISTS name` |

### Adding Enum Values

New enum values are inserted after the preceding existing value to preserve sort order:

```python
# Current enum: mood AS ENUM ('sad', 'ok', 'happy')
# Adding 'ecstatic' after 'happy':
# ALTER TYPE mood ADD VALUE 'ecstatic' AFTER 'happy'
```

### Enum Value Renaming (PG 10+)

```sql
ALTER TYPE mood RENAME VALUE 'sad' TO 'unhappy';
```

### Enum Value Deletion

PostgreSQL does **not** support `ALTER TYPE ... DROP VALUE`. Removing an enum value requires:
1. `ALTER TYPE name RENAME TO name_old`
2. `CREATE TYPE name AS ENUM (...)` without the removed value
3. `ALTER TABLE ... ALTER COLUMN c TYPE name USING c::text::name`
4. `DROP TYPE name_old`

This is **not** automated by dbwarden. Value removal is classified as a manual operation.

### Enum Type Normalization

SQLAlchemy's `Enum` type with `create_type=True` creates a persistent PostgreSQL enum type. Without it, the enum is rendered as a `VARCHAR` with a `CHECK` constraint.

## Domains

**Handler**: `DomainHandler` (PREAMBLE phase)

Domains are config-driven objects declared via `pg_domains`:

```python
pg_domains=[
    {
        "name": "us_postal_code",
        "type": "text",
        "not_null": True,
        "check": r"VALUE ~ '^\d{5}(-\d{4})?$'",
    },
    {
        "name": "positive_int",
        "type": "int",
        "check": "VALUE > 0",
        "default": "1",
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE DOMAIN name AS base_type [DEFAULT default] [NOT NULL] [CHECK (expr)]` |
| Drop | `DROP DOMAIN IF EXISTS name CASCADE;` |
| Set default | `ALTER DOMAIN name SET DEFAULT expr;` |
| Drop default | `ALTER DOMAIN name DROP DEFAULT;` |
| Set not null | `ALTER DOMAIN name SET NOT NULL;` |
| Drop not null | `ALTER DOMAIN name DROP NOT NULL;` |
| Add constraint | `ALTER DOMAIN name ADD CONSTRAINT c CHECK (expr);` |
| Drop constraint | `ALTER DOMAIN name DROP CONSTRAINT c;` |
| Rename | `ALTER DOMAIN name RENAME TO new_name;` |
| Rename constraint | `ALTER DOMAIN name RENAME CONSTRAINT old TO new;` |
| Set schema | `ALTER DOMAIN name SET SCHEMA new_schema;` |

### Domain Constraint Validation

Domain constraints are checked on every use of the domain, not just at column creation. This means changing a domain's CHECK constraint can invalidate existing data in every table that uses the domain.

Changes to domain definition are detected as drop-then-create. When a domain check is relaxed or tightened, existing rows must satisfy the new constraint or be updated first.

## Composite Types

**Handler**: `CompositeTypeHandler` (PREAMBLE phase)

Composite types are config-driven objects declared via `pg_composite_types`:

```python
pg_composite_types=[
    {
        "name": "address",
        "columns": [
            {"name": "street", "type": "text"},
            {"name": "city", "type": "text"},
            {"name": "zip", "type": "text"},
        ],
    },
]
```

### Lifecycle

| Operation | DDL |
|-----------|-----|
| Create | `CREATE TYPE name AS (col1 type1, col2 type2, ...)` |
| Drop | `DROP TYPE IF EXISTS name CASCADE;` |

### Composite Type Modification

PostgreSQL does **not** support `ALTER TYPE` for composite types. To modify a composite type:
1. `DROP TYPE name CASCADE;` (automatically drops dependent columns/tables)
2. `CREATE TYPE name AS (...);` with the new definition
3. Recreate any dropped columns referencing this type

This is **not** automated by dbwarden. Composite type changes are detected as drop-then-create with `CASCADE`.

### Schema

Composite types can be scoped to a schema:

```python
{"name": "address", "schema": "app", "columns": [...]}
```

## Range Types

PostgreSQL supports built-in range types that dbwarden normalizes during schema extraction:

| PostgreSQL Type | SQLAlchemy Type | Example Value |
|----------------|-----------------|---------------|
| `INT4RANGE` | `INT4RANGE` | `[1, 10)` |
| `INT8RANGE` | `INT8RANGE` | `[100, 200)` |
| `NUMRANGE` | `NUMRANGE` | `[0.0, 1.0)` |
| `DATERANGE` | `DATERANGE` | `[2024-01-01, 2024-12-31)` |
| `TSTZRANGE` | `TSTZRANGE` | `["2024-01-01 00:00:00+00", "2024-12-31 23:59:59+00")` |
| `TSRANGE` | `TSRANGE` | `[2024-01-01, 2024-12-31)` |

Range type bounds are canonicalized as `[lower, upper)` (inclusive lower, exclusive upper) by PostgreSQL.

## Type Mapping Summary

See [Type Mapping](type-mapping.md) for the complete SQLAlchemy-to-PostgreSQL type normalization reference.

========================================================================
PAGE: https://docs.dbwarden.org/databases/postgresql/views/
========================================================================

# Views

**Handler**: `ViewHandler` (DIFF phase)

Views are model-derived via `PGViewMeta`.

## Regular Views

```python
from dbwarden.databases.pgsql import PGViewMeta

class ActiveUser(Base):
    __tablename__ = "active_users"

    id: Mapped[int] = mapped_column(Integer)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(PGViewMeta):
        pg_view_query = "SELECT id, email, name FROM users WHERE active = true"
        pg_view_materialized = False
```

Generated DDL:
```sql
CREATE OR REPLACE VIEW active_users AS SELECT id, email, name FROM users WHERE active = true;
```

## Materialized Views

```python
class OrderSummary(Base):
    __tablename__ = "order_summary"

    user_id: Mapped[int] = mapped_column(Integer)
    total: Mapped[float] = mapped_column(Integer)

    class Meta(PGViewMeta):
        pg_view_query = "SELECT user_id, count(*) AS total FROM orders GROUP BY user_id"
        pg_view_materialized = True
```

Generated DDL:
```sql
CREATE MATERIALIZED VIEW order_summary AS SELECT user_id, count(*) AS total FROM orders GROUP BY user_id;
```

## WITH Options

### Security Barrier

Control whether the view acts as a security barrier:

```python
class Meta(PGViewMeta):
    pg_view_query = "SELECT id, email FROM users WHERE active = true"
    pg_view_options = "security_barrier"
```

```sql
CREATE OR REPLACE VIEW active_users WITH (security_barrier) AS ...;
```

Security barrier views prevent leaked subquery optimizations when used for row-level access control. Use this for views that expose a subset of a table based on session variables.

### WITH CHECK OPTION

Control what rows can be inserted/updated through the view:

```python
class Meta(PGViewMeta):
    pg_view_query = "SELECT id, email, name FROM users WHERE active = true"
    pg_view_check_option = "LOCAL"  # or "CASCADED"
```

```sql
CREATE OR REPLACE VIEW active_users AS ... WITH LOCAL CHECK OPTION;
```

| Option | Behaviour |
|--------|-----------|
| `LOCAL` | Check only this view's WHERE clause |
| `CASCADED` (default) | Check this view's WHERE and all underlying views' WHERE clauses |

Without `WITH CHECK OPTION`, rows can be inserted or updated through the view even if they would not satisfy the view's WHERE clause (they become invisible through the view but exist in the underlying table).

## Auto-Refresh

Set `pg_view_auto_refresh = True` to emit `REFRESH MATERIALIZED VIEW` in subsequent migrations:

```python
class Meta(PGViewMeta):
    pg_view_query = "..."
    pg_view_materialized = True
    pg_view_auto_refresh = True
```

```sql
REFRESH MATERIALIZED VIEW order_summary;
```

### REFRESH CONCURRENTLY

To avoid table locking during refresh:

```python
class Meta(PGViewMeta):
    pg_view_query = "..."
    pg_view_materialized = True
    pg_view_auto_refresh = True
    pg_view_refresh_concurrently = True
```

```sql
REFRESH MATERIALIZED VIEW CONCURRENTLY order_summary;
```

`CONCURRENTLY` requires a unique index on the materialized view. It takes longer but allows concurrent reads and writes.

### WITH DATA / WITH NO DATA

Control whether the materialized view is populated on creation:

| Option | Behaviour |
|--------|-----------|
| `WITH DATA` (default) | Populate immediately |
| `WITH NO DATA` | Create empty; must `REFRESH` before querying |

```python
class Meta(PGViewMeta):
    pg_view_query = "..."
    pg_view_materialized = True
    pg_view_with_data = False
```

## Schema-Qualified Views

```python
class Meta(PGViewMeta):
    pg_view_query = "SELECT id, email FROM users"
    pg_view_materialized = False
    pg_schema = "app"
```

```sql
CREATE OR REPLACE VIEW app.active_users AS SELECT id, email FROM users;
```

## Recursive Views

```sql
CREATE RECURSIVE VIEW view_name (col1, col2, ...) AS
    SELECT ...   -- non-recursive term
    UNION ALL
    SELECT ...   -- recursive term
;
```

Recursive views are declared via `pg_view_query` with the full recursive query text.

## Temporary Views

```sql
CREATE TEMPORARY VIEW temp_active_users AS SELECT * FROM active_users;
```

Temporary views are session-scoped and dropped automatically at session end.

## Updatable Views

PostgreSQL automatically makes simple views updatable (single table, no aggregates, no DISTINCT, no set operations). Complex views require `INSTEAD OF` triggers for updates.

## Indexes on Materialized Views

Materialized views support indexes. Create indexes directly on the materialized view table:

```python
class Meta(PGViewMeta):
    pg_view_query = "..."
    pg_view_materialized = True
```

Indexes on materialized views are created via `pg_indexes` on `PGTableMeta` (not `PGViewMeta`). For materialized views with `pg_view_auto_refresh`, indexes persist across refreshes.

## INSTEAD OF Triggers

For complex views that need to support INSERT/UPDATE/DELETE, use `INSTEAD OF` triggers. `pg_triggers` requires the `dbwarden-pgsql-extensions` plugin (`dbwarden plugin add dbwarden-pgsql-extensions`).

```python
pg_triggers=[{
    "name": "trg_active_users_insert",
    "table": "active_users",
    "function": "insert_active_user",
    "timing": "INSTEAD OF",
    "events": ["INSERT"],
    "for_each": "ROW",
}]
```

See [Functions & Triggers](functions-and-triggers.md) for trigger configuration.

## Lifecycle

| Operation | DDL |
|-----------|-----|
| Create regular view | `CREATE OR REPLACE VIEW name AS query;` |
| Create matview | `CREATE MATERIALIZED VIEW name AS query;` |
| Refresh matview | `REFRESH MATERIALIZED VIEW name;` |
| Refresh matview concurrently | `REFRESH MATERIALIZED VIEW CONCURRENTLY name;` |
| Drop | `DROP VIEW IF EXISTS name CASCADE;` / `DROP MATERIALIZED VIEW IF EXISTS name CASCADE;` |

========================================================================
PAGE: https://docs.dbwarden.org/databases/round-trip/
========================================================================

# Round Trip Support

A **round-trip** backend is one where dbwarden can both read schema (via `generate-models`) and write schema (via `make-migrations` / `migrate`).

## Supported Backends

| Backend | `database_type` | Round-Trip |
|---------|------------------|------------|
| PostgreSQL | `postgresql` | Yes |
| MySQL | `mysql` | Yes |
| ClickHouse | `clickhouse` | Yes |
| SQLite | `sqlite` | Yes |
| MariaDB | `mariadb` | Partial |

## How Round-Trip Verification Works

"First-class" means the round-trip is verified: reverse-engineer a live database with `generate-models`, feed the output back into `make-migrations`, and get **zero diff**.

## Per-Backend Details

### PostgreSQL

PostgreSQL is a **first-class backend** with full round-trip support. All metadata (identity columns, collation, storage, compression, generated columns, fillfactor, tablespace, inheritance, exclude constraints, deferrable foreign keys, and advanced index options) is captured by the snapshot, diffed correctly, and emitted as valid DDL.

See [PostgreSQL Deep Dive](postgresql/index.md) for the complete list of supported features.

### MySQL

MySQL is a **first-class backend** with full round-trip support. All metadata (engine, charset, collation, row format, auto_increment, unsigned columns, `ON UPDATE`, and column comments) is captured by the snapshot, diffed correctly, and emitted as valid DDL.

See [MySQL Deep Dive](mysql/index.md) for the complete list of supported features.

### ClickHouse

ClickHouse has full round-trip support: `generate-models` reads schema from a live ClickHouse server, and `make-migrations` / `migrate` auto-generates DDL for table operations.

Known edge cases:

- Materialized views are introspected but are not currently emitted as discoverable model tables.
- Some native types (`Enum8`/`Enum16`, `DateTime64`, `FixedString`, `UUID`) may map to generic SQLAlchemy types in generated models; the underlying CH types are preserved in `CHColumnMeta` where possible.
- Replicated engine string arguments and `SETTINGS` on materialized views require manual review.

See [ClickHouse Deep Dive](clickhouse/index.md) for the complete list of supported features.

### SQLite

SQLite is a **first-class backend** with full round-trip support. Table options (`WITHOUT ROWID`, `STRICT`), generated columns and column collations are captured by the snapshot, diffed correctly, and emitted as valid DDL. Changes SQLite's `ALTER TABLE` cannot express are emitted as a table rebuild, and the rollback is the rebuild in the other direction.

SQLite remains the usual choice for a `dev_database_url` as well; see [SQL Translation](../sql-translation.md).

See [SQLite Deep Dive](sqlite/index.md) for the complete list of supported features.

### MariaDB

MariaDB is supported as a separate `database_type` (`mariadb`), but it does **not** have round-trip support. You can use MariaDB as a target database for migrations, but `generate-models` and full schema introspection are not available. Use `make-migrations` to write migrations manually.

========================================================================
PAGE: https://docs.dbwarden.org/databases/sql-databases/
========================================================================

# SQL Databases

dbwarden supports PostgreSQL, MySQL, MariaDB, and SQLite. While all four share standard SQL DDL, each backend has distinct behaviors that affect generated migrations. This page documents backend-specific syntax, limitations, and edge cases.

## DDL Transactional Behavior

| Backend | Transactional DDL | Impact |
|---------|------------------|--------|
| PostgreSQL | Yes | Entire migration file succeeds or rolls back atomically |
| MySQL / MariaDB | No | Each DDL auto-commits; partial failure leaves schema inconsistent |
| SQLite | Mostly | Per-statement auto-commit outside explicit transactions |

**PostgreSQL**: DDL is transactional. If a migration file contains multiple statements and one fails, all prior DDL in that file is rolled back. This makes PostgreSQL the safest backend for automated migration runs.

PostgreSQL is also a **first-class backend** with full support for identity columns, collation, storage, compression, generated columns, table fillfactor, tablespace, inheritance, EXCLUDE constraints, deferrable FK options, and advanced index parameters. See [PostgreSQL Deep Dive](postgresql/index.md) for complete details.

**MySQL / MariaDB**: DDL statements auto-commit immediately. If a 5-statement migration fails on the 4th statement, the first 3 are already committed and cannot be rolled back. Manual inspection and recovery may be needed. Always test MySQL/MariaDB migrations in a staging environment first.

**SQLite**: DDL is transactional only within explicit `BEGIN/COMMIT` blocks. dbwarden migration files are executed with each statement as a separate implicit transaction. This means SQLite is similar to MySQL in practice; partial failure is possible.

## Column Rename

| Backend | Syntax | Supported |
|---------|--------|-----------|
| PostgreSQL | `ALTER TABLE t RENAME COLUMN old TO new` | Yes |
| SQLite | `ALTER TABLE t RENAME COLUMN old TO new` | Yes (3.25+) |
| MySQL | `ALTER TABLE t CHANGE old new type` | Workaround needed |
| MariaDB | `ALTER TABLE t CHANGE old new type` | Workaround needed |

PostgreSQL and SQLite (3.25+) support `RENAME COLUMN` natively. dbwarden emits the standard form for all backends. If you need a backend that does not support native column rename (e.g., MySQL < 8.0, MariaDB), you must write a manual migration or verify the generated SQL.

## Column Type Change

| Backend | Syntax | Supported |
|---------|--------|-----------|
| PostgreSQL | `ALTER TABLE t ALTER COLUMN c TYPE newtype` (commented-out `USING` by default, active with `--postgres-auto-using`) | Yes |
| MySQL / MariaDB | `ALTER TABLE t MODIFY COLUMN c newtype` | Yes |
| SQLite | Table rebuild | Yes |

**PostgreSQL**: Emits `ALTER TABLE t ALTER COLUMN c TYPE newtype` with a commented-out `-- USING col::newtype` line. Pass `--postgres-auto-using` on `make-migrations` to emit an active `USING` clause.

**MySQL / MariaDB**: Emits `ALTER TABLE t MODIFY COLUMN c newtype`. Note that `MODIFY COLUMN` requires specifying the entire column definition, not just the type. dbwarden includes only the type in the `MODIFY` statement; if you need additional attributes (e.g., `NOT NULL`, `DEFAULT`), add them manually.

**SQLite**: `ALTER COLUMN TYPE` does not exist, so dbwarden rebuilds the table: create the new shape under a temporary name, copy the rows, drop the original, rename into place, recreate the indexes. The rollback is the rebuild in the other direction. See [SQLite Deep Dive](sqlite/index.md#table-rebuilds).

## Column Nullable Change

| Backend | Syntax | Supported |
|---------|--------|-----------|
| PostgreSQL | `ALTER TABLE t ALTER COLUMN c [SET/DROP] NOT NULL` | Yes |
| MySQL / MariaDB | `ALTER TABLE t MODIFY COLUMN c coltype [NOT] NULL` | Yes |
| SQLite | Table rebuild | Yes |

**PostgreSQL**: Uses `SET NOT NULL` / `DROP NOT NULL`. No column type needed.

**MySQL / MariaDB**: Uses `MODIFY COLUMN` which requires the full column type. dbwarden includes the type from the model column definition. If the type is not available, nullable changes for MySQL/MariaDB may produce incomplete SQL.

**SQLite**: Nullability is part of the column definition and can only change by rebuilding the table, which dbwarden generates. See [SQLite Deep Dive](sqlite/index.md#table-rebuilds).

## Column Default Change

PostgreSQL, MySQL and MariaDB support `ALTER TABLE t ALTER COLUMN c SET DEFAULT value` and `ALTER TABLE t ALTER COLUMN c DROP DEFAULT`. SQLite has no `ALTER COLUMN`, so a default change is emitted as a table rebuild.

## Foreign Key Handling

| Backend | ADD FK | DROP FK | Notes |
|---------|--------|---------|-------|
| PostgreSQL | `ADD CONSTRAINT ... FOREIGN KEY` | `DROP CONSTRAINT ...` | Supports `ON DELETE`, `ON UPDATE`, `DEFERRABLE` |
| MySQL | `ADD CONSTRAINT ... FOREIGN KEY` | `DROP FOREIGN KEY ...` | Uses constraint name, not FK name |
| MariaDB | `ADD CONSTRAINT ... FOREIGN KEY` | `DROP FOREIGN KEY ...` | Same as MySQL |
| SQLite | Table rebuild | Table rebuild | Constraint is inlined in the rebuilt CREATE TABLE |
| ClickHouse | Not supported (error) | Not supported (error) | `ForeignKey()` raises `DBWardenConfigError` |

**PostgreSQL FK options**: `ON DELETE`, `ON UPDATE`, and `DEFERRABLE INITIALLY DEFERRED` are fully supported. See [PostgreSQL Deep Dive](postgresql/index.md).

**Validation**: Before emitting `ADD FOREIGN KEY`, dbwarden verifies that the referenced table and columns exist in the snapshot. If they don't, the FK is silently skipped to avoid generating broken SQL. This can happen when the referenced table is added in the same migration batch. If an FK is unexpectedly missing from generated SQL, check whether the referenced table exists in the snapshot.

**Content-based comparison**: FKs are compared by a 6-tuple signature `(columns, ref_table, ref_columns, on_delete, on_update, deferrable)`, not by constraint name. Renaming an FK constraint or changing options produces a drop+add.

## Index Handling

| Backend | CREATE INDEX | DROP INDEX | Notes |
|---------|-------------|------------|-------|
| PostgreSQL | `CREATE [UNIQUE] INDEX [CONCURRENTLY] ... USING <method> INCLUDE (<cols>) WITH (<params>) WHERE <pred> TABLESPACE <ts>` | `DROP INDEX` | Full feature support |
| MySQL / MariaDB | `CREATE [UNIQUE] INDEX` | `DROP INDEX` | Standard |
| SQLite | `CREATE [UNIQUE] INDEX` | `DROP INDEX` | Standard |

**PostgreSQL advanced parameters**: all are supported in `_build_index_sql`. See [PostgreSQL Deep Dive](postgresql/index.md) for full coverage.

**PostgreSQL `CONCURRENTLY`**: dbwarden defaults to `CREATE INDEX CONCURRENTLY` to avoid table locking. Pass `--no-concurrent` when the migration must run inside a transaction block (PostgreSQL requires `CONCURRENTLY` outside a transaction).

**Full-content comparison**: Indexes are compared by **all** attributes (using, unique, where, include, with_params, tablespace, nulls_not_distinct, column_sorting, concurrently), not just columns + name. Any difference produces a drop+add; ALTER INDEX is not used.

**Auto-generated names**: `idx_{table}_{col1}_{col2}` (non-unique), `uq_{table}_{col1}_{col2}` (unique). Non-btree `USING` methods append a suffix: `idx_{table}_{col}_{method}`.

**No ALTER INDEX**: All index parameter changes (adding a WHERE clause, switching USING methods, changing sort order) produce `DROP INDEX` + `CREATE INDEX`. This is intentional; `ALTER INDEX` support varies widely across backends and index attribute types.

## Safe Type Change

| Backend | Supported | Behavior |
|---------|-----------|----------|
| PostgreSQL | Yes | Multi-step: add temp column, backfill comment, verify, drop+rename |
| MySQL / MariaDB | Yes | Multi-step: add temp column, backfill comment, verify, drop+rename |
| SQLite | Not needed | The rebuild already copies the table atomically |

The `--safe-type-change` flag generates a multi-step strategy:
1. Add a temporary column with the new type
2. Emit a `--` comment with an `UPDATE` statement template
3. Emit a verification comment
4. After manual verification, drop the old column and rename the temporary column

On SQLite the flag has no effect: a type change is already emitted as a table rebuild, which copies every row into the new column definition in one statement sequence rather than in manually verified steps.

## Table Rename

All four SQL backends support `ALTER TABLE t RENAME TO newname`. The syntax is uniform. ClickHouse is the only supported backend that does not support table rename (see [ClickHouse](clickhouse/index.md)).

## DROP COLUMN Warning

All DROP COLUMN statements are prefixed with a warning comment:

```sql
-- WARNING: DROPPING COLUMN users.legacy_field

ALTER TABLE users DROP COLUMN legacy_field
```

This applies to all SQL backends. The warning is a comment only and does not affect execution. In MySQL/MariaDB, be especially careful with DROP COLUMN since the DDL auto-commits and cannot be rolled back.

## DROP TABLE

`DROP TABLE` emits a rollback comment that references restoring from snapshot. The actual rollback SQL is a placeholder and must be written manually if needed.

## Statement Ordering

```
CREATE SEQUENCE      (0)
RENAME TABLE         (1)
RENAME COLUMN        (2)
ALTER COLUMN TYPE    (3)
ALTER COLUMN NULLABLE (4)
ALTER COLUMN DEFAULT (5)
CREATE TYPE          (6)
CREATE FUNCTION      (6)
CREATE TABLE         (7)
CREATE VIEW          (8)
ADD COLUMN           (9)
ALTER FOREIGN KEY    (10)
DROP VIEW            (11)
ALTER INDEX          (12)
DROP COLUMN          (13)
DROP TABLE           (14)
ALTER TABLE COMMENT  (15)
ALTER COLUMN COMMENT (16)
ALTER TABLE OPTIONS  (17)
ALTER TABLE CONSTRAINT (18)
```

This ordering ensures safe execution across all SQL backends. Table renames come first so all subsequent ops use the new name. Drops come last to minimize risk of referencing dropped objects.

## Migration Name Generation

Auto-generated migration names are truncated to 72 characters. Operation words (`add`, `drop`, `alter`, `create`, `rename`, `add_columns`, etc.) are preserved during truncation; table and column names are shortened as needed. This applies uniformly across all backends.

## Resolved From Values

The plan JSON `resolved_from` field indicates how a rename was confirmed:

| Value | Meaning |
|-------|---------|
| `"rename_flag"` | Explicitly declared via `--rename` or `--rename-table` CLI flag |
| `"prompt"` | Confirmed interactively by the user |
| (absent) | Auto-detected and kept without explicit confirmation |

## Known Backend-Specific Limitations

### PostgreSQL
- `USING` clause for type casts is not auto-generated (write a manual migration)
- `CREATE INDEX CONCURRENTLY` may not work within multi-statement transactions
- `ALTER COLUMN ADD GENERATED ... AS (expr) STORED` is not supported by PostgreSQL; dbwarden emits a comment placeholder

### MySQL / MariaDB
- DDL is not transactional; partial failure leaves schema inconsistent
- `MODIFY COLUMN` requires full column definition; auto-generated nullable change SQL includes the column type but may omit other attributes
- Column rename is not natively supported (requires `CHANGE` syntax with type)
- Foreign key drop uses `DROP FOREIGN KEY` (constraint name is still the auto-generated name)

### SQLite
- Column type, nullability, default and constraint changes are emitted as a table rebuild; the migration copies every row and holds a write lock for its duration
- `DROP COLUMN` in a rebuild discards the column's data; the rollback restores the column, not the values
- Column rename supported since 3.25.0; `DROP COLUMN` since 3.35.0
- Declared types are preserved as written; inside a `STRICT` table they collapse to `INTEGER` / `REAL` / `TEXT` / `BLOB`
- Foreign key enforcement is off by default, which the rebuild relies on; do not enable `PRAGMA foreign_keys` on the migration connection
- Limited to single-writer; no concurrent writes

========================================================================
PAGE: https://docs.dbwarden.org/databases/sqlite/alter-semantics/
========================================================================

# ALTER semantics

SQLite ALTERs behave differently from every other database dbwarden supports. The native grammar is deliberately minimal, the schema is stored as plain text that ALTER rewrites, and DDL is fully transactional. Understanding these properties explains why dbwarden emits a native `ALTER TABLE` for some changes and a full table rebuild for everything else.

## Why SQLite ALTERs are special

Three properties set SQLite apart:

1. **A minimal native grammar.** Each `ALTER TABLE` statement performs exactly one action, and only a handful of actions exist. There is no `ALTER COLUMN TYPE`, no multi-action comma form, and (before 3.53.0) no way to add or drop a constraint after creation. Anything outside the native forms requires rebuilding the table, and dbwarden generates that rebuild for you.

2. **The schema is stored as text.** SQLite keeps the original `CREATE` statements in `sqlite_schema` (`sqlite_master`). Every `ALTER TABLE` modifies that text and reparses the entire schema; the statement only succeeds if the schema is still valid afterwards. Renames therefore rewrite the stored SQL of *other* objects (triggers, views, foreign key clauses), not just the table's own row.

3. **DDL is transactional, but there is only one writer.** SQLite transactions are atomic — "all changes within a single transaction in SQLite either occur completely or not at all" ([sqlite.org](https://sqlite.org/transactional.html)) — so a failed migration rolls back cleanly, unlike MySQL or ClickHouse. The cost is concurrency: one writer at a time, and a rebuild holds the write lock while it copies every row.

## Native ALTER support

Feature-introduction timeline from the [SQLite changelog](https://sqlite.org/changes.html):

| Form | Since |
|------|-------|
| `ALTER TABLE ... RENAME TO` | 3.1.2 (2005) |
| `ALTER TABLE ... ADD COLUMN` | 3.2.0 (2005) |
| `ALTER TABLE ... RENAME COLUMN` | 3.25.0 (2018) |
| `ALTER TABLE ... DROP COLUMN` | 3.35.0 (2021) |
| `ALTER COLUMN ... SET/DROP NOT NULL`, `ADD/DROP CONSTRAINT ... CHECK` | 3.53.0 (2026) |

Two consequences:

- **No batching syntax.** One statement, one action. dbwarden never combines SQLite ALTERs — instead its collapse pass folds every change that needs a rebuild into a *single* rebuild per table (see [below](#how-dbwarden-decides-native-alter-vs-rebuild)).
- **dbwarden does not version-sniff.** It emits the same SQL against every SQLite version: the four classic forms natively, everything else as a rebuild. The 3.53.0 `SET/DROP NOT NULL` syntax is not used, so behavior is identical across versions; a native `DROP COLUMN` does require SQLite ≥ 3.35 at apply time (see [version gotchas](#version-specific-gotchas)).

## ADD COLUMN restrictions

From the [SQLite ALTER TABLE documentation](https://sqlite.org/lang_altertable.html), the new column may take any form permissible in `CREATE TABLE`, with these restrictions:

- The column may not have a `PRIMARY KEY` or `UNIQUE` constraint.
- The column may not have a default of `CURRENT_TIME`, `CURRENT_DATE`, `CURRENT_TIMESTAMP`, or an expression in parentheses.
- If `NOT NULL` is specified, the column must have a default other than `NULL`.
- With foreign key enforcement on, a column with a `REFERENCES` clause must default to `NULL`.
- The column may not be `GENERATED ALWAYS ... STORED` (`VIRTUAL` is allowed).

Two runtime subtleties:

- Since **3.37.0**, adding a column with a `CHECK` constraint — or a `NOT NULL` constraint on a generated column — validates the new constraint against **all preexisting rows** and fails the `ADD COLUMN` if any row violates it. Before 3.37.0 the violation went undetected at ALTER time.
- That validation makes the ALTER take time proportional to table size. Constraint-free additions only rewrite schema text and run in constant time.

### How dbwarden handles it

`add_column_requires_rebuild` in `sql_build.py` mirrors the SQLite rules: a primary key, `UNIQUE`, `STORED` generated, `NOT NULL` without a default, or non-constant-default column is emitted as a rebuild; anything else is a native `ALTER TABLE ... ADD COLUMN`.

## DROP COLUMN

`DROP COLUMN` was added in **3.35.0** and works only when the column is not referenced elsewhere in the schema. Per the [documentation](https://sqlite.org/lang_altertable.html), it fails when the column:

- is a `PRIMARY KEY` or part of one,
- has a `UNIQUE` constraint,
- is indexed, or named in a partial index's `WHERE` clause,
- is named in a `CHECK` constraint not associated with the column itself,
- is used in a foreign key constraint,
- is used in a generated column expression,
- appears in a trigger or view.

Unlike `ADD COLUMN`, dropping a column rewrites every row, so it takes time proportional to table size even when it is natively supported.

!!! warning "DROP COLUMN corrupted databases before 3.35.5"

    The `DROP COLUMN` implementation in 3.35.0–3.35.4 "could corrupt the database file" when the table was rewritten ([changelog](https://sqlite.org/changes.html), fixed in 3.35.5). Do not rely on native `DROP COLUMN` against SQLite older than **3.35.5** — and never against < 3.35.0, where the statement does not exist.

### How dbwarden handles it

`drop_column_requires_rebuild` in `sql_build.py` checks the same reference list (primary key, index, constraint, generated expression) and falls back to a rebuild when any apply. A rebuild that drops a column discards its data; the rollback restores the column but not its contents, and the migration is reported as a conditional rollback for that reason.

## RENAME semantics and stored schema text

Renames are the subtlest SQLite ALTERs because they rewrite stored SQL text:

- Since **3.25.0**, `RENAME TO` and `RENAME COLUMN` also rewrite references inside triggers and views that mention the renamed object.
- Foreign key `REFERENCES` clauses in *other* tables are rewritten too — only when `PRAGMA foreign_keys=ON` before 3.26.0, unconditionally since 3.26.0:

| `foreign_keys` | `legacy_alter_table` | FK references rewritten | Version |
|---|---|---|---|
| Off | Off | No | < 3.26.0 |
| Off | Off | Yes | ≥ 3.26.0 |
| On | Off | Yes | all |
| any | On | No | all |

- `PRAGMA legacy_alter_table=ON` (3.25.2+) restores the pre-3.25 behavior where only the object's own `CREATE` statement is edited and references inside trigger/view bodies are left pointing at the old name. It is a per-connection, non-persistent setting intended as a workaround for programs that depend on the old behavior. dbwarden never sets it.

Because the rewrite is textual, an `ALTER TABLE` also fails — changing nothing — if *any* row in `sqlite_schema` does not parse (that strictness can be disabled with `PRAGMA writable_schema=ON` since 3.38.0, which dbwarden does not use).

### The ordering trap

The [official 12-step procedure](https://sqlite.org/lang_altertable.html) for arbitrary schema changes is explicit about ordering:

> The safe procedure constructs the revised table definition using a new temporary name, then renames the table into its final name... Renaming the original table away first "might corrupt references to that table in triggers, views, and foreign key constraints."

```sql
-- Correct (what dbwarden generates)
CREATE TABLE users__dbw_new (...);
INSERT INTO users__dbw_new SELECT ... FROM users;
DROP TABLE users;
ALTER TABLE users__dbw_new RENAME TO users;

-- Incorrect: renaming the original first can corrupt
-- trigger, view, and FK references to it (3.25.0+)
ALTER TABLE users RENAME TO users_old;
CREATE TABLE users (...);
```

dbwarden's rebuild always uses the safe create → copy → drop → rename order, with the staging table named `<table>__dbw_new`. See [Table rebuilds](index.md#table-rebuilds) for the full generated sequence.

## PRAGMA foreign_keys is a no-op inside a transaction

From the [PRAGMA documentation](https://sqlite.org/pragma.html):

> This pragma is a no-op within a transaction; foreign key constraint enforcement may only be enabled or disabled when there is no pending BEGIN or SAVEPOINT.

This is why the official procedure disables foreign keys *before* starting the transaction, and it constrains migration tooling: dbwarden runs each migration inside a single transaction (see [locking](#locking-and-busy_timeout) below), so a `PRAGMA foreign_keys` emitted mid-migration would silently do nothing. dbwarden therefore never emits it and relies on the SQLite default (`OFF`). If your deployment enables foreign keys on the migration connection, turn them off before running the migration — with enforcement on, the `DROP TABLE` step of a rebuild can be blocked or cascade into referencing tables.

A related pragma, `PRAGMA defer_foreign_keys=ON`, delays FK enforcement until commit for the current transaction — useful in hand-written data migrations, and automatically reset at each `COMMIT` or `ROLLBACK`.

## Locking and busy_timeout

SQLite allows any number of readers but exactly one writer:

- In rollback-journal mode, a writer must escalate to an `EXCLUSIVE` lock to commit; active readers block that escalation and the writer fails with `SQLITE_BUSY` once the busy timeout expires ([locking architecture](https://sqlite.org/lockingv3.html)).
- In [WAL mode](https://sqlite.org/wal.html), readers never block the writer and vice versa, but there is still only one writer — a concurrent migration still gets `SQLITE_BUSY`.

dbwarden's SQLite lock strategy (`dbwarden/lock/sqlite.py`) runs the **entire migration in one `BEGIN IMMEDIATE` transaction** on the lock connection. That acquires the write lock up front, makes the whole migration atomic, and means a crash mid-migration releases everything with no schema change applied. The busy timeout is controlled by the `sqlite_busy_timeout` config key and defaults to `0` (fail fast with `SQLITE_BUSY` rather than queue behind another writer).

The practical consequence for rebuilds: a rebuild copies every row while holding the database-wide write lock. On a large table that is a maintenance-window operation, which is why `check-impact` flags rebuild-triggering option changes as warnings.

## How dbwarden decides: native ALTER vs rebuild

The SQLite backend runs a **collapse pass** (`collapse_sqlite_ops` in `collapse.py`) over the diff before any SQL is emitted:

1. Ops that SQLite can express natively pass through unchanged: `rename_table`, `rename_column`, `add_index`/`drop_index`, and the `add_column`/`drop_column` variants described above.
2. Every remaining op on a table — type, nullability, or default changes, constraint adds/drops, `WITHOUT ROWID`/`STRICT` changes, generated-column or collation changes — is folded into **one** `recreate_sq_table` op per table, regardless of how many individual changes were requested.
3. Index ops on a table being rebuilt are dropped from the op list, because the rebuild recreates the table's indexes itself.
4. If the before/after table shapes needed to build the rebuild are unavailable, the original ops are kept so the failure is visible: emit falls back to comment-only statements such as `-- SQLite: ADD CONSTRAINT ... (not supported)`.

The full trigger matrix is documented in [What triggers a rebuild](index.md#what-triggers-a-rebuild).

### dbwarden's batching behavior

| Handler | Emits natively | Batching |
|---------|----------------|----------|
| SqTableHandler | Rebuild script only | All table/column-meta changes collapse into one rebuild per table |
| ColumnHandler | `RENAME COLUMN`, `ADD COLUMN`, `DROP COLUMN` | One statement per op |
| RenameTableHandler | `ALTER TABLE ... RENAME TO` | One statement per op |
| IndexHandler | `CREATE [UNIQUE] INDEX` / `DROP INDEX` | One statement per op; folded into the rebuild when the table rebuilds |
| ConstraintHandler | — (rebuild via collapse) | Comment-only fallback when the rebuild can't be constructed |

A rebuild is a single `MigrationStatement` whose SQL is a multi-statement script (create, copy, drop, rename, recreate indexes); the statements execute individually, in order, inside the migration transaction.

## Operation catalog

Complete reference of the ALTER shapes dbwarden generates for SQLite, with their classification. Two systems apply: the **`--force` gate** (blocks migration generation/apply on dangerous changes; shown as the Safety column below, `—` = not gated) and the **impact-plan severity** reported by `dbwarden check-impact`, which additionally rates `drop_column` as ERROR and `drop_index` / `drop_foreign_key` / `recreate_sq_table` as WARNING. Legacy `WARNING` / `ERROR` display labels map to `WARN` / `CRITICAL`. Use the generated plan for operation-specific decisions and the [safety scope contract](../../correctness/safety-scoped-migrations.md) for gates.

### Native ALTER operations

| Operation | Handler | Emitted as | Safety | Notes |
|-----------|---------|-----------|--------|-------|
| Rename table | RenameTableHandler | `ALTER TABLE old RENAME TO new;` | — | Rewrites trigger/view/FK text (3.25.0+) |
| Rename column | ColumnHandler | `ALTER TABLE t RENAME COLUMN a TO b` | — | 3.25.0+ |
| Add column | ColumnHandler | `ALTER TABLE t ADD COLUMN c <def>` | INFO | Restrictions above; else rebuild |
| Drop column | ColumnHandler | `ALTER TABLE t DROP COLUMN c` | WARNING, `--force` | 3.35.0+ (≥3.35.5 advised); rewrites all rows |
| Add index | IndexHandler | `CREATE [UNIQUE] INDEX ix ON t (...)` | INFO | Partial/expression indexes preserved |
| Drop index | IndexHandler | `DROP INDEX ix` | — | Not gated; impact plan rates WARNING |

### Rebuild operations (one rebuild per table, any mix of triggers)

| Trigger | Safety | Notes |
|---------|--------|-------|
| Column type change | WARNING, `--force` | Generic `change_column_type` classification |
| `sq_without_rowid` / `sq_strict` change | WARNING, `--force` | Flagged with "(rebuilds the table and copies every row)" |
| `sq` generated column / collation change | WARNING, `--force` | Same message |
| Column nullability / default change | — | Not separately flagged; rebuild is visible in the migration |
| Add/drop `UNIQUE`, `CHECK`, `FOREIGN KEY` | — | Constraints can only enter via the CREATE statement |
| `ADD COLUMN` hitting a native restriction | INFO (as add column) | PK, UNIQUE, NOT NULL w/o default, non-constant default, STORED generated |
| `DROP COLUMN` hitting a native restriction | WARNING, `--force` | PK, indexed, constraint-referenced, generated-referenced |

The rebuild's rollback is the rebuild in the other direction (`rollback_kind="real"`), except when columns were dropped — then the rollback restores the columns but not their data and is reported as conditional.

## Version-specific gotchas

| Version | Issue |
|---------|-------|
| 3.25.0 | Rename sanity-check false positives roll back valid ALTERs; fixed in 3.25.1 |
| 3.25.0–3.25.x | FK references rewritten on rename only with `foreign_keys=ON`; unconditional from 3.26.0 |
| 3.35.0–3.35.4 | `DROP COLUMN` can corrupt the database file; fixed in 3.35.5 |
| < 3.37.0 | `ADD COLUMN` does not validate new `CHECK`/`NOT NULL` constraints against existing rows |
| 3.7.0–3.51.2 | WAL-mode reset bug can corrupt the database when a writer and a checkpoint race; fixed in 3.51.3 (backports 3.44.6, 3.50.7) |

Recommended floor: **3.26.0** for predictable rename behavior, **3.35.5** if you rely on native `DROP COLUMN`, **3.37.0** for `STRICT` tables and `ADD COLUMN` validation, **3.51.3** (or a backport) for WAL deployments.

## Related

- [SQLite overview](index.md): table rebuilds, `WITHOUT ROWID`/`STRICT`, generated columns, type fidelity
- [Migration locking](../../advanced/migration-locking.md): how the `BEGIN IMMEDIATE` lock coordinates concurrent runs
- [Rollback generation](../../correctness/rollback-generation.md): the rollback contract rebuilds satisfy
- [Common SQL databases](../sql-databases.md): cross-backend DDL behavior comparison

========================================================================
PAGE: https://docs.dbwarden.org/databases/sqlite/
========================================================================

# SQLite

dbwarden treats SQLite as a **first-class backend**: every natively supported feature is reverse-engineered, diffed, and emitted as correct DDL, and every change SQLite's `ALTER TABLE` cannot express is emitted as a table rebuild rather than as a comment telling you to write the migration yourself.

Database declarations use `DbwardenDatabase` subclasses by default. The equivalent `database_config(...)` function API remains supported for existing projects.

## First-Class Features

"First-class" means the round-trip is verified: reverse-engineer a live database with `generate-models`, feed the output back into `make-migrations`, and get **zero diff**.

```bash
# Step 1: reverse-engineer your live SQLite database
$ dbwarden generate-models -d primary

# Step 2: feed the generated models back in, zero diff
$ dbwarden make-migrations -d primary
# -> "No new migrations to generate"  (your models match the DB exactly)
```

| Category | Features |
|----------|----------|
| Table options | `WITHOUT ROWID`, `STRICT` via `sq_without_rowid`, `sq_strict` |
| Generated columns | `GENERATED ALWAYS AS (expr) STORED / VIRTUAL` via `sq.field(generated=..., generated_mode=...)` |
| Collation | Per-column `COLLATE` via `sq.field(collate="NOCASE")` |
| Keys | `INTEGER PRIMARY KEY AUTOINCREMENT`, composite primary keys, `UNIQUE` and `CHECK` table constraints |
| Foreign keys | Column-level and multi-column `FOREIGN KEY` clauses with `ON DELETE` / `ON UPDATE` |
| Indexes | `CREATE [UNIQUE] INDEX`, partial indexes (`WHERE`), expression indexes |
| Native ALTER | `RENAME TO`, `RENAME COLUMN`, `ADD COLUMN`, `DROP COLUMN` |
| Everything else | Emitted as a table rebuild (see below) |

## Installation

SQLite ships with Python, so no extra dependency group is needed:

```bash
uv add dbwarden
```

## Configuration

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "sqlite"
    database_url_sync = "sqlite:///./app.db"
```

## Table Rebuilds

SQLite's `ALTER TABLE` supports only `RENAME TO`, `RENAME COLUMN`, `ADD COLUMN` and `DROP COLUMN`. Anything else - a column's type, its nullability, its default, a table constraint, `WITHOUT ROWID`, `STRICT` - is expressed by rebuilding the table.

dbwarden generates that rebuild for you, following the procedure from the SQLite documentation:

```sql
CREATE TABLE users__dbw_new (
    id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
    email VARCHAR(255) NOT NULL,
    age TEXT,
    CONSTRAINT uq_users_email UNIQUE (email)
);
INSERT INTO users__dbw_new (id, email, age) SELECT id, email, age FROM users;
DROP TABLE users;
ALTER TABLE users__dbw_new RENAME TO users;
CREATE INDEX IF NOT EXISTS ix_users_email ON users (email);
```

Notes on the generated sequence:

- **All changes to one table collapse into one rebuild.** Changing three columns and adding a constraint produces a single copy of the table, not four.
- **The rollback is a rebuild in the other direction**, so a SQLite migration satisfies the [rollback contract](../../correctness/rollback-generation.md) instead of emitting a placeholder.
- **The staging table is created without `IF NOT EXISTS`.** A leftover `*__dbw_new` table from a failed run fails the migration rather than silently receiving the copied rows.
- **Indexes are recreated** after the rename, because `DROP TABLE` takes the original table's indexes with it.
- **Generated columns are not copied**; SQLite recomputes them.
- **Dropped columns lose their data.** The rollback restores the column but not its contents, and the migration is reported as a conditional rollback for that reason.
- **Foreign key enforcement.** dbwarden never issues `PRAGMA foreign_keys=ON`, and SQLite defaults it to off, so the `DROP TABLE` step is not blocked by other tables referencing this one. If your deployment enables foreign keys on the migration connection, turn them off for the duration of the migration.
- **Index statements for a rebuilt table are folded into the rebuild.** A rebuild recreates the table's indexes itself, so a migration that both rebuilds a table and changes one of its indexes emits the rebuild alone rather than an index statement that the rebuild would immediately discard.

### What the rebuild preserves

A rebuild reproduces the table from what dbwarden reads back from the database. Reflection alone is lossy, so several things are read from the `CREATE TABLE` and `CREATE INDEX` text SQLite stores in `sqlite_master`:

| Preserved | Read from |
|-----------|-----------|
| Declared type, exactly as written (`VARCHAR(255)`, not `varchar`) | stored table DDL |
| `AUTOINCREMENT` | stored table DDL |
| Generated columns and their `STORED` / `VIRTUAL` mode | stored table DDL |
| Column `COLLATE` | stored table DDL |
| `ON DELETE` / `ON UPDATE` on foreign keys | `PRAGMA foreign_key_list` |
| Unnamed `UNIQUE (...)` table constraints | reflection, named `uq_<table>_<columns>` |
| Partial indexes (`WHERE`), `DESC` and `COLLATE` inside an index, expression indexes | stored index DDL |

An index the migration does not change is recreated from its stored DDL rather than from the reflected shape, so index details SQLAlchemy cannot report survive the rebuild. An index the model changes is rendered from the model.

### What Triggers a Rebuild

| Change | Emitted as |
|--------|-----------|
| Column type change | Rebuild |
| Column nullability change | Rebuild |
| Column default change | Rebuild |
| Add / drop `UNIQUE`, `CHECK` or `FOREIGN KEY` constraint | Rebuild |
| `WITHOUT ROWID` or `STRICT` change | Rebuild |
| Generated expression or collation change | Rebuild |
| Add column that is a primary key, `UNIQUE`, `NOT NULL` without a default, a STORED generated column, or has a non-constant default | Rebuild |
| Drop column that is part of the primary key, is indexed, is named by a table constraint, or is referenced by a generated column | Rebuild |
| Add column (otherwise) | `ALTER TABLE ... ADD COLUMN` |
| Drop column (otherwise) | `ALTER TABLE ... DROP COLUMN` |
| Rename table / rename column | `ALTER TABLE ... RENAME` |
| Add / drop index | `CREATE INDEX` / `DROP INDEX` |

A rebuild copies every row and holds a write lock for the duration. `dbwarden check-impact` reports SQLite table-option, generated-column and collation changes as warnings for that reason.

For the semantics behind these statements — native ALTER restrictions, rename rewriting, version-specific behavior, and locking — see [ALTER Semantics](alter-semantics.md).

## Table Metadata

```python
from dbwarden import SqTableMeta
from dbwarden.databases.sqlite import sq


class Session(Base):
    __tablename__ = "sessions"

    key: Mapped[str] = mapped_column(String, primary_key=True)
    value: Mapped[str] = mapped_column(String)

    class Meta(SqTableMeta):
        sq_without_rowid = True
        sq_strict = True
```

| Attribute | Effect |
|-----------|--------|
| `sq_without_rowid` | Appends `WITHOUT ROWID`. The table must have a primary key, and `AUTOINCREMENT` is not available on it. |
| `sq_strict` | Appends `STRICT`. Column types are collapsed to the set SQLite accepts (see below). |
| `sq_indexes` | SQLite-specific index specs, merged with `indexes` |

## Column Metadata

```python
from dbwarden import SqColumnMeta, SqTableMeta
from dbwarden.databases.sqlite import sq


class Article(Base):
    __tablename__ = "articles"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String)
    slug: Mapped[str] = mapped_column(String)
    code: Mapped[str] = mapped_column(String)

    class Meta(SqTableMeta):
        class slug(SqColumnMeta):
            sq = sq.field(generated="lower(title)")

        class code(SqColumnMeta):
            sq = sq.field(collate="NOCASE")
```

| `sq.field(...)` argument | Effect |
|--------------------------|--------|
| `generated` | `GENERATED ALWAYS AS (<expr>)` |
| `generated_mode` | `STORED` (default) or `VIRTUAL` |
| `collate` | `COLLATE <name>` |

A `VIRTUAL` generated column can be added to an existing table with `ALTER TABLE ADD COLUMN`; a `STORED` one cannot, and forces a rebuild.

## Types and STRICT Tables

Outside a `STRICT` table, SQLite accepts any type name and reads it only to pick a column affinity, so dbwarden emits the declared type unchanged: `VARCHAR(255)` stays `VARCHAR(255)`. This keeps the schema readable and keeps `generate-models` round-trips stable.

Inside a `STRICT` table only `INT`, `INTEGER`, `REAL`, `TEXT`, `BLOB` and `ANY` are legal, so the declared type is collapsed to the closest one:

| Declared | STRICT |
|----------|--------|
| `VARCHAR(n)`, `CHAR(n)`, `CLOB` | `TEXT` |
| `BOOLEAN` | `INTEGER` |
| `DATE`, `DATETIME`, `TIMESTAMP`, `TIME` | `TEXT` |
| `NUMERIC(p,s)`, `DECIMAL(p,s)` | `REAL` |
| `BIGINT`, `SMALLINT`, `INT` | `INTEGER` |
| `FLOAT`, `DOUBLE PRECISION` | `REAL` |

## AUTOINCREMENT

`AUTOINCREMENT` is emitted only where SQLite permits it: a single-column `INTEGER PRIMARY KEY` on a rowid table. On a composite key, a non-integer key, or a `WITHOUT ROWID` table it is omitted, because the declaration would be rejected.

A SQLite `INTEGER PRIMARY KEY` is the rowid alias whether or not `AUTOINCREMENT` is written, so dbwarden does not generate autoincrement changes for primary key columns on SQLite.

## Primary Key Nullability

SQLite reports a primary key column as nullable unless it was declared `NOT NULL`, while a mapped model always declares its primary key `NOT NULL`. dbwarden records primary key columns as `NOT NULL` when reading a SQLite schema, so a table does not report a nullability change on its key column on every run.

## SQLite as a Development Database

SQLite is also the usual choice for `dev_database_url`, where the production schema is generated for another backend and translated for local use. That path is unchanged and documented separately in [SQL Translation](../../sql-translation.md).

## Data Conversion in a Rebuild

The rebuild copies rows with `INSERT INTO ... SELECT`, so SQLite's own rules apply to the data:

- In an ordinary table, SQLite converts values to the new column's affinity where it can and stores them as-is where it cannot; a type change never fails for the data's sake.
- In a `STRICT` table, SQLite rejects a value that does not fit the new type, and the migration fails:

    ```
    cannot store REAL value in INTEGER column sales__dbw_new.total
    ```

    That is the table doing its job. Convert the data first - in a `data_op()` or a hand-written migration - and then narrow the column.

- A column that becomes `NOT NULL` while existing rows hold NULL fails the same way. Give the column a default, or backfill first.

## Limitations

- DDL runs inside the migration transaction, but a rebuild rewrites the whole table: expect the migration to take time proportional to table size.
- SQLite allows a single writer; a rebuild blocks other writers for its duration.
- Column comments have no SQLite equivalent and are emitted as SQL comments.
- Schemas (`ATTACH`ed databases) are not modelled; `pg_schema`-style qualification does not apply.
- Dropping a column in a rebuild discards its data; the rollback restores the column, not the values.
- Views are not rebuilt with their table. A view over a rebuilt table survives because SQLite resolves it by name, but a view that names a dropped column breaks at query time.
- `--safe-type-change` has no effect: the rebuild is already a single copy of the table.

========================================================================
PAGE: https://docs.dbwarden.org/declarative-data-migrations/
========================================================================

# Declarative data migrations

Declarative data migrations describe intended rows and transformations beside SQLAlchemy models. DBWarden compiles declarations into a canonical data specification, SQL, a checksummed plan, and a frozen `.data.py` artifact. Applying a migration uses those artifacts. It does not import current declarations.

Use this for reviewed changes to data that belongs with a schema release. Declarations generate versioned migrations; they cannot be placed in repeatable migration files. Keep one-off repair SQL and application backfills separate.

## Configure discovery

`model_paths` contains mapped SQLAlchemy models. `data_paths` optionally contains modules with `DataTransition` declarations. DBWarden does not treat `*.data.py` files as live declarations.

```python
from dbwarden import database_config

database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://localhost/app",
    model_paths=["app/models.py"],
    data_paths=["app/data"],
    data_snapshot_dir=".dbwarden/data",
    snapshot_registry=".dbwarden/snapshots/registry.json",
)
```

`model_paths` and `data_paths` name Python files or directories, not dotted module names. Missing configured paths fail discovery. Symlink paths, duplicate declaration IDs, duplicate owners of a table, and reserved DBWarden tables are rejected. `model_tables` restricts model discovery, including model data declarations. Two database entries cannot use the same live data path unless their existing `overlap_models=True` opt-in permits it. `data_snapshot_dir` and `snapshot_registry` are relative paths. They hold compiled data snapshots and historical schema registry metadata.

Declaration IDs default to the project-relative module path and class name. Set `declaration_id` on a mapped model or transition to retain its identity when moving code. Keep an ID only for the same logical owner. Changing managed key columns or owned columns under an existing ID fails planning.

## Managed rows

Put a `Data` inner class on a mapped model. `rows()` declares key columns, literal values, columns owned by the declaration, missing-row handling, and rollback policy.

```python
from sqlalchemy import String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.data import DataMeta, rows

class Base(DeclarativeBase):
    pass

class Country(Base):
    __tablename__ = "countries"
    code: Mapped[str] = mapped_column(String(2), primary_key=True)
    name: Mapped[str] = mapped_column(String, nullable=False)

    class Data(DataMeta):
        managed_rows = rows(
            key="code",
            rows=[{"code": "UY", "name": "Uruguay"}],
            owned_columns=["name"],
            rollback="restore_previous",
        )
```

`rows()` accepts exactly one of `rows=` and `source=`. `source=` names a static CSV or JSON file. It is read during compilation and becomes canonical values in the frozen specification. Keys and owned columns must exist on the target table. `on_missing="delete"` requires a scope expression and `acknowledge_delete=True`. Choose `rollback="restore_previous"` when the generated migration can restore prior values, or `"irreversible"` when it cannot.

Omitted keys use the model's primary key. Omitted `owned_columns` use the declared non-key columns. DBWarden records ownership separately from application rows. Missing-row deletion only removes previously owned rows inside the explicit scope; it does not adopt every row with a matching key. A pre-existing row is accepted only under the declaration's value and ownership checks. Rollback checks current owned values before changing them, so later application edits cause a conflict instead of being overwritten.

CSV and JSON inputs must stay inside the project. CSV values are converted using column types. JSON rejects duplicate keys and non-finite numbers. Normalized strings and typed literals determine checksums, not source-file formatting. Removing a managed-row declaration does not silently delete its rows: first declare the intended deletion and generate its migration.

## Derived values and expressions

`derive()` owns one target column inside its declared domain. Expressions are typed ASTs, not Python callbacks or raw SQL.

```python
from dbwarden.data import DataMeta, col, derive, func

class Data(DataMeta):
    transformations = [
        derive("normalized_name", func.lower(col("name")), rollback="clear"),
    ]
```

Use `col(name, table=None)`, `literal(value)`, `param(name)`, `case(...)`, `cast(...)`, `mapping(...)`, and `func.<name>(...)` to build expressions. Arithmetic, comparisons, null tests, membership, and Boolean composition are supported. Combine predicates with `&`, `|`, and `~`; Python `and`, `or`, and `not` cannot build expressions. The function allowlist is `lower`, `upper`, `trim`, `concat`, `coalesce`, `json_extract`, `split_part`, and PostgreSQL `date_trunc`. Arbitrary functions, Python callbacks, raw SQL, volatile values, and unsupported casts fail compilation.

The compiler checks column types, nullability, ownership, and expression domains. A partial `case` or `mapping` needs an explicit fallback or a domain that proves it complete. `derive(..., when=..., on_unmatched="error" | "keep", max_rows=...)` controls its domain and affected-row limit. Rollback is required: `clear` requires a nullable target; `recompute` requires a previous expression that can be restored; `capture` records a guarded typed preimage; `irreversible` blocks rollback.

`date_trunc` requires `settings` containing nonempty `backend_version`, `timezone`, `collation`, `search_path`, and `encoding` strings. These settings become part of expression identity and are checked against PostgreSQL before execution. A numeric version pin matches the server major version; other version strings must match exactly.

Use `func.json_extract(value, "$.path")` for scalar JSON extraction and
`func.split_part(value, delimiter, index)` for one-based string splitting.
JSON paths and delimiters must be nonempty literals; the path begins with `$`.
JSON strings are decoded, numbers become text, booleans become `true` or
`false`, and JSON null, missing paths, objects, and arrays become SQL null.
ClickHouse rejects `json_extract` because its native result cannot preserve
that contract. The split index is a literal integer from 1 through 64. SQL
null stays null and a missing component returns an empty string. `split_part`
works on all five backends. DBWarden freezes both canonical expressions.

Supply parameters through repeated `--param NAME=VALUE` options on generation and data inspection commands, or through the Python generation API's `parameters` mapping. Values parse as JSON when possible, otherwise as strings. Missing or duplicate names fail. Frozen artifacts contain resolved canonical values, so execution does not reread parameters.

## Validation rules

`Data.validations` holds `validate(expression, message=...)` declarations. Each expression must hold for every target row; false and null results fail. Validation runs after relevant data writes and during `check --data`. Validation-only declarations need no rollback policy because they do not change rows.

```python
from dbwarden.data import DataMeta, col, validate

class Data(DataMeta):
    validations = [validate(col("name").is_not(None), message="Name is required")]
```

## Historical transitions

Use `historical_table()` to name a source table pinned to a snapshot. `DataTransition` sends source rows to one or more current mapped models with `into()`.

```python
from dbwarden.data import DataTransition, historical_table, into
from app.models import Customer, Person

legacy = historical_table("legacy_people", snapshot="primary__0007")

class SplitLegacyPeople(DataTransition):
    source = legacy
    source_identity = [legacy.id]
    overlap = "fan_out"
    targets = [
        into(
            Person,
            map={Person.id: legacy.id, Person.name: legacy.full_name},
            key=[Person.id],
            on_conflict="ignore_if_equivalent",
        ),
        into(
            Customer,
            map={Customer.person_id: legacy.id},
            key=[Customer.person_id],
            on_conflict="ignore_if_equivalent",
        ),
    ]
```

An explicit snapshot ID is recommended and required when compatible lineage is
ambiguous. Omitting it invokes the generation-time latest-compatible selection
described below; the frozen bundle still contains one exact ID. Current ORM
models describe targets. Historical source tables come from the pinned snapshot
and are not rediscovered at execution. A transition may preserve its source or
drop it only with the declaration's explicit completion and rollback rules. The
unified generation path removes the duplicate schema `drop_table` operation
when the transition owns that source removal.

Python target mappings use `{Target.column: source_expression}`. Target keys must identify stable rows. `on_conflict` is `error`, `ignore_if_equivalent`, or `overwrite`; overwrite requires `acknowledge_overwrite=True`. Equivalent pre-existing rows remain unowned, so rollback does not delete them. Identity ledgers bind each source key to its target key and mapped values.

| Setting | Behavior |
|---|---|
| `coverage="all"` | Every source row must match a target. |
| `coverage="subset", on_unmatched="ignore"` | Unmatched source rows are permitted. |
| `coverage="exactly_once_match"` | Each source row must match exactly one raw target predicate. |
| `coverage="all_assigned_once", overlap="priority"` | Each source row must have exactly one assignment after priority resolution. |
| `overlap="error"` | A source row matching multiple targets fails. |
| `overlap="fan_out"` | One source may populate several targets. |
| `overlap="priority"` | Distinct integer target priorities choose the first match; requires `all_assigned_once`. |
| `on_complete="keep"` | Keep a mapped current-model source. Requires `source_snapshot=...` and `on_unmatched="error"`. |
| `on_complete="preserve"` | Rename the source to its deterministic preservation table. |
| `on_complete="drop"` | Drop the source; requires `acknowledge_drop=True` and `rollback="irreversible"`. |
| `rollback="restore_preserved_source"` | Restore preserved source and remove only verified writes owned by the transition. |
| `max_rows` | Abort when source count exceeds the declared limit. |

Target foreign keys determine write order. Unsupported cycles, ambiguous identities, overlapping ownership, or references that would be broken by source removal fail before execution. Nullable expansion, data writes, validation, and non-null contraction share one dependency-ordered plan. A transition owns its source removal, preventing duplicate schema drops.

The older `exactly_once` spelling normalizes to `exactly_once_match`; `coverage="all"` with priority normalizes to `all_assigned_once`. Frozen plans store the canonical mode. Priority values on a non-priority transition are rejected.

To copy from a model that remains part of desired schema, assign the mapped
model class to `source`, set its exact `source_snapshot`, and select
`on_complete="keep"`. DBWarden requires complete coverage with
`on_unmatched="error"` and leaves the source unchanged on rollback. PostgreSQL
and SQLite keep the source name throughout. MySQL and MariaDB temporarily
freeze it with an atomic rename, verify the target, and restore its name before
success. A `historical_table(...)` or merged source cannot use `keep`.

## Deterministic merges

Use `merge_sources()` when several historical tables contribute to one current
target. Every source needs an exact snapshot, a stable identity, the same named
projection, and a distinct priority. Every non-key output needs an explicit
aggregate or winner rule.

```python
from dbwarden.data import (
    DataTransition,
    aggregate,
    batch,
    col,
    from_source,
    historical_table,
    into,
    merge_sources,
    winner,
)
from app.models import Customer

crm = historical_table("crm_customers", snapshot="primary__0010")
billing = historical_table("billing_customers", snapshot="primary__0010")

customers = merge_sources(
    from_source(
        crm,
        identity=[crm.id],
        map={
            "customer_id": crm.id,
            "display_name": crm.name,
            "balance": crm.balance,
        },
        priority=10,
    ),
    from_source(
        billing,
        identity=[billing.customer_id],
        map={
            "customer_id": billing.customer_id,
            "display_name": billing.name,
            "balance": billing.balance,
        },
        priority=20,
    ),
    key=["customer_id"],
    values={
        "display_name": winner(col("display_name")),
        "balance": aggregate("sum", col("balance")),
    },
    allow_empty=False,
)

class MergeCustomers(DataTransition):
    source = customers
    targets = [
        into(
            Customer,
            map={
                Customer.id: col("customer_id"),
                Customer.name: col("display_name"),
                Customer.balance: col("balance"),
            },
            key=[Customer.id],
            on_conflict="ignore_if_equivalent",
        )
    ]
    execution = batch(size=1000, key=["customer_id"], max_duration=60)
    on_complete = "preserve"
    rollback = "restore_preserved_source"
```

Supported aggregate names are `sum`, `avg`, `min`, `max`, `count`, and
`require_equal`. `count` counts non-null expression results.
`require_equal` fails when contributions differ or mix null with a non-null
value. For strings it compares UTF-8 bytes instead of the database's default
collation. `winner` selects the lowest numeric source priority, then orders by
the source identity's UTF-8 byte encoding. Source identities and grouping keys
must be non-null integers or strings; cast other types explicitly. Source
priorities must be distinct.

`allow_empty=False` is the default and stops before source retirement when all
inputs are empty. Set `allow_empty=True` only when an empty merged target is a
reviewed outcome. The flag changes merge identity. A merge supports preserved
or acknowledged dropped sources. Archive completion belongs to a normal
single-source transition.

DBWarden rejects arbitrary reducer functions, different projection fields,
nullable or duplicate source identities, missing grouping keys, ambiguous
field rules, repeated input relations, ambiguous qualified snapshot entries,
and merge revisions that reuse the old declaration. Create a new declaration
for changed merge semantics. ClickHouse rejects merges because it cannot
establish the required source-write boundary.

## Capture, archive, and batching

Use `rollback="capture"` on a transformation when rollback must restore the
actual typed value that existed before the generated write.

```python
from dbwarden.data import DataMeta, batch, col, derive, func

class Data(DataMeta):
    transformations = [
        derive(
            "normalized_name",
            func.lower(col("name")),
            rollback="capture",
            execution=batch(
                size=500,
                key=["id"],
                max_duration=30,
                statement_timeout=10,
                lock_timeout=5,
            ),
        )
    ]
```

Capture requires a target primary key. Apply stores the preimage before the
write in the same consistency boundary. Rollback checks that the current value
still matches the generated postimage. If application code changed it,
rollback stops rather than overwriting the later value.

`check --data` verifies the authenticated capture receipt and recorded
postimage. It does not reevaluate a captured expression or its domain on the
post-write row. That would misclassify valid one-shot expressions such as
`derive("value", col("value") + 1, rollback="capture")`.

Archive policies use `archive_table()` and an explicit archive acknowledgement.
They are available for scoped missing managed rows, unmatched transition rows,
and completed transition sources. An archive move writes or verifies the
archive row before removing the source row. The archive destination cannot be
a current managed model or overlap another incompatible declaration.

```python
from dbwarden.data import DataMeta, archive_table, col, rows

class Data(DataMeta):
    managed_rows = rows(
        key="code",
        rows=[{"code": "UY", "name": "Uruguay"}],
        owned_columns=["name"],
        on_missing="archive",
        scope=col("managed"),
        archive_to=archive_table("country_archive"),
        acknowledge_archive=True,
        rollback="restore_previous",
    )
```

For a transition, set `coverage="subset"`, `on_unmatched="archive"`, and
`unmatched_archive_to=archive_table(...)` to move unmatched rows. Set
`on_complete="archive"` and `archive_to=archive_table(...)` to retain the
completed source under the declared archive name. Either form requires
`acknowledge_archive=True`. `rollback="capture"` records overwritten target
preimages; `restore_preserved_source` restores a preserved or archived source
after verified target cleanup. Archive destinations cannot reuse an internal
name, a desired model, the source or target itself, or another declaration's
archive destination.

`batch()` uses ascending keyset ranges and never uses `OFFSET`. The key must be
the source identity for a transition or the target primary key for a
transformation, must be non-null and unique, and cannot be modified by the
statement. All chunks remain in one statement transaction and ownership
receipt. They do not commit or resume independently. A timeout or lag violation
rolls back that statement.

`max_duration` is one cooperative budget across every step and chunk in the
declaration attempt. A retry starts a new budget. The executor checks the
budget before and after SQL and between chunks; `statement_timeout` interrupts
supported statements independently. Duration, statement, lock, and lag limits
are finite positive seconds.

Backend controls differ. PostgreSQL supports statement, lock, duration, and
replication-lag limits. MariaDB supports its native statement limit; MySQL
rejects a requested DML statement timeout and supports the other admitted
controls. SQLite supports duration, statement interruption, and lock timeout,
but rejects replication-lag limits. ClickHouse rejects keyset batching.

## Historical snapshots

Use an explicit snapshot ID when branch intent matters. You may omit
`snapshot=` during authoring to request the newest compatible snapshot on one
unambiguous registered lineage. Generation then freezes the concrete snapshot
ID, database, backend, schema checksum, and parents into the bundle. Multiple
heads, multiple newest compatible candidates, a missing table, or invalid
lineage fails and asks for `snapshot=`. Execution never performs this search or
substitutes another snapshot. `latest_previous` is not an accepted alias.

For external snapshot import, use `dbwarden.data.snapshots.register_snapshot(snapshot_id, schema_state, database=..., backend=..., parent_snapshot_ids=..., registry_path=...)`. `resolve_snapshot(snapshot_id, database=..., backend=..., registry_path=..., schema_dir=...)` reads the pinned entry without changing the registry. Reusing an ID with different content fails. Existing schema snapshot files must carry their own valid checksum before they can be resolved without a registry entry.

Registry writes use a local operating-system file lock and atomic replacement, including offline imports. Parent IDs are sorted and must identify registered snapshots in the same database and backend. Resolution rejects missing parents, cycles, and duplicate IDs. Pass both branch snapshot IDs as parents when importing a merge or squash snapshot; an imported root may have no parents.

## Generate and inspect

Use the shared generator so schema and transition work become one migration when a source table changes.

```text
dbwarden make-migrations "introduce country data" --offline --database primary
dbwarden data render --database primary --format operations
dbwarden data describe --database primary --format markdown
dbwarden data docs --database primary --output docs/data --diagrams mermaid
dbwarden data transition validate SplitLegacyPeople --database primary
dbwarden data transition plan SplitLegacyPeople --database primary --format sql
dbwarden data transition dry-run SplitLegacyPeople --database primary --probes
dbwarden migrate --database primary --dry-run --data
dbwarden check --database primary --data
dbwarden diff --database primary --data
```

`data render` and `data describe` accept `--model`, `--include`, `--only-with-data`, and `--relationship-depth` (0–5). Include sections are `schema`, `constraints`, `relationships`, `data`, and `transitions`. Render formats are `text`, `json`, `mapping`, `sql`, and `operations`; describe formats are `text`, `json`, and `markdown`, with optional `--output`. Values are redacted by default. `--show-managed-values` reveals declaration literals. SQL output intentionally shows SQL literals because it is for review. `data docs` writes an index, model and transition pages, and `checksums.json`.

Render, describe, and docs can read the latest non-superseded frozen bundle when configured live inputs have been removed. This preserves the stored model schema and declarations without importing a frozen Python file. Validation and generation still require live inputs; malformed live code is not silently replaced by old artifacts.

`transition new --manual Name` writes a disabled draft beneath a configured `data_paths` directory. Fill in its source, identity, and targets before enabling it. Its direct form accepts `--from-table` or `--from-model`, `--source-snapshot`, repeated `--to-model`, `--map source:Model.target`, `--where`, source and target keys, coverage, overlap, and preserve or drop-source flags. Direct predicates accept column equality with a literal; complex predicates belong in the Python declaration. Interactive authoring requires a terminal. Existing files are not overwritten.

For direct priority authoring, pass `--coverage all_assigned_once --overlap priority` and one `--priority Model:INTEGER` per target. Values must be distinct; lower numbers win.

The transition group contains `audit`, `describe`, `new`, `render`, `validate`,
`plan`, `dry-run`, and `reconcile`. The compatibility group
`dbwarden data-transition` exposes the same commands.
For direct transition creation, use `--source-key`, `--key`, `--coverage`,
`--overlap`, `--preserve-source` or `--drop-source`, and `--acknowledge-drop`
when source removal needs acknowledgement. `dbwarden make-data-migration` and
`dbwarden make-migrations` accept repeatable `--param NAME=VALUE` generation
parameters.

Offline generation reads saved model state, declarations, and pinned snapshots. It does not need a target database connection. A historical source without a valid snapshot fails before SQL generation. `make-data-migration` uses the same generator with schema changes excluded. Use `make-migrations` when target tables or columns must change with the data.

## Complete review and execution flow

The following sequence uses the same compiled semantics from declaration review
through convergence.

1. Configure `model_paths`, `data_paths`, `data_snapshot_dir`, and
   `snapshot_registry`. Register any externally supplied historical snapshot.
2. Add the declaration beside its model or under `data_paths`. Give declarations
   explicit stable IDs before moving or renaming their Python definitions.
3. Validate and inspect the live semantics:

   ```text
   dbwarden data transition validate --database primary
   dbwarden data render --database primary --format operations
   dbwarden data transition plan SplitLegacyPeople --database primary --format json
   ```

4. Run read-only probes against the intended database. Review the database
   identity, snapshot, isolation level, observed counts, and accepted limits:

   ```text
   dbwarden data transition dry-run SplitLegacyPeople --database primary --probes
   ```

5. Generate one unified schema/data migration. Omit `--offline` when generation
   must collect live review evidence:

   ```text
   dbwarden make-migrations "split legacy people" --database primary
   ```

6. Review all three generated members together. Audit verifies their manifest
   and statically reads the frozen declaration:

   ```text
   dbwarden data transition audit primary__0012_split_legacy_people --database primary
   ```

7. Preview the pending frozen bundle selected by version and safety policy. This
   does not import the current declarations:

   ```text
   dbwarden migrate --database primary --dry-run --data
   ```

8. Apply under the normal migration lock. Use `--force` only when the reviewed
   severity requires destructive acknowledgement:

   ```text
   dbwarden migrate --database primary --force
   ```

9. Verify current state from desired declarations and applied frozen history:

   ```text
   dbwarden check --database primary --data
   dbwarden diff --database primary --data
   ```

   `check --data` succeeds only when the schema, pending migration SQL, and
   declared data converge. `--force` can acknowledge reviewed `WARN` and
   `CRITICAL` safety findings, but it cannot bypass unknown SQL classification
   or data drift.

If apply fails before any durable effect, the journal records a retryable
failure and a later ordinary migrate may retry it. If an effect may have
occurred, the state is `UNKNOWN_REQUIRES_RECONCILIATION` and ordinary migrate
stops. Inspect first:

```text
dbwarden data transition reconcile primary__0012_split_legacy_people --database primary
```

After repairing or verifying database state, record only a decision the
reported state permits:

```text
dbwarden data transition reconcile primary__0012_split_legacy_people \
  --database primary --apply --decision verified_retry
```

`verified_retry` does not ignore conflicts or infer ownership from equivalent
values. `--decision abandon` is an audited terminal decision where supported.
It does not mark the declared state converged.

Rollback uses the frozen rollback plan and refuses irreversible declarations or
changed owned values:

```text
dbwarden rollback --database primary --count 1
```

A completed data rollback records `ROLLED_BACK`. Ordinary migrate does not
silently replay it. Reapply is a separate operator decision and creates a new
linked epoch:

```text
dbwarden migrate --database primary --reapply-data
```

Baseline records history without executing the data plan. It requires an exact
target version and stores the explicit acknowledgement, reason, and skipped
checks. It does not prove current data convergence:

```text
dbwarden migrate --database primary --baseline --to-version 0012
```

## Artifacts and integrity

A data migration writes three paired files:

- `.sql` contains reviewable upgrade and rollback SQL.
- `.plan.json` contains typed operations, canonical data specification, execution steps, safety information, and integrity manifest.
- `.data.py` contains the frozen canonical declaration data.

The manifest binds ordered member names, media types, byte lengths, and SHA-256 checksums for SQL, plan, and frozen artifact. The plan records semantic identity separately from probe evidence. Canonical JSON normalizes Unicode and rejects duplicate normalized keys, non-finite values, and unsupported literal types. Static input rows are embedded; replay does not depend on the original CSV or JSON file.

The plan member's hash and length cover its canonical JSON payload with `data_bundle` omitted; they do not measure the indented file that also stores the manifest. SQL and frozen Python hashes cover their UTF-8 text. This keeps each checksum outside its own hash input.

`transition audit` reads the frozen file with the AST reader, verifies the bundle, and does not import it. A merge that supersedes a data bundle leaves its SQL unchanged and writes a `.superseded.json` sidecar bound to the complete original SQL checksum. Tampering makes the marker fail closed. File publication restores prior contents on failure and publishes the plan last.

## Apply, journal, rollback, and reconciliation

Execution verifies the complete bundle before writes and uses the existing database migration lock. Backend behavior differs:

| Backend | Execution boundary |
|---|---|
| PostgreSQL | One transaction includes data and success history. Existing referenced tables are locked against competing writes through guards and mutation. |
| SQLite | An explicit immediate transaction covers guards, writes, and success history. |
| MySQL / MariaDB | Durable per-statement checkpoints account for implicit DDL commits. Insert ownership requires transactional DML and InnoDB target/checkpoint tables. Interrupted statements require reconciliation; a transaction rollback is not treated as proof that all effects vanished. |
| ClickHouse | Versioned append-only journal records, synchronous mutations, and mutation-status checks. Managed rows and transformations require irreversible rollback policy. Historical transitions, merges, archive, capture rollback, batching, and managed-row `on_missing="delete"` are unsupported. |

Autocommit or concurrent schema steps cannot be combined with a data bundle. Unsupported backend operations fail during planning rather than emitting placeholder SQL.

Recorded native evidence includes six lifecycle flows on MySQL 8.4.11 and the
same six independently on MariaDB 11.4.4: managed revision and reapply,
two-revision archive rollback, captured derivation rollback, preserved
transition rollback, batched merge round trip, and retained-source tamper
refusal. Connection-loss, process-crash, and broader-version qualification
remain release gates. No live ClickHouse result is claimed.

`_dbwarden_data_runs` records migration ID, application epoch, bundle checksum, run ID, status, last completed operation, sanitized error, baseline flag, and timestamps. `_dbwarden_data_events` records state transitions; nontransactional backends also persist statement checkpoints. A successful checksum-matched migration returns as already applied. Changed checksums fail. A fully rolled-back transactional failure is retryable; uncertain durable effects produce `UNKNOWN_REQUIRES_RECONCILIATION` and block automatic replay.

On MySQL and MariaDB, claimed insert counts must match the driver's affected-row count. The executor commits a `WRITE_VERIFIED` checkpoint in the same DML transaction as the inserted rows. An equivalent concurrent application insert cannot become migration-owned merely by passing a value comparison. Count mismatches produce terminal `FAILED_FINAL`; a crash with only a `PENDING` ownership insert cannot be reconciled from equivalent target values alone.

Generated MySQL/MariaDB rollback steps persist intent before acquiring row locks. Pure DML steps then keep `SELECT ... FOR UPDATE` locks through identity verification, guards, mutation, and the `DONE` checkpoint commit. Target and checkpoint tables must use InnoDB with autocommit disabled. DDL runs in separate steps because it can commit implicitly.

MySQL and MariaDB rename a historical source to a reserved internal name before capturing its rows. Writes through the original name then fail while the transition runs. A failed attempt can leave that internal source in place for explicit reconciliation. Rollback restores the original name after target cleanup succeeds.

A MySQL/MariaDB merge freezes all input sources in one multi-table rename, so
no source remains writable after another source has reached the merge snapshot.
Merge staging, aggregate guards, target ownership, contribution edges, and
source retirement remain ordered journal steps.

Captured values, archive receipts, and merge contribution edges live in
reserved internal tables and are authenticated or checksum-bound to the frozen
plan. Back them up with the execution journal. Removing them can make rollback
or convergence unverifiable.

Batched statements call fencing and progress hooks for each emitted chunk. The
operation becomes complete only after every chunk, guard, edge record, and
postcondition succeeds. Chunks never create independent application epochs.

Dry-run probes read the original source name without renaming it. Apply-only DDL existence checks run during execution. Later convergence checks use the preserved source and omit temporary rename-state checks.

ClickHouse managed-row deletion is rejected because the backend does not provide the atomic ownership receipt needed to distinguish migration inserts from concurrent application inserts. Historical transitions are also rejected: its nonblocking rename cannot establish the source-write barrier needed for complete capture and safe source retirement.

Retained identity edges bind source identity, target identity, and mapped values with HMAC authentication using a per-database key. Convergence and rollback verify those edges against current target rows. Treat the key and journal tables as part of the database backup. Baseline application records `baseline=True` plus an acknowledgement, reason, and skipped check identifiers in its event. The Python executor accepts `baseline_reason`; the CLI records that baseline was explicitly requested. This does not prove that declared data already exists.

Distinct source keys that collapse to the same normalized identity are rejected. On transactional backends, that failure rolls back the transition. On a nontransactional backend, any durable effects still require reconciliation.

Rollback follows each declaration's policy. Irreversible data operations refuse rollback. A completed rollback records `ROLLED_BACK`. Ordinary migrate refuses to replay that data; `migrate --reapply-data` explicitly starts a later epoch linked to the previous one. `--force` does not authorize reapply, and `--reapply-data` does not rerun an already successful migration. The direct Python executor uses `reapply=True` for the same decision.

When the same bundle creates a target table, rollback removes owned rows before checking whether the table is empty. Any remaining application row prevents the table drop. PostgreSQL and SQLite keep that check inside the locked transaction; SQLite also rejects remaining dependent views, foreign keys, and triggers. MySQL/MariaDB first rename the target into a reserved rollback table, then check and drop it. If that check fails, the renamed table retains its rows for inspection and recovery.

Use `dbwarden data transition reconcile MIGRATION_ID` to inspect journal state. `--apply --decision abandon` records an explicit abandonment where supported. `--apply --decision verified_retry` requires backend-specific proof: transactional rollback evidence, or matching checkpoints and verified postconditions for durable statements. An ambiguous partial statement remains blocked. The command takes the migration lock before it writes.

A partially completed MySQL/MariaDB rollback can remain blocked when it has already removed or changed target rows needed by the original identity proof. Verified checkpoints alone do not bypass that proof. Inspect and repair the affected state before retrying; DBWarden does not guess the missing before-image.

Persistent-environment `dbwarden reconcile` is separate: it generates a new reconciliation bundle from the declarations actually applied on that database to the merged frozen target. It retains original migration history, checks schema and data convergence, then marks branch versions reconciled. Repeating reconciliation after convergence produces no new change.

Data guards are read-only count queries run before or after a step. They enforce maximum or minimum rows, validation predicates, coverage, and conflict policies. `transition dry-run --probes` executes those queries and records count, scope checksum, database, isolation level, observation time, thresholds, and evidence checksum. It does not predict post-write values when the required schema or rows do not yet exist.

`migrate --dry-run --data` inspects pending frozen bundles after version and severity selection. It reports unavailable probes without creating a missing SQLite database. Probe success describes the observed database at that time; apply reruns guards inside its execution boundary.

## Safety and convergence

Data operations carry INFO, WARN, or CRITICAL severity based on their declaration and rollback behavior. Destructive source handling, overwrite behavior, or irreversible work deserve CRITICAL review. A passing compile and a successful guard prove only the declared checks ran. They do not prove application semantics or data quality outside declared ownership.

`category=` on `rows`, `derive`, or `DataTransition` selects a named migration group. Groups use the existing category and CLI plugin API; they do not add severity levels. SAFE, INFO, WARN, and CRITICAL retain their ordering. Safety-scoped generation defers dependent data and schema operations together, and each generated group has its own verified bundle. See [safety-scoped migrations](correctness/safety-scoped-migrations.md) and [plugin API](plugins/developing/overview.md).

`check --data` and `diff --data` add managed-row, derived-value, validation,
transition, merge, archive, and capture convergence checks to schema drift
checks. Preserved sources allow source-to-target comparison. Dropped sources
require authenticated retained edges and matching applied journal evidence.
Merge checks validate grouped values and contribution provenance. Archive and
capture checks validate their receipts and current postimages. Missing or
changed evidence reports an unverifiable declaration instead of assuming
convergence. Reserved ownership, edge, capture, archive, merge, and journal
tables are excluded from schema drift.

Re-run `data render`, `transition plan`, and bundle audit after a declaration change. Once a migration exists, edit the live declaration and generate a new migration. Do not edit frozen data files or migration SQL.

## Compared with legacy seeds

Seed files are useful for optional initial content and ad hoc loading. Declarative data migrations are versioned migration operations with canonical identity, ownership, rollback policy, guards, a durable journal, and frozen source artifacts. Use them when data change belongs to the same release contract as schema change.

The [design specification](design/declarative-data-migrations.md) defines the artifact and execution contract. The generated [Python API reference](reference/python-api.md) and [CLI option reference](reference/cli-options.md) list signatures and command options.

========================================================================
PAGE: https://docs.dbwarden.org/design/declarative-data-migrations/
========================================================================

# Declarative data migrations

Status: normative contract; implementation and evidence status recorded below

This document is the corrected normative design for DBWarden declarative data migrations. It replaces ambiguous or conflicting requirements in the 2026-08-19 proposal. `MUST`, `MUST NOT`, `SHOULD`, and `MAY` have their usual standards meaning.

## Product boundary

DBWarden compiles reviewed data declarations into immutable migration artifacts. Users declare desired managed rows, deterministic transformations, and one-shot historical transitions. They do not provide arbitrary migration callbacks, sessions, network calls, or Python execution hooks.

The current SQLAlchemy model remains the only current schema definition. A data declaration may refer to current columns, but it does not redefine table shape. Historical structure comes from one explicitly pinned schema snapshot.

The production contract includes:

- inline and static-file managed rows;
- deterministic transformations and backfills;
- one-source, one-target historical transitions;
- one-source, multi-target splits;
- deterministic many-source, one-target merges with explicit grouping and field rules;
- captured transformation preimages and declared archive destinations;
- keyset-batched data statements with explicit duration, statement, lock, and replication-lag limits;
- unified schema and data dependency planning;
- immutable SQL, plan, and frozen audit artifacts;
- source and target identity analysis;
- explicit ownership, coverage, overlap, conflict, completion, and rollback policy;
- operation journals, reconciliation, rollback epochs, and convergence checks;
- PostgreSQL production execution;
- MySQL and MariaDB production execution after their non-transactional journal gate passes;
- SQLite development, test, and sandbox parity;
- the limited ClickHouse subset defined below.

The following remain unsupported and MUST fail validation:

- arbitrary merge reducers, implicit winners, and merges without complete source provenance;
- independently committed or resumable batch chunks;
- generic source reconstruction and inverse-mapping rollback without a preserved or captured source;
- plain conflict `ignore`;
- cross-database transitions;
- execution-scoped expressions;
- unregistered user-defined functions;
- execution above explicitly declared row limits.

Compiler preview for an unsupported backend is allowed. It MUST be labelled preview, MUST NOT emit an applicable artifact set, and MUST NOT count as backend conformance.

## Conformance status

The checked-in implementation covers the original safety core: declarations,
canonical identity, exact snapshot lineage, manifest-bound artifacts, ownership,
transactional and checkpointed execution, reconciliation, rollback epochs, and
convergence. Deterministic merge, archive, capture, and keyset batching are part
of this contract and require the same artifact, ownership, recovery, rollback,
and convergence rules as the original primitives.

Release evidence is backend-specific. SQLite has real local execution evidence.
PostgreSQL has real local execution and concurrency evidence for the implemented
operation set. MySQL 8.4.11 and MariaDB 11.4.4 each passed six native lifecycle
flows covering managed revisions, archive, capture, preserved transitions,
batched merge, rollback, reapply, and retained-source tamper refusal.
Connection-loss, process-crash, and broader-version qualification remain open.
ClickHouse has native lowering and an instrumented append-version journal, but
no live-server conformance. These limits are release gates; generated SQL and
adapter tests do not satisfy them.

## Core declarations

### Managed rows

A model may declare a finite set of owned rows. A managed-row declaration contains:

- a stable `declaration_id`;
- target table;
- one or more non-null key columns;
- canonical inline rows or a frozen static source;
- owned columns;
- an explicit row scope predicate;
- `on_missing`, initially `keep` or `delete`;
- adoption and conflict policy;
- destructive acknowledgement when deletion or overwrite is possible;
- rollback policy.

If no key is declared, DBWarden MAY use the model primary key. An explicitly selected natural or composite key MUST be backed by a unique constraint or an equivalent backend proof.

`on_missing="delete"` applies only to rows already claimed by the same declaration and still inside its explicit scope. It never means "delete every unlisted row in the table." The runtime ownership ledger distinguishes a row created by DBWarden, adopted as equivalent, or overwritten with acknowledgement.

An existing target row is accepted only when all owned values are equivalent or the declaration explicitly acknowledges overwrite. Columns outside `owned_columns` are never changed.

### Static row sources

At generation, DBWarden resolves a static source inside the project root, rejects symlink escape, parses and type-checks it, canonicalizes its rows, and freezes the canonical content into the artifact set. The original relative path is provenance only. Apply never reads the original project file.

The source format contract specifies encoding, newline normalization, CSV dialect when applicable, null syntax, number parsing, duplicate-key rejection, deterministic row order, and maximum size. Static sources containing secrets are unsupported.

### Transformations

A transformation declares a target column or columns, a deterministic expression, a complete domain, unmatched-row behavior, maximum affected rows, ownership, and rollback policy.

Expression-language version 1 permits:

- column references and literals;
- bound parameters frozen at generation;
- versioned arithmetic and explicit casts;
- lower, upper, trim, and concatenation under pinned string semantics;
- `coalesce` and `case`;
- complete static mappings;
- scalar JSON extraction through `json_extract(value, literal_path)`;
- one-based deterministic string components through
  `split_part(value, literal_delimiter, literal_index)`;
- reviewed deterministic backend functions with pinned settings.

`json_extract` requires a literal path beginning with `$`. Strings are decoded,
numbers become text, booleans become `true` or `false`, and JSON null, a missing
path, an object, or an array becomes SQL null. PostgreSQL, MySQL, MariaDB, and
SQLite implement those semantics. ClickHouse rejects `json_extract` because
its available lowering cannot preserve the missing-versus-empty distinction.

`split_part` requires a nonempty literal delimiter and an integer index from 1
through 64. SQL null remains null and an absent component returns an empty
string. All five backends lower it. Both functions are additive version-1 AST
nodes; existing version-1 nodes retain their meaning.

It rejects filesystem and environment reads, network calls, random values, sequence reads, current time, arbitrary Python callables, mutable globals, unordered windows, and unknown functions.

A partial expression requires an explicit predicate and unmatched policy. A generated reconciliation statement MUST be idempotent, but idempotency does not authorize rerunning a successful migration.

The canonical expression is a backend-independent AST with inferred types, referenced columns, functions, bound parameters, determinism class, and semantic versions. Backend SQL appears only after lowering.

Version 1 preserves AST operand order. It does not algebraically reassociate operations. A later semantic version may add a rewrite only with an explicit proof covering nulls, types, collation, overflow, timezone, and backend coercion.

### Historical and current-model transitions

A transition has one source and one or more current targets. A
`historical_table(...)` source is excluded from final desired-model discovery
and may be preserved, archived, or dropped. A mapped current-model source
remains in desired schema and requires `source_snapshot=<id>`,
`on_complete="keep"`, and `on_unmatched="error"`. Its snapshot freezes the
source columns used by the one-shot mapping, while completion leaves the live
source table in place. Any other use of `keep`, or any attempt to retire a
mapped desired source, fails validation.

PostgreSQL and SQLite read the kept source under their normal transactional
lock boundary. MySQL and MariaDB atomically rename it to a reserved frozen name,
complete and verify target writes, then rename it back. This is a temporary
write barrier, not source retirement. Rollback removes verified owned target
effects and leaves the current source in place. ClickHouse rejects transitions.

A transition declares:

- stable declaration identity;
- exact source snapshot lineage;
- source identity columns;
- one or more targets with target identity, mapping, predicate, and conflict policy;
- relationship cardinality;
- coverage and overlap policy;
- source completion policy;
- rollback policy;
- execution limits.

Single-source cardinalities are `one_to_one` and `one_to_many_split`.
`many_to_one` is admitted only through the deterministic `merge_sources()`
contract below. Unstructured many-to-one mappings remain rejected.

Source identity is backed by a schema uniqueness constraint or an apply-time duplicate proof in the same consistency boundary as the write. Target identity is non-null and stable. It MUST have a database uniqueness constraint unless the backend proves an equivalent serialized conditional write.

One source identity may fan out to several declared target types. It may not create the same target identity twice inside one target.

### Deterministic merges

A merge combines two or more explicitly pinned historical sources into exactly
one current target. Every input declares its own source identity, projection,
and distinct priority. The merge declares non-null grouping keys and one rule
for every non-key output field.

The supported field rules are:

| Rule | Contract |
|---|---|
| `sum` | Sum non-null numeric contributions using the lowered backend numeric type. |
| `avg` | Average non-null numeric contributions using the lowered backend numeric type. |
| `min` / `max` | Select the ordered minimum or maximum under the plan's pinned backend semantics. |
| `count` | Count non-null expression results. |
| `require_equal` | Require every contribution, including null placement, to be equal. String equality uses UTF-8 bytes rather than the database's default collation. |
| `winner` | Select the lowest-priority source, then order by its source identity encoded as UTF-8 bytes. |

Arbitrary reducers, implicit input order, and implicit last-write wins are
invalid. Every input must project the same named fields with compatible types.
Grouping keys must exist in every projection. Source identities and grouping
keys are non-null integer or string values, and source identities are unique at
apply. Other types require an explicit cast into one of those stable families.
String grouping uses UTF-8 byte equality. The plan records
`priority_then_identity_utf8_hex_v1` as its ordering rule.

The combined input must contain at least one row by default. A declaration may
set `allow_empty=True` to acknowledge that an empty result is valid and that
source completion may still proceed. The setting is part of merge identity.

The plan retains source-to-group and group-to-target provenance. Runtime edge
records authenticate each contribution without logging raw identities. A merge
uses the same target conflict and ownership rules as a transition. A revision
that changes grouping, sources, or field rules requires a new declaration with
explicitly pinned retained sources.

MySQL and MariaDB MUST freeze every merge source under one metadata-lock
boundary before reading any source. Freezing sources one at a time is invalid
because writes between renames would create an inconsistent merge cut.

### Capture and archive

`capture` transformation rollback records the previous owned value before the
write. The capture store distinguishes a missing row, null, and a typed value,
and binds the record to the migration, epoch, declaration, target identity,
owned column, preimage, and generated postimage. Capture and mutation share the
same consistency boundary. Rollback restores a capture only while the current
owned value still matches the recorded postimage.

Archive is an explicit move to a named archive table. It is valid for missing
managed rows, unmatched transition rows, or a completed historical source only
when the declaration specifies the archive destination and acknowledgement.
The archive plan records source and archive identities, mapped columns,
ownership, conflict behavior, and rollback behavior. It writes or verifies the
archive record before removing the source record. A backend that cannot make
that sequence atomic MUST use durable intent, verification, and reconciliation.

Archive never means unscoped deletion. An archive destination is reserved by
the declaration and must not overlap a current desired model or another
declaration with incompatible ownership.

### Keyset batching

`batch(...)` may split a generated data statement into ordered keyset chunks.
Its key is a non-null unique identity and may not be changed by the statement.
The compiler proves that key against the source identity or target primary or
unique key. `OFFSET` batching is invalid.

Supported controls are chunk size, maximum declaration-attempt duration,
statement timeout, lock timeout, and maximum observed replication lag.
`max_duration` is one cooperative budget shared by every step and chunk in the
declaration attempt; a retry starts a new budget. The executor checks it before
and after SQL and between chunks. `statement_timeout` provides interruption on
backends that support it. All four limit values are finite positive seconds.
Unsupported controls fail during backend negotiation.

All chunks for one generated statement remain in the statement's existing
transaction and ownership receipt. They do not commit independently and cannot
be resumed independently. A failure rolls back the complete statement on a
backend admitted for batching. The normal operation checkpoint records the
statement effect. Global coverage, postconditions, edge recording, and source
retirement occur only after all chunks succeed.

## Coverage and conflicts

Initial coverage modes are:

| Mode | Meaning |
|---|---|
| `all` | Every source identity is assigned to at least one declared target edge. |
| `subset` | Unmatched source identities are allowed and remain untouched. |
| `exactly_once_match` | Every source identity matches exactly one raw target predicate. |
| `all_assigned_once` | Every source identity is assigned once after explicit target priority rewrites overlapping predicates. |

`on_unmatched="ignore"` or `on_unmatched="archive"` is valid only with `subset`. `all` and both exactly-once modes use `error`.

Overlap policies are:

- `error`, which rejects any raw overlap;
- `fan_out`, which permits only the explicitly listed cross-target edges;
- `priority`, which requires unique explicit priorities and is valid only with `all_assigned_once`.

The plan records raw predicates and effective predicates after priority rewriting.

Initial conflict policies are:

| Policy | Meaning |
|---|---|
| `error` | Abort before mutation when a target identity already exists, except when a verified same-migration edge proves recovery state. |
| `ignore_if_equivalent` | Accept an existing identity only when every owned value equals the declared result. |
| `overwrite` | Replace owned values after explicit CRITICAL acknowledgement and ownership validation. |

Plain `ignore` is invalid. Retry machinery MUST NOT implement `error` as ignore.

## Ownership

Every mutating declaration contains:

- owner declaration ID and revision;
- target table;
- key columns and canonical key domain;
- row scope predicate;
- owned columns;
- adoption policy;
- acknowledgement state;
- runtime ownership or identity-edge ledger reference.

A transformation owns its target columns only inside its domain. A transition owns only the target row identities and columns recorded by its verified identity edges. No declaration owns a table merely because it writes to it.

The planner compares live declarations for overlapping row and column ownership. A possible overlap fails unless a deterministic ownership order is explicit in semantics. Runtime conflicts outside declared ownership fail closed.

## Canonical IR

The compiler has these phases:

```text
discovery
validation
resolution
canonicalization
identity analysis
coverage and conflict analysis
dependency planning
safety classification
backend capability negotiation
backend lowering
artifact generation
execution
convergence
```

Each phase has a typed input and output. A phase does not reinterpret pre-canonical Python objects. Backend SQL does not exist before lowering. Execution consumes verified artifacts and recorded state; it does not rediscover live declarations.

Canonical values use NFC strings, lexicographically sorted object keys, explicit nulls, and type-preserving encodings for dates, timestamps, decimals, and finite floats. Semantic arrays define their own order. Environment-specific absolute paths are forbidden.

`canonical_bytes(value)` returns the UTF-8 strict canonical JSON encoding. `digest(value, domain)` hashes a typed array containing `dbwarden`, the domain, semantic format version, and value. Every identity domain is distinct.

### Identity derivation

Derived fields are excluded from their own preimages. Derivation order is:

1. Canonicalize declaration semantics without derived IDs, checksums, probe evidence, timestamps, or runtime state.
2. Compute the declaration semantic checksum.
3. Compute transition ID from database identity, declaration ID, and declaration semantic checksum.
4. Compute each data operation ID from its kind and canonical declaration payload. Dependency IDs are excluded. Derive execution-step IDs from the parent operation ID, direction, and local step index. The journal scopes those IDs by migration and epoch.
5. Resolve dependency references to operation IDs.
6. Compute the complete plan and artifact manifest checksums.

String concatenation is not a hash encoding. All tuples and arrays use canonical typed encoding.

The identity model distinguishes:

- declaration ID;
- declaration semantic checksum;
- transition ID;
- migration ID;
- operation ID;
- static edge definition ID;
- runtime edge instance hash;
- run ID and application epoch.

An edge definition ID hashes transition ID, source key definitions, target reference, target key definitions, mapping, and predicate. It contains no row value.

An edge instance hash uses HMAC-SHA-256 over versioned typed source-key and target-key tuples. The checkpoint stores its `key_id` and encoding version. Reconciliation keys remain available for the journal retention lifetime.

## Snapshot lineage

Generation resolves a historical source to one exact snapshot. The frozen lineage contains:

- database identity;
- globally unique snapshot ID within that database;
- schema checksum;
- optional data-declaration checksum;
- creating migration ID;
- canonical parent snapshot IDs;
- snapshot format version.

Branch label, creation time, storage URI, and import provenance are registry metadata. They do not affect transition semantic identity.

Authoring may omit `snapshot=` to request the latest compatible snapshot on one
unambiguous registry lineage. Compilation walks the single lineage head,
selects the newest snapshot containing the source table with the requested
schema, and freezes that concrete snapshot ID and lineage. Multiple heads,
multiple newest compatible candidates, a missing source, or malformed lineage
fails and requires an explicit snapshot ID. `latest_previous` is not a magic
snapshot name. Execution never searches again or substitutes another snapshot.
A missing or corrupt bundle member is an integrity failure.

The registry rejects duplicate IDs, unknown or cyclic parents, invalid checksums, database or backend mismatch, and ambiguous matches. An operating-system file lock serializes registry updates before atomic publication; migration-driven registration also runs under the database migration lock. Offline imports need no database connection. Read-only resolution does not create directories or files.

The implementation API is:

```text
resolve_snapshot(
    snapshot_id,
    database,
    backend,
    registry_path=None,
    schema_dir=None,
)
```

It returns the frozen lineage plus `schema_state` after verifying the existing core schema-snapshot checksum.

## Planning and ordering

Schema and data operations form one dependency DAG. It includes target creation, intermediate column creation, transfers, transformations, validation, constraint enforcement, source preservation, and destructive work.

For a non-null derived column on an existing table, the planner either emits:

1. a safe nullable or equivalent intermediate column;
2. backfill;
3. validation;
4. constraint enforcement;

or fails and requests an explicit staged release.

A destructive operation depends on all reads from the object, all transfers, coverage validation, convergence checks required by policy, and any preservation operation. Data transfer never follows dependent drop or rename work.

The planner topologically sorts the DAG before lowering. `operation_id` is the deterministic tie-breaker. Lowering and execution preserve this order across transaction boundaries. They MUST NOT regroup all transactional statements before all autocommit statements.

A cycle error includes the concrete operation path. It may recommend a multi-migration release only when a valid intermediate state can break the cycle. If one transition spans several files, a release record links their migration IDs and states.

## Review evidence and safety

The plan has separate `semantics` and `review_evidence` sections.

Semantic identity and SQL depend only on `semantics`. Live probes may populate evidence, thresholds, severity, and guards, but they never change mappings, expressions, policies, identity, or SQL shape.

Evidence records probe ID, canonical query scope, database identity, isolation level, observation time, values, accepted thresholds, and checksum. The final artifact manifest covers the evidence reviewed by the operator.

Generation-time evidence is advisory at apply. Apply reruns required probes. Transactional plans run probes and writes in one documented isolation boundary, with constraints or locks where needed. Non-transactional plans use conditional writes, durable intent, per-operation verification, and final convergence.

Destructive behavior requires both declaration-specific acknowledgement and the normal operator acknowledgement such as `--force`. A stale acknowledgement whose semantic checksum no longer matches is invalid.

## Backend capability negotiation

Lowering evaluates the complete plan for:

- atomic DML and metadata commit;
- transactional DDL;
- ordered mixed transactional and autocommit execution;
- conditional write and equivalence support;
- durable operation checkpoints;
- synchronous effect visibility;
- source preservation support;
- convergence-query support.

One autocommit, external, or asynchronous operation makes the affected execution segment non-transactional. Backend name alone never proves atomicity.

### Initial capability matrix

| Capability | PostgreSQL | MySQL and MariaDB | SQLite | ClickHouse |
|---|---|---|---|---|
| Support level | Production | Backend-observed lifecycle subset; crash and version gates open | Development, CI, sandbox | Implemented limited subset; live conformance pending |
| Managed rows | Full initial policies | Full initial policies with journal | Full initial policies | Insert and synchronous update mutations; irreversible rollback only; scoped deletion rejected |
| Transformations | Full initial expressions | Full initial expressions with journal as required | Supported expression subset | Synchronous `ALTER ... UPDATE` mutations; irreversible rollback only |
| One-to-one transition | Full | Full with edge checkpoints | Full | Rejected; source-write barrier unavailable |
| Split | Full | Full with per-target checkpoints | Full | Rejected; source-write barrier unavailable |
| Deterministic merge | Full after source locks | Full only with atomic multi-source freeze and journal | Full | Rejected; source-write barrier unavailable |
| Captured transformation rollback | Full | Full with transactional capture/write receipt | Full | Rejected |
| Archive | Full when archive/write/delete share the transaction | Journaled archive receipt before source removal | Full | Rejected |
| Keyset batching | Transactional chunks, PostgreSQL limits | Transactional DML chunks; backend-specific limits | Transactional chunks; no replication-lag control | Rejected |
| `ignore_if_equivalent` | Yes | Yes | Yes | Historical transitions unsupported |
| `overwrite` | Acknowledged and owned values only | Same, journaled | Same | No |
| Source drop | After same-boundary validation or verified journal state | After source freeze and verified journal state | Yes | Historical transitions unsupported |
| Retry after uncertainty | Reconcile first | Reconcile first | Reconcile first | Reconcile first, never blind retry |

MySQL and MariaDB require separate real-backend evidence. MySQL 8.4.11 and
MariaDB 11.4.4 each passed the six lifecycle flows described above; neither
backend inherits the other's result. Their crash and broader-version gates
remain open. ClickHouse uses `mutations_sync = 2`, probes `system.mutations`
before a checkpoint becomes `DONE`, and stores append-only run and checkpoint
versions in `ReplacingMergeTree` tables. Unit tests cover generated native SQL
and an instrumented ClickHouse connection; no live ClickHouse conformance run
has been completed. Asynchronous mutations, historical transitions,
deterministic merges, archive, capture rollback, batching, overwrite, and
scoped managed-row deletion remain outside the ClickHouse subset.

MySQL and MariaDB rename a transition source into the reserved migration namespace before capturing rows. This waits for existing table users and prevents later writes through the original name. Rollback restores the source name only after owned target cleanup. ClickHouse's Atomic engine performs rename without waiting for existing queries, so rename alone cannot establish that source boundary. Historical transitions fail planning until a supported write barrier exists. See the backend documentation for [MySQL rename and metadata locking](https://dev.mysql.com/doc/refman/8.4/en/rename-table.html) and [ClickHouse Atomic rename](https://clickhouse.com/docs/engines/database-engines/atomic#rename-table).

## Artifact set

Every data migration has:

```text
<migration>.sql
<migration>.plan.json
<migration>.data.py
```

Inline CSV and JSON inputs are normalized into the frozen `DATA_SPEC`; they do not add copied source files beside the bundle. Any separately retained source artifact is an ordered manifest member with its media type, byte length, and checksum.

The SQL file is executable output. The plan is the machine-readable semantic and execution contract. The frozen Python file is human-readable audit data and is never an execution input.

The artifact manifest contains format version, migration ID, semantic checksum, and ordered members. Every member has normalized relative name, media type, byte length, and SHA-256. The manifest checksum is the domain-separated digest of the canonical manifest without its checksum field. The plan's embedded `data_bundle` field is excluded from the plan component preimage to avoid self-reference.

Generation writes members to staging and publishes the valid manifest last. Discovery ignores incomplete sets. Verification occurs before plan, baseline, apply, retry, reconcile, audit, rollback, and convergence. Applied history stores the manifest checksum.

The implementation APIs are:

```python
build_manifest(sql_content, plan, frozen_content)
verify_bundle(sql_path, plan=None)
```

Verification requires all three core members, checks every component, statically reads the frozen artifact, verifies its guard migration ID, and compares canonical `DATA_SPEC` with `plan["data_spec"]`.

## Frozen audit artifacts

The generated audit file imports and immediately calls `guard_frozen_data_artifact(migration_id)`, then assigns one literal `DATA_SPEC`. It contains no callbacks or executable migration logic.

Normal import raises `FrozenDataArtifactImportError`. Intentional Python audit tooling may use the context-local `allow_frozen_data_imports()` context manager. The process-wide environment override is disabled unless its value is exactly `1`. Every allowed import emits an audit warning.

Normal audit uses `read_frozen(path)`, which parses the AST, requires the guard, permits only the generated import, guard call, and literal assignment, and never imports the file.

## Public implementation API

The supported authoring surface is exported from `dbwarden.data`:

```text
rows(*, key=None, rows=None, source=None, owned_columns=None,
     on_missing="keep", scope=None, acknowledge_delete=False,
     archive_to=None, acknowledge_archive=False,
     rollback="irreversible", category=None)

derive(target, expression, *, rollback, when=None,
       on_unmatched="error", max_rows=None, category=None,
       settings=None, execution=None)

historical_table(table, *, snapshot=None, schema=None)
archive_table(table, *, schema=None)
into(model, *, map, key=None, where=None,
     on_conflict="error", acknowledge_overwrite=False, priority=None)
validate(expression, *, message="Data validation failed")

from_source(source, *, identity, map, priority)
aggregate(policy, expression)
winner(expression)
merge_sources(*inputs, key, values, allow_empty=False)

batch(*, size, key, max_duration=None, statement_timeout=None,
      lock_timeout=None, max_replication_lag=None)
```

`DataMeta` owns `managed_rows`, `transformations`, and `validations`.
`DataTransition` owns the source, source identity, targets, coverage, unmatched,
overlap, completion, rollback, maximum-row, category, and execution policies.
A merged source is assigned to `DataTransition.source`; its grouping key becomes
the default transition source identity.

The stable artifact and execution helpers are:

```text
resolve_snapshot(snapshot_id, *, database, backend,
                 registry_path=None, schema_dir=None)
register_snapshot(snapshot_id, schema_state=None, *, database=None,
                  backend=None, parent_snapshot_ids=(), registry_path=None,
                  data_checksum=None, created_by_migration=None)
build_manifest(sql_content, plan, frozen_content)
verify_bundle(sql_path, plan=None)
render_frozen(data_spec, migration_id)
read_frozen(path)

execute_data_plan(connection, plan, *, migration_id,
                  direction="upgrade", record_success=None,
                  baseline=False, baseline_reason=None, reapply=False,
                  before_statement=None, after_statement=None)
reconcile_data_plan(connection, migration_id, *, apply=False,
                    decision=None, plan=None)
```

`execute_data_plan` assumes the caller holds the shared migration lock. Normal
application uses `execute_migration_bundle`, which verifies the frozen bundle,
connects schema history recording, and forwards fencing and progress callbacks.
Direct callers must preserve those responsibilities.

## End-to-end lifecycle

1. Discovery imports trusted live project declarations from `model_paths` and
   `data_paths`. Frozen migration files are excluded.
2. Validation rejects ambiguous identities, unsupported policies, overlapping
   ownership, incomplete expressions, unsafe archive/capture rules, and invalid
   batch or merge declarations.
3. Resolution loads each exact snapshot and static row source. Static rows are
   normalized and embedded. Snapshot lineage is frozen.
4. Canonicalization creates the versioned data specification and content-based
   declaration checksum. Identity analysis derives declaration, transition,
   operation, step, and edge-definition identities without self-reference.
5. Planning combines schema and data operations in one dependency DAG. Backend
   negotiation rejects any operation whose transaction, lock, receipt,
   preservation, or convergence requirements cannot be met.
6. Lowering emits ordered upgrade and rollback steps, guards, lock tables,
   ownership proofs, edge metadata, batch descriptors, and recovery behavior.
7. Generation stages SQL and frozen audit data, computes the manifest, and
   publishes the plan last. A dry run stops before publication and mutation.
8. Apply acquires the normal migration lock, verifies the bundle and backend
   settings, creates or reads journal state, and reruns required guards.
9. Transactional backends execute guards, writes, edge recording, and schema
   history in one boundary. Nontransactional segments persist intent before a
   possible effect and verify each durable effect before proceeding.
10. Success commits schema history and `APPLIED_SUCCESS` together where the
    backend permits it. A checksum-matched repeat performs no migration writes.
11. A known rolled-back failure becomes retryable. A possible durable effect
    becomes `UNKNOWN_REQUIRES_RECONCILIATION` and blocks apply.
12. Reconciliation verifies checkpoints, ownership receipts, authenticated
    edges, and current postconditions. Only explicit `verified_retry` or
    `abandon` decisions allowed by the current state can change it.
13. Rollback verifies current ownership and postimages before reversing effects.
    A successful rollback records `ROLLED_BACK`; later application requires an
    explicit linked reapply epoch.
14. `check --data` and `diff --data` compare current desired declarations with
    actual owned values, retained sources, archives, captures, merge provenance,
    and applied frozen history.

## Execution journal

Execution state uses three logical stores:

1. An immutable artifact record keyed by migration ID and manifest checksum.
2. An append-only event journal identifying each run and application epoch, with a current-state projection for inspection.
3. Durable operation checkpoints and authenticated identity-edge instance records. SQL backends update checkpoint state in place; ClickHouse appends checkpoint versions.

A current-state projection may exist for status queries. It does not replace history.

Migration states are:

```text
NOT_STARTED
APPLYING
FAILED_RETRYABLE
FAILED_FINAL
UNKNOWN_REQUIRES_RECONCILIATION
APPLIED_SUCCESS
ABANDONED
ROLLING_BACK
ROLLED_BACK
```

Normal transitions are:

```text
NOT_STARTED -> APPLYING
APPLYING -> APPLIED_SUCCESS
APPLYING -> FAILED_RETRYABLE
APPLYING -> FAILED_FINAL
APPLYING -> UNKNOWN_REQUIRES_RECONCILIATION
UNKNOWN_REQUIRES_RECONCILIATION -> FAILED_RETRYABLE
UNKNOWN_REQUIRES_RECONCILIATION -> FAILED_FINAL
FAILED_RETRYABLE -> APPLYING
APPLIED_SUCCESS -> ROLLING_BACK -> ROLLED_BACK
```

Baseline records an `APPLIED_SUCCESS` run with mode `baseline`. It first proves convergence or stores explicit operator acknowledgement, reason, and skipped or failed checks.

`APPLIED_SUCCESS`, `ROLLED_BACK`, `FAILED_FINAL`, and `ABANDONED` are terminal inside one application epoch. Reapply after rollback creates a new explicit epoch and references the prior epoch. Normal discovery never reruns a migration merely because it was rolled back.

Every run records migration ID, manifest checksum, epoch, run ID, attempt number, state transitions, timestamps, current operation, completed operations, failure, reconciliation evidence, and operator decisions.

### Non-transactional protocol

Before a non-transactional statement, DBWarden commits a `PENDING` checkpoint. It applies the effect, verifies it, and commits `DONE`. Each frozen execution step has a distinct identity within its operation and direction, so statement indexes cannot collide across transition targets. A crash after the possible effect and before verification produces unknown state.

MySQL and MariaDB ownership inserts require InnoDB tables and disabled autocommit. The affected-row count must match the frozen plan's expected count. The inserted rows and a `WRITE_VERIFIED` checkpoint commit in the same DML transaction before later value checks mark the statement `DONE`. Count mismatches are terminal failures. A `PENDING` ownership insert cannot be recovered from equivalent target values alone, because those values may belong to an application writer.

Generated rollback steps containing only DML first persist their intent checkpoints, then lock the target rows with `SELECT ... FOR UPDATE`. Identity verification, guards, writes, afterguards, and `DONE` checkpoint updates share that transaction. DDL occupies separate steps. Preserved-source restoration runs only after target cleanup succeeds.

ClickHouse does not claim insertion ownership. Managed-row scoped deletion is unsupported without an atomic insertion-ownership receipt, and historical transitions are rejected without a supported source-write barrier.

Minimum checkpoint fields are migration ID, manifest checksum, epoch, run ID, operation ID, transition ID where applicable, edge definition ID, source and target edge instance hashes where applicable, state, record checksum, key ID, and encoding version.

Runtime journal rows cannot be covered by a generation-time migration checksum. The manifest covers journal schema and protocol versions. Each runtime record has its own checksum or authenticated hash and artifact-manifest reference.

Automatic retry is allowed only when the whole attempted segment rolled back atomically or reconciliation proves every completed effect and the next action. A stale `APPLYING` record whose lock is gone becomes unknown.

## Rollback

Rollback is a state-changing run with its own journal. It is conditional on current ownership and state.

For managed rows:

- delete only a row created by the migration and unchanged from its recorded postimage;
- restore a value only from a previous DBWarden desired state or an explicitly captured preimage;
- do not delete an adopted row;
- stop for reconciliation when current owned values no longer match the recorded postimage.

For transformations, `clear` is valid only for an owned nullable target and only while the current value matches the generated postimage. `recompute` restores the previous frozen desired expression from retained inputs after checking the current expression's postcondition. It requires a previous declaration with compatible ownership and domain; it does not reconstruct arbitrary pre-migration values. `capture` restores the authenticated typed preimage recorded in the same consistency boundary as the original write, again only while the current value matches the recorded postimage. Generic inverse reconstruction remains unsupported.

For transitions, `restore_preserved_source` restores the recorded fully qualified preserved source name and reverses only ledger-owned target effects. It does not generically drop target tables. A target table may be dropped only when the same migration created it and verification proves that it has no unowned rows or dependent objects.

Preserved names include a collision-resistant transition or migration suffix and are recorded in the plan. Registered preserved internal tables are excluded from desired-schema diff but remain visible to audit and convergence.

Archive rollback first verifies the archive receipt and the absence of a
conflicting live row. It restores only the row and columns owned by the archived
declaration, then removes only its own unchanged archive record. An archive
record changed or adopted by application code stops rollback for reconciliation.

An irreversible migration causes rollback to fail clearly. A comment or empty SQL section is not successful reversal.

## Convergence

Convergence compares current canonical desired state with actual state. It does not regenerate or replay old migration code.

Checks cover:

- managed-row keys and owned values inside scope;
- transformation violations inside domain;
- expected transition target identities;
- coverage of retained sources;
- duplicate target identities;
- conflict-equivalence claims;
- merge group values, deterministic winner evidence, and contribution edges;
- archive receipts, archive values, and source absence required by policy;
- capture preimage/postimage records needed for reversible history;
- foreign keys and constraints;
- preserved-source and ownership-ledger consistency.

`check --data` reports each declaration separately and redacts row values by default.

A captured transformation is a frozen one-shot write. Convergence verifies its
authenticated ledger and recorded postimage against the current target. It
does not reevaluate the expression or domain against post-write rows, because
self-references and target-dependent domains can change meaning after the
write.

Sandbox execution requires an explicit fixture, sanitized clone, or separately approved data snapshot for historical source rows. With only a schema snapshot, sandbox proves SQL syntax and empty-state schema convergence. It does not prove production coverage, conflicts, row limits, or performance.

## CLI contract

The initial safety-critical commands are:

```text
dbwarden data-transition validate
dbwarden data-transition plan
dbwarden data-transition dry-run
dbwarden data-transition render
dbwarden data-transition audit
dbwarden make-data-migration
dbwarden make-migrations
dbwarden migrate --dry-run
dbwarden migrate
dbwarden data-transition reconcile
dbwarden diff --data
dbwarden check --data
```

Dry run resolves and reports one exact snapshot, validates the canonical IR, runs allowed read-only probes, negotiates backend capabilities, renders the operation DAG, and mutates no file, metadata, or database row.

Non-interactive execution never prompts. Ambiguous mappings, snapshots, policies, and acknowledgements fail. JSON output uses the same versioned plan contract as human output.

Interactive scaffolding, model descriptions, and generated Markdown are useful product features but are not prerequisites for execution safety. Generated documentation is never a discovery source.

## Security model

Live declarations are trusted project Python code. Import-time validation occurs after Python import side effects and is not a sandbox. The trust boundary includes code review, the process, credentials, and database permissions.

The compiler pipeline is a correctness and mutation-safety boundary. It rejects unsupported expressions, ambiguous ownership, and unsafe operations. Frozen artifacts use static parsing by default. Plans and logs redact raw row values unless local debugging explicitly requests them.

## Required verification

The production contract is incomplete until it passes:

- canonical hash golden vectors and self-preimage exclusion tests;
- identity domain, typed-key, HMAC key-rotation, and collision cases;
- branch, squash, import, corrupt, missing, and ambiguous snapshot cases;
- ownership overlap, adoption, scoped deletion, and later-write rollback cases;
- coverage and overlap truth tables;
- merge grouping, aggregate null/type behavior, winner ordering, and ambiguous-contribution cases;
- archive write-before-delete, adoption, conflict, rollback, and interrupted-move cases;
- capture missing/null/value, postimage conflict, tamper, and rollback cases;
- keyset uniqueness, null, non-advancing key, timeout, lag, mid-chunk failure, and full-statement rollback cases;
- concurrent conflict and probe-to-write race tests;
- unified DAG and mixed transaction/autocommit order tests;
- artifact member tamper and publication-crash tests;
- failure injection before effect, after effect, before verification, and before success metadata;
- reconciliation tests that keep `error` conflict semantics;
- apply, rollback, ordinary migrate, and explicit reapply epoch tests;
- real PostgreSQL, MySQL, MariaDB, SQLite, and limited ClickHouse conformance runs;
- convergence after applying history to an empty database with suitable source fixtures;
- compatibility tests proving existing schema migrations and seed behavior are unchanged.

Tests assert that arbitrary merge reducers, implicit merge winners, independently committed batch chunks, cross-database operations, execution-scoped expressions, and plain-ignore declarations fail with stable unsupported-capability diagnostics. Backend-specific unsupported combinations, including all four advanced operations on ClickHouse, fail before an applicable artifact is emitted.

Repeated apply after success executes zero mutation statements. Journal and integrity inspection reads remain necessary. It is not tested by allowing idempotent mutation SQL to run again.

## Completion criteria

The production contract is complete only when:

1. Managed rows, transformations, validations, transitions, deterministic merges, archive, capture, and batching compile through the same typed IR and unified DAG.
2. Identity hashes are acyclic, domain-separated, and stable across processes.
3. Historical sources resolve to exact, checksummed lineage with no apply-time fallback.
4. Every mutation has enforceable ownership and scope.
5. Every artifact set has a verified manifest covering SQL, plan, frozen audit data, lineage, and frozen static sources.
6. Successful execution occurs once per epoch and every attempt remains auditable.
7. Non-transactional backends cannot retry without durable checkpoints or reconciliation.
8. Rollback preserves prior success history and touches only proven owned effects.
9. Schema and data operations retain dependency order through execution.
10. Unsupported backend or operation combinations fail closed.
11. `check --data` proves or reports current managed-data convergence.
12. Existing migrations and seeds retain their documented semantics.
13. Merge contributions and winners are deterministic and retain authenticated provenance.
14. Archive and capture rollback refuse changed or unowned state.
15. Batched statements preserve the unbatched result and roll back as one statement effect.

Shipping compiler preview, generated SQL, or green unit tests without the journal, ownership, manifest, rollback, and real-backend evidence does not satisfy these criteria.

========================================================================
PAGE: https://docs.dbwarden.org/features/
========================================================================

# Features

This page gives a compact overview of the main dbwarden features, with short examples. dbwarden is a schema compiler. Each feature maps to a compilation stage.

For the complete workflow index and implementation boundaries, see the [feature map](reference/feature-map.md). It includes safety scopes, repeatables, state composition, merge/reconcile, plugin groups, and operational diagnostics.

## SQL-First Compilation

dbwarden compiles migrations as plain SQL files. Each file contains both an `--upgrade` section and a `--rollback` section.

```sql
-- upgrade
CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE
);

-- rollback
DROP TABLE users;
```

This keeps schema changes reviewable in code review and runnable without hidden ORM magic.

## Typed Database Configuration

Configure one or many databases with explicit `DbwardenDatabase` subclasses.

```python
from dbwarden import DbwardenDatabase


class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/main"
    model_paths = ["app.models"]
    model_tables = ["users", "posts", "comments"]

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://default:@localhost:8123/analytics"
    model_paths = ["app.analytics_models"]
    model_tables = ["events", "page_views"]
```

Each entry is validated before use, including database names, table names, `model_paths`, `model_tables`, and duplicate target detection.
The supported `database_config(...)` function alternative is useful when a
plugin or integration supplies function-style configuration.

## Model-Driven Migration Generation

dbwarden reads SQLAlchemy models, diffs them against the live schema or an offline model state, and emits SQL.

```text
$ dbwarden make-migrations "add posts table" --database primary
Created migration: migrations/primary/primary__0002_add_posts_table.sql
```

## Declarative Data Migrations

Managed rows, typed derived values, and historical table transitions compile into versioned SQL plus a canonical frozen data artifact. The executor journals each application and checks guards in the migration transaction. See [Declarative data migrations](declarative-data-migrations.md) for declarations, artifacts, rollback, and reconciliation.

## Backend-Specific Metadata

PostgreSQL, MySQL/MariaDB, and ClickHouse support first-class metadata through `class Meta`.

PostgreSQL tables:

```python
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg


class Meta(PGTableMeta):
    pg_fillfactor = 80
    pg_schema = "app"

    class id(PGColumnMeta):
        pg = pg.field(identity="always")
```

PostgreSQL views and materialized views:

```python
from dbwarden.databases.pgsql import PGViewMeta


class Meta(PGViewMeta):
    pg_view_query = "SELECT id, email FROM users WHERE active = true"
    pg_view_materialized = False
    pg_schema = "app"
```

Set `pg_view_auto_refresh = True` for materialized views that need `REFRESH MATERIALIZED VIEW` on every migration cycle.

MySQL

```python
from dbwarden.databases.mysql import MyTableMeta, MyColumnMeta, my


class Meta(MyTableMeta):
    my_engine = "InnoDB"
    my_charset = "utf8mb4"

    class id(MyColumnMeta):
        my = my.field(unsigned=True)
```

ClickHouse example:

```python
from dbwarden.databases.clickhouse import CHTableMeta, ChEngineSpec, ChIndexSpec


class Meta(CHTableMeta):
    ch_engine = ChEngineSpec("MergeTree")
    ch_order_by = ["event_date", "id"]
    ch_indexes = [
        ChIndexSpec("ix_payload", ["payload"], type="bloom_filter", granularity=1),
    ]
```

## Safety Classification

Use `check` to classify changes before generating or applying SQL.

```text
$ dbwarden check --database primary
SAFE      add column users.bio
WARN      shrink varchar users.email
CRITICAL  drop table audit_log
```

Safety levels are `SAFE`, `INFO`, `WARN`, and `CRITICAL`.

## Read-Only Schema Diffing

Use `diff` when you want to inspect differences without writing migration files.

```bash
$ dbwarden diff --database primary
```

This is useful during reviews, debugging, and CI checks.

## Reverse-Engineering Models

Generate SQLAlchemy model code from a live database.

```bash
$ dbwarden generate-models --database primary --tables users,posts
```

This is useful when adopting dbwarden in an existing project, documenting an inherited schema, or recovering model definitions.

## Offline Migrations

dbwarden can generate migrations without connecting to a live database, by diffing current models against an exported model state.

```bash
$ dbwarden export-models --database primary
$ dbwarden make-migrations "offline change" --offline --database primary
```

This is useful for CI pipelines and restricted environments.

## Multi-Database Workflows

Manage multiple backends from one repository, each with its own migration directory and model set.

```bash
$ dbwarden migrate --database primary
$ dbwarden migrate --database analytics
```

You can also show status across all configured databases:

```bash
$ dbwarden status --all
```

## Dev Mode

Use `--dev` to point a configured database at a separate development target.

```python
primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

```bash
$ dbwarden --dev make-migrations "test locally" --database primary
$ dbwarden --dev migrate --database primary
```

## Seed Management

dbwarden tracks SQL and Python seed files separately from schema migrations.

```bash
$ dbwarden seed create "load countries" --type sql --database primary
$ dbwarden seed apply --database primary
$ dbwarden seed list --database primary
```

This is useful for reference data, lookup tables, and repeatable environment setup.

## FastAPI Integration

`database_config(...)` returns a `DatabaseHandle` that can be used directly in FastAPI dependencies.

```python
from fastapi import APIRouter
from .dbwarden import primary


router = APIRouter()


@router.get("/users")
async def list_users(session: primary.async_session):
    ...
```

This gives one shared source of truth for migrations, runtime connections, and session injection.

## Sandbox Testing

Apply migrations in a temporary sandbox database before applying them for real.

```bash
$ dbwarden migrate --sandbox --database primary
```

This is useful when validating generated SQL against a throwaway environment.

## Status, History, and Rollback

dbwarden includes built-in commands for operational visibility.

```bash
$ dbwarden status --database primary
$ dbwarden history --database primary
$ dbwarden rollback --count 1 --database primary
```

These commands make the migration lifecycle inspectable and reversible.

## Next Steps

- Follow [Get Started](getting-started/setup.md)
- Explore [Cookbook & Examples](cookbook/index.md)
- Use [CLI Reference](cli-reference.md) for command lookup

========================================================================
PAGE: https://docs.dbwarden.org/getting-started/developing-locally/
========================================================================

# Developing Locally

This guide covers the local development workflow: using a development database, checking diffs safely, reverse-engineering models, and generating offline migrations.

## Use Dev Mode

Dev mode swaps the configured database target for `dev_database_url` and `dev_database_type`.

Configuration example:

```python
from dbwarden import DbwardenDatabase


class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/main"
    model_paths = ["app.models"]
    model_tables = ["users", "posts", "comments"]
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./development.db"
```

Run local commands against the development target:

```text
$ dbwarden --dev make-migrations "test local change" --database primary
Created migration: migrations/primary/primary__0002_test_local_change.sql
$ dbwarden --dev migrate --database primary
Applying migration: primary__0002_test_local_change.sql
Migration applied successfully
```

## SQLite Translation

Using SQLite in dev mode is common, but not every type or default translates perfectly from server databases. dbwarden handles this by translating backend-specific types and warning when fidelity is reduced.

If you want those warnings to become hard failures, use:

```bash
$ dbwarden --dev --strict-translation make-migrations "validate translation" --database primary
```

## Check the Planned Changes

Use `diff` when you want to inspect differences without writing files:

```bash
$ dbwarden diff --database primary
```

Use `check` when you want a safety classification:

```text
$ dbwarden check --database primary
SAFE      add column users.bio
WARN      shrink varchar users.email
CRITICAL  drop table audit_log
```


## Offline Migrations

Offline migrations let you generate SQL without connecting to a live database. The workflow is:

1. Export the current model state.
2. Change your models.
3. Generate a migration with `--offline`.

Commands:

```bash
$ dbwarden export-models --database primary
$ dbwarden make-migrations "offline schema change" --offline --database primary
```

This is useful for CI, restricted environments, and workflows where the migration plan should not depend on a live database connection.

For a full walkthrough, see [Cookbook: Offline & CI](../cookbook/04-offline-ci.md).

## Local Validation Loop

A practical local loop looks like this:

```bash
$ dbwarden --dev diff --database primary
$ dbwarden --dev check --database primary
$ dbwarden --dev make-migrations "local change" --database primary
$ dbwarden --dev migrate --database primary
$ dbwarden --dev status --database primary
```

This keeps feedback fast while still using the same toolchain you use in production.

Next, continue with [Workflows](workflows.md).

========================================================================
PAGE: https://docs.dbwarden.org/getting-started/first-migration/
========================================================================

# Your First Migration

This guide walks through the core dbwarden workflow: define models, generate SQL, apply the migration, inspect the result, and roll it back.

## Create the Models

Create `app/models.py`:

```python
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases import IndexSpec, TableMeta


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    bio: Mapped[str | None] = mapped_column(Text, nullable=True)

    class Meta(TableMeta):
        comment = "Core user accounts"


class Post(Base):
    __tablename__ = "posts"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String(255), nullable=False)
    body: Mapped[str] = mapped_column(Text, nullable=False)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False)

    class Meta(TableMeta):
        indexes = [
            IndexSpec(name="ix_posts_created_at", columns=["created_at"]),
        ]
```

## Generate the Migration

Run:

```text
$ dbwarden make-migrations "create core tables" --database primary
Created migration: migrations/primary/primary__0001_create_core_tables.sql
```

dbwarden compares your current models against the live schema, or snapshot state, and writes a new SQL migration file.

The `make-migrations` and `migrate` command names deliberately mirror Django's `makemigrations` and `migrate`, so the generate-then-apply loop is familiar if you have used Django. The rest of the CLI follows its own conventions.

## Review the Generated SQL

Open the new file. It will look roughly like this:

```sql
-- upgrade

CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    bio TEXT
);

CREATE TABLE IF NOT EXISTS posts (
    id SERIAL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    user_id INTEGER NOT NULL REFERENCES users(id),
    created_at TIMESTAMP NOT NULL
);

CREATE INDEX IF NOT EXISTS ix_posts_created_at ON posts (created_at);

-- rollback

DROP INDEX IF EXISTS ix_posts_created_at;
DROP TABLE posts;
DROP TABLE users;
```

The exact SQL depends on the backend, but the structure is always the same:

- `-- upgrade` contains the forward change
- `-- rollback` contains the reverse change

## Apply the Migration

Run:

```text
$ dbwarden migrate --database primary
Applying migration: primary__0001_create_core_tables.sql
Migration applied successfully
```

Internally, dbwarden resolves the config, acquires the migration lock, executes the upgrade SQL, records the checksum, and releases the lock.

## Verify the Result

Run:

```text
$ dbwarden status --database primary
Database: primary
Applied migrations: 1
Pending migrations: 0
$ dbwarden history --database primary
1  primary__0001_create_core_tables.sql  applied
```

Use `status` to see the current state of the migration queue. Use `history` to see what has been applied and in what order.

You can also inspect the live schema directly:

```bash
$ dbwarden check-db --database primary
```

This is useful when you want a read-only view of what the database currently contains.

## Roll Back the Migration

Run:

```text
$ dbwarden rollback --count 1 --database primary
Rolling back migration: primary__0001_create_core_tables.sql
Rollback completed successfully
```

After that, the database is back to its previous schema state.

## Step by Step

### Step 1: Define the Base Class

```python
from sqlalchemy.orm import DeclarativeBase


class Base(DeclarativeBase):
    pass
```

dbwarden does not export a shared `Base`. You define a local SQLAlchemy declarative base in your project.

### Step 2: Define the Models

```python
class User(Base):
    __tablename__ = "users"
```

Every model maps to a table. Columns come from normal SQLAlchemy field declarations. Table-level migration metadata lives in `class Meta`.

### Step 3: Add Table Metadata

```python
class Meta(TableMeta):
    comment = "Core user accounts"
```

`TableMeta` is the cross-database surface for comments, indexes, checks, and unique constraints.

### Step 4: Generate SQL

```bash
$ dbwarden make-migrations "create core tables" --database primary
```

This command inspects the configured models, compares them with the current schema, and emits a SQL file. The file becomes part of your normal code review and deployment workflow.

### Step 5: Review Upgrade and Rollback

Every migration file contains both directions. This is one of dbwarden's core design choices: a migration is not complete until the rollback exists.

### Step 6: Apply the Migration

```bash
$ dbwarden migrate --database primary
```

This executes pending migrations in order and records them in the migration table.

### Step 7: Verify the State

```bash
$ dbwarden status --database primary
$ dbwarden history --database primary
```

Verification is part of the workflow, not optional cleanup.

### Step 8: Roll It Back

```bash
$ dbwarden rollback --count 1 --database primary
```

Rolling back the first migration confirms that the file contains valid reverse SQL, not just valid forward SQL.

## Manual Migrations

Not every schema or data change should be auto-generated. When the change is not model-driven, create a manual migration file:

```bash
$ dbwarden new "manual hotfix" --database primary
```

Use manual migrations for cases like:

- data backfills
- type changes that require custom `USING` expressions
- backend-specific operations that need hand-written SQL

dbwarden will track these files the same way it tracks generated migrations.

Next, continue with [Developing Locally](developing-locally.md).

========================================================================
PAGE: https://docs.dbwarden.org/getting-started/first-steps/
========================================================================

# First Steps

This walkthrough is the foundation of the dbwarden workflow.

The goal is not just to run commands, but to understand why each step exists and how it fits the migration lifecycle.

## Step 1: Initialize the project

```bash
$ dbwarden init
```

This creates:

- a migrations directory structure
- a declarative Python configuration scaffold (`dbwarden.py`)

Why it matters: dbwarden expects a project-local migration layout and config source so migration behavior is deterministic per repository.

## Step 2: Define one explicit database entry

```python
from dbwarden import DbwardenDatabase


class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/main"
    model_paths = ["app.models"]
    model_tables = ["users"]
```

Why it matters: dbwarden resolves migration targets from explicit typed entries, not inferred environment state. The equivalent `database_config(...)` function API remains supported and may appear in plugin integrations.

## Step 3: Add SQLAlchemy models

dbwarden uses model metadata to generate migration SQL. A minimal model example:

```python
from sqlalchemy import DateTime, Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from datetime import datetime


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
```

Why it matters: model metadata is the input to `make-migrations`.

## Step 4: Generate migration SQL

```bash
$ dbwarden make-migrations -d "create users table" --database primary
```

dbwarden creates a versioned SQL file under `migrations/primary/`.

Why it matters: this file is now part of your code review process and deployment artifact.

## Step 5: Review the generated migration

Open the file and validate both sections:

```sql
-- upgrade

-- rollback
```

Why it matters: rollback quality determines recovery quality.

## Step 6: Apply migrations

```bash
$ dbwarden migrate --database primary
```

During execution dbwarden:

1. resolves config and target database
2. acquires migration lock
3. executes pending SQL
4. stores migration record and checksum
5. releases lock

## Step 7: Verify the result

```bash
$ dbwarden status --database primary
$ dbwarden history --database primary
```

Use status to confirm pending/applied counts and history to confirm execution order.

## Common first-run issues

- `No configuration found`: ensure your project has one discovered config source with a concrete `DbwardenDatabase` subclass or a `database_config(...)` call
- `Database '<name>' not found`: ensure `--database` matches configured `database_name`
- `No SQLAlchemy models found`: set `model_paths` explicitly in config

## Next Steps

- [Configuration](../configuration/index.md)
- [Your First Migration](first-migration.md)
- [Developing Locally](developing-locally.md)

========================================================================
PAGE: https://docs.dbwarden.org/getting-started/migrating-from-alembic/
========================================================================

# Migrating from Alembic

This guide is for teams with an existing Alembic setup who want to switch to dbwarden. The migration is low-risk by design: your SQLAlchemy models stay exactly where they are, your database is never rebuilt, and your Alembic history remains in git. You are replacing the migration workflow, not the schema.

If you are evaluating rather than migrating, read [Why dbwarden](../index.md#why-dbwarden) first. The short version: Alembic maintains schema truth in a chain of revision scripts, dbwarden maintains it in your models and derives plain SQL migrations from them.

## Concept mapping

Every Alembic concept has a dbwarden counterpart. This table is the mental model for the whole guide.

| Alembic | dbwarden |
|---|---|
| `alembic.ini` + `env.py` | `dbwarden.py` config file |
| Revision script (`.py`) | Migration file (`.sql`) with `-- upgrade` / `-- rollback` sections |
| Revision chain as source of truth | SQLAlchemy models as source of truth |
| `alembic revision --autogenerate` | `dbwarden make-migrations "description"` |
| `alembic upgrade head` | `dbwarden migrate` |
| `alembic downgrade -1` | `dbwarden rollback` |
| `alembic stamp <rev>` | `dbwarden migrate --baseline` |
| `alembic current` | `dbwarden status` |
| `alembic history` | `dbwarden history` |
| `alembic upgrade head --sql` (offline mode) | `dbwarden make-migrations --sql`, or full offline generation via `export-models` + `--offline` |
| `alembic_version` table | dbwarden migration table |
| Hand-written revision | `dbwarden new "description"` (manual SQL migration) |

Two Alembic features have no direct equivalent, and you should know that before starting:

- **Python data migrations.** Alembic revisions can run arbitrary Python. dbwarden manual migrations (`dbwarden new`) are SQL. Most backfills express well in SQL; if yours do not, plan for how you will run them (application scripts, one-off jobs) before switching.
- **Revision branching and merging.** dbwarden uses a linear versioned sequence per database. If your team relies on `alembic merge` workflows, the linear model is a real change to your process.

## Prerequisites

- Python 3.12+ and SQLAlchemy 2.0+
- A database whose schema currently matches your models (run your pending Alembic migrations first, and resolve any drift)
- Your SQLAlchemy models importable from the project

## Step 1: Install and configure

```bash
uv add dbwarden
```

Create `dbwarden.py` in your project root. This replaces `alembic.ini` and `env.py`:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:pass@localhost:5432/myapp"
    database_url_async = "postgresql+asyncpg://user:pass@localhost:5432/myapp"
```

The existing `database_config(...)` function API is also supported if your
project or a plugin already uses that form.

Model discovery is automatic. If you want explicit control over which modules are scanned, pass `model_paths`. See [Setup](setup.md) and [Model Discovery](../configuration/model-discovery.md).

Then initialize:

```bash
dbwarden init
```

## Step 2: Generate the baseline migration

Your first `make-migrations` run has no prior dbwarden state, so it generates the full schema from your models:

```bash
dbwarden make-migrations "baseline from alembic"
```

Review the generated `.sql` file. It should describe the schema you already have. If something looks wrong here, your models and your database disagree, and it is far better to learn that now than later. Fix the models (or the database) and regenerate before continuing.

## Step 3: Baseline the existing database

Your database already has the schema, so the baseline migration must be recorded as applied without executing:

```bash
dbwarden migrate --baseline
```

This marks the migration as applied in dbwarden's migration table. Nothing runs against the database. This is the equivalent of `alembic stamp head` for a fresh setup.

## Step 4: Verify convergence

```bash
dbwarden status
dbwarden check
dbwarden diff
```

`status` should show no pending migrations. `diff` compares your models against the live database and should report no differences. If `diff` is clean, dbwarden and reality agree, and the migration is effectively done.

For extra confidence, make a trivial model change (add a nullable column), run `dbwarden make-migrations`, inspect the generated SQL and its rollback, then revert the change and delete the generated file. That exercise shows you the full loop before you rely on it.

## Step 5 (optional): Set up offline generation for CI

If your CI previously needed a database service for `alembic revision --autogenerate` checks, you can drop it entirely:

```bash
dbwarden export-models --database primary
git add .dbwarden/model_state.primary.json
```

The state file is named after the database, so a database called `primary` produces `.dbwarden/model_state.primary.json`.

From then on, any machine can generate migrations without a database connection:

```bash
dbwarden make-migrations "description" --offline
```

See [Cookbook: Offline & CI](../cookbook/04-offline-ci.md). Do not delete the model state file; it is the offline source of truth. If it is ever lost, restore it from git or regenerate with `export-models`.

## Step 6: Retire Alembic

Once you trust the new workflow:

- Remove `alembic.ini`, `env.py`, and the `versions/` directory from the active workflow. They stay in git history, which is where migration archaeology belongs.
- Remove `alembic` from your dependencies and any `alembic upgrade head` calls from deploy scripts, replacing them with `dbwarden migrate`.
- The `alembic_version` table in your database is inert. Drop it when convenient, or leave it; dbwarden does not touch it.

If you prefer a transition period, both tools can coexist: they track state in separate tables. Generate the same change with both and compare the SQL until you are confident. Keep the overlap short, since two sources of truth defeats the point.

## Workflow changes to communicate to your team

- **Renames are explicit.** Where autogenerate guessed (usually as drop-and-create), dbwarden requires `--rename table.old:new` or `--rename-table old:new`. A rename never becomes silent data loss, but your team must know the flags exist.
- **Rollback is a contract.** Generated migrations include an executable `-- rollback` section, and placeholder rollbacks are refused. Irreversible changes must be declared with `-- dbwarden: irreversible`. See [Rollback Generation](../correctness/rollback-generation.md).
- **Table metadata moves into `class Meta`.** Comments, advanced indexes, and backend-specific options that previously lived in revision scripts or raw SQL are declared on the model. See [Modeling](modeling.md).
- **Destructive changes can be checked first.** `dbwarden check-impact` reports which code still references a column or table you are about to drop. Make it part of review for destructive migrations.

========================================================================
PAGE: https://docs.dbwarden.org/getting-started/modeling/
========================================================================

# Modeling Guide

This guide walks through the process of defining SQLAlchemy models that dbwarden can read to generate migration SQL. For the complete reference of all supported Meta attributes, see [SQLAlchemy Models Reference](../models.md).

## How dbwarden Reads Models

dbwarden discovers models in the directories specified by `model_paths` in your `database_config(...)`. It reads two sources of metadata from each model:

1. **Column definitions**: typed SQLAlchemy `Mapped[...] = mapped_column(...)` fields, nullability, defaults, primary keys
2. **`class Meta` inner class**: backend-specific options like engine specs, partitioning, codecs

All backend-specific metadata uses the `class Meta` pattern. The `__table_args__` approach is not supported for PostgreSQL metadata. Using `mapped_column(info=...)` for backend-specific options raises `DBWardenConfigError`.

The `class Meta` convention is borrowed from Django's model metadata, so the pattern should read as familiar if you have used Django. dbwarden's version is typed and validated at import time: unknown attributes raise `DBWardenConfigError` when the module loads rather than producing incorrect DDL later.

## Common Meta Attributes

Every backend supports a core set of cross-database attributes. These work with any `database_type`.

### Table-Level

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases import TableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(TableMeta):
        comment = "Core user accounts"
        indexes = [
            {"name": "ix_users_email", "columns": ["email"]},
        ]
```

Available table-level attributes: `comment`, `indexes`, `checks`, `uniques`. See [Common Meta Attributes](../models.md#common-meta-attributes) for details.

### Column-Level

```python
class Meta(TableMeta):
    class internal_note:
        comment = "Internal system note"
        public = False
```

Available column-level attributes: `comment`, `public`. Fields named with a leading `_` are implicitly `public=False`.

For backend-specific column options, use `pg = pg.field(...)` for PostgreSQL. See [Column-Level Meta Base Class](../models.md#column-level-meta-base-class) for details.

## PostgreSQL Models

When `database_type="postgresql"`, use `class Meta(PGTableMeta)` for table-level metadata and `PGColumnMeta` inner classes for column-level metadata.

```python
from sqlalchemy import Integer, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    bio: Mapped[str] = mapped_column(Text)

    class Meta(PGTableMeta):
        pg_fillfactor = 80

        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", compression="pglz")
```

See the [reference](../models.md#postgresql-model-metadata) for the full list of `PGTableMeta` and `PGColumnMeta` attributes, or the [PostgreSQL Deep Dive](../databases/postgresql/index.md) for DDL behavior and snapshot format.

### PostgreSQL Views and Schemas

dbwarden supports PostgreSQL views and materialized views via `PGViewMeta`. Define a view as a model with `__tablename__` matching the view name:

```python
from dbwarden.databases.pgsql import PGViewMeta

class ActiveUser(Base):
    __tablename__ = "active_users"

    id: Mapped[int] = mapped_column(Integer)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(PGViewMeta):
        pg_view_query = "SELECT id, email FROM users WHERE active = true"
        pg_view_materialized = False
```

For materialized views, set `pg_view_materialized = True` and `pg_view_auto_refresh = True` to emit `REFRESH MATERIALIZED VIEW` automatically:

```python
class OrderSummary(Base):
    __tablename__ = "order_summary"

    user_id: Mapped[int] = mapped_column(Integer)
    total: Mapped[float] = mapped_column(Integer)

    class Meta(PGViewMeta):
        pg_view_query = "SELECT user_id, count(*) AS total FROM orders GROUP BY user_id"
        pg_view_materialized = True
        pg_view_auto_refresh = True
```

Scope tables or views to a PostgreSQL schema with `pg_schema`:

```python
class Meta(PGTableMeta):
    pg_schema = "app"

# DDL uses app.users instead of public.users
```

At the config level, set `pg_schema` in `database_config(...)` to set the connection `search_path`. See [Schema Support](../databases/postgresql/schemas.md#config-level-schema) for details.


## Using `generate-models` as a Starting Point

> **Note**: `generate-models` only works for databases with round trip support (PostgreSQL, SQLite, MySQL, ClickHouse). See [Round Trip Support](../databases/round-trip.md) for details.

The fastest way to get a correct model is to reverse-engineer it from your live database:

```bash
$ dbwarden generate-models -d primary --tables users,orders
```

dbwarden produces one `.py` file per table (or a single `models.py` with `--single-file`). The generated output includes `class Meta` with all detected backend-specific metadata.

Review the generated code before using it:

- Column types are mapped from database types to SQLAlchemy types. Verify the mapping is correct for your use case.
- Generated `class Meta` attributes are complete but may need adjustment (for example, you might want different index names or additional column hints).
- Partitioning, TTL, and engine settings are captured from the live database. If the database schema has drifted from what you intend, edit the model before running `make-migrations`.

## Auto-Generated Pydantic Schemas with `@auto_schema`

Use `@auto_schema` to generate four Pydantic schema classes on your model:

| Attribute | Contents |
|-----------|---------|
| `Model.Schema` | All mapped columns |
| `Model.CreateSchema` | Excludes server-defaulted columns (PKs with identity, `server_default`) |
| `Model.UpdateSchema` | All fields optional |
| `Model.PublicSchema` | Excludes fields where `public=False` or name starts with `_` |

Requires the `dbwarden-fastapi` plugin:

```bash
dbwarden plugin add dbwarden-fastapi
```

```python
from dbwarden_fastapi import auto_schema

@auto_schema
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    password_hash: Mapped[str] = mapped_column(String(255))

    class Meta:
        class email:
            comment = "Primary contact email"
            public = True

        class password_hash:
            public = False

# PublicSchema excludes password_hash and any _prefixed fields
public = User.PublicSchema(email="alice@example.com")
```

The decorator reads `class Meta` to infer `SchemaConfig`, then calls `schemap` to build the Pydantic models. Column `comment` values are injected into Pydantic field descriptions, and backend-specific metadata (`pg_*`, `my_*`, `ch_*`, `mdb_*`, `sq_*`) is included in `json_schema_extra.dbwarden_backend_meta`.

To customize schema generation, pass a `SchemaConfig` explicitly:

```python
from dbwarden_fastapi import auto_schema, SchemaConfig

@auto_schema(config=SchemaConfig(exclude_public=["internal_note"]))
class Order(Base):
    ...
```

`SchemaConfig` supports the following fields:

| Field | Type | Description |
|-------|------|-------------|
| `exclude_always` | `list[str]` | Excluded from all schemas |
| `exclude_create` | `list[str]` | Excluded from CreateSchema only |
| `exclude_update` | `list[str]` | Excluded from UpdateSchema only |
| `exclude_public` | `list[str]` | Excluded from PublicSchema only |
| `field_overrides` | `dict` | Override field types in generated schemas |
| `required_always` | `list[str]` | Fields that are always required |
| `optional_always` | `list[str]` | Fields that are always optional |

## When to Use Manual Migrations

Auto-generated migrations handle most cases, but some schema changes still need manual intervention via `dbwarden new`:

- PostgreSQL `USING` clause for type casts (e.g., casting `TEXT` to `INTEGER`). dbwarden emits `ALTER COLUMN ... TYPE` with a commented-out `-- USING col::newtype` line. Pass `--postgres-auto-using` to emit an active `USING` clause.
- Column renames not caught by the heuristic auto-detection. Use `--rename old_name:new_name` flags for deterministic renames, or rename in a manual migration.
- Data migrations (backfilling, transforming existing data). dbwarden emits a SQL comment placeholder.

For these cases run `dbwarden new` and write the SQL by hand, or use the relevant flag for auto-generation.

## Best Practices

- **One model class per table**: dbwarden discovers models by scanning directories. Each table should have exactly one model class.
- **Use `model_paths`**: always set `model_paths` explicitly in `database_config(...)`. Auto-discovery is available but explicit paths are more predictable.
- **Review generated migrations**: always read the `.sql` file before running `dbwarden migrate`.
- **Use `--dev` for local development**: configure a `dev_database_url` (SQLite works well) and use `dbwarden --dev` to iterate quickly without touching your real database.
- **Keep Meta classes minimal**: only set attributes that differ from the default. Default values are omitted from generated migrations, reducing noise.
- **Use `@auto_schema` for API projects** (requires `dbwarden-fastapi` plugin): generates Pydantic schemas from your model annotations. Fields with `public=False` or a leading `_` are excluded from `PublicSchema`.

See also: [Cookbook: Models & Migrations](../cookbook/02-models-and-migrations.md)

Next, continue with [Your First Migration](first-migration.md).

========================================================================
PAGE: https://docs.dbwarden.org/getting-started/setup/
========================================================================

# Setup

This guide shows the initial project setup for dbwarden. By the end, you will have dbwarden installed, a project-local config file, and one verified database entry.

## Requirements

- Python 3.12.7 or higher
- A project that uses SQLAlchemy models, or plans to
- A supported backend: PostgreSQL, MySQL, MariaDB, SQLite, or ClickHouse

## Install dbwarden

Install the base package:

```bash
uv add dbwarden
```

Optional dependency groups for database drivers:

| Group | Command | Use case |
|---|---|---|
| `postgres` | `uv add "dbwarden[postgres]"` | PostgreSQL driver |
| `mysql` | `uv add "dbwarden[mysql]"` | MySQL/MariaDB driver |
| `clickhouse` | `uv add "dbwarden[clickhouse]"` | ClickHouse driver |

You can combine them:

```bash
uv add "dbwarden[postgres,mysql]"
```

## Initialize the Project

```text
$ dbwarden init
Initialized dbwarden project structure
Created migrations directory
Created dbwarden.py
```

`init` creates the local migration layout and a declarative config scaffold. It is safe to run again, dbwarden will not destroy existing `database_config(...)` definitions.

## Create the Configuration File

By default, `dbwarden init` creates `dbwarden.py` with a `DbwardenDatabase` subclass, and that is the simplest place to start. The function alternative, `database_config(...)`, can also live in any discovered Python file inside your project.

The simplest `dbwarden.py` looks like this:

```python
from dbwarden import DbwardenDatabase


class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/main"
    model_paths = ["app.models"]
    model_tables = ["users", "posts", "comments"]
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./development.db"
```

Copy this into `dbwarden.py`, or use `database_config(...)` in another discovered Python module, then adjust the URLs, `model_paths`, and `model_tables` for your project.

## Step by Step

### Step 1: Import `DbwardenDatabase`

```python
from dbwarden import DbwardenDatabase
```

Define each concrete `DbwardenDatabase` subclass with a unique name and URL. Every configured database becomes part of the validated runtime config. The equivalent `database_config(...)` function API remains supported, including for plugin examples and integrations.

### Step 2: Define `database_name`

```python
database_name="primary"
```

This is the stable name you will use in CLI commands such as:

```bash
$ dbwarden status --database primary
```

### Step 3: Mark the default database

```python
default=True
```

Exactly one configured database must be the default. Commands without `--database` use that entry.

### Step 4: Set the backend type

```python
database_type="postgresql"
```

This controls backend-specific SQL generation, schema inspection, and metadata behavior.

Supported values:

- `postgresql`
- `mysql`
- `mariadb`
- `sqlite`
- `clickhouse`

### Step 5: Set the runtime URL

```python
database_url_sync="postgresql://user:password@localhost:5432/main"
```

This is the database URL used by CLI commands such as `make-migrations`, `migrate`, `status`, and `check`.

If your application also uses async SQLAlchemy sessions, you can define an async URL too:

```python
database_url_async="postgresql+asyncpg://user:password@localhost:5432/main"
```

dbwarden keeps sync and async URLs separate so the CLI and FastAPI runtime can share one config source without forcing a single driver choice.

### Step 6: Point to your models

```python
model_paths=["app.models"]
```

This tells dbwarden where to discover SQLAlchemy models. In multi-database projects, explicit `model_paths` are required.

### Step 7: Filter tables for this database

```python
model_tables=["users", "posts", "comments"]
```

Optional. When set, dbwarden only includes the listed tables from the discovered models. All other discovered tables are ignored.

This is useful when multiple databases share the same `model_paths` but own different subsets of tables:

```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/main"
    model_paths = ["app.models"]
    model_tables = ["users", "posts", "comments"]

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://default:@localhost:8123/analytics"
    model_paths = ["app.models"]
    model_tables = ["events", "page_views"]

class Logs(DbwardenDatabase):
    database_name = "logs"
    database_type = "mysql"
    database_url_sync = "mysql+pymysql://user:password@localhost:3306/logs"
    model_paths = ["app.models"]
    model_tables = ["audit_log", "error_log"]
```

If `model_tables` is not set, all discovered tables in `model_paths` belong to that database.

### Step 8: Configure a dev database

```python
dev_database_type="sqlite"
dev_database_url="sqlite:///./development.db"
```

This is optional, but recommended. It lets you run commands locally with `--dev`, without touching your main database.

## Verify the Configuration

Run:

```text
$ dbwarden settings show --all
Database: PRIMARY (default)
  Default: True
  Type: PostgreSQL
  URL: postgresql://user:password@localhost:5432/main
  Migrations Directory: migrations/primary
  Model Paths: ['app.models']
```

If this command works, dbwarden can resolve and validate your config.

## Common Problems

### `No configuration found`

dbwarden could not locate a config source. Make sure your project contains at least one discovered file with a concrete `DbwardenDatabase` subclass or a `database_config(...)` call. `dbwarden.py` is the default convention, but it does not have to be the only location.

### `Exactly one default=True required`

If you configure more than one database, only one can be the default.

### `model_paths is required when more than one database is configured`

In multi-database setups, each database must declare the models that belong to it.

Next, continue with [Modeling](modeling.md).

========================================================================
PAGE: https://docs.dbwarden.org/getting-started/workflows/
========================================================================

# Workflows

This guide covers larger day-to-day workflows once the basics are in place.

## Multi-Database Projects

dbwarden can manage more than one database from one config source.

```python
from dbwarden import DbwardenDatabase


class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/main"
    model_paths = ["app.models"]
    model_tables = ["users", "posts", "comments"]

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://default:@localhost:8123/analytics"
    model_paths = ["app.analytics_models"]
    model_tables = ["events", "page_views"]
```

The equivalent `database_config(...)` function form remains supported and is
used by some plugins and integration examples.

Apply migrations per database:

```bash
$ dbwarden migrate --database primary
$ dbwarden migrate --database analytics
```

Show status across all configured databases:

```bash
$ dbwarden status --all
```

## Separate Model Sets

Each database should usually own a distinct model set through `model_paths`. When databases share the same models package, use `model_tables` to split ownership by table name. dbwarden validates overlapping paths unless `overlap_models=True` is set explicitly.

This prevents one model tree from being interpreted as belonging to multiple databases by accident.

## CI Workflows

A common CI pattern is:

```bash
$ dbwarden export-models --database primary
$ dbwarden make-migrations "ci validation" --offline --database primary
$ dbwarden check --database primary
```

This keeps schema generation deterministic and avoids depending on a live database in every pipeline step.

For a full example, see [Cookbook: Offline & CI](../cookbook/04-offline-ci.md).

## Sandbox Validation

Before applying migrations to a real environment, you can validate them in a temporary sandbox database.

```bash
$ dbwarden migrate --sandbox --database primary
```

This is especially useful for complex migrations, risky type changes, and CI gates.

See the [Architecture Deep Dive](../architecture-deep-dive.md) for a thorough explanation of sandbox validation.

## Baselines and Partial Applies

When integrating dbwarden into an existing environment, or when applying only part of a migration sequence, these patterns are common:

- `--baseline` marks the target migration as already applied without actually running it, useful for onboarding an existing database.
- `--partial` (via `--count` or `--to-version`) applies a subset of pending migrations instead of all of them.

```bash
$ dbwarden migrate --database primary --baseline --to-version 0005
$ dbwarden migrate --database primary --count 2
$ dbwarden rollback --database primary --to-version 0007
```

See the [CLI Reference](../cli-reference.md) for a full breakdown of these flags. Use these modes carefully. They are operational tools, not everyday authoring commands.

## Operational Command Pattern

A typical production-safe pattern is:

```bash
$ dbwarden check --database primary
$ dbwarden make-migrations "release change" --database primary
$ dbwarden migrate --database primary
$ dbwarden status --database primary
$ dbwarden history --database primary
```

This keeps planning, execution, and verification as separate visible steps.

## Rollback Command Pattern

When validating rollback quality, use a loop like this:

```bash
$ dbwarden migrate --database primary
$ dbwarden rollback --count 1 --database primary
$ dbwarden migrate --database primary
```

This verifies both directions of the migration before a release depends on them.

## Where to Go Next

- Use [Cookbook Overview](../cookbook/index.md) for full working flows
- Use [Configuration](../configuration/index.md) for deeper config behavior
- Use [CLI Reference](../cli-reference.md) for command details

========================================================================
PAGE: https://docs.dbwarden.org/glossary/
========================================================================

# Glossary

## A

**Auto Schema**
: A feature that generates Pydantic schemas from SQLAlchemy model annotations using `@auto_schema` (from the `dbwarden-fastapi` plugin), eliminating duplication between ORM and API layers in FastAPI applications.

## B

**Backend**
: A supported database type (PostgreSQL, MySQL, MariaDB, SQLite, ClickHouse). Each backend has specific DDL syntax, feature support, and round-trip capability.

**Baseline**
: A migration version used as a starting point, marking the point up to which existing migrations are considered already applied without executing the upgrade SQL.

## C

**Checksum**
: A SHA-256 hash stored when a migration file is applied. On subsequent runs, the checksum is recalculated to detect file tampering or accidental edits.

**Check (command)**
: Analyzes schema differences between SQLAlchemy models and a live database, classifying every operation by danger level.

**Code Seed**
: A seed defined as a Python class extending `Seed`, with `forward(connection, session)` and `reverse(connection, session)` methods. The recommended way to manage seed data.

**Configuration (dbwarden.py)**
: dbwarden uses a Python file (`dbwarden.py`) with concrete `DbwardenDatabase` classes by default. The equivalent `database_config()` function API is also supported, providing type safety, runtime flexibility, and IDE support for configuring databases.

## D

**Database Config**
: A concrete `DbwardenDatabase` class, or equivalent `database_config()` call, that defines one database. Each config includes the database name, type, connection URL, model paths, and optional dev mode settings.

**Dev Mode**
: Using a different database type (typically SQLite) for local development while targeting a production database (e.g., PostgreSQL) in deployment, enabled via the `dev_database_type` and `dev_database_url` config options.

**Diff**
: A read-only command showing structural differences between SQLAlchemy models and a live database, with table, json, and sql output formats.

**Downgrade**
: Revert applied migrations to reach a specific target version by reading `-- rollback` sections and applying them in reverse order.

## F

**FastAPI Integration**
: Support for FastAPI including async sessions, health endpoints, migration endpoints, Prometheus metrics, and distributed locking. Provided by the [`dbwarden-fastapi`](https://pypi.org/p/dbwarden-fastapi) plugin. Configured once via a `DbwardenDatabase` class or the equivalent `database_config()` function.

## G

**generate-models**
: A command that reverse-engineers SQLAlchemy model code from a live database, supporting all backends.

## H

**Health Endpoints**
: Production-ready HTTP endpoints for database connectivity checks, migration state monitoring, and Kubernetes liveness/readiness probes.

## I

**Impact Analysis**
: The ability to find affected Python code references (function calls, class references, variable names) before deploying a migration, via the `check-impact` command.

**Index Spec**
: A class (`IndexSpec`) used in model `Meta` to define database indexes, including columns, uniqueness, and types.

## L

**Lock (Migration Lock)**
: A database-level lock that prevents concurrent schema mutations across multiple application instances or CLI invocations.

## M

**make-migrations**
: The command that generates SQL migration files by comparing current SQLAlchemy model definitions against either a live database or stored schema snapshots.

**Manual Migration**
: A migration file created by hand (via `dbwarden new`) rather than auto-generated. Useful for data migrations, stored procedures, or any DDL outside model diffs.

**Meta (class Meta)**
: An inner class on SQLAlchemy models that provides dbwarden with backend-specific metadata like table comments, indexes, partitioning, engine options, and more.

**Migration File**
: A plain SQL file with `-- upgrade` and `-- rollback` sections. Each file represents one atomic schema change.

**Multi-Database**
: Managing multiple database configurations (e.g., primary + analytics, or microservice per database) from a single `dbwarden.py` config file.

## O

**Observability**
: dbwarden's monitoring capabilities including Prometheus metrics (migration counters, schema version gauges, connection pool health) and structured JSON logging.

**Offline Mode**
: Generating migrations using stored JSON schema snapshots instead of connecting to a live database, enabling CI/CD pipelines without database access.

## P

**Pydantic Schema (auto-generated)**
: Request/response schemas automatically generated from SQLAlchemy model annotations using `@auto_schema` (from the `dbwarden-fastapi` plugin), keeping API contracts in sync with database models.

## R

**Rename Detection**
: dbwarden can detect column and table renames by comparing schema snapshots, generating `ALTER TABLE ... RENAME` instead of `DROP` + `ADD`.

**Rollback**
: The `-- rollback` section of a migration file containing SQL to undo the upgrade. dbwarden enforces that every migration has a corresponding rollback.

**Round-Trip**
: A backend that supports both reading schema (via `generate-models`) and writing schema (via `make-migrations`/`migrate`). Verified when reverse-engineering a database and re-generating produces zero diff.

**RA (Runs-Always) Migration**
: A migration type that runs every time `migrate` is executed, regardless of previous application. Useful for idempotent operations like views or functions.

**ROC (Runs-on-Change) Migration**
: A migration type that runs only when its content has changed (detected via checksum). Useful for stored procedures that evolve over time.

## S

**Safety Check**
: A feature that classifies every migration operation into danger levels (SAFE, INFO, WARN, CRITICAL) so teams can review high-risk changes before production.

**Sandbox**
: An isolated test environment using Testcontainers to validate migrations against a real database before applying them to production or staging.

**Schema Snapshot**
: A JSON file recording the full DDL state of a database at the point a migration was applied. Enables offline migration generation, rename detection, and CI workflows.

**Seed**
: Data population mechanism using either Python code seeds (recommended) or file-based SQL/Python seeds. Seeds are tracked and versioned like migrations.

**SQL-First**
: A design philosophy where all schema changes are expressed as explicit SQL files that can be reviewed, tested, and rolled back, rather than being abstracted away by an ORM.

**SQL Translation**
: The ability to generate SQL for one dialect (e.g., PostgreSQL) while operating against a different development database (e.g., SQLite), enabling local development without full infrastructure.

## T

**TableMeta**
: The base class for model `Meta` inner classes, providing type-safe configuration of table-level metadata like comments, indexes, partitioning, and engine options.

## U

**Unlock**
: The `dbwarden unlock` command to recover from a stale migration lock when no migration is actually running and the lock was not released properly.

========================================================================
PAGE: https://docs.dbwarden.org/
========================================================================

<p align="center">
  <img src="https://raw.githubusercontent.com/dbwarden-org/dbwarden/refs/heads/main/assets/icon.png" alt="dbwarden" width="128"/>
</p>
<p align="center">
  <strong style="font-size: 2.5em;">dbwarden</strong>
</p>
<p align="center">
  <em>Your SQLAlchemy models are your migrations.</em>
</p>
<p align="center">
  <a href="https://www.python.org/downloads/">
    <img src="https://img.shields.io/badge/Python-3.12.7%2B-3776AB?logo=python&logoColor=white&style=for-the-badge" alt="Python">
  </a>
  <a href="https://pypi.org/project/dbwarden/">
    <img src="https://img.shields.io/pypi/v/dbwarden?logo=pypi&logoColor=white&style=for-the-badge" alt="PyPI">
  </a>
  <a href="https://github.com/dbwarden-org/dbwarden/blob/main/LICENSE">
    <img src="https://img.shields.io/badge/License-MIT-10AC84?style=for-the-badge" alt="License">
  </a>
  <a href="https://deepwiki.com/dbwarden-org/dbwarden/">
    <img src="https://img.shields.io/badge/DeepWiki-8A2BE2?logo=readthedocs&logoColor=white&style=for-the-badge" alt="DeepWiki">
  </a>
  <a href="https://codecov.io/gh/dbwarden-org/dbwarden">
    <img src="https://img.shields.io/codecov/c/github/dbwarden-org/dbwarden?logo=codecov&logoColor=white&style=for-the-badge" alt="Codecov">
  </a>
</p>

<p align="center">
  <strong><a href="https://docs.dbwarden.org/">Full documentation</a></strong>
  &nbsp;|&nbsp;
  <strong><a href="https://github.com/dbwarden-org/dbwarden">Source Code</a></strong>
</p>

---

dbwarden is a declarative schema compiler for SQLAlchemy. You declare the schema you want in your SQLAlchemy models, and dbwarden compiles everything else: migration SQL, rollbacks, snapshots, and safety checks.

There are no migration scripts to write or maintain. There is no migration runtime. Your models are the contract. The database is kept in sync with them.

## Features and limits

- **Migration authoring:** SQLAlchemy models, backend metadata, rename detection, reverse engineering, explicit SQL files, and rollback generation.
- **Safety:** operation classification, expand-contract type changes, severity splits, execution ceilings, pending-plan checks, and impact reports.
- **State and history:** offline model-state JSON, schema snapshots, checksums, repeatable migrations, merge reconciliation, and rebase support.
- **Operation:** multiple database configurations, optional database skipping, locks, sandbox providers, SQLite dev translation, JSON output, and tracing.
- **Extensions:** plugin configuration, handlers, hooks, CLI commands, and named migration groups. FastAPI, seed extensions, and backend object plugins have separate installation and compatibility requirements.

Use the [feature map](reference/feature-map.md) to find each workflow and its implementation boundary. The [CLI inventory](reference/cli-options.md) lists every built-in option; the [Python inventory](reference/python-api.md) lists exports, signatures, declared fields, and methods.

## At a glance
- Migrations generated from your models, not written by hand
- Plain SQL output: reviewable, committable, executable anywhere
- Rollback contract with executable rollback, strict placeholder refusal, and explicit irreversible declarations
- Pre-deploy impact analysis: know what breaks before it ships
- Offline migration generation for CI pipelines without a live database
- Schema snapshots for deterministic diffs and rename detection
- Typed `class Meta` system with import-time validation
- Multi-database support: PostgreSQL, MySQL, ClickHouse, MariaDB, SQLite
- Extensible plugin system with official plugins for seeds, RBAC, FastAPI, sandbox testing, and PostgreSQL/ClickHouse extensions
- Reverse-engineer live databases into models with `generate-models`

## Why dbwarden

Schema management tools fall into two camps. Imperative tools have you author *changes*: revision scripts that describe how to get from one schema version to the next. Declarative tools have you author the *desired state* and derive the changes for you. dbwarden is declarative: your SQLAlchemy models are the single definition of what the schema should be.

Most imperative tools ask you to maintain two representations of your schema: your ORM models and your migration files. When they drift, you find out at deploy time.

dbwarden eliminates the second representation. Your SQLAlchemy models are the schema definition. dbwarden reads them, diffs them against the current database state, and generates the SQL to close the gap (including rollback) without you writing a line of migration code.

This also means:

- No migration runtime to install or version
- No generated Python scripts that quietly do the wrong thing
- No schema drift discovered in production: drift is caught at `make-migrations` time
- Migrations that can be generated in CI without a database connection

dbwarden is not a wrapper around Alembic. It is a different approach to the same problem. Alembic asks you to describe *how* to change the database; dbwarden asks you to describe *what* the schema should be. Alembic can autogenerate revisions, but each one becomes an imperative Python artifact you own, edit, and chain; the revision history is the source of truth. With dbwarden the models stay the source of truth, and the output is plain SQL.

Unlike tools that apply declarative diffs directly to the database, dbwarden still produces versioned, reviewable migration files with explicit rollbacks: declarative authoring without giving up auditable deploy artifacts.

## From zero to production

Typical adoption path in an existing project:

1. Point dbwarden at your existing SQLAlchemy models
2. Run initial `make-migrations` to generate a baseline schema
3. Commit generated migrations as your source of truth
4. Replace your current migration workflow with the dbwarden CLI
5. Optionally enable:
   - Migration impact analysis for safer deploys
   - Offline mode for CI pipelines without a database service

## Installation

```bash
uv add dbwarden
```

Requirements: Python 3.12+, SQLAlchemy 2.0+.

Optional dependency groups:

| Group        | Default | Provides                             |
|--------------|---------|--------------------------------------|
| `[postgres]` |         | `psycopg2-binary`                    |
| `[mysql]`    |         | `pymysql`                            |
| `[clickhouse]` |       | `clickhouse-connect`, `aiohttp`      |
| `[dev]`      |         | `pytest`, `pytest-asyncio`, `pytest-cov`, `zensical`, `seoslug`, `httpx2`, `prometheus-client` |

## Quick start

### 1. Configure

Create a file named `dbwarden.py` in your project root:

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:pass@localhost:5432/myapp"
    database_url_async = "postgresql+asyncpg://user:pass@localhost:5432/myapp"
```

The function alternative, `database_config(...)`, remains supported. Some
plugins use it in examples or integration code.

### 2. Define your models

```python
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases import IndexSpec, TableMeta


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    bio: Mapped[str | None] = mapped_column(Text, nullable=True)

    class Meta(TableMeta):
        comment = "Core user accounts"


class Post(Base):
    __tablename__ = "posts"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    title: Mapped[str] = mapped_column(String(255), nullable=False)
    body: Mapped[str] = mapped_column(Text, nullable=False)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
    created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False)

    class Meta(TableMeta):
        indexes = [
            IndexSpec(name="ix_posts_created_at", columns=["created_at"]),
        ]
```

### 3. Generate a migration

```bash
dbwarden init
dbwarden make-migrations "create initial tables"
```

Output: both upgrade and rollback in the same file.

```sql
-- upgrade
CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    bio TEXT
);
COMMENT ON TABLE users IS 'Core user accounts';

CREATE TABLE IF NOT EXISTS posts (
    id INTEGER PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    user_id INTEGER NOT NULL REFERENCES users(id),
    created_at TIMESTAMP NOT NULL
);
CREATE INDEX IF NOT EXISTS ix_posts_created_at ON posts (created_at);

-- rollback
DROP TABLE posts;
DROP TABLE users;
```

### 4. Apply

```bash
dbwarden migrate
```

### 5. Check status

```bash
dbwarden status
```

## Typical workflow

1. Define or update your SQLAlchemy models with `class Meta` annotations
2. Run `dbwarden make-migrations` to generate SQL
3. Review the generated `.sql` file and its rollback section
4. Run `dbwarden migrate` to apply
5. Verify with `dbwarden status`

---

## Migration engine

**Model-driven compilation**: dbwarden reads your SQLAlchemy models directly. When you change a model, it diffs the new state against the last snapshot and compiles the SQL to reconcile them.

**Plain SQL output**: Compiled migrations are `.sql` files. No migration runtime, no generated Python. Review them, commit them, execute them directly against any environment.

**Rollback contract**: Generated migrations carry both upgrade and rollback sections. dbwarden emits executable rollback when it is safe, refuses placeholder rollback by default, and requires an explicit irreversible declaration when rollback cannot be produced.

**Schema snapshots**: After every migration, a checksummed JSON snapshot is written to `.dbwarden/schemas/`. Snapshots power rename detection, offline diffing, and column-level comparisons without querying the live database.

**Column-level diffing**: Type, nullability, default, and comment changes generate precise `ALTER COLUMN` statements.

**Typed `class Meta`**: The `_MetaValidator` metaclass validates every attribute on `class Meta` at import time. Typos that would have silently produced wrong DDL now raise `DBWardenConfigError` immediately.

```python
class Meta(MyTableMeta):
    my_engin = "InnoDB"  # DBWardenConfigError: unknown attr 'my_engin'
```

Supported index features:

- Partial indexes (`WHERE` clause)
- Covering indexes (`INCLUDE` columns)
- `USING` access methods
- `NULLS NOT DISTINCT` (PostgreSQL 15+)
- Per-column sort order
- Storage parameters (`WITH (fillfactor=...)`)
- ClickHouse skip indexes via `ChIndexSpec`

---

## Pre-deploy impact analysis

Before applying schema changes, dbwarden can scan your codebase to identify what will be affected. It uses AST analysis with a grep fallback, so results reflect actual code structure rather than text matches.

```bash
dbwarden check-impact 0042 --database primary
```

Output:

```
drop_column on users.username
  References: 2
    app/routes/users.py:34  attribute_access
      .username
    app/templates/profile.jinja2:12  grep
      user.username
```

Run this before any destructive deploy to surface breaking changes before they reach production.

---

## Offline migrations

Export model state once, then compile migrations on any machine without a database connection. Designed for CI pipelines and local development without a running database.

```bash
dbwarden export-models --database primary
git add .dbwarden/model_state.json
```

Then on any machine, with no database required:

```bash
dbwarden make-migrations "add bio column" --offline
```

The model state file is updated in place after each migration.

> **Important:** The model state file (`.dbwarden/model_state.*.json`) is used for offline migration generation. It is auto-generated and committed to version control. If accidentally deleted, restore it from git (`git checkout .dbwarden/model_state.*.json`) or regenerate it by running `dbwarden export-models --database <db>` against a live database. Without it, offline commands like `make-migrations --offline` will not work, but online operations are unaffected.

---

## Reverse-engineer models

Decompile a live database (PostgreSQL, MySQL, ClickHouse, SQLite) into SQLAlchemy models:

```bash
dbwarden generate-models --database primary --tables users,posts
dbwarden generate-models --database primary --base app.database:Base
```

By default each generated file declares its own `Base = declarative_base()`. Use `--base` to import a custom Base class from your project instead (e.g. `--base app.database:Base` or `--base app.database:DeclarativeBase`). The generated output includes `class Meta` blocks with all detected backend-specific metadata.

---

## Supported databases

| Database   | Round-trip | Notes                                       |
|------------|------------|---------------------------------------------|
| PostgreSQL | Full       | Primary backend, full schema fidelity       |
| MySQL      | Full       | DDL parity focus                            |
| ClickHouse | Full       | Analytics backend, MergeTree engine family  |
| SQLite     | Full       | Table rebuilds, WITHOUT ROWID/STRICT, generated columns |
| MariaDB    | No         | Schema layer complete; snapshot gaps remain |

### PostgreSQL

First-class support with full round-trip schema fidelity. Supported features include identity and generated columns, partitioning, table inheritance, exclusion constraints, deferrable constraints, advanced indexes via `PgIndexSpec`, per-column storage and collation, enum type creation, and full type normalization (SERIAL, TIMESTAMPTZ, NUMERIC, JSONB, UUID, ARRAY, TSTZRANGE).

### MySQL

Full round-trip support with `MyTableMeta` / `MyColumnMeta` and `my.field()` spec objects. Engine-level options (`my_engine`, `my_charset`, `my_collate`, `my_row_format`), column-level options (`unsigned`, `charset`, `collate`, `on_update`), and model reverse-engineering via `generate-models`.

```bash
uv add "dbwarden[mysql]"
```

### ClickHouse

First-class analytics backend support. MergeTree engine family via `ChEngineSpec`, replicated engines, projections, dictionaries, materialized views, skip indexes via `ChIndexSpec`, column codecs, `LowCardinality` and `Nullable` type wrappers.

```bash
uv add "dbwarden[clickhouse]"
```

### SQLite

Full round-trip support with `SqTableMeta` / `SqColumnMeta` and `sq.field()` spec objects. `WITHOUT ROWID` and `STRICT` tables, generated columns (`STORED` / `VIRTUAL`), per-column collation, and model reverse-engineering via `generate-models`. Changes SQLite's `ALTER TABLE` cannot express - column types, nullability, defaults, table constraints - are emitted as a table rebuild with a rebuild in the other direction as the rollback.

### MariaDB

Schema layer is complete with `MdbTableMeta` / `MdbColumnMeta` and `mdb.field()` spec objects including MariaDB-specific features (`page_compressed`, `invisible`, `without_overlaps`). Snapshot capture and reverse-engineering of MariaDB-specific features are not yet complete.

---

## Developer experience

**Dev mode**: Run SQLite locally against a PostgreSQL production schema with automatic SQL translation.

**Multi-database**: One project, multiple databases, full isolation between them. Use `model_tables` to assign table ownership per database when sharing model paths.

**Generate models**: Reverse-engineer a live database (PostgreSQL, MySQL, ClickHouse) into SQLAlchemy models with `dbwarden generate-models`.

**`dbwarden diff`**: Read-only comparison tool. Outputs as Rich table, JSON, or raw SQL. Supports `--offline` mode.

**Graceful disconnection**: Automatic retry logic and clear error messages when a database is unreachable.

---

## Official plugins

dbwarden features a plugin system with three trust tiers (official, verified, community). Official plugins extend core with features that were previously built-in, now maintained independently:

| Plugin | PyPI | Purpose |
|---|---|---|
| `dbwarden-ch-rbac` | [`dbwarden-ch-rbac`](https://pypi.org/p/dbwarden-ch-rbac) | ClickHouse RBAC: roles, users, grants, row policies, quotas, settings profiles |
| `dbwarden-fastapi` | [`dbwarden-fastapi`](https://pypi.org/p/dbwarden-fastapi) | FastAPI session dependencies, health endpoints, migration routes |
| `dbwarden-pgsql-extensions` | [`dbwarden-pgsql-extensions`](https://pypi.org/p/dbwarden-pgsql-extensions) | PostgreSQL extensions, event triggers, functions, triggers, storage parameters |
| `dbwarden-pgsql-rbac` | [`dbwarden-pgsql-rbac`](https://pypi.org/p/dbwarden-pgsql-rbac) | PostgreSQL RBAC: roles, grants, default privileges, policies |
| `dbwarden-pgsql-types` | [`dbwarden-pgsql-types`](https://pypi.org/p/dbwarden-pgsql-types) | PostgreSQL custom types: ENUMs, domains, composite types, sequences |
| `dbwarden-sandbox` | [`dbwarden-sandbox`](https://pypi.org/p/dbwarden-sandbox) | Testcontainers sandbox providers for safe migration replay |
| `dbwarden-seeds` | [`dbwarden-seeds`](https://pypi.org/p/dbwarden-seeds) | Seed data management with code seeds and file-based SQL/Python seeds |

See the [plugin documentation](plugins/) for installation, development guides, and the full Verified standard.

---

## License

MIT

---

dbwarden is built for teams that want declarative, reviewable, reproducible database changes, derived from the models they already maintain, not from migration scripts they have to write.

## Next Steps

- Start with [Features](features.md)
- Follow the guides in [Get Started](getting-started/setup.md)
- Explore [Cookbook & Examples](cookbook/index.md)
- Browse [Plugins](plugins/) to extend dbwarden's capabilities
- Use [CLI Reference](cli-reference.md) as command lookup

========================================================================
PAGE: https://docs.dbwarden.org/installation/
========================================================================

# Installation

This guide covers installing dbwarden in your project and verifying it works correctly.

## Requirements

- Python 3.12.7 or higher
- A project that uses SQLAlchemy for database models
- uv or another Python package manager

## Install using uv

```bash
uv add dbwarden
```

### Development dependencies

To also install testing and linting tools:

```bash
uv add "dbwarden[dev]"
```

### Optional dependency groups

The `[postgres]` extra is the most commonly used. Install it if you are targeting PostgreSQL.

| Group                  | Command | Provides |
|------------------------|-------|-------|
| `postgres`             | `uv add "dbwarden[postgres]"` | PostgreSQL driver (`psycopg2-binary`) |
| `mysql`                | `uv add "dbwarden[mysql]"` | MySQL/MariaDB driver (`pymysql`) |
| `clickhouse`           | `uv add "dbwarden[clickhouse]"` | ClickHouse driver (`clickhouse-connect`) |

Combine groups as needed:

```bash
uv add "dbwarden[postgres,mysql]"
```

## Database drivers

dbwarden uses SQLAlchemy under the hood. The recommended way to install drivers is via the extras above:

```bash
# PostgreSQL
uv add "dbwarden[postgres]"

# MySQL/MariaDB
uv add "dbwarden[mysql]"

# ClickHouse
uv add "dbwarden[clickhouse]"

# SQLite comes bundled with Python
```

You can also install drivers directly if you prefer:

```bash
uv add psycopg2-binary    # PostgreSQL
uv add pymysql            # MySQL / MariaDB
uv add clickhouse-connect # ClickHouse
```

## Verify installation

After installing, confirm dbwarden is available:

```bash
$ dbwarden version
```

You should see output:

```
0.18.0
```

## Initialize in your project

Create the dbwarden structure in your project directory:

```bash
$ dbwarden init
```

This creates:

- a `migrations/` directory structure
- a `dbwarden.py` config scaffold (or discovers your existing config source)

## What happens during init

When you run `init`, dbwarden:

1. Creates `migrations/` if missing
2. Creates `dbwarden.py` (or updates existing config source) with declarative class scaffolding
3. Does not overwrite existing `database_config(...)` definitions you have added

You can run `init` safely on an existing project - it is idempotent.

## Quick configuration

After init, configure your first database by editing `dbwarden.py` (or your existing config file):

```python
from dbwarden import DbwardenDatabase


class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/mydb"
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./development.db"
```

The `dev_database_*` fields are optional but recommended - they enable fast local iterations with `--dev`.
The function alternative, `database_config(...)`, remains supported, and some
plugins use it in examples or integration code.

## Verify configuration loads

```bash
$ dbwarden settings show --all
```

You should see your database entry printed with type and URL.

## Common installation issues

**Command not found after uv add**

- Ensure your virtual environment is activated
- Try removing and re-adding: `uv remove dbwarden && uv add dbwarden`

**Import errors or missing module warnings**

- Upgrade uv and reinstall: `uv add --upgrade dbwarden`

**Database driver errors**

- Install the appropriate driver for your target database (see Database drivers section above)

## Upgrading

To update to a newer version:

```bash
uv add --upgrade dbwarden
```

Or with poetry:

```bash
poetry update dbwarden
```

Check the release notes when upgrading major versions - there may be configuration or workflow changes.

========================================================================
PAGE: https://docs.dbwarden.org/migration-files/
========================================================================

# Migration File Format

Migration files are the execution contract in dbwarden.

Everything that changes your database should be represented in explicit SQL files that can be reviewed, tested, and rolled back.

## File naming and location

Versioned migrations are stored under each database migrations directory (default: `migrations/<database_name>`).

Canonical filename pattern:

```text
{database_name}__{version}_{description}.sql
```

A legacy format without the `{database_name}__` prefix (e.g. `0001_create_users_table.sql`) is also accepted for backward compatibility.

Examples:

```text
primary__0001_initial_schema.sql
primary__0002_add_users_table.sql
analytics__0001_create_events.sql
```

When a migration is auto-generated with `make-migrations`, dbwarden also writes a companion plan file:

```text
primary__0001_initial_schema.plan.json
```

That file captures machine-readable metadata for CI and debugging and is not executed by `migrate`.

Schema 1.1 plans retain legacy fields and add full typed `upgrade_ops`, `base_checksum`, `target_checksum`, `content_hash`, and `severity`. The content hash covers both SQL sections and headers, normalizing line endings. Missing, stale, inconsistent, or unmapped severity becomes UNKNOWN.

A severity split uses ordinary consecutive versions; `__deferred` is description text. Both files contain their own rollback. The deferred header names its base with `-- depends_on: ["0007"]` and records `-- dbwarden: split-from 0007 (threshold WARN)`. Both files record `-- dbwarden: file-severity LEVEL`. See [Safety-scoped migrations](correctness/safety-scoped-migrations.md).

Plugins may add named migration groups. Files in a custom group end in `__<category>.sql`; their plans include `category: {name, order, plugin}` and SQL includes `-- dbwarden: category <name>`. Groups receive consecutive versions and retain the same severity/force rules. Categories never allow skipping earlier versions. See [plugin categories](plugins/developing/object-plugins.md#safety-and-migration-categories).

## Required sections

Each migration file must define both:

```sql
-- upgrade

-- rollback
```

- `-- upgrade`: statements applied during `migrate`
- `-- rollback`: statements applied during `rollback`

If rollback is weak or incomplete, production recovery is weak or incomplete.

## Rollback contract

Generated rollback is part of the migration contract. dbwarden classifies rollback before a migration is accepted:

| Kind | Meaning | Generation behavior |
|------|---------|---------------------|
| `real` | Executable rollback SQL restores the prior schema state for the operation. | Allowed. |
| `conditional` | Executable rollback SQL is allowed only when prior state is captured and dbwarden can prove the inverse is structurally safe. | Allowed with verbose warning. |
| `irreversible` | Rollback is genuinely unsafe or impossible, such as lossy ClickHouse engine changes or PostgreSQL enum value additions. | Allowed only for known irreversible operations or explicit acknowledgement. |
| `placeholder` | Rollback is only a comment or manual instruction. | Refused by default. |

Placeholder rollback is not a working rollback. If dbwarden cannot emit executable rollback SQL, generation fails unless the migration is intentionally declared irreversible.

Explicit irreversible declaration:

```sql
-- dbwarden: irreversible
```

Use this only when the team accepts that the migration cannot be rolled back automatically. See [Rollback Coverage](correctness/rollback-coverage-matrix.md) for the backend operation matrix.

## Migration classes

dbwarden supports three execution classes:

| Prefix | Class | Behavior |
|--------|-------|----------|
| `NNNN_` | Versioned | Runs once in ordered version sequence |
| `RA__` | Runs always | Runs on every `migrate` execution |
| `ROC__` | Runs on change | Runs when checksum changed |

### When to use each

- `NNNN_`: schema evolution (tables, columns, indexes, constraints)
- `RA__`: objects that should always be refreshed (views, grants)
- `ROC__`: routines/policies that should apply only when content changes

## Execution model

At runtime, dbwarden builds an execution plan from file discovery + migration metadata:

1. read versioned files and filter already-applied versions
2. include `RA__` files
3. include changed `ROC__` files
4. execute with lock protection
5. record metadata and checksums

Conceptual plan:

```python
def build_plan(directory, applied_versions):
    versioned = parse_versioned_files(directory)
    repeatable = parse_repeatable_files(directory)
    pending_versioned = [m for m in versioned if m.version not in applied_versions]
    pending_ra = repeatable.runs_always
    pending_roc = changed_only(repeatable.runs_on_change)
    return pending_versioned + pending_ra + pending_roc
```

## Examples

### Versioned migration

```sql
-- upgrade

CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    created_at DATETIME
);

-- rollback

DROP TABLE users;
```

### Runs-always migration (`RA__`)

Filename example: `primary__RA__refresh_active_users_view.sql`

```sql
-- upgrade

CREATE OR REPLACE VIEW active_users AS
SELECT id, email FROM users WHERE is_active = TRUE;

-- rollback

DROP VIEW IF EXISTS active_users;
```

### Runs-on-change migration (`ROC__`)

Filename example: `primary__ROC__update_timestamp_trigger.sql`

```sql
-- upgrade

CREATE OR REPLACE FUNCTION update_updated_at()
RETURNS TRIGGER AS $$
BEGIN
  NEW.updated_at = NOW();
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- rollback

DROP FUNCTION IF EXISTS update_updated_at();
```

## Metadata headers

Headers are parsed from migration file comments. The `-- seed` marker is recognised
by tools; `-- depends_on` parsing is implemented but **not yet enforced** during
migration execution (migrations run in filesystem-sort order by version).

Dependency header (parsed but not enforced):

```sql
-- depends_on: ["0004", "0005"]
```

Seed marker:

```sql
-- seed
```

## Authoring guidelines

- One logical change per migration file
- Keep DDL explicit; avoid hidden application-side schema effects
- Keep rollback idempotent when possible (`IF EXISTS`, safe predicates)
- Do not commit placeholder rollback. Use executable rollback SQL or an explicit irreversible declaration.
- For data migrations, use bounded, reversible operations
- Prefer small migrations over large monolithic SQL scripts

## Review checklist

Before merge:

- upgrade section matches intended schema change
- rollback section restores prior valid state
- indexes/constraints/defaults are explicit
- no environment-specific literals accidentally committed

Before release:

```bash
$ dbwarden status --database primary
$ dbwarden migrate --database primary
$ dbwarden rollback --database primary --count 1
$ dbwarden migrate --database primary
```

========================================================================
PAGE: https://docs.dbwarden.org/models/
========================================================================

# SQLAlchemy Models Reference

This page is the **reference** for all supported Meta attributes across every backend. For a step-by-step walkthrough of defining models, see the [Modeling Guide](getting-started/modeling.md).

dbwarden reads SQLAlchemy model metadata to generate migration SQL. Use
`model_paths` on your `DbwardenDatabase` classes, or in equivalent
`database_config(...)` entries, to control discovery.

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:pass@localhost:5432/main"
    model_paths = ["app.models"]
    model_tables = ["users", "posts", "comments"]
```

## Common Meta Attributes

Every backend supports a core set of cross-database attributes via `class Meta(TableMeta)`:

### Table-level

| Attribute | Type | SQL | Backends |
|-----------|------|-----|----------|
| `comment` | `str` | `COMMENT ON TABLE t IS '...'` | All |
| `indexes` | `list[IndexSpec]` | `CREATE INDEX ...` | All |
| `checks` | `list[CheckSpec]` | `ALTER TABLE t ADD CONSTRAINT ... CHECK (...)` | All |
| `uniques` | `list[UniqueSpec]` | `ALTER TABLE t ADD CONSTRAINT ... UNIQUE (...)` | All |

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases import TableMeta, IndexSpec, CheckSpec, UniqueSpec

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    age: Mapped[int] = mapped_column(Integer)

    class Meta(TableMeta):
        comment = "Core user accounts"
        indexes = [
            IndexSpec(name="ix_users_email", columns=["email"]),
        ]
        checks = [
            CheckSpec(name="ck_users_age", expression="age >= 0"),
        ]
        uniques = [
            UniqueSpec(name="uq_users_email", columns=["email"]),
        ]
```

Typed spec classes (`IndexSpec`, `UniqueSpec`, `CheckSpec`) provide full IDE autocomplete and are the recommended way to declare model metadata. Plain dicts are also accepted for all three lists using the same field names.

### Column-level

| Attribute | Type | SQL | Backends |
|-----------|------|-----|----------|
| `comment` | `str` | `COMMENT ON COLUMN t.c IS '...'` | All |
| `public` | `bool` | Controls field visibility in `@auto_schema` (requires `dbwarden-fastapi` plugin) | All |

```python
class Meta(TableMeta):
    class internal_note:
        comment = "Internal system note"
        public = False
```

These attributes work with any `database_type`. Backend-specific subclasses (`PGTableMeta`, `MyTableMeta`, `CHTableMeta`) inherit all common attributes and add their own.

### Typed Spec Classes (Recommended)

Use `IndexSpec`, `UniqueSpec`, and `CheckSpec` from `dbwarden.databases` for cross-backend model metadata. These provide full IDE autocomplete and type checking:

```python
from dbwarden.databases import IndexSpec, UniqueSpec, CheckSpec

class Meta(TableMeta):
    indexes = [
        IndexSpec(name="ix_users_email", columns=["email"]),
    ]
    checks = [
        CheckSpec(name="ck_users_age", expression="age >= 0"),
    ]
    uniques = [
        UniqueSpec(name="uq_users_email", columns=["email"]),
    ]
```

Plain dicts using the same field names (`name`, `columns`, `expression`, `sql`) are also accepted for backwards compatibility, but typed specs are the idiomatic form.

### Column-Level Meta Base Class

For IDE autocomplete on column-level inner classes, use `PGColumnMeta` for PostgreSQL, `MyColumnMeta` for MySQL, `MdbColumnMeta` for MariaDB, or `CHColumnMeta` for ClickHouse. Each defines the cross-database attributes (`comment`, `public`) alongside its backend-specific spec object (`pg`, `ch`, `my`, `mdb`, `sq`). There is no shared `FieldMeta` base: the `*FieldMeta` classes were removed and their fields inlined into each backend's column Meta.

```python
from dbwarden.databases.pgsql import PGColumnMeta, pg

# Use typed spec objects for backend-specific column attributes:
#   pg = pg.field(collation=..., storage=..., ...)
#   ch = ch.field(codec=..., nullable=..., ...)
```

Backend-specific options are always set via a typed spec object attribute, never as flat attributes. For example, use `pg = pg.field(collation="en_US.UTF-8")` instead of the old `pg_collation = "en_US.UTF-8"`.

### Backend Subpackages

dbwarden organizes backend-specific types into subpackages under `dbwarden.databases`, also available there as short aliases:

| Alias | Subpackage | Key types |
|-------|------------|-----------|
| `pg` | `dbwarden.databases.pgsql` | `PgFieldSpec`, `PgIndexSpec`, `PgTableSpec` |
| `ch` | `dbwarden.databases.clickhouse` | `ChFieldSpec`, `ChIndexSpec`, `ChTableSpec` |
| `my` | `dbwarden.databases.mysql` | `MyFieldSpec`, `MyTableSpec` |
| `mdb` | `dbwarden.databases.mariadb` | `MdbFieldSpec`, `MdbTableSpec` |
| `sq` | `dbwarden.databases.sqlite` | `SqFieldSpec`, `SqTableSpec`, `SqTableMeta`, `SqColumnMeta` |

Only `IndexSpec`, `PgIndexSpec`, and `ChIndexSpec` exist as typed index spec classes. MySQL, MariaDB, and SQLite use the base `IndexSpec` with the `indexes` attribute or plain dicts in their backend-specific index list (`my_indexes`, `sq_indexes`).

```python
from dbwarden.databases.pgsql import pg
from dbwarden.databases.clickhouse import ch
from dbwarden.databases.mysql import my
from dbwarden.databases.mariadb import mdb
from dbwarden.databases.sqlite import sq

# Use pg.field(), ch.field() for column-level metadata
pg_spec = pg.field(collation="en_US.UTF-8", storage="PLAIN")
ch_spec = ch.field(codec="ZSTD(3)", nullable=True)
```

## PostgreSQL Model Metadata

When `database_type="postgresql"`, dbwarden supports first-class PostgreSQL metadata via `class Meta(PGTableMeta)` inner classes. This is the **only** supported surface: `mapped_column(info=...)` raises `DBWardenConfigError`.

### Table-Level Meta

Inherit from `PGTableMeta` on your `class Meta`:

```python
from sqlalchemy import Integer
from sqlalchemy.orm import DeclarativeBase,     Mapped, mapped_column
from dbwarden.databases.pgsql import PGTableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)

    class Meta(PGTableMeta):
        pg_fillfactor = 80
        pg_tablespace = "fastspace"
        pg_storage_params = {
            "fillfactor": 80,
            "autovacuum_enabled": "false",
        }
```

`PGTableMeta` inherits all common `TableMeta` attributes (`comment`, `indexes`, `checks`, `uniques`) and adds PostgreSQL-specific ones (`pg_schema`, `pg_fillfactor`, `pg_tablespace`, `pg_storage_params`, `pg_unlogged`, `pg_partition`, `pg_inherits`, `pg_excludes`, `pg_indexes`, `pg_checks`, `pg_uniques`).

For PostgreSQL-specific indexes, use `PgIndexSpec` in `pg_indexes`:

```python
from dbwarden.databases.pgsql import PgIndexSpec

class Meta(PGTableMeta):
    pg_indexes = [
        PgIndexSpec("ix_users_email", ["email"],
            unique=True, using="gin"),
    ]
```

`PgIndexSpec` supports operator classes via `postgresql_ops` for GIN indexes on JSONB columns:

```python
PgIndexSpec("ix_users_data", ["data"],
    using="gin",
    postgresql_ops={"data": "jsonb_path_ops"})
```

This generates `CREATE INDEX ... ON users USING GIN (data jsonb_path_ops)`.

Full `PgIndexSpec` constructor fields: `name`, `columns`, `unique`, `using`, `where`, `include`, `with_params`, `tablespace`, `nulls_not_distinct`, `column_sorting`, `postgresql_ops`, `concurrently`. See [Indexes](databases/postgresql/indexes.md) for indexing options and [Declaring Metadata](databases/postgresql/declaring-metadata.md) for the full `PGTableMeta` attribute list.

### Column-Level Meta

Use `PGColumnMeta` inner classes named after the column. Use `pg = pg.field(...)` to set column-level options:

```python
from sqlalchemy import Integer, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.pgsql import PGTableMeta, PGColumnMeta, pg

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    bio: Mapped[str] = mapped_column(Text)

    class Meta(PGTableMeta):
        class id(PGColumnMeta):
            pg = pg.field(identity="always", identity_start=100)

        class bio(PGColumnMeta):
            pg = pg.field(storage="EXTENDED", collation="en_US.UTF-8")
```

`PGColumnMeta` includes the common `comment` and `public` attributes plus a `pg` attribute of type `PgFieldSpec` that bundles all PostgreSQL-specific column options.

For the full list of supported attributes, see [Tables & Columns](databases/postgresql/tables-and-columns.md) for column handler details and [Declaring Metadata](databases/postgresql/declaring-metadata.md) for PGColumnMeta options.

### PostgreSQL Views

Use `PGViewMeta` for models that represent PostgreSQL views or materialized views. Set `__tablename__` to the view name:

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.pgsql import PGViewMeta

class Base(DeclarativeBase):
    pass

class ActiveUser(Base):
    __tablename__ = "active_users"

    id: Mapped[int] = mapped_column(Integer)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(PGViewMeta):
        pg_view_query = "SELECT id, email FROM users WHERE active = true"
        pg_view_materialized = False
```

`PGViewMeta` attributes:

| Attribute | Type | Default | SQL |
|-----------|------|---------|-----|
| `pg_view_query` | `str` | `None` | `AS <query>` in CREATE VIEW |
| `pg_view_materialized` | `bool` | `False` | `CREATE MATERIALIZED VIEW` instead of `CREATE VIEW` |
| `pg_view_auto_refresh` | `bool` | `False` | Emits `REFRESH MATERIALIZED VIEW` in subsequent migrations |
| `pg_schema` | `str \| None` | `None` | Fully qualifies the view as `schema.view_name` |

Set `pg_view_auto_refresh = True` to keep a materialized view current after each schema change:

```python
class Meta(PGViewMeta):
    pg_view_query = "SELECT user_id, count(*) AS total FROM orders GROUP BY user_id"
    pg_view_materialized = True
    pg_view_auto_refresh = True
```

Each `make-migrations` run after the initial CREATE produces a `REFRESH MATERIALIZED VIEW` statement.

See [Views](databases/postgresql/views.md) for the full view lifecycle and auto-refresh configuration.

## ClickHouse Model Metadata

When `database_type="clickhouse"`, dbwarden supports first-class ClickHouse metadata via `class Meta(CHTableMeta)` inner classes. This is the **only** supported surface. Pass options via `mapped_column(info=...)` raises `DBWardenConfigError`.

### Table-Level Meta

Inherit from `CHTableMeta` on your `class Meta`:

```python
from datetime import date
from sqlalchemy.orm import DeclarativeBase,     Mapped, mapped_column
from dbwarden.databases.clickhouse import CHTableMeta, ChEngineSpec

class Base(DeclarativeBase):
    pass

class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(Int64, primary_key=True)
    event_date: Mapped[date] = mapped_column(Date)
    payload: Mapped[str] = mapped_column(String)

    class Meta(CHTableMeta):
        ch_engine = ChEngineSpec("ReplacingMergeTree", args=("version_column",))
        ch_order_by = ["region", "event_time"]
        ch_primary_key = "region"
        ch_partition_by = "toYYYYMM(event_time)"
        ch_sample_by = "intHash64(user_id)"
        ch_ttl = [
            "event_time + INTERVAL 1 MONTH DELETE",
            "event_time + INTERVAL 1 YEAR TO DISK 'cold'",
        ]
        ch_settings = {"index_granularity": "8192"}
```

`CHTableMeta` inherits all common `TableMeta` attributes (`comment`, `indexes`, `checks`, `uniques`) and adds ClickHouse-specific ones (`ch_engine`, `ch_order_by`, `ch_primary_key`, `ch_partition_by`, `ch_sample_by`, `ch_ttl`, `ch_settings`, `ch_object_type`, `ch_select_statement`, `ch_to_table`, `ch_dictionary`, `ch_dict_layout`, `ch_dict_source`, `ch_dict_lifetime`, `ch_dict_primary_key`, `ch_projections`, `ch_zookeeper_path`, `ch_replica_name`).

For the full list of supported attributes, see [ClickHouse Deep Dive](databases/clickhouse/index.md).

### Column-Level Meta

Use `CHColumnMeta` inner classes named after the column. Use `ch = ch.field(...)` to set column-level options:

```python
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.clickhouse import CHTableMeta, CHColumnMeta, ChEngineSpec, ch

class Base(DeclarativeBase):
    pass

class Event(Base):
    __tablename__ = "events"

    id: Mapped[int] = mapped_column(Int64, primary_key=True)
    payload: Mapped[str] = mapped_column(String)
    tags: Mapped[list[str]] = mapped_column(ARRAY(String))

    class Meta(CHTableMeta):
        ch_engine = ChEngineSpec("MergeTree")
        ch_order_by = "event_time"

        class payload(CHColumnMeta):
            ch = ch.field(codec="ZSTD(3)", nullable=False)

        class tags(CHColumnMeta):
            ch = ch.field(low_cardinality=True)
```

`CHColumnMeta` includes the common `comment` and `public` attributes plus a `ch` attribute of type `ChFieldSpec` that bundles all ClickHouse-specific column options.

### Engine Spec

Use `ChEngineSpec` for the table engine:

```python
from dbwarden.databases.clickhouse import ChEngineSpec

# Simple engine
ch_engine = ChEngineSpec("MergeTree")

# Engine with arguments
ch_engine = ChEngineSpec("ReplacingMergeTree", args=("version_column",))

# Replicated engine
ch_engine = ChEngineSpec("ReplicatedMergeTree",
    zookeeper_path="/clickhouse/tables/shard1/events",
    replica_name="{replica}")

# Distributed engine with settings
ch_engine = ChEngineSpec("Distributed",
    args=("cluster", "db", "events", "rand()"),
    settings={"insert_distributed_sync": "1"})
```

For replicated engines, `ch_zookeeper_path` and `ch_replica_name` are injected as the first two engine arguments. If `args` contains existing positional arguments, they come after the ZooKeeper path and replica name.

### Projections

Use `ProjectionSpec` in `ch_projections`:

```python
from dbwarden.databases.clickhouse import ProjectionSpec

class Meta(CHTableMeta):
    ch_order_by = ["author", "created_at"]
    ch_projections = [
        ProjectionSpec("by_author", "SELECT * ORDER BY author"),
        ProjectionSpec("daily_stats",
            "SELECT toDate(created_at) AS day, count() GROUP BY day"),
    ]
```

Current behavior:
- projection definitions are rendered into generated ClickHouse DDL
- safety checks classify added projections as `INFO`
- removed projections are classified as `WARNING`

### Skip Indexes

Use `ChIndexSpec` in `ch_indexes`:

```python
from dbwarden.databases.clickhouse import ChIndexSpec

class Meta(CHTableMeta):
    ch_indexes = [
        ChIndexSpec("ix_payload", ["payload"],
            type="bloom_filter", granularity=1),
    ]
```

### Materialized Views

Use `materialized_view()` builder with `CHViewMeta`:

```python
from dbwarden.databases.clickhouse import CHViewMeta, materialized_view

class EventRollup(Base):
    __tablename__ = "event_rollup_mv"

    event_date: Mapped[date] = mapped_column(Date)
    total: Mapped[int] = mapped_column(Int64)

    class Meta(CHViewMeta):
        ch = materialized_view(
            select=(
                "SELECT toDate(event_time) AS event_date, count() AS total "
                "FROM events GROUP BY event_date"
            ),
            to="mv_target",
        )
```

Two storage modes:

| Shape | `to` | `engine` / `order_by` |
|-------|-----------|----------------------|
| Explicit target | Set | Not needed (target owns storage) |
| Implicit `.inner` | `None` | Required on `materialized_view()` |

Expression fields (select, order_by, partition_by, ttl) accept
SQLAlchemy `ColumnElement`, `ChRaw`, or plain `str`.

### Dictionaries

ClickHouse dictionaries use `ch_dictionary = True` with related `ch_dict_*` fields:

```python
class CountryCode(Base):
    __tablename__ = "country_codes"

    code: Mapped[str] = mapped_column(String)
    name: Mapped[str] = mapped_column(String)

    class Meta(CHTableMeta):
        ch_dictionary = True
        ch_dict_layout = "FLAT()"
        ch_dict_source = "CLICKHOUSE(HOST 'localhost' TABLE 'countries')"
        ch_dict_lifetime = "MIN 0 MAX 3600"
        ch_dict_primary_key = "code"
```

Required fields when `ch_dictionary = True`:

| Field | Description | Example |
|-------|-------------|---------|
| `ch_dict_layout` | Dictionary layout | `"FLAT()"`, `"COMPLEX_KEY_HASHED()"` |
| `ch_dict_source` | Source configuration | `"CLICKHOUSE(HOST '...' TABLE '...')"` |
| `ch_dict_lifetime` | Cache lifetime | `"MIN 0 MAX 3600"` or `3600` |

Optional field:

| Field | Description | Default |
|-------|-------------|---------|
| `ch_dict_primary_key` | Primary key expression | First column |

Column types render as CH-native types (`Int64`, `String`).

### Column Hints

Use `CHColumnMeta` inner classes for per-column hints instead of `info={}`:

```python
from dbwarden.databases.clickhouse import ch

class Meta(CHTableMeta):
    class payload(CHColumnMeta):
        ch = ch.field(codec="ZSTD(3)", low_cardinality=True, nullable=False)
```

Supported `ch.field()` options:

| Keyword | Type | Description | Example |
|---------|------|-------------|---------|
| `codec` | `str` | Compression codec | `"ZSTD(3)"` |
| `default_expression` | `str` | Default value expression | `"now()"` |
| `materialized` | `str` | Materialized expression | `"lower(name)"` |
| `alias` | `str` | Alias expression | `"concat(a, b)"` |
| `ttl` | `str` | Column TTL expression | `"event_time + INTERVAL 1 YEAR"` |
| `low_cardinality` | `bool` | Wrap type in LowCardinality | `True` |
| `nullable` | `bool` | Wrap type in Nullable | `True` |

## MySQL Model Metadata

When `database_type="mysql"` (or `"mariadb"`), dbwarden supports first-class MySQL metadata via `class Meta(MyTableMeta)` inner classes. This is the **only** supported surface: `mapped_column(info=...)` raises `DBWardenConfigError`.

### Table-Level Meta

Inherit from `MyTableMeta` on your `class Meta`:

```python
from sqlalchemy import Integer, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))

    class Meta(MyTableMeta):
        my_engine = "InnoDB"
        my_charset = "utf8mb4"
        my_collate = "utf8mb4_unicode_ci"
        my_row_format = "DYNAMIC"
        my_auto_increment = 1000
        comment = "Core user accounts"
```

`MyTableMeta` inherits all common `TableMeta` attributes (`comment`, `indexes`, `checks`, `uniques`) and adds MySQL-specific ones (`my_engine`, `my_charset`, `my_collate`, `my_row_format`, `my_auto_increment`).

For MariaDB, use `MdbTableMeta` which extends `MyTableMeta` with `mdb_page_compressed` and `mdb_page_compression_level`.

### Column-Level Meta

Use `MyColumnMeta` inner classes named after the column. Use `my = my.field(...)` to set column-level MySQL options:

```python
from sqlalchemy import Integer, String, TIMESTAMP
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from dbwarden.databases.mysql import MyTableMeta, MyColumnMeta, my

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    email: Mapped[str] = mapped_column(String(255))
    updated_at: Mapped[str] = mapped_column(TIMESTAMP)

    class Meta(MyTableMeta):
        class id(MyColumnMeta):
            comment = "Primary key"
            my = my.field(unsigned=True)

        class email(MyColumnMeta):
            my = my.field(charset="utf8mb4", collate="utf8mb4_unicode_ci")

        class updated_at(MyColumnMeta):
            my = my.field(on_update="CURRENT_TIMESTAMP")
```

Supported `my.field()` options:

| Keyword | Type | Description | Example |
|---------|------|-------------|---------|
| `unsigned` | `bool` | `UNSIGNED` on integer columns | `unsigned=True` |
| `charset` | `str` | Per-column character set | `charset="utf8mb4"` |
| `collate` | `str` | Per-column collation | `collate="utf8mb4_unicode_ci"` |
| `on_update` | `str` | `ON UPDATE` expression (typically for TIMESTAMP) | `on_update="CURRENT_TIMESTAMP"` |

For MariaDB, use `MdbColumnMeta` and `mdb.field()` which extends `my.field()` with `invisible` and `sequence` options.

Cross-backend column attributes (`comment`, `public`) are set directly on the inner class, not on the spec object.

See [Type Mapping](databases/postgresql/type-mapping.md) for the full SQLAlchemy-to-PostgreSQL type normalization table.

========================================================================
PAGE: https://docs.dbwarden.org/observability/
========================================================================

# Observability

dbwarden provides Prometheus metrics and structured JSON logging for monitoring and debugging.

## Prometheus metrics

### Installation

Install the optional metrics dependency:

```bash
uv add "dbwarden[metrics]"
```

This installs `prometheus-client` which is required for metric collection and exposition.

### Enabling metrics

Set the `DBWARDEN_METRICS` environment variable to `true`:

```bash
export DBWARDEN_METRICS=true
```

When enabled, dbwarden instruments the `migrate` and `seed apply` commands with Prometheus metric recording. When disabled (or when `prometheus_client` is not installed), all metric functions are safe no-ops.

### Available metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `dbwarden_migrations_total` | Counter | `database`, `version`, `success` | Total migrations applied |
| `dbwarden_migration_duration_seconds` | Histogram | `database`, `version` | Duration of migration operations |
| `dbwarden_schema_version` | Gauge | `database` | Current schema version |
| `dbwarden_seed_version` | Gauge | `database` | Current seed version |
| `dbwarden_migrations_pending` | Gauge | `database` | Number of pending migrations |
| `dbwarden_migration_errors_total` | Counter | `database` | Total migration errors |

### FastAPI metrics endpoint

The `MetricsRouter` exposes a `GET /metrics` endpoint in Prometheus text format:

```python
from fastapi import FastAPI
from dbwarden_fastapi import MetricsRouter

app = FastAPI()
app.include_router(MetricsRouter(), prefix="/metrics")
```

The endpoint returns:

```
# HELP dbwarden_migrations_pending Number of pending migrations
# TYPE dbwarden_migrations_pending gauge
dbwarden_migrations_pending{database="primary"} 0
# HELP dbwarden_schema_version Current schema version
# TYPE dbwarden_schema_version gauge
dbwarden_schema_version{database="primary"} 5.0
```

Only active when `prometheus_client` is installed and `DBWARDEN_METRICS=true` is set. Returns 404 when disabled.

### MetricsMiddleware

The `MetricsMiddleware` is an ASGI middleware that refreshes pending-migration gauges on each HTTP request:

```python
from fastapi import FastAPI
from dbwarden_fastapi import MetricsMiddleware, MetricsRouter

app = FastAPI()
app.add_middleware(MetricsMiddleware)
app.include_router(MetricsRouter(), prefix="/metrics")
```

The middleware also records HTTP request duration via the migration duration histogram.

## JSON logging

dbwarden supports structured JSON logging for integration with log aggregation systems (ELK, Loki, Datadog, etc.).

### Enabling JSON logging

Set the `DBWARDEN_LOG_JSON` environment variable to `true`:

```bash
export DBWARDEN_LOG_JSON=true
```

For one-off CLI runs, the global `--json` flag has the same effect on the
command's log output (and also switches display commands to JSON):

```bash
dbwarden --json migrate --database primary
```

When enabled, all dbwarden log output uses newline-delimited JSON format:

```json
{"timestamp": "2025-06-01T10:00:00.123456", "level": "INFO", "logger": "dbwarden", "message": "Applying migration 0003", "db_name": "primary", "db_type": "postgresql"}
{"timestamp": "2025-06-01T10:00:01.234567", "level": "INFO", "logger": "dbwarden", "message": "Migration 0003 applied successfully", "db_name": "primary", "db_type": "postgresql"}
```

### JSON log fields

| Field | Description |
|-------|-------------|
| `timestamp` | ISO-8601 timestamp with microseconds |
| `level` | Log level (DEBUG, INFO, WARNING, ERROR) |
| `logger` | Logger name |
| `message` | Log message text |
| `db_name` | Database name (when applicable) |
| `db_type` | Database type (when applicable) |
| `exception` | Exception traceback (when applicable) |

## TRACE logging

The `trace` level (numeric `5`, below `debug`) is enabled with
`--debug-level trace` and logs every SQL statement as it executes during
`migrate`, `rollback`, and `downgrade`:

```bash
dbwarden --debug-level trace migrate --database primary
```

Each statement is logged as `SQL (2.1ms): CREATE TABLE ...` and respects the
current log format (plain text or JSON).

## Migration phase timing

Migration commands always log phase timings at INFO level: lock acquisition,
schema snapshot write, and model state write. Add `--perf` to also log each
individual SQL statement with its duration:

```bash
dbwarden migrate --perf --database primary
```

```bash
dbwarden make-migrations "add column" --perf
```

## Environment variables reference

| Variable | Value | Effect |
|----------|-------|--------|
| `DBWARDEN_METRICS` | `true` | Enable Prometheus metric recording and exposition |
| `DBWARDEN_LOG_JSON` | `true` | Enable JSON-formatted log output |
| `DBWARDEN_MIGRATE_AUTH` | API key string | Require `X-API-Key` header for `POST /migrate` endpoint |
| `DBWARDEN_HEALTH_AUTH` | API key string | Require `X-API-Key` header for health endpoints |

See also: [Cookbook: Observability](cookbook/11-observability.md)

========================================================================
PAGE: https://docs.dbwarden.org/plugins/developing/object-plugins/
========================================================================

# Object Plugins

An object plugin adds a database object **type** to dbwarden's schema convergence pipeline. Core drives your handler through the same extract → canonicalize → diff → emit flow it uses for tables, so the object is diffed against the live database and emitted in the right SQL order.

## Scope Test

An object plugin is the right tool when the feature is **database state declared by the model (or config) and converged by dbwarden**, not a one-off script. Good candidates: extensions, roles, grants, policies, triggers, functions, sequences, and backend-specific declarative objects. If the thing you want isn't schema state dbwarden should diff and emit, use a value hook or a migration instead.

## Safety and Migration Categories

`Op` and `MigrationStatement` accept optional `safety` and `category` keywords. Plugins must classify custom operation kinds with `SAFE`, `INFO`, `WARN`, or `CRITICAL`; missing classifications stop generation. Core classifications remain a minimum: a plugin cannot lower a known operation's severity. If both operation and statement declare safety, the highest declaration wins. The generated plan records the plugin identity and classification, so deployment does not require importing the original plugin to interpret an existing plan.

```python
def setup(registrar):
    registrar.register_migration_category("cleanup", order=200)
    registrar.register_object_handler(MyHandler())

# In diff(): each independently deployable change is a separate Op.
Op("create_custom_view", {"name": "report"}, category="base", safety="INFO")
Op("replace_custom_view", {"name": "legacy"}, category="deferred", safety="WARN")
Op("remove_custom_view", {"name": "old"}, category="cleanup", safety="CRITICAL")

# Or set the same fields on emitted statements:
MigrationStatement(
    order=self.statement_order,
    upgrade_sql="CREATE VIEW report AS SELECT 1 AS value;",
    rollback_sql="DROP VIEW report;",
    category="base",
    safety="INFO",
)
```

Built-in groups are `base` (order 0) and `deferred` (order 100). Custom names must match `[a-z][a-z0-9_]{0,47}` and have a unique positive integer order other than 100. `base`, `deferred`, and `unsplit` are reserved. Registration is idempotent only for the same plugin, name, and order; conflicting registrations fail.

Generation writes one versioned SQL/plan pair per nonempty group, in group order. Custom filenames end in `__<category>.sql`. Every plan carries its actual group, plugin owner, order, typed operations, and consecutive state checksums. Dependencies move operations into later groups when necessary. `--split-at-severity` also moves high-severity operations to at least `deferred`; selecting `base` cannot bypass it. Execution still follows the version prefix and `--max-severity` ceiling.

An operation remains atomic. All statements emitted for that operation must agree on category; split independently deployable statements into separate operations. Statement category overrides the operation's default. Use explicit `id` and `depends_on` operation attributes for plugin dependencies core cannot infer. Repeatables accept unsplit/base operations only. Generated repeatable SQL must be safe to rerun; category and safety declarations do not make SQL idempotent.

Inspect registrations with `dbwarden plugin categories --format json`, or `dbwarden plugin info <distribution> --load --format json`.

## The `ObjectHandler` Contract

A handler is any object exposing these attributes and methods (see `dbwarden.engine.core.protocol.ObjectHandler`):

```python
object_type: str                 # unique registration key, e.g. "pg_extension"
op_types: tuple[str, ...]        # op names this handler emits, e.g. ("create_pg_extension", ...)
run_phase: RunPhase              # PREAMBLE or DIFF
ordering: OrderingConstraint     # public anchors (optional)

def extract(self, snapshot) -> dict: ...           # DB snapshot -> current spec
def model_spec_from_config(self, config) -> dict: ...  # config -> desired spec
def model_spec_from_tables(self, model_tables) -> dict: ...  # models -> desired spec
def canonicalize(self, spec) -> dict: ...          # normalize for comparison
def diff(self, snap_spec, model_spec) -> tuple[list[Op], list[Op]]: ...  # (upgrade, rollback)
def emit(self, op, db_name=None, **kwargs) -> list[MigrationStatement]: ...  # Op -> SQL
```

## Step By Step: A `CREATE EXTENSION` Handler

The goal: converge the PostgreSQL extensions declared in config (`config.pg_extensions`) with those present in the database. This mirrors the real [`dbwarden-pgsql-extensions`](https://github.com/dbwarden-org/dbwarden-pgsql-extensions) `PgExtensionHandler`.

### 1. Import the public types

```python
from dbwarden.engine.core import (
    Anchor,
    MigrationStatement,
    Op,
    OrderingConstraint,
    RunPhase,
)
```

### 2. Declare identity, phase, and ordering

Extensions must exist before anything that uses them, so pin the handler between `PREAMBLE` and `BEFORE_TABLES`:

```python
class PgExtensionHandler:
    object_type = "pg_extension"
    op_types = ("create_pg_extension", "drop_pg_extension")
    run_phase = RunPhase.PREAMBLE
    ordering = OrderingConstraint(
        after=(Anchor.PREAMBLE,),
        before=(Anchor.BEFORE_TABLES,),
    )
```

### 3. Read current state from the snapshot

```python
    def extract(self, snapshot):
        return snapshot.get("pg_extensions", {})
```

### 4. Read desired state

```python
    def model_spec_from_config(self, config):
        return {name: {} for name in getattr(config, "pg_extensions", [])}

    def model_spec_from_tables(self, model_tables):
        return {}  # extensions come from config, not model tables
```

### 5. Canonicalize and diff

`canonicalize` normalizes both sides so comparison is apples-to-apples (the real handler lowercases names and sorts). `diff` returns `(upgrade_ops, rollback_ops)`, emitting a create for what's missing and a drop for what's extra, with the inverse recorded for rollback:

```python
    def canonicalize(self, spec):
        return {str(n).lower(): dict(v or {}) for n, v in sorted((spec or {}).items())}

    def diff(self, snap_spec, model_spec):
        upgrade_ops, rollback_ops = [], []
        snap, model = snap_spec or {}, model_spec or {}
        for name in sorted(set(model) - set(snap)):        # missing -> create
            attrs = {"name": name}
            upgrade_ops.append(Op("create_pg_extension", attrs, attrs))
            rollback_ops.insert(0, Op("drop_pg_extension", attrs, attrs))
        for name in sorted(set(snap) - set(model)):        # extra -> drop
            attrs = {"name": name}
            upgrade_ops.append(Op("drop_pg_extension", attrs, attrs))
            rollback_ops.insert(0, Op("create_pg_extension", attrs, attrs))
        return upgrade_ops, rollback_ops
```

### 6. Emit SQL

`emit` turns each `Op` into ordered SQL, branching on `op.object_type`. Use `self.statement_order`, which dbwarden sets from your `ordering` anchors (see below):

```python
    def emit(self, op, db_name=None, **kwargs):
        name = '"' + str(op.upgrade_attrs["name"]).replace('"', '""') + '"'
        if op.object_type == "create_pg_extension":
            return [MigrationStatement(
                order=self.statement_order,
                upgrade_sql=f"CREATE EXTENSION IF NOT EXISTS {name};",
                rollback_sql=f"DROP EXTENSION IF EXISTS {name};",
            )]
        if op.object_type == "drop_pg_extension":
            return [MigrationStatement(
                order=self.statement_order,
                upgrade_sql=f"DROP EXTENSION IF EXISTS {name};",
                rollback_sql=f"CREATE EXTENSION IF NOT EXISTS {name};",
            )]
        return []
```

### 7. Register it

`setup` lives in the package `__init__.py` and imports the handler module lazily:

```python
# src/dbwarden_example/__init__.py
def setup(registrar) -> None:
    from dbwarden_example.handler import PgExtensionHandler

    registrar.register_object_handler(PgExtensionHandler())
```

### 8. Declare your config keys

If your handler is driven by `database_config(...)` keys, register them too. A plugin owns its config keys, not just its handlers.

```python
CONFIG_KEYS = ("pg_extensions",)


def setup(registrar) -> None:
    from dbwarden_example.handler import PgExtensionHandler

    registrar.register_object_handler(PgExtensionHandler())
    registrar.register_config_key(*CONFIG_KEYS)
```

Core validates every keyword argument passed to `database_config(...)` against the registered keys. A key no plugin owns is rejected as an unknown argument, and a key belonging to a plugin that is not installed raises `DBWardenConfigError` naming the plugin to install. Both fire when `dbwarden.py` loads, so a missing plugin surfaces immediately rather than as a migration that silently produces nothing.

Read the config values off the `config` object your handler already receives, using the key name directly:

```python
extensions = getattr(config, "pg_extensions", []) or []
```

To stay loadable against cores that predate the config-key registry, guard the call:

```python
    register_config_key = getattr(registrar, "register_config_key", None)
    if register_config_key is not None:
        register_config_key(*CONFIG_KEYS)
```

## Ordering And The DAG

dbwarden orders all object handlers into a single directed acyclic graph, then emits their statements in that order.

- **Anchors** place a handler relative to core milestones. At registration, dbwarden validates your `OrderingConstraint` and derives a private `statement_order` from the anchors: you never touch the integers.
- **Object-to-object** constraints (`after_object`, `before_object`) place your handler relative to *other* handlers by `object_type`:

```python
ordering = OrderingConstraint(after_object=("role",))  # emit after the "role" handler
```

Anchors are the **public contract**: `PREAMBLE`, `BEFORE_TABLES`, `AFTER_TABLES`, `AFTER_CONSTRAINTS`, `AFTER_INDEXES`, `POSTAMBLE`. See [ordering anchors](../reference/ordering-anchors.md) for the full map and failure modes (unknown references, cycles, impossible pairs).

## Shared SQL Helpers

Emitting SQL that matches core's byte-for-byte matters: a plugin that quotes identifiers differently, or that skips `ON CLUSTER`, produces migrations that disagree with the ones core generates for neighbouring objects. Rather than have you copy those rules, the shared implementations are public in `dbwarden.engine.core.plugin_api`:

| Name | Use |
|---|---|
| `quote_pg(name)` | Quote a PostgreSQL identifier the way core does (reserved words only). |
| `qualified_name(name, schema)` | Render `schema.name`, or bare `name` when unqualified. |
| `build_create_policy_sql`, `build_alter_policy_sql` | Render row-level-security policy DDL. |
| `build_grant_sql`, `build_revoke_sql` | Render `GRANT` / `REVOKE`. |
| `emit_with_cluster`, `ClusterableStatement` | Wrap ClickHouse DDL so it honours the configured `ON CLUSTER`. |
| `strip_secret_values`, `has_visible_secrets`, `REDACTED` | Redact secret values before they reach a snapshot. |

The module sits under `dbwarden.engine.core`, so importing from it satisfies the public-API check in the [Verified standard](verified-standard.md).

## Conflicts And Overrides

`object_type` is the registration key. Two *different* plugins registering the same `object_type` raises `ObjectHandlerConflictError`. A plugin handler and a core handler with the same type is allowed: the plugin handler overrides core, which is how official plugins replace built-in fallbacks (e.g. the core `CREATE EXTENSION` preamble).

Overriding is a real transfer of responsibility, so dbwarden announces it rather than swapping silently:

```
WARNING  dbwarden.registry - Plugin 'dbwarden-pgsql-rbac' overrides the built-in
handler for object type 'role'. Migration SQL for 'role' now comes from the
plugin, not dbwarden core.
```

The warning is emitted once per plugin and object type per run. If you see one you did not expect, a plugin has taken over DDL generation for that object type, and the SQL in your migrations for it is the plugin's, not core's. Plugins requesting the Verified tier must declare their overrides in the verification issue.

## Tests

Object plugins run the value-plugin conformance checks plus two handler-specific ones from the shared harness (`dbwarden.plugin_conformance`); see the [Verified standard](verified-standard.md).

- `test_object_handler_conformance`: `assert_object_handler_conformance(handler, config=...)` exercises `extract`, `canonicalize`, `diff`, and `emit` against a minimal fixture and validates the returned types and SQL shape.
- `test_ordering_constraint_satisfiable`: `assert_ordering_constraint_satisfiable(handler)` confirms the constraint is not statically impossible (no impossible anchor pair; no unknown/cyclic object references).

========================================================================
PAGE: https://docs.dbwarden.org/plugins/developing/overview/
========================================================================

# Plugin Development Overview

A dbwarden plugin is a Python package with one entry point in the `dbwarden.plugins` group. The entry point points to a `setup(registry)` callable that registers hooks or object handlers.

```toml
[project.entry-points."dbwarden.plugins"]
example = "dbwarden_example:setup"
```

`setup` is defined in the package's `__init__.py`, so the entry point is `dbwarden_example:setup` (not `...plugin:setup`). Hook functions live at module top level, and heavy or dbwarden-specific imports go inside the function body so importing the package stays side-effect-free:

```python
# src/dbwarden_example/__init__.py
import importlib.util
import sys
from pathlib import Path
from typing import Any


def load_model_module(path: Path, base_dir: Path) -> Any:
    spec = importlib.util.spec_from_file_location(Path(path).stem, path)  # deferred/lazy work
    module = importlib.util.module_from_spec(spec)
    sys.modules[spec.name] = module
    spec.loader.exec_module(module)
    return module


def setup(registrar) -> None:
    registrar.register("load_model_module", load_model_module)
```

## Two Kinds Of Plugin

- **[Value plugins](value-plugins.md)** supply a value at a named hook point: session factories, FastAPI routes, lifespans, seed commands, module loaders. Registered with `registry.register(name, callable)`.
- **[Object plugins](object-plugins.md)** add a database object type to the schema diff pipeline: extensions, roles, grants, triggers, types. Registered with `registry.register_object_handler(handler)`.

A single package may register both.

## Lifecycle

1. dbwarden discovers entry-point metadata (name, version, value) without importing your code.
2. dbwarden classifies the distribution into a trust tier by name.
3. If trust rules allow loading, dbwarden imports the entry point and calls `setup(registry)`.
4. `setup(registry)` registers value hooks and/or object handlers.

The `registry` argument is a `PluginRegistrar` bound to your distribution name, so dbwarden attributes every hook to your plugin (used for conflict reporting and `plugin list`).

## Naming Conventions

All plugins are distributed as `dbwarden-<name>`. The import package and entry-point key derive from the distribution name: `slug = distribution.removeprefix("dbwarden-").replace("-", "_")`, then import package = `dbwarden_<slug>` and entry key = `<slug>`.

| Role | Convention | Example |
|------|------------|---------|
| PyPI project | `dbwarden-<name>` | `dbwarden-audit` |
| Import package | `dbwarden_<slug>` | `dbwarden_audit` |
| Entry-point key | `<slug>` | `audit` |

The `dbwarden-` prefix is shared by every plugin; a plugin's trust tier comes from the curated Official/Verified lists in core, not from its name. See [publishing](publishing.md#naming) for the full rule.

## Project Structure

Start from the [`dbwarden-plugin-template`](https://github.com/dbwarden-org/dbwarden-plugin-template) GitHub template (see [publishing](publishing.md#starting-from-the-template)):

```text
dbwarden-example/
├── pyproject.toml
├── src/dbwarden_example/
│   ├── __init__.py        # hook functions + setup(registrar)
│   └── handler.py         # object plugins only
├── tests/
│   └── test_example_plugin.py
├── README.md
└── LICENSE
```

`setup(registrar)` goes at the bottom of `__init__.py`. There is no `plugin.py`.

## The Core Contract

- **You may import anything from `dbwarden`.** There is no allowlist and no import check to satisfy. If your plugin needs `dbwarden.output` to render like the rest of the CLI, or `dbwarden.repositories.*` to read a tracking table, import it.
- Three surfaces are **stable** and change only on a major version: `dbwarden.plugin` (`PluginRegistrar`, the registries, hook errors), `dbwarden.exceptions`, and `dbwarden.engine.core` including `dbwarden.engine.core.plugin_api`. Prefer them where they cover your need, since that is what you will not have to revisit on a core upgrade.
- Anything deeper is supported but moves faster. Pin your `dbwarden` dependency to the range you test against, and run CI against those versions. `plugin_conformance.core_imports_outside_stable_api(pkg)` lists your deeper imports so an upgrade has a checklist.
- Use public **ordering anchors**, never private `StatementOrder` integers, those are renumbered freely.

## Rules

- **No import-time side effects.** Registering a hook must happen inside `setup()`, never at module import. Importing your package should register nothing.
- **Defer heavy imports to call time.** Value hooks import their dbwarden internals (and optional deps like FastAPI or drivers) inside the hook function body; object plugins import their `handler` module inside `setup()`. That way merely importing the package pulls in nothing heavy.
- Match documented hook signatures exactly; see the [hook catalog](../reference/hook-catalog.md).

========================================================================
PAGE: https://docs.dbwarden.org/plugins/developing/publishing/
========================================================================

# Publishing Plugins

## Starting From The Template

Use the [`dbwarden-plugin-template`](https://github.com/dbwarden-org/dbwarden-plugin-template) GitHub template repository: click **Use this template** to create your own repo, then rename the placeholders:

```bash
python bootstrap.py dbwarden-yourname
```

`bootstrap.py` applies the naming rule below, renames the package and test, and removes itself. The template is a working value plugin with a passing test and CI; adapt it to your hooks.

Every plugin uses the same shape: a setuptools `src/` layout, `setup(registrar)` defined at the bottom of the package `__init__.py`, and a single `dbwarden.plugins` entry point.

```text
dbwarden-example/
├── pyproject.toml
├── LICENSE
├── README.md
├── .gitignore
├── .github/workflows/test.yml
├── src/dbwarden_example/
│   └── __init__.py        # hook functions + setup(registrar)
└── tests/
    └── test_example_plugin.py
```

Object plugins add a `handler.py` next to `__init__.py` and import it lazily inside `setup()`. The official plugins are the reference implementations: [`dbwarden-fastapi`](https://github.com/dbwarden-org/dbwarden-fastapi) (value), [`dbwarden-pgsql-extensions`](https://github.com/dbwarden-org/dbwarden-pgsql-extensions) (object).

## Naming

All plugins, community and official alike, are distributed as `dbwarden-<name>`. The import package and entry-point key derive from the distribution name by one rule:

```
slug        = distribution_name.removeprefix("dbwarden-").replace("-", "_")
import pkg  = "dbwarden_" + slug
entry key   = slug
```

| Role | Convention | Example (`dbwarden-audit`) |
|------|------------|---------------------------|
| PyPI project | `dbwarden-<name>` | `dbwarden-audit` |
| Import package | `dbwarden_<slug>` | `dbwarden_audit` |
| Entry-point key | `<slug>` | `audit` |

The `dbwarden-` prefix is shared by every plugin. A plugin's trust tier is **not** inferred from its name; it comes from the curated Official and Verified lists in core (`dbwarden/_official.py`, `dbwarden/_verified.py`), and everything else is Community. Use dbwarden-owned names such as `dbwarden-fastapi` only for packages published by the dbwarden organization.

## PyPI Metadata

This mirrors what the official plugins ship (setuptools, `src/` layout, one entry point):

```toml
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "dbwarden-example"
version = "0.2.0"
description = "Example dbwarden plugin"
readme = "README.md"
requires-python = ">=3.12.7"
dependencies = ["dbwarden>=0.15.0,<0.18.0"]

[project.entry-points."dbwarden.plugins"]
example = "dbwarden_example:setup"

[tool.setuptools.packages.find]
where = ["src"]

[tool.pytest.ini_options]
testpaths = ["tests"]
```

The entry point is `dbwarden_example:setup` because `setup` is defined in the package's `__init__.py`; there is no separate `plugin.py`. Add `[project.urls]`, `license`, and `keywords` if you want them; the official plugins keep their metadata minimal.

## Compatibility

Declare both the oldest core version you test and the next untested compatibility boundary:

```toml
dependencies = ["dbwarden>=0.15.0,<0.18.0"]
```

`DBWARDEN_PLUGIN_API` protects the explicit plugin contract, but it does not prove compatibility with every core release. Update the upper bound after testing each new DBWarden minor release. Official plugins should follow the same policy.

## CI/CD Example

`.github/workflows/test.yml`:

```yaml
name: tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -e . pytest
      - run: pytest -q
```

Link a green run of this workflow when you [submit for verification](verified-standard.md#submit-for-verification).

## Trusted Publishing

Official dbwarden plugins use provenance-backed publishing (PyPI Trusted Publishing via GitHub Actions OIDC), which is what `dbwarden plugin add` verifies at install time. Community plugins can (and should) adopt Trusted Publishing too, but on its own it does not make a plugin Official: the Official tier is the curated list in `dbwarden/_official.py`.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/developing/value-plugins/
========================================================================

# Value Plugins

A value plugin supplies a value at a named hook point. Core calls the hook where optional behavior is needed, and, for single hooks, exactly one plugin may provide it. See the [hook catalog](../reference/hook-catalog.md) for every hook, its signature, and whether it is single or multi.

Common value hooks:

- `session_factory`, `sync_session_factory`: SQLAlchemy session dependency factories
- `clickhouse_session_factory`, `clickhouse_sync_session_factory`: ClickHouse client factories
- `load_model_module`, `load_config_module`: custom module loaders
- `lifespan`: FastAPI lifespan builder
- `health_routes`, `migration_routes`: route bundles (multi hooks)
- `seed_create`, `seed_apply`, `seed_list`, `seed_rollback`, `seed_export`: seed commands

## Registering A Hook

Define the hook function at module top level and register it in `setup()`:

```python
# src/dbwarden_example/__init__.py
def session_factory(database: str | None = None, *, dev: bool = False):
    from dbwarden_fastapi.engines import _async_session_factory  # deferred

    async def _dependency():
        async with _async_session_factory(database, dev=dev)() as session:
            yield session

    return _dependency


def setup(registrar) -> None:
    registrar.register("session_factory", session_factory)
```

`registrar.register(name, callable)` raises `ValueError` if `name` is not a known hook, so typos fail fast at load time.

## Walkthrough: The `dbwarden-fastapi` `setup()`

The official `dbwarden-fastapi` plugin defines each hook as a top-level function in its `__init__.py` and registers them all at the end. Its `setup()` is exactly:

```python
def setup(registrar) -> None:
    registrar.register("session_factory", session_factory)
    registrar.register("sync_session_factory", sync_session_factory)
    registrar.register("clickhouse_session_factory", clickhouse_session_factory)
    registrar.register("clickhouse_sync_session_factory", clickhouse_sync_session_factory)
    registrar.register("lifespan", lifespan)
    registrar.register("health_routes", health_routes)       # multi hook
    registrar.register("migration_routes", migration_routes) # multi hook
```

Each registered function imports what it needs from dbwarden (and FastAPI) **inside its body**, so importing the package registers the hooks without pulling in FastAPI. Core then resolves them where needed: `dbwarden_fastapi.get_session("primary")` calls the registered `session_factory`; if no plugin registered it, core uses its built-in fallback.

## The Import-Deferred Pattern

`setup` is exported from `__init__.py`, and registration happens only when dbwarden calls it. Keep heavy imports out of module scope by deferring them into each hook function:

```python
# src/dbwarden_example/__init__.py
def health_routes(*, auth_mode: str = "open", api_key: str | None = None):
    from fastapi import APIRouter  # deferred: FastAPI loads only when this hook runs

    router = APIRouter()
    # ... define routes ...
    return router


def setup(registrar) -> None:
    registrar.register("health_routes", health_routes)


__all__ = ["health_routes", "setup"]
```

Importing `dbwarden_example` must not import FastAPI or register anything until `setup()` runs.

## Multi vs Single Hooks

`health_routes` and `migration_routes` are **multi** hooks: several plugins may each contribute a router, and core collects them with `execute_all`. Every other value hook is **single**: if two plugins register it, core raises `HookConflictError` when the hook is invoked.

## Tests

Use the shared conformance harness (`dbwarden.plugin_conformance`) so your suite matches the [Verified standard](verified-standard.md) and earns the Verified badge; the [template](publishing.md#starting-from-the-template) wires it up for you. At minimum, value plugins should include:

- `test_setup_registers_hooks`: `setup(PluginRegistrar("dist"))` registers the declared hooks.
- `test_import_has_no_side_effects`: importing the package/module registers nothing and mutates no plugin global state.
- `test_entry_point_is_declared`: `pyproject.toml` exposes a discoverable `dbwarden.plugins` entry point.
- `test_hook_signature_compliance`: each registered callable accepts the documented arguments.

The official plugins use a lightweight fake registrar, matching how core calls `setup`:

```python
from dbwarden.plugin import HookRegistry
from dbwarden_example import setup


def setup_function() -> None:
    HookRegistry.clear()


def test_setup_registers_hooks() -> None:
    class Registrar:
        def register(self, hook_name, fn) -> None:
            HookRegistry.register(hook_name, fn, plugin="dbwarden-example")

    setup(Registrar())

    assert HookRegistry.is_registered("session_factory") is True
```

========================================================================
PAGE: https://docs.dbwarden.org/plugins/developing/verified-standard/
========================================================================

# Verified Plugin Standard

The **Verified** tier earns a plugin the **Verified** badge: it loads without per-user consent once its installed version meets the verified minimum. To qualify, the plugin must pass a focused, automatable conformance suite that proves it respects the dbwarden contract, does not compromise the user's environment, and (for object plugins) integrates correctly with the diff pipeline.

The suite is deliberately minimal. It does **not** review business logic, performance, or security; it ensures the plugin is a well-behaved citizen.

!!! warning "Verification is not a security audit"
    The Verified badge is a statement of contract adherence, not fitness for purpose. It says the plugin respects dbwarden's rules, will not break core, and will not execute code before you consent. It does **not** certify the plugin is free of vulnerabilities or malicious behavior, and it grants no sandboxing. A loaded plugin runs with full process privileges. Vet the source yourself for anything sensitive.

## The Conformance Harness

dbwarden ships the checks as `dbwarden.plugin_conformance`, so verification is reproducible rather than a one-time manual event. Each function raises `ConformanceError` with an explanation on failure. The [plugin template](publishing.md#starting-from-the-template) already wires them into `tests/test_conformance.py`; copy that file and adjust the distribution/package names.

## Required Tests

| # | Test | Harness call | Protects against |
|---|------|--------------|------------------|
| 1 | `test_entry_point_is_declared` | `assert_entry_point_declared(dist)` | Misconfigured packaging, missing/unresolvable entry point, a plugin that can never be discovered. |
| 2 | `test_import_has_no_side_effects` | `assert_import_has_no_side_effects(pkg)` | The #1 security-model violation: registering at import time bypasses the consent gate. Enforces classify-before-load. |
| 3 | `test_setup_registers_hooks` | `assert_setup_registers(setup, value_hooks=...)` | A broken `setup()` (wrong signature, swallowed exception, forgot to register) that installs but does nothing. |
| 4 | `test_hook_signature_compliance` | `assert_hook_signatures(setup)` | Runtime `TypeError` when core calls a hook with the documented arguments; caught at build time instead of migration time. |
| 5 | `test_core_imports_resolve` | `assert_core_imports_resolve(pkg)` | A plugin importing a dbwarden module the installed core does not have, which would otherwise fail on a user's machine instead of in CI. |
| 6 | `test_api_version_is_declared` | `assert_api_version_declared(pkg)` | A plugin with no declared contract version, which silently keeps loading after the contract changes under it. |
| 7 | `test_object_handler_conformance` (object plugins) | `assert_object_handler_conformance(handler, config=...)` | Handlers that crash the diff pipeline or emit malformed statements: `extract`/`canonicalize`/`diff`/`emit` must return the right types. |
| 8 | `test_ordering_constraint_satisfiable` (object plugins) | `assert_ordering_constraint_satisfiable(handler)` | Ordering constraints that can never be satisfied (impossible anchor pair, unknown/cyclic object references) and crash generation. |
| 9 | `test_idempotent_setup` (recommended) | `assert_idempotent_setup(setup)` | Double-loading in reloads/tests/interactive sessions: calling `setup()` twice must not raise or add new registrations. |

Tests 1 to 6 apply to every plugin. Tests 7 and 8 apply to object plugins; value-only plugins mark them not applicable and say so. Test 9 is recommended and may become required.

### What the harness checks for test 5

**Plugins may import anything from dbwarden core.** There is no allowlist. A plugin that needs `dbwarden.output` for consistent CLI rendering, or `dbwarden.repositories.*` to read tracking tables, should just import them.

What test 5 verifies is that every `dbwarden` module you import actually exists in the core you are built against. That catches the failure that actually bites users: a plugin pinned to `dbwarden>=0.15` importing a module that moved in 0.16, discovered at load time on their machine rather than in your CI.

Three surfaces are documented as stable and change only on a major version:

- `dbwarden.plugin` (`PluginRegistrar`, the registries, hook errors)
- `dbwarden.exceptions`
- `dbwarden.engine.core`, including `dbwarden.engine.core.plugin_api`

Building on those means less to revisit when core releases. Anything deeper is fair game, but pin your `dbwarden` dependency accordingly and keep CI running against the versions you claim to support. `plugin_conformance.core_imports_outside_stable_api(pkg)` lists your deeper imports so you know what to re-check on an upgrade; it reports, it does not fail.

## What This Does Not Cover

- **Business logic**: whether the emitted SQL does what the user wants (their own tests).
- **Performance**: no benchmark required.
- **Security audit**: no static analysis for malicious code beyond the import restriction.
- **Inter-plugin interaction**: the resolver's job.

## Submit For Verification

Open an issue in the dbwarden repository using the **Plugin Verification** template. It asks for the repository, the distribution name and requested minimum version, what the plugin provides, the core versions you test against, a link to a green conformance run, and two things worth explaining here.

### The plugin API version

The plugin contract is versioned separately from dbwarden's release version, as `dbwarden.plugin.PLUGIN_API_VERSION`. Declare the one you target on your package:

```python
DBWARDEN_PLUGIN_API = 1
```

Core refuses to load a plugin declaring a version it does not provide, reporting it as `incompatible` with both versions named, rather than letting it register and generate migrations under assumptions that no longer hold. Declaring nothing means "version 1", so plugins written before the contract was versioned keep working; the conformance suite still requires the declaration, because a plugin that never declares gets no protection when the contract moves.

The version changes only when a plugin could otherwise be *wrong* rather than loudly broken: a hook signature change, different semantics for a registered handler, a rename on the stable surface. Adding a hook or a new `plugin_api` helper does not change it.

### Declaring deep imports

The template asks you to paste the output of:

```bash
python -c "from dbwarden.plugin_conformance import core_imports_outside_stable_api as f; print('\n'.join(f('your_package')) or 'none')"
```

and justify each line. This is a review conversation, not a test: no automated check can tell a good reason from a bad one, which is why it belongs to a reviewer.

It also runs in your favour. A justification that generalises is an argument that the stable surface is missing something, and the outcome is usually that the helper becomes public in `dbwarden.engine.core.plugin_api` rather than that you are told to stop. That is exactly how `plugin_api` acquired `quote_pg`, `qualified_name`, the grant and policy builders, and `emit_with_cluster`: official plugins needed them, so they stopped being internal. If you would switch to a public equivalent, say so.

### Declaring object type overrides

A plugin handler that claims an `object_type` core already handles **replaces** core for that type: migration DDL for those objects comes from the plugin. dbwarden logs a warning when this happens, but a reviewer needs to know up front. Declare any overrides and explain why replacing core is right rather than adding a new type. Overriding is legitimate (it is how official plugins supersede built-in fallbacks) and it raises the review bar.

## CI Enforcement

To be Verified, the plugin must run the conformance suite in public CI (GitHub Actions or similar) on every push, with a visible status badge. The template's `.github/workflows/test.yml` does this. The reviewer confirms CI is green and that `tests/test_conformance.py` matches the standard, then does a quick sanity pass for obviously unsafe behavior (malware, core monkey-patching, network exfiltration). On verification, the distribution name and minimum version are added to `dbwarden/_verified.py` and ship in the next dbwarden release.

## Version Re-Verification

The verified minimum is a **floor**: patch and minor releases above it load automatically. A **new major version** (or any release that changes hooks, signatures, or object semantics) must request re-verification, and the floor in `dbwarden/_verified.py` is updated accordingly. Until then, a version below the recorded floor is treated as **Community** and requires consent.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/faq/
========================================================================

# Plugin FAQ

## Can I Use A Plugin Without The CLI?

Yes. Install it with `uv add` or `pip install`. dbwarden still applies trust rules when loading it, so community plugins installed this way still require consent (`dbwarden plugin trust <name>` or the interactive prompt).

## What Happens If An Official Plugin's Provenance Fails?

Installation aborts and nothing is installed. Official plugin install is fail-closed: if provenance cannot be verified, dbwarden refuses rather than installing unverified code.

## How Do I Know A Community Plugin Is Safe?

There is no automatic guarantee. Review the source, prefer Verified plugins, and only consent to plugins you have vetted yourself. Trust gates whether a plugin loads, not what it can do once loaded.

## Can Two Plugins Provide The Same Hook?

For **multi** hooks (`health_routes`, `migration_routes`), yes: core collects all providers. For **single** hooks, no: a second provider causes a `HookConflictError` when the hook runs. Two plugins registering the same object handler `object_type` raise `ObjectHandlerConflictError`.

## Can Plugins Depend On Each Other?

Yes. Use normal Python package dependencies. Object handlers from different plugins are ordered relative to each other with `OrderingConstraint` (`after_object` / `before_object`).

## Do Plugins Work In CI?

Yes. Install them as project dependencies. Use `--format json` on `plugin list` and `plugin info` for machine-readable output; Rich tables degrade gracefully when output is captured. Note that non-interactive runs never auto-consent, so trust community plugins ahead of time (commit `.dbwarden/consent.toml`).

## How Do I Get My Plugin Verified?

Follow the [Verified Plugin Standard](developing/verified-standard.md): pass the seven mandatory tests and open a review issue in the dbwarden repository.

## What Happens When I Upgrade A Community Plugin?

Consent is version-specific. After upgrading, the recorded consent no longer matches the new version, so the plugin is treated as unconsented until you run `dbwarden plugin trust <name>` again.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/
========================================================================

# dbwarden Plugins

Plugins let dbwarden keep a clean core while still supporting framework integrations, backend-specific schema objects, and community extensions. They are normal Python packages distributed through PyPI (or another package source) and discovered through the `dbwarden.plugins` entry point group.

dbwarden supports two plugin kinds:

- **Value plugins** supply values at named hook points: session factories, FastAPI routes, lifespans, seed commands, and module loaders.
- **Object plugins** add database object types to the schema diff pipeline: PostgreSQL extensions, roles, grants, policies, triggers, and other backend-specific objects.

Before plugin code is imported, dbwarden classifies the package by distribution name. Community plugin code is not imported until you explicitly consent to that exact version.

## Trust Tiers

| Tier | Meaning | Trust guarantee |
|------|---------|-----------------|
| Official | Built and maintained by the dbwarden organization. Published under organization-owned package names with provenance verified at install time. | Cryptographic and organizational. |
| Verified | Community-maintained, but passed the dbwarden plugin test standard and manual review. | Community review and technical compliance. |
| Community | Any `dbwarden.plugins` entry point not listed as Official or Verified. | Explicit consent only. |

Verification is not a security audit. Once loaded, a plugin runs with normal Python process privileges (it is not sandboxed). See [consent and trust](using-plugins/consent-and-trust.md).

## Philosophy

- Core defines stable contracts and migration semantics.
- Plugins use standard Python packaging and entry points.
- dbwarden classifies plugins before loading them (classify-before-load).
- Plugins register through `setup(registry)`, not import-time side effects.
- Object plugins use public ordering anchors, not private statement-order integers.

## Example

```bash
dbwarden plugin add dbwarden-fastapi
dbwarden plugin list
```

Then use the integration exposed by the plugin according to its documentation.

## Next Steps

- Start with the [quickstart](quickstart.md).
- Learn how to [install plugins](using-plugins/installing.md).
- Read the [trust model](using-plugins/consent-and-trust.md).
- Build your first plugin with the [developer overview](developing/overview.md).

========================================================================
PAGE: https://docs.dbwarden.org/plugins/quickstart/
========================================================================

# Quickstart: Add FastAPI Support

This quickstart walks through installing the official `dbwarden-fastapi` plugin, handling the consent prompt for community plugins, listing what dbwarden discovered, and using a session dependency in an endpoint.

## 1. Install The Plugin

Use dbwarden's plugin installer:

```bash
dbwarden plugin add dbwarden-fastapi
```

`dbwarden-fastapi` is an **Official** plugin, so `plugin add` verifies its provenance before installing and records the result in `.dbwarden/plugins.lock`.

You can also install directly with your package manager:

```bash
uv add dbwarden-fastapi
# or
pip install dbwarden-fastapi
```

To route `plugin add` through `uv` instead of pip, pass `--uv`:

```bash
dbwarden plugin add dbwarden-fastapi --uv
```

Preview the plan without touching your environment:

```bash
dbwarden plugin add dbwarden-fastapi --dry-run
```

```text
              Plugin Add (dry run): dbwarden-fastapi
  Tier            official
  Installer       python -m pip install dbwarden-fastapi
  Version         latest
  After install   provenance-verified, lockfile updated
```

## 2. Consent For Community Plugins

Official and Verified plugins load automatically. A **Community** plugin, any distribution not listed in core, is discovered but not imported until you consent to the exact installed version. When you run a dbwarden command in an interactive terminal, you are prompted:

```text
Enable community plugin 'dbwarden-example' version 0.1.0? [y/N]:
```

Answering `y` records consent in `.dbwarden/consent.toml` and loads the plugin. You can also consent ahead of time:

```bash
dbwarden plugin trust dbwarden-example
```

## 3. List Plugins

```bash
dbwarden plugin list
```

```text
                                     dbwarden Plugins
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Distribution     ┃ Version ┃ Tier     ┃ Trusted ┃ State  ┃ Hooks           ┃ Objects ┃ Lock       ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━┩
│ dbwarden-fastapi │ 0.1.0   │ official │ yes     │ loaded │ health_routes,  │ -       │ provenance │
│                  │         │          │         │        │ session_factory │         │            │
└──────────────────┴─────────┴──────────┴─────────┴────────┴─────────────────┴─────────┴────────────┘
```

For machine-readable output, add `--format json`:

```bash
dbwarden plugin list --format json
```

## 4. Use The Integration

The FastAPI integration exposes session dependencies through dbwarden's FastAPI API. A minimal endpoint that uses the async session for the `primary` database:

```python
from fastapi import Depends, FastAPI
from sqlalchemy.ext.asyncio import AsyncSession

from dbwarden_fastapi import get_session

app = FastAPI()


@app.get("/users")
async def users(session: AsyncSession = Depends(get_session("primary"))):
    result = await session.execute(...)
    return {"users": result.scalars().all()}
```

`get_session` resolves the dependency through the plugin's `session_factory` hook. This import path is provided by the `dbwarden-fastapi` plugin package, not by dbwarden core. See the plugin's README for configuration.

## Next Steps

- [Installing plugins](using-plugins/installing.md)
- [Consent and trust](using-plugins/consent-and-trust.md)
- [Plugin CLI reference](reference/plugin-cli.md)

========================================================================
PAGE: https://docs.dbwarden.org/plugins/reference/hook-catalog/
========================================================================

# Hook Catalog

Value hooks are registered with `registry.register(name, callable)`. Object handlers are registered with `registry.register_object_handler(handler)`. The known hook names are defined by `KNOWN_VALUE_HOOKS` in `dbwarden.plugin`; registering an unknown name raises `ValueError`.

## Value Hooks

| Hook | Multi | Signature | Returns | Provided by (example) |
|------|-------|-----------|---------|-----------------------|
| `session_factory` | no | `(database: str \| None = None, *, dev: bool = False)` | async session dependency (callable) | `dbwarden-fastapi` |
| `sync_session_factory` | no | `(database: str \| None = None, *, dev: bool = False)` | sync session dependency (callable) | `dbwarden-fastapi` |
| `clickhouse_session_factory` | no | `(database: str \| None = None, *, dev: bool = False)` | async ClickHouse dependency (callable) | `dbwarden-fastapi` |
| `clickhouse_sync_session_factory` | no | `(database: str \| None = None, *, dev: bool = False)` | sync ClickHouse dependency (callable) | `dbwarden-fastapi` |
| `load_model_module` | no | `(path: Path, base_dir: Path)` | loaded module | `dbwarden-sandbox` |
| `load_config_module` | no | `(path: Path, base_dir: Path)` | `None` | `dbwarden-sandbox` |
| `lifespan` | no | `(app=None, *, mode="check", **kwargs)` | async context manager | `dbwarden-fastapi` |
| `health_routes` | yes | `(*, auth_mode: str = "open", api_key: str \| None = None)` | `APIRouter` | `dbwarden-fastapi` |
| `migration_routes` | yes | `(*, auth_mode: str = "open", api_key: str \| None = None)` | `APIRouter` | `dbwarden-fastapi` |
| `seed_create` | no | `(description, *, seed_type="sql", database=None, verbose=False)` | `None` | `dbwarden-seeds` |
| `seed_apply` | no | `(*, version=None, dry_run=False, database=None, all_databases=False, verbose=False)` | `None` | `dbwarden-seeds` |
| `seed_list` | no | `(*, database=None, all_databases=False, verbose=False, prune=False)` | `None` | `dbwarden-seeds` |
| `seed_rollback` | no | `(*, count=None, to_version=None, database=None, all_databases=False, verbose=False)` | `None` | `dbwarden-seeds` |
| `seed_export` | no | `(*, database=None, all_databases=False, output_dir="seeds")` | `None` | `dbwarden-seeds` |
| `sandbox_provider_start` | no | `(db_name: str \| None = None)` | `None` | `dbwarden-sandbox` |
| `sandbox_provider_stop` | no | `(db_name: str \| None = None)` | `None` | `dbwarden-sandbox` |

**Multi** hooks (`health_routes`, `migration_routes`) may be provided by several plugins; core collects all of them. Every other value hook is **single**: two providers cause a `HookConflictError` when the hook runs.

## Object Handlers

Object handlers are not named hooks; they register through `registry.register_object_handler(handler)` and are keyed by the handler's `object_type`. See [object plugins](../developing/object-plugins.md). Registering the same `object_type` from two different plugins raises `ObjectHandlerConflictError`.

## How Core Calls Hooks

Core resolves value hooks through the `HookRegistry`:

```python
from dbwarden.plugin import HookRegistry, HookNotRegisteredError

# Single hook: exactly one provider, or fall back to core behavior.
try:
    dependency = HookRegistry.execute_single("session_factory", "primary", dev=False)
except HookNotRegisteredError:
    dependency = core_default_session("primary")

# Multi hook: collect every provider's contribution.
routers = HookRegistry.execute_all("health_routes", auth_mode="open")
```

- `execute_single(name, *args, **kwargs)` raises `HookNotRegisteredError` when no plugin provides the hook and `HookConflictError` when more than one does.
- `execute_all(name, *args, **kwargs)` returns a list of every provider's result (empty if none).

Plugin authors do not call these; they are shown here so you can see exactly how and with what arguments core will invoke your callable.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/reference/ordering-anchors/
========================================================================

# Ordering Anchors

Object plugins position their SQL using public **anchors**, not private statement-order integers. At registration, dbwarden validates your `OrderingConstraint` and derives the internal `statement_order` from the anchors for you.

## The `Anchor` Enum

| Anchor | Meaning |
|--------|---------|
| `PREAMBLE` | Earliest setup statements (extensions, roles, schemas). |
| `BEFORE_TABLES` | Objects that must exist before tables (types, domains). |
| `AFTER_TABLES` | Objects emitted after table creation (added columns). |
| `AFTER_CONSTRAINTS` | Objects emitted after table constraints. |
| `AFTER_INDEXES` | Objects emitted after indexes. |
| `POSTAMBLE` | Final statements (views, grants that depend on everything). |

Anchors run in the order listed above.

## `OrderingConstraint`

```python
@dataclass(frozen=True)
class OrderingConstraint:
    after: tuple[Anchor, ...] = ()
    before: tuple[Anchor, ...] = ()
    after_object: tuple[str, ...] = ()   # by another handler's object_type
    before_object: tuple[str, ...] = ()
```

- `after` / `before` place the handler relative to core milestones.
- `after_object` / `before_object` place it relative to other object handlers by their `object_type`, forming a DAG that dbwarden topologically sorts.

## Examples

Extensions, first thing (PostgreSQL):

```python
ordering = OrderingConstraint(after=(Anchor.PREAMBLE,), before=(Anchor.BEFORE_TABLES,))
```

Roles, before the objects that reference them:

```python
ordering = OrderingConstraint(after=(Anchor.PREAMBLE,), before=(Anchor.BEFORE_TABLES,))
```

Grants, after everything they depend on:

```python
ordering = OrderingConstraint(after=(Anchor.AFTER_INDEXES,), before=(Anchor.POSTAMBLE,))
```

Triggers, after tables and their constraints exist:

```python
ordering = OrderingConstraint(after=(Anchor.AFTER_CONSTRAINTS,))
```

A grant handler that must run after a role handler, by object type:

```python
ordering = OrderingConstraint(after_object=("role",))
```

## Failure Modes

dbwarden rejects invalid ordering at registration or run time by raising `OrderingError`:

- **Impossible anchor pair**, `after` an anchor that comes at or after `before`:

  ```python
  OrderingConstraint(after=(Anchor.POSTAMBLE,), before=(Anchor.PREAMBLE,))
  # OrderingError: Impossible ordering: after postamble and before preamble
  ```

- **Unknown object reference**, naming an `object_type` no registered handler provides:

  ```python
  OrderingConstraint(after_object=("missing",))
  # OrderingError: Unknown object ordering reference 'missing' for '<your type>'
  ```

- **Cycle**, two handlers that each require running after the other:

  ```python
  # first: after_object=("second",);  second: after_object=("first",)
  # OrderingError: Object handler ordering cycle detected: first, second
  ```

## Why `StatementOrder` Is Private

The internal `StatementOrder` integers (and their exact values) are an implementation detail and are renumbered as core evolves. Anchors are the **public, stable contract**. Never read or set `statement_order` yourself; declare `ordering` with anchors and let dbwarden compute the rest.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/reference/plugin-cli/
========================================================================

# Plugin CLI

All commands live under `dbwarden plugin`. Unlike other dbwarden commands, `dbwarden plugin` does not auto-load plugins first, so it can inspect and manage plugins without importing them. Management and inspection do not import plugins by default. `categories` loads trusted plugins without prompting; `list` and `info` do so only with `--load`.

## Subcommands

| Command | Purpose |
|---------|---------|
| `list` | Show discovered plugins as a table or JSON. |
| `info <name>` | Show one plugin's metadata and provenance. |
| `add <name>` | Install a plugin (provenance-verified for official). |
| `remove <name>` | Uninstall a plugin and clean up its consent/lock state. |
| `trust <name>` | Record consent for a community plugin's installed version. |
| `untrust <name>` | Revoke consent for a community plugin. |
| `categories` | List built-in and loaded plugin migration groups, order, and owner. |

## Flags

| Command | Flag | Effect |
|---------|------|--------|
| `list`, `info`, `categories` | `--format`, `-f` `table\|json` | Output format (default `table`). |
| `list`, `info` | `--load` | Load trusted plugins without prompting to inspect registrations. |
| `add` | `--uv` | Install with `uv add` instead of pip. |
| `add` | `--version <v>` | Pin an exact version (`dist==<v>`). |
| `add` | `--dry-run` | Print the install plan without installing. |
| `remove` | `--uv` | Uninstall with `uv remove` instead of pip. |
| `remove` | `--dry-run` | Print the uninstall plan without uninstalling. |

## list

```bash
dbwarden plugin list
dbwarden plugin list --format json
```

Columns: Distribution, Version, Tier, Trusted, State, Hooks, Objects, Lock.

```text
                                     dbwarden Plugins
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Distribution     ┃ Version ┃ Tier     ┃ Trusted ┃ State  ┃ Hooks           ┃ Objects ┃ Lock       ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━┩
│ dbwarden-fastapi │ 0.1.0   │ official │ yes     │ loaded │ health_routes,  │ -       │ provenance │
│                  │         │          │         │        │ session_factory │         │            │
└──────────────────┴─────────┴──────────┴─────────┴────────┴─────────────────┴─────────┴────────────┘
```

## info

```bash
dbwarden plugin info dbwarden-fastapi
dbwarden plugin info dbwarden-fastapi --format json
```

Shows entry point, tier, trust/load state, registered hooks and object handlers, official repository, verified minimum version, and lockfile provenance fields when present. Exits `1` if the plugin is not found.

## add

```bash
dbwarden plugin add dbwarden-fastapi
dbwarden plugin add dbwarden-fastapi --version 0.2.0 --uv
dbwarden plugin add dbwarden-example --dry-run
```

Official plugin installation verifies provenance and fails closed when verification is unavailable. Community plugins are installed but not trusted; the command prints the follow-up `trust` hint. `--dry-run` prints tier, installer command, and post-install behavior without installing.

## remove

```bash
dbwarden plugin remove dbwarden-example
dbwarden plugin remove dbwarden-example --dry-run --uv
```

Uninstalls the distribution and removes its consent entry and lockfile record.

## trust / untrust

```bash
dbwarden plugin trust dbwarden-example
dbwarden plugin untrust dbwarden-example
```

`trust` records consent for the currently installed version in `.dbwarden/consent.toml`. `untrust` removes it.

## categories

```bash
dbwarden plugin categories --format json
dbwarden plugin info dbwarden-example --load --format json
```

`list` and `info` JSON include `migration_categories`, containing each registered category's `name`, `order`, and `plugin`. Use `--load` to populate these registrations in a fresh CLI process. `info` also displays category names in its table.

Categories are named migration groups; safety levels remain SAFE/INFO/WARN/CRITICAL. See [object plugin categories](../developing/object-plugins.md#safety-and-migration-categories).

## File Formats

### Consent file: `.dbwarden/consent.toml`

Written by `trust` and by accepting the interactive prompt. Consent is version-specific.

```toml
[consent."dbwarden-example"]
version = "0.1.0"
consented_at = "2026-07-22T14:03:11.482915+00:00"
```

### Lockfile: `.dbwarden/plugins.lock`

Written by `plugin add` for official (provenance-verified) installs. The section key is the distribution name with the `dbwarden-` prefix stripped and dashes replaced by underscores (for example `dbwarden-pgsql-extensions` becomes `pgsql_extensions`).

```toml
[pgsql_extensions]
distribution = "dbwarden-pgsql-extensions"
version = "0.3.0"
filename = "dbwarden_pgsql_extensions-0.3.0-py3-none-any.whl"
sha256 = "abc123..."
tier = "official"
verified = "provenance"
identity = "https://github.com/dbwarden-org/dbwarden-pgsql-extensions"
installed_at = "2026-07-22T14:05:40.001122+00:00"
```

Commit both files so teammates and CI inherit your trust and provenance decisions.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/using-plugins/consent-and-trust/
========================================================================

# Consent And Trust

dbwarden classifies every plugin entry point **before importing any plugin code**. Classification is by distribution name, so a plugin's tier is known without running it.

## The Three Tiers

| Tier | Loads automatically? | Basis of trust |
|------|----------------------|----------------|
| Official | Yes | Organization-owned name + provenance verified at install. |
| Verified | Yes, when version ≥ verified minimum | Community review against the test standard. |
| Community | No, consent required | Your explicit, version-specific consent. |

### Official

Official plugins are dbwarden-maintained. `plugin add` verifies provenance and **fails closed** when verification is unavailable or invalid. They load without consent.

### Verified

Verified plugins are community-maintained but reviewed against the dbwarden plugin standard. They load **without consent** only when the installed version is at or above the verified minimum recorded in `dbwarden/_verified.py`. If the installed version is below the floor, dbwarden logs a warning and treats the plugin as **Community** (consent required).

### Community

Community plugins require explicit consent before loading. In an interactive terminal, dbwarden prompts on first use:

```text
Enable community plugin 'dbwarden-example' version 0.1.0? [y/N]:
```

Accepting records consent. Non-interactive runs (CI, scripts) never auto-consent: an unconsented community plugin is skipped with a warning:

```text
Skipping community plugin 'dbwarden-example' (not consented).
Run `dbwarden plugin trust dbwarden-example` to enable it.
```

## Consent Commands

```bash
dbwarden plugin trust dbwarden-example      # record consent for the installed version
dbwarden plugin untrust dbwarden-example    # revoke consent
```

## The Consent File

Consent is stored per-project in `.dbwarden/consent.toml`:

```toml
[consent."dbwarden-example"]
version = "0.1.0"
consented_at = "2026-07-22T14:03:11.482915+00:00"
```

Consent is bound to the exact `version`. If you upgrade the plugin, the recorded version no longer matches and the plugin is treated as unconsented until you trust the new version. Commit this file to share consent decisions with your team.

## Security Boundary

Classify-before-load ensures untrusted community plugin code is **not imported** automatically: the security decision happens before `import`.

!!! warning "Not a sandbox"
    Trust gates *whether* a plugin loads, not *what it can do*. Once a plugin is loaded it runs with **full Python process privileges**: it can read files, open network connections, and call any API your process can. Consent is a statement that you have reviewed and trust the code, not a containment mechanism. Approval is likewise a review, **not a security audit**.

Verified and Official tiers exist to reduce how often you must make that trust decision manually, not to make loading arbitrary code safe.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/using-plugins/finding-plugins/
========================================================================

# Finding Plugins

dbwarden plugins are standard Python packages that expose a `dbwarden.plugins` entry point. dbwarden classifies each discovered distribution into a [trust tier](consent-and-trust.md) before importing it, by matching its distribution name against the curated Official and Verified lists in core.

## Search PyPI

All plugins are distributed as:

```text
dbwarden-<name>
```

Search PyPI for `dbwarden-` to find them. The name alone does not confer trust: Official plugins are the dbwarden-owned packages listed in `dbwarden/_official.py` (for example `dbwarden-fastapi`, `dbwarden-pgsql-extensions`); every other `dbwarden-*` package is Community until it reaches the Verified list.

## Official Plugins

Official plugins are built and maintained by the dbwarden organization under the [`dbwarden` GitHub organization](https://github.com/dbwarden). They are enumerated in core (`dbwarden/_official.py`) and are eligible for provenance verification during `dbwarden plugin add`.

| Package | Repository | Purpose |
|---------|------------|---------|
| `dbwarden-fastapi` | [dbwarden-org/dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi) | FastAPI sessions, health/migration routes, lifespan. |
| `dbwarden-pgsql-extensions` | [dbwarden-org/dbwarden-pgsql-extensions](https://github.com/dbwarden-org/dbwarden-pgsql-extensions) | PostgreSQL `CREATE EXTENSION` diffing. |
| `dbwarden-pgsql-rbac` | [dbwarden-org/dbwarden-pgsql-rbac](https://github.com/dbwarden-org/dbwarden-pgsql-rbac) | PostgreSQL roles, grants, policies. |
| `dbwarden-pgsql-types` | [dbwarden-org/dbwarden-pgsql-types](https://github.com/dbwarden-org/dbwarden-pgsql-types) | PostgreSQL enum/domain/composite types. |
| `dbwarden-ch-rbac` | [dbwarden-org/dbwarden-ch-rbac](https://github.com/dbwarden-org/dbwarden-ch-rbac) | ClickHouse roles and grants. |
| `dbwarden-seeds` | [dbwarden-org/dbwarden-seeds](https://github.com/dbwarden-org/dbwarden-seeds) | Seed command implementations. |
| `dbwarden-sandbox` | [dbwarden-org/dbwarden-sandbox](https://github.com/dbwarden-org/dbwarden-sandbox) | Isolated model/config module loading. |

The authoritative list is always `OFFICIAL_PLUGINS` in `dbwarden/_official.py`.

## Verified Plugins

Verified plugins are community-maintained packages that passed dbwarden's [plugin test standard](../developing/verified-standard.md) and manual review. They load without consent once the installed version meets the verified minimum.

| Package | Minimum verified version | Description |
|---------|--------------------------|-------------|
| _None yet_ – feel free to [be the first](../developing/verified-standard.md#submit-for-verification)! | – | Verified entries are listed in `dbwarden/_verified.py` and this table as the ecosystem grows. |

Verification is not a security audit. See [consent and trust](consent-and-trust.md).

## Community Plugins

Any other package with a `dbwarden.plugins` entry point is treated as **Community**. Community plugins require explicit consent before loading. A community-curated index may be maintained as a Markdown page in the dbwarden repository; until then, discover them via PyPI search and review the source before trusting.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/using-plugins/installing/
========================================================================

# Installing Plugins

## dbwarden CLI

```bash
dbwarden plugin add <distribution-name>
```

What `plugin add` does depends on the [trust tier](consent-and-trust.md) of the distribution:

- **Official**: dbwarden verifies provenance first. On success it installs the package and writes an entry to `.dbwarden/plugins.lock` recording the version, filename, SHA-256, and verifying identity. If provenance cannot be verified, installation aborts and nothing is installed (fail-closed).
- **Verified**: installed like any package; it will load automatically once the installed version meets the verified minimum.
- **Community**: the package is installed but **not** trusted. You must run `dbwarden plugin trust <name>` (or accept the interactive consent prompt) before dbwarden loads it.

```text
              Plugin Installed
  Installed community plugin 'dbwarden-example'.
Run `dbwarden plugin trust dbwarden-example` to allow it to load.
```

### Flags

| Flag | Applies to | Effect |
|------|------------|--------|
| `--uv` | `add`, `remove` | Use `uv add` / `uv remove` instead of pip. |
| `--version <v>` | `add` | Pin an exact version to install (`dist==<v>`). |
| `--dry-run` | `add`, `remove` | Print the plan (tier, installer command, post-install behavior) without changing anything. |

```bash
dbwarden plugin add dbwarden-fastapi --version 0.2.0 --uv
dbwarden plugin add dbwarden-fastapi --dry-run
```

## uv Or pip

You can install plugins directly without the CLI:

```bash
uv add dbwarden-fastapi
```

```bash
pip install dbwarden-fastapi
```

dbwarden still applies its trust model when loading. Community plugins installed this way still require consent, and Official plugins installed this way are **not** provenance-locked (no `.dbwarden/plugins.lock` entry is written unless you use `plugin add`).

## Version Pinning

Pin versions in your project dependencies for repeatable environments:

```bash
uv add "dbwarden-fastapi==0.2.0"
```

Community consent is version-specific: upgrading a community plugin invalidates prior consent and requires consent for the new version.

## What Happens During Official Install

Provenance verification uses PyPI's [PEP 740](https://peps.python.org/pep-0740/) attestations. dbwarden:

1. Looks up the distribution in `OFFICIAL_PLUGINS` for its expected GitHub repository and publishing workflow.
2. Resolves the target version on PyPI (the pinned/`--version` release, or the highest stable release) and selects its distribution file and SHA-256.
3. Fetches that file's attestation from PyPI's Integrity API and checks that the recorded Trusted-Publishing publisher is the expected GitHub repository and workflow, and that the attestation covers the file's exact digest.
4. On success, installs that **exact** version and writes a lockfile entry with `verified = "provenance"`, the identity, filename, and SHA-256.

Any missing attestation, publisher mismatch, digest mismatch, or network error makes the install **fail closed**: nothing is installed. Trust is anchored in PyPI's server-side attestation verification and TLS, the same root pip relies on.

## Updating Plugins

Update with your package manager, then reconcile dbwarden's view:

```bash
uv add "dbwarden-fastapi@latest"
dbwarden plugin list
dbwarden plugin info dbwarden-fastapi
```

For community plugins, re-run `dbwarden plugin trust <name>` after upgrading so consent matches the new version.

## Removing Plugins

```bash
dbwarden plugin remove <distribution-name>
```

This uninstalls the distribution and removes its consent entry and lockfile record. Use `--dry-run` to preview or `--uv` to uninstall via `uv remove`.

========================================================================
PAGE: https://docs.dbwarden.org/plugins/using-plugins/plugin-list-and-info/
========================================================================

# Plugin List And Info

## List Plugins

```bash
dbwarden plugin list
```

The Rich table shows every discovered distribution with its version, tier, trust state, load state, the hooks and object handlers it registered, and its lockfile verification status.

```text
                                     dbwarden Plugins
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Distribution           ┃ Version ┃ Tier      ┃ Trusted ┃ State   ┃ Hooks           ┃ Objects      ┃ Lock       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ dbwarden-fastapi       │ 0.1.0   │ official  │ yes     │ loaded  │ health_routes,  │ -            │ provenance │
│                        │         │           │         │         │ session_factory │              │            │
│ dbwarden-pgsql-exten…  │ 0.3.0   │ official  │ yes     │ loaded  │ -               │ pg_extension │ provenance │
│ dbwarden-acme          │ 1.0.0   │ community │ no      │ skipped │ -               │ -            │ -          │
└────────────────────────┴─────────┴───────────┴─────────┴─────────┴─────────────────┴──────────────┴────────────┘
```

Column meanings:

- **Trusted**: `yes` if the plugin is allowed to load (official, verified-and-current, or consented community).
- **State**: `loaded`, `skipped` (untrusted community), `failed` (raised during `setup`), `incompatible` (built against a different plugin API version), or `discovered`.

An `incompatible` plugin is not broken: it targets a different version of the plugin contract than this dbwarden provides, so it was refused rather than allowed to register and produce migrations under the wrong assumptions. The **Error** row names both versions. Upgrade the plugin, or pin dbwarden to a release that provides the version it wants.
- **Hooks / Objects**: the value hooks and object handler types the plugin registered. A discovered-but-skipped plugin registers nothing, so these stay empty.
- **Lock**: the lockfile `verified` value (e.g. `provenance`) for provenance-locked installs.

### JSON Output

For CI or scripting, use `--format json`:

```bash
dbwarden plugin list --format json
```

```json
[
  {
    "distribution": "dbwarden-fastapi",
    "version": "0.1.0",
    "tier": "official",
    "trusted": true,
    "state": "loaded",
    "hooks": ["health_routes", "session_factory"],
    "object_handlers": [],
    "error": null,
    "lock": {"verified": "provenance", "identity": "https://github.com/dbwarden-org/dbwarden-fastapi", "...": "..."}
  }
]
```

## Plugin Details

```bash
dbwarden plugin info dbwarden-fastapi
```

```text
Plugin Info
  Distribution       dbwarden-fastapi
  Version            0.1.0
  Entry point        fastapi = dbwarden_fastapi:setup
  Tier               official
  Trusted            yes
  State              loaded
  Hooks              health_routes, session_factory
  Object handlers    -
  Error
  Repository         https://github.com/dbwarden-org/dbwarden-fastapi
  Verified minimum
  Lock verified      provenance
  Lock identity      https://github.com/dbwarden-org/dbwarden-fastapi
```

- For **Official** plugins, details include the repository and lockfile provenance fields.
- For **Verified** plugins, the **Verified minimum** row shows the version floor.
- For **Community** plugins, trust reflects consent state; `Repository` and `Verified minimum` are empty.

`plugin info` also supports `--format json`.

## Checking Registered Hooks

Both `plugin list` and `plugin info` report the hooks and object handlers registered **after load**. If a plugin is discovered but skipped (untrusted) or failed during `setup`, those columns are empty and, for failures, the **Error** row explains why.

========================================================================
PAGE: https://docs.dbwarden.org/reference/cli-options/
========================================================================

# CLI option inventory

This page lists every built-in command, argument, and option from the current CLI.
For workflows and exit codes, see the [CLI guide](../cli-reference.md).
Plugin commands are added at startup by installed, approved plugins; inspect
their help separately. Global options precede the command name.

Regenerate with `python scripts/check-docs.py --cli-reference`.

## `dbwarden`

dbwarden - Professional database migration system for SQLAlchemy models

All commands support the --verbose / -v flag for detailed output and the
--debug / --debug-level flags for DEBUG-level diagnostics.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --dev | boolean | False | Use development database settings (dev_database_url/dev_database_type) |
| --strict-translation | boolean | False | Fail when a type/default cannot be translated for target backend |
| --disable-skip | boolean | False | Do not skip databases configured with skip_if_missing. |
| --debug | boolean | False | Enable DEBUG-level logging |
| --debug-level | str | None | Exact log level: trace, debug, info, warning, error, critical, or 5/10/20/30/40/50 |
| --json, -j | boolean | False | Render command output (and logs) as structured JSON |
| --log-level | str (repeatable) | [] | Per-component log level (repeatable). Example: --log-level snapshot:debug --log-level plugin:warning. Known components: plugin, lock, registry, snapshot |

`--help` displays command help.

## `dbwarden init`

Initialize the migrations directory.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Database name to create migration directory for |

`--help` displays command help.

## `dbwarden generate-models`

Reverse-engineer SQLAlchemy model code from a live database.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --output, -o | str | 'models' | Output directory for generated model files |
| --tables | str | None | Comma-separated list of tables to include |
| --exclude-tables | str | None | Comma-separated list of tables to exclude |
| --clickhouse-engines | boolean | False | Include ClickHouse engine metadata |
| --relationships | boolean | False | Generate relationship attributes |
| --dialect | str | None | SQL dialect for type mapping (auto-detected by default) |
| --single-file | boolean | False | Generate a single models.py file |
| --base | str | None | Custom Base class import path (e.g. 'app.core.database:Base' or 'app.database:DeclarativeBase') |
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden export-models`

Export current model definitions to a JSON state file for offline diffs.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --output, -o | str | None | Output path for the model state JSON file |
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden recover-model-state`

Recover a deleted model state file by replaying migrations in a sandbox.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden make-migrations`

Auto-generate SQL migration from SQLAlchemy models.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| description | str | None | Description for the migration |
| --verbose, -v | boolean | False | Enable verbose logging |
| --plan | boolean | False | Output migration plan JSON without writing files |
| --sql | boolean | False | Output raw migration SQL to stdout without writing files |
| --database, -d | str | None | Target database name |
| --rename | str (repeatable) | [] | Explicitly rename column (format: table.old_name:new_name). Can be repeated. |
| --safe-type-change | boolean | False | Use multi-step safe type change (add temp column, backfill, swap, drop old) |
| --rename-table | str (repeatable) | [] | Declare a table rename as old_table:new_table. Repeatable. |
| --concurrent, --no-concurrent | boolean | True | Use CREATE INDEX CONCURRENTLY on PostgreSQL (disable inside transactions) |
| --offline | boolean | False | Use model state file instead of live database (requires export-models first) |
| --clickhouse-engine-recreate | boolean | False | Allow automatic ClickHouse table rebuild when engine changes require recreation |
| --drop-preserved-clickhouse-table, --keep-preserved-clickhouse-table | boolean | None | Drop the preserved old ClickHouse table after swap; if omitted, prompt in TTY and preserve by default |
| --postgres-auto-using | boolean | False | Emit active USING clause on PostgreSQL ALTER COLUMN TYPE (default: commented-out) |
| --type, -t | str | 'versioned' | Output prefix: versioned (default), runs_always/ra, runs_on_change/roc |
| --perf | boolean | False | Log SQL-generation phase timing |
| --split-at-severity | str | None | Defer operations at or above SAFE, INFO, WARN, or CRITICAL. |
| --strict-pending, --no-strict-pending | boolean | None | Refuse generation when pending files lack composable plans. |
| --dry-run | boolean | False | Preview generated migrations without writing files. |
| --param | str (repeatable) | [] | Freeze a data expression parameter as name=JSON or name=text. |
| --show-managed-values | boolean | False | Include data literals in --plan output. |

`--help` displays command help.

## `dbwarden make-data-migration`

Generate versioned managed-row and transformation migrations.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| description | str | None |  |
| --database, -d | str | None |  |
| --offline | boolean | False |  |
| --dry-run | boolean | False |  |
| --plan | boolean | False |  |
| --sql | boolean | False |  |
| --param | str (repeatable) | [] |  |
| --show-managed-values | boolean | False |  |
| --split-at-severity | str | None |  |

`--help` displays command help.

## `dbwarden new`

Create a new manual migration file.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| description | str | required | Description of the migration |
| --version | str | None | Version of the migration |
| --database, -d | str | None | Target database name |
| --type, -t | str | 'versioned' | Migration type: versioned (default), runs_always/ra, runs_on_change/roc |

`--help` displays command help.

## `dbwarden migrate`

Apply pending migrations to the database.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --count, -c | int | None | Number of migrations to apply |
| --to-version, -t | str | None | Migrate to a specific version |
| --verbose, -v | boolean | False | Enable verbose logging |
| --database, -d | str | None | Target database name |
| --all, -a | boolean | False | Run migrations on all databases sequentially |
| --baseline | boolean | False | Mark migrations as applied without executing |
| --reapply-data | boolean | False | Explicitly reapply data migrations that were successfully rolled back |
| --with-backup, -b | boolean | False | Create a backup before migrating |
| --backup-dir | str | None | Directory for backup files |
| --dry-run | boolean | False | Show what would be applied without executing |
| --data | boolean | False | With --dry-run, verify and render frozen data bundle details without writes |
| --sandbox | boolean | False | Apply migrations in a temporary sandbox database |
| --apply-seeds | boolean | False | Apply pending seeds after migrations (overrides config) |
| --perf | boolean | False | Log per-SQL-statement timing breakdowns |
| --defer-snapshots | boolean | False | Write one final schema snapshot instead of one after every migration |
| --max-severity | str | None | Severity ceiling: SAFE, INFO, WARN, CRITICAL. Stops before higher/UNKNOWN files; exit 3. |
| --force | boolean | False | Acknowledge operation risks; does not raise the severity ceiling. |

`--help` displays command help.

## `dbwarden rollback`

Rollback the last applied migration.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --count, -c | int | None | Number of migrations to rollback |
| --to-version, -t | str | None | Rollback to a specific version |
| --verbose, -v | boolean | False | Enable verbose logging |
| --database, -d | str | None | Target database name |
| --perf | boolean | False | Log per-SQL-statement timing breakdowns |

`--help` displays command help.

## `dbwarden downgrade`

Downgrade to a specific migration version by reverting applied migrations.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --to, -t | str | required | Target version to downgrade to |
| --verbose, -v | boolean | False | Enable verbose logging |
| --database, -d | str | None | Target database name |
| --perf | boolean | False | Log per-SQL-statement timing breakdowns |

`--help` displays command help.

## `dbwarden make-rollback`

Generate a rollback SQL file for a given migration file.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| migration_file | str | required | Path to migration SQL file |

`--help` displays command help.

## `dbwarden snapshot`

Snapshot the DDL schema of a specific table.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| table_name | str | required | Table name to snapshot |
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden history`

Show the full migration history.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden status`

Show migration status (applied and pending).

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Target database name |
| --all, -a | boolean | False | Show status for all databases |
| --all-environments | boolean | False | Show status for all registered environments |

`--help` displays command help.

## `dbwarden check`

Run the schema safety analyzer.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| version | str | None | Version to classify with --write-plan. |
| --out, -o | str | 'txt' | Output format (json, txt) |
| --force | boolean | False | Allow warning-level changes to pass |
| --database, -d | str | None | Target database name |
| --write-plan | boolean | False | Statically classify SQL into hash-bound plans; no live database. |
| --all | boolean | False | Classify all migration files; unresolved files produce exit 4. |
| --data | boolean | False | Also verify live declarative data convergence. |

`--help` displays command help.

## `dbwarden check-db`

Inspect the live database schema.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --out, -o | str | 'txt' | Output format (json, yaml, sql, txt) |
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden check-impact`

Analyze impact of a migration on your codebase.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| migration | str | required | Migration version or plan file path |
| --out, -o | str | 'text' | Output format: text (default) or json |
| --scan-path | str | '.' | Directory to scan for affected code |
| --deep | boolean | False | Enable deep introspection (imports models live) |
| --verbose, -v | boolean | False | Show INFO-level operations in scan |
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden diff`

Show structural differences between models and database (read-only, no files written).

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --out, -o | str | 'table' | Output format: table (default), json, sql |
| --verbose, -v | boolean | False | Enable verbose logging |
| --database, -d | str | None | Target database name |
| --offline | boolean | False | Use model state file instead of live database snapshot |
| --data | boolean | False | Also compare live declared data values. |

`--help` displays command help.

## `dbwarden config`

Display current Python configuration.

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden version`

Display dbwarden version and compatibility information.

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden lock-status`

Check if migration is currently locked.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden unlock`

Release the migration lock.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Target database name |
| --force, -f | boolean | False | Skip confirmation and force-release the lock |

`--help` displays command help.

## `dbwarden merge`

Merge divergent migration histories after a branch merge.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --dry-run | boolean | False | Preview reconciliation and superseded files without writing. |
| --split-at-severity | str | None | Split reconciliation at SAFE, INFO, WARN, or CRITICAL. |
| --database, -d | str | None | Target database name |
| --rename-column | str (repeatable) | None | Column rename to confirm (format: table.old=new) |
| --rename-table | str (repeatable) | None | Table rename to confirm (format: old=new) |
| --force, -f | boolean | False | Force marking hand-edited migrations |
| --commit | boolean | False | Create a git commit with the changes |
| --json | boolean | False | Output results as JSON |
| --verbose, -v | boolean | False | Enable verbose logging |

`--help` displays command help.

## `dbwarden rebase`

Recover a disposable environment after a merge.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Target database name |
| --yes, -y | boolean | False | Skip confirmation prompts |
| --force, -f | boolean | False | Force operation even against persistent environments |
| --check, --dry-run | boolean | False | Only check what would happen, don't make changes |
| --verbose, -v | boolean | False | Enable verbose logging |

`--help` displays command help.

## `dbwarden reconcile`

Recover a persistent environment after a dirty merge.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --force | boolean | False | Acknowledge operation risks in reconciliation. |
| --split-at-severity | str | None | Split reconciliation by severity. |
| environment | str | required | Environment name to reconcile |
| --database, -d | str | None | Target database name |
| --rename-column | str (repeatable) | None | Column rename to confirm |
| --dry-run | boolean | False | Only show what would happen |
| --verbose, -v | boolean | False | Enable verbose logging |

`--help` displays command help.

## `dbwarden database`

List configured databases

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden database list`

List all configured databases.

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden settings`

View dbwarden settings

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden settings show`

Show current settings configuration.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| database | str | None | Database name |
| --all | boolean | False | Show all databases |

`--help` displays command help.

## `dbwarden seed`

Manage seed data

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden seed create`

Create a new seed file.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| description | str | required | Description for the seed |
| --type, -t | str | 'sql' | Seed type: sql or python |
| --verbose, -v | boolean | False | Enable verbose logging |
| --database, -d | str | None | Target database name |

`--help` displays command help.

## `dbwarden seed apply`

Apply pending seeds.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --version, -v | str | None | Apply a specific seed version |
| --dry-run | boolean | False | Show what would be applied without executing |
| --database, -d | str | None | Target database name |
| --all, -a | boolean | False | Apply seeds on all databases |
| --verbose, -v | boolean | False | Enable verbose logging |

`--help` displays command help.

## `dbwarden seed list`

List seed files and their applied status.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Target database name |
| --all, -a | boolean | False | Show seeds for all databases |
| --verbose, -v | boolean | False | Enable verbose logging |
| --prune | boolean | False | Remove tracking records for seed files that no longer exist |

`--help` displays command help.

## `dbwarden seed export`

Export code seeds to ROC SQL files for stateless application.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None | Target database name |
| --all, -a | boolean | False | Export seeds for all configured databases |
| --output-dir, -o | str | 'seeds' | Output directory for ROC seed files (default: seeds/) |

`--help` displays command help.

## `dbwarden seed rollback`

Rollback seed tracking records (allows re-application).

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --count, -c | int | None | Number of seeds to rollback |
| --to-version, -t | str | None | Rollback to a specific version |
| --database, -d | str | None | Target database name |
| --all, -a | boolean | False | Rollback seeds in all databases |
| --verbose, -v | boolean | False | Enable verbose logging |

`--help` displays command help.

## `dbwarden plugin`

Manage dbwarden plugins

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden plugin categories`

List built-in and loaded plugin migration groups in execution order.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --format, -f | str | 'table' | Output format: table or json |

`--help` displays command help.

## `dbwarden plugin list`

List installed dbwarden plugins.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --format, -f | str | 'table' | Output format: table (default) or json |
| --load | boolean | False | Load trusted plugins to inspect registered handlers and migration categories. |

`--help` displays command help.

## `dbwarden plugin info`

Show plugin metadata.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| dist_name | str | required | Plugin distribution name |
| --format, -f | str | 'table' | Output format: table (default) or json |
| --load | boolean | False | Load trusted plugins to inspect registered handlers and migration categories. |

`--help` displays command help.

## `dbwarden plugin trust`

Allow a community plugin to load.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| dist_name | str | required | Community plugin distribution name |

`--help` displays command help.

## `dbwarden plugin untrust`

Remove trust for a community plugin.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| dist_name | str | required | Community plugin distribution name |

`--help` displays command help.

## `dbwarden plugin add`

Install a dbwarden plugin.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| dist_name | str | required | Plugin distribution name |
| --uv | boolean | False | Install with `uv add` instead of pip |
| --version | str | None | Pin an exact version to install (e.g. 1.2.0) |
| --dry-run | boolean | False | Show the install plan without installing |

`--help` displays command help.

## `dbwarden plugin remove`

Uninstall a dbwarden plugin.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| dist_name | str | required | Plugin distribution name |
| --uv | boolean | False | Uninstall with `uv remove` instead of pip |
| --dry-run | boolean | False | Show the uninstall plan without uninstalling |

`--help` displays command help.

## `dbwarden data`

Render and document declarative data migrations

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden data render`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None |  |
| --model | str | None |  |
| --format | str | 'text' |  |
| --include | str | 'schema,constraints,relationships,data,transitions' |  |
| --only-with-data | boolean | False |  |
| --relationship-depth | int range | 0 |  |
| --show-managed-values | boolean | False |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data describe`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --database, -d | str | None |  |
| --model | str | None |  |
| --format | str | 'text' |  |
| --output | path | None |  |
| --include | str | 'schema,constraints,relationships,data,transitions' |  |
| --only-with-data | boolean | False |  |
| --relationship-depth | int range | 0 |  |
| --show-managed-values | boolean | False |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data docs`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| --output | path | required |  |
| --database, -d | str | None |  |
| --diagrams | str | None |  |
| --show-managed-values | boolean | False |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data transition`

Validate, plan, and audit data transitions

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden data transition validate`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | None |  |
| --database, -d | str | None |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data transition plan`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --format | str | 'text' |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data transition dry-run`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --probes | boolean | False |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data transition render`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --format | str | 'text' |  |
| --show-managed-values | boolean | False |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data transition describe`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --format | str | 'text' |  |
| --output | path | None |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data transition audit`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --format | str | 'json' |  |

`--help` displays command help.

## `dbwarden data transition reconcile`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| migration_id | str | required |  |
| --database, -d | str | None |  |
| --apply | boolean | False |  |
| --decision | str | None |  |

`--help` displays command help.

## `dbwarden data transition new`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --manual | boolean | False |  |
| --from-table | str | None |  |
| --from-model | str | None |  |
| --source-snapshot | str | None |  |
| --to-model | str (repeatable) | None |  |
| --map | str (repeatable) | None |  |
| --source-key | str | None |  |
| --key | str (repeatable) | None |  |
| --where | str (repeatable) | None |  |
| --preserve-source, --drop-source | boolean | True |  |
| --acknowledge-drop | boolean | False |  |
| --coverage | str | 'all' |  |
| --overlap | str | 'error' |  |
| --priority | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data-transition`

Validate, plan, and audit data transitions

| Argument or option | Type | Default | Description |
|---|---|---|---|

`--help` displays command help.

## `dbwarden data-transition validate`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | None |  |
| --database, -d | str | None |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data-transition plan`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --format | str | 'text' |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data-transition dry-run`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --probes | boolean | False |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data-transition render`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --format | str | 'text' |  |
| --show-managed-values | boolean | False |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data-transition describe`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --format | str | 'text' |  |
| --output | path | None |  |
| --param | str (repeatable) | None |  |

`--help` displays command help.

## `dbwarden data-transition audit`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --format | str | 'json' |  |

`--help` displays command help.

## `dbwarden data-transition reconcile`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| migration_id | str | required |  |
| --database, -d | str | None |  |
| --apply | boolean | False |  |
| --decision | str | None |  |

`--help` displays command help.

## `dbwarden data-transition new`

Command group.

| Argument or option | Type | Default | Description |
|---|---|---|---|
| name | str | required |  |
| --database, -d | str | None |  |
| --manual | boolean | False |  |
| --from-table | str | None |  |
| --from-model | str | None |  |
| --source-snapshot | str | None |  |
| --to-model | str (repeatable) | None |  |
| --map | str (repeatable) | None |  |
| --source-key | str | None |  |
| --key | str (repeatable) | None |  |
| --where | str (repeatable) | None |  |
| --preserve-source, --drop-source | boolean | True |  |
| --acknowledge-drop | boolean | False |  |
| --coverage | str | 'all' |  |
| --overlap | str | 'error' |  |
| --priority | str (repeatable) | None |  |

`--help` displays command help.

========================================================================
PAGE: https://docs.dbwarden.org/reference/configuration-api/
========================================================================

# Configuration API Reference

Complete reference for the `DbwardenDatabase` declarative API and the equivalent
`database_config()` function.

This is a reference page. For step-by-step guides, see
[Quick Start](../configuration/quick-start.md), [Concepts](../configuration/concepts.md),
or [Production Patterns](../configuration/production-patterns.md).

Per-database safety scope settings:

| Setting | Default | Meaning |
|---|---|---|
| `split_at_severity` | `None` | Optional generation split threshold |
| `max_severity` | `"CRITICAL"` | Execution ceiling; CRITICAL is uncapped |
| `strict_pending` | `False` | Refuse generation with uncomposable pending plans |

Thresholds accept SAFE, INFO, WARN, or CRITICAL. CLI values override configuration.
Both declarative classes and `database_config()` expose the same fields.

## Declarative API (recommended)

Define a concrete subclass of `DbwardenDatabase`. Every field is available as a
class attribute, and concrete subclasses register automatically when imported.

```python
from dbwarden import DbwardenDatabase


class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:pass@localhost:5432/myapp"
    model_paths = ["app.models"]
```

### Inheritance

For configuration that benefits from inheritance, use an abstract base class:

```python
from dbwarden import DbwardenDatabase


class Shared(DbwardenDatabase):
    __abstract__ = True
    database_type = "postgresql"
    model_paths = ["app.models"]
    plugin_config = {
        "pg_roles": ["app_owner"],
    }


class Primary(Shared):
    database_name = "primary"
    database_url_sync = "postgresql://user:pass@localhost/myapp"
    default = True
```

### Plugin keys as class attributes

Plugin keys may be declared directly as class attributes:

```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    database_url_sync = "postgresql://user:pass@localhost/myapp"
    pg_roles = ["app_owner"]
```

`Primary.handle` is the `DatabaseHandle` returned by the function API. Mutable
values such as `model_paths`, `model_tables`, and `plugin_config` are copied for
each registered subclass, so child classes can override them without mutating
their base class.

## Function API

The equivalent `database_config()` function:

```python
def database_config(
    *,
    database_name: str,
    database_type: Literal["sqlite", "postgresql", "mysql", "mariadb", "clickhouse"] = "sqlite",
    database_url_sync: str | None = None,
    database_url_async: str | None = None,
    default: bool = False,
    migrations_dir: str | None = None,
    migration_table: str | None = None,
    seed_table: str | None = None,
    auto_apply_seeds: bool = False,
    model_paths: list[str] | None = None,
    model_tables: list[str] | None = None,
    data_paths: list[str] | None = None,
    data_snapshot_dir: str = ".dbwarden/data",
    snapshot_registry: str = ".dbwarden/snapshots/registry.json",
    dev_database_type: str | None = None,
    dev_database_url: str | None = None,
    overlap_models: bool = False,
    secure_values: bool = False,
    skip_if_missing: bool = False,
    pg_schema: str | None = None,
    pg_migration_lock_timeout: int | None = None,
    ch_cluster: str | None = None,
    ch_replicated_database: bool = False,
    clickhouse_lock_ttl: int | None = None,
    lock_namespace: str | None = None,
    migration_hooks: dict[str, list[Callable[..., Any]]] | None = None,
    environments: list[Any] | None = None,
    split_at_severity: str | None = None,
    max_severity: str = "CRITICAL",
    strict_pending: bool = False,
    **plugin_config: Any,
) -> DatabaseHandle:
    """Register a database in dbwarden and return a handle with session dependencies."""
```

`data_paths` names live modules containing `DataTransition` declarations. `*.data.py` artifacts are frozen migration inputs and are excluded from normal model discovery. `data_snapshot_dir` and `snapshot_registry` must be relative project paths. See [Declarative data migrations](../declarative-data-migrations.md).

## Required arguments

| Argument | Type | Description |
|----------|------|-------------|
| `database_name` | `str` | unique name for this database in your project |
| `database_type` | `str` | backend type: `sqlite`, `postgresql`, `mysql`, `mariadb`, or `clickhouse` (default: `"sqlite"`) |

At least one of `database_url_sync` or `database_url_async` must be provided.

## Optional arguments

| Argument | Type | Default | Description |
|----------|------|---------|-------------|
| `database_url_sync` | `str | None` | `None` | synchronous connection URL (used by migrations, CLI, and sync sessions) |
| `database_url_async` | `str | None` | `None` | async connection URL (used by async sessions; falls back to `database_url_sync` if omitted) |
| `default` | `bool` | `False` | if `True`, this database is used when `--database` is omitted |
| `skip_if_missing` | `bool` | `False` | if `True`, skip this database when its connection URL is not available (e.g., optional databases in multi-DB setups) |
| `migrations_dir` | `str | None` | `None` | custom migration directory path (defaults to `migrations/<database_name>`) |
| `migration_table` | `str | None` | `None` | custom migration tracking table name (defaults to `_dbwarden_migrations`) |
| `seed_table` | `str | None` | `None` | custom seed tracking table name (defaults to `_dbwarden_seeds`) |
| `auto_apply_seeds` | `bool` | `False` | if `True`, automatically apply pending code seeds after `migrate` |
| `model_paths` | `list[str] | None` | `None` | list of Python import paths containing SQLAlchemy models for this database |
| `model_tables` | `list[str] | None` | `None` | optional filter: only these table names are owned by this database |
| `dev_database_type` | `str | None` | `None` | backend type for local development (used with `--dev`) |
| `dev_database_url` | `str | None` | `None` | connection URL for local development (used with `--dev`) |
| `overlap_models` | `bool` | `False` | if `True`, allow model path overlap with other databases |
| `secure_values` | `bool` | `False` | if `True`, display commands show variable names instead of resolved values |
| `pg_schema` | `str | None` | `None` | PostgreSQL schema name; sets `search_path` on connection so all unqualified references use that schema |
| `pg_migration_lock_timeout` | `int | None` | `None` | PostgreSQL lock timeout in seconds for migration DDL; prevents indefinite waits on conflicting locks |
| `ch_cluster` | `str | None` | `None` | ClickHouse cluster name; appends `ON CLUSTER '<name>'` to every DDL statement. Mutually exclusive with `ch_replicated_database` |
| `ch_replicated_database` | `bool` | `False` | ClickHouse replicated database engine; DDL propagates automatically via ZooKeeper, `ON CLUSTER` must be omitted. Mutually exclusive with `ch_cluster` |
| `clickhouse_lock_ttl` | `int | None` | `None` | ClickHouse lease TTL in seconds (CH-0/CH-1). Defaults to 120 if not set. |
| `lock_namespace` | `str | None` | `None` | Lock scope. Allows independent lock streams. Defaults to `"default"` if not set. |
| `recovery_policy` | `str` | `"halt"` | Recovery policy for dead predecessor detection: `halt`, `resume_idempotent`, or `force` |
| `assume_session_pooling` | `bool` | `False` | If `True`, skip PgBouncer transaction-pooling detection (escape hatch for known-good session-pool deployments) |
| `tcp_keepalive` | `bool` | `True` | If `True`, enable TCP keepalive on migration connections (60s idle, 3 probes, 30s interval) |
| `sqlite_busy_timeout` | `int | None` | `None` | SQLite busy timeout in milliseconds. Default is 0 (fail fast). Set to a positive value to wait for the write lock. |
| `per_statement_history` | `bool` | `False` | If `True`, record history per-statement on non-transactional engines (MySQL, ClickHouse). Enables finer-grained recovery but may impact performance. |
| `rename_policy` | `str` | `"prompt"` | Rename handling policy for merges: `strict` (always require flags), `prompt` (interactive), `auto-high-confidence` (auto-accept high-confidence renames outside merges) |

## Field descriptions

### `database_name`

A unique identifier for this database within your project.

**Requirements:**

- Must be unique across all entries in your config source
- Used in CLI `--database` / `-d` flags to select this database
- Becomes part of migration filename prefix (for versioned migrations)

**Examples:**
```python
database_name="primary"
database_name="analytics"
database_name="legacy"
```

Use descriptive names that reflect the database's purpose: `primary`, `analytics`, `audit_logs`, etc.

### `database_type`

The database backend technology. Each value determines:

- URL parsing behavior
- SQL dialect and syntax handling
- Available features (transactions, DDL, constraints)

Valid values: `sqlite`, `postgresql`, `mysql`, `mariadb`, `clickhouse`

### `database_url_sync` and `database_url_async`

Connection URL strings in the format:

```
[dialect+driver://user:password@]host[:port][/database][?options]
```

- **`database_url_sync`**  used by CLI commands (`migrate`, `init`, etc.) and any sync session
- **`database_url_async`**  used by async sessions (FastAPI); falls back to `database_url_sync` if omitted

At least one must be provided. If only `database_url_sync` is given, async sessions will use it (with async driver substitution like `postgresql://...`  `postgresql+asyncpg://...`).

Examples:

```python
# Both sync and async (recommended for FastAPI projects)
database_url_sync = "postgresql://user:password@localhost:5432/mydb"
database_url_async = "postgresql+asyncpg://user:password@localhost:5432/mydb"

# Sync only (CLI-only projects)
database_url_sync = "postgresql://user:password@localhost:5432/mydb"

# SQLite (relative path)
database_url_sync = "sqlite:///./development.db"

# MySQL
database_url_sync = "mysql://user:password@localhost:3306/mydb"

# ClickHouse
database_url_sync = "http://user:password@clickhouse-host:8123/mydb"
```

### `default`

When `True`, this database is selected when `--database` / `-d` is not specified.

**Rule:** Exactly one entry must have `default=True`.

**Example:**
```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:pass@localhost:5432/mydb"

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://localhost:8123/analytics"
    # default=False implied
```

Exactly one database must have `default=True`. Having zero or multiple defaults will cause a validation error.

### `migrations_dir`

Path where this database's migration files are stored.

- Defaults to `migrations/<database_name>`
- Each database should have its own directory to avoid collision
- Versioned migration files go here (`NNNN_description.sql`)
- Repeatable migration files go here (`RA__*.sql`, `ROC__*.sql`)

### `model_paths`

A list of Python import paths where dbwarden should discover SQLAlchemy model definitions.

**When required:**

- **Single database:** Optional (dbwarden scans entire codebase)
- **Multiple databases:** Required for each database

**How it works:** dbwarden imports each path and inspects classes inheriting from `DeclarativeBase` or `declarative_base()`.

**Examples:**
```python
# Single module
model_paths=["app.models"]

# Multiple modules
model_paths=["app.models.primary", "app.legacy"]

# Nested modules
model_paths=["app.models.api.v1", "app.models.api.v2"]
```

Specifying `model_paths` makes discovery faster and more predictable, even for single-database projects.

See [Multi-Database Guide](../configuration/multi-database.md) for organizing models across databases.

### `model_tables`

A downstream filter applied after model discovery. Only tables whose string
name appears in this list are owned by this database. All other discovered
tables are ignored.

**When to use:**

- **Multi-database shared `model_paths`:** Two databases share the same
  import path but own different subsets of tables.
- **Selective deployment:** A microservice owns only a few tables from a
  shared models package.

**How it works:**
1. dbwarden discovers all models via `model_paths`
2. If `model_tables` is set, it validates every name exists among the
   discovered tables
3. Only the matching tables participate in migrations, diffs, and exports

**Example:**
```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    model_paths = ["app.models"]
    model_tables = ["users", "posts", "comments"]

class Audit(DbwardenDatabase):
    database_name = "audit"
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/audit"
    model_paths = ["app.models"]
    model_tables = ["audit_logs"]
```

**Overlap validation:** If two databases both set `model_tables` with
overlapping names, dbwarden raises an error (same behavior as
`model_paths` overlap). Set `overlap_models=True` to allow it.

**Must be valid SQL identifiers.** Dotted (schema-qualified) names are not
supported in the initial release.

### `migration_table`

Name of the table dbwarden uses to record applied migrations and repeatable migration checksums.

- Defaults to `_dbwarden_migrations`
- Must be a valid SQL identifier
- Applies per database entry
- Only affects migration tracking metadata; lock tables are separate

**Example:**

```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    migration_table = "custom_migrations"
```

Use this when:

- integrating with an existing database that already reserves a migrations table name
- isolating dbwarden metadata under a project-specific convention

### `seed_table`

Name of the table dbwarden uses to record applied seeds.

- Defaults to `_dbwarden_seeds`
- Must be a valid SQL identifier
- Applies per database entry

**Example:**

```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    seed_table = "custom_seeds"
```

Use this when integrating with an existing database that already reserves the seed table name.

### `auto_apply_seeds`

When `True`, dbwarden automatically applies pending code seeds after each successful `migrate` run.

- Defaults to `False`
- Applies per database entry
- Can be overridden per-run with `--apply-seeds` CLI flag on `migrate`

**Example:**

```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://localhost/myapp"
    auto_apply_seeds = True
```

Use this when:

- you want seeds to stay in sync with schema changes without manual `seed apply` steps
- deploying code seeds that define reference data or lookup tables
- running in CI/CD where every migration cycle should also re-seed

### `dev_database_type` and `dev_database_url`

These define an alternate connection for local development workflows.

When `--dev` is passed to any dbwarden command:

- `database_type` is swapped to `dev_database_type`
- `database_url_sync` / `database_url_async` are swapped to `dev_database_url`

**Benefits:**

- Use SQLite locally for speed (if production is PostgreSQL)
- Target a separate development database instance
- Test migrations safely before running against production
- Each developer has isolated database
- Easy to reset (just delete the file)

**Example:**
```python
class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://prod-host/myapp"
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./dev.db"
```

Use with:
```bash
$ dbwarden --dev migrate  # Uses SQLite
$ dbwarden migrate        # Uses PostgreSQL
```

Use SQLite with `dev_database_url="sqlite:///./dev.db"` for the fastest local iteration loop.

See [Dev Mode](../configuration/dev-mode.md) for complete workflow and patterns.

### `overlap_models`

By default, dbwarden prevents model path overlap between databases.

Set `overlap_models=True` when:

- Two databases legitimately share model definitions
- You understand the behavior implications (both databases will include overlapping tables)

### `secure_values`

When enabled, CLI display commands show the original variable/expression for non-literal arguments instead of resolved values.

**Use when:**

- Your config uses environment variables or expressions for secrets
- You want terminal output to avoid exposing credentials
- Running commands in CI/CD with logged output

**Example:**

```python
import os
from dbwarden import DbwardenDatabase

DATABASE_URL = os.getenv("DATABASE_URL")

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = DATABASE_URL
    secure_values = True  # Enable secure display
```

**Without `secure_values`:**
```bash
$ dbwarden settings show
URL: postgresql://user:SECRET_PASSWORD@prod-host/myapp
```

**With `secure_values=True`:**
```bash
$ dbwarden settings show --all
URL: DATABASE_URL (expression)
```

Always set `secure_values=True` in production to prevent credential exposure in logs.

### `ch_cluster`

ClickHouse cluster name. When set, dbwarden appends `ON CLUSTER '<name>'` to every DDL statement (CREATE, ALTER, DROP, RENAME, DETACH, ATTACH) for this database.

**Use when:**

- You have a multi-node ClickHouse cluster and want DDL distributed automatically
- You want explicit control over which DDL goes to which cluster

**Example:**

```python
class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://clickhouse1:8123/analytics"
    ch_cluster = "production_cluster"
```

**Validation:**

- Mutually exclusive with `ch_replicated_database` (setting both raises `ConfigurationError`)
- Must be a non-empty string when set
- Cluster name must match a cluster defined in ClickHouse's `remote_servers.xml`

See [ON Cluster](../databases/clickhouse/on-cluster.md) for full details on DDL injection, supported statement types, and the recreate pipeline.

### `ch_replicated_database`

When `True`, dbwarden uses the ClickHouse `Replicated` database engine. DDL propagates automatically through ZooKeeper, so `ON CLUSTER` must be omitted.

**Use when:**

- You want automatic DDL replication via ZooKeeper without explicit `ON CLUSTER` clauses
- Your tables use `Replicated*` engine variants

**Example:**

```python
class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://clickhouse1:8123/analytics"
    ch_replicated_database = True
```

**Validation:**

- Mutually exclusive with `ch_cluster` (setting both raises `ConfigurationError`)
- When `True`, use `Replicated*` engine variants (e.g., `replicated_merge_tree()`) in your models

See [ON Cluster](../databases/clickhouse/on-cluster.md) for the comparison between ON CLUSTER and replicated database modes.

### `clickhouse_lock_ttl`

Lease TTL in seconds for ClickHouse migration locking (CH-0 and CH-1 profiles). Controls how long a migration lock is held before auto-expiring. Default is 120 seconds.

**Use when:**

- Your migrations take longer than 2 minutes and the lock expires mid-run
- You want a shorter lock window for faster failover

**Example:**

```python
class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://clickhouse1:8123/analytics"
    clickhouse_lock_ttl = 600  # 10 minutes
```

### `lock_namespace`

Lock scope for migration locking. Allows independent lock streams so different operations (schema migrations, data operations) can run concurrently without conflicting.

**Use when:**

- You run schema migrations and data operations as separate processes
- You need independent lock streams for different deployment stages

**Example:**

```python
# Schema migrations
class SchemaDb(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://clickhouse1:8123/analytics"
    lock_namespace = "schema"

# Data operations
class DataDb(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://clickhouse1:8123/analytics"
    lock_namespace = "data"
```

## Return value: `DatabaseHandle`

`database_config()` returns a `DatabaseHandle` object with two
properties designed as FastAPI dependency annotations:

| Property | Resolves to (SQL) | Resolves to (ClickHouse) |
|----------|-------------------|--------------------------|
| `.async_session` | `Annotated[AsyncSession, Depends(...)]` | `Annotated[AsyncClient, Depends(...)]` |
| `.sync_session` | `Annotated[Session, Depends(...)]` | `Annotated[Client, Depends(...)]` |

Use them as type hints in FastAPI route parameters. The handle serves as
a namespace so you never confuse which database a session belongs to:

```python
# dbwarden.py
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:pass@localhost/myapp"

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://clickhouse1:8123/analytics"
```

```python
# routes.py
from ..dbwarden import Primary, Analytics

@router.get("/users")
async def get_users(session: Primary.handle.async_session):
    return await session.execute(...)

@router.get("/reports")
def get_reports(session: Analytics.handle.sync_session):
    return session.execute(...)
```

Use `.async_session` for async route handlers and `.sync_session` for sync
handlers. The deprecated `.session` property (aliased to `.async_session`)
will be removed in a future version.

`DatabaseHandle` is still useful as a typed container even without FastAPI. Access
`handle._name` and `handle._db_type` for the raw config values.

## Configuration rules (enforced at load time)

dbwarden validates your config to prevent dangerous misconfigurations:

| Rule | Error message (if violated) |
|------|---------------------------|
| Exactly one `default=True` | `Exactly one default=True required` |
| Unique `database_name` across all entries | `Duplicate database_name` |
| Unique `database_url_sync` across all entries | `Duplicate database_url_sync` |
| Unique physical target (even across credentials) | `Duplicate database target detected` |
| Required `model_paths` when multiple databases | `model_paths is required when more than one database is configured` |
| Explicit `overlap_models` when paths overlap | `model_paths overlap detected` |
| `model_tables` (if set) must not overlap across databases | `model_tables overlap detected` |
| If `dev_database_type` set, `dev_database_url` also required | `dev_database_url is required when dev_database_type is set` |

## Loading and resolution

Config is loaded by importing your Python config source and executing `database_config(...)` calls.

The resolution priority is:

1. Walk project tree for `dbwarden.py` files
2. If `DBWARDEN_CONFIG_MODULE` environment variable is set, use that module
3. Full scan for any file containing `database_config(...)` calls

If more than one discovery source is found, dbwarden fails with an ambiguity error.

`dbwarden.py` is the default convention, but it is not the only valid location. Any discovered Python file inside the project can call `database_config(...)`.

### Security sandbox

Config files are loaded with path traversal protection that ensures the file is within the project tree. See [Configuration Concepts  Config Loading Security](../configuration/concepts.md#config-loading-security-sandbox).

## Examples

### Minimal single-database setup

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/mydb"
```

### With local development (recommended)

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/mydb"
    dev_database_type = "sqlite"
    dev_database_url = "sqlite:///./development.db"
```

### Multi-database setup

```python
from dbwarden import DbwardenDatabase

class Primary(DbwardenDatabase):
    database_name = "primary"
    default = True
    database_type = "postgresql"
    database_url_sync = "postgresql://user:password@localhost:5432/main"
    model_paths = ["app.models.api"]

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "http://clickhouse:password@clickhouse-host:8123/analytics"
    model_paths = ["app.models.analytics"]
```

### ClickHouse with ON CLUSTER

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://clickhouse1:8123/analytics"
    model_paths = ["app.models.analytics"]
    ch_cluster = "production_cluster"
```

### ClickHouse with replicated database

```python
from dbwarden import DbwardenDatabase

class Analytics(DbwardenDatabase):
    database_name = "analytics"
    database_type = "clickhouse"
    database_url_sync = "clickhouse://clickhouse1:8123/analytics"
    model_paths = ["app.models.analytics"]
    ch_replicated_database = True
```

## Quick Reference

| Parameter | Required? | Default | Use When |
|-----------|-----------|---------|----------|
| `database_name` |  Yes | - | Always |
| `database_type` |  No | `"sqlite"` | Non-SQLite backends |
| `database_url_sync` |  Conditional | `None` | CLI or sync sessions |
| `database_url_async` |  No | `None` | Async sessions (FastAPI) |
| `default` |  No | `False` | Mark one database as default |
| `migrations_dir` |  No | `migrations/<name>` | Custom migration directory |
| `seed_table` |  No | `_dbwarden_seeds` | Custom seed tracking table |
| `auto_apply_seeds` |  No | `False` | Auto-apply seeds after migrate |
| `migration_table` |  No | `_dbwarden_migrations` | Custom migration tracking table |
| `model_paths` |  Conditional | `None` | Multi-database or explicit discovery |
| `model_tables` |  No | `None` | Filter discovered tables by name |
| `dev_database_type` |  No | `None` | Local development |
| `dev_database_url` |  No | `None` | Local development |
| `overlap_models` |  No | `False` | Shared models (read replicas) |
| `secure_values` |  No | `False` | Hide credentials in output |
| `pg_schema` |  No | `None` | PostgreSQL schema name for search_path |
| `pg_migration_lock_timeout` |  No | `None` | PostgreSQL lock timeout (seconds) for migrations |
| `ch_cluster` |  No | `None` | ClickHouse cluster name for ON CLUSTER DDL |
| `ch_replicated_database` |  No | `False` | ClickHouse replicated database engine |
| `clickhouse_lock_ttl` |  No | `None` | ClickHouse lease TTL in seconds |
| `lock_namespace` |  No | `None` | Lock scope for independent streams |

## Related Documentation

**Getting Started:**

- [Quick Start](../configuration/quick-start.md) - Your first configuration
- [Concepts](../configuration/concepts.md) - How configuration works

**Guides:**

- [Connection URLs](../configuration/connection-urls.md) - Database URL formats
- [Model Discovery](../configuration/model-discovery.md) - How `model_paths` works
- [Dev Mode](../configuration/dev-mode.md) - Local development
- [Multi-Database](../configuration/multi-database.md) - Multiple databases
- [Production Patterns](../configuration/production-patterns.md) - Real-world examples

**Help:**

- [Troubleshooting](../configuration/troubleshooting.md) - Common issues

========================================================================
PAGE: https://docs.dbwarden.org/reference/feature-map/
========================================================================

# Feature map

Use this page to choose a workflow. The [CLI option inventory](cli-options.md)
lists every built-in command and flag. The [Python API inventory](python-api.md)
lists declared exports, signatures, fields, and methods, including contributor
interfaces. An exported descriptor does not imply that core manages its server
object; plugin boundaries are listed below.

## Configuration and models

| Capability | Guide | Boundary |
|---|---|---|
| Declarative `DbwardenDatabase`, function `database_config`, project `DbwardenConfig` | [Configuration API](configuration-api.md) | One default database; plugin keys require registration |
| URLs, credentials, optional databases, environments, config discovery | [Configuration](../configuration/index.md) | Imported project code can execute; redacted output does not encrypt stored artifacts |
| Multiple databases and model ownership | [Multiple databases](../configuration/multi-database.md) | Separate targets and migration directories; overlap requires explicit configuration |
| SQLAlchemy models, table/column Meta, backend specs | [Models](../models.md) | Accepted fields and emitted operations vary by backend |
| Model discovery and selective tables | [Discovery](../configuration/model-discovery.md) | Importable mapped classes; avoid application startup side effects |
| Declarative managed rows, derivations, validations, historical transitions, frozen artifacts, and data journal | [Declarative data migrations](../declarative-data-migrations.md) | Versioned bundles use transactional or checkpointed execution; backend conformance evidence is separate from SQL generation |
| Reverse engineering with `generate-models` | [Schema inspection](../cookbook/05-schema-inspection.md) | Review inferred types, defaults, and unsupported objects |
| Pydantic schema generation with `auto_schema` | [Auto schemas](../cookbook/10-auto-schemas.md) | Runtime validation schemas are separate from migration state |

## Migration authoring and state

| Capability | Guide | Boundary |
|---|---|---|
| Model-based SQL generation, rename detection, explicit `new` migrations | [First migration](../getting-started/first-migration.md) | Generated SQL requires review; name similarity does not prove a rename |
| Versioned SQL, upgrade/rollback sections, RA/ROC repeatables | [Migration files](../migration-files.md) | Repeatables use execution/checksum rules; versioned migrations run in order |
| Offline export, diff, generation, and state recovery | [Offline workflow](../cookbook/04-offline-ci.md) | Saved state must correspond to migration history |
| Expand-contract type changes and severity splits | [Safety-scoped migrations](../correctness/safety-scoped-migrations.md) | SAFE/INFO/WARN/CRITICAL are safety levels; categories are named groups |
| Pending-plan composition, base/deferred state, strict pending checks | [Safety-scoped migrations](../correctness/safety-scoped-migrations.md) | Checksums and recorded before/after states must agree |
| Manual SQL classification with `check --write-plan` | [CLI guide](../cli-reference.md) | Static analysis can return UNKNOWN; it cannot establish live data safety |
| Merge detection, SQL/plan renumbering, reconcile, and rebase | [Merge handling](../advanced/merge-handling.md) | Persistent environments need reconciliation; plans and JSON state remain paired |
| Snapshots, checksums, history, status, recover-model-state | [Checksum integrity](../advanced/checksum-integrity.md) | Checksums detect changes; they do not prove the SQL is correct |

## Execution and inspection

| Capability | Guide | Boundary |
|---|---|---|
| Apply, rollback, downgrade, and make-rollback | [Apply and inspect](../cookbook/03-apply-and-inspect.md) | An inverse schema change cannot recover deleted data |
| Execution severity ceilings and repeatables | [Safety-scoped migrations](../correctness/safety-scoped-migrations.md) | A ceiling stops at a version prefix; `--force` does not raise it |
| Read-only diff, check-db, check-impact | [Safety and impact](../cookbook/06-safety-impact.md) | Reports use available schema/code evidence, not application correctness proofs |
| Data probes, convergence, journal reconciliation, and explicit reapply | [Declarative data migrations](../declarative-data-migrations.md) | `--data` adds live data checks; rolled-back data needs explicit reapply; uncertain durable effects require proof before retry |
| Lock-status, unlock, native locks, ClickHouse leases | [Locking](../advanced/migration-locking.md) | ClickHouse lease checks have weaker guarantees than native mutual exclusion |
| Sandboxed migration replay | [Workflows](../getting-started/workflows.md) | Core includes SQLite; external database providers require plugins |
| SQLite dev mode and SQL translation | [SQL translation](../sql-translation.md) | SQLite execution does not verify another server's behavior |
| JSON output, debugging, performance tracing, metrics, and logs | [Observability](../observability.md) | Dependencies and enabled hooks determine available telemetry |
| Skipped databases, partial success, and exit codes | [CLI guide](../cli-reference.md) | Exit code 3 needs result inspection; unresolved static plans use exit code 4 |

## Extensions and backend-specific operations

| Capability | Guide | Boundary |
|---|---|---|
| Plugin install, consent, trust, provenance, discovery, and removal | [Using plugins](../plugins/index.md) | Plugins execute in-process; trust labels are not a sandbox |
| Plugin configuration, value hooks, object handlers, ordering anchors | [Developing plugins](../plugins/developing/overview.md) | Follow the core/plugin API version contract |
| Base/deferred statements and custom named migration groups | [Plugin CLI](../plugins/reference/plugin-cli.md) | Registered group names do not change safety levels |
| Plugin CLI commands and category introspection | [Hook catalog](../plugins/reference/hook-catalog.md) | Installed plugins determine extra commands and groups |
| Seed descriptors, SQL/Python seed files, export/apply/rollback | [Seeds](../seeds.md) | Seed extensions may require `dbwarden-seeds`; inspect installed support |
| FastAPI sessions, lifespan, health, and management routes | [FastAPI](../cookbook/09-fastapi-integration.md) | External `dbwarden-fastapi` package; not implemented in this repository |
| PostgreSQL types, functions, extensions, RBAC, and object metadata | [PostgreSQL](../databases/postgresql/index.md) | Several config-owned objects require their corresponding plugins |
| MySQL metadata and MariaDB-specific fields | [MySQL and MariaDB](../databases/mysql/index.md) | MariaDB-specific emission is partial; review MODIFY preservation limits |
| SQLite rebuilds, generated columns, constraints, and table options | [SQLite](../databases/sqlite/index.md) | Rebuilds copy rows and can fail on changed constraints |
| ClickHouse engines, views, aggregation, dictionaries, projections, and indexes | [ClickHouse](../databases/clickhouse/index.md) | Server acceptance and convergence depend on version and schema |
| ClickHouse DataOp descriptors and populate helpers | [Data operations](../databases/clickhouse/data-operations.md) | Descriptors do not execute or automatically register SQL |
| ClickHouse RBAC and named collections | [RBAC](../databases/clickhouse/rbac.md) | Core exports specs; external plugin owns DDL and policy |

## Maintaining coverage

Run `python scripts/check-docs.py` after API, command, or documentation changes.
It checks declared exports, generated inventories, snippet syntax and supported
call signatures, CLI flags, links, anchors, and navigation. It does not execute
documentation snippets against external services. Use the relevant runtime
tests and a target-server integration run for behavior claims.

========================================================================
PAGE: https://docs.dbwarden.org/reference/migrate-from-toml/
========================================================================

# Migrate from TOML

If your project currently uses `warden.toml` for dbwarden configuration, this guide walks through transitioning to Python-based configuration with the default `DbwardenDatabase` API. The equivalent `database_config(...)` function remains supported.

## Why migrate

The Python configuration model offers:

- **Type validation** - catches misconfigurations at import time, not runtime
- **Runtime flexibility** - use environment variables, conditional logic, and expressions in your config
- **IDE support** - autocomplete, type hints, and inline documentation
- **Consistency** - same Python codebase powers your app and your migrations

## Before: TOML configuration

```toml
default = "primary"

[database.primary]
database_type = "postgresql"
sqlalchemy_url = "postgresql://user:password@localhost:5432/main"
migrations_dir = "migrations/primary"
model_paths = ["app.models.api"]
```

## After: Python configuration

```python
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    migrations_dir="migrations/primary",
    model_paths=["app.models.api"],
)
```

## What changed in field names

| TOML field | Python field | Notes |
|------------|--------------|-------|
| `default` | `default=True` | Boolean flag per entry instead of top-level |
| `database.<name>.database_type` | `database_type` | Passed directly to function |
| `database.<name>.sqlalchemy_url` | `database_url_sync` | Note: renamed for clarity |
| `database.<name>.migrations_dir` | `migrations_dir` | Same concept, different syntax |
| `database.<name>.model_paths` | `model_paths` | List syntax in Python |
| `database.<name>.dev_database_type` | `dev_database_type` | Optional dev swap entries |
| `database.<name>.dev_database_url` | `dev_database_url` | Optional dev swap entries |

## Migration checklist

### Step 1: Identify current databases

Check your existing `warden.toml`:

```bash
$ dbwarden database list
```

Note each database entry and its configuration.

### Step 2: Create new config source

Create your new config file (or update an existing config file that will contain `database_config(...)` calls).

Typical options:

- `dbwarden.py` (recommended for new projects)
- `app/core/config.py` (if you already have one)
- Any Python file that dbwarden can discover

### Step 3: Map each database entry

For each database in your TOML config, create a corresponding `database_config(...)` call.

**Example transformation:**

```python
# From TOML:
# [database.primary]
# database_type = "postgresql"
# sqlalchemy_url = "postgresql://user:password@localhost:5432/main"
# dev_database_type = "sqlite"
# dev_database_url = "sqlite:///./development.db"

# To Python:
from dbwarden import database_config


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

### Step 4: Verify configuration loads

```bash
$ dbwarden settings show --all
```

You should see all your migrated databases listed with correct types and URLs.

### Step 5: Test core commands

Confirm the migration works by running key commands:

```bash
$ dbwarden status --database primary
$ dbwarden history --database primary
```

### Step 6: Remove TOML file

Once verified, delete your old `warden.toml`:

```bash
rm warden.toml
```

dbwarden now uses only your Python configuration source.

## Advanced migration patterns

### Using environment variables

```python
import os


DATABASE_URL = os.getenv("DATABASE_URL")


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=DATABASE_URL,
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db",
)
```

This replaces the old pattern of reading URL from environment via TOML (which TOML cannot do natively).

### Using conditional configuration

```python
import os


ENV = os.getenv("ENVIRONMENT", "development")


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync="postgresql://user:password@localhost:5432/main",
    dev_database_type="sqlite",
    dev_database_url="sqlite:///./development.db" if ENV == "development" else None,
)
```

This is impossible with TOML but natural with Python.

### Adding `secure_values` for sensitive URLs

If your configuration uses variables or expressions for credentials:

```python
import os


DB_USER = os.getenv("DB_USER")
DB_PASS = os.getenv("DB_PASS")


primary = database_config(
    database_name="primary",
    default=True,
    database_type="postgresql",
    database_url_sync=f"postgresql://{DB_USER}:{DB_PASS}@localhost:5432/main",
    secure_values=True,
)
```

With `secure_values=True`, display commands show the expression rather than resolved credentials.

## Common issues during migration

### "Exactly one default=True required"

Ensure exactly one entry has `default=True`. Unlike TOML's top-level `default` field, you must set this on exactly one entry.

### "model_paths is required when more than one database is configured"

When using multiple databases, each needs explicit `model_paths` to keep model discovery boundaries clear.

### "Duplicate database_name" or "Duplicate database_url"

Python loads all `database_config(...)` calls, so ensure each call has a unique `database_name` and no duplicate URLs.

### TOML-specific features not supported

Some TOML features don't map 1:1 to Python:

- Inline tables → use separate key-value dictionary in Python (more verbose but explicit)
- Multiline strings → use triple-quoted strings in Python
- Arrays of tables → use separate `model_paths=[...]` list entries

If you used advanced TOML features, manually translate those to equivalent Python constructs.

## Verification commands after migration

After completing the migration, run these to confirm everything works:

```bash
# Confirm all databases visible
$ dbwarden settings show --all

# Confirm status works
$ dbwarden status --database <name>

# Confirm history works
$ dbwarden history --database <name>

# If using --dev, confirm it works
$ dbwarden --dev status --database <name>
```

## Rollback if needed

If migration causes issues, you can always:

1. Recreate `warden.toml` with the original configuration
2. Run dbwarden version that supports TOML (pre-0.5)

However, the Python-based approach is the recommended direction (0.9+) and offers significant benefits.

========================================================================
PAGE: https://docs.dbwarden.org/reference/python-api/
========================================================================

# Python API inventory

This inventory covers declared package exports, configuration, and runtime handles.
Signatures and declared fields come from the current source. Internal engine exports
are contributor interfaces, not a compatibility promise. Use the
[plugin development guide](../plugins/developing/overview.md) when choosing extension points.

For usage, see [configuration](configuration-api.md), [models](../models.md),
[plugin hooks](../plugins/reference/hook-catalog.md), and the backend guides.

Regenerate this inventory with `python scripts/check-docs.py --api-reference`.

## `dbwarden`

### `database_config`

```text
database_config(*, database_name: 'str', database_type: 'DatabaseType' = 'sqlite', database_url_sync: 'str | None' = None, database_url_async: 'str | None' = None, secure_values: 'bool' = False, skip_if_missing: 'bool' = False, default: 'bool' = False, migrations_dir: 'str | None' = None, migration_table: 'str | None' = None, model_paths: 'list[str] | None' = None, model_tables: 'list[str] | None' = None, data_paths: 'list[str] | None' = None, data_snapshot_dir: 'str' = '.dbwarden/data', snapshot_registry: 'str' = '.dbwarden/snapshots/registry.json', dev_database_type: 'DatabaseType | None' = None, dev_database_url: 'str | None' = None, overlap_models: 'bool' = False, auto_apply_seeds: 'bool' = False, seed_table: 'str | None' = None, pg_schema: 'str | None' = None, pg_migration_lock_timeout: 'int | None' = None, ch_cluster: 'str | None' = None, ch_replicated_database: 'bool' = False, clickhouse_lock_ttl: 'int | None' = None, lock_namespace: 'str | None' = None, migration_hooks: 'dict[str, list[Callable[..., Any]]] | None' = None, environments: 'list[Any] | None' = None, split_at_severity: 'str | None' = None, max_severity: 'str' = 'CRITICAL', strict_pending: 'bool' = False, **plugin_config: 'Any') -> 'DatabaseHandle'
```

### `EnvironmentConfig`

```text
EnvironmentConfig(name: 'str', url_env: 'str', persistent: 'bool') -> None
```

Declared fields: `name`, `url_env`, `persistent`.

### `DbwardenConfig`

```text
DbwardenConfig()
```

### `DbwardenDatabase`

```text
DbwardenDatabase()
```

### `ChEngineSpec`

```text
ChEngineSpec(name: 'str', args: 'tuple[str, ...]' = (), zookeeper_path: 'str | None' = None, replica_name: 'str | None' = None, settings: 'dict[str, str] | None' = None) -> None
```

Declared fields: `name`, `args`, `zookeeper_path`, `replica_name`, `settings`.

Methods declared on this class:

```text
to_dict(self) -> 'dict[str, Any]'
from_dict(d: 'dict') -> 'ChEngineSpec'
from_engine_string(engine_str: 'str') -> 'ChEngineSpec'
```

### `CHTableMeta`

```text
CHTableMeta()
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`, `ch_indexes`, `ch`, `ch_engine`, `ch_order_by`, `ch_primary_key`, `ch_partition_by`, `ch_sample_by`, `ch_ttl`, `ch_settings`, `ch_object_type`, `ch_select_statement`, `ch_to_table`, `ch_dictionary`, `ch_dict_layout`, `ch_dict_source`, `ch_dict_lifetime`, `ch_dict_primary_key`, `ch_projections`, `ch_zookeeper_path`, `ch_replica_name`.

### `MyColumnMeta`

```text
MyColumnMeta()
```

Declared fields: `comment`, `public`, `my`.

### `MyTableMeta`

```text
MyTableMeta()
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`, `my_engine`, `my_charset`, `my_collate`, `my_row_format`, `my_auto_increment`, `my_indexes`, `my_checks`, `my_uniques`.

### `PGViewMeta`

```text
PGViewMeta()
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`, `pg_view_query`, `pg_view_materialized`, `pg_view_auto_refresh`, `pg_schema`.

### `SqColumnMeta`

```text
SqColumnMeta()
```

Declared fields: `comment`, `public`, `sq`.

### `SqTableMeta`

```text
SqTableMeta()
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`, `sq_without_rowid`, `sq_strict`, `sq_indexes`.

### `load_plugins`

```text
load_plugins(*, interactive: 'bool' = False) -> 'None'
```

### `register_migration_hooks`

```text
register_migration_hooks(*, pre_migration_run: 'list[Callable[..., Any]] | None' = None, pre_migration: 'list[Callable[..., Any]] | None' = None, migration_progress: 'list[Callable[..., Any]] | None' = None, post_migration: 'list[Callable[..., Any]] | None' = None, on_migration_failure: 'list[Callable[..., Any]] | None' = None, post_migration_run: 'list[Callable[..., Any]] | None' = None) -> 'None'
```

### `Seed`

```text
Seed()
```

Declared fields: `model`, `rows`.

### `SeedRow`

```text
SeedRow(**kwargs: 'Any') -> 'None'
```

Declared fields: .

Methods declared on this class:

```text
to_dict(self) -> 'dict'
```

### `seed_data`

```text
seed_data(database: 'str', version: 'str' = '', description: 'str' = '', on_conflict: 'str' = 'ignore', conflict_columns: 'list[str] | None' = None)
```

## `dbwarden.cli`

### `app`

```text
app(*args: Any, **kwargs: Any) -> Any
```

## `dbwarden.commands.make_migrations`

### `RenameIntent`

```text
RenameIntent(table: str, old_name: str, new_name: str) -> None
```

Declared fields: `table`, `old_name`, `new_name`.

### `auto_discover_model_paths`

```text
auto_discover_model_paths() -> List[str]
```

### `build_migration_plan`

```text
build_migration_plan(migration_id: str, changes: list[dbwarden.engine.migration_name.Change], upgrade_sql: str, *, typed_ops: list[dict] | None = None, content: str | None = None, backend: str = '') -> dict[str, object]
```

### `generate_migration_sql`

```text
generate_migration_sql(tables: list, migrations_dir: str | None = None, database: str | None = None, db_name: str | None = None, confirmed_renames: set[tuple[str, str, str]] | None = None, resolved_from_map: dict[tuple[str, str, str], str] | None = None, safe_type_change: bool = False, confirmed_table_intents: set[tuple[str, str]] | None = None, table_resolved_from_map: dict[tuple[str, str], str] | None = None, concurrent: bool = True, clickhouse_engine_recreate: bool = False, drop_preserved_clickhouse_table: bool | None = None, postgres_auto_using: bool = False) -> tuple[str, str, list[dbwarden.engine.migration_name.Change]]
```

### `get_all_model_tables`

```text
get_all_model_tables(model_paths: List[str] | None = None, db_name: str | None = None) -> List[dbwarden.engine.core.models.ModelTable]
```

### `get_current_model_state_path`

```text
get_current_model_state_path(db_name: str | None = None) -> pathlib.Path
```

### `get_database`

```text
get_database(name: 'str | None' = None) -> 'DatabaseConfig'
```

### `get_model_state_path`

```text
get_model_state_path(db_name: str | None = None, legacy: bool = False) -> pathlib.Path
```

### `get_pending_migration_statements`

```text
get_pending_migration_statements(migrations_dir: str) -> set[str]
```

### `make_migrations_cmd`

```text
make_migrations_cmd(description: str | None = None, verbose: bool = False, database: str | None = None, output_plan: bool = False, output_sql: bool = False, rename_flags: list[str] | None = None, safe_type_change: bool = False, rename_table_flags: list[str] | None = None, concurrent: bool = True, offline: bool = False, migration_type: str = 'versioned', clickhouse_engine_recreate: bool = False, drop_preserved_clickhouse_table: bool | None = None, postgres_auto_using: bool = False, perf: bool = False, split_at_severity: str | None = None, strict_pending: bool | None = None, dry_run: bool = False, parameters: dict | None = None, show_managed_values: bool = False) -> None
```

### `new_migration_cmd`

```text
new_migration_cmd(description: str, version: str | None = None, database: str | None = None, migration_type: str = 'versioned') -> None
```

## `dbwarden.config_registry`

### `register_reset_hook`

```text
register_reset_hook(hook: 'Callable[[], None]') -> 'None'
```

### `reset_registry`

```text
reset_registry() -> 'None'
```

### `registered_entries`

```text
registered_entries() -> 'list[DatabaseEntry]'
```

### `registered_project_config`

```text
registered_project_config() -> 'ProjectConfigEntry | None'
```

### `database_config`

Re-export of `dbwarden.config_registry.database_config`.

### `DbwardenDatabase`

Re-export of `dbwarden.config_registry.DbwardenDatabase`.

### `DbwardenConfig`

Re-export of `dbwarden.config_registry.DbwardenConfig`.

## `dbwarden.config_schema`

### `DatabaseEntry`

```text
DatabaseEntry(database_name: 'str', database_type: 'DatabaseType', database_url_sync: 'str | None' = None, database_url_async: 'str | None' = None, secure_values: 'bool' = False, skip_if_missing: 'bool' = False, default: 'bool' = False, migrations_dir: 'str | None' = None, migration_table: 'str | None' = None, model_paths: 'list[str] | None' = None, model_tables: 'list[str] | None' = None, data_paths: 'list[str]' = NOTHING, data_snapshot_dir: 'str' = '.dbwarden/data', snapshot_registry: 'str' = '.dbwarden/snapshots/registry.json', dev_database_type: 'DatabaseType | None' = None, dev_database_url: 'str | None' = None, overlap_models: 'bool' = False, auto_apply_seeds: 'bool' = False, seed_table: 'str | None' = None, pg_schema: 'str | None' = None, pg_migration_lock_timeout: 'int | None' = None, ch_cluster: 'str | None' = None, ch_replicated_database: 'bool' = False, clickhouse_lock_ttl: 'int | None' = None, lock_namespace: 'str | None' = None, recovery_policy: 'str' = 'halt', assume_session_pooling: 'bool' = False, tcp_keepalive: 'bool' = True, sqlite_busy_timeout: 'int | None' = None, per_statement_history: 'bool' = False, rename_policy: 'str' = 'prompt', split_at_severity: 'str | None' = None, max_severity: 'str' = 'CRITICAL', strict_pending: 'bool' = False, migration_hooks: 'dict[str, list] | None' = None, environments: 'list[Any] | None' = None, plugin_config: 'dict[str, Any]' = NOTHING) -> None
```

Declared fields: `database_name`, `database_type`, `database_url_sync`, `database_url_async`, `secure_values`, `skip_if_missing`, `default`, `migrations_dir`, `migration_table`, `model_paths`, `model_tables`, `data_paths`, `data_snapshot_dir`, `snapshot_registry`, `dev_database_type`, `dev_database_url`, `overlap_models`, `auto_apply_seeds`, `seed_table`, `pg_schema`, `pg_migration_lock_timeout`, `ch_cluster`, `ch_replicated_database`, `clickhouse_lock_ttl`, `lock_namespace`, `recovery_policy`, `assume_session_pooling`, `tcp_keepalive`, `sqlite_busy_timeout`, `per_statement_history`, `rename_policy`, `split_at_severity`, `max_severity`, `strict_pending`, `migration_hooks`, `environments`, `plugin_config`.

### `ProjectConfigEntry`

```text
ProjectConfigEntry(pre_migrate_safety: 'ProjectPolicy' = 'off', pre_migrate_impact: 'ProjectPolicy' = 'off', missing_plan: 'ProjectPolicy' = 'off', impact_paths: 'list[str]' = NOTHING) -> None
```

Declared fields: `pre_migrate_safety`, `pre_migrate_impact`, `missing_plan`, `impact_paths`.

### `MultiDatabaseConfig`

```text
MultiDatabaseConfig(default: 'str', databases: 'dict[str, DatabaseEntry]') -> None
```

Declared fields: `default`, `databases`.

### `structure_database_entry`

```text
structure_database_entry(kwargs: 'dict') -> 'DatabaseEntry'
```

### `structure_project_config`

```text
structure_project_config(kwargs: 'dict[str, Any]') -> 'ProjectConfigEntry'
```

## `dbwarden.connection`

### `get_db_connection`

```text
get_db_connection(db_name: str | None = None) -> collections.abc.Generator[typing.Any, None, None]
```

### `reset_connection_logging`

```text
reset_connection_logging() -> None
```

### `QueryMethod`

```text
QueryMethod(*values)
```

### `get_query`

```text
get_query(method: dbwarden.connection.queries.store.QueryMethod, db_name: str | None = None, **kwargs) -> str
```

## `dbwarden.data`

### `ArchiveTable`

```text
ArchiveTable(table: 'str', schema: 'str | None' = None) -> None
```

Declared fields: `table`, `schema`.

### `DataMeta`

```text
DataMeta()
```

### `DataTransition`

```text
DataTransition()
```

### `aggregate`

```text
aggregate(policy, expression)
```

### `archive_table`

```text
archive_table(table: 'str', *, schema=None) -> 'ArchiveTable'
```

### `batch`

```text
batch(*, size, key, max_duration=None, statement_timeout=None, lock_timeout=None, max_replication_lag=None)
```

### `case`

```text
case(*whens: 'tuple[Any, Any]', else_: 'Any' = <object object>) -> 'Expr'
```

### `cast`

```text
cast(value: 'Any', type_: 'Any') -> 'Expr'
```

### `col`

```text
col(name: 'str', table: 'str | None' = None) -> 'Expr'
```

### `derive`

```text
derive(target, expression, *, rollback, when=None, on_unmatched='error', max_rows=None, category=None, settings=None, execution=None) -> 'Derivation'
```

### `from_source`

```text
from_source(source, *, identity, map, priority)
```

### `func`

```text
func
```

### `historical_table`

```text
historical_table(table: 'str', *, snapshot: 'str | None' = None, schema=None) -> 'HistoricalTable'
```

### `into`

```text
into(model, *, map, key=None, where=None, on_conflict='error', acknowledge_overwrite=False, priority=None) -> 'TransitionTarget'
```

### `literal`

```text
literal(value: 'Any') -> 'Expr'
```

### `mapping`

```text
mapping(value: 'Any', values: 'Mapping[Any, Any]', *, else_: 'Any' = <object object>) -> 'Expr'
```

### `merge_sources`

```text
merge_sources(*inputs, key, values, allow_empty=False)
```

### `param`

```text
param(name: 'str', type: 'Any' = None) -> 'Expr'
```

### `rows`

```text
rows(*, key=None, rows=None, source=None, owned_columns=None, on_missing='keep', scope=None, acknowledge_delete=False, archive_to=None, acknowledge_archive=False, rollback='irreversible', category=None) -> 'ManagedRows'
```

### `validate`

```text
validate(expression, *, message='Data validation failed') -> 'Validation'
```

### `winner`

```text
winner(expression)
```

## `dbwarden.database`

### `get_db_connection`

Re-export of `dbwarden.connection.connection.get_db_connection`.

### `reset_connection_logging`

Re-export of `dbwarden.connection.connection.reset_connection_logging`.

### `QueryMethod`

Re-export of `dbwarden.connection.queries.store.QueryMethod`.

### `get_query`

Re-export of `dbwarden.connection.queries.get_query`.

## `dbwarden.databases`

### `CheckSpec`

```text
CheckSpec(expression: 'str', name: 'str | None' = None, no_inherit: 'bool' = False) -> None
```

Declared fields: `expression`, `name`, `no_inherit`.

Methods declared on this class:

```text
as_dict(self) -> 'dict[str, Any]'
```

### `DBWardenMeta`

```text
DBWardenMeta(comment: 'str | None' = None, indexes: 'list[Any]' = <factory>, checks: 'list[Any]' = <factory>, uniques: 'list[Any]' = <factory>, primary_key: 'list[str]' = <factory>, partition: 'Any' = None, backend_table: 'Any' = None, pg_indexes: 'list[Any]' = <factory>, pg_checks: 'list[Any]' = <factory>, pg_uniques: 'list[Any]' = <factory>, pg_excludes: 'list[Any]' = <factory>, ch_indexes: 'list[Any]' = <factory>, my_indexes: 'list[Any]' = <factory>, my_checks: 'list[Any]' = <factory>, my_uniques: 'list[Any]' = <factory>, sq_indexes: 'list[Any]' = <factory>, table_attrs: 'dict[str, Any]' = <factory>) -> None
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`, `partition`, `backend_table`, `pg_indexes`, `pg_checks`, `pg_uniques`, `pg_excludes`, `ch_indexes`, `my_indexes`, `my_checks`, `my_uniques`, `sq_indexes`, `table_attrs`.

### `IndexSpec`

```text
IndexSpec(columns: 'list[str]', name: 'str | None' = None, unique: 'bool' = False, using: 'str | None' = None, where: 'str | None' = None, include: 'list[str] | None' = None, with_params: 'dict[str, Any] | None' = None, tablespace: 'str | None' = None, nulls_not_distinct: 'bool' = False, column_sorting: 'dict[str, str] | None' = None, comment: 'str | None' = None, concurrently: 'bool' = True, clickhouse_type: 'str | None' = None, clickhouse_granularity: 'int | None' = None) -> None
```

Declared fields: `columns`, `name`, `unique`, `using`, `where`, `include`, `with_params`, `tablespace`, `nulls_not_distinct`, `column_sorting`, `comment`, `concurrently`, `clickhouse_type`, `clickhouse_granularity`.

Methods declared on this class:

```text
to_dict(self) -> 'dict[str, Any]'
from_dict(d: 'dict') -> 'IndexSpec'
```

### `Seed`

Re-export of `dbwarden.seed.Seed`.

### `SeedRow`

Re-export of `dbwarden.seed.SeedRow`.

### `TableMeta`

```text
TableMeta()
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`.

### `UniqueSpec`

```text
UniqueSpec(columns: 'list[str]', name: 'str | None' = None, nulls_not_distinct: 'bool' = False, deferrable: 'bool' = False, initially_deferred: 'bool' = False, include: 'list[str] | None' = None) -> None
```

Declared fields: `columns`, `name`, `nulls_not_distinct`, `deferrable`, `initially_deferred`, `include`.

Methods declared on this class:

```text
as_dict(self) -> 'dict[str, Any]'
```

### `apply_meta`

```text
apply_meta(cls: 'type') -> 'None'
```

### `attach_meta`

```text
attach_meta(cls, incoming: 'DBWardenMeta') -> 'None'
```

### `ch`

```text
ch
```

### `check`

```text
check(name: 'str', expression: 'str', *, no_inherit: 'bool' = False) -> 'dict[str, Any]'
```

### `mdb`

```text
mdb
```

### `my`

```text
my
```

### `pg`

```text
pg
```

### `read_meta`

```text
read_meta(cls) -> 'DBWardenMeta | None'
```

### `seed_data`

Re-export of `dbwarden.seed.seed_data`.

### `sq`

```text
sq
```

### `unique`

```text
unique(name: 'str', columns: 'list[str]', *, nulls_not_distinct: 'bool' = False, deferrable: 'bool' = False, initially_deferred: 'bool' = False, include: 'list[str] | None' = None) -> 'dict[str, Any]'
```

## `dbwarden.databases.clickhouse`

### `AggregatingView`

```text
AggregatingView()
```

Declared fields: .

### `AggregatingViewSpec`

```text
AggregatingViewSpec(source: 'Any' = '', group_by: 'tuple[Any, ...]' = (), aggregates: 'tuple[Any, ...]' = (), order_by: 'tuple[Any, ...]' = (), partition_by: 'Any | None' = None, ttl: 'tuple[Any, ...] | None' = None, settings: 'dict[str, str] | None' = None, target_name: 'str | None' = None) -> None
```

Declared fields: `source`, `group_by`, `aggregates`, `order_by`, `partition_by`, `ttl`, `settings`, `target_name`.

Methods declared on this class:

```text
mv_name (property)
target_columns (property)
to_dict(self) -> 'dict[str, Any]'
```

### `AggExpr`

```text
AggExpr(func: 'str', arg: 'Any', arg_types: 'tuple[str, ...]', alias: 'str | None' = None) -> None
```

Declared fields: `func`, `arg`, `arg_types`, `alias`.

Methods declared on this class:

```text
as_(self, alias: 'str') -> 'AggExpr'
target_type(self) -> 'str'
state_combinator(self, source_column_type: 'str | None' = None) -> 'str'
```

### `ChAggStateType`

```text
ChAggStateType(func: 'str', *types: 'str') -> 'None'
```

### `ChEngineSpec`

Re-export of `dbwarden.databases.clickhouse.engine.ChEngineSpec`.

### `ChFieldSpec`

```text
ChFieldSpec(codec: 'str | None' = None, default_expression: 'str | None' = None, materialized: 'str | None' = None, alias: 'str | None' = None, ephemeral: 'str | None' = None, ttl: 'str | None' = None, low_cardinality: 'bool' = False, nullable: 'bool' = False, type: 'str | None' = None) -> None
```

Declared fields: `codec`, `default_expression`, `materialized`, `alias`, `ephemeral`, `ttl`, `low_cardinality`, `nullable`, `type`.

Methods declared on this class:

```text
to_col_info(self) -> 'dict[str, Any]'
```

### `ChGrantSpec`

```text
ChGrantSpec(privileges: 'tuple[str, ...]', on: 'str', to: 'str', with_grant_option: 'bool' = False) -> None
```

Declared fields: `privileges`, `on`, `to`, `with_grant_option`.

### `ChIndexSpec`

```text
ChIndexSpec(name: 'str', columns: 'list[str]', type: "Literal['minmax', 'set', 'bloom_filter', 'ngrambf_v1', 'tokenbf_v1', 'hypothesis'] | str" = 'minmax', granularity: 'int' = 1, expr: 'str | None' = None) -> None
```

Declared fields: `name`, `columns`, `type`, `granularity`, `expr`.

Methods declared on this class:

```text
to_dict(self) -> 'dict[str, Any]'
from_dict(d: 'dict') -> 'ChIndexSpec'
```

### `ChQuotaSpec`

```text
ChQuotaSpec(name: 'str', interval: 'str', limits: 'dict[str, int]', to_roles: 'tuple[str, ...]' = ('ALL',)) -> None
```

Declared fields: `name`, `interval`, `limits`, `to_roles`.

### `ChRaw`

```text
ChRaw(sql: 'str') -> None
```

Declared fields: `sql`.

### `ChRoleSpec`

```text
ChRoleSpec(name: 'str', settings: 'dict[str, str] | None' = None) -> None
```

Declared fields: `name`, `settings`.

### `ChRowPolicySpec`

```text
ChRowPolicySpec(name: 'str', table: 'str', using: 'str', to_roles: 'tuple[str, ...]' = ('ALL',), permissive: 'bool' = True) -> None
```

Declared fields: `name`, `table`, `using`, `to_roles`, `permissive`.

### `ChSettingsProfileSpec`

```text
ChSettingsProfileSpec(name: 'str', settings: 'dict[str, str]', to_roles: 'tuple[str, ...]' = ()) -> None
```

Declared fields: `name`, `settings`, `to_roles`.

### `ChTableSpec`

```text
ChTableSpec(engine: 'ChEngineSpec | str | None' = None, order_by: 'list[Any] | Any | None' = None, primary_key: 'list[Any] | Any | None' = None, partition_by: 'Any | None' = None, sample_by: 'Any | None' = None, ttl: 'list[Any] | Any | None' = None, settings: 'MergeTreeSettings | None' = None, projections: 'list[ProjectionSpec] | None' = None, indexes: 'list[ChIndexSpec] | None' = None, zookeeper_path: 'str | None' = None, replica_name: 'str | None' = None, select_statement: 'Any | None' = None, to_table: 'str | None' = None) -> None
```

Declared fields: `engine`, `order_by`, `primary_key`, `partition_by`, `sample_by`, `ttl`, `settings`, `projections`, `indexes`, `zookeeper_path`, `replica_name`, `select_statement`, `to_table`.

### `ChUserSpec`

```text
ChUserSpec(name: 'str', auth: 'str' = 'no_password', roles: 'tuple[str, ...]' = (), default_roles: 'tuple[str, ...]' = (), host: 'str' = 'ANY', settings_profile: 'str | None' = None) -> None
```

Declared fields: `name`, `auth`, `roles`, `default_roles`, `host`, `settings_profile`.

### `CHColumnMeta`

```text
CHColumnMeta()
```

Declared fields: `comment`, `public`, `ch`.

### `CHTableMeta`

Re-export of `dbwarden.schema.table_meta.CHTableMeta`.

### `CHViewMeta`

```text
CHViewMeta()
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`, `ch`.

### `ChView`

```text
ChView()
```

Declared fields: .

### `ClusterMode`

```text
ClusterMode(*values)
```

### `DataOp`

```text
DataOp(name: 'str', forward: 'str', rollback: 'str | None' = None, requires_confirmation: 'bool' = False) -> None
```

Declared fields: `name`, `forward`, `rollback`, `requires_confirmation`.

### `DictSpec`

```text
DictSpec(layout: 'str', source: 'dict[str, Any] | str', lifetime: 'int | str | None' = None, primary_key: 'str | list[str] | None' = None) -> None
```

Declared fields: `layout`, `source`, `lifetime`, `primary_key`.

Methods declared on this class:

```text
to_dict(self) -> 'dict[str, Any]'
```

### `KafkaSettings`

```text
KafkaSettings
```

Declared fields: `kafka_broker_list`, `kafka_topic_list`, `kafka_group_name`, `kafka_format`, `kafka_row_delimiter`, `kafka_schema`, `kafka_num_consumers`, `kafka_max_block_size`, `kafka_skip_broken_messages`, `kafka_commit_every_batch`, `kafka_client_id`, `kafka_poll_timeout_ms`, `kafka_flush_interval_ms`, `kafka_thread_per_consumer`, `kafka_handle_error_mode`.

### `MaterializedView`

```text
MaterializedView()
```

Declared fields: .

### `MaterializedViewSpec`

```text
MaterializedViewSpec(name: 'str | None' = None, select: 'Any' = None, to: 'str | None' = None, refresh: 'str | None' = None, populate: 'bool' = False, engine: 'Any' = None, order_by: 'Any' = None, partition_by: 'Any' = None, ttl: 'Any' = None, settings: 'dict[str, str] | None' = None) -> None
```

Declared fields: `name`, `select`, `to`, `refresh`, `populate`, `engine`, `order_by`, `partition_by`, `ttl`, `settings`.

Methods declared on this class:

```text
to_dict(self) -> 'dict[str, Any]'
```

### `MergeTreeSettings`

```text
MergeTreeSettings
```

Declared fields: `index_granularity`, `index_granularity_bytes`, `min_index_granularity_bytes`, `enable_mixed_granularity_parts`, `ttl_only_drop_parts`, `merge_with_ttl_timeout`, `merge_with_recompression_ttl_timeout`, `write_final_mark`, `merge_max_block_size`, `max_part_loading_threads`, `max_parts_in_total`, `replicated_deduplication_window`, `replicated_deduplication_window_seconds`, `replicated_can_become_leader`, `min_bytes_for_wide_part`, `min_rows_for_wide_part`, `max_parts_in_memory`, `max_bytes_to_merge_at_max_space_in_pool`, `min_bytes_to_use_direct_io`, `merge_tree_clear_old_broken_detached`, `parts_to_throw_insert`, `parts_to_delay_insert`, `inactive_parts_to_throw_in_insert`, `inactive_parts_to_delay_in_insert`, `max_delay_to_insert`, `max_suspicious_broken_parts`.

### `NamedCollectionSpec`

```text
NamedCollectionSpec(name: 'str', entries: 'dict[str, str]' = <factory>, overridable: 'dict[str, bool] | None' = None) -> None
```

Declared fields: `name`, `entries`, `overridable`.

Methods declared on this class:

```text
to_dict(self) -> 'dict'
```

### `NATSSettings`

```text
NATSSettings
```

Declared fields: `nats_url`, `nats_subjects`, `nats_format`, `nats_num_consumers`, `nats_max_block_size`, `nats_skip_broken_messages`, `nats_flush_interval_ms`, `nats_commit_every_batch`, `nats_username`, `nats_password`.

### `ProjectionSpec`

```text
ProjectionSpec(name: 'str', query: 'str') -> None
```

Declared fields: `name`, `query`.

Methods declared on this class:

```text
to_dict(self) -> 'dict[str, Any]'
from_dict(d: 'dict') -> 'ProjectionSpec'
```

### `RabbitMQSettings`

```text
RabbitMQSettings
```

Declared fields: `rabbitmq_host`, `rabbitmq_port`, `rabbitmq_virtual_host`, `rabbitmq_username`, `rabbitmq_password`, `rabbitmq_routing_key_list`, `rabbitmq_exchange_name`, `rabbitmq_format`, `rabbitmq_exchange_type`, `rabbitmq_routing_key`, `rabbitmq_num_consumers`, `rabbitmq_max_block_size`, `rabbitmq_flush_interval_ms`, `rabbitmq_skip_broken_messages`, `rabbitmq_commit_every_batch`, `rabbitmq_queue_base`, `rabbitmq_persistent`.

### `RedisSettings`

```text
RedisSettings
```

Declared fields: `redis_host`, `redis_port`, `redis_password`, `redis_storage`.

### `S3QueueSettings`

```text
S3QueueSettings
```

Declared fields: `s3queue_buckets`, `s3queue_processing_threads`, `s3queue_tracking_poll_timeout_ms`, `s3queue_tracking_window_ms`, `s3queue_last_processed_node`.

### `S3Settings`

```text
S3Settings
```

Declared fields: `s3_access_key_id`, `s3_secret_access_key`, `s3_format`, `s3_compression`, `s3_compression_method`, `s3_compression_level`.

### `URLSettings`

```text
URLSettings
```

Declared fields: `url`, `url_format`.

### `agg`

```text
agg
```

### `aggregating_merge_tree`

```text
aggregating_merge_tree(*, settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `aggregating_view`

```text
aggregating_view(*, source: 'Any', group_by: 'Any' = None, aggregates: 'Any' = None, order_by: 'Any' = None, partition_by: 'Any' = None, ttl: 'Any' = None, settings: 'dict[str, str] | None' = None, name: 'str | None' = None, **deprecated: 'Any') -> 'AggregatingViewSpec'
```

### `buffer`

```text
buffer(database: 'str', table: 'str', num_layers: 'int', min_time: 'int', max_time: 'int', min_rows: 'int', max_rows: 'int', min_bytes: 'int', max_bytes: 'int', *, settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `ch`

```text
ch
```

### `ch_agg_state`

```text
ch_agg_state(func: 'str', *types: 'str') -> 'ChAggStateType'
```

### `ch_raw`

```text
ch_raw(sql: 'str') -> 'ChRaw'
```

### `ch_table`

```text
ch_table(*, engine: 'ChEngineSpec | str | None' = None, order_by: 'Any' = None, primary_key: 'Any' = None, partition_by: 'Any' = None, sample_by: 'Any' = None, ttl: 'Any' = None, settings: 'MergeTreeSettings | None' = None, projections: 'list[ProjectionSpec] | None' = None, indexes: 'list[ChIndexSpec] | None' = None, zookeeper_path: 'str | None' = None, replica_name: 'str | None' = None) -> 'ChTableSpec'
```

### `ch_view_tables_from_models`

```text
ch_view_tables_from_models(model_paths: 'list[str] | None' = None, db_name: 'str | None' = None) -> 'list[ModelTable]'
```

### `collapsing_merge_tree`

```text
collapsing_merge_tree(sign_col: 'str', *, settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `column_name_from_expr`

```text
column_name_from_expr(expr: 'ColumnElement | str | Any') -> 'str'
```

### `data_op`

```text
data_op(*, name: 'str', forward: 'str', rollback: 'str | None' = None, requires_confirmation: 'bool' = False) -> 'DataOp'
```

### `data_ops`

```text
data_ops
```

### `data_ops_populate`

```text
data_ops_populate(spec: 'MaterializedViewSpec | AggregatingViewSpec | dict[str, Any]', *, name: 'str | None' = None, rollback: 'str | None' = None) -> 'DataOp'
```

### `derive_agg_target_columns`

```text
derive_agg_target_columns(agg_result: 'AggregatingViewSpec | dict') -> 'list[str]'
```

### `dictionary`

```text
dictionary(*, layout: 'str', source: 'dict[str, Any] | str', lifetime: 'int | str | None' = None, primary_key: 'str | list[str] | None' = None) -> 'DictSpec'
```

### `dictionary_engine`

```text
dictionary_engine(dict_name: 'str') -> 'ChEngineSpec'
```

### `distributed`

```text
distributed(cluster: 'str', database: 'str', table: 'str', sharding_key: 'str | None' = None, *, policy_name: 'str | None' = None, settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `field`

```text
field(*, codec: 'str | None' = None, default_expression: 'str | None' = None, materialized: 'str | None' = None, alias: 'str | None' = None, ephemeral: 'str | None' = None, ttl: 'str | None' = None, low_cardinality: 'bool' = False, nullable: 'bool' = False, type: 'str | None' = None) -> 'ChFieldSpec'
```

### `file_engine`

```text
file_engine(format: 'str', path: 'str | None' = None) -> 'ChEngineSpec'
```

### `get_all_ch_views`

```text
get_all_ch_views(model_paths: 'list[str] | None' = None, db_name: 'str | None' = None) -> 'list[dict[str, Any]]'
```

### `graphite_merge_tree`

```text
graphite_merge_tree(config_section: 'str' = 'default', *, settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `hdfs`

```text
hdfs(uri: 'str', format: 'str') -> 'ChEngineSpec'
```

### `join_engine`

```text
join_engine(strictness: 'str', kind: 'str', *key_cols: 'str') -> 'ChEngineSpec'
```

### `kafka`

```text
kafka(*, named_collection: 'str | None' = None, broker_list: 'str | None' = None, topic_list: 'str | None' = None, group_name: 'str | None' = None, format: 'str | None' = None, num_consumers: 'int | None' = None) -> 'ChEngineSpec'
```

### `log`

```text
log() -> 'ChEngineSpec'
```

### `materialized_view`

```text
materialized_view(*, name: 'str | None' = None, select: 'Any' = None, to: 'str | None' = None, refresh: 'str | None' = None, populate: 'bool' = False, engine: 'Any' = None, order_by: 'Any' = None, partition_by: 'Any' = None, ttl: 'Any' = None, settings: 'dict[str, str] | None' = None, **deprecated: 'Any') -> 'MaterializedViewSpec'
```

### `memory`

```text
memory() -> 'ChEngineSpec'
```

### `merge`

```text
merge(db_regex: 'str', table_regex: 'str') -> 'ChEngineSpec'
```

### `merge_tree`

```text
merge_tree(*, settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `mongodb`

```text
mongodb(host: 'str', port: 'int', database: 'str', table: 'str', user: 'str', password: 'str') -> 'ChEngineSpec'
```

### `mysql_engine`

```text
mysql_engine(host: 'str', port: 'int', database: 'str', table: 'str', user: 'str', password: 'str') -> 'ChEngineSpec'
```

### `named_collection`

```text
named_collection(name: 'str', *, overridable: 'dict[str, bool] | None' = None, **entries: 'str') -> 'NamedCollectionSpec'
```

### `nats`

```text
nats(*, named_collection: 'str | None' = None, url: 'str | None' = None, subjects: 'str | None' = None, format: 'str | None' = None) -> 'ChEngineSpec'
```

### `null`

```text
null() -> 'ChEngineSpec'
```

### `postgresql_engine`

```text
postgresql_engine(host: 'str', port: 'int', database: 'str', table: 'str', user: 'str', password: 'str') -> 'ChEngineSpec'
```

### `projection`

```text
projection(name: 'str', query: 'str') -> 'ProjectionSpec'
```

### `rabbitmq`

```text
rabbitmq(*, named_collection: 'str | None' = None, host: 'str | None' = None, format: 'str | None' = None, exchange_name: 'str | None' = None, routing_key: 'str | None' = None) -> 'ChEngineSpec'
```

### `redis`

```text
redis(host: 'str', port: 'int', password: 'str', storage: 'str | int') -> 'ChEngineSpec'
```

### `render_expr`

```text
render_expr(expr: 'ColumnElement | str | Any') -> 'str'
```

### `render_expr_list`

```text
render_expr_list(items: 'list[ColumnElement | str | Any] | None') -> 'list[str]'
```

### `replicated_merge_tree`

```text
replicated_merge_tree(zookeeper_path: 'str', replica_name: 'str', *args: 'str', settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `replacing_merge_tree`

```text
replacing_merge_tree(version_col: 'str | None' = None, *, settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `s3`

```text
s3(*, named_collection: 'str | None' = None, path: 'str | None' = None, format: 'str | None' = None, compression: 'str | None' = None, access_key_id: 'str | None' = None, secret_access_key: 'str | None' = None) -> 'ChEngineSpec'
```

### `s3_queue`

```text
s3_queue(*, named_collection: 'str | None' = None, path: 'str | None' = None, format: 'str | None' = None, compression: 'str | None' = None) -> 'ChEngineSpec'
```

### `set_engine`

```text
set_engine() -> 'ChEngineSpec'
```

### `skip_index`

```text
skip_index(name: 'str', columns: 'list[str]', type: "Literal['minmax', 'set', 'bloom_filter', 'ngrambf_v1', 'tokenbf_v1', 'hypothesis'] | str" = 'minmax', *, granularity: 'int' = 1, expr: 'str | None' = None) -> 'ChIndexSpec'
```

### `stripe_log`

```text
stripe_log() -> 'ChEngineSpec'
```

### `summing_merge_tree`

```text
summing_merge_tree(*columns: 'str', settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

### `tiny_log`

```text
tiny_log() -> 'ChEngineSpec'
```

### `url_engine`

```text
url_engine(url: 'str', format: 'str') -> 'ChEngineSpec'
```

### `versioned_collapsing_merge_tree`

```text
versioned_collapsing_merge_tree(sign_col: 'str', version_col: 'str', *, settings: 'MergeTreeSettings | None' = None) -> 'ChEngineSpec'
```

## `dbwarden.databases.mariadb`

### `MdbColumnMeta`

```text
MdbColumnMeta()
```

Declared fields: `comment`, `public`, `mdb`.

### `MdbFieldSpec`

```text
MdbFieldSpec(invisible: 'bool' = False, without_overlaps: 'bool' = False, sequence: 'str | None' = None) -> None
```

Declared fields: `invisible`, `without_overlaps`, `sequence`.

Methods declared on this class:

```text
to_col_info(self) -> 'dict[str, Any]'
```

### `MdbTableMeta`

```text
MdbTableMeta()
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`, `my_engine`, `my_charset`, `my_collate`, `my_row_format`, `my_auto_increment`, `my_indexes`, `my_checks`, `my_uniques`, `mdb_page_compressed`, `mdb_page_compression_level`.

### `MdbTableSpec`

```text
MdbTableSpec(engine: 'str' = 'InnoDB', charset: 'str' = 'utf8mb4', collate: 'str' = 'utf8mb4_unicode_ci', row_format: 'str | None' = None, auto_increment: 'int | None' = None, page_compressed: 'bool' = False, page_compression_level: 'int | None' = None) -> None
```

Declared fields: `engine`, `charset`, `collate`, `row_format`, `auto_increment`, `page_compressed`, `page_compression_level`.

### `field`

```text
field(*, invisible: 'bool' = False, without_overlaps: 'bool' = False, sequence: 'str | None' = None) -> 'MdbFieldSpec'
```

### `mdb`

```text
mdb
```

## `dbwarden.databases.mysql`

### `MyColumnMeta`

Re-export of `dbwarden.schema.table_meta.MyColumnMeta`.

### `MyFieldSpec`

```text
MyFieldSpec(charset: 'str | None' = None, collate: 'str | None' = None, unsigned: 'bool' = False, on_update: 'str | None' = None) -> None
```

Declared fields: `charset`, `collate`, `unsigned`, `on_update`.

Methods declared on this class:

```text
to_col_info(self) -> 'dict[str, Any]'
```

### `MyTableMeta`

Re-export of `dbwarden.schema.table_meta.MyTableMeta`.

### `MyTableSpec`

```text
MyTableSpec(engine: 'str' = 'InnoDB', charset: 'str' = 'utf8mb4', collate: 'str' = 'utf8mb4_unicode_ci', row_format: 'str | None' = None, auto_increment: 'int | None' = None) -> None
```

Declared fields: `engine`, `charset`, `collate`, `row_format`, `auto_increment`.

### `field`

```text
field(*, charset: 'str | None' = None, collate: 'str | None' = None, unsigned: 'bool' = False, on_update: 'str | None' = None) -> 'MyFieldSpec'
```

### `my`

```text
my
```

## `dbwarden.databases.pgsql`

### `ExcludeSpec`

```text
ExcludeSpec(name: 'str | None' = None, using: 'str' = 'gist', where: 'str | None' = None, elements: 'list[dict[str, Any]] | None' = None) -> None
```

Declared fields: `name`, `using`, `where`, `elements`.

Methods declared on this class:

```text
as_dict(self) -> 'dict[str, Any]'
```

### `PGColumnMeta`

```text
PGColumnMeta()
```

Declared fields: `comment`, `public`, `pg`.

### `PGTableMeta`

```text
PGTableMeta()
```

Declared fields: `comment`, `indexes`, `checks`, `uniques`, `primary_key`, `pg_schema`, `pg_fillfactor`, `pg_tablespace`, `pg_inherits`, `pg_partition_of`, `pg_partition_bound`, `pg_unlogged`, `pg_checks`, `pg_uniques`, `pg_excludes`, `pg_indexes`, `pg_partition`, `pg_rls`, `pg_rls_force`, `pg_policies`, `pg_grants`, `pg_storage_params`.

### `PGViewMeta`

Re-export of `dbwarden.schema.table_meta.PGViewMeta`.

### `PgFieldSpec`

```text
PgFieldSpec(collation: 'str | None' = None, storage: 'str | None' = None, compression: 'str | None' = None, generated: 'str | None' = None, identity: 'str | None' = None, identity_start: 'int | None' = None, identity_increment: 'int | None' = None, identity_min: 'int | None' = None, identity_max: 'int | None' = None) -> None
```

Declared fields: `collation`, `storage`, `compression`, `generated`, `identity`, `identity_start`, `identity_increment`, `identity_min`, `identity_max`.

Methods declared on this class:

```text
to_col_info(self) -> 'dict[str, Any]'
```

### `PgIndexSpec`

```text
PgIndexSpec(name: 'str', columns: 'list[str]', unique: 'bool' = False, using: 'str | None' = None, where: 'str | None' = None, include: 'list[str] | None' = None, with_params: 'dict[str, Any] | None' = None, tablespace: 'str | None' = None, nulls_not_distinct: 'bool' = False, column_sorting: 'dict[str, str] | None' = None, postgresql_ops: 'dict[str, str] | None' = None, concurrently: 'bool' = True, expression: 'str | None' = None) -> None
```

Declared fields: `name`, `columns`, `unique`, `using`, `where`, `include`, `with_params`, `tablespace`, `nulls_not_distinct`, `column_sorting`, `postgresql_ops`, `concurrently`, `expression`.

Methods declared on this class:

```text
to_dict(self) -> 'dict[str, Any]'
from_dict(d: 'dict') -> 'PgIndexSpec'
```

### `PgTableSpec`

```text
PgTableSpec(tablespace: 'str | None' = None, fillfactor: 'int | None' = None, unlogged: 'bool' = False, inherits: 'list[str] | None' = None, schema: 'str | None' = None, partition: 'dict | None' = None) -> None
```

Declared fields: `tablespace`, `fillfactor`, `unlogged`, `inherits`, `schema`, `partition`.

### `PgViewSpec`

```text
PgViewSpec(query: 'str | None' = None, materialized: 'bool' = False, schema: 'str | None' = None, auto_refresh: 'bool' = False) -> None
```

Declared fields: `query`, `materialized`, `schema`, `auto_refresh`.

### `exclude`

```text
exclude(name: 'str', *, using: 'str' = 'gist', where: 'str | None' = None, elements: 'list[dict[str, Any]] | None' = None) -> 'dict[str, Any]'
```

### `field`

```text
field(*, collation: 'str | None' = None, storage: 'str | None' = None, compression: 'str | None' = None, generated: 'str | None' = None, identity: 'str | None' = None, identity_start: 'int | None' = None, identity_increment: 'int | None' = None, identity_min: 'int | None' = None, identity_max: 'int | None' = None) -> 'PgFieldSpec'
```

### `index`

```text
index(name: 'str', columns: 'list[str]', *, unique: 'bool' = False, using: 'str | None' = None, where: 'str | None' = None, include: 'list[str] | None' = None, with_params: 'dict[str, Any] | None' = None, tablespace: 'str | None' = None, nulls_not_distinct: 'bool' = False, column_sorting: 'dict[str, str] | None' = None, postgresql_ops: 'dict[str, str] | None' = None, concurrently: 'bool' = True, expression: 'str | None' = None) -> 'dict[str, Any]'
```

### `partition_by_hash`

```text
partition_by_hash(column: 'str', partitions: 'int') -> 'dict[str, Any]'
```

### `partition_by_list`

```text
partition_by_list(column: 'str') -> 'dict[str, Any]'
```

### `partition_by_range`

```text
partition_by_range(column: 'str', *, interval: 'str | None' = None) -> 'dict[str, Any]'
```

### `pg`

```text
pg
```

## `dbwarden.databases.sqlite`

### `SqColumnMeta`

Re-export of `dbwarden.schema.table_meta.SqColumnMeta`.

### `SqFieldSpec`

```text
SqFieldSpec(generated: 'str | None' = None, generated_mode: 'str' = 'STORED', collate: 'str | None' = None) -> None
```

Declared fields: `generated`, `generated_mode`, `collate`.

Methods declared on this class:

```text
to_col_info(self) -> 'dict[str, Any]'
```

### `SqTableMeta`

Re-export of `dbwarden.schema.table_meta.SqTableMeta`.

### `SqTableSpec`

```text
SqTableSpec(without_rowid: 'bool' = False, strict: 'bool' = False) -> None
```

Declared fields: `without_rowid`, `strict`.

### `field`

```text
field(*, generated: 'str | None' = None, generated_mode: 'str' = 'STORED', collate: 'str | None' = None) -> 'SqFieldSpec'
```

### `sq`

```text
sq
```

## `dbwarden.db_handle`

### `DatabaseHandle`

```text
DatabaseHandle(name: 'str', db_type: 'str') -> 'None'
```

Methods declared on this class:

```text
session (property)
```

## `dbwarden.engine.backends.clickhouse`

### `Safety`

```text
Safety(*values)
```

### `analyze_clickhouse_options`

```text
analyze_clickhouse_options(table_snapshot: 'dict[str, Any]', model_table: 'ModelTable') -> 'list[SafetyIssue]'
```

### `classify_ch_column_change`

```text
classify_ch_column_change(key: 'str', *, change: 'dict | None' = None) -> 'Safety'
```

### `classify_ch_options_change`

```text
classify_ch_options_change(key: 'str') -> 'Safety'
```

### `classify_ch_safety`

```text
classify_ch_safety(op: 'dict', model_column: 'Any' = None, snapshot_column: 'Any' = None) -> 'list[SafetyIssue]'
```

### `extract_balanced_parens`

```text
extract_balanced_parens(match: 're.Match') -> 'str | None'
```

### `parse_dict_layout`

```text
parse_dict_layout(create_query: 'str') -> 'str | None'
```

### `parse_dict_lifetime`

```text
parse_dict_lifetime(create_query: 'str') -> 'int | str | None'
```

### `parse_dict_primary_key`

```text
parse_dict_primary_key(create_query: 'str') -> 'str | None'
```

### `parse_dict_source`

```text
parse_dict_source(create_query: 'str') -> 'str | None'
```

### `parse_mv_query`

```text
parse_mv_query(create_query: 'str') -> 'str | None'
```

### `parse_mv_to_table`

```text
parse_mv_to_table(create_query: 'str') -> 'str | None'
```

### `parse_projection_names`

```text
parse_projection_names(create_query: 'str') -> 'list[str]'
```

### `parse_projection_queries`

```text
parse_projection_queries(create_query: 'str') -> 'list[dict[str, str]]'
```

### `parse_replica_name`

```text
parse_replica_name(create_query: 'str', engine: 'str') -> 'str | None'
```

### `parse_settings`

```text
parse_settings(create_query: 'str') -> 'dict[str, str] | None'
```

### `parse_ttl_expressions`

```text
parse_ttl_expressions(create_query: 'str') -> 'list[str]'
```

### `parse_tuple_or_list`

```text
parse_tuple_or_list(value: 'Any') -> 'str | list[str] | None'
```

### `parse_zookeeper_path`

```text
parse_zookeeper_path(create_query: 'str', engine: 'str') -> 'str | None'
```

## `dbwarden.engine.backends.clickhouse.handlers`

### `ChAggTargetHandler`

```text
ChAggTargetHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ChColumnHandler`

```text
ChColumnHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ChCommentHandler`

```text
ChCommentHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ChDataOpHandler`

```text
ChDataOpHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'tuple[list[Op], list[Op]]'
emit(self, op: 'Op', db_name: 'str | None' = None, **kwargs: 'Any') -> 'list[MigrationStatement]'
```

### `ChDictionaryHandler`

```text
ChDictionaryHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ChMaterializedViewHandler`

```text
ChMaterializedViewHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ChProjectionHandler`

```text
ChProjectionHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ChSkipIndexHandler`

```text
ChSkipIndexHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ChTableHandler`

```text
ChTableHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`, `clickhouse_engine_recreate`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

## `dbwarden.engine.backends.mariadb`

### `append_mariadb_column_attrs`

```text
append_mariadb_column_attrs(sql: 'str', meta: 'dict[str, Any]') -> 'str'
```

### `assert_complete_mariadb_type`

```text
assert_complete_mariadb_type(col_type: 'str') -> 'None'
```

### `extract_mariadb_meta`

```text
extract_mariadb_meta(connection, table_name: 'str', indexes: 'list[dict] | None' = None, checks: 'list[dict] | None' = None, uniques: 'list[dict] | None' = None) -> 'tuple[dict[str, Any], dict[str, dict[str, Any]]]'
```

### `mariadb_column_definition_for_meta`

```text
mariadb_column_definition_for_meta(col_type: 'str', meta: 'dict[str, Any]', nullable: 'bool | None' = None, default: 'str | None' = None, comment: 'str | None' = None, autoincrement: 'bool | None' = None) -> 'str'
```

### `normalize_mariadb_default`

```text
normalize_mariadb_default(d: 'Any') -> 'str | None'
```

### `normalize_mariadb_table_value`

```text
normalize_mariadb_table_value(key: 'str', value: 'Any') -> 'Any'
```

### `render_mariadb_column_type`

```text
render_mariadb_column_type(type_str: 'str', meta: 'dict[str, Any]') -> 'str'
```

### `resolve_mariadb_imports`

```text
resolve_mariadb_imports(columns: 'list[dict]') -> 'set[str]'
```

## `dbwarden.engine.backends.mysql`

### `append_mysql_column_attrs`

```text
append_mysql_column_attrs(sql: 'str', meta: 'dict[str, Any]') -> 'str'
```

### `assert_complete_mysql_type`

```text
assert_complete_mysql_type(col_type: 'str') -> 'None'
```

### `build_mysql_alter_default_sql`

```text
build_mysql_alter_default_sql(table: 'str', column: 'str', default: 'Any', col_type: 'str | None' = None, nullable: 'bool | None' = None, my_meta: 'dict[str, Any] | None' = None) -> 'tuple[str, str]'
```

### `extract_mysql_meta`

```text
extract_mysql_meta(connection, table_name: 'str', indexes: 'list[dict] | None' = None, checks: 'list[dict] | None' = None, uniques: 'list[dict] | None' = None) -> 'tuple[dict[str, Any], dict[str, dict[str, Any]]]'
```

### `mysql_column_definition_for_meta`

```text
mysql_column_definition_for_meta(col_type: 'str', meta: 'dict[str, Any]', nullable: 'bool | None' = None, default: 'str | None' = None, comment: 'str | None' = None, autoincrement: 'bool | None' = None) -> 'str'
```

### `normalize_mysql_default`

```text
normalize_mysql_default(d: 'Any') -> 'str | None'
```

### `normalize_mysql_table_value`

```text
normalize_mysql_table_value(key: 'str', value: 'Any') -> 'Any'
```

### `render_mysql_column_type`

```text
render_mysql_column_type(type_str: 'str', meta: 'dict[str, Any]') -> 'str'
```

### `resolve_mysql_imports`

```text
resolve_mysql_imports(columns: 'list[dict]') -> 'set[str]'
```

## `dbwarden.engine.backends.mysql.handlers`

### `MyTableHandler`

```text
MyTableHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

## `dbwarden.engine.backends.postgresql`

### `classify_enum_change`

```text
classify_enum_change(from_values: 'list[str]', to_values: 'list[str]') -> 'str'
```

### `classify_pg_type_change`

```text
classify_pg_type_change(from_type: 'dict[str, Any]', to_type: 'dict[str, Any]') -> 'str'
```

## `dbwarden.engine.backends.postgresql.handlers`

### `ColumnHandler`

```text
ColumnHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ConstraintHandler`

```text
ConstraintHandler() -> 'None'
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `IndexHandler`

```text
IndexHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `PartitionHandler`

```text
PartitionHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `PgTableHandler`

```text
PgTableHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `RenameTableHandler`

```text
RenameTableHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `SchemaHandler`

```text
SchemaHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `StatisticsHandler`

```text
StatisticsHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `TableHandler`

```text
TableHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `TypeHandler`

```text
TypeHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `ViewHandler`

```text
ViewHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

## `dbwarden.engine.backends.sqlite`

### `extract_sqlite_column_meta`

```text
extract_sqlite_column_meta(inspector, connection, table_name: 'str', raw_columns: 'list[dict] | None' = None) -> 'dict[str, dict[str, Any]]'
```

### `extract_sqlite_meta`

```text
extract_sqlite_meta(connection, table_name: 'str', indexes: 'list[dict] | None' = None, checks: 'list[dict] | None' = None, uniques: 'list[dict] | None' = None, raw_columns: 'list[dict] | None' = None) -> 'tuple[dict[str, Any], dict[str, dict[str, Any]]]'
```

### `extract_sqlite_table_meta`

```text
extract_sqlite_table_meta(connection, table_name: 'str') -> 'dict[str, Any]'
```

### `resolve_sqlite_imports`

```text
resolve_sqlite_imports(columns: 'list[dict]') -> 'set[str]'
```

## `dbwarden.engine.backends.sqlite.handlers`

### `SqTableHandler`

```text
SqTableHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`, `op_types`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

## `dbwarden.engine.core`

### `RunPhase`

```text
RunPhase(*values)
```

### `Op`

```text
Op(object_type: 'str', upgrade_attrs: 'dict[str, Any]' = <factory>, rollback_attrs: 'dict[str, Any]' = <factory>, irreversible: 'bool' = False, category: 'str | None' = None, safety: 'str | None' = None) -> None
```

Declared fields: `object_type`, `upgrade_attrs`, `rollback_attrs`, `irreversible`, `category`, `safety`.

### `ObjectHandler`

```text
ObjectHandler(*args, **kwargs)
```

Declared fields: `object_type`, `run_phase`, `statement_order`, `ordering`.

Methods declared on this class:

```text
extract(self, snapshot: 'dict[str, Any]') -> 'dict[str, Any]'
model_spec_from_config(self, config: 'Any') -> 'dict[str, Any]'
model_spec_from_tables(self, model_tables: 'list[Any]') -> 'dict[str, Any]'
canonicalize(self, spec: 'dict[str, Any]') -> 'dict[str, Any]'
diff(self, snap_spec: 'dict[str, Any]', model_spec: 'dict[str, Any]') -> 'Tuple[List[Op], List[Op]]'
emit(self, op: 'Op', db_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[MigrationStatement]'
```

### `Anchor`

```text
Anchor(*values)
```

### `OrderingConstraint`

```text
OrderingConstraint(after: 'tuple[Anchor, ...]' = (), before: 'tuple[Anchor, ...]' = (), after_object: 'tuple[str, ...]' = (), before_object: 'tuple[str, ...]' = ()) -> None
```

Declared fields: `after`, `before`, `after_object`, `before_object`.

### `OrderingError`

```text
OrderingError
```

### `StatementOrder`

```text
StatementOrder(*values)
```

### `MigrationStatement`

```text
MigrationStatement(order: 'StatementOrder', upgrade_sql: 'str', rollback_sql: 'str', rollback_kind: 'str' = 'real', rollback_reason: 'str | None' = None, category: 'str | None' = None, safety: 'str | None' = None) -> None
```

Declared fields: `order`, `upgrade_sql`, `rollback_sql`, `rollback_kind`, `rollback_reason`, `category`, `safety`.

### `Change`

```text
Change(operation: str, table: str, target: str | None = None, resolved_from: str | None = None, index_type: str | None = None) -> None
```

Declared fields: `operation`, `table`, `target`, `resolved_from`, `index_type`.

### `RegistryDriver`

```text
RegistryDriver(*, include_plugins: 'bool' = True) -> 'None'
```

Methods declared on this class:

```text
register(self, handler: 'ObjectHandler') -> 'None'
run(self, snapshot: 'dict[str, Any]', model_tables: 'list[Any]', config: 'Any') -> 'Tuple[List[Op], List[Op]]'
emit_all(self, ops: 'List[Op]', db_name: 'Optional[str]' = None, cluster_ctx: 'Any' = None) -> 'List[MigrationStatement]'
emit_op_to_sql(self, upgrade_ops: 'List[Op]', rollback_ops: 'List[Op]', db_name: 'Optional[str]' = None, cluster_ctx: 'Any' = None) -> 'Tuple[str, str, List[Change]]'
```

### `ModelColumn`

```text
ModelColumn(name: 'str', type: 'str', nullable: 'bool', primary_key: 'bool', unique: 'bool', default: 'Optional[str]', foreign_key: 'Optional[str]', codec: 'Optional[str]' = None, comment: 'Optional[str]' = None, pg_meta: 'Optional[dict[str, Any]]' = None, ch_meta: 'Optional[dict[str, Any]]' = None, my_meta: 'Optional[dict[str, Any]]' = None, sq_meta: 'Optional[dict[str, Any]]' = None, autoincrement: 'Optional[bool]' = None, fk_on_delete: 'Optional[str]' = None, fk_on_update: 'Optional[str]' = None)
```

Methods declared on this class:

```text
to_dict(self) -> 'dict'
```

### `IndexInfo`

```text
IndexInfo(columns: 'list[str]', name: 'str | None' = None, unique: 'bool' = False, using: 'str | None' = None, where: 'str | None' = None, include: 'list[str] | None' = None, with_params: 'dict[str, Any] | None' = None, tablespace: 'str | None' = None, nulls_not_distinct: 'bool' = False, column_sorting: 'dict[str, str] | None' = None, postgresql_ops: 'dict[str, str] | None' = None, comment: 'str | None' = None, concurrently: 'bool' = True, clickhouse_type: 'str | None' = None, clickhouse_granularity: 'int | None' = None, expression: 'str | None' = None) -> None
```

Declared fields: `columns`, `name`, `unique`, `using`, `where`, `include`, `with_params`, `tablespace`, `nulls_not_distinct`, `column_sorting`, `postgresql_ops`, `comment`, `concurrently`, `clickhouse_type`, `clickhouse_granularity`, `expression`.

Methods declared on this class:

```text
to_dict(self) -> 'dict'
from_dict(d: 'dict') -> "'IndexInfo'"
```

### `ModelTable`

```text
ModelTable(name: 'str', columns: 'List[ModelColumn]', clickhouse_options: 'Optional[dict]' = None, object_type: 'str' = 'table', foreign_keys: 'Optional[list[dict]]' = None, indexes: 'Optional[list[dict | IndexInfo]]' = None, comment: 'str | None' = None, checks: 'Optional[list[dict[str, Any]]]' = None, uniques: 'Optional[list[dict[str, Any]]]' = None, excludes: 'Optional[list[dict[str, Any]]]' = None, pg_table: 'Optional[dict[str, Any]]' = None, my_table: 'Optional[dict[str, Any]]' = None, sq_table: 'Optional[dict[str, Any]]' = None, schema: 'str | None' = None, pg_view_definition: 'str | None' = None, pg_view_materialized: 'bool' = False, pg_view_auto_refresh: 'bool' = False, pg_policies: 'Optional[list[dict[str, Any]]]' = None, pg_grants: 'Optional[list[dict[str, Any]]]' = None)
```

Methods declared on this class:

```text
to_dict(self) -> 'dict'
```

### `STATE_FORMAT_VERSION`

```text
STATE_FORMAT_VERSION
```

### `model_state_to_dict`

```text
model_state_to_dict(tables: 'list[ModelTable]', dbwarden_version: 'str' = '') -> 'dict'
```

### `normalize_model_state`

```text
normalize_model_state(state: 'dict[str, Any]') -> 'dict[str, Any]'
```

### `reconstruct_model_table`

```text
reconstruct_model_table(table_entry: 'dict[str, Any]') -> 'ModelTable'
```

### `reconstruct_model_column`

```text
reconstruct_model_column(col_entry: 'dict[str, Any]') -> 'ModelColumn'
```

### `compute_checksum`

```text
compute_checksum(snapshot: 'dict[str, Any]') -> 'str'
```

### `get_schemas_directory`

```text
get_schemas_directory(database: 'str | None' = None) -> 'str'
```

### `write_snapshot`

```text
write_snapshot(snapshot: 'dict[str, Any]', database: 'str | None' = None, migration_id: 'str' = '') -> 'str'
```

### `read_snapshot`

```text
read_snapshot(filepath: 'str') -> 'dict[str, Any] | None'
```

### `find_latest_snapshot`

```text
find_latest_snapshot(database: 'str | None' = None) -> 'dict[str, Any] | None'
```

### `extract_snapshot_tables`

```text
extract_snapshot_tables(snapshot: 'dict[str, Any]') -> 'dict[str, set[str]]'
```

### `TableRenameIntent`

```text
TableRenameIntent(old_table: 'str', new_table: 'str') -> None
```

Declared fields: `old_table`, `new_table`.

### `RENAME_TABLE_OVERLAP_THRESHOLD`

```text
RENAME_TABLE_OVERLAP_THRESHOLD
```

### `detect_renames`

```text
detect_renames(table_name: 'str', dropped_columns: 'list[tuple[str, dict[str, Any]]]', added_columns: 'list[tuple[str, ModelColumn]]') -> 'list[tuple[str, str]]'
```

## `dbwarden.engine.core.plugin_api`

### `ClusterableStatement`

```text
ClusterableStatement(prefix: 'str', suffix: 'str', supports_cluster: 'bool' = True) -> None
```

Declared fields: `prefix`, `suffix`, `supports_cluster`.

Methods declared on this class:

```text
render(self, ctx: 'ClusterContext') -> 'str'
from_sql(sql: 'str') -> "'ClusterableStatement'"
to_migration(self, order: "'StatementOrder'", ctx: 'ClusterContext', rollback: "'ClusterableStatement | str | None'" = None) -> 'MigrationStatement'
```

### `REDACTED`

```text
REDACTED
```

### `build_alter_policy_sql`

```text
build_alter_policy_sql(policy: 'dict', qname: 'str') -> 'str'
```

### `build_create_policy_sql`

```text
build_create_policy_sql(policy: 'dict', qname: 'str') -> 'str'
```

### `build_grant_sql`

```text
build_grant_sql(grant_entry: 'dict', qname: 'str', object_type: 'str' = 'TABLE') -> 'str'
```

### `build_revoke_sql`

```text
build_revoke_sql(grant_entry: 'dict', qname: 'str', object_type: 'str' = 'TABLE') -> 'str'
```

### `emit_with_cluster`

```text
emit_with_cluster(method)
```

### `has_visible_secrets`

```text
has_visible_secrets(spec: 'dict') -> 'bool'
```

### `qualified_name`

```text
qualified_name(name: 'str', schema: 'str | None') -> 'str'
```

### `quote_pg`

```text
quote_pg(name: 'str') -> 'str'
```

### `strip_secret_values`

```text
strip_secret_values(spec: 'dict', secret_keys: 'frozenset[str]') -> 'dict'
```

## `dbwarden.engine.model_discovery`

### `IndexInfo`

Re-export of `dbwarden.engine.core.models.IndexInfo`.

### `ModelColumn`

Re-export of `dbwarden.engine.core.models.ModelColumn`.

### `ModelTable`

Re-export of `dbwarden.engine.core.models.ModelTable`.

### `DBWardenConfigError`

```text
DBWardenConfigError
```

### `auto_discover_model_paths`

Re-export of `dbwarden.engine.model_discovery.path_discovery.auto_discover_model_paths`.

### `discover_models_in_directory`

```text
discover_models_in_directory(directory: str) -> List[str]
```

### `load_model_from_path`

```text
load_model_from_path(filepath: str) -> module | None
```

### `extract_column_info`

```text
extract_column_info(column, db_name: str | None = None, backend: str | None = None) -> dbwarden.engine.core.models.ModelColumn | None
```

### `extract_table_from_model`

```text
extract_table_from_model(model_class: type, db_name: str | None = None) -> dbwarden.engine.core.models.ModelTable | None
```

### `extract_tables_from_database`

```text
extract_tables_from_database(sqlalchemy_url: str) -> dict[str, set[str]]
```

### `extract_tables_from_migrations`

```text
extract_tables_from_migrations(migrations_dir: str) -> dict[str, set[str]]
```

### `filter_model_tables_by_name`

```text
filter_model_tables_by_name(tables: list[dbwarden.engine.core.models.ModelTable], allowed_names: list[str] | None) -> list[dbwarden.engine.core.models.ModelTable]
```

### `validate_model_tables_exist`

```text
validate_model_tables_exist(discovered_tables: list[dbwarden.engine.core.models.ModelTable], configured_names: list[str] | None, db_name: str) -> None
```

### `generate_add_column_sql`

```text
generate_add_column_sql(table_name: str, column: dbwarden.engine.core.models.ModelColumn, db_name: str | None = None, schema: str | None = None) -> str
```

### `generate_create_table_sql`

```text
generate_create_table_sql(table: dbwarden.engine.core.models.ModelTable, db_name: str | None = None) -> str
```

### `generate_drop_object_sql`

```text
generate_drop_object_sql(table: dbwarden.engine.core.models.ModelTable) -> str
```

### `generate_drop_table_sql`

```text
generate_drop_table_sql(table_name: str, schema: str | None = None) -> str
```

### `get_all_model_tables`

Re-export of `dbwarden.engine.model_discovery.get_all_model_tables`.

### `get_model_table_by_name`

```text
get_model_table_by_name(table_name: str, model_paths: list[str] | None = None, db_name: str | None = None) -> dbwarden.engine.core.models.ModelTable | None
```

## `dbwarden.exceptions`

### `ConfigurationError`

```text
ConfigurationError
```

### `DBDisconnectedError`

```text
DBDisconnectedError
```

### `DBWardenConfigError`

Re-export of `dbwarden.exceptions.core.DBWardenConfigError`.

### `DBWardenError`

```text
DBWardenError
```

### `DatabaseError`

```text
DatabaseError
```

### `DirectoryNotFoundError`

```text
DirectoryNotFoundError
```

### `HookConflictError`

```text
HookConflictError(hook_name: 'str', plugins: 'list[str]') -> 'None'
```

### `HookNotRegisteredError`

```text
HookNotRegisteredError(hook_name: 'str') -> 'None'
```

### `HookValidationError`

```text
HookValidationError
```

### `HookVetoError`

```text
HookVetoError
```

### `ImmutableChangeError`

```text
ImmutableChangeError
```

### `LockAcquireTimeout`

```text
LockAcquireTimeout
```

### `LockError`

```text
LockError
```

### `LockStuck`

```text
LockStuck
```

### `NoMigrationsError`

```text
NoMigrationsError
```

### `NoSeedsError`

```text
NoSeedsError
```

### `ObjectHandlerConflictError`

```text
ObjectHandlerConflictError(object_type: 'str', plugins: 'list[str]') -> 'None'
```

### `OrderingError`

Re-export of `dbwarden.exceptions.engine.OrderingError`.

### `PendingMigrationsError`

```text
PendingMigrationsError
```

### `PluginApiMismatchError`

```text
PluginApiMismatchError(dist_name: 'str', declared: 'object') -> 'None'
```

### `PluginInstallError`

```text
PluginInstallError
```

### `RecoveryRequired`

```text
RecoveryRequired
```

### `RollbackContractError`

```text
RollbackContractError
```

### `SeedError`

```text
SeedError
```

### `VersionNotFoundError`

```text
VersionNotFoundError
```

## `dbwarden.merge`

### `SupersededMarker`

```text
SupersededMarker(merged_into: 'str', merged_at: 'str', merge_base: 'str', branch: 'str', applied_persistent: 'str', file_checksum: 'str') -> None
```

Declared fields: `merged_into`, `merged_at`, `merge_base`, `branch`, `applied_persistent`, `file_checksum`.

### `parse_superseded_marker`

```text
parse_superseded_marker(file_path: 'str | Path') -> 'Optional[SupersededMarker]'
```

### `write_superseded_marker`

```text
write_superseded_marker(file_path: 'str | Path', marker: 'SupersededMarker') -> 'None'
```

### `is_superseded`

```text
is_superseded(file_path: 'str | Path') -> 'bool'
```

### `get_file_checksum`

```text
get_file_checksum(file_path: 'str | Path') -> 'str'
```

### `ReconciliationHeader`

```text
ReconciliationHeader(merge_base: 'str', merge_base_checksum: 'str', supersedes: 'list[str]', probe_results: 'dict[str, str]', generated_by: 'str', renames: 'list[dict] | None' = None) -> None
```

Declared fields: `merge_base`, `merge_base_checksum`, `supersedes`, `probe_results`, `generated_by`, `renames`.

### `parse_reconciliation_header`

```text
parse_reconciliation_header(file_path: 'str | Path') -> 'Optional[ReconciliationHeader]'
```

### `write_reconciliation_header`

```text
write_reconciliation_header(file_path: 'str | Path', header: 'ReconciliationHeader') -> 'None'
```

### `is_reconciliation`

```text
is_reconciliation(file_path: 'str | Path') -> 'bool'
```

### `MergeSignal`

```text
MergeSignal(*values)
```

### `detect_merge_signals`

```text
detect_merge_signals(db_name: 'str | None' = None) -> 'list[MergeSignal]'
```

### `EnvironmentConfig`

Re-export of `dbwarden.merge.environments.EnvironmentConfig`.

### `load_environments`

```text
load_environments(db_name: 'str | None' = None) -> 'list[EnvironmentConfig]'
```

### `is_persistent`

```text
is_persistent(environment: 'str', db_name: 'str | None' = None) -> 'bool'
```

### `get_persistent_environments`

```text
get_persistent_environments(db_name: 'str | None' = None) -> 'list[str]'
```

## `dbwarden.models`

### `MigrationType`

```text
MigrationType(*values)
```

### `MigrationDirection`

```text
MigrationDirection(*values)
```

### `MigrationRecord`

```text
MigrationRecord(order_executed: int, version: str | None, description: str, filename: str, migration_type: str, applied_at: datetime.datetime, checksum: str | None) -> None
```

Declared fields: `order_executed`, `version`, `description`, `filename`, `migration_type`, `applied_at`, `checksum`.

### `MigrationFile`

```text
MigrationFile(version: str | None, filename: str, filepath: str, upgrade_sql: list[str], rollback_sql: list[str], checksum: str) -> None
```

Declared fields: `version`, `filename`, `filepath`, `upgrade_sql`, `rollback_sql`, `checksum`.

### `SchemaDifference`

```text
SchemaDifference(type: str, table_name: str, column_name: str | None = None, sql: str = '') -> None
```

Declared fields: `type`, `table_name`, `column_name`, `sql`.

### `SeedRecord`

```text
SeedRecord(version: str, description: str, filename: str, seed_type: str, applied_at: datetime.datetime, checksum: str | None = None) -> None
```

Declared fields: `version`, `description`, `filename`, `seed_type`, `applied_at`, `checksum`.

### `SafetyIssue`

```text
SafetyIssue(severity: str, change_type: str, table_name: str, message: str, column_name: str | None = None, sql: str = '', required_flag: str | None = None) -> None
```

Declared fields: `severity`, `change_type`, `table_name`, `message`, `column_name`, `sql`, `required_flag`.

## `dbwarden.plugin`

### `HOOK_CALL_SPECS`

```text
HOOK_CALL_SPECS
```

### `KNOWN_VALUE_HOOKS`

```text
KNOWN_VALUE_HOOKS
```

### `MULTI_VALUE_HOOKS`

```text
MULTI_VALUE_HOOKS
```

### `PLUGIN_API_ATTR`

```text
PLUGIN_API_ATTR
```

### `PLUGIN_API_VERSION`

```text
PLUGIN_API_VERSION
```

### `PLUGIN_CONFIG_KEY_OWNERS`

```text
PLUGIN_CONFIG_KEY_OWNERS
```

### `PLUGIN_GROUP`

```text
PLUGIN_GROUP
```

### `ConfigKeyRegistry`

```text
ConfigKeyRegistry()
```

Declared fields: .

Methods declared on this class:

```text
register(key: 'str', *, plugin: 'str') -> 'None'
is_registered(key: 'str') -> 'bool'
owner(key: 'str') -> 'str | None'
keys() -> 'tuple[str, ...]'
reset() -> 'None'
```

### `HookConflictError`

Re-export of `dbwarden.exceptions.plugin.HookConflictError`.

### `HookNotRegisteredError`

Re-export of `dbwarden.exceptions.plugin.HookNotRegisteredError`.

### `HookRegistry`

```text
HookRegistry()
```

Declared fields: .

Methods declared on this class:

```text
register(hook_name: 'str', fn: 'Callable[..., Any]', *, plugin: 'str') -> 'None'
execute_single(hook_name: 'str', *args: 'Any', **kwargs: 'Any') -> 'Any'
execute_all(hook_name: 'str', *args: 'Any', **kwargs: 'Any') -> 'list[Any]'
is_registered(hook_name: 'str') -> 'bool'
providers(hook_name: 'str') -> 'list[str]'
hooks() -> 'dict[str, list[tuple[str, Callable[..., Any]]]]'
clear() -> 'None'
```

### `ObjectHandlerConflictError`

Re-export of `dbwarden.exceptions.plugin.ObjectHandlerConflictError`.

### `ObjectHandlerRegistration`

```text
ObjectHandlerRegistration(plugin: 'str', handler: 'Any') -> None
```

Declared fields: `plugin`, `handler`.

### `ObjectPluginRegistry`

```text
ObjectPluginRegistry()
```

Declared fields: .

Methods declared on this class:

```text
register_category(name: 'str', *, order: 'int', plugin: 'str') -> 'None'
categories() -> 'dict[str, dict[str, Any]]'
register(handler: 'Any', *, plugin: 'str') -> 'None'
handlers() -> 'dict[str, ObjectHandlerRegistration]'
has_handler(object_type: 'str') -> 'bool'
clear() -> 'None'
```

### `PluginApiMismatchError`

Re-export of `dbwarden.exceptions.plugin.PluginApiMismatchError`.

### `PluginRegistrar`

```text
PluginRegistrar(plugin_name: 'str') -> 'None'
```

Methods declared on this class:

```text
register(self, hook_name: 'str', fn: 'Callable[..., Any]') -> 'None'
register_object_handler(self, handler: 'Any') -> 'None'
register_migration_category(self, name: 'str', *, order: 'int') -> 'None'
register_config_key(self, *keys: 'str') -> 'None'
```

## `dbwarden.repositories`

### `create_migrations_table_if_not_exists`

```text
create_migrations_table_if_not_exists(db_name: str | None = None) -> None
```

### `fetch_latest_versioned_migration`

```text
fetch_latest_versioned_migration(db_name: str | None = None) -> dbwarden.models.MigrationRecord | None
```

### `get_existing_runs_always_filenames`

```text
get_existing_runs_always_filenames(db_name: str | None = None) -> set[str]
```

### `get_existing_runs_on_change_filenames_to_checksums`

```text
get_existing_runs_on_change_filenames_to_checksums(db_name: str | None = None) -> dict[str, str]
```

### `get_latest_versions`

```text
get_latest_versions(db_name: str | None = None, limit: int | None = None, starting_version: str | None = None) -> list[str]
```

### `get_migration_records`

```text
get_migration_records(db_name: str | None = None) -> list[dbwarden.models.MigrationRecord]
```

### `get_migrated_versions`

```text
get_migrated_versions(db_name: str | None = None) -> list[str]
```

### `migrations_table_exists`

```text
migrations_table_exists(db_name: str | None = None) -> bool
```

### `run_migration`

```text
run_migration(sql_statements: list[str], version: str | None, migration_operation: str, filename: str, migration_type: str = 'versioned', db_name: str | None = None, perf: bool = False, connection: Any | None = None, namespace: str = 'default', fencing_token: int = 0, progress_callback: Any | None = None, migration_path: str | None = None, reapply_data: bool = False) -> None
```

### `run_repeatable_migration`

```text
run_repeatable_migration(sql_statements: list[str], filename: str, migration_type: str, db_name: str | None = None, perf: bool = False, connection: Any | None = None) -> None
```

### `acquire_lock`

```text
acquire_lock(db_name: 'str | None' = None, *, namespace: 'str | None' = None, migration_version: 'str | None' = None, migration_checksum: 'str | None' = None) -> 'LockAcquisition'
```

### `check_lock`

```text
check_lock(db_name: 'str | None' = None, *, namespace: 'str' = 'default') -> 'bool'
```

### `create_lock_table_if_not_exists`

```text
create_lock_table_if_not_exists(db_name: 'str | None' = None) -> 'None'
```

### `force_release_lock`

```text
force_release_lock(db_name: 'str | None' = None, *, namespace: 'str' = 'default') -> 'bool'
```

### `release_lock`

```text
release_lock(db_name: 'str | None' = None, *, namespace: 'str' = 'default', strategy: 'Any' = None) -> 'bool'
```

## `dbwarden.schema`

### `DBWardenSeed`

```text
DBWardenSeed(database: 'str', description: 'str', on_conflict: 'str' = 'ignore', conflict_columns: 'list[str] | None' = None, source_hash: 'str' = '', version: 'str' = '', **kwargs: 'Any') -> 'None'
```

Declared fields: `database`, `description`, `on_conflict`, `conflict_columns`, `source_hash`.

## `dbwarden.schema.table_meta`

### `TableMeta`

Re-export of `dbwarden.schema.table_meta.TableMeta`.

### `PGTableMeta`

Re-export of `dbwarden.schema.table_meta.PGTableMeta`.

### `PGColumnMeta`

Re-export of `dbwarden.schema.table_meta.PGColumnMeta`.

### `PGViewMeta`

Re-export of `dbwarden.schema.table_meta.PGViewMeta`.

### `CHViewMeta`

Re-export of `dbwarden.schema.table_meta.CHViewMeta`.

### `CHTableMeta`

Re-export of `dbwarden.schema.table_meta.CHTableMeta`.

### `CHColumnMeta`

Re-export of `dbwarden.schema.table_meta.CHColumnMeta`.

### `MyTableMeta`

Re-export of `dbwarden.schema.table_meta.MyTableMeta`.

### `MyColumnMeta`

Re-export of `dbwarden.schema.table_meta.MyColumnMeta`.

### `MdbTableMeta`

Re-export of `dbwarden.schema.table_meta.MdbTableMeta`.

### `MdbColumnMeta`

Re-export of `dbwarden.schema.table_meta.MdbColumnMeta`.

### `SqTableMeta`

Re-export of `dbwarden.schema.table_meta.SqTableMeta`.

### `SqColumnMeta`

Re-export of `dbwarden.schema.table_meta.SqColumnMeta`.

========================================================================
PAGE: https://docs.dbwarden.org/seeds/
========================================================================

# Seed Management

**Requires the `dbwarden-seeds` plugin:** `dbwarden plugin add dbwarden-seeds`. Seed management was previously built into core and now ships as an official plugin. The `dbwarden seed` commands still exist in core, but without the plugin installed each one exits with a message telling you to install it.

dbwarden provides seed data management for populating databases with initial or reference data. Seeds complement migrations by handling data that belongs in version control.

## Overview

There are two ways to define seeds, listed in order of preference:

1. **Code seeds** (recommended): define seeds inline alongside your SQLAlchemy models using the `Seed` base class. Each seed has an immutable explicit version.
2. **File seeds**: traditional `.sql` or `.py` files in a `seeds/` directory, useful for complex multi-statement SQL.

Both are tracked in the `_dbwarden_seeds` table and applied via `dbwarden seed apply`.

---

## Code Seeds (Recommended)

Code seeds live alongside your models in your `model_paths` directories. They are the recommended way to define seed data because they stay in sync with your schema, support IDE autocompletion, and do not require manual version management.

### Seed Base Class

Inherit from `Seed` and set a `model` + `rows`:

```python
from dbwarden.seed import Seed

class CountrySeed(Seed):
    __seed_version__ = "C0001"
    __seed_database__ = "primary"
    __seed_description__ = "initial countries"
    __seed_on_conflict__ = "update"
    __seed_conflict_columns__ = ["code"]

    model = Country
    rows = [
        Country(code="UY", name="Uruguay"),
        Country(code="AR", name="Argentina"),
    ]
```

Key advantages over the old decorator approach:

- **Full IDE autocompletion**: `rows` uses model instances directly, so your editor knows the column names and types
- **Immutable versioning**: declare a permanent `__seed_version__` such as `C0001`; never change or reuse a released version
- **No manual import of `SeedRow`**: though `SeedRow` is still available if you prefer dict-like rows

### Seed Class Reference

| Attribute | Default | Description |
|-----------|---------|-------------|
| `__seed_database__` | `"default"` | Routes the seed to the named database handle configured by a `DbwardenDatabase` class or `database_config(...)`. |
| `__seed_description__` | `""` | Human-readable label shown in `dbwarden seed list` output. |
| `__seed_on_conflict__` | `"ignore"` | What to do when a row with matching columns exists: `"ignore"` (skip silently), `"update"` (overwrite), or `"error"` (raise). |
| `__seed_conflict_columns__` | `None` | List of column names used for conflict detection. Required when `__seed_on_conflict__` is `"update"`. |

### Model Instances in Rows

Because `rows` accepts model instances, you get full autocompletion from your `Mapped` annotations:

```python
from dbwarden.seed import Seed

class RepoSeed(Seed):
    __seed_database__ = "clickhouse"
    __seed_description__ = "Tracked Repos"

    model = Repo
    rows = [
        Repo(name="dbwarden", owner="anomalyco", is_org=True, default_branch="main"),
        Repo(name="vigil", owner="anomalyco", is_org=True, default_branch="master"),
    ]
```

Your editor will suggest `name`, `owner`, `is_org`, `default_branch` etc. as you type.

> SQLAlchemy 2.0's `DeclarativeBase` does not accept positional arguments in the constructor. Always use keyword arguments when instantiating models in `rows`: `Repo(name="dbwarden", ...)` instead of `Repo("dbwarden", ...)`.

### SeedRow (Alternative)

If you prefer dict-like rows, `SeedRow` still works:

```python
from dbwarden.seed import Seed, SeedRow

class CountrySeed(Seed):
    __seed_database__ = "primary"
    __seed_description__ = "initial countries"
    __seed_on_conflict__ = "update"
    __seed_conflict_columns__ = ["code"]

    model = Country
    rows = [
        SeedRow(code="UY", name="Uruguay"),
        SeedRow(code="AR", name="Argentina"),
    ]
```

### `on_conflict` Behavior

| Value | Behavior |
|-------|----------|
| `"ignore"` (default) | Skips existing rows silently |
| `"update"` | Updates existing rows with new values |
| `"error"` | Raises an error on conflict |

### PostgreSQL Schema Resolution

When a model uses `pg_schema` in its Meta (via `PGTableMeta` or `PGViewMeta`), code seeds automatically qualify the table name with that schema:

```python
from dbwarden.databases.pgsql import PGTableMeta

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)

    class Meta(PGTableMeta):
        pg_schema = "app"

class UserSeed(Seed):
    __seed_database__ = "primary"
    __seed_description__ = "initial users"

    model = User
    rows = [User(email="alice@example.com", name="Alice")]
```

The generated INSERT becomes:

```sql
INSERT INTO app.users (email, name) VALUES ('alice@example.com', 'Alice')
```

The schema is resolved in this order: `Meta.pg_schema`, then `Meta.backend_table.schema`, then `__table__.schema`. The seed tracking table (default `_dbwarden_seeds`) stays in the schema set by the connection's `search_path` (config-level `pg_schema`).

### Logic-Based Seeds

Define `forward(connection, session)` and `reverse(connection, session)` for procedural data, analogous to Django's `RunPython`:

```python
class PermissionSeed(Seed):
    __seed_version__ = "C0002"
    __seed_database__ = "primary"
    __seed_description__ = "load permissions"
    __seed_on_conflict__ = "ignore"

    model = Permission

    @staticmethod
    def forward(connection, session):
        for resource in ["users", "orders"]:
            for action in ["read", "write", "delete"]:
                session.add(Permission(name=f"{resource}:{action}"))

    @staticmethod
    def reverse(connection, session):
        session.execute("DELETE FROM permissions")
```

### `@seed_data` Decorator (Deprecated)

The old decorator still works but is deprecated in favour of the `Seed` base class:

```python
from dbwarden.seed import seed_data, SeedRow

@seed_data(
    database="primary",
    description="initial countries",
    on_conflict="update",
    conflict_columns=["code"],
)
class CountrySeed:
    model = Country
    rows = [SeedRow(code="UY", name="Uruguay")]
```

The deprecated decorator does not provide the required immutable version contract; use `Seed` subclasses for new code seeds.

### Discovery and Ordering

Code seeds are discovered through the same `model_paths` scan as models. They declare immutable versions in the `C` namespace (`C0001`, `C0002`, ...) and duplicate versions are rejected. Pending detection compares versions against the `_dbwarden_seeds` tracking table.

---

## File Seeds (Traditional)

File seeds live in a `seeds/` directory and are useful for complex multi-statement SQL or when you need to hand-craft seed files.

### Directory Structure

```
seeds/
  V0001__seed_initial_users.sql
  V0002__seed_lookup_tables.sql
  V0003__seed_sample_data.py
```

Each file follows the naming convention:

```
V<4-digit-version>__<description>.<sql|py>
```

### Creating File Seeds

```bash
$ dbwarden seed create "seed initial users" --database primary
```

Creates a file like `seeds/V0001__seed_initial_users.sql`:

```sql
-- INSERT statements go here
```

### Python File Seeds

```bash
$ dbwarden seed create "generate sample data" --database primary --type python
```

Creates `seeds/V0001__generate_sample_data.py` with `forward(connection, session)` and `reverse(connection, session)` functions.

Both functions receive a raw SQLAlchemy `Connection` and an ORM `Session` bound to the same transaction:

```python
# Using raw connection
def forward(connection, session):
    for i in range(100):
        connection.execute(
            "INSERT INTO users (name) VALUES (:name)",
            {"name": f"user_{i}"},
        )

# Using ORM session
def forward(connection, session):
    for i in range(100):
        session.add(User(name=f"user_{i}"))
    session.flush()
```

---

## Applying Seeds

Apply all pending seeds (file + code seeds are both discovered):

```bash
$ dbwarden seed apply --database primary
```

Apply a specific version:

```bash
$ dbwarden seed apply --database primary --version 0003
```

Apply to all databases:

```bash
$ dbwarden seed apply --all
```

### Dry Run

Preview what would be applied without executing:

```bash
$ dbwarden seed apply --database primary --dry-run
```

### Auto-Apply After Migrations

Configure seeds to be applied automatically after each `dbwarden migrate`:

```python
database_config(
    database_name="primary",
    default=True,
    database_type="sqlite",
    database_url_sync="sqlite:///./app.db",
    model_paths=["models"],
    auto_apply_seeds=True,
)
```

Or apply seeds once after a migration without changing config:

```bash
$ dbwarden migrate --apply-seeds
```

---

## Listing Seeds

```bash
$ dbwarden seed list --database primary
```

Output:

```
Seeds for database 'primary':
  V0001  seed_initial_users                   applied  2025-06-01 10:00:00
  C0001  initial countries                    pending   (code seed)
```

List across all databases:

```bash
$ dbwarden seed list --all
```

### Pruning Orphaned Records

Remove tracking records for seed files that no longer exist on disk:

```bash
$ dbwarden seed list --prune
```

---

## Rolling Back Seeds

Rollback executes the seed's inverse operation and removes its tracking record only after that succeeds. SQL files require a `-- rollback` section; Python and procedural code seeds require `reverse(connection, session)`. Missing reverse logic makes a seed irreversible.

```bash
# Rollback the most recent seed
$ dbwarden seed rollback --database primary

# Rollback a specific number
$ dbwarden seed rollback --database primary --count 2

# Rollback to a specific version
$ dbwarden seed rollback --database primary --to-version 0002
```

---

## Seed Tracking Table

dbwarden tracks applied seeds in `_dbwarden_seeds` (configurable via `seed_table`):

| Column | Description |
|--------|-------------|
| `version` | 4-digit seed version (`V0001`) or code seed ID (`C0001`) |
| `description` | Human-readable description |
| `filename` | File path or code seed identifier |
| `seed_type` | `sql`, `python`, or `code` |
| `checksum` | SHA-256 hash of file/class source |
| `applied_at` | Timestamp of application |

The tracking table is created automatically on first seed apply. Each version can only be applied once until rolled back.

### Checksum Drift

When a seed file has been modified since it was last applied, dbwarden raises an error and blocks seed operations:

```
SeedError: Seed V0001 has been modified since it was applied (checksum mismatch).
```

This helps detect accidental changes to already-applied seeds.

---

## Exporting Seeds for Production

Code seeds require your full application environment to execute. For Dockerized deployments where you don't want to copy the application code into a container just to seed data, use `dbwarden seed export` to produce stateless ROC (runs-on-change) SQL files.

```bash
$ dbwarden seed export --database clickhouse
```
This writes `seeds/ROC__clickhouse__code_seeds.sql` containing `INSERT ... ON CONFLICT` statements rendered in the target database dialect. In production, apply with:
```bash
$ dbwarden seed apply --database clickhouse
```

Because the file is ROC, updating the code seed and re-exporting produces a new content checksum, which triggers re-application. The `ON CONFLICT DO UPDATE` clause handles updating existing rows; no need to delete and recreate.

**Non-handled problems:**

- Rows removed from a code seed are not automatically deleted in the target database
- Procedural `forward()` seeds cannot be exported, because executing arbitrary code against SQLite would not reliably reproduce behavior on the configured backend

**Dialect requirement:** Exporting requires the same dialect packages as connecting to that database. For ClickHouse, install `clickhouse-sqlalchemy`. Missing packages produce a clear error at export time.

## Seeds and Migrations

Seeds are independent from migrations. You can:

- Apply migrations without seeds
- Apply seeds without migrations
- Mix both in your workflow

The `dbwarden status` command and the FastAPI `GET /status` endpoint report both pending migrations and pending seeds.

---

## Seeds in FastAPI

The `DBWardenRouter` includes seed status in its `GET /status` response:

```json
{
  "databases": {
    "primary": {
      "status": "ok",
      "connected": true,
      "pending_migrations": 0,
      "applied_migrations": 5,
      "pending_seeds": 2,
      "applied_seeds": 1,
      "lock_active": false,
      "error": null
    }
  }
}
```

FastAPI integration ships separately as the `dbwarden-fastapi` plugin, so its reference lives with the plugin: [dbwarden-fastapi](https://github.com/dbwarden-org/dbwarden-fastapi).

See also: [Cookbook: Seeds](cookbook/07-seeds.md)

========================================================================
PAGE: https://docs.dbwarden.org/sql-translation/
========================================================================

# SQL Translation

dbwarden includes a SQL translation layer to support development workflows where your primary database differs from your development database.

The most common case is:

- Primary database: PostgreSQL/MySQL/MariaDB/ClickHouse
- Development database: SQLite (`--dev` mode)

This keeps local development fast while still allowing production-targeted schemas.

## Why SQL Translation Exists

SQLite does not support all backend-specific SQL types and default expressions used by other databases.

Without translation, generated migrations can fail in local development when they contain backend-specific types like `UUID`, `JSONB`, or default expressions like `now()`.

dbwarden translation solves this by adapting generated SQL for SQLite compatibility.

## When translation is active

Translation runs whenever the **resolved backend is SQLite** and SQL is being
generated from models (`make-migrations`, `export-models`). That covers both:

- a `--dev` run whose `dev_database_url` is SQLite, and
- a database whose `database_type` is `sqlite` in its own right.

It is not a runtime SQL proxy for arbitrary manual SQL, and it is not applied to
migration files you write by hand.

!!! note "Translation and first-class SQLite are different layers"

    Translation adapts a *type or default written for another backend* so it is
    legal in SQLite. [First-class SQLite support](databases/sqlite/index.md) is about
    the shape of the migration itself - table rebuilds, `WITHOUT ROWID`,
    `STRICT`, generated columns. They compose: a translated type is what the
    rebuild then renders.

## How It Works

When you run commands in development mode and target a SQLite dev database:

```bash
$ dbwarden --dev make-migrations "sync models" -d primary
```

dbwarden uses this flow:

1. Loads the selected database config and resolves `dev_database_url`.
2. Detects that the active target backend is SQLite.
3. Extracts model metadata from SQLAlchemy models.
4. Translates backend-specific types/defaults to SQLite-compatible SQL.
5. Generates migration SQL with translated definitions.

Translation is applied during migration generation, not as a post-processing regex pass.

## Type conversion behavior

Common conversions:

| Source type | SQLite output |
|-------------|---------------|
| `UUID` | `TEXT` |
| `JSON` / `JSONB` | `TEXT` |
| `BYTEA` | `BLOB` |
| `TIMESTAMPTZ` / `TIMESTAMP WITH TIME ZONE` | `DATETIME` |
| `SERIAL` / `BIGSERIAL` | `INTEGER` |
| `BOOL` | `BOOLEAN` |
| `INT8` / `INT16` / `INT32` / `INT64` | `INTEGER` |
| `UINT8` / `UINT16` / `UINT32` / `UINT64` | `INTEGER` |
| `FLOAT32` / `FLOAT64` | `REAL` |
| `DATETIME64` / `DATETIME64(n)` | `DATETIME` |
| `DECIMAL(p,s)` / `NUMERIC(p,s)` | `NUMERIC` |
| `FIXEDSTRING(n)` | `TEXT` |
| `ARRAY(...)` | `TEXT` (or an error in strict mode) |

ClickHouse `Nullable(...)` and `LowCardinality(...)` wrappers are unwrapped
before the inner type is translated.

Types SQLite already understands - `INTEGER`, `VARCHAR(n)`, `TEXT`, `BOOLEAN`,
`REAL`, `DATE`, `DATETIME`, `BLOB`, `NUMERIC` and friends - pass through
unchanged, keeping their declared length.

If a type cannot be translated safely:

- non-strict mode: fallback to `TEXT` + warning
- strict mode: fail migration generation

## Default expression handling

Backend expressions such as `now()` or `gen_random_uuid()` may not have direct SQLite equivalents.

`CURRENT_TIMESTAMP`, `CURRENT_DATE` and `CURRENT_TIME` pass through unchanged.
`NOW()`, `UUID_GENERATE_V4()`, `GEN_RANDOM_UUID()` and `NEXTVAL(...)` have no
SQLite equivalent.

In non-strict mode, unsupported defaults are dropped with warning.

In strict mode, unsupported defaults fail generation.

## Strict Translation Mode

If you want hard failures instead of fallback behavior:

```bash
$ dbwarden --dev --strict-translation make-migrations "sync models" -d primary
```

In strict mode:

- Unknown/unsupported type conversions raise errors
- Unsupported default expression conversions raise errors

Use this when you want to catch every lossy conversion early.

!!! warning "`--strict-translation` is not a `STRICT` table"

    `--strict-translation` controls what dbwarden does with a type it cannot
    translate. SQLite's `STRICT` keyword is a table option, set with
    `sq_strict` on the model, that makes SQLite itself enforce column types.
    See [SQLite](databases/sqlite/index.md#types-and-strict-tables).

## Recommended team workflow

1. iterate quickly with `--dev` (SQLite)
2. keep strict checks in CI (`--strict-translation`)
3. validate release candidate migrations against production-like database

This balances speed and correctness.

## Troubleshooting

`--dev mode is enabled, but database '<name>' has no dev_database_url configured`:

- add `dev_database_url` for that database entry

Unexpected type fallback to `TEXT`:

- inspect model type for backend-specific declaration
- re-run with `--strict-translation` to fail fast and fix explicitly

Generated SQL differs from production expectations:

- expected in SQLite compatibility mode; validate final release migrations on production-like backend

## Notes and Limitations

- Translation focuses on compatibility for local development.
- Some backend features cannot be represented exactly in SQLite.
- For production accuracy, always test migrations against your production-like database too.
- Translation converts a type name; it does not convert stored data. Narrowing a
  column against existing rows can still be rejected by SQLite when the target
  table is `STRICT`.
