Metadata-Version: 2.4
Name: migkit
Version: 0.2.0
Summary: Verify, repair, and move databases across engines - without trusting the mover.
Author: arkashira
License: AGPL-3.0-or-later
Project-URL: Homepage, https://github.com/arkashira/migkit
Project-URL: Repository, https://github.com/arkashira/migkit
Project-URL: Changelog, https://github.com/arkashira/migkit/blob/master/CHANGELOG.md
Keywords: database,migration,cdc,replication,verification,postgres,mysql,mongodb,dms,reconciliation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: click>=8.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: psycopg2-binary>=2.9
Requires-Dist: pymysql>=1.1
Requires-Dist: pymssql>=2.3
Requires-Dist: clickhouse-connect>=0.8
Requires-Dist: oracledb>=3.4
Requires-Dist: ibm_db>=3.2
Requires-Dist: pyodbc>=5.1
Requires-Dist: mapepire-python>=1.0
Requires-Dist: ebcdic>=2.0
Requires-Dist: pyodata>=1.12
Requires-Dist: dlt>=1.30
Requires-Dist: cassandra-driver>=3.29
Requires-Dist: opensearch-py>=3.0
Requires-Dist: snowflake-connector-python>=3.6
Requires-Dist: google-cloud-bigquery>=3.11
Requires-Dist: google-cloud-bigquery-storage>=2.41
Requires-Dist: snowpipe-streaming>=1.8
Requires-Dist: redshift_connector>=2.1
Requires-Dist: pyiceberg[pyarrow,sql-sqlite]>=0.12
Requires-Dist: deltalake>=1.6
Requires-Dist: zstandard>=0.22
Requires-Dist: mysql-replication>=1.0
Requires-Dist: pymongo>=4.6
Requires-Dist: redis>=5.0
Requires-Dist: kafka-python>=2.2.15
Requires-Dist: confluent-kafka>=2.5
Requires-Dist: fastavro>=1.9
Requires-Dist: reladiff
Requires-Dist: datacompy
Requires-Dist: pandas
Requires-Dist: pyarrow
Requires-Dist: sqlglot
Requires-Dist: hypothesis
Requires-Dist: z3-solver
Requires-Dist: ast-grep-py>=0.45
Requires-Dist: duckdb>=1.1
Requires-Dist: results
Requires-Dist: psycopg[binary,pool]>=3.1
Requires-Dist: psutil>=5.9
Requires-Dist: pyrage>=1.2
Requires-Dist: croniter>=2.0
Requires-Dist: boto3
Requires-Dist: google-cloud-pubsub>=2.20
Requires-Dist: azure-identity
Requires-Dist: azure-mgmt-postgresqlflexibleservers
Requires-Dist: azure-mgmt-mysqlflexibleservers
Requires-Dist: google-api-python-client
Requires-Dist: azure-mgmt-sql
Requires-Dist: azure-mgmt-redis
Requires-Dist: azure-mgmt-cosmosdb
Requires-Dist: tencentcloud-sdk-python-common
Requires-Dist: tencentcloud-sdk-python-cdb
Requires-Dist: tencentcloud-sdk-python-postgres
Requires-Dist: tencentcloud-sdk-python-mongodb
Requires-Dist: tencentcloud-sdk-python-redis
Requires-Dist: tencentcloud-sdk-python-cynosdb
Requires-Dist: tencentcloud-sdk-python-sqlserver
Requires-Dist: tencentcloud-sdk-python-mariadb
Requires-Dist: alibabacloud_credentials
Requires-Dist: alibabacloud_rds20140815
Requires-Dist: alibabacloud_dds20151201
Requires-Dist: alibabacloud_polardb20170801
Requires-Dist: alibabacloud_r_kvstore20150101
Requires-Dist: alibabacloud_polardbx20200202
Requires-Dist: oci
Requires-Dist: ibm-cloud-sdk-core
Requires-Dist: keyring>=25
Requires-Dist: couchbase>=4.3
Requires-Dist: azure-cosmos>=4.9.1
Requires-Dist: google-cloud-spanner>=3.71
Requires-Dist: google-cloud-bigtable>=2.48
Requires-Dist: google-cloud-firestore>=2.33
Requires-Dist: google-cloud-datastore>=2.26
Requires-Dist: gremlinpython>=3.8
Requires-Dist: cassandra-sigv4>=4.0
Requires-Dist: apprise>=2.0
Requires-Dist: opentelemetry-sdk>=1.40
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.40
Requires-Dist: azure-storage-blob>=12.20
Requires-Dist: google-cloud-storage>=2.18
Requires-Dist: werkzeug>=3.0
Requires-Dist: jinja2>=3.1
Requires-Dist: cheroot>=10.0
Requires-Dist: authlib>=1.3
Requires-Dist: joserfc>=1.0
Requires-Dist: signxml>=4.0
Requires-Dist: argon2-cffi>=23.1
Requires-Dist: scim2-filter-parser>=0.7
Provides-Extra: mssql-bulk
Requires-Dist: mssql-python>=1.15; extra == "mssql-bulk"
Provides-Extra: mysql
Provides-Extra: mongo
Provides-Extra: kafka
Provides-Extra: redis
Provides-Extra: diff
Provides-Extra: dataframe
Provides-Extra: s3
Provides-Extra: pgschema
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: faker; extra == "dev"
Requires-Dist: hypothesis; extra == "dev"
Requires-Dist: pexpect; extra == "dev"
Requires-Dist: moto[dynamodb,ec2,rds]; extra == "dev"
Requires-Dist: pytest-xdist; extra == "dev"
Requires-Dist: pytest-timeout; extra == "dev"
Requires-Dist: flask; extra == "dev"
Requires-Dist: flask-cors; extra == "dev"
Requires-Dist: prometheus-client; extra == "dev"
Dynamic: license-file

# migkit

Tools for moving databases, checking the result and repairing differences.

<!-- gen_docs: badge -->
[![ci](https://github.com/arkashira/migkit/actions/workflows/ci.yml/badge.svg)](https://github.com/arkashira/migkit/actions/workflows/ci.yml)
[![engines](https://img.shields.io/badge/engines-64-2a78d6)](#supported-engines)
[![pairs](https://img.shields.io/badge/cross--engine-any_pair-0ca30c)](#cross-engine-hetero)
[![python](https://img.shields.io/badge/python-3.12+-3776ab)](pyproject.toml)
[![license](https://img.shields.io/badge/license-AGPL--3.0--or--later-0ca30c)](LICENSE)
<!-- /gen_docs: badge -->

migkit is a personal project built from database migration work. It helps
prepare the target, move or follow data, compare both sides and repair supported
differences. It can work alongside a managed migration service or handle data
movement itself.

A successful copy is not the end of a migration. I want to check what arrived,
what was left out and whether the target is ready for the application.

[Usage reference](https://arkashira.github.io/migkit/) | [Changelog](CHANGELOG.md)

## Table of contents

- [Why](#why)
- [Case study: what a "consistent" report does not cover](docs/case-study-what-a-consistent-report-does-not-cover.md)
- [Features](#features)
- [Quickstart](#quickstart)
- [Install](#install)
- [Commands](#commands)
- [One result shape](#one-result-shape)
- [Supported engines](#supported-engines)
- [Cross-engine (hetero)](#cross-engine-hetero)
- [How it compares](#how-it-compares)
- [Safety model](#safety-model)
- [Telemetry](#telemetry)
- [License](#license)

## Why

A migration can finish while sequences, indexes, permissions or other database
objects still need attention. Row counts alone also miss changes to the values
inside those rows.

migkit checks the source and target separately from the copy process. Reports
include the results from both sides so you can investigate differences and keep
a record of the run. Support varies by engine; check the capability tables for
your pair before planning a migration.

The [case study](docs/case-study-what-a-consistent-report-does-not-cover.md)
shows findings from migrations that had already been reported as consistent.

## Features

- Compare schema, row counts, data checksums and sequence or identity values.
- Locate differences by key range, primary key or column where supported.
- Repair rows and counters, saving target state for undo before changes.
- Resume supported copies and long checks after an interruption.
- Follow source changes and verify changed rows on supported engines.
- Use consistent snapshots and replication fences to investigate differences
  caused by replication lag.
- Adjust verification concurrency when supported database load signals show
  pressure on the server.
- Show expected locks for generated repair DDL before applying it.
- Keep local state, write records and reports, with an optional web dashboard.

The available checks, copy methods and recovery options depend on the engine,
permissions and installed components. `migkit doctor` reports what is available
on your machine. The [usage reference](https://arkashira.github.io/migkit/) covers the
individual workflows and limitations.

## Quickstart

Start by installing migkit and defining a source and target. The move commands
below show a plan first; `--go` allows the write:

<!-- gen_docs: quickstart -->
```
uv tool install git+https://github.com/arkashira/migkit.git
migkit doctor --install
migkit init
migkit assess shop
migkit move shop --mode full+cdc
migkit move shop --mode full+cdc --go
migkit check shop
migkit move shop --mode cdc --drop --go
```
<!-- /gen_docs: quickstart -->

What a hop can be for - its `goal`, one word
([what each does on each side](docs/goals-and-modes.md)):

<!-- gen_docs: goals -->
| Goal | What it is for |
|---|---|
| `copy` | move the data once and prove every row arrived |
| `cutover` | move, follow the source's changes, prove both equal, and switch with the least downtime - a stream back ready from the moment of the switch |
| `verify` | prove two databases hold the same; nothing is written to either |
| `two-way` | both sides written, each side's changes carried to the other |
| `keep-in-sync` | a copy that follows the source for as long as it runs, the application kept off it |
| `rehearse` | the whole path the real run will take - copy, follow, prove, cut over - on a copy or a sample of the source into a target of its own, every phase timed, so the hop it rehearses (rehearses:) is forecast from what it measured |
| `upgrade` | the same engine at a newer version, a higher edition (SQL Server Web to Enterprise), or both, with the least downtime: what the new version or edition changes checked first, then the data moved into it and followed, or a copy of its pages taken to the target where the hop allows it - upgraded there where the version moves - and switched over |
| `subset` | a copy for testing: the rows chosen and every row they refer to, masked where the hop says, into a staging or developer database, proved row by row |
| `consolidate` | many databases or tenants into one: each source's rows under a column the hop names, filled with its own value, keys kept apart by the hop's rule, proved source by source |
| `split` | one database into many - a service extracted, a database split by region: each part's rows by its rule, every row placed exactly once, proved, and followed until every part is cut over at once |
| `reshard` | N sources onto M targets by one shard function - shards split or merged, a new shard key, a cluster's own distribution - every row placed exactly once, proved, followed until every source and target is cut over at once |
| `archive` | old rows to cheaper storage: the rows each rule picks added to an archive and proved there, then removed from the source only once proved and only where the hop allows it |
<!-- /gen_docs: goals -->

## Install

```bash
uv tool install migkit     # or: pipx install migkit
```

Either one puts `migkit` on your PATH in its own isolated environment. Plain
`pip install migkit` works too if you would rather manage the environment
yourself.

There are no extras to choose. Every engine driver and every comparison
library is a hard dependency, so the install either gives you all of them or
fails telling you why - a missing driver should not be something you discover
at cutover.

Some capabilities also need a command-line program from the platform (parallel
dump/restore, cross-engine load, schema DDL). Nothing is required to start:

```bash
migkit doctor            # capabilities: ready / reduced / unavailable
migkit doctor --install  # install what is missing via brew or apt
```

`doctor` runs before you have configured anything - that is what it is for.
When you are ready:

```bash
migkit init              # writes ~/.config/migkit/hops.yaml, mode 600
```

In a terminal, `init` asks where the data is and where it goes - an
address each, or a task file from another tool (a DMS task, a
change-capture connector's configuration, a LOAD DATABASE file) - and the
passwords, which it keeps in the OS keychain or leaves in environment
variables. It connects, says what it found (software and versions, who runs
each side, sizes, tables without a key, types that need care across two
engines, whether changes can be followed, what the users may do), proposes
the plan, writes a script for the DBA where a grant or a setting is
missing, and writes the hop. Without a terminal it reads `MIGKIT_SOURCE`
and `MIGKIT_TARGET` instead, or writes a template.

migkit finds that file on its own. `MIGKIT_CONF=/path/to/hops.yaml` overrides
it, and a `conf/hops.yaml` in the working directory takes precedence, so a
project can keep its hops beside its code. The file looks like this:

```yaml
hops:
  shop:
    goal: cutover                 # copy (left out) | cutover | verify | two-way | keep-in-sync
    source: postgres://app@db1.example.com:5432/shop?sslmode=verify-full
    target: mysql://app@10.0.0.10/shop
```

The engines, the pair, the ports, the databases and TLS are read from the
addresses; the number of workers, the size of a range and the way each
table goes are decided from the servers, so the file says none of them. A
password is never in an address - `init` keeps it in the keychain, or
write a reference in its place (`postgres://app:env:SRC_PW@db1/shop`). The
long form (`source: {host: ..., port: ..., user: ...}`, `databases`,
`db_map`) stays valid, and every configuration is checked before anything
runs: an unknown key is refused with the closest known one, a wrong type or
a contradiction with where it is.

A move reads every range it writes back from the target and holds it to
what it copied, and compares a bulk program's result with the source before
it calls the move complete. `options: { verify_batches: false }` turns that
off for a hop where the time matters more.

## Commands

<!-- gen_docs: commands -->
| Command | What it does |
|---|---|
| `init` | Set up a hop: in a terminal, asks for the two addresses and the passwords, connects, says what it found and what it will do, and writes the hop. |
| `doctor` | What this machine can do, and whether the configured hops connect. |
| `scope` | Say what the hop takes and leaves out - its include and exclude lists, the rows, columns and names it narrows, the changes it does not carry - and the target's tables left out as a transfer service's own, by name and by whose they are. |
| `leftovers` | Give back the writes a run held on a side and left behind: each exactly as its record took it, where its holder is gone; one whose holder still runs is left, with why. |
| `drain` | Drain the running change tail to a position and hold it there: it cannot be taken back, so it is shown first and taken with --go against the target the plan named. |
| `change-sheet` | Write the plan's steps as a change sheet (CSV and JSON beside the hop's reports) with stable ids a team's task sheet can hold, or, with --back, write a run's evidence back to one step. |
| `advise` | Show the migration playbook for this hop's service. |
| `assess` | Premigration readiness check: CDC prerequisites, risky tables, encoding, extensions, accounts. |
| `schema` | Target schema workflows: preparation plan (default), cross-engine DDL conversion (--convert), versioned migration files (--migration). |
| `check` | Read-only validation of target vs source. |
| `sync` | Make target equal to source, or run a whole migration flow. |
| `migrate` | Two addresses and a goal in; the whole migration run to a verified outcome: the assessment, the copy, the catch-up, the proof and the cutover's readiness, every way chosen by migkit, a database that fails held apart while the others go on, and asked only what migkit cannot find out (`autonomous.QUESTIONS`). |
| `move` | Move data for this hop. |
| `cutover` | The cutover: live switch with --go, or rehearse with --dry-run. |
| `watch` | Watch a running migration load: row counts, rate, ETA, replication state. |
| `report` | Render the HTML report from the last check run and export the hop's evidence (what moved, every verdict with both sides' counts and digests and how to take them again, decisions, approvals, repairs, outages - HTML, Markdown, JSON, CSV), or --serve the web view: every hop's outcomes, progress, approvals, offers and evidence for the people each hop names, the REST API, and a Prometheus /metrics endpoint (`migkit.web`). |
| `history` | Audit trail: saved rollback states and every write migkit made, read from the local changelog and state journals. |
| `autopilot` | Execute turn-key autonomous end-to-end migration pipeline. |
| `failback` | Execute zero-loss reverse failback to source within 48h post-cutover. |
| `rollback` | Restore target sequences from a saved state, put back the rows the repairs since it wrote, where the target still holds what they wrote; where the hop streamed back or cut over, the old source's counters raised past the new primary's. |
| `users` | Sync logins source -> target keeping the same password. |
| `telemetry` | Product telemetry: what migkit did, never what it carried. |
| `secrets` | The user's own keys: replace the masking key, describe it. |
| `export-manifest` | Export neutral portable manifest package for exit portability (Row 668: NEED20). |
| `diagnostics-bundle` | Generate bounded, sanitized support diagnostic bundle (Row 671: NEED23). |
| `verify-partial-access` | Verify partial access export against upstream objects with honest coverage (Row 672: NEED24). |
| `estate` | Every database these addresses, networks and accounts reach: its engine, version and size, the time its move takes by migkit's own measured runs, and its cost on the catalog's machines. |
<!-- /gen_docs: commands -->

Every check is read-only and rerunnable. Every repair is dry-run unless
`--apply`/`--go`, saves an undo first, and converges to the same end state on
re-run. Add `-q/--quiet` before any command to silence per-table output.

The pre-0.2 command names (`hops`, `setup-target`, `repair`, `replicate`,
`tail`, `convert-schema`, `gen-migration`, `sample-diff`, `ui`, `state`,
`monitor`) still work as hidden aliases, so existing scripts keep running.

## One result shape

Every engine words its own checks differently: PostgreSQL reports `encoding`
where MySQL reports `charset`, and MongoDB reports `null-missing` where
PostgreSQL reports `nullempty`. Those are the same two failures. `check`
writes `verdict.json`, in which they carry the same name on every engine, so a
report can be read - and aggregated across hops - without knowing which
engine produced it.

```json
{
  "format_version": 1,
  "tool": "migkit",
  "hop": "prod-cutover",
  "status": "different",
  "has_differences": true,
  "totals": {"ok": 231, "warn": 0, "diff": 3, "error": 0, "skip": 0},
  "by_category": {"access.sequence-grants": {"diff": 1}},
  "fingerprint": "9f2c...",
  "findings": [{"category": "identity.sequence-collision", "...": "..."}]
}
```

- `status` is one of `same`, `different`, `error`, `incomplete`
- `category` is engine-independent and stable: `value.charset`,
  `value.collation`, `value.null-empty`, `value.timezone`, `value.precision`,
  `identity.sequence-collision`, `access.sequence-grants`,
  `structure.partitions`, `movement.target-ahead`, `parity.row-count`, and so
  on. Categories are only renamed with a `format_version` bump.
- `findings` holds what is not `ok`, so an empty list means a clean run
- `fingerprint` covers the verdicts, not the wording, so a repeated check
  reports "identical to the previous run" instead of making you diff two
  reports by eye

`summary.json` is still written unchanged next to it.

## Supported engines

<!-- gen_docs: engines -->
64 engines, any two of them as a pair; migkit's own decision engine finds the fastest, exact, consistent way for each source -> target pair. What each can do, cell by cell, is on [the engines page](docs/site/engines.html).

| Role | Engines |
|---|---|
| either side | PostgreSQL (`postgres`), MySQL (`mysql`), MongoDB (`mongodb`), SQL Server (`mssql`), Redis (`redis`), Kafka (`kafka`), SQLite (`sqlite`), Parquet files (`parquet`), ClickHouse (`clickhouse`), DynamoDB (`dynamodb`), Oracle Database (`oracle`), Db2 (Linux, UNIX, Windows) (`db2`), Db2 for i (`db2i`), Db2 for z/OS (`db2zos`), SAP ASE (`ase`), Teradata (`teradata`), OpenSearch (`opensearch`), Cassandra (`cassandra`), Amazon Redshift (`redshift`), Snowflake (`snowflake`), BigQuery (`bigquery`), DuckDB (`duckdb`), Files described by COBOL copybooks (`copybook`), Iceberg tables (`iceberg`), Delta tables (`delta`), TiDB (`tidb`), CockroachDB (`cockroachdb`), Aurora DSQL (`dsql`), Couchbase (`couchbase`), Cosmos DB for NoSQL (`cosmosdb`), Google Cloud Spanner (`spanner`), Google Cloud Bigtable (`bigtable`), Firestore (native mode) (`firestore`), Firestore in Datastore mode (`datastore`), GaussDB (openGauss) (`gaussdb`), OceanBase (MySQL mode) (`oceanbase`), OceanBase (Oracle mode) (`oceanbase-oracle`), PolarDB-X (`polardbx`), TDSQL (distributed) (`dcdb`), Amazon Keyspaces (`keyspaces`), Amazon Neptune (`neptune`), Amazon Timestream (`timestream`), Lindorm (`lindorm`), GeminiDB (Cassandra API) (`geminidb-cassandra`), GeminiDB (Redis API) (`geminidb-redis`), GeminiDB (MongoDB API) (`geminidb-mongo`), Qdrant (`qdrant`), Neo4j (`neo4j`), AnalyticDB for MySQL (`analyticdb-mysql`), AnalyticDB for PostgreSQL (`analyticdb-postgres`), Azure Synapse (dedicated SQL pool) (`synapse`), YugabyteDB (`yugabytedb`), SingleStore (`singlestore`), Netezza (`netezza`), SAP HANA (`saphana`), Informix (`informix`), Progress OpenEdge (`progress`), Vertica (`vertica`), Exasol (`exasol`), Databricks SQL (`databricks`) |
| a stream a pair delivers its changes into | Kinesis Data Streams (`kinesis`), Google Pub/Sub (`pubsub`) |
| a source, read and never written | SAP application data (ODP) (`sapodp`), SaaS applications, read by a connector program (`saas`) |
| the pair: a source of one engine, a target of another | Any pair of two engines (`hetero`) |
| either side, compared by the tables the hop lists | Any other SQL engine (`generic`) |

Also written: `adb-mysql`, `adb-pg`, `alloydb`, `amazon-keyspaces`, `amazon-neptune`, `amazon-timestream`, `analyticdb-pg`, `as400`, `aurora-dsql`, `aurora-mysql`, `aurora-postgres`, `aws-keyspaces`, `azure-cosmos`, `azure-sql`, `azure-synapse`, `cloud-bigtable`, `cloud-datastore`, `cloud-spanner`, `cobol`, `cockroach`, `cosmos`, `cosmosdb-mongo`, `cosmosdb-nosql`, `crdb`, `databricks-sql`, `db2-for-i`, `db2-zos`, `db2z`, `delta-lake`, `documentdb`, `drds`, `elasticsearch`, `firestore-datastore`, `firestore-native`, `gaussdb-opengauss`, `geminidb`, `hana`, `ibmi`, `iseries`, `mariadb`, `memsql`, `neo4j-aura`, `oceanbase-mysql`, `odp`, `openedge`, `opengauss`, `percona`, `pgsql`, `polardb-x`, `postgresql`, `program`, `progress-openedge`, `pure-data-netezza`, `sap-ase`, `sap-hana`, `sap-odp`, `scylla`, `scylladb`, `spanner-pg`, `sqlserver`, `sybase`, `td`, `tdsql`, `tdsql-c`, `tdsql-distributed`, `tdsql-shard`, `valkey`, `vantage`, `vsam`, `ysql`, `yugabyte`.

Managed services recognised: Aiven for Apache Cassandra (Aiven); Aiven for Apache Kafka (Aiven); Aiven for ClickHouse (Aiven); Aiven for MySQL (Aiven); Aiven for OpenSearch (Aiven); Aiven for PostgreSQL (Aiven); Aiven for Valkey (Aiven); Alibaba Cloud Elasticsearch (Alibaba Cloud); AlloyDB for PostgreSQL (Google Cloud); Amazon Aurora DSQL (AWS); Amazon Aurora MySQL (AWS); Amazon Aurora PostgreSQL (AWS); Amazon DocumentDB (AWS); Amazon DynamoDB (AWS); Amazon ElastiCache (Valkey, Redis OSS) (AWS); Amazon Keyspaces (for Apache Cassandra) (AWS); Amazon Kinesis Data Streams (AWS); Amazon MSK (AWS); Amazon MemoryDB (AWS); Amazon Neptune (AWS); Amazon OpenSearch Service (AWS); Amazon RDS for Db2 (AWS); Amazon RDS for MariaDB (AWS); Amazon RDS for MySQL (AWS); Amazon RDS for Oracle (AWS); Amazon RDS for PostgreSQL (AWS); Amazon RDS for SQL Server (AWS); Amazon Redshift (AWS); Amazon Timestream for InfluxDB (AWS); Amazon Timestream for LiveAnalytics (AWS); AnalyticDB for MySQL (Alibaba Cloud); AnalyticDB for PostgreSQL (Alibaba Cloud); ApsaraDB RDS for MariaDB (Alibaba Cloud); ApsaraDB RDS for MySQL (Alibaba Cloud); ApsaraDB RDS for PostgreSQL (Alibaba Cloud); ApsaraDB RDS for SQL Server (Alibaba Cloud); ApsaraDB for MongoDB (Alibaba Cloud); ApsaraDB for OceanBase (MySQL mode) (Alibaba Cloud); ApsaraDB for OceanBase (Oracle mode) (Alibaba Cloud); ApsaraMQ for Kafka (Alibaba Cloud); Azure Cache for Redis (Azure); Azure Cosmos DB for Apache Cassandra (Azure); Azure Cosmos DB for MongoDB (RU) (Azure); Azure Cosmos DB for NoSQL (Azure); Azure Cosmos DB for PostgreSQL (Azure); Azure Database for MySQL flexible server (Azure); Azure Database for PostgreSQL flexible server (Azure); Azure DocumentDB (formerly Azure Cosmos DB for MongoDB vCore) (Azure); Azure Event Hubs (Kafka endpoint) (Azure); Azure Managed Redis (Azure); Azure SQL Database (Azure); Azure SQL Managed Instance (Azure); Azure Synapse Analytics (dedicated SQL pool) (Azure); BigQuery (Google Cloud); Bigtable (Google Cloud); Cloud SQL for MySQL (Google Cloud); Cloud SQL for PostgreSQL (Google Cloud); Cloud SQL for SQL Server (Google Cloud); CockroachDB Cloud (CockroachDB Cloud); Confluent Cloud (Confluent Cloud); Crunchy Bridge (Crunchy Bridge); Firestore (Google Cloud); Firestore in Datastore mode (Google Cloud); Huawei Cloud Distributed Cache Service (DCS) for Redis (Huawei Cloud); Huawei Cloud Distributed Message Service (DMS) for Kafka (Huawei Cloud); Huawei Cloud Document Database Service (DDS) (Huawei Cloud); Huawei Cloud GaussDB (Huawei Cloud); Huawei Cloud GeminiDB (Cassandra API) (Huawei Cloud); Huawei Cloud GeminiDB (DynamoDB, HBase and InfluxDB APIs) (Huawei Cloud); Huawei Cloud GeminiDB (MongoDB API) (Huawei Cloud); Huawei Cloud GeminiDB (Redis API) (Huawei Cloud); Huawei Cloud RDS for MySQL (Huawei Cloud); Huawei Cloud RDS for PostgreSQL (Huawei Cloud); Huawei Cloud RDS for SQL Server (Huawei Cloud); Huawei Cloud Search Service (CSS) (Huawei Cloud); Huawei Cloud TaurusDB (formerly GaussDB(for MySQL)) (Huawei Cloud); IBM Cloud Databases for Elasticsearch (IBM Cloud); IBM Cloud Databases for MongoDB (IBM Cloud); IBM Cloud Databases for MySQL (IBM Cloud); IBM Cloud Databases for PostgreSQL (IBM Cloud); IBM Cloud Databases for Redis (IBM Cloud); IBM Db2 Warehouse on Cloud (IBM Cloud); IBM Db2 on Cloud (Db2 SaaS) (IBM Cloud); IBM Event Streams (IBM Cloud); Lindorm (Alibaba Cloud); Memorystore for Redis (Google Cloud); Memorystore for Redis Cluster (Google Cloud); Memorystore for Valkey (Google Cloud); MongoDB Atlas (MongoDB Atlas); MySQL HeatWave (Oracle Cloud); Neon (Neon); OCI Cache (Oracle Cloud); OCI Database with PostgreSQL (Oracle Cloud); OCI Streaming (Kafka-compatible) (Oracle Cloud); Oracle Autonomous Database (Oracle Cloud); Oracle Base Database Service (Oracle Cloud); PlanetScale (MySQL, on Vitess) (PlanetScale); PlanetScale for Postgres (PlanetScale); PolarDB for MySQL (Alibaba Cloud); PolarDB for PostgreSQL (Alibaba Cloud); PolarDB-X (Alibaba Cloud); Pub/Sub (Google Cloud); Redis Cloud (Redis Cloud); SingleStore Helios (SingleStore Helios); Spanner (Google Cloud); Supabase (Supabase); TDSQL (distributed) (Tencent Cloud); TDSQL-C for MySQL (Tencent Cloud); TDSQL-C for PostgreSQL (Tencent Cloud); Tair (ApsaraDB for Redis) (Alibaba Cloud); Tencent Cloud CKafka (Tencent Cloud); Tencent Cloud Elasticsearch Service (Tencent Cloud); TencentDB for MariaDB (Tencent Cloud); TencentDB for MongoDB (Tencent Cloud); TencentDB for MySQL (Tencent Cloud); TencentDB for PostgreSQL (Tencent Cloud); TencentDB for Redis (Tencent Cloud); TencentDB for SQL Server (Tencent Cloud); Teradata VantageCloud Lake (Teradata VantageCloud); Tiger Cloud (Timescale) (Tiger Cloud (Timescale)); YugabyteDB Aeon (YugabyteDB Aeon).
<!-- /gen_docs: engines -->

## Cross-engine (hetero)

MySQL -> PostgreSQL is verified end to end:

```bash
migkit schema my2pg --convert --apply    # transpile the DDL and apply it
migkit move   my2pg --go                 # resumable chunked copy
migkit move   my2pg --mode cdc --db X --go   # CDC from the binlog, checkpointed
migkit check  my2pg                      # cross-dialect row verify
```

The `hetero` engine is an orchestrator that reuses the per-side native engines,
so every pair follows the same shape: `source_engine` and `target_engine` in
the hop's options name any two of the engines above.

## How it compares

migkit combines migration, verification and repair in one workflow. The tables
below summarize the intended scope of the tool categories. Actual support varies
by engine, service version and configuration; they are not a benchmark.

| Capability | migkit | AWS DMS / Tencent DTS | a change-capture connector | a table-sync script | GoldenGate + Veridata |
|---|:--:|:--:|:--:|:--:|:--:|
| Bulk load (drives best mover) | yes | yes | no | partial | yes |
| Change data capture | yes | yes | yes | no | yes |
| Row-level verify with proof (LSN fence) | yes | partial | no | partial | yes |
| Consistency fence (no false diffs) | yes | no | no | partial | partial |
| Continuous delta verify, O(changes) | yes | no | no | no | partial |
| Column-level diff localization | yes | no | no | no | partial |
| Schema-object verify (views/routines/FK/index) | yes | no | no | no | partial |
| Sequence / identity carry and verify | yes | no | no | no | partial |
| Row-level repair with undo | yes | no | no | partial | yes |
| Schema repair (apply DDL, with undo) | yes | no | no | no | partial |
| Rollback / state snapshots | yes | no | no | no | partial |
| Self-hosted, no vendor lock-in | yes | no | yes | yes | partial |
| Zero footprint on the target | yes | no | yes | yes | partial |
| Engines | [every one below](#supported-engines), and any pair | cloud-scoped | several | one | oracle-centric |
| License | free (AGPL-3.0) | paid | free | free | commercial |

### Silent-corruption detection

These checks look for problems that a completed load or matching row count can
miss. Which checks apply depends on the database engine and the hop.

| Silent failure (data looks "present", is wrong) | migkit | DMS/DTS | Veridata | a row-diff tool | a table-sync script |
|---|:--:|:--:|:--:|:--:|:--:|
| Sequence / identity collision (`nextval <= max(pk)`) | yes | no | no | no | no |
| Type-narrowing / silent truncation (varchar, scale, int) | yes | no | no | no | no |
| Charset corruption (utf8mb4 cut, latin1 mojibake, U+FFFD) | yes | no | no | no | no |
| Uniform timezone offset (systematic, not row noise) | yes | no | no | no | no |
| Collation unique-collapse + glibc/ICU version drift | yes | no | no | no | no |
| Partition routing (rows stranded in default/MAXVALUE) | yes | no | no | no | no |
| Generated/computed column drift (stored != expression) | yes | no | no | no | no |
| RLS partial-dump / default-deny lockout | yes | no | no | no | no |
| NULL vs empty-string flip (Oracle `''`=NULL) | yes | no | no | no | no |
| No-PK table (CDC drops updates/deletes) | yes | no | partial | no | no |
| NOT VALID / untrusted constraints + FK orphans | yes | no | no | no | no |
| Deferrable-constraint drift | yes | no | no | no | no |
| Extensions + sequence-level grant parity | yes | no | no | no | no |
| Replication slot bloat / abandoned slot / long-txn | yes | partial | no | no | no |

PostgreSQL and MySQL have the deepest coverage. See the
[engine tables](https://migkit.axentx.cloud/site/engines.html) for the checks
available on each engine and the gaps still being worked on.

## Safety model

- `check` (incl. `--deep`/`--drill`), `assess`, `watch`, `report`, `history`
  are read-only and can run anytime.
- `sync`, `move` (all modes), `schema --convert` write to the target; all are
  dry-run by default and require `--apply`/`--go`.
- The source is written only where the hop asks for it itself: the `two-way`
  goal, the cutover migkit drives (`cutover:`), or a change on the source a
  faster way needs and the operator agreed to (`source_changes`). What each
  goal does on each side is said before it runs
  ([goals and modes](docs/goals-and-modes.md)).
- A lock file prevents concurrent writes; every write is recorded in the local
  changelog ledger.
- Movers are self-hosted only on a trusted network (or a cloud VM); managed
  services remain the recommended path over long-haul links.

## Telemetry

migkit keeps product telemetry about how it behaved - the command, the
engines, the ways chosen and why (as codes), timings, resources, progress
and estimate accuracy, error categories, recoveries, verified coverage and
guarantees, how a run ended - so the product can learn which ways work.
It never keeps rows, values, digests of them, statements, schemas, names
of databases, tables, columns or accounts, addresses, paths, credentials,
logs, stack traces or error messages: every field is a closed code, a
bucket or a bounded number, and anything else is refused where the event
is made.

- Default: on, kept on this machine only, in a spool of a fixed size under
  the state directory. Nothing is sent: no destination is chosen.
- `migkit telemetry show` prints the state and every event kept
  (`--schema` lists every field).
- Off: `migkit telemetry off`, `MIGKIT_TELEMETRY=off`, `MIGKIT_EGRESS=none`,
  or an organisation policy file (`/etc/migkit/policy.yaml`, or
  `MIGKIT_TELEMETRY_POLICY`) saying `telemetry: off`. Off deletes what was
  kept.

## Testing

```bash
pip install -e ".[dev]"
pytest tests/ -q -m "not docker"    # unit + fail-case only, no docker
python tools/final_suite.py         # the whole suite: pure tests side by side,
                                    # docker tests in lanes, a report
```

100+ tests: pure-logic units, CLI-surface tests (every visible command, legacy
aliases stay invocable), mover and pipeline selection, test_decoding
parsing, end-to-end integration against throwaway Postgres containers
(including the full delta-verify loop: touch -> flag -> replay -> repair ->
advance, and the consistent-snapshot pass), exact repair-undo restore against
a MySQL pair, Faker-generated data covering every column type, and
failure-mode tests (bad credentials, missing state, locks, no-PK tables,
credential drift). CI runs the no-docker subset on every push and the full
suite on Ubuntu runners.

## License

AGPL-3.0-or-later - see [LICENSE](LICENSE). migkit takes in, as its own
dependencies, every open tool and library that adds a capability,
copyleft ones included, and the Affero GPL is the licence all of them
can be combined under. The programs and libraries it drives keep their
own licences; those that are not open source are installed only after
the operator accepts their terms (`migkit doctor --install`) and used
only for what those terms allow.
