Metadata-Version: 2.4
Name: terminusdb-migrations
Version: 0.2.1
Summary: Alembic-style migration history and schema migration tooling for TerminusDB
License-Expression: MIT
Project-URL: Source, https://gitlab.com/ed-tech6370840/terminusdb-migrations
Project-URL: Issues, https://gitlab.com/ed-tech6370840/terminusdb-migrations/-/issues
Project-URL: PyPI, https://pypi.org/project/terminusdb-migrations/
Keywords: terminusdb,migration,schema,pydantic,graph-database,alembic
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: terminusdb-async<1.0.0,>=0.1.0
Requires-Dist: terminusdb-pydantic<1.0.0,>=0.1.0
Requires-Dist: typer<1.0.0,>=0.12
Dynamic: license-file

# terminusdb-migrations

[![PyPI](https://img.shields.io/pypi/v/terminusdb-migrations.svg)](https://pypi.org/project/terminusdb-migrations/)
[![Python](https://img.shields.io/pypi/pyversions/terminusdb-migrations.svg)](https://pypi.org/project/terminusdb-migrations/)

Alembic-style migration history for TerminusDB.

Version 0.2 adds repository-backed migration files, explicit revision chains,
upgrade and downgrade execution, database revision tracking, interrupted
migration detection, and a Python operation DSL on top of the native
TerminusDB Migration API.

> Status: pre-1.0. Review migration files and dry-run destructive changes
> before using them against production data.
>
> This is a community package, not an official TerminusDB project.

## Installation

With uv:

~~~bash
uv add terminusdb-migrations
~~~

With pip:

~~~bash
python -m pip install terminusdb-migrations
~~~

Python 3.11+ is supported.

Internal package compatibility remains:

~~~text
terminusdb-async    >=0.1.0,<1.0.0
terminusdb-pydantic >=0.1.0,<1.0.0
~~~

Patch releases do not raise these lower bounds unless they actually need newer
API.

## Repository-backed migrations

A project keeps migration files in Git:

~~~text
migrations/
├── README.md
└── versions/
    ├── 20261004_001_initial.py
    ├── 20261005_002_add_description.py
    └── 20261006_003_rename_title.py
~~~

Each file declares revision, down_revision, upgrade, and downgrade.

Migration callbacks are deliberately synchronous and declarative:

~~~python
def upgrade(op: Operations) -> None:
    ...
~~~

They only build an in-memory migration plan. Database I/O remains asynchronous
inside the runner. Async functions, generator functions, and callbacks that
return a value/coroutine are rejected before migration state is changed.

The same chain can be applied independently to development, staging, and
production databases.

## Initialize

~~~bash
tdb-migrate init
~~~

Use another location when needed:

~~~bash
tdb-migrate init --migrations-dir db/migrations
~~~

Runtime commands require an initialized `<migrations-dir>/versions` directory.
A missing directory is treated as configuration error, not as an empty migration
history. This prevents a typo in `--migrations-dir` from producing a false
successful deployment.

## Create a revision

~~~bash
tdb-migrate revision -m "add discipline description"
~~~

Or provide a stable custom revision id:

~~~bash
tdb-migrate revision \
  --rev-id 20261004_001 \
  -m "add discipline description"
~~~

Generated migration:

~~~python
from terminusdb_migrations import Operations, optional

revision = "20261004_001"
down_revision = None
message = "add discipline description"


def upgrade(op: Operations) -> None:
    op.create_class_property(
        "Discipline",
        "description",
        optional("xsd:string"),
    )


def downgrade(op: Operations) -> None:
    op.delete_class_property(
        "Discipline",
        "description",
    )
~~~

The next generated revision automatically points at the current repository head.

## Linear revision graph

Version 0.2 intentionally supports one linear history:

~~~text
base
  |
  v
20261004_001
  |
  v
20261005_002
  |
  v
20261006_003
~~~

Disconnected, duplicated, cyclic, or branched histories are rejected before any
database migration is attempted.

Branching histories and merge revisions are planned for a later release.

## Current revision

~~~bash
tdb-migrate current --db edtech
~~~

Example:

~~~text
current: 20261005_002
head:    20261006_003
~~~

## History

~~~bash
tdb-migrate history --db edtech
~~~

Example:

~~~text
20261006_003 <- 20261005_002  rename title
20261005_002 <- 20261004_001 *  add description
20261004_001 <- base  initial schema
~~~

The asterisk marks the current database revision.

## Upgrade

Upgrade to head:

~~~bash
tdb-migrate upgrade head --db edtech
~~~

Upgrade one step:

~~~bash
tdb-migrate upgrade +1 --db edtech
~~~

Upgrade to an explicit revision:

~~~bash
tdb-migrate upgrade 20261006_003 --db edtech
~~~

Dry run:

~~~bash
tdb-migrate upgrade head \
  --db edtech \
  --dry-run
~~~

## Downgrade

Revert one revision:

~~~bash
tdb-migrate downgrade -1 --db edtech
~~~

Return to an explicit revision:

~~~bash
tdb-migrate downgrade 20261004_001 --db edtech
~~~

Return to the state before the first migration:

~~~bash
tdb-migrate downgrade base --db edtech
~~~

Downgrade executes the migration file's downgrade function. It does not move the
TerminusDB branch pointer backwards. The rollback therefore becomes new,
auditable TerminusDB history.

## Operation DSL

Migration functions receive an Operations object.

### Create and delete a class

~~~python
def upgrade(op: Operations) -> None:
    op.create_class(
        {
            "@id": "Discipline",
            "@key": {
                "@type": "Lexical",
                "@fields": ["code"],
            },
            "code": "xsd:string",
            "name": "xsd:string",
        }
    )


def downgrade(op: Operations) -> None:
    op.delete_class("Discipline")
~~~

### Create and delete a property

~~~python
op.create_class_property(
    "Discipline",
    "description",
    optional("xsd:string"),
)

op.delete_class_property(
    "Discipline",
    "description",
)
~~~

### Rename a property

~~~python
op.move_class_property(
    "Discipline",
    "title",
    "name",
)
~~~

### Cast a property

By default a cast is strict: if any existing value cannot be converted,
TerminusDB rejects the migration.

~~~python
op.cast_class_property(
    "Discipline",
    "credits",
    "xsd:integer",
)
~~~

Fallback replacement is intentionally not exposed in 0.2 for TerminusDB
12.0.7. The server accepts a `Default/value` descriptor at the JSON parser
boundary but currently does not execute that fallback correctly. Passing
`default=...` therefore raises `UnsupportedCastDefault` locally instead of
sending a migration that ends in an HTTP 500.

Tracking issue:
[issue #2](https://gitlab.com/ed-tech6370840/terminusdb-migrations/-/work_items/2).

### Change a key

~~~python
op.change_key(
    "Discipline",
    {
        "@type": "Lexical",
        "@fields": ["code"],
    },
)
~~~

Key changes are marked destructive.

### Expand an enum

~~~python
op.expand_enum(
    "EducationLevel",
    ["specialist"],
)
~~~

### Raw native operation

~~~python
op.raw(
    {
        "@type": "SomeFutureMigrationOperation",
        "...": "...",
    }
)
~~~

This keeps the wrapper open to new TerminusDB migration operations without
waiting for a convenience method in this package.

### Type helpers

~~~python
from terminusdb_migrations import list_of, optional, set_of

optional("xsd:string")
list_of("LearningOutcome")
set_of("Competency")
~~~

## Irreversible migrations

Not every destructive change has a meaningful downgrade.

Mark that explicitly:

~~~python
def downgrade(op: Operations) -> None:
    op.irreversible(
        "legacy_code values were deleted and cannot be reconstructed"
    )
~~~

The downgrade fails before any migration request is sent.

## Destructive migrations

Deleting classes or properties and changing keys are marked destructive.

They require explicit permission:

~~~bash
tdb-migrate upgrade head \
  --db edtech \
  --allow-destructive
~~~

The same flag is available for downgrade.

Use dry-run first when possible.

For a chain of native Migration API operations, `--dry-run` validates the
whole path in one TerminusDB dry-run request. This preserves the virtual
intermediate schema, so a revision may depend on classes or properties created
by an earlier revision in the same path.

A path containing `op.schema_document(...)` cannot currently be represented
inside that same native transaction. Version 0.2 fails explicitly with
`UnsupportedDryRun` instead of reporting a false successful validation.
Full mixed-operation dry-run via a temporary TerminusDB branch is tracked in
[issue #1](https://gitlab.com/ed-tech6370840/terminusdb-migrations/-/work_items/1).

## Database revision state

The library stores migration state in an internal TerminusDB document of type:

~~~text
TerminusDBMigrationState
~~~

It tracks:

~~~text
revision
pending_revision
direction
owner
~~~

State acquisition uses TerminusDB's `TerminusDB-Data-Version` header as an
optimistic compare-and-set. Two runners cannot both acquire the same clean
revision: one wins the state update and the other observes the changed branch
version. The runner also checks that the stored revision still matches the
revision it planned from, so a stale deployment cannot replay an older
migration.

Completion is tied to the owner token that acquired the pending state.

The internal schema class is automatically excluded from the legacy
Pydantic-to-schema diff.

### Dirty state protection

Schema migration and revision-state update use separate TerminusDB API requests.
To make interruptions detectable, the runner first records the migration as
pending, then runs it, then advances the current revision.

If the process stops in the middle, current shows a dirty state and further
upgrade or downgrade commands refuse to continue.

This prevents the database from silently pretending that an interrupted
migration never started.

## Recovering from an interrupted migration

First inspect the actual database schema and determine what was applied.

Then reconcile the revision marker explicitly:

~~~bash
tdb-migrate stamp 20261005_002 \
  --db edtech \
  --force
~~~

Or return the marker to base:

~~~bash
tdb-migrate stamp base \
  --db edtech \
  --force
~~~

Stamp changes only migration metadata. It does not execute schema operations.

## Python API

~~~python
from terminusdb_async import AsyncTerminusClient
from terminusdb_migrations import (
    MigrationContext,
    MigrationRepository,
)


async def migrate() -> None:
    repository = MigrationRepository("migrations")

    async with AsyncTerminusClient(
        "http://localhost:6363",
        organization="admin",
        database="edtech",
        username="admin",
        password="root",
    ) as db:
        context = MigrationContext(
            db,
            repository,
            author="schema-bot",
        )

        state = await context.current()
        print(state.revision)

        await context.upgrade("head")
~~~

## Connection configuration

Database commands accept:

~~~text
--url
--org
--db
--branch
--username
--password
--token
--author
~~~

Environment variables:

~~~text
TERMINUSDB_URL
TERMINUSDB_ORG
TERMINUSDB_USER
TERMINUSDB_PASSWORD
TERMINUSDB_TOKEN
~~~

Example:

~~~bash
export TERMINUSDB_URL=http://localhost:6363
export TERMINUSDB_ORG=admin
export TERMINUSDB_USER=admin
export TERMINUSDB_PASSWORD=root

tdb-migrate current --db edtech
tdb-migrate upgrade head --db edtech
~~~

## Legacy model diff

The 0.1 API remains available:

~~~bash
tdb-migrate plan \
  --models app.models:MODELS \
  --db edtech
~~~

And:

~~~bash
tdb-migrate apply \
  --models app.models:MODELS \
  --db edtech \
  --dry-run
~~~

For repeatable deployment, explicit migration files are recommended.

## Scope of 0.2

Included:

- migration files stored in the application repository;
- revision and down_revision metadata;
- validated linear revision graph;
- current and history commands;
- forward upgrades;
- explicit downgrades;
- relative targets such as +1 and -1;
- revision state stored in TerminusDB;
- dirty-state detection;
- optimistic concurrency control for migration-state acquisition;
- stale-revision rejection;
- sequential chained dry-run for native migration operations;
- recovery with stamp;
- operation DSL;
- irreversible downgrade support;
- legacy model diff compatibility.

Deferred:

- revision --autogenerate;
- branching migration histories;
- merge revisions;
- automatic rename intent inference;
- automatic backfill generation;
- hard TerminusDB branch reset as ordinary downgrade;
- mixed schema-document/native dry-run via a temporary branch (issue #1).

Autogeneration can be layered on top of the existing Pydantic schema diff in a
future release.

## Package family

| Package | Purpose |
| --- | --- |
| [terminusdb-async](https://pypi.org/project/terminusdb-async/) | Async TerminusDB HTTP client |
| [terminusdb-pydantic](https://pypi.org/project/terminusdb-pydantic/) | Pydantic v2 to JSON-LD and schema generation |
| [terminusdb-migrations](https://pypi.org/project/terminusdb-migrations/) | Repository-backed migration history |

## Development

~~~bash
uv sync --group dev
uv run pytest -q
uv build --no-sources
~~~

CI tests Python 3.11, 3.12, and 3.13.

Development follows GitHub Flow on GitLab: short-lived branches are merged into
main through Merge Requests.

## License

MIT License. See [LICENSE](LICENSE) for the full text.
