Metadata-Version: 2.5
Name: airflow-migration-helper
Version: 1.1.0
Summary: Utility for migrating Airflow code from version 2 to version 3.
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.10
Requires-Dist: packaging
Requires-Dist: ruff<0.16,>=0.15.5
Requires-Dist: tomli>=1.1.0; python_version < '3.11'
Provides-Extra: airflow
Requires-Dist: apache-airflow==3.2.1; extra == 'airflow'
Provides-Extra: dev
Requires-Dist: apache-airflow==3.2.1; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# Migration Helper

`airflow-migration-helper` is a Python project for helping migrate Apache
Airflow code from Airflow 2 to Airflow 3. The PyPI distribution installs the
command-line tool as `migration-helper`.

The current implementation is focused on static analysis and migration
assistance for DAG code. It includes checks based on Ruff AIR rules, Airflow
configuration linting, DAG import checks, dependency compatibility checks, base
reporting, shared subprocess helpers, file selection helpers, and tests.

## How it works

`migration-helper` migrates code from Airflow 2 to Airflow 3, so it is run
against a target project that is currently on Airflow 2:

- The analyzed project (DAGs, configs, dependencies) stays on Airflow 2.
- The tool runs in its own dedicated virtual environment (see
  [Requirements and setup](#requirements-and-setup)).

Which Airflow version the checker environment needs depends on the checker:

- `ruff` and `dep_check` analyze the target statically and do not require
  Airflow at all.
- `airflow_config` checks config keys against a built-in static rule table, so
  it also works without Airflow. When Airflow is installed in the checker
  environment, the checker additionally runs `airflow config lint` and reports
  more findings. Note that the Airflow CLI commands used by this checker
  operate on the active configuration of the checker environment
  (`AIRFLOW_HOME` / `AIRFLOW_CONFIG`), not on `airflow.cfg` files found in the
  target; only the static scan covers target files. To lint or update a
  specific config file through the CLI path, point `AIRFLOW_CONFIG` at it.
- `dag_check` runs `airflow dags list-import-errors` from the checker
  environment, so it parses the Airflow 2 DAGs **under the Airflow installed in
  that environment** (Airflow 3 for full compatibility checks).
- `db_migrate` runs the one-shot, read-only `airflow db check-migrations`
  command from the checker environment. No running Airflow processes are
  needed, but a reachable metadata database must be configured for that
  environment. The stage is only enabled with `--db-migrate`.

Airflow's CLI does not run on native Windows (`airflow` imports POSIX-only
modules). On Windows, `dag_check` and `db_migrate` always report `FAILED`; run
the tool in WSL or a container when those stages matter.

Every checker reports an execution status alongside its findings:

- `success`: the check ran; its findings (including zero) are trustworthy.
- `partial`: the check ran with reduced capability, for example `airflow_config`
  with only the static rule table because Airflow is not installed.
- `failed`: the check could not run or its output could not be read. A failed
  checker is **never** rendered as `0 issues`.
- `skipped`: the stage is intentionally not run (for example `db_migrate`
  without `--db-migrate`) or there is nothing to check.

For full functionality (all checkers, including `dag_check`), install the tool
together with Airflow 3 in the same virtual environment:

```bash
python -m venv .venv-mh
.venv-mh/bin/pip install "airflow-migration-helper[airflow]"
```

## Expected Output

After running `migration-helper`, you get a practical migration report for the
selected Airflow project. The report lists compatibility issues by file and rule,
shows error/warning severity, and explains whether each finding can be handled
automatically or needs manual review.

Reports can be written in JSON and HTML formats. The JSON report is intended
for automation and CI, while the HTML report is intended for manual review.
Both formats expose the same issue details.

The JSON report has this top-level structure:

- `generated_at`: UTC timestamp when the report was created.
- `target`: analyzed file or directory.
- `summary`: total number of findings, split into `errors` and `warnings`, plus
  `checkers_failed`: the list of checkers that could not run.
- `checkers`: execution status of every checker (see below).
- `by_file`: findings grouped by affected file path.
- `by_type`: findings grouped by `rule_id`.

Each `checkers` entry describes one checker:

- `status`: `success`, `partial`, `failed`, or `skipped`.
- `reason`: machine-readable cause for non-success, for example
  `airflow_unavailable`, `output_unparseable`, or `disabled`.
- `issues`: the number of findings, present **only** for `success` and
  `partial`. A `failed` or `skipped` entry carries no count, so a check that
  did not run can never be read as `0 issues`.
- For `failed` entries the diagnostic detail is included: `error`,
  `command`, and `stdout_excerpt` when available.

The terminal summary prints the same information per checker:

```text
Per-checker breakdown:
  ruff            13 issue(s)
  airflow_config  0 issue(s)
  dag_check       FAILED       airflow is not available - install with the [airflow] extra
  dep_check       SKIPPED      no dependency file found
  db_migrate      SKIPPED      not enabled - pass --db-migrate
```

The green `No issues found. Ready for Airflow 3.` message is printed only when
every checker ran; if any checker failed, the summary instead reports that the
results are incomplete.

Each finding contains:

- `file`: affected file or logical target, for example `airflow.cfg` or
  `requirements.txt`.
- `line` and `col`: source location when the checker can provide it; otherwise
  `null`.
- `rule_id`: checker-specific rule or error identifier, for example `AIR301`,
  `CFG001`, or `DEP_RESOLUTION_FAILED`.
- `message`: short explanation of the problem.
- `severity`: `error`, `warning`, or `info`.
- `source`: checker that produced the finding, for example `ruff`,
  `airflow_config`, `dag_check`, `dep_check`, or `db_migrate`.

Reports use these fields to describe fixability:

- `fix_safety: "safe"` means the checker knows a safe mechanical fix or safe
  migration action. `migration-helper fix TARGET --safe-only` applies safe Ruff
  fixes and, when Airflow is available in the checker environment, also runs
  `airflow config update --fix` on that environment's active config (see
  [How it works](#how-it-works)). With `--dry-run`, Ruff applies those fixes
  on a temporary copy so `TARGET` is unchanged; config update stays in
  Airflow CLI dry-run mode and does not write.
- `fix_safety: "unsafe"` means the tool knows the likely change, but a human
  should review it before applying.
- `fix_safety: null` (or `fix_safety: "manual"` in HTML report) means there is
  no automatic fix path in the current implementation. Treat it as manual work.
- `fix_description` is the human-readable hint for what to change or which
  command/path is relevant.

Today, DAG import errors, dependency conflicts, and DB migration findings are
diagnostic/manual. Source-level DAG fixes outside Ruff are still planned.

## Requirements and setup

- Python `>=3.10`
- pip version `>=22.2`
- `git` available in `PATH`
- Work in a dedicated virtual environment (see below)

Install `airflow-migration-helper` and its dependencies (`ruff` and others)
into a dedicated virtual environment, not into the system Python. This keeps
the tool's dependencies isolated from the analyzed project.

```bash
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
```

Install from PyPI:

```bash
pip install airflow-migration-helper[airflow]
```

### Offline installation

To install on a machine without access to PyPI, build a wheelhouse on a machine
with internet access and install from it:

```bash
# On a machine with internet access
pip download airflow-migration-helper[airflow]==1.1.0 -d wheelhouse/

# On the offline machine
pip install --no-index --find-links ./wheelhouse airflow-migration-helper
```

Note: `ruff` is a platform-specific binary package. Download the wheelhouse on
a machine with the same OS and the same Python version as the target machine.

If the wheelhouse must be prepared for a different platform than the download
machine, pass the target platform tags explicitly. This works only with
pre-built wheels, so `--only-binary=:all:` is required:

```bash
# Build a wheelhouse for manylinux x86_64 + CPython 3.11 from any machine
pip download airflow-migration-helper[airflow]==1.1.0 -d wheelhouse/ \
  --only-binary=:all: \
  --platform manylinux2014_x86_64 \
  --python-version 311 \
  --implementation cp \
  --abi cp311
```

`ruff` ships platform-specific wheels (`py3-none-manylinux2014_x86_64`,
`macosx_11_0_arm64`, `win_amd64`, ...) so the `--platform` tag matters most;
`packaging` and `tomli` are pure Python and work everywhere. Repeat the command
per target platform (for example `macosx_11_0_arm64` or `win_amd64`) or build
the wheelhouse on a machine matching the target configuration.

Alternatives: copy a prepared virtual environment as a whole (same OS /
architecture / Python required), or serve the packages from an internal PyPI
mirror (devpi, Nexus, Artifactory) and use `pip config set global.index-url
http://<mirror>/`.

Offline usage notes: Airflow 3.2.1 constraints for Python 3.10-3.14 are bundled
with the package, so the default dependency check works offline. Other Airflow
versions require network access to download constraints, and the deep
dependency check (`pip install --dry-run` / `uv pip compile`) needs access to a
package index; point it at an internal one via `PIP_INDEX_URL` / `UV_INDEX_URL`
if needed. `git` must still be available in `PATH`.

For development, install with test and Airflow extras instead:

```bash
pip install -e ".[dev]"
```

This editable install is required if you want to run the CLI as
`migration-helper` from the repository checkout. Without it, use
`python -m migration_helper.cli ...` from the repository root.

## Usage

```bash
migration-helper COMMAND [TARGET] [OPTIONS]
```

Use the built-in help commands to see available options:

```bash
migration-helper --help
migration-helper analyze --help
migration-helper fix --help
```

### Commands

- `analyze`: scan `TARGET` for Airflow 3 compatibility issues in read-only mode.
- `fix`: scan `TARGET` and apply or dry-run available auto-fix paths.

### Shared Options

- `TARGET`: directory or file to scan. Defaults to the current directory.
- `--report-dir PATH`, `-r PATH`: write `migration_report.json` and/or
  `migration_report.html` into `PATH`.
- `--output-format json|html|all`: report format when `--report-dir` is set.
  Default: `all`.
- `--modified-only`: only check files reported by `git status --porcelain`
  under the target path. If git state cannot be resolved, all files are checked.
  With `fix --dry-run`, git selection runs on the original `TARGET` because the
  temporary copy has no `.git`.
- `--target-airflow-version VERSION`: Airflow version for dependency checks.
  Default: `3.2.1`.
- `--target-python-version VERSION`: Python major/minor version for dependency
  constraints. Defaults to the current interpreter version.
- `--dependency-file FILE`: dependency file to check at the target root.
  Supports requirements-style `.txt` files and `pyproject.toml`. Default:
  `requirements.txt`.
- `--use-uv`: use `uv` instead of `pip` for dependency resolver checks.
- `--db-migrate`: enable the DB migration stage. The stage runs the read-only
  `airflow db check-migrations` command and reports a warning (`DB001`) when
  the configured metadata database has pending migrations. A check that could
  not run - for example an unreachable database - is reported as a checker
  failure (`FAILED`), not as a `DB001` finding. By default, it uses
  the checker environment's own database; to check the Airflow 2 cluster
  metadata database, point the environment's connection string (for example
  `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`) at it. Applying migrations is not
  automated; run `airflow db migrate` yourself after reviewing the report.
- `--quiet`, `-q`: suppress progress and summary output. Exit codes are still
  set.

### Fix Options

- `--safe-only`: enable safe mechanical fixes where they are currently
  implemented. Today this mainly drives Ruff/config fix paths; the source-level
  DAG fixer is still a placeholder.
- `--unsafe`: also enable unsafe fixes that require human review. Implies
  safe fixes.
- `--dry-run`: do not modify `TARGET` files or the database. Ruff still applies
  `--safe-only` / `--unsafe` fixes, but on a temporary copy of `TARGET`;
  `dag_check` then runs against that already-fixed copy. The report lists
  leftover issues after those fixes (closer to a real `fix` than to `analyze`),
  with paths rewritten back to `TARGET`. The copy is deleted afterwards. Config
  update remains a non-writing Airflow CLI dry-run. The copy includes only
  `TARGET`. Imports that resolve to files outside that tree (a sibling
  `plugins/` package, a repo-root helper module) may fail in `dag_check`
  during dry-run even if a real `fix` against the original layout would
  succeed.
- `--fix-config`: request Airflow config updates explicitly. In the current
  implementation any fix mode (`--safe-only` / `--unsafe`) already enables the
  config update path, so the flag is effectively redundant. Config update
  still respects `--dry-run`.
- Where config fixes are applied: `airflow config update` runs against the
  checker environment's active config - `$AIRFLOW_CONFIG`, else
  `$AIRFLOW_HOME/airflow.cfg`, else `~/airflow/airflow.cfg`. `airflow.cfg`
  files under `TARGET` are checked but never rewritten; to migrate a config,
  point `AIRFLOW_CONFIG` at it (or run inside the Airflow environment itself).
  Airflow regenerates the file: options whose value equals the Airflow 3
  default are written commented out, and renamed keys carry a
  `# Renamed from <section>.<key>` note. Keys removed in Airflow 3 that
  Airflow's own updater does not drop (for example `webserver.rbac`,
  `core.store_serialized_dags`, `core.store_dag_code` on Airflow 3.2.1) are
  removed by migration-helper itself. Applied changes are reported as `CFG004`
  (`info`); `--dry-run` lists the would-be changes and removals as `CFG005`
  (`warning`), and findings that describe the environment config carry its
  full path in reports.
- Config-update resilience: after the update rewrites the checker environment's
  `airflow.cfg`, migration-helper re-reads it. If Airflow's writer produced an
  unparseable file (a known Airflow defect with multiline values such as
  `dag_bundle_config_list`), the file is repaired in place - semantically
  identical, only indentation restored - or, when that is impossible, the
  pre-update state is restored. The checker environment is also repaired at
  startup on every run, so a previously broken environment self-heals.

### Exit Codes

- `0`: no error-severity issues found and every checker ran. Warnings may still
  be present.
- `1`: one or more error-severity issues found.
- `2`: invalid arguments or the `TARGET` path does not exist.
- `3`: incomplete run - at least one checker could not execute, so the report
  cannot be trusted as a complete answer even when it contains no findings.
  `partial` and `skipped` stages do not trigger this code.

### Examples

```bash
# Read-only scan of all DAGs in ./dags/
migration-helper analyze ./dags/

# Scan and write HTML + JSON reports
migration-helper analyze ./dags/ --report-dir ./reports/

# Check dependencies against a specific Airflow/Python target
migration-helper analyze ./dags/ --target-airflow-version 3.2.1 --target-python-version 3.10

# Only report issues in files changed in the current git working tree
migration-helper analyze ./dags/ --modified-only

# Preview leftover issues after safe fixes without changing TARGET
migration-helper fix ./dags/ --safe-only --dry-run

# Apply currently available safe fixes in-place
migration-helper fix ./dags/ --safe-only

# Quiet mode for CI
migration-helper analyze ./dags/ --quiet
echo $?   # 0 = clean, 1 = errors
```

## License

Licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).
See the `LICENSE` file for the full license text.

## Common Commands

Run unit tests:

```bash
pytest tests/unit
```

Run integration tests:

```bash
pytest -m integration tests/integration
```

Run all regular tests, excluding integration and e2e by default:

```bash
pytest
```

Run e2e tests explicitly:

```bash
python -m pytest tests/e2e -m e2e
```

If you would like to test the tool with your own DAGs, place them in `test_dags`
or another local folder and run the tool based on the examples above.

## E2E Testing

The project keeps a small Airflow 2 DAG corpus under
`tests/e2e/airflow2-dag-corpus/` and separate runtime provider fixtures under
`tests/e2e/airflow2-provider-fixtures/`. The corpus is the `migration-helper`
target; provider fixtures are import support only.

Run all e2e checks explicitly:

```bash
python -m pytest tests/e2e -m e2e
```

Run the Airflow 2 baseline import smoke:

```bash
cd tests/e2e
docker compose -f docker-compose.airflow2.yml run --rm airflow2-smoke
```

For the current e2e layout, DAG corpus contents, dry-run artifacts under
`tests/e2e/runs/`, and the difference between `analyze` and `fix --dry-run`, see
`tests/e2e/README.md`.

## Docker

The project includes a multi-stage `Dockerfile` and `docker-compose.yml`.
Two images are built:

- **production**: runtime dependencies used for running `migration-helper`
  commands.
- **dev**: development/test dependencies used for running tests.

### Build

```bash
docker compose build
```

### Prepare DAG files

Place your DAG files in `test_dags/` (this directory is git-ignored):

```bash
mkdir -p test_dags
cp path/to/your_dag.py test_dags/
```

### Run analysis (read-only)

The `migration-helper-analyze` service mounts `./test_dags` at `/dags` as
read-only, which is appropriate for `analyze`:

```bash
docker compose run --rm migration-helper-analyze
```

Or pass any command and options directly:

```bash
docker compose run --rm migration-helper-analyze analyze /dags
docker compose run --rm migration-helper-analyze analyze /dags --modified-only
docker compose run --rm migration-helper-analyze analyze /dags/demo.py
```

### Apply fixes (writable)

The `migration-helper-fix` service mounts `./test_dags` at `/dags` as writable
so Ruff can rewrite `.py` files in place:

```bash
docker compose run --rm migration-helper-fix
```

Examples:

```bash
docker compose run --rm migration-helper-fix fix /dags --safe-only --dry-run
docker compose run --rm migration-helper-fix fix /dags/demo.py --safe-only
docker compose run --rm migration-helper-fix fix /dags/demo.py --unsafe
```

### Run tests

Unit tests:

```bash
docker compose run --rm test
```

Integration tests:

```bash
docker compose run --rm test -m integration tests/integration
```

E2E tests:

```bash
docker compose run --rm test tests/e2e -m e2e
```

E2E dry-run reports are written to:
`tests/e2e/runs/<UTC timestamp>/`
Open the latest run directory and check
`analyze/analyze-report/migration_report.html`,
`fix-dry-run/fix-dry-run-report/migration_report.html`,
`fix-safe/fix-safe-report/migration_report.html`, or
`fix-unsafe/fix-unsafe-report/migration_report.html`.

The main `docker-compose.yml` mounts `./tests` to `/app/tests`, so reports
created in the `test` container are available on the host immediately. No
`docker cp` is needed.
