Metadata-Version: 2.5
Name: airflow-migration-helper
Version: 1.0.3
Summary: Utility for migrating Airflow code from version 2 to version 3.
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.10
Requires-Dist: packaging
Requires-Dist: ruff<0.16,>=0.15.5
Requires-Dist: tomli>=1.1.0; python_version < '3.11'
Provides-Extra: airflow
Requires-Dist: apache-airflow==3.2.1; extra == 'airflow'
Provides-Extra: dev
Requires-Dist: apache-airflow==3.2.1; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# Migration Helper

`airflow-migration-helper` is a Python project for helping migrate Apache
Airflow code from Airflow 2 to Airflow 3. The PyPI distribution installs the
command-line tool as `migration-helper`.

The current implementation is focused on static analysis and migration
assistance for DAG code. It includes checks based on Ruff AIR rules, Airflow
configuration linting, DAG import checks, dependency compatibility checks, base
reporting, shared subprocess helpers, file selection helpers, and tests.

## How it works

`migration-helper` migrates code from Airflow 2 to Airflow 3, so it is run
against a target project that is currently on Airflow 2:

- The analyzed project (DAGs, configs, dependencies) stays on Airflow 2.
- The tool runs in its own dedicated virtual environment (see
  [Requirements and setup](#requirements-and-setup)).

Which Airflow version the checker environment needs depends on the checker:

- `ruff` and `dep_check` analyze the target statically and do not require
  Airflow at all.
- `airflow_config` checks config keys against a built-in static rule table, so
  it also works without Airflow. When Airflow is installed in the checker
  environment, the checker additionally runs `airflow config lint` and reports
  more findings. Note that the Airflow CLI commands used by this checker
  operate on the active configuration of the checker environment
  (`AIRFLOW_HOME` / `AIRFLOW_CONFIG`), not on `airflow.cfg` files found in the
  target; only the static scan covers target files. To lint or update a
  specific config file through the CLI path, point `AIRFLOW_CONFIG` at it.
- `dag_check` runs `airflow dags list-import-errors` from the checker
  environment, so it parses the Airflow 2 DAGs **under the Airflow installed in
  that environment** (Airflow 3 for full compatibility checks).
- `db_migrate` runs the one-shot, read-only `airflow db check-migrations`
  command from the checker environment. No running Airflow processes are
  needed, but a reachable metadata database must be configured for that
  environment. The stage is only enabled with `--db-migrate`.

For full functionality (all checkers, including `dag_check`), install the tool
together with Airflow 3 in the same virtual environment:

```bash
python -m venv .venv-mh
.venv-mh/bin/pip install "airflow-migration-helper[airflow]"
```

## Expected Output

After running `migration-helper`, you get a practical migration report for the
selected Airflow project. The report lists compatibility issues by file and rule,
shows error/warning severity, and explains whether each finding can be handled
automatically or needs manual review.

Reports can be written in JSON and HTML formats. The JSON report is intended
for automation and CI, while the HTML report is intended for manual review.
Both formats expose the same issue details.

The JSON report has this top-level structure:

- `generated_at`: UTC timestamp when the report was created.
- `target`: analyzed file or directory.
- `summary`: total number of findings, split into `errors` and `warnings`.
- `by_file`: findings grouped by affected file path.
- `by_type`: findings grouped by `rule_id`.

Each finding contains:

- `file`: affected file or logical target, for example `airflow.cfg` or
  `requirements.txt`.
- `line` and `col`: source location when the checker can provide it; otherwise
  `null`.
- `rule_id`: checker-specific rule or error identifier, for example `AIR301`,
  `CFG001`, or `DEP_RESOLUTION_FAILED`.
- `message`: short explanation of the problem.
- `severity`: `error`, `warning`, or `info`.
- `source`: checker that produced the finding, for example `ruff`,
  `airflow_config`, `dag_check`, `dep_check`, or `db_migrate`.

Reports use these fields to describe fixability:

- `fix_safety: "safe"` means the checker knows a safe mechanical fix or safe
  migration action. `migration-helper fix TARGET --safe-only` applies safe Ruff
  fixes and, when Airflow is available in the checker environment, also runs
  `airflow config update --fix` on that environment's active config (see
  [How it works](#how-it-works)). Both paths are blocked by `--dry-run`.
- `fix_safety: "unsafe"` means the tool knows the likely change, but a human
  should review it before applying.
- `fix_safety: null` (or `fix_safety: "manual"` in HTML report) means there is
  no automatic fix path in the current implementation. Treat it as manual work.
- `fix_description` is the human-readable hint for what to change or which
  command/path is relevant.

Today, DAG import errors, dependency conflicts, and DB migration findings are
diagnostic/manual. Source-level DAG fixes outside Ruff are still planned.

## Requirements and setup

- Python `>=3.10`
- `git` available in `PATH`
- Work in a dedicated virtual environment (see below)

Install `airflow-migration-helper` and its dependencies (`ruff` and others)
into a dedicated virtual environment, not into the system Python. This keeps
the tool's dependencies isolated from the analyzed project.

```bash
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
```

Install from PyPI:

```bash
pip install airflow-migration-helper[airflow]
```

### Offline installation

To install on a machine without access to PyPI, build a wheelhouse on a machine
with internet access and install from it:

```bash
# On a machine with internet access
pip download airflow-migration-helper[airflow]==1.0.3 -d wheelhouse/

# On the offline machine
pip install --no-index --find-links ./wheelhouse airflow-migration-helper
```

Note: `ruff` is a platform-specific binary package. Download the wheelhouse on
a machine with the same OS and the same Python version as the target machine.

If the wheelhouse must be prepared for a different platform than the download
machine, pass the target platform tags explicitly. This works only with
pre-built wheels, so `--only-binary=:all:` is required:

```bash
# Build a wheelhouse for manylinux x86_64 + CPython 3.11 from any machine
pip download airflow-migration-helper[airflow]==1.0.3 -d wheelhouse/ \
  --only-binary=:all: \
  --platform manylinux2014_x86_64 \
  --python-version 311 \
  --implementation cp \
  --abi cp311
```

`ruff` ships platform-specific wheels (`py3-none-manylinux2014_x86_64`,
`macosx_11_0_arm64`, `win_amd64`, ...) so the `--platform` tag matters most;
`packaging` and `tomli` are pure Python and work everywhere. Repeat the command
per target platform (for example `macosx_11_0_arm64` or `win_amd64`) or build
the wheelhouse on a machine matching the target configuration.

Alternatives: copy a prepared virtual environment as a whole (same OS /
architecture / Python required), or serve the packages from an internal PyPI
mirror (devpi, Nexus, Artifactory) and use `pip config set global.index-url
http://<mirror>/`.

Offline usage notes: Airflow 3.2.1 constraints for Python 3.10-3.14 are bundled
with the package, so the default dependency check works offline. Other Airflow
versions require network access to download constraints, and the deep
dependency check (`pip install --dry-run` / `uv pip compile`) needs access to a
package index; point it at an internal one via `PIP_INDEX_URL` / `UV_INDEX_URL`
if needed. `git` must still be available in `PATH`.

For development, install with test and Airflow extras instead:

```bash
pip install -e ".[dev]"
```

This editable install is required if you want to run the CLI as
`migration-helper` from the repository checkout. Without it, use
`python -m migration_helper.cli ...` from the repository root.

## Usage

```bash
migration-helper COMMAND [TARGET] [OPTIONS]
```

Use the built-in help commands to see available options:

```bash
migration-helper --help
migration-helper analyze --help
migration-helper fix --help
```

### Commands

- `analyze`: scan `TARGET` for Airflow 3 compatibility issues in read-only mode.
- `fix`: scan `TARGET` and apply or dry-run available auto-fix paths.

### Shared Options

- `TARGET`: directory or file to scan. Defaults to the current directory.
- `--report-dir PATH`, `-r PATH`: write `migration_report.json` and/or
  `migration_report.html` into `PATH`.
- `--output-format json|html|all`: report format when `--report-dir` is set.
  Default: `all`.
- `--modified-only`: only check files reported by `git status --porcelain`
  under the target path. If git state cannot be resolved, all files are checked.
- `--target-airflow-version VERSION`: Airflow version for dependency checks.
  Default: `3.2.1`.
- `--target-python-version VERSION`: Python major/minor version for dependency
  constraints. Defaults to the current interpreter version.
- `--dependency-file FILE`: dependency file to check at the target root.
  Supports requirements-style `.txt` files and `pyproject.toml`. Default:
  `requirements.txt`.
- `--use-uv`: use `uv` instead of `pip` for dependency resolver checks.
- `--db-migrate`: enable the DB migration stage. The stage runs the read-only
  `airflow db check-migrations` command and reports a warning (`DB001`) when
  the configured metadata database has pending migrations. By default it uses
  the checker environment's own database; to check the Airflow 2 cluster
  metadata database, point the environment's connection string (for example
  `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`) at it. Applying migrations is not
  automated; run `airflow db migrate` yourself after reviewing the report.
- `--quiet`, `-q`: suppress progress and summary output. Exit codes are still
  set.

### Fix Options

- `--safe-only`: enable safe mechanical fixes where they are currently
  implemented. Today this mainly drives Ruff/config fix paths; the source-level
  DAG fixer is still a placeholder.
- `--unsafe`: also enable unsafe fixes that require human review. Implies
  safe fixes.
- `--dry-run`: guarantee that files and the database are not modified. In the
  current implementation, dry-run disables Ruff/config fix application before the
  pipeline runs, so it is mostly a scan/report safety check rather than a full
  diff preview.
- `--fix-config`: request Airflow config updates explicitly. In the current
  implementation any fix mode (`--safe-only` / `--unsafe`) already enables the
  config update path, so the flag is effectively redundant. Config update
  still respects `--dry-run`.

### Exit Codes

- `0`: no error-severity issues found. Warnings may still be present.
- `1`: one or more error-severity issues found.
- `2`: invalid arguments or the `TARGET` path does not exist.

### Examples

```bash
# Read-only scan of all DAGs in ./dags/
migration-helper analyze ./dags/

# Scan and write HTML + JSON reports
migration-helper analyze ./dags/ --report-dir ./reports/

# Check dependencies against a specific Airflow/Python target
migration-helper analyze ./dags/ --target-airflow-version 3.2.1 --target-python-version 3.10

# Only report issues in files changed in the current git working tree
migration-helper analyze ./dags/ --modified-only

# Dry-run the current fix command path without writing files
migration-helper fix ./dags/ --safe-only --dry-run

# Apply currently available safe fixes in-place
migration-helper fix ./dags/ --safe-only

# Quiet mode for CI
migration-helper analyze ./dags/ --quiet
echo $?   # 0 = clean, 1 = errors
```

## License

Licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).
See the `LICENSE` file for the full license text.

## Common Commands

Run unit tests:

```bash
pytest tests/unit
```

Run integration tests:

```bash
pytest -m integration tests/integration
```

Run all regular tests, excluding integration and e2e by default:

```bash
pytest
```

Run e2e tests explicitly:

```bash
python -m pytest tests/e2e -m e2e
```

If you would like to test the tool with your own DAGs, place them in `test_dags`
or another local folder and run the tool based on the examples above.

## E2E Testing

The project keeps a small Airflow 2 DAG corpus under
`tests/e2e/airflow2-dag-corpus/` and separate runtime provider fixtures under
`tests/e2e/airflow2-provider-fixtures/`. The corpus is the `migration-helper`
target; provider fixtures are import support only.

Run all e2e checks explicitly:

```bash
python -m pytest tests/e2e -m e2e
```

Run the Airflow 2 baseline import smoke:

```bash
cd tests/e2e
docker compose -f docker-compose.airflow2.yml run --rm airflow2-smoke
```

For the current e2e layout, DAG corpus contents, dry-run artifacts under
`tests/e2e/runs/`, and the difference between `analyze` and `fix --dry-run`, see
`tests/e2e/README.md`.

## Docker

The project includes a multi-stage `Dockerfile` and `docker-compose.yml`.
Two images are built:

- **production**: runtime dependencies used for running `migration-helper`
  commands.
- **dev**: development/test dependencies used for running tests.

### Build

```bash
docker compose build
```

### Prepare DAG files

Place your DAG files in `test_dags/` (this directory is git-ignored):

```bash
mkdir -p test_dags
cp path/to/your_dag.py test_dags/
```

### Run analysis (read-only)

The `migration-helper-analyze` service mounts `./test_dags` at `/dags` as
read-only, which is appropriate for `analyze`:

```bash
docker compose run --rm migration-helper-analyze
```

Or pass any command and options directly:

```bash
docker compose run --rm migration-helper-analyze analyze /dags
docker compose run --rm migration-helper-analyze analyze /dags --modified-only
docker compose run --rm migration-helper-analyze analyze /dags/demo.py
```

### Apply fixes (writable)

The `migration-helper-fix` service mounts `./test_dags` at `/dags` as writable
so Ruff can rewrite `.py` files in place:

```bash
docker compose run --rm migration-helper-fix
```

Examples:

```bash
docker compose run --rm migration-helper-fix fix /dags --safe-only --dry-run
docker compose run --rm migration-helper-fix fix /dags/demo.py --safe-only
docker compose run --rm migration-helper-fix fix /dags/demo.py --unsafe
```

### Run tests

Unit tests:

```bash
docker compose run --rm test
```

Integration tests:

```bash
docker compose run --rm test -m integration tests/integration
```

E2E tests:

```bash
docker compose run --rm test tests/e2e -m e2e
```

E2E dry-run reports are written to:
`tests/e2e/runs/<UTC timestamp>/`
Open the latest run directory and check
`analyze/analyze-report/migration_report.html`,
`fix-dry-run/fix-dry-run-report/migration_report.html`,
`fix-safe/fix-safe-report/migration_report.html`, or
`fix-unsafe/fix-unsafe-report/migration_report.html`.

The main `docker-compose.yml` mounts `./tests` to `/app/tests`, so reports
created in the `test` container are available on the host immediately. No
`docker cp` is needed.
