Metadata-Version: 2.5
Name: migration-helper
Version: 1.0.1
Summary: Utility for migrating Airflow code from version 2 to version 3.
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.10
Requires-Dist: packaging
Requires-Dist: ruff<0.16,>=0.15.5
Requires-Dist: tomli>=1.1.0; python_version < '3.11'
Provides-Extra: airflow
Requires-Dist: apache-airflow==3.2.1; extra == 'airflow'
Provides-Extra: dev
Requires-Dist: apache-airflow==3.2.1; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# Migration Helper

`migration-helper` is a Python project for helping migrate Apache Airflow code
from Airflow 2 to Airflow 3.

The current implementation is focused on static analysis and migration
assistance for DAG code. It includes checks based on Ruff AIR rules, Airflow
configuration linting, DAG import checks, dependency compatibility checks, base
reporting, shared subprocess helpers, file selection helpers, and tests.

## Expected Output

After running `migration-helper`, you get a practical migration report for the
selected Airflow project. The report lists compatibility issues by file and rule,
shows error/warning severity, and explains whether each finding can be handled
automatically or needs manual review.

Reports can be written in JSON and HTML formats. The JSON report is intended
for automation and CI, while the HTML report is intended for manual review.
Both formats expose the same issue details.

The JSON report has this top-level structure:

- `generated_at`: UTC timestamp when the report was created.
- `target`: analyzed file or directory.
- `summary`: total number of findings, split into `errors` and `warnings`.
- `by_file`: findings grouped by affected file path.
- `by_type`: findings grouped by `rule_id`.

Each finding contains:

- `file`: affected file or logical target, for example `airflow.cfg` or
  `requirements.txt`.
- `line` and `col`: source location when the checker can provide it; otherwise
  `null`.
- `rule_id`: checker-specific rule or error identifier, for example `AIR301`,
  `CFG001`, or `DEP_RESOLUTION_FAILED`.
- `message`: short explanation of the problem.
- `severity`: `error`, `warning`, or `info`.
- `source`: checker that produced the finding, for example `ruff`,
  `airflow_config`, `dag_check`, `dep_check`, or `db_migrate`.

Reports use these fields to describe fixability:

- `fix_safety: "safe"` means the checker knows a safe mechanical fix or safe
  migration action. Safe Ruff fixes can be applied with
  `migration-helper fix TARGET --safe-only`; Airflow config fixes require the
  config fix path and are still blocked by `--dry-run`.
- `fix_safety: "unsafe"` means the tool knows the likely change, but a human
  should review it before applying.
- `fix_safety: null` (or `fix_safety: "manual"` in HTML report) means there is
  no automatic fix path in the current implementation. Treat it as manual work.
- `fix_description` is the human-readable hint for what to change or which
  command/path is relevant.

Today, DAG import errors, dependency conflicts, and DB migration findings are
diagnostic/manual. Source-level DAG fixes outside Ruff are still planned.

## Requirements and setup

- Python `>=3.10`
- `git` available in `PATH`

Install from PyPI:

```bash
pip install migration-helper
```

### Offline installation

To install on a machine without access to PyPI, build a wheelhouse on a machine
with internet access and install from it:

```bash
# On a machine with internet access
pip download migration-helper==1.0.1 -d wheelhouse/

# On the offline machine
pip install --no-index --find-links ./wheelhouse migration-helper
```

Note: `ruff` is a platform-specific binary package. Download the wheelhouse on
a machine with the same OS and the same Python version as the target machine.

If the wheelhouse must be prepared for a different platform than the download
machine, pass the target platform tags explicitly. This works only with
pre-built wheels, so `--only-binary=:all:` is required:

```bash
# Build a wheelhouse for manylinux x86_64 + CPython 3.11 from any machine
pip download migration-helper==1.0.1 -d wheelhouse/ \
  --only-binary=:all: \
  --platform manylinux2014_x86_64 \
  --python-version 311 \
  --implementation cp \
  --abi cp311
```

`ruff` ships platform-specific wheels (`py3-none-manylinux2014_x86_64`,
`macosx_11_0_arm64`, `win_amd64`, ...) so the `--platform` tag matters most;
`packaging` and `tomli` are pure Python and work everywhere. Repeat the command
per target platform (for example `macosx_11_0_arm64` or `win_amd64`) or build
the wheelhouse on a machine matching the target configuration.

Alternatives: copy a prepared virtual environment as a whole (same OS /
architecture / Python required), or serve the packages from an internal PyPI
mirror (devpi, Nexus, Artifactory) and use `pip config set global.index-url
http://<mirror>/`.

Offline usage notes: Airflow 3.2.1 constraints for Python 3.10-3.14 are bundled
with the package, so the default dependency check works offline. Other Airflow
versions require network access to download constraints, and the deep
dependency check (`pip install --dry-run` / `uv pip compile`) needs access to a
package index; point it at an internal one via `PIP_INDEX_URL` / `UV_INDEX_URL`
if needed. `git` must still be available in `PATH`.

For development, install with test and Airflow extras instead:

```bash
pip install -e ".[dev]"
```

This editable install is required if you want to run the CLI as
`migration-helper` from the repository checkout. Without it, use
`python -m migration_helper.cli ...` from the repository root.

## Usage

```bash
migration-helper COMMAND [TARGET] [OPTIONS]
```

Use the built-in help commands to see available options:

```bash
migration-helper --help
migration-helper analyze --help
migration-helper fix --help
```

### Commands

- `analyze`: scan `TARGET` for Airflow 3 compatibility issues in read-only mode.
- `fix`: scan `TARGET` and apply or dry-run available auto-fix paths.

### Shared Options

- `TARGET`: directory or file to scan. Defaults to the current directory.
- `--report-dir PATH`, `-r PATH`: write `migration_report.json` and/or
  `migration_report.html` into `PATH`.
- `--output-format json|html|all`: report format when `--report-dir` is set.
  Default: `all`.
- `--modified-only`: only check files reported by `git status --porcelain`
  under the target path. If git state cannot be resolved, all files are checked.
- `--target-airflow-version VERSION`: Airflow version for dependency checks.
  Default: `3.2.1`.
- `--target-python-version VERSION`: Python major/minor version for dependency
  constraints. Defaults to the current interpreter version.
- `--dependency-file FILE`: dependency file to check at the target root.
  Supports requirements-style `.txt` files and `pyproject.toml`. Default:
  `requirements.txt`.
- `--use-uv`: use `uv` instead of `pip` for dependency resolver checks.
- `--db-migrate`: enable the DB migration stage. The current implementation is
  still a placeholder, so this flag is part of the CLI contract but does not yet
  produce a real DB migration report.
- `--quiet`, `-q`: suppress progress and summary output. Exit codes are still
  set.

### Fix Options

- `--safe-only`: enable safe mechanical fixes where they are currently
  implemented. Today this mainly drives Ruff/config fix paths; the source-level
  DAG fixer is still a placeholder.
- `--unsafe`: also enable unsafe fixes that require human review. Implies
  safe fixes.
- `--dry-run`: guarantee that files and the database are not modified. In the
  current implementation, dry-run disables Ruff/config fix application before the
  pipeline runs, so it is mostly a scan/report safety check rather than a full
  diff preview.
- `--fix-config`: request Airflow config updates. Config update still respects
  `--dry-run`.

### Exit Codes

- `0`: no error-severity issues found. Warnings may still be present.
- `1`: one or more error-severity issues found.
- `2`: invalid arguments or the `TARGET` path does not exist.

### Examples

```bash
# Read-only scan of all DAGs in ./dags/
migration-helper analyze ./dags/

# Scan and write HTML + JSON reports
migration-helper analyze ./dags/ --report-dir ./reports/

# Check dependencies against a specific Airflow/Python target
migration-helper analyze ./dags/ --target-airflow-version 3.2.1 --target-python-version 3.10

# Only report issues in files changed in the current git working tree
migration-helper analyze ./dags/ --modified-only

# Dry-run the current fix command path without writing files
migration-helper fix ./dags/ --safe-only --dry-run

# Apply currently available safe fixes in-place
migration-helper fix ./dags/ --safe-only

# Quiet mode for CI
migration-helper analyze ./dags/ --quiet
echo $?   # 0 = clean, 1 = errors
```

## License

Licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).
See the `LICENSE` file for the full license text.

## Common Commands

Run unit tests:

```bash
pytest tests/unit
```

Run integration tests:

```bash
pytest -m integration tests/integration
```

Run all regular tests, excluding integration and e2e by default:

```bash
pytest
```

Run e2e tests explicitly:

```bash
python -m pytest tests/e2e -m e2e
```

If you would like to test the tool with your own DAGs, place them in `test_dags`
or another local folder and run the tool based on the examples above.

## E2E Testing

The project keeps a small Airflow 2 DAG corpus under
`tests/e2e/airflow2-dag-corpus/` and separate runtime provider fixtures under
`tests/e2e/airflow2-provider-fixtures/`. The corpus is the `migration-helper`
target; provider fixtures are import support only.

Run all e2e checks explicitly:

```bash
python -m pytest tests/e2e -m e2e
```

Run the Airflow 2 baseline import smoke:

```bash
cd tests/e2e
docker compose -f docker-compose.airflow2.yml run --rm airflow2-smoke
```

For the current e2e layout, DAG corpus contents, dry-run artifacts under
`tests/e2e/runs/`, and the difference between `analyze` and `fix --dry-run`, see
`tests/e2e/README.md`.

## Docker

The project includes a multi-stage `Dockerfile` and `docker-compose.yml`.
Two images are built:

- **production**: runtime dependencies used for running `migration-helper`
  commands.
- **dev**: development/test dependencies used for running tests.

### Build

```bash
docker compose build
```

### Prepare DAG files

Place your DAG files in `test_dags/` (this directory is git-ignored):

```bash
mkdir -p test_dags
cp path/to/your_dag.py test_dags/
```

### Run analysis (read-only)

The `migration-helper-analyze` service mounts `./test_dags` at `/dags` as
read-only, which is appropriate for `analyze`:

```bash
docker compose run --rm migration-helper-analyze
```

Or pass any command and options directly:

```bash
docker compose run --rm migration-helper-analyze analyze /dags
docker compose run --rm migration-helper-analyze analyze /dags --modified-only
docker compose run --rm migration-helper-analyze analyze /dags/demo.py
```

### Apply fixes (writable)

The `migration-helper-fix` service mounts `./test_dags` at `/dags` as writable
so Ruff can rewrite `.py` files in place:

```bash
docker compose run --rm migration-helper-fix
```

Examples:

```bash
docker compose run --rm migration-helper-fix fix /dags --safe-only --dry-run
docker compose run --rm migration-helper-fix fix /dags/demo.py --safe-only
docker compose run --rm migration-helper-fix fix /dags/demo.py --unsafe
```

### Run tests

Unit tests:

```bash
docker compose run --rm test
```

Integration tests:

```bash
docker compose run --rm test -m integration tests/integration
```

E2E tests:

```bash
docker compose run --rm test tests/e2e -m e2e
```

E2E dry-run reports are written to:
`tests/e2e/runs/<UTC timestamp>/`
Open the latest run directory and check
`analyze/analyze-report/migration_report.html`,
`fix-dry-run/fix-dry-run-report/migration_report.html`,
`fix-safe/fix-safe-report/migration_report.html`, or
`fix-unsafe/fix-unsafe-report/migration_report.html`.

The main `docker-compose.yml` mounts `./tests` to `/app/tests`, so reports
created in the `test` container are available on the host immediately. No
`docker cp` is needed.
