/migrating-dagster-to-airflow

name: migrating-dagster-to-airflow
description: Guide for migrating Dagster projects to Apache Airflow 3 on Astro. Use when the user mentions migrating, converting, or porting Dagster (or Dagster+) code to Airflow or Astro, wants to plan or assess such a migration, or asks what a Dagster construct maps to in Airflow. Covers assets, partitions, schedules, sensors, declarative automation, resources, IO managers, ops/jobs, dbt, Pipes, Components, and Dagster+ platform config. Always load this skill as the first step for any Dagster-to-Airflow request.
hooks:
  PostToolUse:
    - matcher: "Edit"
      hooks:
        - type: command
          command: "echo 'Migrated DAG edited: consider re-running gates 1-3: python3 scripts/validate_dag.py <astro_project> --manifest manifest.json'"

Dagster → Airflow 3 (Astro) migration

Migrate a Dagster project to Airflow 3 on Astro Runtime, honestly. The migration is asset-first (Dagster asset graphs translate to Airflow assets and asset-aware schedules, not flattened DAGs), incremental (domain by domain, Dagster stays authoritative until parity), and honest (every definition gets an explicit disposition; semantic deltas are documented, never papered over).

First time driving this? Read `reference/quickstart.md` first: hour-one commands, the glossary, and what can and cannot break.

Migration at a glance

1. Baseline the source project's tests, then inventory it read-only (`scripts/inventory.py` → manifest). 2. Review classifications (MECH/JUDG/REDESIGN/NONE per `reference/mapping.md`); make the go/no-go call (three outcomes; migrate-with-conditions is the common case, stay is the narrow one); plan DAG boundaries, per-edge IO decisions, and Gate 3 expectations into the manifest. 3. Trial-migrate 2-3 representative units end-to-end through every validation gate. 4. Migrate domain by domain through the six-gate ladder (`reference/validation.md`), tracking per-unit state (`scripts/status.py`); fix failure classes via `reference/troubleshooting.md`, never stub. 5. Map the platform layer (secrets, alerts, CI/CD, Deployments) per `reference/astro-deployment.md`. 6. Run side by side, then cut over per domain (consumers unpause first; see the checklist), keeping rollback one step away. 7. Deliver the migration report: every definition dispositioned, an equivalence row per trigger, losses stated plainly.

Version drift

Verified against Airflow 3.3.0 / Astro Runtime 3.3-2 / astronomer-cosmos 1.15 / Dagster 1.13 (2026-07). Version-sensitive rows in the references carry their floor (notably the 3.2-vs-3.3 partition surface). Before relying on a version-gated claim: check the target (`airflow version`, `astro deployment inspect`), probe imports for sdk surface (`python3 -c "from airflow.sdk import X"`), and prefer `--help` / API spec discovery over assuming verbatim CLI/REST contracts on newer versions. Playbook entries are version-scoped per entry.

Requirements

• Target **Astro Runtime 3.3+** (Airflow 3.3+); the native asset-partition surface requires it. Below 3.2 the mapping degrades badly; say so and recommend upgrading before migrating.
• The Dagster repo, and ideally a running Dagster instance (its materialization metadata provides parity-test fixtures).
• `astro` CLI for the target project.

Hard rules

1. **Never stub.** A translated unit either works through its validation gate or is deferred with a written reason. Fake-success bodies and workaround code with long justifying comments are failures. 2. **No silent omissions.** Every record in the inventory manifest ends `complete` or `deferred (reason)`. `scripts/status.py summary` exits nonzero otherwise; run it before claiming done. 3. **Equivalence rows for every trigger.** Each schedule/sensor/automation condition gets a report row: source spelling, target spelling, delta in one sentence. Semantic deltas exist (catchup, on_cron inversion, eager guarantees); the sin is not the delta, it is the undocumented delta. 4. **Fix classes, not instances.** When a translation pattern fails validation, fix the pattern (and record it in `reference/troubleshooting.md`), then re-apply; do not hand-patch one unit. 5. **Do not invent APIs.** The references contain verified names only. Anything not covered there gets verified against official docs before use.

Workflow

Phase 0: Preflight

Confirm target Runtime version, `astro` CLI presence, and repo access. Detect the project layout: classic (`@repository`/`workspace.yaml`), modern (`Definitions`), or Components (`pyproject.toml [tool.dg]`, `defs.yaml` files); all three occur, sometimes together. Baseline the source project's test suite now: pre-existing failures are recorded and excluded from migration blame.

Phase 1: Inventory (read-only)

python3 scripts/inventory.py <dagster_repo> --out manifest.json          # static scan
python3 scripts/inventory.py <dagster_repo> --runtime --out manifest.json # + runtime introspection when the project imports

The manifest lists every definition with file:line, captured params, current-vs-deprecated spelling, and dependency edges with their IO manager; every record starts `classification: "pending"`. Classifying is YOUR first judgment task: assign each record MECH / JUDG / REDESIGN / NONE from its row in `reference/mapping.md` and write it into the manifest. The scanner enumerates (deterministic completeness); the agent classifies (judgment). A record you cannot map to a mapping.md row is itself a finding: record it, do not guess. Also grep for `DAGSTER_CLOUD_` and `EnvVar(` (platform layer, Phase 5).

Manifest conventions: the canonical manifest lives in the migration run directory. Once the Astro project exists (Phase 2 scaffold), copy the manifest to its `include/inventory/manifest.json` so the Gate 3 pytest and `status.py` defaults find it; until then it just stays in the run dir (keep the two in sync afterward, the run-dir copy wins). Static records are the canonical migration units