How do I migrate from Airflow DAGs to Lakeflow Jobs without rewriting all my pipeline logic?

Summary

• Airflow DAG logic can be reused in Lakeflow Jobs by wrapping tasks as Python functions or notebooks that Lakeflow orchestrates.
• Lakeflow Jobs handles scheduling, retries, error handling, and monitoring, so you move the operational concerns from Airflow to Lakeflow.
• Both Airflow and Lakeflow Jobs support task dependencies, but Lakeflow's declarative syntax is simpler and requires no XML or complex Python for scheduling.
• You can run both systems in parallel during migration, gradually moving workloads to Lakeflow as you gain confidence.
• Migration typically involves refactoring Airflow DAGs into Lakeflow Job definitions or Python/Scala notebooks, not rewriting the core business logic.

How do I migrate from Airflow DAGs to Lakeflow Jobs without rewriting all my pipeline logic?
Migrating from Airflow DAGs to Lakeflow Jobs preserves most of your business logic while shifting the orchestration and operational layer to Databricks. You refactor Airflow tasks into Lakeflow Job steps (Python, SQL, or notebook tasks), and Lakeflow handles scheduling, retries, alerts, and lineage tracking. The core transformation logic in your DAGs stays the same; the change is in how tasks are defined and executed.

What is Airflow?

Apache Airflow is an open-source workflow orchestration platform. It uses Directed Acyclic Graphs (DAGs) to define pipelines, where each node is a task (Python function, shell command, or sensor) and edges represent dependencies. Airflow executes tasks on a schedule (cron, interval, or event-triggered) and provides retry logic, alerting, and a web UI for monitoring.

What are Lakeflow Jobs?

Lakeflow Jobs is Databricks' serverless orchestration service for data pipelines. Like Airflow, it orchestrates workflows defined as task graphs, but it is purpose-built for the Databricks lakehouse. Lakeflow Jobs integrate with Delta Lake, Unity Catalog, and Databricks compute, and they eliminate the need to manage a separate Airflow scheduler or workers.

Migration approach: how they compare

Aspect	Airflow DAGs	Lakeflow Jobs
Definition	Python code with @dag and @task decorators or DAG class	Declarative YAML or Python job config
Scheduling	Native cron, interval, or external triggers	Native cron, interval, or event-triggered via API
Task types	PythonOperator, BashOperator, KubernetesPodOperator, sensors, etc.	Python notebook, SQL notebook, Python inline, shell, or Databricks job cluster command
Compute	External workers, Kubernetes, or local executor	Databricks clusters (serverless SQL Warehouse, all-purpose, or job compute)
Retries	Built-in retry logic and backoff strategies	Built-in retry logic with exponential backoff and failure notifications
Monitoring	Web UI, logs, sensors	Databricks workspace Jobs UI, email alerts, webhook integration
State management	XCom for inter-task communication	Parameters and task results via Databricks API

The key insight: Airflow and Lakeflow Jobs solve the same problem (orchestrating dependent tasks) but with different execution models. Airflow is cluster-agnostic; Lakeflow is lakehouse-native.

How Databricks approaches pipeline migration

Refactor, don't rewrite. Each Airflow task becomes a Lakeflow Job task. If your Airflow task runs a Python script, that logic moves into a Python notebook or a notebook task in Lakeflow. If it runs SQL, you move the SQL into a SQL notebook or inline SQL task. The business logic stays the same.
Map task dependencies. Airflow's task_id and set_downstream/set_upstream become task dependencies in Lakeflow (defined in the job config or via the Databricks API). Task ordering logic translates directly.
Preserve retry and error handling. Both systems support retry policies. Lakeflow's retry configuration maps to Airflow's retry_count and retry_delay. If you have Airflow sensors (wait_for_condition), you can implement equivalent logic with Lakeflow's task dependencies or polling notebooks.
Use the dual-run period. Run Airflow and Lakeflow in parallel for a time. Trigger the same data pipeline in both systems, compare outputs, and gradually shift traffic to Lakeflow as you verify correctness. This reduces risk and lets you test incrementally.

Common migration patterns

Pattern 1: Lightweight Python or SQL logic. Airflow tasks that run simple Python or SQL can become single Lakeflow notebook tasks with minimal refactoring. The Python code moves into the notebook; the orchestration moves to Lakeflow.
Pattern 2: Multi-step pipelines with complex branching. If your Airflow DAG has branching (e.g., if-else logic), reimplement it using Lakeflow's task dependencies and conditional triggers, or use dynamic task generation via the Databricks API.
Pattern 3: External service calls. Airflow tasks that call external APIs (REST, webhooks) can become Python notebook tasks in Lakeflow. Import the same libraries and call the same endpoints; Lakeflow provides the scheduler.
Pattern 4: Incremental and CDC workloads. If your Airflow DAG includes CDC (change data capture) or incremental sync tasks, Lakeflow Jobs + Lakeflow Pipelines is a natural fit. You can run incremental transformations as Lakeflow Pipelines (continuous or triggered) and orchestrate them with Lakeflow Jobs.

Use cases

Consolidating orchestration and compute. Replace separate Airflow and data warehouse management with a unified Lakeflow + Databricks lakehouse stack. Reduces operational overhead.
Migrating off on-premises schedulers. If you are currently running Airflow on self-managed infrastructure (Kubernetes, on-prem servers), moving to Lakeflow is a cloud-native, serverless alternative that eliminates cluster management.
Unifying batch and real-time workloads. Lakeflow Jobs can orchestrate both batch pipelines (Lakeflow Pipelines) and real-time transformation notebooks, all in one job definition.

FAQs

Can I run Airflow and Lakeflow Jobs in parallel during migration?

Yes. Deploy the same business logic as a Lakeflow Job and run it on a staggered schedule or on-demand while keeping Airflow active. Compare outputs to validate correctness, then cut over to Lakeflow once you are confident.

Do I need to rewrite my Airflow Python operators?

Usually not. If your operator runs Python code (not Airflow-specific), that code moves into a Lakeflow notebook task. Airflow-specific logic (like custom sensors, operators, or plugins) may need refactoring or replacement with Lakeflow equivalents.

How do I handle Airflow XCom (cross-task communication)?

Lakeflow Jobs pass data between tasks using task parameters and the task results API. If your Airflow DAG uses XCom heavily, you will need to refactor task communication to use Lakeflow's parameter and output model.

What if my Airflow DAG uses external triggers or sensors?

Lakeflow Jobs support event-triggered execution via the API and webhooks. For Airflow sensors (e.g., waiting for a file or external condition), you can implement equivalent logic using Lakeflow's task dependencies or create a sensor task that polls a condition and reports completion.