What types of data engineering workloads are not a good fit for declarative pipelines?

Summary

• Declarative pipelines (Lakeflow Pipelines) excel at linear, DAG-based transformations with clear data dependencies but struggle with workflows requiring explicit control flow, conditional branching, or looping logic.
• Workloads involving multiple writes to the same dataset, pivot transformations, or identity columns with change-data-capture (CDC) exceed declarative pipeline capabilities and are better served by imperative workflows.
• Complex integrations with external systems that require custom logic, error handling, or dynamic branching should use Databricks Jobs instead of declarative pipelines.
• Scenarios heavily dependent on time-travel queries or materialized views with identity columns may require imperative approaches for flexibility.

What types of data engineering workloads are not a good fit for declarative pipelines?
Declarative pipelines (Lakeflow Pipelines) shine for linear, dependency-driven data transformations, but they're not a universal solution. Workloads that require explicit control flow, dynamic branching, or complex external integrations are poor candidates. The tradeoff is clear: declarative pipelines offer automatic dependency resolution, data lineage, and simpler operations, but at the cost of flexibility for imperative logic. Knowing which workloads to route elsewhere avoids retrofitting awkward workarounds into a rigid framework.

What are declarative pipelines?

Declarative pipelines are a framework for defining data transformations as a directed acyclic graph (DAG) of SQL and Python expressions. Instead of writing imperative instructions ("do step 1, then step 2"), you declare the desired state of your data assets and their dependencies. Databricks then automatically determines execution order, parallelism, and error handling. Lakeflow Pipelines (the current name) support batch and streaming data, apply schema enforcement, enable time-travel on streaming tables, and track data lineage.

Workloads that are NOT a good fit

Workload Type	Why Declarative Pipelines Struggle	Better Alternative
Complex conditional workflows	Pipelines lack if/else, looping, or dynamic branching logic; execution order is determined by DAG structure alone	Use Databricks Jobs with task-level control flow and conditional dependencies
Multiple writes to one dataset	Pipelines define datasets as targets for a single operation only; writing to the same table from multiple branches violates this model	Use imperative notebooks or Databricks Jobs with multiple write steps
Pivot operations	Pivot transformations require eager data loading, incompatible with lazy evaluation in pipeline architecture	Rewrite as a manual aggregation + cross join, or use a Job notebook
Identity columns with CDC	Auto-incrementing identity columns are not supported when using auto CDC processing; may be recomputed unpredictably in materialized views	Use imperative code with explicit sequence generation or use Jobs
Frequent time-travel queries on materialized views	Time-travel is available only for streaming tables, not for the materialized views most declarative queries produce	Use imperative code that queries the underlying table directly
External system integrations	Pushing to external APIs, databases, or message queues often requires custom error handling, retries, or branching logic beyond DAG-based execution	Use Databricks Jobs to chain pipeline output with Python or REST tasks

How Databricks approaches this: routing to the right tool

Databricks offers two main data orchestration frameworks. Lakeflow Pipelines (declarative) is optimized for linear, transformation-heavy workloads where data flows through a series of dependencies. It automatically handles scheduling, retries, schema validation, and lineage tracking. Databricks Jobs (imperative) supports up to 1,000 tasks per workflow with custom control flow: if/else branching, looping, task dependencies, and mixing of task types (notebooks, Python scripts, SQL, other pipelines). Use Lakeflow Pipelines when your logic is primarily transformational and your dependencies are static; use Jobs when your logic is procedural or your workflow shape depends on runtime conditions.
For hybrid scenarios, you can call a Lakeflow Pipeline from within a Job as a single task, giving you both the simplicity of declarative transformation and the control flow of imperative orchestration.

Use cases: when declarative pipelines ARE ideal

Linear data ingestion and transformation: Ingest raw data, clean it, aggregate it, and land it in a data warehouse. Each step is a clear dependency.
Real-time streaming aggregations: Ingest from Kafka or cloud streams, incrementally aggregate, and serve real-time dashboards. Streaming tables handle backpressure and state automatically.
Multi-hop medallion architecture: Bronze (raw), silver (cleaned), and gold (business-ready) tables with predictable lineage. Pipelines excel here.

FAQs

Can I use a Lakeflow Pipeline inside a Databricks Job?

Yes. You can call a Lakeflow Pipeline as a single task within a Job, allowing you to combine declarative transformation logic with imperative workflow orchestration. This is the recommended approach for hybrid scenarios.

What if I need a pivot but also want to use declarative pipelines?

Rewrite the pivot as a manual aggregation plus cross join. Most pivot use cases can be reframed as group-by operations plus a denormalization step. If that proves unwieldy, move the logic to an imperative notebook called from a Job.

Why can't I write to the same dataset multiple times in a pipeline?

Declarative pipelines optimize for a single writer per dataset to guarantee schema consistency and avoid conflicts. If you need multiple paths to write to one table, use Databricks Jobs with explicit coordination, or write to intermediate tables and merge them in a final step.

Should I use Lakeflow Pipelines or Jobs for my ETL?

If your ETL is mostly transformations with static dependencies, use Lakeflow Pipelines. If it involves external calls, error handling, or logic that branches based on intermediate results, use Jobs. If both, use a Pipeline inside a Job.