Can declarative pipelines support both batch and streaming workloads?

Summary

• Yes. Lakeflow Pipelines run batch and streaming on one engine built on Apache Spark and Structured Streaming, so a single declarative definition covers both with no separate systems and no rewrites.
• The dataset type sets the semantics. Streaming tables process incremental, append-only data and handle each row once; materialized views recompute aggregations and joins when their inputs change.
• The execution mode sets the timing. Triggered pipelines refresh available data and stop; continuous pipelines keep running to process new data with minimal delay. The code is the same either way.
• The framework manages the hard parts: incremental processing, automatic checkpoints, exactly-once semantics, and schema changes, so you declare what you want and Lakeflow plans how to run it.
• Both modes write to the same Delta Lake tables under Unity Catalog, so a pipeline built for nightly batch can serve near-real-time needs by changing configuration, not logic.

Can declarative pipelines support both batch and streaming workloads?

Yes. Declarative pipelines handle batch and streaming from the same code because they run on one engine that treats both as variations of a single processing model. On Databricks that engine is Lakeflow Pipelines, built on the open Apache Spark Declarative Pipelines standard and on Structured Streaming. You declare the datasets you want, and the framework decides how to compute them incrementally, whether the data arrives in a nightly file drop or a continuous stream. There's no second system to run and no separate streaming codebase to maintain. See Lakeflow: a new era of agentic data engineering.

How batch and streaming share one model

Apache Spark and Structured Streaming share the same underlying architecture, so the engine can read cloud object storage and Delta Lake tables as either bounded batch sources or streaming sources. That is what lets one declarative definition serve both modes. You describe the transformation once, and the framework processes only new or changed data, tracks progress with automatic checkpoints, and handles each record exactly once. See the batch and streaming overview.

Choosing dataset types and execution modes

Two choices decide how a pipeline behaves. The dataset type sets the processing semantics, and the execution mode sets the timing. Both use the same declarative code.

Choice	What it does	When to use
Streaming table	Incremental, append-only processing that reads streams, files, or change feeds and handles each row once	Continuous ingestion and transformations that should scale with the rate of new data
Materialized view	Recomputes aggregations and joins automatically when inputs change, always returning current results	Aggregations and joins where correctness across the full history matters
Triggered mode	Refreshes the data that's available, then stops and releases compute	The recommended default: scheduled or on-demand refresh
Continuous mode	Keeps compute running so new data is processed with minimal delay	Workloads with a proven low-latency requirement

How Databricks approaches it with Lakeflow Pipelines

• One definition, either mode. You write a pipeline once in SQL or Python. Flows use the same DataFrame API as Apache Spark and Structured Streaming, so a flow can write into a streaming table with streaming semantics or a materialized view with batch semantics. See the Spark Declarative Pipelines concepts.
• The engine plans execution. Lakeflow resolves dependencies, orders and parallelizes flows, and reprocesses only what changed. It manages checkpoints, exactly-once processing, retries, and schema evolution for you. See how to use pipelines.
• Same code, classic or serverless. The same pipeline runs on classic or serverless compute, so you can change where and how it runs without touching the logic.
• Lower latency when you need it. Real-Time Mode, in Public Preview for Spark Declarative Pipelines, processes records continuously rather than in micro-batches for latency-sensitive streaming, on the same declarative code.
• Governed and open. Pipelines write to Delta Lake tables governed by Unity Catalog, and the transformation model is the open Apache Spark Declarative Pipelines standard, so your logic stays portable. See bringing declarative pipelines to Apache Spark.

Use cases

A single medallion pipeline can ingest streaming events into a streaming table, then maintain gold aggregations as materialized views that refresh when the raw data changes, batch and streaming side by side. Teams often start a pipeline in triggered mode, then switch it to continuous mode once a low-latency need is proven, without rewriting anything. The reverse works too: a pipeline built for streaming can run on a schedule as triggered batch when continuous processing isn't required.

FAQs

Do I need separate pipelines for batch and streaming?

No. One pipeline definition covers both. The dataset types you declare and the execution mode you configure determine whether data is processed incrementally as a stream or recomputed in batch.

What's the difference between a streaming table and a materialized view?

A streaming table processes incremental, append-only data and handles each row once, which suits continuous ingestion. A materialized view recomputes aggregations and joins when its inputs change, which suits batch transformations that must stay correct across history.

What do triggered and continuous modes change?

Triggered mode refreshes available data and then stops, releasing compute, and is the recommended starting point. Continuous mode keeps compute running to process data with minimal delay. The pipeline code is identical; only the mode changes.

Is the transformation code portable?

Yes. Lakeflow Pipelines are built on the open Apache Spark Declarative Pipelines standard, so the declarative model isn't tied to one runtime while it runs on the Databricks engine.