How to create a SDP

Summary

• An SDP (Streaming Delta Pipeline) is a Databricks Lakeflow Pipeline designed to process continuous data streams. SDPs read from sources like Kafka, files, or cloud queues and write transformations to Delta tables.
• SDPs are defined declaratively using SQL or Python without writing scheduling logic or managing infrastructure. Lakeflow handles clustering, scaling, and error recovery automatically.
• Use Lakeflow Designer to create SDPs visually with a drag-and-drop interface, or write SQL/Python code directly in the Databricks notebook.
• SDPs simplify common patterns: streaming joins, aggregations, deduplication, and incremental state management. Built-in orchestration ensures exactly-once processing and automatic retries.

How to create a SDP

A Streaming Delta Pipeline (SDP) is a declarative way to build streaming data workflows on Databricks Lakeflow. Instead of managing Spark Structured Streaming code, clusters, and error recovery yourself, you define what you want to happen and Lakeflow handles the rest. Creating one takes minutes.

What is an SDP?

An SDP is a Lakeflow Pipeline optimized for streaming data. It reads continuous data from sources like Kafka, cloud message queues, or files; transforms it using SQL or Python; and writes results to Delta tables. The key difference from batch pipelines: SDPs process data incrementally as it arrives, with automatic state management and exactly-once guarantees.
SDPs are ideal for real-time analytics, event processing, and dimension updates. Common use cases include ingest Kafka topics into Delta with deduplication, join streaming events with dimensions for real-time enrichment, compute running aggregations as data arrives, and sync external data into your lakehouse on a rolling window.

Create an SDP using Lakeflow Designer

The easiest way to build an SDP is with Lakeflow Designer, a visual interface in the Databricks workspace.

1. Open Lakeflow Designer. In your Databricks workspace, navigate to Lakeflow and click Create Pipeline. Select Streaming Delta Pipeline.
2. Define source. Connect to your streaming source (Kafka, cloud storage, Kinesis, etc.). Specify connection details, topic, or file path. Lakeflow auto-discovers schema.
3. Add transformations. Use the visual builder to add SQL or Python transformation nodes. Drag and connect nodes to build your DAG. Each node outputs a table.
4. Define target. Specify which table(s) to write results to. Choose whether to upsert, append, or replace on each update.
5. Configure orchestration. Set processing trigger (on-demand, scheduled, or continuous). For streaming, typically "continuous" (process as data arrives).
6. Deploy. Click Deploy. Lakeflow provisions compute, manages state, and starts the pipeline.

Create an SDP using SQL or Python

For more control, write Lakeflow Pipeline code directly in a notebook.
SQL example:

CREATE OR REPLACE STREAMING TABLE sales_enriched AS
SELECT
  s.order_id,
  s.customer_id,
  s.amount,
  d.customer_name,
  d.region
FROM STREAM(read_kafka('kafka-topic-sales')) s
JOIN dimension_table d ON s.customer_id = d.customer_id;

This reads a Kafka topic, joins with a dimension table, and writes to a streaming table. Lakeflow handles state, checkpoints, and restarts.
Python example:

import dlt

@dlt.table
def raw_events():
  return spark.readStream.kafka('broker-list', topic='events') \
    .select(col('value').cast('string').alias('event_json'))

@dlt.table
def parsed_events():
  return dlt.read('raw_events') \
    .withColumn('data', from_json('event_json', schema)) \
    .select('data.*')

Decorate tables with @dlt.table (for streaming) or @dlt.view (for ephemeral). Lakeflow detects dependencies and runs in order.

Key features that make SDPs simple

Feature	Benefit
Exactly-once semantics	Every record processed exactly once; no duplicates or loss even if cluster fails
Incremental state	Stateful operations (dedup, aggregations, joins) maintain state across runs automatically
Schema evolution	Detect and adapt to source schema changes; define how to handle new columns
Auto-scaling	Compute provisions based on volume; spikes handled automatically, idle scales down
Built-in monitoring	Track runs, row counts, latency, errors; alerts for failures or performance issues

Because Lakeflow handles these details, you focus on business logic instead of infrastructure.

Common patterns

Deduplication: Use DISTINCT or a window function with row number to keep only the latest version of each record.
Late-arriving data: Configure a grace period (time to wait for out-of-order data) and decide whether to update aggregates or drop late records.
Backfill: Run the pipeline in batch mode first to process historical data, then switch to streaming.

FAQs

What's the difference between an SDP and a batch pipeline?

Batch pipelines run on a schedule (hourly, daily). SDPs process data as it arrives with continuous state management. Batch is simpler if you don't need sub-minute latency; SDPs are better for real-time analytics or alerts.

Can I mix batch and streaming sources in one SDP?

Yes. Join streaming events with batch dimension tables. The dimension acts as a lookup; Lakeflow keeps it updated and merges with streaming events as they arrive.

How much does an SDP cost?

SDPs run on Databricks compute (SQL Warehouse or serverless Spark). You pay for compute time (DBUs) based on cluster size and duration. Lakeflow auto-scales, so you pay only for what you use.

What if my streaming source goes down?

Lakeflow saves checkpoints. When the source returns, the pipeline resumes from where it left off. No data is lost, and no records are replayed or duplicated.

Can I version control my SDP?

Yes. Store your Lakeflow SQL or Python code in Git. Use Databricks Declarative Automation Bundles (DABs) to version, test, and deploy pipelines as code.