What's the best way to stream data into Databricks?

Summary

• The best way to stream data into Databricks depends on the source: Auto Loader for files in cloud storage, Structured Streaming or Lakeflow Pipelines for message buses such as Apache Kafka, and Zerobus Ingest for producers that push records directly.
• Databricks documents three ingestion layers, ordered from most customizable to most managed, and recommends starting with the most managed layer that supports your source.
• Structured Streaming offers end-to-end fault tolerance with exactly-once processing guarantees, and a checkpoint lets a restarted query continue where it left off.
• Real-time mode, a Structured Streaming trigger type announced as Generally Available on March 19, 2026, serves operational workloads that need ultra-low latency.

What's the best way to stream data into Databricks?

Start from where the data lives, then pick the most managed tool that supports it. The Databricks ingestion docs say it directly: "Databricks recommends starting with the most managed layer. If it doesn't satisfy your requirements (for example, if it doesn't support your data source), drop down to the next layer." In practice that means Auto Loader for files, Lakeflow Pipelines or Structured Streaming for message buses, and Zerobus Ingest when your applications can push records themselves.

What are the layers of streaming ingestion on Databricks?

The docs describe three layers, ordered from most customizable to most managed:

• Structured Streaming: "a streaming engine that offers end-to-end fault tolerance with exactly-once processing guarantees using Spark APIs."
• Lakeflow Pipelines: they "extend Structured Streaming, offering a declarative framework for creating data pipelines" in SQL and Python.
• Managed connectors: "Fully-managed connectors build on Lakeflow pipelines, offering even more automation for the most popular data sources."

Alongside these, Zerobus Ingest, part of Lakeflow Connect, is a push-based streaming API that writes data straight into Unity Catalog Delta tables.

Which ingestion path fits your source?

Source	Databricks path	What the docs say
Files in cloud storage	Auto Loader (cloudFiles)	Processes new data files incrementally as they arrive; file metadata kept in the checkpoint location ensures data is processed exactly once
Apache Kafka	kafka source, or read_kafka in streaming tables	Kafka works as a source or a sink for Structured Streaming
Amazon Kinesis	kinesis source	Structured Streaming reads and writes data to Kinesis
Google Pub/Sub	pubsub source, or read_pubsub in a streaming table	Exactly-once processing semantics for rows from the subscriber
Applications that push records	Zerobus Ingest	Writes into Unity Catalog Delta tables with no message bus to run
SaaS applications and databases	Lakeflow Connect managed connectors	Pipelines governed by Unity Catalog and powered by serverless compute and Lakeflow Pipelines
Files loaded by SQL	COPY INTO	Retriable and idempotent; files already loaded are skipped on later runs

How do Lakeflow Pipelines handle streaming?

A Lakeflow pipeline defines streaming tables and materialized views. For a streaming table, "Each record is processed exactly one time, assuming an append-only source." Pipelines run flows "in the correct order with maximum parallelism, and retry transient failures progressively", from the Spark task to the flow to the entire pipeline. A pipeline runs in triggered mode, which refreshes available data and stops, or continuous mode, which keeps tables fresh as new data arrives.
Databricks recommends Auto Loader in Lakeflow Pipelines for incremental data ingestion. Expectations add data quality checks: EXPECT writes invalid records to the target, ON VIOLATION DROP ROW drops them, and ON VIOLATION FAIL UPDATE stops the update.

When do you need real-time mode?

Real-time mode is a trigger type for Structured Streaming that enables ultra-low latency data processing. Databricks announced it as Generally Available on March 19, 2026, and the docs aim it at operational workloads that need an immediate response to streaming data, "such as fraud detection and real-time personalization." For analytical processing "where latency requirements are measured in seconds or minutes," the docs point to micro-batch mode, the default trigger.
The reference lists Apache Kafka, Event Hubs through the Kafka connector, Kinesis, and MSK as sources, supports update output mode only, and doesn't support serverless compute. Auto Loader and Delta tables aren't on the source list.

How do you control throughput and recover from failures?

Admission controls limit the input rate of a streaming query: cloudFiles.maxFilesPerTrigger and cloudFiles.maxBytesPerTrigger for Auto Loader, maxFilesPerTrigger and maxBytesPerTrigger for Delta sources, and maxOffsetsPerTrigger for Kafka. The docs say they "can help maintain a consistent batch size and prevent large batches from causing spill and cascading micro-batch processing delays."
For recovery, set a checkpointLocation. The checkpoint stores the query's offsets, commits, state, and metadata, and "Each query must have a different checkpoint location." The Spark guide says that after "a failure or intentional shutdown, you can recover the previous progress and state of a previous query, and continue where it left off."

FAQs

What's the difference between Auto Loader and COPY INTO?

Auto Loader is a Structured Streaming source that incrementally processes new data files as they arrive in cloud storage. COPY INTO is a SQL command that loads files from a file location into a Delta table. It is retriable and idempotent, so files already loaded are skipped on later runs. For a more scalable file ingestion experience, Databricks recommends that SQL users use streaming tables.

Can I stream from Kafka using SQL?

Yes. The docs say "Streaming with SQL is supported only in Lakeflow pipelines or with streaming tables in Databricks SQL," and the read_kafka function applies to Databricks SQL and Databricks Runtime 13.3 LTS and above.

Do I need to write imperative Spark code?

No. In Lakeflow Pipelines you declare streaming tables and materialized views in SQL or Python, and the pipeline runs flows in the correct order and retries transient failures.

Does streaming into Delta Lake give exactly-once results?

The Structured Streaming docs describe using Delta Lake tables as streaming sources and sinks with exactly-once processing guarantees.