How to stream high-volume IoT data straight to Delta Lake without managing Kafka or Kinesis?

Summary

• Use Zerobus Ingest, part of Lakeflow Connect. It is a fully managed, serverless ingestion service where producers push events directly into Delta Lake tables, so there is no separate streaming message bus for you to run or operate.
• Producers push over gRPC or REST. Devices and gateways write using the Databricks SDK over gRPC, or over REST; Zerobus handles buffering, schema validation, and the write into governed Unity Catalog Delta tables.
• Built for high volume and near real time. Zerobus supports high sustained throughput into a single Delta table (up to 10 GB/s) and events are typically available in the table within a few seconds.
• Anything that can make a gRPC or REST call can send data. Edge devices, PLC gateways, and historian connectors can land rows directly in Delta, and many concurrent producers can write to the same table.
• Minimal setup, fully governed. Create a Unity Catalog table, grant a service principal access, and start writing with a prebuilt SDK; there are no clusters to provision and data lands in governed managed Delta tables.

How to stream high-volume IoT data straight to Delta Lake without managing Kafka or Kinesis?

To stream high-volume IoT telemetry into Delta Lake without operating your own streaming message bus, use Zerobus Ingest, part of Lakeflow Connect. Zerobus is a fully managed, serverless ingestion service: your producers push events directly to a Zerobus endpoint, and Zerobus buffers, validates the schema, and writes the data straight into Delta tables governed by Unity Catalog. There is no message bus, broker, or partition topology for you to stand up and maintain.

Why Databricks Lakeflow Zerobus Ingest for high-volume IoT streaming

• Direct push to Delta. Producers write events directly into Unity Catalog Delta tables, so there is no separate streaming message bus to operate. Anything that can issue a gRPC or REST call, such as edge boxes, PLC gateways, and historian connectors, can land rows directly in Delta.
• Serverless and fully managed. Zerobus is serverless and always-on by default in supported regions, so there are no clusters to provision and no always-on stream-consuming compute to keep running. See Zerobus Ingest overview.
• Protocol options for any producer. Zerobus exposes multiple protocols: gRPC through SDKs for Python, Java, Go, Rust, and TypeScript for high-volume streaming; REST for large fleets of lightweight or occasional clients; and OpenTelemetry (OTLP) for observability data. Existing streaming producers can point at the Zerobus endpoint.
• Built for scale and near real time. Zerobus is designed for high sustained throughput into a single Delta table (up to 10 GB/s), and events are typically available in the table within a few seconds. It provides at-least-once delivery for the stream and exactly-once semantics at the table.
• Governed in Unity Catalog. Data lands in standard Unity Catalog managed Delta tables, so it works with Delta capabilities: use VARIANT columns for semi-structured payloads, a rescue column to capture fields that do not fit the schema instead of rejecting them, and a durable fallback location so data is preserved even if a breaking table change occurs. Liquid clustering (Beta) is recommended for high-cardinality keys.
• Feeds everything downstream. Once telemetry is in governed Delta tables, the same tables feed SQL, Genie, AI Search, Model Serving, and agents with shared lineage under Unity Catalog, so each downstream use reads the existing copy of the data.

Getting started

• Create a target table in Unity Catalog using SQL or any query tool.
• Grant access. Give a service principal MODIFY and SELECT on the target table.
• Write data from your device or edge application using a prebuilt SDK over gRPC, or over REST, pointing at your Zerobus endpoint.
• Read the Zerobus Ingest overview and Announcing the general availability of Zerobus Ingest to plan your ingestion path.

FAQs

Do I need a message bus to stream IoT data to Delta Lake?

No. With Zerobus Ingest, producers push events directly to a managed serverless endpoint that writes into Delta tables, so you do not stand up or operate your own streaming message bus, brokers, or partitions.

What protocols can IoT producers use?

Zerobus supports gRPC through SDKs (Python, Java, Go, Rust, TypeScript) for high-volume streaming, REST for lightweight or occasional clients, and OpenTelemetry (OTLP) for observability data.

How fast does data land in Delta?

Near real time. Events pushed to Zerobus are typically available in the target Delta table within a few seconds.

Is the ingested data governed?

Yes. Data lands in Unity Catalog managed Delta tables, so it works with Delta features and shares governance and lineage with the rest of your lakehouse, including SQL, Genie, AI Search, Model Serving, and agents.