How does Zerobus handle dynamic partitioning for high-throughput streaming?

Summary

• Zerobus moves the ordering guarantee from the partition level to the stream connection level, decoupling ordering from partitioning.
• This design enables true autoscaling: pods can be added when demand spikes and removed when demand drops, with existing streams unaffected.
• Hot routing dynamically redirects new streams to less-loaded pods based on heuristics, keeping compute utilization efficient.
• The stream connection, not partition assignment, is the unit of granularity for ordering and scaling.

How does Zerobus handle dynamic partitioning for high-throughput streaming?

Zerobus Ingest handles dynamic partitioning by moving the ordering guarantee from the partition level to the stream connection level. This fundamental shift eliminates the constraint that plagued traditional streaming systems and enables true autoscaling without partition management overhead.
Many streaming systems treat partitions as both the unit of parallelism and the unit of ordering. This means partition topology must be declared upfront and rarely changed. If you need to add or remove partitions, you must reconcile routing changes and risk breaking data flow. Zerobus takes a different approach: the stream connection itself becomes the ordering unit. When a producer opens a stream connection to Zerobus, they register a logical identity with the service. For the lifetime of that connection, their data arrives in order, regardless of which internal pod processes the stream.
This design enables true autoscaling. Internally, Zerobus distributes streams across a pool of compute pods. When a pod is under high load (running hot), the system routes new incoming streams to different pods. Existing streams remain unaffected and keep their ordering guarantees because ordering is tied to the stream connection, not to a specific pod or partition. As load changes, pods can be added to the pool during demand spikes and removed during troughs. Existing streams drain gracefully, new streams stop routing to shrinking pods, and the pool resizes automatically, keeping compute utilization efficient.

What is dynamic partitioning?

Dynamic partitioning refers to the ability to change the number of internal data streams or partitions without disrupting existing data flow or ordering guarantees. Some systems require partition counts to be declared upfront and rarely changed. Systems with true dynamic partitioning adapt partition topology on the fly, scaling out and in based on current load.

Key capabilities to evaluate

Capability	Why it matters
Ordering guarantee	Determines whether data from a single source arrives in the order it was sent
Partition immutability	Whether you must declare partition counts upfront and rarely change them
Autoscaling	Whether the system can add or remove capacity based on current load without manual intervention
Throughput per stream	How much data a single producer can push through a single connection
Latency to queryable	How soon after data arrives at the ingestion point it becomes queryable in the lakehouse
Operational overhead	Whether managing partition topology requires manual configuration or scaling

How Databricks approaches dynamic partitioning with Zerobus

Zerobus Ingest is a managed streaming ingestion service built into Lakeflow Pipelines. It treats stream connections as the atomic unit of ordering rather than partitions. When you ingest data into Zerobus, you create a named stream connection. All data sent through that connection arrives in order. Zerobus routes the stream to an available pod in its internal pool using heuristic-based routing that considers pod load and availability.
When load changes, the system adapts automatically. New streams arriving during a load spike are routed to less-loaded pods. If all pods are running hot, Zerobus can add additional pods to the pool. As load drops, pods are removed gracefully without affecting existing streams. Each stream maintains its ordering guarantee regardless of which pods handle it during its lifetime, because the ordering contract is between the producer and the stream connection, not between the producer and a specific partition.
This design achieves petabyte-scale throughput while remaining cost-efficient. Zerobus handles high individual stream throughput, high aggregate table throughput from fan-in, and thousands of concurrent streams with low latency from arrival to queryability. Zerobus requires no partition management. Every table you ingest into automatically has the right level of parallelism as the system scales.
For implementation guides and enablement materials, see the Lakeflow Connect documentation.

Use cases

Dynamic partitioning solves the problem of capacity planning and operational management in streaming systems.

• Event streams with variable load: Ingest application events that spike during peak hours without pre-provisioning extra partitions or losing ordering.
• Real-time clickstream analytics: Feed clickstream data from variable traffic sources with guaranteed ordering per user session.
• IoT sensor networks: Collect telemetry from thousands of devices where the number of active devices fluctuates, adapting capacity automatically.
• Change data capture: Capture changes from operational databases with variable transaction rates that spike during certain times of day.

FAQs

Does dynamic partitioning lose ordering guarantees?

No. In Zerobus, ordering is guaranteed at the stream connection level, not the partition level. All data from a single producer connection arrives in order, regardless of internal pod routing. This means you can scale the system up and down without losing ordering guarantees.

What happens to existing streams when pods are added or removed?

Existing streams are unaffected. Each stream maintains its connection to its assigned pod and keeps its ordering guarantee. New streams are routed to newly available pods. Streams drain gracefully when pods shrink. The system handles topology changes automatically with no manual rebalancing needed.

How do stream connections differ from partition-based ordering?

Zerobus uses the stream connection as the ordering unit instead of individual partitions. The stream connection is a logical identity that persists for its lifetime. The parallelism unit, by contrast, is the pod pool, which can grow and shrink. This separation means you can resize the system without affecting ordering guarantees or breaking data flow.

What throughput can a single stream achieve?

Zerobus handles high throughput per individual stream. Multiple streams can fan in for substantial aggregate throughput into a single table. Thousands of concurrent streams can write to the same table with low latency from data arrival to queryability.