How do I size a cluster for a streaming workload?

Summary

• Size a streaming cluster by measurement. A query keeps up when it processes rows at least as fast as they arrive and each micro-batch finishes within its trigger interval. Read inputRowsPerSecond and processedRowsPerSecond from the query progress.
• The Spark docs give no fixed ratio of memory to cores. They do fix some inputs: a Kafka topic partition maps to one Spark partition by default, and spark.sql.shuffle.partitions (default 200) cannot change for a stateful query after its first run.
• Memory follows state. The default HDFS state store provider holds state in executor JVM memory. The RocksDB provider uses native memory and local disk, with an optional per-node cap.
• Bound batch size with source options such as maxOffsetsPerTrigger, which you can change between restarts.
• On Databricks, run streams as Lakeflow Jobs on jobs compute and do not enable autoscaling for them. Lakeflow Pipelines offer enhanced autoscaling for streaming workloads.

How do I size a cluster for a streaming workload?
Sizing a Structured Streaming cluster is a loop: measure, change one setting, measure again. The Apache Spark docs give no universal ratio of memory to cores or of executors to partitions. This page uses the numbers the docs do publish and the signals a running query reports.

What does "keeping up" mean?

With no trigger set, Spark starts each micro-batch as soon as the previous one finishes. With a fixed interval, a batch that finishes early waits for the interval to end. If a batch runs past the interval, the next batch starts "as soon as the previous one completes." A cluster keeps up when batches finish inside the interval and rows are processed as fast as they arrive.

Signal	Where to read it	What it tells you
inputRowsPerSecond and processedRowsPerSecond	query.lastProgress; Input Rate and Process Rate in the Structured Streaming tab	Whether processing matches the arrival rate
durationMs.triggerExecution	Query progress	Time to plan and run one micro-batch
numRowsTotal, memoryUsedBytes under stateOperators	Query progress	Whether state is growing
GC time	Executors tab	JVM heap pressure
avgOffsetsBehindLatest	Kafka source metrics on Databricks	Backlog behind the latest Kafka offsets

How many cores do I need?

Start from the parallelism that the source and the shuffle impose.

• Source tasks. A Kafka source maps each topic partition to one Spark partition by default. minPartitions can split large Kafka partitions into more Spark partitions.
• Shuffle tasks. Joins and aggregations use spark.sql.shuffle.partitions, which defaults to 200. The tuning guide recommends 2-3 tasks per CPU core, so a 48-core cluster points to about 96 to 144 shuffle partitions.
• Locked in. For stateful queries this value cannot change after the first run, because state is partitioned by hashing the key. To change it you "discard the checkpoint and start a new query," so set it with the largest cluster you plan to run in mind. If you later want fewer tasks, coalesce avoids unnecessary repartitioning.
• Per machine. The hardware provisioning guide says you should likely provision at least 8-16 cores per machine.

How much memory do I need?

Memory follows state. A stateful operator keeps state for each shuffle partition, and a stream-stream join creates four state store instances per partition.

• HDFS state store provider (the Spark default). State sits in executor JVM memory, so millions of keys can cause long GC pauses.
• RocksDB provider. State sits in native memory and on local disk. Set spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage and spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB to cap it per node.
• Watermarks. A watermark lets Spark clean up old state. A larger threshold keeps more state and might require more compute resources.
• Node memory. Allocate at most three quarters of a machine's memory to Spark.

How do I keep each batch a predictable size?

Admission control puts an upper bound on each micro-batch. Use maxOffsetsPerTrigger for Kafka, and maxFilesPerTrigger or maxBytesPerTrigger for files.

spark.conf.set("spark.sql.shuffle.partitions", "96")  # before the query's first run

df = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "host1:port1")
    .option("subscribe", "topic1")
    .option("maxOffsetsPerTrigger", "100000")
    .load())

You can add, remove or change rate limits between restarts without resetting the checkpoint. After a failure, the micro-batch planned before the stop runs first, and a change to admission control applies only after it completes. If that batch failed with an out-of-memory error, you might need to scale up the compute.

What changes on Databricks?

• Run production streams as Lakeflow Jobs in continuous mode on jobs compute, not all-purpose compute. Databricks says: "Don't enable autoscaling for compute for Structured Streaming jobs."
• Compute autoscaling has limitations scaling down for streaming workloads, so Databricks recommends Lakeflow Pipelines with enhanced autoscaling. It scales on task slot utilization and task queue size.
• The Spark UI reports Structured Streaming metrics under the Streaming tab.
• Photon supports stateless streaming only.

FAQs

Where does micro-batch time go?

The Structured Streaming tab breaks Operation Duration into addBatch, getBatch, latestOffset, queryPlanning and walCommit. The docs say addBatch "should take the bulk of the micro-batch's time."

Can I raise the trigger interval to give batches more time?

The trigger interval controls how frequently Structured Streaming checks for new data. Databricks recommends a trigger mode that balances latency and cost, and notes that a processing time trigger reduces overhead by preventing the system from checking for data too frequently.