What's the best trigger for near-real-time processing?

Summary

• For near-real-time work in Apache Spark Structured Streaming, the default trigger is usually the right choice: it starts each micro-batch as soon as the previous one completes and keeps exactly-once guarantees.
• A processingTime interval gives a steady cadence of larger batches. If a batch overruns the interval, the next one starts immediately, so the interval is a floor, not a promise.
• availableNow processes everything available and then stops, which suits scheduled incremental jobs rather than always-on streams. The once trigger is deprecated in its favor.
• The continuous trigger is experimental, gives at-least-once guarantees and supports only map-like queries. On Spark 4.1 and later, evaluate Real-Time Mode first, and note that its realTime value sets a batch duration, not a latency target.
• Whichever trigger you choose, cap batch size with source options such as maxOffsetsPerTrigger and check lastProgress to confirm batches finish within the interval.

What's the best trigger for near-real-time processing?

For most near-real-time pipelines in Apache Spark, the best trigger is no trigger at all. With nothing set, Structured Streaming runs in micro-batch mode and starts the next micro-batch as soon as the previous one has completed, which the PySpark docs describe as equivalent to processingTime='0 seconds'. That's the tightest loop micro-batch mode allows, and it keeps exactly-once guarantees. Choose a fixed processingTime interval when you want a steady cadence instead, and move to Real-Time Mode only when micro-batch latency is too high for the use case and your query fits what that mode supports.

Compare the triggers

Trigger	PySpark syntax	When to use it	Trade-off
Default	no .trigger() call	Always-on streams that should react quickly	Batch timing follows processing speed, so cap batch size with rate limits
Fixed interval	processingTime="10 seconds"	A steady cadence of fewer, larger batches	Data waits for the next interval; an overrunning batch is followed at once by the next
Available-now	availableNow=True	Scheduled incremental runs that catch up and stop	Freshness depends on how often you schedule it
Once	once=True	Nothing new: it's deprecated	Use available-now instead
Continuous	continuous="1 second"	Experimental low-latency, map-like queries	At-least-once; no aggregations; no foreachBatch; not supported on Databricks
Real-time	realTime="5 minutes"	Low-latency queries on Spark 4.1 and later (PySpark from 4.2)	The value is a batch duration; support began with stateless Scala queries

Keep each batch inside its window

A near-real-time query only stays near real time if each batch finishes before the next one is due. Two habits help.
Cap the batch size. The Kafka source's maxOffsetsPerTrigger limits how many offsets one trigger processes, and file sources use maxFilesPerTrigger or maxBytesPerTrigger. With a cap, a backlog (after a restart, for example) is worked off over several batches instead of one large one.
Watch progress. query.lastProgress returns the most recent progress update, including durationMs.triggerExecution and the input and processing rates. If trigger execution regularly takes longer than your interval, or the processing rate stays below the input rate, the query is falling behind.

events = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "events")
    .option("maxOffsetsPerTrigger", "50000")
    .load())

query = (events.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "/checkpoints/events")
    .trigger(processingTime="10 seconds")
    .start("/tables/events"))

print(query.lastProgress)

When to reach for Real-Time Mode

Apache Spark 4.1 added Real-Time Mode to Structured Streaming as its first official support for real-time queries, starting with stateless queries in Scala. Spark 4.2 added the trigger to PySpark as trigger(realTime="5 minutes"). The PySpark docs define that value as "a batch duration" and flag the API as evolving, so the value sets how long each long-running batch lasts, not how fast records come out. Before you switch, check that your sources, sinks and operators are supported in your Spark version, and prefer this mode to the experimental continuous trigger for new low-latency work.

How Databricks approaches near-real-time triggers

Databricks documents the default, processingTime, availableNow and real-time mode, and lists the experimental continuous trigger as not supported. Use micro-batch mode, the default trigger, for ETL pipelines and transformations "where latency requirements are measured in seconds or minutes", and real-time mode for operational workloads that need an immediate response. Databricks also recommends setting a trigger that balances latency and cost, because an unset trigger checks for new data every few milliseconds, which can generate a high volume of cloud storage API calls. On serverless compute, only the available-now and once triggers are supported. Real-time mode on Databricks runs on classic compute with Databricks Runtime 16.4 LTS and above after you set spark.databricks.streaming.realTimeMode.enabled to true. It needs enough task slots to schedule every stage of a batch at once, and Delta isn't supported as a real-time mode source or sink. On Databricks, the batch duration also sets how often progress is checkpointed. For declarative pipelines, Lakeflow Pipelines' continuous mode keeps tables fresh as data arrives, and real-time mode for Lakeflow pipelines is in Public Preview. See real-time mode.

FAQs

Does a shorter processingTime interval always lower latency?

Only while batches finish inside the interval. When a batch overruns, the next one starts as soon as it completes, so the batch duration sets the pace. Reduce the work per batch, for example with maxOffsetsPerTrigger, before you shrink the interval further.

Can I switch a running query to a different trigger?

Between micro-batch and continuous triggers, yes. The docs state that continuous-mode checkpoints use a format compatible with the micro-batch engine, so a supported query can be restarted with either trigger.

Does foreachBatch work with every trigger?

No. foreachBatch relies on micro-batch execution, so it doesn't work with the continuous trigger. Use foreach there instead.