processingTime versus availableNow triggers?

Summary

• processingTime keeps a Structured Streaming query running and starts a micro-batch at a fixed interval, as in .trigger(processingTime='1 minute'). availableNow processes all data available when the query starts, in one or more micro-batches, then stops: .trigger(availableNow=True).
• With processingTime, Spark waits for the next interval if a batch finishes early, starts the next batch immediately if one runs long, and skips the interval when no new data has arrived.
• availableNow, introduced in Spark 3.3.0, supersedes the deprecated one-time trigger. It honors rate limits such as maxFilesPerTrigger, processes uncommitted batches from an earlier run first, and advances the watermark per batch.
• Choose processingTime when results must stay fresh and compute can stay up. Choose availableNow when a scheduler can start the job periodically and the cluster can shut down in between.
• On Databricks serverless compute, only Trigger.AvailableNow() and the deprecated Trigger.Once() are supported.

processingTime versus availableNow triggers?

Both are micro-batch triggers in Apache Spark Structured Streaming, and they differ in the one thing that drives every other choice: whether the query stays alive. A processingTime query runs until you stop it and starts a micro-batch at a fixed interval. An availableNow query processes everything available when it starts, then terminates, and a scheduler decides when the next run happens. Use processingTime for always-on freshness and availableNow for scheduled incremental batch on compute that can shut down between runs. The rest of the query, including its sources, sinks and checkpoint, stays the same.

Side by side

Aspect	processingTime	availableNow
Python syntax	.trigger(processingTime='1 minute')	.trigger(availableNow=True)
Scala and Java syntax	Trigger.ProcessingTime("1 minute")	Trigger.AvailableNow()
Lifecycle	Runs until stopped or until it fails	Stops on its own once the data available at start is processed
Batch timing	One micro-batch per interval; skipped when no new data arrives	Back-to-back micro-batches until the backlog is done
Batch size	Rate limits cap each micro-batch	Rate limits split the backlog into several micro-batches
Scheduling	The trigger is the schedule	An external scheduler starts each run
Watermarks	Advance with each micro-batch	Advance per batch, with a final no-data batch when the last batch moved the watermark
Source support	All streaming sources	Sources that support Trigger.AvailableNow; otherwise Spark falls back to a one-time micro-batch
Available since	The trigger API dates to Spark 2.0	Spark 3.3.0

How processingTime behaves

The Structured Streaming guide calls this trigger "fixed interval micro-batches". If a micro-batch completes within the interval, the engine waits until the interval is over before starting the next one. If a batch runs past the interval, the next batch starts as soon as the previous one completes rather than waiting for the next boundary. If no new data is available, no micro-batch starts. The query holds its compute for as long as it runs.

How availableNow behaves

availableNow processes all the data available at the time of execution and then stops on its own, splitting the work into micro-batches according to source options such as maxFilesPerTrigger or maxBytesPerTrigger. The guide describes a strong processing guarantee: regardless of how many batches were left over from a previous run, all uncommitted batches are processed first, and all available data is processed before termination. The watermark advances per batch, which keeps state for stateful operators smaller and more predictable. One caveat: if any source in the query doesn't support Trigger.AvailableNow, Spark deactivates the trigger and performs a one-time micro-batch instead.

One query, either trigger

def start_orders_stream(always_on):
    writer = (spark.readStream
              .format("json")
              .schema(order_schema)
              .option("maxFilesPerTrigger", 100)
              .load("/landing/orders")
              .writeStream
              .format("parquet")
              .option("path", "/lake/orders")
              .option("checkpointLocation", "/checkpoints/orders"))
    if always_on:
        return writer.trigger(processingTime="1 minute").start()
    return writer.trigger(availableNow=True).start()

File sources need an explicit schema by default, hence order_schema. When you run the availableNow form from a scheduler, call awaitTermination() on the returned query so the job exits when the backlog is done. Only one trigger can be set per query.

Which one should you choose?

• Freshness target. If consumers expect results within minutes all day, keep a processingTime query running. If a periodic refresh is enough, availableNow does the same work in scheduled bursts.
• Compute model. The guide's own use case for run-and-stop triggers is to spin up a cluster periodically, process everything since the last run, and shut it down.
• Backlogs. After an outage or a backfill, availableNow works through the backlog in rate-limited batches instead of one oversized batch.
• Source support. File sources, Apache Kafka (since Spark 3.3.0) and Delta Lake tables (since Delta Lake 2.1.0) support Trigger.AvailableNow. Check any other source before you switch.

How Databricks approaches triggers

On serverless compute, Databricks supports Trigger.AvailableNow(), which it recommends, and the deprecated Trigger.Once(). Trigger.ProcessingTime(interval) raises INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED. For always-on work on serverless, the docs point to continuous mode in Lakeflow Pipelines or Trigger.AvailableNow() inside a continuous Lakeflow Jobs run. On classic compute both triggers work. Databricks recommends always setting one: without a trigger, a query runs as processingTime with a zero interval and checks for new data constantly, which can drive up cloud storage API calls. You can change the trigger between runs while keeping the same checkpoint, and the docs describe how an in-flight micro-batch finishes during the switch. In Databricks Runtime 11.3 LTS and above, Trigger.Once is deprecated in favor of Trigger.AvailableNow.

FAQs

Does processingTime guarantee one micro-batch per interval?

No. When no new data arrives, no micro-batch starts. When a batch overruns the interval, the next one starts immediately instead of waiting for the next boundary.

Should I still use the once trigger?

No. Apache Spark marks it deprecated, and the guide says availableNow gives a better processing guarantee, finer-grained batches and better watermark handling, including no-data batches.

What is the default trigger?

If you set none, Spark runs micro-batches as soon as the previous one completes. The PySpark docs describe this as equivalent to processingTime='0 seconds'.