Will real-time ingestion become the default instead of nightly batch ETL?

Summary

• We believe real-time ingestion will become the default for operational workloads such as fraud detection and personalization, while scheduled or incremental batch stays common for analytics. This is a forecast, so it is our view and not a documented fact.
• On Databricks the choice is largely a setting. A Structured Streaming trigger can run a query as scheduled incremental batch, at a fixed interval, or in real-time mode, and you can change the trigger interval between runs while using the same checkpoint.
• Lakeflow Pipelines offers triggered and continuous modes. Triggered mode runs the cluster only long enough to update the pipeline, and continuous mode needs an always-running cluster to reduce processing latency.
• Ingestion options span the range: Auto Loader for files, Lakeflow Connect connectors with change data capture and streaming sources, and Zerobus Ingest for direct writes.

Will real-time ingestion become the default instead of nightly batch ETL?

We believe real-time ingestion will become the default for operational use cases, but not a replacement for scheduled batch in analytics. The Databricks docs steer users the same way. They say to "Use real-time mode for operational workloads that require immediate response to streaming data, such as fraud detection and real-time personalization," and to use micro-batch mode, the default trigger, for "Analytical processing: ETL pipelines, data transformations, and medallion architecture implementations where latency requirements are measured in seconds or minutes." Because one engine can run either way, on Databricks the choice is largely a per-pipeline setting.

How do the Databricks processing modes differ?

Mode	What the Databricks docs describe	Where the docs point it
Structured Streaming, AvailableNow trigger	Consumes all available records as an incremental batch	Scheduled incremental batch processing
Structured Streaming, processingTime trigger	Checks for new data at a fixed interval	Balancing cost and performance
Structured Streaming, real-time mode	A trigger type for Structured Streaming, generally available on Databricks since March 2026	Operational workloads that require an immediate response
Lakeflow Pipelines, triggered	Stops after refreshing all tables based on the data available when the update started	The cluster runs only long enough to update the pipeline, and new data is not processed until the pipeline is triggered
Lakeflow Pipelines, continuous	Processes new data as it arrives in data sources	Requires an always-running cluster and reduces processing latency

For freshness, the pipeline-mode docs pair triggered mode with data updates that "run every 10 minutes, hourly, or daily," and continuous mode with updates that "are desired between every 10 seconds and a few minutes." For always-on streaming workloads, Databricks recommends continuous mode in Lakeflow Jobs.

Where does scheduled batch still fit?

• Incremental by design. The AvailableNow trigger processes the data available when the job is triggered, so a scheduled job can still read only new records.
• Event-driven schedules. Lakeflow Jobs can start a run on a schedule, on a source table update, or on the arrival of files to a Unity Catalog storage location.
• Incremental refresh. Incremental refresh on a materialized view "detects changes in the source data and recomputes only the affected results, rather than recomputing the entire query."
• Resource use. Triggered pipelines run the cluster "only long enough to update the pipeline," continuous pipelines "require an always-running cluster," and real-time mode "requires dedicated compute resources."

Which ingestion options span batch and real-time?

• Auto Loader "incrementally and efficiently processes new data files as they arrive in cloud storage."
• Lakeflow Connect offers database connectors that use change data capture, SaaS and file connectors, query-based connectors, and streaming connectors that continuously ingest from message buses and event streaming sources such as Apache Kafka. Some sources or specific tables don't support incremental ingestion, and connectors "are in various release states," so check the connector docs.
• Zerobus Ingest is a push-based streaming API that writes data into Unity Catalog Delta tables. Its gRPC (SDKs), REST, and OpenTelemetry interfaces are listed as Generally Available.

FAQs

When should I move a pipeline from batch to real-time?

We believe the latency requirement should decide. If a workload needs an immediate response, such as blocking a transaction, the docs point to real-time mode. If results can arrive in seconds or minutes, micro-batch mode is the documented fit.

Can I start with batch and add real-time later?

On Databricks, yes. A Structured Streaming query can start with the AvailableNow trigger for scheduled incremental batch, and "You can change the trigger interval between runs while using the same checkpoint." In Lakeflow Pipelines, both materialized views and streaming tables can be updated in either triggered or continuous mode.

Does real-time mode run on every compute type?

No. The Real-time mode reference currently lists dedicated compute and, for Python only, standard compute. It lists serverless as not supported, so check the reference for the latest limits before you plan around it.