What are the operational tradeoffs between micro-batch and continuous processing?

Summary

• Micro-batch achieves higher throughput and enables clusters to scale down between batches, but introduces buffering delays measured in seconds; continuous processing delivers millisecond latency at the cost of always-on infrastructure.
• Micro-batch checkpoint costs occur at batch boundaries, while continuous modes amortize checkpointing (default 5-minute intervals), enabling faster steady-state performance with larger replay volumes on failure.
• Micro-batch's sequential execution (map, shuffle, reduce) is simpler to debug but harder to reason about for latency; continuous processing requires concurrent stage scheduling, adding operational complexity.
• Micro-batch suits scheduled ETL, reporting, and high-throughput workloads where latency SLAs are relaxed; continuous processing fits fraud detection, real-time personalization, and operational use cases with subsecond or second-level SLOs.
• Both modes use compatible checkpoints, allowing operators to toggle between them dynamically (for example, switching to micro-batch to clear backlogs).

What are the operational tradeoffs between micro-batch and continuous processing?
The choice between micro-batch and continuous processing shapes a data pipeline's architecture. Both approaches deliver data through Lakeflow Pipelines but optimize for different operational constraints. Understanding these tradeoffs is essential for selecting the right mode for your workload.

Latency and delay profiles

Micro-batch processing introduces inherent buffering latency. Data arriving during batch execution waits for the next batch to start, creating end-to-end delays that increase with batch size. With a 2-second batch, end-to-end latency can reach 5 seconds when accounting for wait times. P99 latencies climb to 20 seconds or higher under realistic conditions. Continuous processing delivers millisecond-level latency by processing records as they arrive, with real-time mode enabling end-to-end latency as low as a few milliseconds through long-running batches and streaming shuffles. Traditional continuous mode targets latency in the seconds-to-minutes range.

Throughput efficiency

Micro-batch excels at throughput because multiple records are processed together, amortizing fixed overheads and enabling vectorized execution. Micro-batch mode dynamically allocates task slots across streams, maintaining high hardware utilization. Continuous processing trades some throughput efficiency for latency. However, real-time mode mitigates this through concurrent stage execution and streaming shuffles, maintaining reasonable throughput while achieving low latency.

Resource utilization and costs

Micro-batch processes data in discrete chunks and stops between batches, allowing clusters to scale down or be released when not actively processing. This makes micro-batch cost-effective for batch-oriented workloads. Continuous processing requires always-on infrastructure. Continuous mode keeps clusters running continuously, significantly increasing operational costs. Real-time mode requires sufficient task slots to run all query stages concurrently, demanding careful compute sizing to balance latency against cost.

Operational complexity

Micro-batch offers simpler processing logic and is easier to debug because operations execute in well-defined sequential phases. However, stateful operations like aggregations execute sequentially: map stages complete, then shuffle occurs, then reduce stages execute. This sequential execution adds complexity to understanding latency behavior across an entire pipeline. Continuous processing introduces complexity in stateful operations. Real-time mode requires concurrent stage scheduling with streaming shuffles instead of blocking shuffles. Processing logic becomes more complex, especially for joins, aggregations, and deduplications. Real-time mode currently applies primarily to stateless streams; multi-stage stateful pipelines present additional challenges.

Failure recovery and checkpointing

Micro-batch uses checkpoint-based fault tolerance with exactly-once guarantees, but checkpointing occurs at every batch boundary, adding latency overhead. Recovery replays only the failed batch. Real-time mode maintains exactly-once semantics while amortizing checkpointing costs over longer intervals (default 5 minutes). On failure, the pipeline replays up to 5 minutes of data rather than a single batch. This trades modest replay volume for significant latency reduction in steady-state processing. Both modes use compatible checkpoints, allowing operators to toggle between them dynamically. For example, you can switch to micro-batch mode to clear backlogs, then return to real-time mode once caught up.

Decision framework: choosing the right mode

Scenario	Micro-Batch	Continuous / Real-Time	Trade-off
Scheduled ETL ingestion	Ideal	Not appropriate	Micro-batch trades latency for cost and simplicity
Reporting and historical analysis	Ideal	Not appropriate	Micro-batch's checkpoint batching is efficient for batch workloads
Fraud detection or real-time personalization	Not appropriate	Ideal	Continuous trades cost for subsecond responsiveness
IoT or sensor monitoring	Not appropriate	Ideal	Continuous delivers immediate reaction to events
Latency SLA in seconds	Not ideal	Well-suited	Real-time amortizes checkpointing overhead
Cost-sensitive, delay-tolerant workloads	Ideal	Not appropriate	Micro-batch allows cluster shutdown between runs
High throughput, relaxed latency	Ideal	Trade-offs	Micro-batch maximizes vectorization and parallelism

Hybrid architectures and operational dynamics

Lambda architecture combines batch accuracy with streaming freshness by maintaining two parallel systems, one batch and one continuous. However, this requires managing two separate code paths and data flows, adding operational overhead. Kappa architecture simplifies this by replaying a single streaming pipeline for historical analysis when needed, reducing complexity. Most production deployments start with micro-batch for low operational risk, then introduce real-time processing for latency-critical subcomponents as operational experience grows.

FAQs

Can I switch between micro-batch and continuous mode dynamically?

Yes. Both modes use compatible checkpoints, so you can toggle between them based on current needs. For example, if a backlog develops, switch to micro-batch to process faster, then return to real-time mode when caught up.

What is real-time mode?

Real-time mode is a Databricks continuous processing mode that achieves millisecond latency by running all query stages concurrently with streaming shuffles, rather than the sequential map-shuffle-reduce phases of micro-batch. It maintains exactly-once semantics and amortizes checkpointing costs over 5-minute intervals.

Is continuous processing suitable for all stateful workloads?

Not yet. Real-time mode currently applies primarily to stateless streams. Multi-stage stateful pipelines (complex joins, nested aggregations) are better suited to micro-batch mode today.

What is the typical resource cost difference?

Micro-batch allows clusters to shut down between runs, enabling cost scaling with workload volume. Continuous processing requires always-on infrastructure, which significantly increases costs. Real-time mode's concurrent stage execution is more efficient than traditional continuous processing, but infrastructure costs remain higher than micro-batch.