Best engine for petabyte-scale analytics

Summary

• No single open-source engine is best at petabyte scale. Apache Spark runs batch and streaming in one engine, and Apache Flink describes itself as an engine for stateful computations over unbounded and bounded data streams.
• The Spark docs describe network, data locality, and skew as constraints to plan around, and Adaptive Query Execution splits skewed join tasks.
• Cut the data an engine reads before adding nodes: Spark prunes partitions from partition filters and pushes Parquet filters down by default.
• Both engines can keep large streaming state in RocksDB on local disk, but their defaults and recovery tools differ.
• On Databricks, Photon, predictive optimization, serverless compute, and Auto Loader add managed capabilities around Spark.

Best engine for petabyte-scale analytics
No single engine is best at petabyte scale. The right choice depends on whether the work is batch SQL and ETL, continuous stateful stream processing, or both. Apache Spark handles batch and streaming in one engine. Apache Flink describes itself as "a framework and distributed processing engine for stateful computations over unbounded and bounded data streams." Pick by workload, then test on your own data.

What should I plan for at petabyte scale?

The Spark docs describe three constraints worth planning around:

• Network: "when the data is in memory, a lot of Spark applications are network-bound." A 10 Gigabit or higher network is "the best way to make these applications faster," especially for group-bys, reduce-bys, and SQL joins.
• Data locality: Spark "prefers to schedule all tasks at the best locality level, but this is not always possible."
• Skew: by default, Adaptive Query Execution (AQE) treats a partition as skewed when it is larger than 5.0 times the median partition size and larger than 256MB.

How do Spark and Flink differ on the criteria that matter?

Criterion	Apache Spark	Apache Flink
Processing model	Batch SQL and DataFrames plus Structured Streaming. Micro-batch is the default trigger, and Spark 4.1.0 added first official Real-Time Mode support.	The DataStream API has STREAMING (the default) and BATCH runtime modes.
Large state	Executor JVM memory by default. The RocksDB state store provider (Spark 3.2 and later) keeps state in native memory and local disk.	HashMapStateBackend (JVM heap, the default) or EmbeddedRocksDBStateBackend, where "the amount of state that you can keep is only limited by the amount of disk space available."
Recovery	Streaming queries use checkpointing and write-ahead logs. Lost partitions are recomputed from the transformations that created them.	Checkpointing is disabled by default. Checkpoints "allow Flink to recover state and positions in the streams." A savepoint is "a consistent image of the execution state of a streaming job" for stop-and-resume, fork, or update.
Bottleneck tools	AQE coalesces post-shuffle partitions and splits skewed join tasks. The web UI Stages tab reports shuffle and GC metrics.	Back pressure is judged from output buffer availability and exposed as metrics such as backPressuredTimeMsPerSecond.
Batch caveats	One engine for batch and streaming.	In BATCH mode, "the configured state backend is ignored" and checkpointing does not work.

How does Spark cut the data it reads?

Reduce reads before adding nodes:

• Partition pruning: the PruneFileSourcePartitions optimizer rule prunes the partitions of file-source tables using partition filters.
• Dynamic partition pruning: spark.sql.optimizer.dynamicPartitionPruning.enabled defaults to true (since Spark 3.0.0) and generates a predicate for a partition column used as a join key.
• Filter push-down: spark.sql.parquet.filterPushdown defaults to true.
• AQE: on by default since Spark 3.2.0. It coalesces post-shuffle partitions, converts a sort-merge join to a broadcast hash join when runtime statistics are small enough, and splits skewed join tasks.

How does Flink handle very large state?

Flink bundles two state backends. With the RocksDB backend, in-flight data lives in a RocksDB database in the TaskManager local data directories, and it is "currently the only backend that offers incremental checkpoints." Flink does not send each record individually. It buffers records into network buffers, and a shorter buffer timeout trades throughput for latency.

How does Databricks approach petabyte-scale analytics?

Databricks runs managed Apache Spark and adds:

• Photon: "the Databricks-native vectorized query engine that accelerates your SQL workloads, DataFrame API calls, ETL pipelines, and stateless streaming workloads." It is enabled on serverless compute, SQL warehouses, and serverless Lakeflow pipelines.
• Predictive optimization: automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables (Delta Lake and Iceberg).
• Serverless compute: Databricks "automatically allocates and manages the necessary compute resources," so you run workloads "without provisioning any compute resources in your cloud account."
• Auto Loader: incrementally and efficiently processes new data files as they arrive in cloud storage, using the cloudFiles Structured Streaming source.

FAQs

Do I need special hardware for petabyte-scale analytics?

The Spark hardware provisioning guide recommends 4-8 disks per node without RAID, at least 8-16 cores per machine, and giving Spark at most three quarters of a machine's memory. It also recommends a 10 Gigabit or higher network. Size your cluster from these starting points and measure.

What is data skew and why does it matter?

The Spark docs say data skew "can severely downgrade the performance of join queries." AQE handles it in sort-merge joins by splitting, and replicating if needed, skewed tasks into roughly evenly sized tasks (spark.sql.adaptive.skewJoin.enabled, default true).

How do I choose between Spark and Flink?

Start from the workload. If most of the work is batch SQL and ETL with some streaming, Spark covers both in one engine. If the core job is long-running stateful stream processing, evaluate Flink's state backends, checkpoints, and savepoints. Prototype both on a representative slice of your data.