I want to process data as fast as possible. What should I use to do this?

Summary

• "Fast" means different things: finishing a large batch job or SQL query sooner, processing each streaming event sooner, or reading less data per query. Pick tools for the goal you have.
• For SQL, ETL and DataFrame work, Photon, the Databricks-native vectorized query engine, is the default engine on SQL warehouses and is enabled on serverless compute. It needs no code changes.
• Data layout affects how much data a query reads. Databricks recommends liquid clustering for all new tables and predictive optimization for Unity Catalog managed tables.
• Spark tuning still applies: adaptive query execution is on by default, small tables can be broadcast in joins, and Databricks recommends automatic disk caching for repeated reads.
• For per-event latency in streaming, use Structured Streaming real-time mode. Micro-batch stays the default trigger.

I want to process data as fast as possible. What should I use to do this?

Start by naming what "fast" means, then match a capability to it. For SQL, ETL and DataFrame throughput, run on Photon using serverless compute or a SQL warehouse. To read less data per query, store tables with liquid clustering and let predictive optimization maintain them. For event-by-event latency in streaming, use real-time mode. Tune joins and caching after you have read the query plan.

What does "fast" mean for your workload?

Goal	What to use	What the docs say
Large SQL, ETL and DataFrame jobs finish sooner	Photon on serverless compute, SQL warehouses or classic compute	A native vectorized engine that processes data in columnar batches; Catalyst still plans the query, and unsupported operations fall back to Spark
Queries read less data	Liquid clustering and predictive optimization	Clustering keys taken from commonly filtered columns improve data skipping; predictive optimization runs OPTIMIZE, VACUUM and ANALYZE for you
Joins adapt to the data at run time	Adaptive query execution (AQE)	On by default; can turn a sort-merge join into a broadcast hash join, coalesce small partitions and split skewed tasks
Repeated reads of the same files	Automatic disk cache	Copies remote Parquet files, including Delta tables, to local storage on worker nodes
Each streaming event handled immediately	Real-time mode	A Structured Streaming trigger type for operational workloads; micro-batch remains the default

How does Photon speed up processing, and where does it not help?

For supported operations, Photon replaces the JVM-based Spark SQL execution engine with a native C++ runtime, and it processes data in columnar batches instead of row by row. It is the default engine for all SQL warehouses and is enabled on serverless compute. Databricks also enables it by default on classic all-purpose compute, jobs compute and classic Lakeflow Pipelines. The docs say Photon gives the greatest benefit to longer-running queries over large datasets, so very short queries see little improvement. It does not support UDFs, RDD APIs or Dataset APIs, and it does not support stateful streaming.

How should I lay out the data?

Databricks recommends liquid clustering for all new tables, and it is generally available for Delta Lake tables. It replaces table partitioning and ZORDER, and you can change clustering keys without rewriting existing data. Choose keys from the columns most commonly used in query filters, or let Databricks choose with automatic liquid clustering, which relies on predictive optimization and applies to Unity Catalog managed tables.

CREATE TABLE table1 (col0 INT, col1 STRING) CLUSTER BY (col0);
ALTER TABLE table1 CLUSTER BY AUTO;

The first statement sets a manual key. The second hands key selection to Databricks. Predictive optimization also runs ANALYZE, which collects the table statistics the optimizer depends on to pick good plans.

When should I broadcast a table or cache data?

Spark broadcasts a join input up to spark.sql.autoBroadcastJoinThreshold in size, which defaults to 10485760 bytes (10 MB). You can request a broadcast yourself with the BROADCAST hint, as in SELECT /*+ BROADCAST(r) */ * FROM src s JOIN records r ON s.key = r.key, or with pyspark.sql.functions.broadcast.
For repeated reads, Databricks recommends automatic disk caching, which needs no code changes. The Apache Spark cache (dataFrame.cache() or spark.catalog.cacheTable) is manual, and DataFrame.cache() uses the storage level MEMORY_AND_DISK_DESER by default.

How do I know which change is helping?

Read the plan before and after each change. In the Spark UI, Photon operators appear in orange in the query DAG and standard Spark operators appear in blue. In SQL warehouses, the query profile's Execution Details view shows the percentage of task time spent in Photon. For adaptive query execution, the Spark UI shows AdaptiveSparkPlan nodes whose diagram evolves as the query runs, and DataFrame.explain() shows the initial plan alongside the current or final plan.

FAQs

Should I always use the largest compute?

Not necessarily. On classic compute you can enable autoscaling and set a minimum and maximum number of workers, and Databricks chooses the number of workers within that range. Serverless compute lets Databricks allocate and manage the compute resources, which the docs say speeds up start-up and scaling times. Compare run times as you resize.

Does real-time mode make batch jobs faster?

No. Real-time mode is a trigger type for Structured Streaming that targets ultra-low latency for operational workloads such as fraud detection. The docs point to micro-batch, the default trigger, for analytical processing such as ETL and for cost-sensitive workloads that do not need sub-second latency, because real-time mode requires dedicated compute resources. It runs on classic compute with Photon and autoscaling turned off.

Should I use Spark caching or the disk cache?

Prefer the automatic disk cache for repeated reads of Parquet and Delta data. Databricks recommends it, it fills automatically on first read, and it detects file changes. Use Spark caching to reuse a DataFrame result, and call unpersist() when you are done.