What is Velox, and does it replace Spark?

Summary

• Velox is an open-source C++ library, "a composable execution engine," that provides reusable data processing components: columnar vectors, vectorized expression evaluation, functions with Presto and Spark semantics, relational operators, I/O connectors and resource management.
• Velox doesn't replace Apache Spark. It has no SQL parser, dataframe layer or query optimizer, and in common usage it takes a fully optimized query plan as input, so a host engine still plans and coordinates the work.
• Velox reaches Spark through Apache Gluten, which converts Spark physical plans to Substrait, runs supported operators in a native backend such as Velox, and falls back to vanilla Spark for unsupported operators.
• Presto C++ (Prestissimo) also builds its workers on Velox, while the Java coordinator stays in place.
• On Databricks, native execution comes from Photon: a Databricks-native vectorized engine, compatible with Spark APIs, that falls back to the Spark runtime for unsupported operations.

What is Velox, and does it replace Spark?

Velox is "a composable execution engine distributed as an open source C++ library," and no, it doesn't replace Apache Spark. It covers the innermost layer of a query engine: evaluating expressions and running relational operators over columnar data. Its README is explicit that Velox "does not provide a SQL parser, a dataframe layer, or a query optimizer" and that in common usage it "takes a fully optimized query plan as input." When Velox appears in a Spark deployment, it sits behind a plugin such as Apache Gluten, which reuses "Spark's control flow, while offloading compute-intensive data processing to the native side." Your DataFrame and SQL code, Spark's planner and your cluster manager stay where they are.

What is Velox?

Velox provides reusable components for building data systems "focused on different analytical workloads, including batch, interactive, stream processing, and AI/ML." The README lists eight: Type, Vector (an Arrow-compatible columnar memory layout), Expression Eval (a fully vectorized expression evaluation engine), Functions (scalar, aggregate and window functions following Presto and Spark semantics), Operators (scans, writes, projections, filtering, grouping, ordering, shuffle, joins and more), I/O, Network Serializers and Resource Management. Velox "is usually not meant to be used directly by end-users; rather, it is mostly used by developers integrating and optimizing their compute engines." It's licensed under Apache 2.0.

Apache Spark vs Velox: how they compare

Capability	Apache Spark	Velox
What it is	"A unified analytics engine for large-scale data processing"	"A composable execution engine distributed as an open source C++ library"
Who calls it	Applications written in Python, Scala, Java, SQL or R (deprecated)	Engine developers embedding it in a host system
Parsing and optimization	Spark SQL uses information about the data's structure and the computation to optimize queries	None: no SQL parser, dataframe layer or query optimizer
Execution	An engine that "supports general execution graphs," running on the JVM	Vectorized expression evaluation and operators over Arrow-compatible columnar vectors, in C++
Cluster management	Spark Standalone, Hadoop YARN or Kubernetes	Resource primitives inside a process (memory arenas, tasks, drivers, thread pools, spilling); the host engine distributes the work
Functions	Spark SQL built-in functions	Function sets following Presto and Spark semantics
Data access	Data sources for Parquet, ORC, JSON, CSV, JDBC and more	Connectors for ORC/DWRF, Parquet and Nimble, with S3, HDFS, GCS and ABFS storage adapters

Where Velox shows up

• Apache Gluten. Gluten describes itself as "a middle layer responsible for offloading JVM-based SQL engines' execution to native engines." It turns Spark's physical plan into a Substrait plan, then into the native engine's plan, with Velox and ClickHouse as backends. Operators it can't offload fall back to vanilla Spark, with ColumnarToRow and RowToColumnar converters at the boundaries. Gluten supports Spark 3.4, 3.5, 4.0 and 4.1, and the Apache Software Foundation announced its graduation from the Incubator on March 5, 2026.
• Presto C++. Presto's native worker, "sometimes referred to by the development name Prestissimo, is a drop-in replacement for Presto workers written in C++ and based on the Velox library." Workers talk to the Java coordinator over REST and don't need a JVM.

Should you add Velox to Spark? Criteria to weigh

• Scope of the change. Gluten with Velox swaps the execution layer under Spark SQL and DataFrame workloads. It doesn't change the APIs your code uses.
• Operator coverage. Check how much of each plan runs natively. Unsupported operators fall back to vanilla Spark, with row and columnar conversions where native and JVM stages meet.
• Semantics. Velox implements Spark-semantics functions, but verify results on your own queries before switching jobs.
• Versions and operations. Gluten targets specific Spark releases, and you take on building and deploying a native library.
• Evidence. Measure on your own workloads; this page makes no performance claims.

How Databricks approaches native execution

On Databricks, native execution is built in. Photon is "the Databricks-native vectorized query engine," and it's "compatible with Apache Spark APIs, so it works with your existing code with no changes required." It runs in native C++ instead of the JVM and is on by default for serverless compute and SQL warehouses. "When Photon encounters an unsupported operation during query execution, it transparently falls back to the Spark runtime for the remainder of that operation." The docs list what it doesn't cover: UDFs, RDD APIs and Dataset APIs, and it supports stateless streaming only. See the Photon documentation.

FAQs

Is Velox a database?

No. Velox is a C++ library of execution components. Without a SQL parser, dataframe layer or optimizer, it's embedded inside engines rather than used on its own.

Can I use Velox with PySpark?

Yes, through Apache Gluten with the Velox backend. Gluten keeps DataFrame and SQL code unchanged and supports Spark 3.4, 3.5, 4.0 and 4.1.

Is Velox the same as Photon?

No. Velox is an open-source library that projects such as Gluten and Presto C++ build on. Photon is the Databricks-native vectorized engine built into Databricks compute.

Does Velox work only with Spark?

No. Its functions follow Presto and Spark semantics, its serializers support PrestoPage and Spark's UnsafeRow, and Presto C++ workers are built on it.