Sponsored by: Onehouse | From JVM Gymnastics to Vectorized Performance in One K8s Operator Deploy

Summary

This video introduces Onehouse Quanton, a native vectorized execution engine for Apache Spark built as an optimized fork of Velox, that can be deployed on an existing Spark-on-Kubernetes cluster by installing a single Kubernetes operator with no code changes or API rewrites required.
Quanton is fully compatible with the Apache Spark API and supports Delta Lake, Apache Iceberg, and Apache Hudi table formats, enabling vectorized predicate pushdown and late materialization that deliver 3x or more faster workloads and 40–60% lower cloud spend.
The deployment process uses a Helm install to drop the Kubernetes operator into an existing Spark cluster, after which it automatically reconciles SparkApplication custom resources and SparkExecutorPod resources to replace JVM execution with native vectorized performance.

Chapters

FAQs

What is Onehouse Quanton and how does it relate to Apache Spark?

Onehouse Quanton is a native vectorized execution engine for Apache Spark, built as an optimized fork of the Velox project. It replaces the JVM-based execution layer of Spark with a native columnar execution engine that leverages SIMD instructions, enabling significantly higher throughput for the same workloads without any changes to existing Spark application code.

What performance improvements does Quanton deliver?

The video cites production results of 3x or more faster workloads and 40–60% lower cloud infrastructure spend compared to standard JVM-based Spark execution. These gains come from vectorized execution, predicate pushdown, and late materialization — techniques that native columnar engines apply to dramatically reduce the data read and processed per query.

What table formats does Quanton support?

Quanton is compatible with Delta Lake, Apache Iceberg, and Apache Hudi table formats, making it applicable across a wide range of open lakehouse architectures. It operates as a vectorized execution layer on top of these open table formats, preserving all the transactional and governance benefits those formats provide.

How is Quanton deployed on a Spark-on-Kubernetes cluster?

Deployment is done via a Helm install that drops a Kubernetes operator into an existing Spark-on-Kubernetes cluster. The operator automatically reconciles SparkApplication custom resources and SparkExecutorPod resources, replacing JVM execution with native vectorized processing — with no changes to existing Spark application code, APIs, or data schemas required.

Full transcript

[00:07] Thanks everyone for coming. I I think we're up live now, so we can get started. My name is Kyle. I run product in our technical field team at Onehouse. I'm very excited to chat with each of you after the uh talk today. Come up, let's
[00:19] talk let's chat let's ask questions, all those kind of things. We can make it more interactive at that point since it's hard to reach without the microphone. But today we are talking about a special topic. Uh the title is called JVM
[00:31] gymnastics to vectorized uh performance. So we're going to talk about how Onehouse has developed a uh native vectorized uh compute engine for Apache Spark. It's called Quanton. It's a optimized fork from Velox with some
[00:44] additional details. We'll get into it as we go. But the way that I want to start the conversation here is actually to do a little bit of time travel. Let's go back in time just a little bit to about 2014.
[00:57] I don't know which companies you might have been working at at the time or maybe going to school or something like that. But in 2014 I was at Microsoft and I was reading papers. I was working on data platforms. I was using Hadoop at the time. But I was reading papers like
[01:09] this published in 2014 where it says uh Spark just passed Hadoop in popularity on the web. It's a pretty cool chart you see. Um and you see these from time to time for different technologies, but
[01:21] rewind from 2014, Spark became ubiquitous across across the industry. Now fast forward until today, this is actually a chart of some research that a firm did across data engineering job
[01:34] postings. And they looked at all the data engineering job postings and and uh compiled which technologies were required in the job requirements. And guess what shows up on top? Apache
[01:46] Spark. The number one technology or tool uh listed in uh job requirements for data engineers. So uh even now today the uh relevance of Apache Spark it's out there. It's pretty much the standard way
[01:59] for building data pipelines um that need distributed compute in the cloud today. So, now let's dive into a little bit of the stats behind how Spark is actually used. Right? Databricks has the lion's
[02:12] share majority of it and probably most of you here you're at the Databricks State of AI conference. You're probably either using Databricks, considering using it, or very comfortable, and been using it for a long time. Um after that
[02:24] uh you you see there's various different cloud providers. There's many ways to use Apache Spark, right? You can use uh from AWS EMR, from uh GCP you can use the DataProc product. Um from Azure
[02:36] they've got Fabric. Many many different organizations will package up a Spark. But actually in the number two position here you see Spark being run on Kubernetes. And so that's very popular as well. It's very portable. You can put
[02:50] it in any environment, any cloud, on prem, hybrid environments, doesn't matter, on your laptop, as you see fit. So, as this plays in, like I want to talk about two big shifts that have changed recently. Let's start from 2016
[03:03] to 2024. Um I've been working on data lakes, data platforms, even specifically lakehouses for quite a long time. I helped uh develop and launch the first version of Databricks on Azure. I was a Microsoft employee at the time, helped partner
[03:15] with Databricks. And uh we we launched Delta Lake in uh 2019. Um another thing I know someone in the audience here uh over in this corner, he's the creator of Apache Hudi,
[03:28] actually the world's first data lakehouse uh from Uber. And so, during this time between 2016 and 2024, something large that has changed from Spark. Spark's not item potent by by default, right? So, having this
[03:41] distributed compute, you need to have eventual like you've got eventual consistency in S3, you'll have partial rights that things could fail across time as it goes. But uh table formats like Apache Hoodie, Apache Iceberg,
[03:53] Delta Lake, these table formats brought transactional consistency to data lakes. And now you could actually be immune to the problems of failing rights or or workloads and have acid acid
[04:07] transactional guarantees. This also brought metadata layers on top which helped improve the performance for your pipelines as you build with Apache Spark as well. So this is very exciting. I'm sure
[04:19] you've all read through and have been through the right through table formats, table format wars, all those kind of things. But now let's talk about what's changing in perhaps the next two years. I think it will be a very short stint. Shouldn't take long for the next
[04:32] revolution to come. But the thing that I am really excited about is how many players in the industry are forking from where Spark is at tied into the JVM really tightly to now building native
[04:46] execution engines. Whether this is rewriting parts of the engine in C++ or in Rust or various other technologies that help make all your data pipelines
[04:58] and execution faster. So today I work at Onehouse and so we're going to be talking about Quonton which is our specific version of this this is a optimized fork from Velox that we've
[05:10] been working on some time and we're very proud of what we've built and developed. So the Quonton approach, if you look at this kind of like a layer cake, you have the top and the bottom of this layer cake
[05:22] that you want to keep completely open and interoperable cuz especially all of you in the audience as you're using any type of vendors product, you want to evaluate it to make sure that you can move that workload, move that code, move
[05:34] that job to anywhere else at the click of a button. So we've kept that intact as well. We are 100% compatible with the Apache Spark APIs. And on the bottom we leverage open table formats. Whether whichever is your your
[05:47] flavor of choice, whether you want Hoodie, your Iceberg, or Delta Lake, it doesn't matter to us. We can have that execution runtime and the storage engine that couple together that we package as
[05:59] Quanton, it can run on any of these table formats as well. So, let's talk a little bit about how it works. We got benchmarks under the QR code if you want to scan and see some of those details. And it also has a blog
[06:11] with in-depth descriptions that we won't even have enough time for during this session to get into. But Quanton is faster in some ways that you might be familiar with, things like SIMD,
[06:23] vectorized execution. This is where baselines like Gluten or Velox or Datafusion, these things kind of stop as well. And we've also added on top aggressive query plan optimizations, high-throughput data scanning. But we've
[06:37] also made this engine smarter, not just faster, but also smarter because we put in deep storage-aware optimizations for things like joins or filters or different updates that you need to do. We found ways to amortize execution
[06:50] costs, especially across repetitive ETL patterns. So, we can have automatic rewriting from batch to incremental operations, background metadata optimization.
[07:02] There's some really smart techniques that we put into the details you can read from the QR code in the blog. You'll get really deep into the weeds. So, how do you use it? This is actually an interesting point. Quanton, this
[07:14] engine, we've decoupled from the rest of the Onehouse platform and product. Quanton you can deploy as a standalone like acceleration engine into any Kubernetes environment. So, you see here
[07:28] on the left-hand side is where you can use the full Onehouse platform. And we have like a BYOC model where you can put a serverless data plane in your VPC on any cloud in in particular. But now
[07:40] BYOK, the Quanton on operator, you can actually deploy in any K8s. So that could be on prem, on cloud, hybrid scenarios, on your laptop, whatever you have set up and you want to accelerate. That's where
[07:53] you can pull and have the Quanta on Spark images available. Um so that's something very exciting that we've announced today. We actually have a separate website dedicated just to explaining the engine and the details
[08:06] and to help you get started you can go to quanton.dev. So there's a premium involved. You don't have to talk to any sales people. You can come and try it on your laptop while I'm still talking yapping my mouth over here. You can get a quick demo on your
[08:19] on your laptop before we even end the session. No credit card or any of that stuff needed involved. You can see in the little screenshot on the side it's really easy. It extends the Spark Kubernetes operator. So you
[08:31] can do a helm upgrade, install the operator, pulls from a registry. You put in some details from a YAML configuration. We'll look at that in a in a demo. Um yeah, so it's easy to get started. You once you create an account you'll
[08:43] have a project. You download a YAML template and then you can fill in the details on that YAML template. You can install with what I just described and then you just execute. You do a kubectl apply and then you pass in
[08:57] that configuration parameter and you're able to execute your Spark jobs. What this means you see here there's no Spark code rewrites. If you already have a job written, PySpark application or Scala or Spark SQL, whatever your job might be.
[09:10] You just come in and run a kubectl with a YAML configuration, point to the existing job or code or or what you have already defined and you're running that pipeline in an accelerated way.
[09:23] So let's look at a quick demo what that looks like. Just watching my time as well. So let me come here to the Onehouse platform. After you go to So let me let me actually step you through another uh
[09:36] here. So when you first scan that QR code, you'll come to quanton.dev and you can skip everything else inside here. Uh looks like the Wi-Fi is incredibly slow um for my demo, hopefully it's okay. Um but you can just click the purple
[09:49] button on top, create an account. This moves you right into the uh account creation page. If the Wi-Fi wants to work with me here, yeah, there you go. You just put in your email, could be your Gmail, Yahoo, aol.com, I don't care, whatever you want to put in there.
[10:02] And you're able to create an account in uh minutes. After you do that, you will have a uh the ability to create a project inside OneHouse. So, this is my project switcher. I have many projects, of
[10:15] course, because I work in the product every single day, but I'll create a project and then on the home page, you'll see the setup instructions. Um and these setup instructions, they'll have the place that you can download the
[10:27] YAML configuration file, and it will give you the step-by-step details on how you can get started and and execute the jobs. So, after you down- download that YAML, then you do the kubectl apply, you're running the
[10:40] jobs. Now, you could monitor it all from your existing system. Maybe you're running this on um EMR on EKS. Maybe you're running this on your own uh Kubernetes cluster in the
[10:52] cloud or some kind or on minicube on your laptop. You can monitor the job as you normally might, but we also allow you to inspect the jobs through our UI here. So, you can see, here's a couple jobs that I've submitted with the operator involved. And I can see for
[11:06] this particular job many different runs and whether they were successful or failed, the runtimes for these jobs. If I drill into one of these, I can also see the um detailed like manifest file, the the
[11:19] configurations that I had passed for this job. So, I can come in and remember the history. Um so, from this, you can see that there's uh not a ton of options on the left-hand side of this navigation pane.
[11:31] So, I'll come back to a demo and show you a little bit more about OneHouse cuz this is particularly the Quantone engine that we've decoupled from the rest of our platform that's now flexible for deployment in any environment.
[11:44] So, coming back here, I'll give you a um just a really fast overview of the rest of the OneHouse platform because what I um what we've
[11:56] talked about so far if this laser pointer shows up, maybe not, is the um the purple and pink queue in the middle. That's what we've been talking about this whole time. But, OneHouse, it also offers a collection of services, things
[12:09] like OneFlow, fully managed ingestion platform, uh Spark and SQL ETL. You can run your jobs um against fully serverless uh cluster Spark clusters that will scale up and down according to your needs. And we
[12:22] also have OneHouse lake base, which is an analytics and AI serving layer for your uh traditional analytics or maybe even for a genetic uh AI workloads um as you're running. We have OneSync, which is the ability to synchronize your
[12:34] metadata across many catalogs, even access control policies. You can synchronize, for example, across Lake Formation and Unity catalog. Um we also have the ability to deploy other types of open-source engines if you're ever
[12:47] interested. And to give you a fast uh tour of that, let me come back here. And I'll come to my home page. Now, you see there's a lot more options on this left-hand side because I have the full
[13:00] OneHouse environment deployed. This is the full uh managed uh data plane. And we deploy, uh by the way, I think I mentioned this inside of your VPC. So, if you're on AWS, you create a VPC, we deploy all that compute in your environment, no data ever leaves. All
[13:13] that compute stays with you. Um so, maybe something that I might want to do with OneHouse, I mentioned how we have a a ingestion platform called OneFlow, I might come in and uh create a
[13:25] new data source. So, let's say let's create a source. Maybe I want to connect to a Postgres database. I put in the connection details to that, and now I can come to what we call flows. In the flows, then I can specify that Postgres
[13:38] endpoint. I put in a couple of configurations for how fast I want it to run. Let me show you. Postgres, append only or mutable workloads, how fast I want it to run,
[13:50] what tables I'd like to ingest, and then I pick a catalog and a S3 location. I hit go. This flow is now running in the background. I'll show you one that's in progress right now. And you'll see stats and statistics
[14:03] about how that pipeline is ingesting data, how quickly it's happening. It will also give you details about your data at the source, how much data is pending to write, sync durations, those uh sort of things.
[14:15] That data then lands inside of your S3 bucket, ADLS bucket, GCP, GCS storage container, whatever it is that you've configured. If I show you an example of one of these, I want to look at Apex
[14:28] retail, bronze, retail master, I think is one that's running actively if I remember right. You'll get a bunch of statistics about your tables. Once the slow Wi-Fi here
[14:41] likes to load, it will tell you details of how large this table is, how much data is being written, um and you'll be able to do a lot of different operations on on top of it, yeah. Okay, cool.
[14:53] Um now I'll zoom you into the Spark jobs, perhaps. When you go to run Spark jobs, the first thing you might want to do is uh create a cluster. So, you create a compute cluster here. This might feel familiar. You can choose
[15:05] a cluster type, whether that's a Spark job cluster, or you want it to run a different type of open-source engine. Once you deploy this, you tell it how large or how small you want it to be.
[15:17] Um you could choose the different SKUs uh or options for your EC2 instances. And um then you create the cluster. This comes with uh uh really advanced auto scaling techniques
[15:29] you can learn about from the rest of our blogs and and other webinars that we've done in the past. After that cluster is online, I can come to jobs.
[15:49] Okay, so it's a little bit frozen here, but I think you guys get the experience. You can go into jobs, you can uh point to a PySpark job or a Scala job that is um existing already um in an S3 bucket or other ways that you've deployed and
[16:02] done your CICD workflows for the Spark jobs. Um and then you can deploy those as well. Um Yeah, that kind of takes us through the demo for the Onehouse platform. Let me come back here
[16:15] to my slides. And I'll leave you with just a couple more QR codes if you're interested.
[16:31] >> How you guys been enjoying the Wi-Fi at the conference? Has it been Yeah, so-so? If some thumbs down, all right. This is a live in action. I'll show it to you in this format. I think the QR codes are big enough to scan anyway. So, again, you can go to quanton.dev.
[16:43] Free tiers are there involved. We run it just like how ChatGPT or the others run. You get a certain amount of gigabytes that are accelerated. And then once you um run out of that free quota, then it goes back down to like Gluten or Velox
[16:56] uh uh um accelerated uh speeds. So, you'll still be fast running this even free after you've exhausted your free quota, it's still going to be faster than your vanilla open source uh Spark engines. Um
[17:10] but yeah, we also have a Slack community. If you want to join the Onehouse Slack community, uh you can chat with any of our engineers live there on Slack and get into more details. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.