Delta and Databricks as a Performant Exabyte-Scale Application Backend

Summary

• Real-Time Application Backend: Delta Lake and Databricks Photon power Capital One's interactive security information and event management (SIEM) backend for cost-effective real-time queries.
• Data Skipping Optimization: Databricks' optimization model and data-skipping techniques deliver performant queries across exabyte-scale data, enabling interactive response times for security analysis.
• Lakehouse Scale and Functionality: Unified data platform provides far greater functionality than traditional NoSQL solutions while maintaining performance and cost efficiency at massive scale.

Delta and Databricks as a Performant Exabyte-Scale Application Backend

Talk by Scott Schenkein, VP, Distinguished Engineer, Capital One Financial
The Delta Lake architecture promises to provide a single, highly functional, and high-scale copy of data that can be leveraged by a variety of tools to satisfy a broad range of use cases. To date, most use cases have focused on interactive data warehousing, ETL, model training, and streaming. Real-time access is generally delegated to costly and sometimes difficult-to-scale NoSQL, indexed storage, and domain-specific specialty solutions, which provide limited functionality compared to Spark on Delta Lake. In this video, we will explore the Delta data-skipping and optimization model and discuss how Capital One leveraged it along with Databricks photon and Spark Connect to implement a real-time web application backend. We’ll share how we built a highly-functional and performant security information and event management user experience (SIEM UX) that is cost effective.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Capital One's use case for Delta and Databricks?

Capital One built a real-time web application backend for security information and event management (SIEM) using Delta Lake and Databricks. This provides highly functional and performant security event analysis at scale and cost-effectively.

Why is Delta Lake suitable for application backends?

Delta Lake's data-skipping and optimization capabilities enable fast queries on massive datasets. Combined with Databricks Photon acceleration, this makes interactive response times possible—something typically reserved for costly specialized systems.

What's the most cost-effective way to process 1 TB of data with Spark?

Databricks with Delta Lake and Photon optimization provides cost-effective Spark processing through intelligent data-skipping that avoids scanning unnecessary data. Capital One achieved this by leveraging Delta's optimization model and Spark Connect, reducing infrastructure costs compared to traditional NoSQL or search engine solutions while maintaining high performance on interactive workloads.

What's the cheapest way to process big data (Spark versus alternatives)?

Spark on Delta with Databricks Photon delivers the lowest total cost of ownership for exabyte-scale data by consolidating multiple specialized systems into one unified platform. Capital One's experience shows that a single lakehouse replaces expensive search engines and NoSQL systems, eliminating costly licensing, duplicative infrastructure, and human overhead—achieving fast, big, and cheap simultaneously.

What performance improvements did Capital One achieve?

By leveraging Databricks Data and AI platform and Photon, Capital One created a performant SIEM backend that's more cost-effective than traditional NoSQL, indexed storage, and specialty solutions while providing greater functionality.

Full transcript

Well, hello. Uh, thanks for spending your precious conference time with me, uh, learning about how Capital One's security organization found a way to use Delta and data bricks, uh, to power a scale backend for our interactive security event analysis application or for the security folks, our SIMS UI, right? Um. I'm currently a distinguished engineer at Capital One. I'm the lead engineer for our cyber security organization and I sponsor a lot of work that we do across the enterprise for advancing data engineering causes.
That's sort of how I got involved with all this, right? Um, for most of my career, I've been very heavily attracted to systems optimization problems. Um, and I've always had a bit of an obsession trying to get like the last little bit out of every compute workload possible and get stuff for the cheapest way possible. So, um, that's how I came to be part of the agenda that we'll talk about today. Um, all right. So, in today's talk, we're going to discuss some tradeoffs that I think most of us generally make around scale, cost, and performance when working in a lakehouse architecture.
Um then we'll pivot to talk a bit about Capital 1's journey to adopt a lakehouse architecture along with some of the limitations that we encountered. Uh and then we'll talk about how we overcame those limitations which is the interesting part of the talk. Um and do a deeper dive into the technical inner workings um that allowed us to achieve some uh pretty improbable goals. All right. So my first introduction in our cyber lake ecosystem was in a call with our finance team of all people right uh our cyber data system was one of the most expensive applications in the company and we were operating way outside of our finance uh team's assumptions for spend right the team of security experts that had built our system uh felt that these very high costs were justified and necessary given that we had requirements to satisfy around high productivity ity for our investigators, very fast access um and to have data that was ready to consume in case we needed to do like a deep look back or retrospective analysis of old data um should we have an incident or something similar.
Um so it was like big and expensive the system. Um I started to think about a system of tradeoffs in data that is very similar to the triple constraint that you hear about like in project management. Um, and in this triple constraint, the axises are high performance, scale, relatively low cost. Um, and among those, generally speaking, you can only have two. And I let this become a bit of a crutch for myself. And I've observed fairly ubiquitously like throughout the industry, this is how people approach the problem.
They like sort of take it for granted that they can't get all three. Um so about six years into my connection to uh Capital 1's cyber data ecosystem um I came to meet a friend who had a very similar professional focus to mine and he chatted to me about how much lift his security organization got from adopting um the Delta Lake ecosystem right data bricks and Delta they solved for very inexpensive data at scale and they consolidated streaming batch Hatch interactive warehousing um all using a single data source and at the time we were using numerous technologies to do the same um and we had less capabilities than they did and I always have my hat on like I'm this forever quest to try to um make my system simpler and less expensive.
So after a ton of prototyping, I personally got convinced that this was the right uh approach for my team and we chartered work to consolidate to a lakehouse architecture using Spark, Delta, data bricks. Um we found that we could replace a meaningful portion of our infrastructure using a single inexpensive copy of data, right? The lakehouse promise. uh which provided really a lot more capability than the litany of more specialized cyber security tools that we were using um up to that point.
And this system had in that triple constraint was big and it was cheap, right? Um and it was really a great transition for our team. It's like that's a little bit too much to talk about in a lightning talk, but um I'll leave it at it was the right architecture decision for us and it was one of the best system investments I've ever been part of. Um but there's a complication to all this. Um though we had driven a lot of convergence around Delta and around our single copy of data, we still had a separate system, a separate copy of our data, a separate load process for our fast interactive data uh use cases, right?
And this system was fast and big, right? Remember fast, big, cheap, you get too um but definitely not cheap. Uh our analysts were still using this very expensive search engine. You can probably if you're a cyber person figure out what I'm talking about. Um to retrieve and pivot through our data. And this was costly from infrastructure standpoint. Licenses were expensive. And most importantly to me, it was a huge draw on our human um component, right?
Like our resources were spending a lot of time doing duplicative work that generally just didn't feel good. Um, so I challenged myself and my team to try to figure out if there was a way that we could break through the triple constraint and serve fast scale use cases using all the relatively inexpensive tools that we got from the lakehouse architecture and from Delta and data bricks. Um, the paved path so to speak for making the lakehouse work at scale for low latency interactive use cases weren't particularly great.
Um, one option you have is to use gigantic Spark clusters. If you have enough data and it's big enough, you can essentially throw hardware at the problem and indefinitely parallelize your work uh, until you get the performance that you need. And while this is a lot less moving parts and was less of a burden on the people on my team, um, it really wasn't any more cost-effective than having an actual search engine. Second option would be doing something a little more creative like hyper segmenting your data.
Think about like maybe a table for every person in the organization or every server. Um and again this is not a great option given the enormous support burden and the design and technical complexity that's involved. Imagine hundreds of thousands of tables be very difficult to maintain. So uh clearly the conventional thinking around this problem wasn't going to get us to that goal of having fast, big and cheap, right?
All using one very simple set of systems. So to move forward, we had to get a much deeper understanding of the fundamental problem that we were solving. number one. Um, and number two, we needed to understand the technology that we're using much more deeply than the way that people in the industry generally learn about how Delta and Data Bricks and Spark worked. Um, so our first order of business was to get a much deeper understanding of what our users were actually doing, right?
Our the people in our security operations center, they didn't work at random. So, there had to be some sort of pattern as to what they were doing. um and patterns are generally very useful in computing and in data problems. So the first thing we noted was that nearly every query that our users ran during an investigation revolved around some sort of subject, right? Like a noun. Think of uh cyber security subjects being things like people, hosts, cloud accounts, networks, appliances, and so forth, right?
Um these are high cardality things uh which we'll look at much more deeply in a few minutes but you can think if you're not familiar with it high cardality being something you have a lot of like at Capital One we have thousands of employees and we have hundreds of thousands of hosts or even millions of hosts if we look at a long enough time frame. Um the second thing that we noted was that our users generally were working in a relatively small time frame.
So while we had pabytes and pabytes and months and months of data, the active portion that most people were working on was actually pretty small. Um, and as an added bonus, a day's investigations, we were typically looking at something that happened like, you know, an hour ago. And so they were all like looking at the same small window. um these time frames like hours and days for those who haven't heard these words before they're they're low cardality meaning like you have a relatively small number of them we have millions of servers tens of thousands of users but we don't have but so many days or hours right um and these low cardality fields are extremely useful for zooming into a very large data set and we'll see a lot of detail about like how how you can think about that and visualize the actual data that underpins.
All right. So, bringing together these ideas of a consistency of query pattern um which I think applies to far beyond cyber security, right? Like anything that deals with high human productivity like call centers, right? Customer servicing, um fraud investigations, we're able to leverage some basic capabilities in the Delta stack to make things faster, right? So, the first thing that we're going to talk about is data skipping.
And you've probably seen like in Spark UI something it talks about data skipping and some things in documentation. We're going to go pretty deep. Um data skipping is one of the simplest truly um in in my experience most poorly understood parts of open table technology. Skipping will allow a query engine like Spark um to eliminate unnecessary work completely skipping the processing in question. Right? So if you can prove that a set of files definitely isn't part of your investigation, you don't have to process them and you don't have to spend the time doing that.
Um but most of our data in the wild gets ordered organically based on uh the order that it came in from a source. Um and in these cases you would only get effective skipping by accident. And generally you will get a lift from like this accidental organic sort of skipping but it kind of looks like the picture in the slide, right? like 15ish% um market but not enough to like get us to super high performance search engine sort of speeds.
Uh fortunately the insights that we talked about before would allow us to lay out our files um and colllocate our data in a way that data that's likely to be queried together is very uh approximate very close um in our case is again among time frames and subject. And so we can force our files into a format that'll allow Spark to skip the vast majority of potential files and compute work and really shave off a huge amount of time.
So you look at the diagram, you can see like the difference between an organically laid out data set and a well-laidout data set. And you imagine that if you're only processing two files, it's less work, it's less time, it's much faster. And we'll dig much deeper technically into how that works in a moment. Um, so relaying out our files using delta optimization techniques reduced our query latency from hours or minutes to a few seconds, but it's still too slow.
An interactive application has to be 2 seconds or less or human user will perceive very poor performance. Um so capitalizing on the fact that most of our users would be doing investigations around a generally similar time frame um using data caching became a very attractive option for accelerating workloads. Fortunately, in data bicks, not in open source spark, but in data bicks, they have a feature called DBFS cache, um, that will shortcircuit some of the most expensive data operations like pulling files remotely from S3 or something similar.
Um, and it stores it locally on NVMe storage on the server that's actually going to read the file. Um, and this took another few seconds off our average query. And we're getting pretty close, but it's still too slow for our very finicky uh requirements from our users. So, finally, as a data bricks user, you're probably familiar with this concept of photon acceleration. And most people think like, I turn on Photon, it goes faster.
There's a little more to it than that, though. So, Photon does a couple of very critical things which close the last part of our performance gap. Um, first, Photon will aggressively prefetch data that you're running in your query so that it ends up in that fast NVMe storage on your server before the server even tries to access it. And so, like essentially zero latency. Um, and secondly, Photon employs some very interesting techniques to shortcut uh processing activities for files that couldn't be skipped entirely.
Um, often eliminating entire files after only downloading a small fragment. Right? So more and each of these things cumulatively um adds performance. Uh once we implemented all three of these techniques, we found that query times came down to just a couple of seconds, which was our goal. Um this meant that we were getting essentially the same performance from Delta and data bricks as from our very expensive bespoke uh search engine. um leveraging just the same data, the same copy that we were using for data warehouse, batch processing and streaming.
So you imagine less is less is better, less work, it's cheaper. Um so our last hurdle was to figure out how to get these very quick runtimes um surfaced for an interactive web application rather than in a data bricks notebook. And fortunately at the time there was a new technology called Spark Connect that is a little more ubiquitous now. Um but it allows you to run a spark driver in a process that's external to your data bicks cluster.
Um data bicks releases their own optimized versions that they call data bicks connect. Um but we can use this technology to embed in an API implementation and it exposes those pabytes and pabytes of tables through for us through a restful interface um that we can consume from application frontends. Right? So we have this very fast thing wired into a web application. When I saw the results, I was very very skeptical, right?
Like no one talks about delta at massive scale with hundreds of pedabytes of data being a couple seconds. So I was positive that I messed something up in my tests. And I did ran them over and over again. Got the exact same results every time. So in order to get comfortable that these performance results were real, I had to really understand what was happening under the covers in our lakehouse implementation. Um so we'll spend our last few minutes um doing a whirlwind tour of the internals of the stack.
All right. So the delta log uh is the foundation or a log is the foundation of all open table formats whether it's delta, iceberg, hoodie, everybody has a transaction log. You've probably seen a delta log directory in your own structure. You may or may not know what it really does, but in very simple terms, you can think of it as a set of operations, add a file, replace a file, and so forth that can be replayed during query execution to know what to read in the table and how to interpret the results.
Since stark contrast to the traditional way of listing files in a remote S3 bucket or something, um the delta log gives you a index of pointers that tell Spark in very concrete terms what to do to traverse a set of um data. Um and the result is better cachability and more most importantly space to provide performance hints. This is how foundationally this solution works. So if you were to crack open some of the files um that live in the delta log, they would look like what you see on the screen, right?
So this is a sample of an ad operation like somebody wrote a file and then there's some summary information about the file like what partition is it part of what range of values are contained in that file. Right? So in this file um you can see it contains minmax able to sally right. So alphabetically it contains values and first name from able to sa to sally. So if your wear clause had zachary in it or xander the query planner could just eliminate file name one. parquet and just not do that work.
Um and these skipping operations again account for the majority of our acceleration. As mentioned before data's natural ordering and parquet files rarely provide superefficient skipping. So in this example you see three parquet files each of which contains a range of values and so when the wear clause looks for seven it has to read everything. Um but with optimization and rewriting we can compact everything toward like two common values and so in order to get seven you only have to look at one file and it's one-third as much work in this example.
Um rewriting files in a read path optimized way you have a few practical options. We don't have a lot of time to go into the details um but generally there's three options available. One is what we use. It's the original delta clustering technology called zordering. Um where you provide those high and low cardality fields and it rewrites your files without changing the contents of the table, right? It's all conveniently available through a SQL interface.
Second option is liquid clustering, which you probably heard a bunch about this week. And this is a little bit more flexible. It's newer. I wasn't able to get exactly the same performance out of it, but um this is where most of the investment is in the stack today. And it's getting better over time. And so it's definitely worth spending some time investigating whether it's right for your use case. And then finally, you can like sort your files and write them manually.
And I I don't recommend this in most cases because like there's all sorts of problems that you have to deal with around like consistency of table data over time and so forth. Um but it's available right and like it sort of demonstrates that's how the files are getting ordered and compacted. Okay. So recall high and low cardality um which provide coarse and fine grain elimination of files. So here you see a scan job that has bad skipping and it scans and hits everything. it's very lengthy and slow.
Um, so here we use high cardality fields only. And you notice it's a very small number of files and this sort of takes away the option that you have of somewhat scaling your Spark cluster out because there isn't enough work to do. There isn't enough work to parallelize. Um, and so you're robbing yourself of like some of your last mile uh performance. But if you go by a high cardality and a low cardality field, you get a relatively small number of files, but potentially enough files to scale across a medium-sized Spark cluster.
And so this for us is sort of the happy medium for performance. All right. So when rangebased skipping can't eliminate a file, Photon comes to the rescue, right? Um first there's the um proactive file elimination that we talked about, right? So Photon can look into the physical encoding of a parquet file and make some shortcuts to figure out that like oh although this is in range I don't see the value in this file. So we can just skip the rest of the file and that takes off a huge amount of time.
Um secondly, there's a thing a concept in Spark called predicate pushdown where the wear clause gets enforced when it's when Spark is reading files not actually in the Spark engine. And so it saves copying me or data into memory for later processing. And that saves a ton of time and so in Photon there's a much more robust implementation of predicate push down. And so you get way less rows for Spark to actually deal with.
And then finally, Photon uses CPU instructions that can process many pieces of data at once, very similar to how like a GPU works for an ML workload. Um, and so this makes it so you can do like hundreds of comparisons in one CPU cycle. And so for the few rows that are left after all these things happen, um, it's highly parallel using just the same hardware that you have in your data center, just part of the Intel or the ARM instruction set.
All right, caching uh is the last of our accelerants and this is where data brick stores fragments of parquet files that underly your physical tables again on fast storage on MVME on your servers. Um and again this is particularly effective when data bricks prefetches files. Um and so less network time means less time means better performance. And it's very easy to enable. just choose an instance type when you're provisioning a cluster um that's denoted as delta cache accelerated and so that's all instance types generally that have instance store and VME storage.
So when you combine all three all four of these techniques, you go from in this case we had like a 250 second query that went down to 46 seconds, right? So like a 400x increase in performance. A lot of the stuff you see with liquid clustering was so you get a 10x. If you're very careful and thoughtful about your high and low cardality fields, you can actually get down into tiny fractions of a second like we've done. All right.
So putting all these building blocks together made it possible for us to build this vastly simplified infrastructure using low cost and low complexity components and we got I think is great performance46 seconds instead of 250. Um and I feel like we've effectively defied that triple constraint. So maybe that's not a real thing, right? Um our final architecture looks like this. Kind of like the challenge slide. Um, it's simple, it's easy, it's cost- effective, it's highly performant, and everything's built around just a single copy of data, right, that we can use for warehousing, batch streaming, and fast interactive access.
No OOLTP databases, no copies of anything. Um, and this is what for the most part we have in production today. So, really proud of that. All right, so just a quick recap. Today we demonstrated that very fast inexpensive application data access is possible with Delta and data bricks. Um it's accomplished using data skipping enhancement through clustering, caching um and platform native processing accelerators. All right.
Um please com uh complete your surveys and thank you for uh spending time with me today. I'm I don't know if there's uh people in this room after this, but I'm happy to stick after and answer some questions.

Learn more about the Databricks Data and AI platform.