1000x Faster Retrieval: IndexTables for Lakehouse Search

Summary

IndexTables is an open-source indexing layer built on Tantivy and Quickwit that adds fast point retrieval, full-text search, and filtered aggregation to Delta Lake and Iceberg tables without requiring data migration or abandoning the lakehouse foundation.
Benchmark tests demonstrate up to 1,700x performance improvement over plain Spark, reducing a full-scan query over 70 billion rows from 6,800 seconds to four seconds when NVMe cache pre-warming is combined with IndexTables.
Capital One uses IndexTables in companion mode alongside existing lakehouse tables to power investigative security workflows where analysts need to search, pivot, and sweep through event data in seconds rather than waiting tens of minutes for full table scans.

1000x Faster Retrieval: IndexTables for Lakehouse Search

Delta Lake and Iceberg excel at large-scale analytics but are not optimized for sub-second point lookups or full-text search. In operational workloads like security analytics, log investigation, and telemetry, retrieval queries requiring full scans or high-cardinality filtering take minutes or hours when seconds or milliseconds are required. This misalignment between analytics-optimized lakehouse architecture and retrieval-heavy workloads creates a significant productivity bottleneck.
Learn how IndexTables, an embedded open-source indexing layer built on Tantivy, enables up to 1000x faster query performance while preserving the Lakehouse model. Discover object-storage-hosted indexes with ACID transactions, millisecond-latency aggregations over billions of rows, and NVMe-backed caching with proactive pre-warming. See how Capital One uses IndexTables in companion mode to complement Delta and Iceberg, enabling fast, flexible retrieval for investigative workflows and AI-driven security operations without abandoning the lakehouse foundation.
🤝

Chapters

FAQs

What is IndexTables and how does it work?

IndexTables is an open-source indexing layer embedded in the lakehouse that adds Tantivy-based inverted indexes alongside Delta Lake and Iceberg tables without requiring any data migration. It uses object-storage-hosted indexes with ACID transactions and intelligent range reads to deliver millisecond-latency queries over billions of rows, as described by its author Scott Shanklin in this video.

Why is the standard lakehouse slow for retrieval workloads like security analytics?

Delta Lake and Iceberg are optimized for large-scale analytical scans and throughput, not for the sub-second point lookups and full-text searches that security and log investigation workflows require. This workload mismatch means a simple query to find all events for a specific host can take thousands of seconds on scan-based lakehouse architectures, as illustrated in this video.

How much faster is IndexTables compared to plain Spark on large datasets?

Benchmarks in this video show plain Spark completing a query over 70 billion rows in 6,800 seconds, while IndexTables with an index query reduces this to 39 seconds. With NVMe-backed cache pre-warming applied, the same query completes in four seconds—a 1,700x improvement over the baseline.

How does Capital One use IndexTables for security operations?

Capital One's security engineering team uses IndexTables in companion mode to complement existing Delta and Iceberg tables, enabling the search-pivot-sweep investigation pattern that security analysts follow without rebuilding the lakehouse foundation. The system is designed to support AI-driven security operations where time-to-first-answer must be measured in seconds rather than minutes.

Full transcript

[00:09] Hi everybody. Um thank you for joining me today. I'll introduce myself. My name's Scott Shanklin. Uh I'm Capital One's lead security engineer. Um and in my spare hobbying time, um I'm the author and maintainer, or the original author and maintainer of Index
[00:25] Tables, which is an open-source project that we'll talk about um at some point today. Uh and I'm also a contributor of uh to the Datafusion Comet project, which is a native accelerator for Spark. Um we're about to bring Delta support to it. I'm
[00:40] pretty excited about that. So, I thought I'd use this as like a shameless plug. Um but thank you for uh spending your last day of this conference with me in the morning. All right. So, let's get to it. All right. So, every analyst
[00:58] has run a simple query that looks like this. Innocuous, right?
[01:14] Started considering whether they've made good life choices or not. And then finally they got a very satisfying answer. All right. So, I'd bet that anyone who's worked with lakehouse data for tech
[01:29] or security operations has experienced this, right? And over time, a lot of us have learned to accept uh this frustration, right? But the most infuriating part to me is that nothing's technically broken here.
[01:46] The lakehouse, it's doing exactly what it was designed to do. It's doing large-scale analytical processing on enormous data sets, and it's doing it at a cost profile that some of us older folks like never would have imagined. Um Spark, Delta, Iceberg, Parquet,
[02:02] they're all incredibly uh successful technologies, but they're optimized for throughput, for scans, for large sequential reads, and for analytical workloads. Um so, this example that we went through is not a failure of the lakehouse. The lakehouse is doing a great job. It's
[02:19] really just what I consider a workload mismatch. Um but the moment we move from analytics to fast retrieval, the shape of the workload changes. Um now we're asking we're not asking summarize this huge data set, we're
[02:36] asking find this one thing, or show me every event for a particular host. Uh search this message body, let me pivot quickly from one clue to the next, and those are retrieval questions. They care less about total throughput, and they care about time to first
[02:51] answer, right? Not 25 minutes, a few seconds. Um and that's where the scan-based lakehouse execution that we live with today uh starts to feel really painful. Um to make it worse, uh our operations and cyber investigations
[03:08] teams rarely run one query. They search, they pivot and sweep, um and with each step potentially taking minutes or hours instead of seconds, um imagine hunting for a specific process name or an IP address across 6 months of
[03:23] telemetry, um and then pivoting to find related events or trends. The pain, the frustration, it's real. Um and so is the likelihood that a challenging situation is going to get measurably worse um while you're going through all of your
[03:38] data. Also, I think most importantly, in real work, the odds of staying uninterrupted for hours while waiting on your results is basically zero. You're going to multitask, you're going to get pulled in the meetings, Um, you're going to lose the thread of
[03:53] your investigation. Uh, but the insight's still there, but your flow's gone, right? And that's like the real cost that you're dealing with. Slow retrieval isn't just a waste of time and money and effort. It breaks your concentration, it drains your momentum, and it really limits what you're willing
[04:09] to explore. Um, so before we go into the solution to all of this, I want to be clear about like what we're discussing today. Typically you get like a native accelerator from Spark or something and you're talking about like a percentage
[04:24] increase, maybe two or three times faster. Um, today we're not going to talk about making things a little bit faster. We're not going to talk about making scans a little better for our analysts. We're going to talk about drastic change, right? Moving things
[04:40] from minutes or even hours down to seconds, right? Um, so let's get to it. So let's start by talking about why this has become the reality for so many Lakehouse users.
[04:57] In the lake, we've been missing three very critical capabilities, um, that would make these use cases a lot better. First is fast point retrieval, right? This is like the classic needle in a haystack problem where you need to find one tiny set of rows in some gigantic
[05:13] data set, right? Big log data set, ugly log for J logs that have tons of junk in them and need to find like the one identifying thing for the error that you're looking for. Um, secondly, and most important for cyber and tech ops log retrieval, the
[05:29] things that I spend my time in, um, is the ability to search for a blob of text within a big string, like in a big message field, finding um, a particular command line, like a particular PowerShell flag or something.
[05:46] Um, and then finally is a concept of like fast filtered aggregation. Um in like in today's world even a simple row count can be very slow and expensive. Like maybe minutes or or even hours in some cases
[06:01] um when it has to scan a huge pool of data and it gets especially bad when it gets compounded with one of the other two. Like especially the one in the middle where you're doing pattern searches that are slow and expensive. And generally speaking scans are the problem. And scans the foundation of
[06:18] everything that we're doing in Databricks and in Spark and uh Presto and similar technologies. Practically all the lake formats parquet, delta, iceberg, and hoodie, they're all built around scan based execution. Um in a scan based architecture, this took me a couple years to understand.
[06:35] Uh like it has to retrieve every row of data. It has to linearly traverse them, get them from S3. Um and find like the couple of rows that matter for your query. It does a ton of work with very little output. Um but indexes work differently. Um they can let your engine retrieve a
[06:51] very small set of rows very quickly. Um and so identifying and retrieving rows becomes much simpler. And if you're talking about the right kind of workload, like most of what we do in cyber and most of what you do in tech ops, it can be much faster. Um but general purpose indexes have
[07:08] never really been available in large scale data lakes. They weren't in Hadoop, they're not in any of the Spark um formats up until recently. Um and that's made scanning the default uh fallback even when you're doing retrieval. You're just trying to grab a needle in a haystack, so to speak.
[07:23] Um now there's three common approaches that people use today to try to make scans less painful. Um we'll talk a little bit about each. Bloom filters, partitioning and skipping, clustering and Z-ordering. And to be clear, these absolutely help. Like they're they're fantastic
[07:39] innovative technologies. Last year at this uh summit I gave a talk about how we're using the three of them together to get real-time performance out of Spark at Capital One. Um but what all three approaches have in common is that they give query engine
[07:55] shortcuts. They reduce the amount of time um that is taken for data scanning, but the key point is they're still scan optimization. So, they're still dealing with this retrieve tons more data than you actually need. Um so, Bloom filters,
[08:11] they work really well for extremely selective lookups, um such as finding like a single row by a unique ID, but selectivity drops in more realistic um operational kinds of workloads, and they degrade quickly. For example, like show me all the activity on this host across tens of thousands of
[08:27] machines, it stops being a clean query very quickly because your data gets distributed all throughout um the physical files that sit under your Delta or your Iceberg table. Um partitioning can be incredibly effective, especially when you're dealing with time series data like cyber
[08:43] people do. Um and it can eliminate huge ranges of files immediately. So, if you have a table with a billion files in it, it might get you down to a couple of million. Um but even at that, like there's potentially a ton of data left to scan.
[08:59] Then you layer on clustering and Z-ordering, like liquid clustering, uh to physically colocate data, and this can improve performance substantially, um but it comes with a bunch of tradeoffs. So, like they can limit you in the number of dimensions that you can efficiently query on.
[09:16] If you're not using uh something like managed uh liquid partitioning, your teams have to spend an incredible amount of time like making sure that they're keeping up with the way that people are using their software, and when something new is discovered, it's like an engineering effort. Um but the good news is that the
[09:34] ecosystem's been moving towards like the broader ecosystem towards an answer for multiple directions. There's some great search systems that work out of object storage similar to the Delta Lake um lakehouse formats over the last 8 years or so have evolved transactional
[09:49] metadata. So like at least we know where all the data is when we're doing a query um in our native execution engines have found ways to plug directly into Spark. This is where we'll get much deeper. All right. So what if the lakehouse could answer retrieval queries the same
[10:04] way that databases and search engines do? Like databases from the '80s with indexes, right? And not scanning everything. Um capabilities finally arriving in the lakehouse ecosystem. Indexes can live alongside your data and object storage and they can provide low
[10:20] latency retrieval uh that operational workloads need. Like for investigation, security analytics, telemetry exploration, and agent driven systems, right? Um you might have seen some announcements from the security team at Databricks about
[10:37] new indexing capabilities coming to Delta and Unity Catalog for uh managed tables. And I'm really excited about that. Like I realize it kind of competes with uh the thing that I dreamed up, but like my goal was make it so that people's lives are better, right? And so it's really
[10:52] about the the technology, right? So today I'm going to talk to you about the open source implementation that I built. And I think the concepts will be useful whether you use my software or you use Databricks's or whoever's, right? All right. So index tables, that's my invention. It's a first-class table
[11:09] format that's designed for retrieval uh workloads in the lakehouse. It looks a lot like Delta and Iceberg. It works with inexpensive object storage and it executes inside of Spark. And there's no separate serving cluster. There's no like Elasticsearch thing sitting on the side. Uh and it's free.
[11:26] It's open source and you can see the software and understand how it works. Um and it has a special capability called companion mode that will allow you to index your existing Iceberg and Delta and parquet tables and get very very fast um
[11:43] retrieval without taking on a redundant secondary uh set of data, a second governance domain, and so forth. All right. I want to pause on something that's important because it's easy to miss what
[11:58] changed, right? The industry, like I said, has plenty of fast search and retrieval systems, right? Lots of the SIM vendors have them, Elastic Search, Splunk. That's not new. Um what's new is that it's no longer separate,
[12:14] right? It's part of Spark, it's part of Databricks, it's part of wherever it is that you're running your uh analytical workloads. So, historically, the patterns look like this, right? Analytics lived in the lakehouse, fast index retrieval lived somewhere else, and we had to copy data in mass
[12:31] around between our engines. And then we had to deal with like all the purging and we kind of looked the other way when it came to governance and so forth, right? Um so, what changes the execution model, right? Retriever retrieval becomes a
[12:47] first-class capability inside the lakehouse architecture. The same object storage, right? S3, same governance domain, like Unity Catalog, uh the same Spark environment, and the same source of truth, right? There's no a second copy of your Iceberg table or your Delta table. Um
[13:04] and this isn't about replacing the lakehouse with a search engine, which is what many you um secure legacy security vendors encourage. It's about bringing like these capabilities inside the lakehouse that we've all gotten very accustomed to working with. All right. So, to understand how this
[13:21] works, we need to separate two ideas that often get conflated. That's where the data lives, in S3. Um and how the queries physically executed. So, Index Tables keeps the Lakehouse assumptions, like I said, object storage, open file, Spark table
[13:36] metadata. Um but it changes the retrieval path by adding index access structures beside the data. All right, like almost any complex piece of uh software, Index Tables builds on top of a lot of a lot of open source work. Uh and here, I think the most
[13:52] important credit goes to the Quickwit project, which is a relatively obscure search engine, but it's a um object storage cloud first call it implementation of something similar to OpenSearch or Elasticsearch. But, in its uh bowels, it has this
[14:10] concept of a split format, right? Which is like a file that lives in object storage in S3, um very similar to Parquet, but instead of being optimized for columnar access, right? It's it's for indexes. And so, it's like
[14:26] a very special file format that allows you to access gigantic remote indexes for huge amounts of data without taking it out of S3 and without incurring huge amounts of latency. Right? Secondly, talk about Delta and Iceberg, like I got a lot of inspiration
[14:44] and a lot of implementation tips from their source code. And so, they brought this idea of having a database, right? To to uh open storage in the Spark. And so, like I'd I'd be remiss to not mention that like I'm I'm standing on the shoulders of those giants as well. Um and then in the
[15:01] implementation, Index Tables itself is is built using Rust code, right? It uses it allocates and deallocates an enormous amount of memory, and so doing that in Scala or Java won't work. But so, Arrow provides a bridge that lets our Java
[15:17] Scala base Spark implementation coexist with native code without having a lot of round tripping and wasted time and effort, right? Because our goal is to be fast. All right. So, you bring them all together. Index makes your discovery fast.
[15:33] But, this is really only half the performance story. Right? So, once we know which rows match, we still have to materialize those rows back from the original table. That's what takes really long time when you're scanning. Like, when you get an hour-long query, it's pulling stuff out of S3 or whatever and linearly traversing every row.
[15:50] Um So, if we fell back to our broad parquet scan, we'd lose most of the edge that we've gotten, right? And you would see maybe like an hour query go comes down to 10 minutes, which is awesome, right? But, like, we wanted better. Um so, like, our index will return very precise
[16:06] pointers, right? Low like row links, file paths, coordinates physically within each parquet file. Um that'll allow us to do some intelligent range reads that will make it so that you can get search engine-like performance from your existing data without copying it.
[16:24] Um So, when you put these pieces together, this is what the architectural looks like. Um Spark remains the front door, which I love. Users write Spark SQL and data frame code. Index tables just sort of integrates through the normal data source path that's transparent to the vast majority of our users.
[16:40] Um like I said, under the covers Index tables uses Arrow to cross like the native to Spark legacy barrier, similar to how Photon works. Um and our engine selects the right splits, finds the right rows, and then pulls the data back from the source of truth,
[16:57] which is your existing table that's in your existing governance domain. Um Yeah, I think that's all I'll say there. So, that's the architecture, but the goal for the architecture wasn't to make users understand splits or transaction
[17:13] logs or Arrow or native execution. It was to make index retrieval feel like the lakehouse that they already know, right? Like, I taught my organization, I think, to to build Spark, and I have hundreds of people who
[17:29] have these great competencies. I don't want them to have to learn something new. I just want their jobs to go faster and their lives to be better. Okay. So, from a user experience, um it feels intentionally boring, right? So, like, you're asking analysts not to learn a new query language. You're not
[17:46] asking them to move into a separate search UI. You're not asking them again to think about like all the technical stuff we talked about. They're in Spark, they're writing SQL, they're still working with tables. Um so, for simple retrieval, like in this first example, it looks like plain old query, right?
[18:02] Give me rows for this user, this host, transaction, session, whatever. The difference isn't the SQL surface, it's that underneath the table, um the lookup can be resolved through an index instead of doing a really deep broad scan. Um in the second example, you combine
[18:19] retrieval with analytics, right? Filter to a very selective slice, aggregate the group, summarize, explore it using normal Spark semantics. Stuff that Cloud Code can do for you or Genie, right? Um and that matters because operational workflows, they
[18:36] really aren't usually find one row. They're usually find a thing, pivot around it, summarize it, ask the next question. You imagine 20 minutes plus 20 minutes plus 20 minutes. We're going to fix that. All right. So, for normal predicates,
[18:51] right? Like, in aggregations, projections, limits, we just keep using Spark SQL the way that we always did. Um but index tables adds a custom infix query operator called index query that allows you to feed search engine queries and integrate it inside of your um
[19:09] your query processing pipeline, All And so, you can combine ordinary things like you see the log data is just plain old Spark SQL. Limit's plain Spark SQL, but then you can add additional filters that like in this case is doing a complex give me anything that has the word error in it
[19:25] plus anything that starts with off or has the full word login. All right? And so, like you get away from having these very lengthy chains of like I like and like statements that I think plague Cypher.
[19:42] All right. And so, go a little bit deeper. Um the important thing here again is that Index Tables isn't asking your people to learn a new environment. Like fundamentally, even if you get deeper into the architecture and use Spark UI, it still looks like Spark's parsing the
[19:57] SQL and Catalyst is building a logical plan. Um Spark still sees a data source and thinks it's doing a scan. Um but it uses this concept of the thing called predicate pushdown. Um where like a predicate, most of you
[20:13] know this and so, I'm sorry if I'm re-explaining, but is like the the where clause on your SQL. And in Spark, you get two chances to do the where clause. First, when you're reading the file, you can filter stuff out. And Delta and Iceberg do a good job of
[20:28] doing that in their scan. And then secondly, once all the data is materialized in the Spark, then Spark can do it. And so, what Index Tables does is it like wires into that pushdown boundary and it intercepts all of your where clauses and converts them into a syntax um and query
[20:44] processing language that the underlying index engine that's mostly based on Quickwit, like I said before, can understand. And so, it like kind of tricks Spark into thinking it's doing what it was designed to do. I think it's kind of cool. Okay. So, companion mode.
[21:01] All right. So, there are 19 modes. Companion mode is the most important adoption path for existing lakehouse environments, right? Most of your orgs already have Delta or Iceberg or something, right? Um and so companion mode's built around a simple idea that like you point a
[21:17] Spark engine at an existing table and you create indexes for it, and then you query against it, and it just sort of lives in the same ecosystem together. And it's delivered through custom SQL operand that you can see above. Um But so then the next question that
[21:32] always comes up is what happens when the source data changes? Right? So like I ran this thing, it indexed it, and then it stopped. Um and then I appended to my table, I did a merge, I deleted some data. So companion mode's designed to build
[21:48] and to sync using the same commands. And so like every time that you run that command, it'll look at the table, it'll figure out what files have been added or removed, and then it'll adjust the indexes to record it. So like in the current mode, this companion mode lives like a little bit lagging the data in
[22:04] your table. And so like it'll look at the previous transaction version, and that's getting better over time. Um but so like the idea is that it's simple, right? You just run the same query, it creates your table, you run it again, it catches up. And uh if you don't want to run it again and again, there's a streaming mode that um
[22:20] Zack, I think is here, one of my engineers at Capital One uh contributed to the project. So you just kind of set it and forget it. All right. And then there's a standalone version, which I think most people will probably start with, where um
[22:37] you Yeah, that's the standalone one. Where you can build a table just like you would with Parquet or CSV or whatever, but instead of specifying CSV or Delta, you say index tables. And then you can write standalone, and there's no Delta table, there's no Parquet, it's
[22:53] all sort of self-contained within the index like it is um conventionally in Quickwit, the thing that I like lifted a lot of their incredible open source intellectual property to build this. Good? All right. So, up next, let's make the
[23:09] 1000x claim concrete. All right, like that was kind of a like a shocking headline. Um So, the biggest performance gains from index tables show up when the result set is small relative to the data set. Remember the index slide where it was like you can do this thing across all
[23:25] your files or you can get two pointers and just go grab two little chunks of data. Um and that's exactly the shape of operational retrieval. You find a few relevant rows, messages, hosts, or events inside like a gigantic pool of data. So, for this section, I want to talk through a real benchmark um on
[23:41] Databricks and show the progression from plain Spark to Spark plus Photon. Photon's amazing, by the way. Um and then see what it looks like with index tables used in a couple of different ways. Um And so, the goal is not really just to show big numbers, it's to show where the time goes and what's actually happening.
[23:57] Give you an intuition. Um So, for the benchmark, we use the same kind of query that we talked about at the beginning of the call. It's like a little string. In this case, it's uh it it's a host ID. We use some really ugly host names sometimes. Um it's not a complex query, it's not a
[24:13] big join. Uh it's what our analysts run every day. And sometimes it takes a really long time. And so, we'll start with just a really basic example. Um the benchmark, all the results that you see are running on the exact same hardware using the exact same version of
[24:30] Databricks. Um there's a couple of like special tuning things we did in each, like between the non-Photon and the Photon run, we turned on Photon. And when we did index tables, we like reduced the size of the JVM heat because it runs in a native space and so, it uses off-heap memory.
[24:47] Which would be a really cool talk, but it's too much for today. Um and so, the thing that really changes isn't the hardware. We still have nice i4i.2xl servers with tons of NVMe storage and lots of memory, but we changed the scan method. Whether it's like a
[25:02] plain delta scan on spark or it's using photon accelerated like a C++ implementation or it's using my stuff, which hopefully will become our stuff someday, right? All right. So, the data set itself was like intentionally large enough to
[25:18] amplify the the point here, right? So, like I think you add it all up, it's like a 70 billion record set. And like I didn't cheat, I used optimize right with data bricks so that like we had the ideally sized delta tables that would
[25:33] get the best possible performance. And in index tables, I did some maintenance work to make sure that we didn't have lots of little tiny index um segments. Okay, so plain spark ran kind of slow, right? It was a little painful. 6,800 seconds,
[25:50] that's like 2 hours. Um and it seems silly, but like if you've been a cyber investigator, this is what we live with. Like you do a retrospective query when you want to look back in time and sometimes it could take days even on big clusters. Um and so like this is the analyst
[26:06] frustration story that we talked about at the beginning of the talk. Um spark's doing what spark was built to do in this case. It scanned and filtered and processed the huge amounts of data and gave us the right answer. Um it's just like it's this workload mismatch we're dealing with.
[26:22] And secondly, we turned on photon, it made a huge difference, right? It went from almost 7,000 seconds to about 2,000 seconds. That's a huge improvement. Um and again, it's doing exactly what it was supposed to do. It makes the scan go much much faster. Um but even with that
[26:38] improvement, it's still 39 minutes. You remember like someone's going to distract you, you're going to go to a meeting, you're going to get tired of investigating, you're going to take shortcuts. So, like we're we're not really there yet, but like photon's not ineffective, it's amazing. Um it's just a little of a mismatch for
[26:53] this workload without additional help from my indexes and more shortcuts. That's three times, right? That's impressive. Um, so then we ran the same query through index tables, the same just under 70 billion
[27:09] rows, the exact same data, the same query, um, but we used the index query operator. So, that I cheated a little. Um, but so index tables uses the index, it found the rows like within the first phase, and then it just went and
[27:24] cherry-picked each one of them. And we got down to 39 seconds. And so like that's a I think a substantially better quality of life for our people, right? 235 times faster. Instead of waiting like a half an hour,
[27:41] it's um, like you push the button and you're like, you know, looking at your watch a little bit, and but then it shows up. You don't you don't get out of the flow. But that's not a thousand, right? So like that that doesn't quite get us to where where we promised to take your time.
[27:57] Okay, so there's another feature within index tables that allows you to pre-warm your caches. And so that pre-warm will like pull some data to the fast NVMe storage inside of the server so that it can do the initial pointer
[28:12] resolution very fast. And so typically most of your data sets, there's like a few fields that your people really like to work with. Like our people really like to work with the message blob from Windows event logs. And so we can like when we bring up the cluster, we can tell it to keep that warm.
[28:28] And so like it'll use a little bit of the on-board disk just like Databricks does for its DBIO cache. And then we re-ran the query in 4 seconds. Right? That feels pretty good. And again, this isn't like big humongous cluster. It's an eight server non-Photon
[28:45] enabled i4i.2xl Amazon cluster that like full retail cost is $7 an hour or something. Oh, and when you're done, you just turn it off, right? Like Elasticsearch, you keep your stuff Splunk, you keep your stuff running all the time. You don't have to do that here. And so,
[29:00] there you go. 1,000x, 1,700x. And so, like this is where Oh, thank you. And so, why did this happen? We found the rows that mattered. We just cherry-picked them. We didn't download
[29:16] an entire S3 bucket and traverse through them um in a naive way. Awesome. All right. So, promised in the write-up that we talk about Capital One. So, Capital One, I have a bunch of my people here, I think.
[29:31] Um we're all in on the lakehouse architecture. We had a huge project that moved us from a hodgepodge of legacy technologies into into Delta Lake. And so, we are very heavily invested in Spark and hugely invested in Databricks. Um but our
[29:47] business depends on advanced analytics. Like we think of ourselves as a tech company, and we've always said that like we're the data company. Um and we're starting to deploy AI at massive scale. Right? Like we have corporate imperatives around like being the leader in AI. Hopefully, you should, too. And
[30:03] like as an industry, we'll get great. Um but like the lakehouse openness gives us the foundation to do that like with the good cost, openness, and governance. Um but cyber, like I talked about, has this other requirement to be support really fast investigation and
[30:19] interactive um reaction to things happening in our environment. Um and this is where index tables is beginning to fit. Um So, uh like talk about the industry a little bit. AI-driven threats are changing our
[30:36] operating model. Right? Like our attackers are moving much faster. They're scaling much more broadly. It's no longer about like a few very skilled people that work for the NSA or work for another intelligence agency that can do really sophisticated attacks. Um And so that means that like we're
[30:53] getting there's less time between our signal and our response. Um and it also means that people in our CSOC are no longer able to like operate from a playbook of like things that are expected to happen cuz there's a lot more variability. Um
[31:09] So Capital One, we're productionizing applications that use AI to help respond in real time. Um and so over the last few years, we've completely rebuilt our SIM like I said on a cloud-native platform, and we use streaming analytics for detections, and
[31:25] we have like, I talked about last year, the hand-optimized Delta stuff that gets our investigators faster query time. Um But this next generation is becoming significantly less predictable.
[31:41] Right? Like now that the attackers are not just reusing the same old tactics, like we have to be able to pivot across all the different dimensions in our data, hosts and identities and message tags, and destinations, and weird behaviors. Um
[31:56] And we need to be able to ask the next question without waiting for our engineering teams to like schedule a project where they can optimize data in some different way. Um So now we need fast, flexible retrieval where we can get that first bite, you know, in 4 seconds instead of 1,000.
[32:13] Um Are we Whoops. Did I skip ahead? Okay. Yeah. So like I said, our next generation is it's chaos. Right? And like with introducing agents, which like I I'm sure most of you have
[32:28] like used cloud code, and it does stuff that you never thought about, like that's also going to be true of these investigative workloads when you have a less like purpose-built agent. And so, you want to give it access to everything. You don't want to stifle its creativity, just like our people. And it's going to do it at a much broader
[32:44] scale. So, like, that 3,000-second query that occupied a gigantic, expensive cluster for 3,000 seconds. And so, like, the model just doesn't scale. And so, like, we need something um that's better and faster and more extensible. And so, with integrating
[33:01] index tables in companion mode, we're not having to completely rewrite our architecture, right? We we built an incredible, I think it's a wonderful application that uses Spark Connect to power a front end that feeds uh agentic stuff. It feeds AI
[33:18] summarization. It does orchestration. It automates a bunch of our workload. And we don't want to change that. It's great. Like, it was a huge investment and we're proud of it. But what changes is that, like, instead of pointing directly at our Delta data, we can point at index
[33:33] tables. We can use the like, essentially, our exact same queries. We have the same architecture. We don't have to rebuild our bone structure and all our CI/CD. And so, like, we're able to evolve our architecture to take advantage of, like, whatever design makes the most sense for us. A little data set, you don't worry
[33:49] about it. Something that's kind of simple and doesn't have a lot of columns, we can still choose to use like the clustering and Z-ordering stuff that we talked about before. Or we can optimize um and use indexes. Like, it all because it's architecturally consistent, you have
[34:05] that ability to kind of pivot and evolve in a way that you couldn't when it was like, do I put this in Splunk, or do I do this locally? And so, I think that's really exciting. All right. So, hey, doing on time. Thanks for spending all this time with me. Um so, today we covered why
[34:22] retrieval is different from analytics. Um we talked about how index tables brings index search into the lakehouse. And we've talked a little bit about how Capital One starting to use that pattern for fast flexible cyber investigation. Um
[34:38] So, let me leave you with four things. Right, first scans matter. Like, I'm not taking a dig at Spark. We've made a lot of progress as a data community. Still the workhorse for analytics, it's not going away. Um second, retrieval needs a shortcut. Search, pivot, point lookups, they
[34:54] shouldn't have to be a full scan. Um third, index tables adds that shortcut. Um and it gives us a retrieval native path while keeping the exact same foundation governance model. SQL's almost exactly the same. And fourth, like, you can feather it into your
[35:09] infrastructure. You don't have to have a big bang in a multi-million dollar project, right? Like keep your Delta and Iceberg where they work. Use index tables where you need something extra or special. Um Big idea, keep it simple. Keep the lakehouse, but
[35:25] cut down the friction that uh is keeping you from meeting your goals and is making your co-workers unhappy. Um so, yeah. I I think I have a little bit of time left for uh questions, but you can learn a little more about index
[35:41] tables on the web, index tables.io. You can look at my source code, contribute to the project. There's like features that you want to see. Um and most importantly, like, the last link here. Looking for something exciting to do, like, we're we're building some great stuff at Capital
[35:56] One, so. We'd love to hear from you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.