Sponsored by: Imply | From Splunk to Databricks: Unlocking Open, Scalable Observability

Summary

Teams migrating from Splunk to the Databricks Data and AI platform face a challenge beyond storage: making large volumes of telemetry data interactive at scale without pre-indexing everything or duplicating platforms during the transition.
Imply Lumi Log Lake enables sub-second queries on raw JSON log data directly from S3 with no pre-indexing, bridging the performance gap between Splunk's proprietary SPL workflows and open lakehouse SQL.
The gradual migration approach lets organizations move data into open lakehouse tables and run existing dashboards without maintaining two full platforms simultaneously, eliminating vendor lock-in through open formats and separated storage and compute.

Chapters

FAQs

What are the main limitations of Splunk for security and observability?

Splunk relies on proprietary storage formats and its own query language (SPL), which creates vendor lock-in and couples workflows tightly to the platform. Even when Splunk uses object storage, the files tend to be proprietary and non-portable, preventing teams from reusing data in other tools without a costly migration.

How does Imply Lumi Log Lake work with the Databricks lakehouse?

Imply Lumi Log Lake separates storage from compute, allowing queries to run directly against raw JSON log data in S3 with no pre-indexing required. This enables sub-second query speeds on large historical datasets while keeping data in open lakehouse tables accessible through Unity Catalog on the Databricks Data and AI platform.

Why is standard SQL alone not enough for security log analysis in a lakehouse?

Standard lakehouse SQL lacks the performance characteristics security analysts need for real-time investigations across large, unindexed log datasets. Sub-second query speeds on raw machine data require a specialized query layer on top of the lakehouse, which standard SQL on historical datasets cannot reliably deliver without additional infrastructure.

How can organizations migrate from Splunk to Databricks without disrupting existing workflows?

Teams can adopt a gradual migration path by moving data into open lakehouse tables and using a layer like Imply Lumi Log Lake to run SPL-compatible queries natively on that data. Existing dashboards and workflows do not need to change immediately, eliminating the need to maintain two full platforms side by side during a multi-year transition.

Full transcript

[00:09] So, I'm Gian Merlino. Uh I'll be talking about uh from Splunk to Databricks. Really just a talk about how we can migrate from classic logging and observability systems, classic SIM systems into a lakehouse style architecture powered by Databricks.
[00:23] Unlocking open scalable observability in the lakehouse. Quick show of hands, how many folks are in a situation like that where you are you have a sort of a classic traditional observability SIM system, you're curious about doing more
[00:36] in the lakehouse. Nice. All right, I hope to have something to add to all your lives. >>
[00:50] the company I founded. Imply has been around for 10 years. We are uh recent uh we're recently starting to do things in the observability and security space. Up until now, we've been focused on the
[01:02] real-time analytics space with a open source uh real-time analytics database called Apache Druid. Um we've raised plenty of money from some nice investors, Andreessen Horowitz, Costanoa Ventures, Toma Bravo,
[01:14] and more. Uh across our current products, uh database-oriented products, we're processing 500 trillion plus data rows per year, sub-second average query speed. So, this
[01:26] is all just to uh talk about the credibility of the underlying technology. And I want to talk now about how we're going to apply that to uh try to make life a little bit easier in the um
[01:38] logging and security world in the lakehouse. First off, I want to talk about why observability in the lakehouse is a really good idea. Um it's a really good idea because we are
[01:51] going to be able to store data directly in cloud storage. It'll be accessible through open catalogs. We can query data directly where it lives. We can analyze it in place at cloud scale. In today's world, the most
[02:04] modern tools are all built around querying data in cloud storage. This is the lowest cost, most flexible system. Databricks has this great system where you get to launch multiple warehouses pointed to the same data. That's a
[02:17] really good example of what the lakehouse gives you. Open formats are great because they keep your data portable.
[02:31] One issue with classic SIM solutions, classic observability solutions, um is they almost always come with proprietary storage formats, locked-up storage formats, retention managed by the platform. Even
[02:43] if they use object storage, the files in object storage tend to be proprietary. Many of them don't use object storage. And the workflows are coupled to that platform storage. You're always thinking about um you're always thinking about,
[02:55] you know, how do I do this workflow in the platform? Generally, the same servers are doing storage and compute. In an open lakehouse, you have open table formats based on um binary formats
[03:07] like uh like parquet. Uh you have files stored directly in S3, and they can be portable across applications. This is why it's become so popular.
[03:20] Of course, I would say the biggest reason why the lakehouse has become popular is the cost reason, the economics. The economics of lakehouse are really nice. With a traditional system, you've got always-on infrastructure.
[03:33] Storage and compute must scale together. It's expensive to retain lots of data. You tend to see things like, all right, maybe I'm going to store a week in my traditional system, and then I'll dump the rest in the lakehouse. Lakehouse infrastructure is where you
[03:45] dump the rest of things for a reason. The independent scale the independent scaling of storage and compute means that the data you're not querying, you're not paying for compute for that data. You're just paying relatively cheap storage costs.
[03:58] You've got elastic infrastructure for bursty demands. And you've got relatively low-cost long-term retention. >>
[04:11] systems. You know, one of the issues with traditional system is you're going to have multiple ingestion paths. Separate storage systems across teams. You're going to duplicate your operational infrastructure. This creates
[04:23] a big hassle with everyone managing their own platform. The lakehouse system there's a shared foundation and a shared cloud system, centralized storage, centralized governance.
[04:37] This sounds great. This sounds great. What's the problem? Why are we not all doing observability and security in the lakehouse? Well, I think there's some frictions. And I want to talk about some of the frictions that we see. One is that popular workflows
[04:51] with great tools remain tied to proprietary platforms. One example, Splunk, a very popular tool, the operational workflows in that tool are tied to SPL. SPL only runs on data that's indexed in Splunk. Even in
[05:04] Splunk, if you're doing federated search to S3, you can't use the full SPL language. You use a subset of it. Tooling is tightly coupled to index storage. It's difficult to extend workflows to
[05:16] open data. You got to use You have to use these special commands if you can do it at all. Uh another example, if you're using Grafana, Grafana's got a really nice uh built-in uh log explorer system, but it only
[05:29] works with Loki. Doesn't work with any of the other data sources Grafana supports. This kind of fragmentation is endemic in the traditional world. Um SQL native tooling. This is the kind of
[05:42] stuff that works best in the lakehouse cuz the lakehouse is by and large a system where we're speaking SQL. Works great for analytics. There's great SQL tooling for analytical workflows. But it's built around tables and
[05:54] structured analysis. It's still evolving for observability. The tooling for using SQL for observability and SQL for security investigations is still very nascent. It's not as mature as some of the other systems.
[06:10] SQL also is not always the ideal language. I have here an example um of a search pipeline running SPL, the Splunk language. Many observability systems, uh CrowdStrike, have Dynatrace,
[06:22] a very similar Elasticsearch, similar languages. They tend to be pipe-based. They tend to have regexes built in as a first-class thing. One thing we're doing here is we're searching the app log. I don't You can't really see it very well,
[06:34] so I'll just describe it even though you can't see it very well. >>
[06:46] with data, very concise, very good for uh iterative investigations. SQL's a little clunky. It's a little clunky for doing this stuff. You tend to have um you have regex expressions,
[06:59] but there's no equivalent in SQL of just write me a regex with named capture groups, extract the fields. SQL requires str- it's built for structured analysis. It's a bit awkward to get to work with
[07:11] data that does not innately have structure. It's a bit awkward to work uh with workflows that are very pipeline oriented.
[07:24] Another issue, another reason we're not all doing observability in the lakehouse, is that migration is expensive and disruptive. These systems, workflows are baked into them. It's hard to migrate. Once you are using one of these uh non-open systems, you know,
[07:37] non-lakehouse systems, getting onto the lakehouse is going to be a mul- can be a multi-year effort. Here's what you typically have to do. Because the lakehouse requires schemas,
[07:49] one of the first thing you're going to do is apply and normalize schemas. Really popular choice in the lakehouse world, OCSF. For security purposes, you're going to rewrite your existing queries, alerts, reports, and dashboards
[08:01] to use SQL and to use SQL-based tools. You'll need to run parallel infrastructure during transitions. That means you're going to have redundant copies of logs. You're going to have two vendors for a while, paying them both a ton of money. It's going to be very
[08:13] expensive and very time-consuming to keep them in sync. What you actually want for migration is you want to ideally keep your existing workflows, use schema on read when applicable. You don't have to define a schema for
[08:26] everything. Standardize on open cloud storage, and avoid, most importantly, avoid duplicating infrastructure. You don't want to pay for two copies of the data for the entire migration, which could take over a year.
[08:43] So, this is why we're not all doing observability and security on the lakehouse. And what we're trying to do at Imply uh with this new product Lumi, is we are trying to make it easier to do this stuff in the lakehouse by trying to address some of these frictions I was
[08:55] talking about. So, the idea is query unstructured logs with schema optional directly where they live in multiple languages.
[09:08] So, the idea here, no pre-indexing required, no pipelines or deduplication or rehydration required. Your data is in object storage, so we embrace the lakehouse, embrace object storage, and schema's optional.
[09:22] And you see on the top here, you know, we have Databricks, Splunk, Grafana, all pointing in. The idea is that by speaking multiple languages, we speak SPL, we speak SQL, we speak Log QL. By speaking multiple languages, you're
[09:36] able to have your one copy of your data in object storage. We don't store it in Lumi itself. You have your one copy data in object storage and then a view of that data is exposed to whatever system needs it. You can use your old workflows on it. You
[09:48] can use your new workflows on it and you don't have to pay for multiple copies. >>
[10:01] logs coming in. Logs may come in directly to a data lake. They may come in streaming through a collection system. Either one is fine. Log Lake is the layer that sits on top
[10:13] of your data lake. It sits on top of your lakehouse and it provides that log oriented view on your data that enables the Lumi compute engine to power SPL workflows, LogQL workflows, SQL workflows on top of that data.
[10:27] This compute engine is only running when you're doing queries. So, it's a fully separated storage and compute, uh fully on demand. If you're not querying your data, the compute engine is not running, you're not paying for it.
[10:44] So, the idea here is we can move on to open storage without making these tradeoffs. We can preserve performance cuz we have this compute engine that launches on demand, scales on demand, and scales to zero when you're not using it.
[10:56] Designed for low latency operational investigation. You can get interactive performance across historical logs with this scalable compute engine and you can query directly in open storage in the lakehouse without needing to do things like rehydrate data, ingest
[11:10] data, anything like that. Uh what we find is that when you're running, for example, I talk about Splunk here because that is one of the major systems we support outside of the lakehouse.
[11:22] Um what we find is that when you run, uh you know, a classic workload um using Log Lake on data from object storage, It actually can end up
[11:34] not just being the same speed as your classic workload, but even faster due to these benefits. So, the idea here is you can modernize your architecture
[11:46] incrementally without breaking the bank and without a lot of complexity. You can keep your existing workflows as long as you want. Maybe eventually you think you want to migrate everything to SQL, but you can take your time
[11:58] because you're not maintaining two copies of data the whole time. You're not maintaining two full systems with full data pipelines. You're really just maintaining the UI API layer. Just a little more lightweight.
[12:10] Your historical logs all live in object storage in the lakehouse. Loomi log lake provides a view of them. And then over time you can add more stuff to this.
[12:26] So, now I'm going to do a little bit of a demo. So, here uh let's see. We can kind of see this. So, great. You know, that's the best I can ask for.
[12:39] Um here we have a SQL query that is designed to help us with a security investigation. Um the SQL query is uh searching Windows logs
[12:51] in the lakehouse uh for pro- for programs on the Windows systems that have escalated access to LSASS, which is a Windows security service. Our thought here is that we're looking
[13:03] for credential dumping attacks. This might be uh something we're looking for as part of a security detection. Um and our hypothesis is that elevated access to LSASS might be indicative of a potential credential
[13:16] dumping attack. So, okay. This is a SQL query written against Delta Lake. The files are stored in parquet. They've been structured. We're using uh you know, Databricks' built in stuff top to bottom.
[13:30] I'm not going to run it because it does take 3 minutes. You can kind of see at the top it says 3 minutes. But you can see the results here.
[13:42] Now here I have almost the exact same query. Uh using Lumi with Log Lake. So we're still within the data bricks system. You know, we're still in a data bricks notebook. And the SQL query looks very very similar.
[13:54] Uh and what we've done is we've set up this labs demo data set over here the from. We've set that up to query using Log Lake. I run that. And it works just the same as querying
[14:08] any other table. And it doesn't take 2 minutes so I can actually run it live. Takes 1 second. Um now I did cheat slightly before this talk I made sure the to spin up the compute cluster so we don't have to wait
[14:20] for the compute cluster to turn on. That takes about a minute. So if it was a complete cold start the first query will take about a minute. Um but you know once the compute cluster is spun up it takes a second and if it's idle for a configurable amount of time
[14:33] once it's idle it turns off and you stop paying for it. This query is hitting raw data which in this case is new line delimited JSON. So it's not using the parquet files. It's
[14:46] not using data that's been structured normalized. It is directly hitting the raw new line delimited JSON files and we're still getting this performance. Because we're directly querying the raw data
[14:59] we can actually retrieve the raw data. So here's another example of a query where I am looking at the event level data for the Sophos OS query.exe uh privilege escalations to the LSASS
[15:12] process. So
[15:27] >> So the idea here is you can do your security workflows the way you're accustomed to doing them um in Databricks. Uh you can get the raw data, you can search raw data directly uh and you can um use SQL the entire time.
[15:44] You can also use Splunk. So here's the exact same query. Let's blow it up a little bit. Here's the exact same query uh running in Splunk using SPL.
[15:56] You can see why I like SPL a little more than SQL for this kind of stuff. It's only three lines long. You can actually have a hope of reading it. It's not, you know, 12 lines long like the SQL query. Um but uh this runs in a second the same as the SQL query. It's running on the exact
[16:09] same raw data in the lakehouse, the newline delimited JSON. Running is in the same Lumi compute engine, just using our SQL mode and or using our SPL mode instead of our SQL mode.
[16:23] Setup is relatively easy. You just point Lumi at your bucket in S3. And then we have copy and paste instructions for setting up connections into Databricks, into Splunk, and into other things like Grafana, Tableau, etc.
[16:38] And I want to show the compute configuration for a bit, too. So I mentioned Lumi's got a compute layer. Uh they work very much like Databricks warehouses. So you got extra small,
[16:50] small, medium, large, etc., etc. Uh they have idle times just like Databricks warehouses. They spin up when you need them, they spin down when you're done with them. What makes them different from the Databricks warehouse is that they don't just speak SQL, they speak SPL, they
[17:04] speak Log QL, they speak these observability and security-oriented languages. They work great with unstructured data. And something very important is they are automatically indexing data as we query it.
[17:16] So as we query the data, we create token-based indexes, we create indexes on extracted fields, and this is what allows those queries to complete in 1 second, 2 seconds that took Spark SQL 2 minutes. It's the fact that
[17:29] we're using a very similar warehouse style design, but with the ability to work with unstructured logs, the ability to work with multiple languages, and the capability of creating indexes on that data as it's being queried to speed up
[17:42] those queries. Um So, whoop. So, I hope
[17:54] that was interesting. Um if you are interested, uh our booth is right over there. It's the Imply booth. It's it's uh on the inside row, just two back. >>
[18:06] we really appreciate it if you fill out these surveys. Please do. And I hope that the some of the things I've commented on about doing security and observability in the lakehouse have resonated. Um if there's any questions, or anyone wants to talk, feel free to
[18:18] come to me uh see me at the booth. And if anyone's got a question now, feel free to raise your hands.
[18:30] Thank you so much. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.