How Databricks Powers Real-Time Threat Detection at Barracuda XDR

Summary

• Real-Time Threat Detection: Barracuda XDR leverages Databricks as the foundation of its security analytics platform for real-time threat detection and mitigation.
• Scalable Detection Framework: Using DLTs, Spark Structured Streaming, and detection-as-code CI/CD pipelines enables scalable, accurate, and cost-efficient threat identification.
• Automated Security Operations: The platform combines advanced detection methodologies with automation to deliver unparalleled protection across 50+ integrations.

How Databricks Powers Real-Time Threat Detection at Barracuda XDR

Talk by Alex Dangel, Manager of Detection Engineering, Barracuda Networks ; Merium Khalid, Director, SOC Offensive Security , Barracuda Networks
As cybersecurity threats grow in volume and complexity, organizations must efficiently process security telemetry for best-in-class detection and mitigation. Barracuda’s XDR platform is redefining security operations by layering advanced detection methodologies over a broad range of supported technologies. Our vision is to deliver unparalleled protection through automation, machine learning and scalable detection frameworks, ensuring threats are identified and mitigated quickly. To achieve this, we have adopted Databricks as the foundation of our security analytics platform, providing greater control and flexibility while decoupling from traditional SIEM tools. By leveraging DLTs, Spark Structured Streaming and detection-as-code CI/CD pipelines, we have built a real-time detection engine that enhances scalability, accuracy and cost efficiency. this video explores how Databricks is shaping the future of XDR through real-time analytics and cloud-native security.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Barracuda XDR?

Barracuda XDR is an open extended detection and response platform that combines sophisticated technology with cybersecurity analysts and engineers backed by a 24/7/365 security operations center.

Why did Barracuda choose Databricks for security analytics?

Barracuda adopted Databricks as the foundation of their security analytics platform to provide greater control and flexibility while decoupling from traditional SIEM tools.

What technologies power Barracuda's detection engine?

Barracuda's real-time detection engine leverages DLTs, Spark Structured Streaming, and detection-as-code CI/CD pipelines to enhance scalability, accuracy, and cost efficiency.

How many integrations does Barracuda XDR support?

Barracuda XDR currently supports over 50 integrations ranging from the most advanced technologies to SaaS solutions like Microsoft 365 and Google Workspace.

Full transcript

Hi everyone. My name is Merium Khalid. I'm the director of SOC offensive security at Barracuda XDR. And today we're going to be talking about how data bricks powers real-time threat detection for Barracuda XDR. So before we get into the nitty-gritty, I want to set the stage a little bit about why what we do is so important and talk a little bit about the global threats that we see on a day-to-day. All right. So let's talk a little bit about some of the metrics that we've seen.
So in 2023 we actually saw a 95% increase in ransomware attack year-over-year. This is no longer an outlier. This is something that we are seeing increasing on a year-to-year basis. And we really need to shift to a more of a proactive approach rather than a reactive approach. Some of the other stats that we did see was 48% of organizations experienced fishing or a social engineering attack in the last in the past year with over 40,000 common vulnerabilities that were released on a year-to-year basis.
Again, I'm not going to go through all the metrics, but these are essentially signals for us that we need to basically up a level our approach, right? Because the threat volume is increasing, the cost is increasing, and the complexity is increasing because of things like AI. So what is Barracuda XDR? So Barracuda XDR is an open extended detection and response platform that combines sophisticated technology with cybersecurity analysts, cybersecurity engineers backed up by a security operation center that is 24/7 365.
And when you look at this shield, you want to think of a house, right? So, there's multiple ways a threat actor can break into your house. XDR is built to cover every single gap that you may have. So, it could be either the back door, it could be the front door, it could be a window. XDR is built to cover, make sure that threat actor does not get it. So, we actually support over 50 integrations right now from the most advanced technologies to SAS solutions like Microsoft 365, Google Workspace.
Um and some of the benefits of this, it's a fully managed SOC, so monitoring your um network 24/7 365, um the ability to detect and respond rapidly to threats. So, automated threat response, if we see a user um being compromised in your environment, we'll actually go ahead and disable it. And most importantly is the cyber 911, so the ability to call someone when you're having an incident or a breach. And really gives our customers that peace of mind that we have their back.
So, how does XDR work? And it's honestly very simple. So, we collect your data via API or XDR agent, and we're only collecting data that we care about, um data that is providing security value. Then we normalize that data, parse that data so that it can easily be processed by our seam for detection, um our SOAR platform for automation, and our threat intelligence platforms. And then, once we have your data, we really have you covered from end to end because we're actually going to take that response action as well when we do see a compromise within your network.
Um lastly, but not least, we have um an interactive way for our partners and our customers to see the alerts via our customer security dashboard, and it's completely self-service where they can actually integrate new data sources and set up automated threat response. So, I want to talk a little bit about why we chose Databricks and I think everyone within the business landscape for cybersecurity knows that legacy seams are getting more and more expensive with insane storage costs and insane processing costs with no real-time detection.
Since Databricks is built on Apache Spark, it's automatically optimized for processing huge amounts of data with fast querying. And not to mention that it comes with a fraction of the cost. Also, legacy seams, there's a lot of built-in machine learning capabilities. However, with Databricks, we are able to train and optimize all of our data to our needs. So, I want to talk a little bit about our our Barracuda's journey and Barracuda and Databricks partnership.
So, this actually started back in May 2020, so a little over 5-6 years ago. We integrated our enterprise platform that gave us a unified foundation for managing product enterprise customer data while enabling our teams for advanced reporting business analytics, which was very helpful. In 2021 and 2022, we used Databricks for our advanced technology group, which is a backbone of our journey AI initiatives that we have across Databricks and Databricks essentially helped us build advanced tracing and troubleshooting mechanisms.
In 2023, we used Barracuda we integrated Barracuda email protection into Databricks for the machine learning classifiers. And then fast forward to 2025, we are now integrating continuous integrations with XDR um, automated threat response detection engineering, giving us the ability to scale detections across thousands of our customers. So, this is Sorry, 1 second. Our migration journey. So, this timeline really highlights the transformational journey from legacy seam to a modern, scalable detection engine and data lakehouse we've built.
And we took a very phased and disciplined approach where we tested early, um, we validated at scale, and we honestly improved at each and every milestone that we have. So, back in August 2023, we kicked off things with a small, but very, very powerful POC where our goal was to prove that Databricks could ingest, parse, and trigger alerts for our customers. And this stage really showed us what the different possibilities are with Databricks.
Um, and then in September 2023, this was where we, um, scoped out some of our success criteria. We rolled up our our sleeves, and we were like, this is what success will look like for us. We looked at the different requirements for uh, SLAs, for data transformation, um, for data modeling, storage pipelines, orchestration, and monitoring. And then in August 2023, sorry, January 2024, this is where we started building our, um, Databricks architecture with using Autoloader for ingestion.
We used bronze to silver for data transformation, um, and we also started using DBT for our detection, DBT warehousing for detection engineering. Um, January, sorry, August 2024 is when we started our first integration, which was Microsoft 365. We integrated about 40 plus detections well across five different workspaces across the globe. Um, and we also integrated with some of our downstream notification systems. And then fast forwarding to May 2025, most recently we just deployed Fortigate using streaming workflows and our updated detection engine that will talk about.
We also did We also integrated 20 plus detection detection proving that the system is not just good for Windows or Office 365, we're also able to integrate network data as well. So, from POC to integrating multiple different sources, it wasn't just migrating the data. It was really about evolving our entire detection strategy. So, we were shifting our mindset to more of a detection as code rather than some of what the legacy seams have to offer in terms of detections.
I'm going to hand it off to go through our seam that we've built inside Databricks. Thank you. Um Sound my good level? Perfect. All right. Well, good afternoon, everyone. I know this is a awkward time. It's either before lunch or after lunch, so I either got food comas or stomach's growling. But, um and thank you for coming and listening today. We will be talking about how we built a seam inside of Databricks and going through some of those technical those tech those technical diagrams and system architecture.
But, fundamentally what we're talking about is how we take logs and telemetry and transform them into alerts, which indicate security incidents. So, I just want everyone to remember we talk about seam, that is what we're talking about, transitioning from logs to alerts. It's not lost on me that they let a security guy talk at a data engineering conference. So, I wanted to kind of convince you that what security engineering is is just another flavor of data engineering.
And in order to do build a seam inside of Databricks, you must follow best practices for data engineering. Databricks is a great canvas for us to paint that solution. And hopefully by the end you'll see why we chose Databricks and how it works. What is detection engineering? Fundamentally, we talk about the process of how we identify security incidents is what we actually mean by detection engineering. And we have to do a lot of research in the beginning to understand what threats we're hoping to detect, what's required to do that type of detection.
And that is what we're talking about when I say detection engineering. At Barracuda, my team is the detection engineering team. We have people that focus a good chunk of all of their sprint cycles just on researching threats and understanding how we can deliver these detections to our customers who send us the logs. How has it evolved over time? It's been around for over 30 years as detection engineering. It started from a very static binary signature-based model to more heuristic-based baselining and of course leveraging software engineering best practices with detection as code, which we'll which we'll highlight in a second.
But really what we're doing is taking all of our detections and figuring out how we're going to express them. And that's really what we talk about with detection as code. It's how we express detections, how do we store them, and how do we deploy them to be executed and generate alerts. Some of the trends that we're seeing in the security space today with threat detection is around understanding the data that you want to use for creating those alerts, but how we enrich that data is quite important.
Uh just a basic example is geolocation. Most logs that we collect are just IP addresses. So, adding in something like geolocation paints a better story. It tends to result in better outcomes for threat detection. Another thing is real-time detection, and real-time is in quotes cuz nothing is really real-time. There's always some latency. Uh but real-time shrinks the dwell time that a attacker will be inside of a customer environment.
And decreased dwell time means less of an impact from a security point of view. Uh so, think about detecting ransomware threats. It's kind of important to know about a ransomware threat before encryption occurs. Uh so, real-time detection allows us to do that. Uh finally, we talk about automation. That's going to be something that comes up in a lot of these presentations is how we can take those alerts, how we can take those signals, create really robust automation pipelines to be able to respond in real time to security threats.
And finally, uh taking some of the more binary detections and converting them into ML classifiers. Um how we compute baselines, how we can reduce our false positives, and the fault the volume of the alerts that we send using statistical models. And just to showcase like a finished product that our XDR team uh recently delivered to customers, we have an ML classifier that detects account takeovers. Uh these alerts fire in Data Bricks, and we have a automation pipeline that will reach into that customer environment and suspend the users that we suspect have been involved in a takeover attempt.
All right. Uh so, we are going to start our journey in how we build a SEEM, but first we have to talk about the parts of a SEEM. Uh and I realized I didn't define what seam stands for. Seam is security incident and event management. Uh but all this is this is a data lake full of our uh events and we're trying to figure out which events constitute a security threat. So these are the eight core pieces that I can think of that every enterprise seam that you purchase you would expect supports these features.
Um and I divided them into two sections so the ones that uh Databricks already supports uh really out of the box. There's a lot of options of course when we talk about ingestion there's a lot of it's a lot more simplified on this slide but these are some of the big major components that already exist inside of Databricks. And then we talk about the custom features which is really what we're going to focus on today is the custom features.
And these are our detection rules, our deployment pipeline, the engine itself what executes the detection rules, and finally our alerting. All right. So what is a detection rule? Uh detection rules is what most teams refer to in a detection analytic and these are templates for how a detection should work. Uh and I literally mean templates. These are stored as templates uh prior to them being deployed in Databricks. Um we talk about detection as code.
The code piece in detection as code refers to the markdown language that we're using to store uh these detection rules which are the templates. We support five different types in Databricks as of today and we're not going to go through each one of these but the types give us the macros for how we're going to detect a threat. So let me give you an example. The last one on the list is threat intelligence threat intel. That rule type takes two tables a threat intel table and a customer log table and we match on common fields like an IP address, uh uh hash, uh some common indicator value.
And that is specific to that rule type. So, you can think we support a lot of different uh methodologies for detection, but the template-ized uh macro format allows our engineers to be a little bit creative in terms of which one they select, but also um it's very, you know, apparent what I need to fill out to be successful uh to build my detection. And here is uh a preview into what that looks like. So, we my team uses uh GitHub, uh and we use a GitOps approach where we have our various branches, and the branches corresponds to our workspaces, our Databricks workspaces.
So, if a rule is in development, it's targeting our Databricks development environments, our workspaces. And if you are an engineer on my team, the only thing you need to know is how to build a template. We've abstracted away all the other nuts and bolts. You just have to build the template um in order to create the rule. This is the beauty of detections code. That was a real That was a big goal for us is how do we abstract away a lot of um the technical data engineering for a security team that maybe doesn't have those data engineering skills?
Um and this allows us to do that. So, this is TOML, for those that can't recognize that. This is TOML. This is a a version of it's a a a different flavor of YAML uh for storing mostly configuration templates. And we've actually built a validator on top of these templates that will build dynamic SQL queries and lint them, uh but more importantly, check for any um violations. Like, hey, I'm missing a key-value pair, or I specified uh a column in my SQL query that does not exist in my schema.
So, we've actually built a custom validator to make sure that any template that we introduce into our deployment pipeline is free from errors. Relatively. Can't can't detect everything, but good enough to be able to deploy it to Databricks. Each one of our detection rules is stored as a uh TOML file in a uh in a GitHub directory here. As you can see, we store them by data source then rule type. All right. So, we have a template.
We built the template. It sits in a GitHub repo, but now what? What do we do? So, this is our deployment methodology. It's very simple. Like it doesn't have to be complex. Uh we are again using Databricks workflows, which we're going to talk our detection engine is also Databricks workflows. Uh but this is a I think kind of a creative way to deploy to Databricks. Um we have a GitHub action that we ultimately run on the GitHub side, and it produces what we call workflow inputs.
Uh I've highlighted some of those. Uh we call them they're just notebook parameters. So, what are what are the changes that you're doing? The action identifies those changes and supplies a repo deployment workflow uh with those inputs. And the underlying notebook that's attached to our repo deployment will accept the inputs passed through GitHub and change what we call a detection metadata delta table. And that detection metadata table stores the parsed versions of all of our TOML files that I showed in it showed in GitHub.
Um if you are an engineer, again, all you're doing is running the action. Uh you're not seeing the workflow running in the background. You are getting notified in Slack when your deployment successful. Uh trying to abstract away uh some of the in the weeds Databricks workspace interactions uh and just making this a really smooth integrated experience. Uh there is uh a lot of other things that we can do with repo deployments.
So, we do code if we change the underlying Spark code base, uh we would do another type of deployment. It's all supported through this uh same methodology. All right. Uh so, this is perhaps the slide that interests most of the people in this room. This is uh the detection engine, um and this is what enables us to uh create security alerts from the logs that we get from our customers. So, I'll I'm going to step through this as high level as possible, and then at the end if course if there's any questions, we'll be happy to revisit the slide.
Uh we have inputs. So, I talked about our detection metadata table. It stores how our rules operate. And then we have all the associated logs that those detections operate on top of. We have intermediary tables. So, think about like machine learning scoring, for example. Those are going to produce uh sources that we can operate rules on top of as well. We're going to read all that data um using a read stream into our continuous Databricks workflow.
And the way we've done this is we have a streaming notebook that exists per data source. So, I've given you three examples here. Uh this is uh Microsoft Office Later's not going to work for me. Microsoft Office 365 uh streaming notebook, Microsoft Windows, which is uh server syslog, and uh Fortinet FortiGate, which is a firewall. Um so, we have a a uh notebook for each data source, and they are executing our rules uh in a for each batch.
So, that's a Spark um write stream method, by the way. And that allows us to iterate through all of our rules that we're reading from our detection metadata table and execute them to create what we call the raw signal. We're not done yet. Raw signals is just one step of our overall architecture here. We then do a union stream union stream where we take all of those raw signal tables, which all have the same schema, and we union them together and process them.
And the alert processing is uh required uh to fill three use cases here. So, the first one is suppression, the second one is exceptions, and the final is notification. And we're going to cover that in detail in the next slide. Uh the fundamental fundamentally we have to have a way to get these out of Databricks to our customers cuz our customers don't live in Databricks. So, we send these off to either our SOAR platform for automation or we send them directly to our case management tool that sends them to customers.
Uh and then finally the output of those alerts is written to one table. And in that table includes all of the processing log and logging that's required to make this work. We don't want a lot of separate tables to kind of see what the what the story was on the customer learning. So, we actually keep and capture everything in that one central alert table. Uh all of our um you know, our code's pretty uh resilient to errors, but if we do encounter any specific rule level errors, we actually are able to log those to tables without killing our Spark workflow.
Uh we also do streaming as well. As you know, we're reading from multiple tables that some are bigger than others, some have more columns than others. So, we want to capture some of those uh Spark metrics as well. And we use our Spark query listener uh to be able to do that. Uh all of this runs on job compute uh using a Photon engine. Um and uh we are running this per workspace. So, each uh development workspace, staging, and production workspace is running some version of this workflow with the associated rules.
All right. So, I mentioned some alert handling challenges, and if you've never had to onboard a scene before, you may not know that these are actually challenges. But, we can't just send every alert that we trigger to a customer. If we did that, our customers would be very upset with us. Um, because there is inevitably things that are going to happen in terms of patterns that result in alert flooding. Let's say someone is doing a software deployment in in their environment, and it's triggering a thousand alerts for some some detection that we've created.
We don't want to send a thousand alerts to the customer for the same activity. So, we have this idea of suppression, where we can group alerts based on similar columns and suppress them for some time period, let's say an hour, for example. Uh, so, our processing pipeline allows us to do that. The second one is similar, exceptions. Exceptions are more long-term suppressions. So, that's the big difference. Exception says the customer is acknowledging that the detection rule should not be firing under these pretenses.
Uh, we can then create an exception so that rule never fires for that customer. These are stored in another metadata table that's read in by that Spark job, and we actually have a JNAI powered uh web application that assists our SOC team in being able to create those exceptions um in Databricks without without having to actually do a table insert. Remember, my my goal was to abstract away a lot of the things in Databricks for those that aren't super familiar with data engineering.
And uh the final the final thing I wanted to mention on this slide is uh we aren't using a lot of dependencies. Okay, we we want it to be as simple as possible. So, everything in Databricks 16x runtime, the Python standard library, there is one exception and that is PySpark IP. PySpark IP is not included in Databricks runtime, but it is a library that we needed to be able to build the IP address UDF. This allows us to treat IP addresses like IP addresses and not strings.
IP addresses have subnets. Strings do not have subnets. So, you know, we need to be able to process this type of IP data, so we had to use PySpark IP. The rest all comes for free in Databricks, which was really nice. It means simpler development experience overall. Alert output. So, this is just a sampling of some important columns so you get the idea of how we're storing our alerts. We have about 50 columns, so I could not have fit them all in this slide, but we include metadata about the alert such as like the MITRE information, the type of rule that we're dealing with, the name of the rule.
And then we add in a lot of other extras like dynamic alert summaries, SLA tracking, like how long it took us to generate an alert in minutes. We have rule specific fields. So, I talked about threshold rules and how we do aggregations, storing the aggregation result for a detection it might be very important. So, we've actually done that down here. You can see we store the aggregation result. We also include our processing output.
So, I mentioned we have the idea of a of an exception potentially removing an alert. We have a suppression that also could remove an alert. If we don't store these, we have no idea what actually prevented that alert from firing. Very important for us to have that, as well as the notification status. So, the notification status is where we sent the alert after it triggered. If we try to send it somewhere and we get a 400 error message, we probably should know about it.
So, we're able to do that inside the Spark workflow as well. What does a customer see? Uh customer just sees this. So, uh they get a notification that a security incident occurred inside their network uh or their environment or their cloud uh their cloud system, and it will summarize the event. Uh this is actually using our in-house uh Barracuda AI insights. Um and then they can click on the alert to view it inside the dashboard.
Uh all of the data that we have in there is obviously powered by Databricks. Finally, um you know, success doesn't happen in a vacuum, and I think really one of the things I wanted to mention is that we didn't get everything right on the first go. That's part of our engineering process is trial and error, what works, what doesn't um work. And I think every team is different in how they do this trial and error process.
Uh we did have DBT and warehousing as our first uh detection engine uh 1. 0, so to speak. Um this did not provide us the real-time detection that we were looking at, as you have to schedule models with DBT. Um so, we were looking for a more real-time approach. Uh so, we switched over to DLT after that, um as mentioned in our earlier slides. Uh DLT was great. Uh there was there was no um problems with DLT. Fundamentally, the way our streaming workflow works is very similar to how it worked in DLT, but we wanted to have our team do a deep dive on the streaming side, really become experts.
And DLT, as maybe a lot of you know is a declarative framework and abstracts away a lot of the streaming knowledge which was a shortcut for our team but ultimately didn't allow us to add the customization that we needed. So we use the streaming workflow today. It is a modular code design where we import different classes. All of our detection rules are stored in the class that we import. We're able to make changes without having to restart the streaming jobs which was a huge benefit.
And overall we're very happy with the streaming workflow and we think it's going to set us up for success later down the road. And I will hand it back over to to talk about some challenges. Thanks. Thanks for the deep dive. So we've had a few challenges along the way and we want to be fully transparent and share some of the bumps we hit along the way and the road bumps. Like any major transformation the migration came with its learning opportunities but what stood out to me the most was how resilient our Barracuda team was and also our Databricks support team.
They stayed with our side across the whole migration journey but one of the biggest shifts that we had was training multiple teams within XDR on the Databricks Lakehouse architecture with if that included managing ELT pipelines to building dashboard Databricks SQL queries building visualizations. As Alex mentioned we're now evolving towards a detection as code model where data engineer skills and programming skills are central to maintain and improve our detections.
It's essentially as I mentioned before it's a shift in mindset. So one of the other things that we did struggle with was optimization and best practices. It wasn't something that was plug and play. We experimented a lot with warehouse configurations number of node sizes number of workers to get that best combination with cost and performance. But I did want to give a quick shout out to our Databricks team for the support and the best practice in terms of the best practices that they gave us throughout the whole migration.
So some of the benefits that we encountered um as we moved to Databricks is that we were actually saving a overall 75% cost daily in both processing and storage cost compared to traditional seams. We're also alerting our customers faster. So customers actually receive our alerts within 5 minutes of logged alert detection, which is a significant improvement in terms of retaining our customers and in terms of the response time.
And one of the other things was ease of use. So our detection engineers can actually now focus on studying on new and emerging cyber threats and building new detections rather than managing and scaling new infrastructure. And then also scaling onboarding. So we're actually at a place right now where we're able to integrate and migrate over multiple data sources at a time compared to when we first started. So when we first started we were migrating just one data source and right now we're actually doing three.
So the platform that we've built is is very scalable. I do want to just talk a little bit about some of the future plans that we have and looking ahead So right now we still have some data sources on a legacy seam and we want to migrate everything over to Databricks which is about 50 plus data sources. We also want to do some cross-platform collaboration across Barracuda. So as Barracuda adopts Databricks throughout their different products we want to share intel, how want to share threat intelligence IOCs.
Uh, we want to build new machine learning and AI use cases for detections. And then, um, most im- last but most importantly, we want to continue collaborating with, um, security professionals that are in the Databricks ecosystem because we're all in a shared mission, right? Which is protecting against cyber threats and getting the bad guys out.

Learn more about the Databricks Data and AI platform.