Scaling Cybersecurity Analytics Without Scaling Costs

Summary

JetBlue built a security analytics platform on the Databricks lakehouse using Delta Lake, Auto Loader, and Databricks SQL to ingest high-cardinality telemetry at scale, decoupling storage from compute to dramatically reduce dependence on expensive SIEM indexing.
The platform enables bot and web crawler detection, CAPTCHA effectiveness measurement, and attack trend analysis by IP, geography, and URL path, supporting both real-time monitoring and historical investigations that legacy tools make prohibitively expensive.
JetBlue is integrating with the Databricks Lakewatch private preview and plans to add Genie for natural language security queries and expand threat hunting and detection capabilities across the platform.

Scaling Cybersecurity Analytics Without Scaling Costs

Cybersecurity teams often struggle with the cost and scale of analyzing massive telemetry. JetBlue solved this by building a security-first analytics platform on the Databricks lakehouse. By leveraging Delta Lake, Auto Loader, and Databricks SQL, JetBlue ingests high-cardinality data into low-cost cloud storage, applying compute only when necessary. This enables long-term retention and flexible analysis that legacy tools find prohibitively expensive. Using Databricks SQL, they developed a fastly-focused security layer to: Detect anomalies and bot abuse Measure CAPTCHA effectiveness Analyze attack trends by IP, geography, and path Support real-time and historical investigations Decoupling storage from compute resulted in faster investigations, better visibility, and significant savings. Attendees will learn how to use Databricks to scale detection and investigation workflows cost-effectively.
Talk By: Greg Speranza, Engineer IT Security, JetBlue Airways ;

Chapters

FAQs

How did JetBlue reduce its security tool costs with Databricks?

JetBlue replaced their reliance on expensive SIEM indexing — which charges high costs for data storage and limits long-term retention — by ingesting high-cardinality telemetry directly into Delta Lake using Auto Loader at low cloud storage rates. This decoupling of storage from compute means JetBlue only applies expensive compute when running a specific query or investigation, rather than paying for always-on indexing.

What security use cases did JetBlue build on Databricks?

JetBlue built a security analytics layer using Databricks SQL that detects bot abuse and web crawler activity, measures CAPTCHA effectiveness, and analyzes attack trends by IP address, geography, and URL path. The platform supports both real-time security monitoring and deep historical investigations that would be prohibitively expensive in a traditional SIEM.

What is Databricks Lakewatch and how is JetBlue using it?

Lakewatch is a Databricks security product for native threat detection and SOC analytics that JetBlue is integrating with as part of a private preview program. This represents the next stage of JetBlue's security analytics maturity, moving from custom notebook-based detection toward a more structured platform for threat hunting and investigation workflows.

What challenges does high-cardinality security telemetry create?

High-cardinality security data — such as web access logs with unique IP addresses, session tokens, and URL paths — creates massive storage and indexing costs in traditional SIEM tools not designed for this volume. JetBlue's Databricks architecture handles this by storing raw high-cardinality data cheaply in Delta Lake and applying compute selectively, enabling analyses that were previously too expensive to perform.

Full transcript

[00:07] Good afternoon everyone. I hope you're having a good summit experience. Um really appreciate everyone taking the time to join my session here and I'm really excited to to be at the summit to talk about uh you know the work we've been doing at JetBlue over the past uh
[00:19] year and a halfish around. Um we've been doing a lot to scale our data in a massive way um keeping costs down and also building custom dashboards and applications. Um before I get into more
[00:31] of the nitty-gritty, I do just want to quickly introduce myself. Uh my name is Greg Sparza. I am a security engineer here at JetBlue. Um like anyone else in this audience who works in cyber security, you know, I wear many
[00:43] different hats. Some of my major focuses are are um cloud and web security, den security, um analytics. I also design and maintain um all of our ingestion pipelines and I build custom
[00:57] applications for uh some of our other security teams. So it's no surprise again to anyone else who is in the cyber security field that you know if you work with data um we we're all kind of dealing with a massive
[01:10] um telemetry problem right um not everyone's data is exactly the same of course um but I think a lot of us may have the same problems um you know like massive growth massive and rapid uh
[01:22] telemetry growth right we're always onboarding more and more sources um some of these sources could be hundreds of gigabytes a day right depending depending on on on kind of what you're ingesting. Um, and then those sources,
[01:35] the there could be a lot of uh high cardality data, right? Um, the example source I'm going to be using to talk about my story today um has a lot of these high cardality data fields. So, um, more about that later. Um, and then
[01:49] of course, if you're dealing with SIMs, you're dealing with um, expensive indexing models, right? They charge an arm and a leg to store your data there. Um, as well as uh, limited retention rate. you know, if you want to store your data for more historical purposes, it it starts to get very expensive. Um,
[02:02] and it makes investigations for that historical uh data um very difficult to do. So, here are some of the main things that we need to change at JetBlue. Um, we need to stop relying on just one
[02:14] destination for storage, which was our SIM at the time. Um, we we needed to change the fact that we had a lot less flexible analytics, right? Like, yeah, it's cool in your in your SIM of choice. You can do visualizations, you can build
[02:26] dashboards, but I mean with data bricks, I felt like when we started to switch to data bricks, the the uh the floodgates really opened right and I was able to do a lot more creative stuff with our data. Um as well as some of my other uh some
[02:38] of my other team members who now use it. Um as I mentioned earlier with the historical uh investigations um you know before we went through all this data brick stuff uh we really couldn't at all do anything with our historical data
[02:51] right because I think we were limited to uh a year I think we had some of our indexes set most of them to 90 days right so um and then we had to knock down the expensive long-term retention rates as well
[03:05] so here's a quick uh very high level overview of the of the data flow Um so we start here with the source. This is the source um source log. And for
[03:18] this purpose I'm going to be using um our Fastly data source which if you guys are not familiar with Fastley uh they're a content delivery network. They front um a significant portion of our web services and applications. Um it's basically web traffic logs. So that
[03:32] would be the source and then we send uh those logs as well as all our others through an observability pipeline tool um where we do very light normalization um and manipulation with that data and then they land in um Azure data lake
[03:46] storage or ADLS uh which we use strategically to uh cut costs on that long-term retention. Um from there in ADLS I then uh have a Pispark inest notebook um that uh does a
[04:01] lot of more processing and uh manipulation. I'll get into the Pispark notebook a little bit in a minute. Um once the notebook does its job, it then lands in a normalized security data table. Um and then from there I you know
[04:14] I build the applications on the dashboards that actually make the data usable for my team. Um so a little bit more on how we ingest and pro uh and process our data. So this
[04:26] top picture here is just some markdown within the notebook I have to instruct um some of our other uh teammates on if you know if they need to clone the notebook and and um and make uh you know
[04:39] in uh onboard a different source um because this notebook is highly modular and reusable. So that's the way I developed it um to process and transform data. Um another big component of the
[04:51] notebook is the use of autoloader uh which allows us to incrementally process incoming data. Um that is absolutely key to this notebook. Um and new files are discovered automatically as they land in storage thanks to autoloader. Um and it
[05:05] was really important for us to do this because we needed the architecture to scale uh without having to do manual tweaking, right? because we have tons and tons of data coming in through this. Oh, and this h sorry this second picture
[05:17] here uh this is just a another one of the cells of my notebook. There's about eight cells in total. I just wanted to give a quick example of kind of some of the stuff we're doing. I don't know if you were able to read it um just yet, but these are just um the ways or the
[05:30] way that that you can build your schema um when it comes to the columns to keep. So that's just one of the steps. So a little bit more on the uh pispark injection pipelines um runs on a set set
[05:43] schedule depending on the source. So I have sources running near real time five every five to 15 minutes to um you know every hour once a day whenever it really like I said depends on the source here. So um and then the the notebook is also
[05:56] doing you know uh uh normalization. It's doing enriching. It's also doing time window filtering, which kind of goes back to the uh the schedule, right? Like if we need it to run on every 15
[06:08] minutes, then we'll set a time window that will retrieve that new data every 15 minutes, you know, as it comes in. Um, and then that data goes to Delta Lake tables that are optimized uh for uh
[06:20] investigation workflows. So now let's move on to building the security analytics layer. And this is where really where the work uh all the work starts to shine. Um so you know now
[06:33] that our data is transformed enriched it's nice and neat. It's in the delta tables. Um we actually have to make the data usable. Um and I just wanted to these two pictures here. Uh that top one right is a fastly log a quick little snippet
[06:47] of a fastly log. Uh that's as the data comes into um ADLS. That's what it looks like in the in the blob. Um, and then the bottom is once the notebook does all its magic and puts it in the delta
[07:00] table. So that's that's kind of like the the final output right there that the applications and the dashboards are ultimately going to use.
[07:12] And this one right here uh is a dashboard that I made. Um the the black just excuse the black um boxes there. That's that's you know that's there's some potentially sensitive data I just wanted to to get rid of. Um but this is a uh a dashboard I built that's using
[07:26] that data that table uh that you just saw. Um and this is fullfeatured. You can search uh the fasty logs. Uh I think they go back right now in this table that that this is using I think it's
[07:39] 90day look back um using a materialized view which helps for more rapid retrieval. Um and you can see here that you can mix and match from all these fields. Um IP J3 origin etc etc etc. Um you can it
[07:54] supports full reax so you can really just sky's kind of the limit. So you can really get down you know down and uh you know try to explore what you really need to see. Um uh and then you can see here this will once it runs it will uh retrieve all
[08:08] these top stats. Um there's also another panel here that is for visualization. So this is how I monitor um our website traffic. So this one for example is for uh uh dubdubdub.jetu.com
[08:21] but this supports all of our uh web services and applications. So you can you can um get exactly what you need um by the by again the custom time range. You can choose really any any time frame
[08:33] that you want. Um you can see here it also sorts by origin and response code. So if we need to see if we're, you know, if we're in the middle of attack, an attack or whatever, you know, this is very useful for uh for that kind of stuff.
[08:46] And now I want to talk about uh a more recent initiative that that uh we've been working on uh with with oper operationalizing capture and bot detection. Um this is what I'm about to
[08:59] speak about now is only a few weeks old. Um uh and just a little bit on what I did to to build this application. Uh so I built it using um uh streamllet right in right in
[09:13] house and data bricks took me maybe a few days. Um combine SQL summary tables and historical telemetry. Um so the logs that this application is using are in a
[09:25] summary table a very fast one. So it retrieves things really really quickly. Um the the application supports capture analysis, spot detection, um anomaly investigations, and traffic exploration. It's also very easy to use as as I
[09:39] wanted it to be for the analysts who maybe aren't as technical. Um you know, it's just clicking buttons and and things like that to use it. Um and it's also rich with data insights. So, and I do have a little demo here. I just want to go through. I have a video that I
[09:51] captured just to show it off. Um so, this is it here. This is like when you first load it. This is what it looks like. You have some just some quick stats. Um, this is all these
[10:03] numbers all represent unique IPs. So 7,000 whatever challenged um and then you have uh uh in uh sorry valid requests there uh invalid tokens and so
[10:17] forth. Um and then you have um time range here. So if I can just go show this off. Sorry it's a little hard to see here. So you can see right here I made it so that you can choose really any kind of custom
[10:30] time range. 30 minutes to a custom range, right? If you need to like see a spike for example of a specific time, um specific day in a specific time window, you can get that granular if you need to
[10:43] as is, you know, the case with a lot of the stuff we're doing. Um I also want to show off the tokens by time panel um right here. So this shows uh you know
[10:55] the amount of uh obviously like a line graph of the amount of times people were challenged or uh were were given an invalid token meaning basically that an attacker or whoever a bad actor uh tried
[11:07] to hit one of our API endpoints without a valid capture token and they were blocked by by our W right. Um so this is actually a real a real world example. We were under attack by uh uh a Chinese ge uh a thread actor that was using a
[11:20] Chinese geoloccation. Not sure if they were actually uh you know in China. Very very doubtful. But um you can see here that this is right around the time that we turned on capture for uh dubdubdub.jetblue.com
[11:32] that um it just spiked with the invalid tokens because they were just rapidly trying to hit I think it was our flight search endpoint. Um and of course they never they didn't have a valid token. So they they were just completely blocked and um they it stopped the attack for
[11:46] for quite a bit. So that's awesome. Um, another thing I want to show is just I'll quickly just go through here. Uh, let me let this play for a minute. Uh, and you can just see there's a ton of other stuff I have here that I built out
[11:58] for our teams. Um, all kinds of heat maps. Go down a little more here. Um, sorry. There we go. User uh in paths with
[12:10] invalid requests, countries with uh some errors, uh, things like that. Um, and then one of the other cool things that I have in here too for operations team or whoever needs to report to maybe our
[12:22] senior leadership, um, is this export PDF function at the top. So, what this does is at the click of a button, it just downloads the PDF and you can see I open I'm opening it up
[12:34] here and it gives you the a report of whatever your time search was. So really handy um because our operations team um and some of our other crew uh crew members on the cyber side, they um they
[12:48] do report uh these metrics to our senior leadership. So I wanted to make it really easy for them to do this. So um really really happy with this with with how this turned out. So now let's talk about where we are
[13:00] now. So about a year and a half later since we started this project to build out all the stuff in data bricks um we've now reduced full dependence on expensive
[13:13] index sim storage models right like I mentioned before we were using just the sim now we have even more of our data in ADLS at the moment so we've cut cost significantly by storing it there right um and now long-term telemetry uh you
[13:27] know retention became operationally practical right now I have analysts that reach out to me and say, "Hey, I got to explore this Microsoft Windows event logs data from May 2025, right?" And I just either they use SQL to do it,
[13:40] right? I'll I'll guide them through how to do it or I in some cases I built applications for them um on demand for them to use it. Um so, uh really cool stuff. And also kind
[13:53] of one of my emos here with all this stuff was to make it as easy for them as possible because I understand like you know some of these analysts they're not um super technical right um so they may not know query languages uh like KQL SPL
[14:05] etc etc etc so all my stuff that I built for them is they don't have to know any of that stuff they just click they click around and they can get their results that they need and carry on. Uh and now so what's next in store for us? Um well as of right now we have 200
[14:18] terabytes of telemetry in our ADLS and obviously growing um as we start to onboard more and more sources. We'll probably hit a pabyte sooner than later and then obviously continue on from there. I think we have our ear our
[14:31] earliest data sets are going back to 2024 at this point. Um so it's really awesome to see that. Um we have multiple analysts um and engineers using um the apps and dashboards. Uh we're
[14:43] growing our engineering team. So, uh, we we just got someone who is very familiar with data bricks and he's, um, he's going to be working on building apps, too. So, really excited to get him going on that. Um, as we keep evolving, uh,
[14:56] I'm in the process of shifting all of our alerting to Lakehouse. So, in the past, we used our observability pipeline tool to do our our all of our alerting. Um, wasn't really working out too well. So, I've just started an initiative to um to uh to get all that housed in um in
[15:11] data bricks um and and email notifications for outages uh through through there. Um as well as I have a call next week with one of our compliance analysts. Um she is interested in using PowerBI
[15:24] um to to uh you know using a PowerBI integration to do stuff with the data. Not completely I think it's uh some of our uh might be our threat locker data
[15:36] or something like that that she wants to get uh her hands on but excited to to start that project with her. Um, and then in the future, uh, why not really get into AI and machine learning? Uh, we've done some stuff with Genie.
[15:49] Um, I feel like we could definitely do more with the AI side and as well as the ML side. Um, I just don't think that we necessarily have really strong use cases yet, but um, you know, things could always change. Um, and then, uh, we do
[16:01] we are right now in the private preview of Lakewatch. Uh, just got it integrated with some of the folks at Data Bricks over the last couple weeks. Have not really had time to start using it just yet. Um, but I'm really excited to get my hands on it and um, you know, uh, go
[16:15] through the journey with with data bricks as they keep developing this and make it even better. Um, and then eventually get uh, maybe our instant response team um, on board to use it since uh, it's very obviously cyber security forward and there's a lot of
[16:27] really cool stuff you can do with threat hunting and detections and things like that. Um, and uh, that's basically it. So I just wanted to thank everyone for coming here and listening to me talk. Um, really really appreciate it and I'll open the floor for any questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.