Databricks on Databricks: Automating Service Reliability at Scale

Summary

This video showcases two production reliability systems built using the Databricks Data and AI platform internally at Databricks: a KPI Miss Auto-Investigator that automates regression analysis and an anomaly detection system that identifies error incidents up to three hours before customer reports.
The KPI Miss Auto-Investigator detects metric dips, ranks error contributors, computes customer blast radius, and syncs annotations across dashboards — built using Databricks Jobs and ETL pipelines with inter-job dependencies — cutting on-call investigation time by 67%.
The User Error Anomaly Detection system ingests logs via Zerobus into Delta tables, monitors them with SQL Alerts and AI/BI dashboards, and achieves 95% recall and 98% precision while reducing incident discovery time by 95% and customer impact by 90%.

Databricks on Databricks: Automating Service Reliability at Scale

Rapid product delivery makes service reliability a core engineering challenge. This talk shows how we use Databricks on Databricks to build production systems that automate reliability detection and investigation for user-facing services.
I’ll cover 2 systems. KPI Miss Auto-Investigator automates KPI regression analysis by detecting dips, ranking error contributors, computing customer blast radius, and syncing annotations across team and global dashboards. Built with Databricks Jobs and ETL pipelines with inter-job dependencies, it cuts on-call investigation time by 67%.
User Error Anomaly Detection is a Databricks-based system for logging, detection, and monitoring. User error logs are ingested via Zerobus into Delta tables, analyzed by Jobs using Asset Bundles, and monitored through SQL Alerts and AI/BI dashboards. The system detects incidents up to 3 hours before customer reports, reducing discovery time by 95% and customer impact by 90%, with 95% recall and 98% precision.
Talk By: Hongwen (Olivia) Song, Data Scientist, Databricks ;

Chapters

FAQs

What are gray failures and why are they hard to detect?

Gray failures are incidents where a system appears healthy on standard monitoring dashboards — green CPU, normal latency, no alerts — but a specific subset of users or operations is silently failing. They are hard to detect because they have partial scope, affecting only a fraction of users, and an expanding blast radius that worsens over time without triggering traditional alerting thresholds.

How does the KPI Miss Auto-Investigator work?

The system automatically detects dips in business KPIs, ranks the top contributing error sources, computes the customer blast radius, and syncs incident annotations across team and global dashboards. It is built on Databricks Jobs with ETL pipelines using inter-job dependencies, and has cut on-call investigation time by 67% compared to manual analysis.

How does the User Error Anomaly Detection system detect incidents so early?

User error logs are ingested via Zerobus into Delta tables and analyzed by jobs built with Databricks Asset Bundles. Statistical anomaly detection models identify unusual error patterns well before the volume of customer complaints reaches a threshold that would trigger manual escalation, enabling incident detection up to three hours earlier than previous methods.

What performance metrics define the anomaly detection system's success?

The system achieves 95% recall — catching nearly all real incidents — and 98% precision, meaning very few false alarms for on-call engineers. Combined, these metrics support a 95% reduction in incident discovery time and a 90% reduction in customer impact, making it a high-confidence, low-noise addition to Databricks' reliability engineering toolkit.

Full transcript

[00:08] Hello everyone. It's so great to have you all here today. How's everybody feeling right now? A bit of f coma. I see that. Well, I hope I can have your next 20 m minutes well spent because we're going to talk about a type of
[00:21] issue that every companies could have but not all of them are handling it well. My name is Olivia. I am from data science team at data bricks and I want
[00:33] to start that with a story. Imagine it's a normal Wednesday. Every engineering dashboard is green. Your CPU looks healthy. Latency is fine and servers up. The database is connected.
[00:47] So by every signal your team usually watches, the system appears to be working perfectly. It's not because at 9:30 in the morning, a routine deploy introduced a subtle bug
[01:01] in the check of flow. And since then, one in 20 customers trying to pay with credit card has quietly failed. After a few retries, the fact customers
[01:13] eventually gave up. The first customer complaint comes in around 12:40 saying, "Hey, I can't pay." The complaint goes to support team who's busy dealing with another incident at
[01:26] the moment. They look at the ticket. Their f first thought is maybe this is another customer mistype their credit card number just like the complaints they've seen before.
[01:38] Then two other complaints and the same issue comes in at 2:20 p.m. which catches attention of the lead of support team and they decide to escalate it. By the time engineers track down the
[01:52] actual bug and ships some fix, it's nearly 4 pm in the afternoon. So it took it took seven hours for engineering teams to finally be aware of the issue.
[02:06] For seven hours, your system was telling you everything is fine, but customers were leaving and revenue was leaking. while support was slowly discovering
[02:18] what monitoring missed. And that's the scariest kind of incident. Not the one with loud alarm which which is something you can notice
[02:30] instantly but the one where dashboard are smiling but your business is hurting. So that type of story is an example what we call a great failure. On the surface, everything looks fine, but
[02:43] underneath some specific piece has quietly stopped working and it hurt your customer without chipping an alert. And that's my topic today because once you start looking, you find great failures
[02:55] everywhere in every industry. I bet you already picture one for your own. So I'll walk you through the framework we use for these inside data bricks and how you can point at whichever matters
[03:09] in your business. They've got two feature that makes them sneaky. First, they're only partial. On the previous story, it wasn't everyone.
[03:21] It was just the customers on one type of credit card I cannot pay. It's not like a server crashed through cash instantly. It sits in the messy middle where most people are fine and everything looks
[03:33] healthy while one group has been suffering the whole time. And second, they grow. It might just start with a handful of customers, but if you leave it alone, more and more customers will hit it. A
[03:47] great theater is like a smoke behind the wall. From the outside, the house looks just fine, but inside damage is spreading. And the longer you wait, the bigger blast radius gets.
[04:00] So how do most team handle this today? Honestly, pretty much the way I describe as start. You wait for your customers to tell you. Customer reports are valuable because
[04:12] they are real human pains. But customers should not be your monitoring system. And there are a few problems with that. For one, it's manual. Somebody actually
[04:24] have to spot the P pattern cost a bunch of support tickets and that's easy to miss. It is slow too. By the time enough people have complained for anyone to
[04:37] notice it's been some days or at least a few hours and that whole time most of your customers are suffering in silence. A lot of them never even bother file a
[04:49] ticket. So a better solution isn't to stop watching such tickets. You keep doing that. You want to have something automatic that catches specious
[05:02] anomalies. So it's running all the time. It picks something up way faster like minutes rather than days. And it can notice anomaly on its own. To be more concrete, it should fire and alert when all of a
[05:16] sudden a lot more customers than usual are hitting the same issue at the same time. So at data bricks, we built radar to
[05:28] close that gap. It stands for reliability anomaly detection, alerting and root cause analysis. We call it radar because when
[05:40] reliability when when visibility is low, you do not want to wait until somebody hits you. You scan for weak signals early. And let me walk you through how it works.
[05:52] For us, a great failure usually shows up as a spike in user errors. So, for example, all of a sudden, a bunch of people can't spin up a particular type of cluster over one region in East
[06:05] Coast. They're failing because of an error coded as invalid argument, which sounds like the system is saying their customer is probably your fault. But when a lot of users suddenly get the same your
[06:19] fault error, at the same time, it start looking more like our fault and we want to catch that. So what we do is at every point in time,
[06:31] we record two things. one, how many out how many errors are happening and how many separate users are running into the same error and then we break them down by error code and by region. So now we
[06:45] have a bunch of time series and then we run unsupervised streaming anom detection on top of each of these series so that it can flag anything that looks off without us having to sit there and tune a pile of threshold. And
[06:58] specifically we use spa model as a model and past 14 days of data as a model baseline. And then when whenever something does fire the alerting layer takes over. It
[07:12] puts in some actual context, futures all the stuff that isn't really significant and ddupes it so on call is not getting hammered with the same alert over and over and then it files a ticket straight
[07:26] to the right engineering team who's responsible for that user error. And every ticket comes with anomaly deep dive information and a link to a dashboard that's enhanced by our large
[07:38] language model assistant Genie. So whoever who's on call can go straight into figuring out what actually went wrong in the least amount of time. And that's the whole loop of radar.
[07:52] Before radar we have to wait on customer tickets from problems in gaze which eventually leads to wide blast radius of incidents. With radar, the great failure discovery
[08:04] is automated and 95% is faster than before, achieving more than 90%age precision. And since the incident discovery time is fast, we keep blast
[08:16] radius contained. And now you might be thinking h how can I make use of this radar? The thing is radar really doesn't care
[08:29] what the magic is. we just happen to point it at our user errors, but for you it could be just greater on your billing or on your customer satisfaction, on
[08:41] your model performance. So basically anywhere great failure happens, anywhere you've got something that could quietly go wrong. And the nice part is that everything you
[08:53] need to build this is already sitting there on data bricks. You basically already get all the pieces. With zero robust, you can achieve low latency data inject ingestion, unity
[09:05] catalog for data governance and delta lake for data storage. You train your model on ML flow and serve your model with model survey and orchestrates the whole detection
[09:17] workflow using workflows. The alerting is enabled with SQL alerts and root cause analysis is enabled on Genie enhanced AIBI dashboard
[09:36] and all of these can be deployed using data bricks asset bundle that's just be skipped let me see if I can get it showed here that complete my slide so with just one command you can deploy everything on
[09:49] your workspace. So, the one catch is that wiring all these things together could be a pain because there are lots of moving pieces. But don't worry, we got you. So, what we
[10:04] did is that we took our internal radar system, we distilled it down and published a scaffold. It's really just a single markdown file, but it works like a recipe. It maps
[10:17] every part of a radar to a specific data bricks configuration. All you really have to do is to bring your own metric whatever your signal
[10:29] happens to live and hand it to and hand the magic and scaffold to an AI agent and it will build the whole radar system for you live on data bricks
[10:46] and I'll show you how it works. A quick thing before I get into it, everything I'm about to show you, including the code, the prompt, and the scaffold, is all going to be in a cure code at the end of the presentation. So, you don't have to copy and paste anything down.
[11:04] I am going to demo how to let me know if you cannot hear it clearly. >> This mockup file is the scaffold. It documents the system structure that we just went through and explicitly maps
[11:16] each component to a datab bricks product. You can think of it as a recipe the agent will use to cook the meal. And this is the prompt that I will use.
[11:28] This prompt tells the agent to read the scaffold and build the system. It says read the scaffold at this task and build radar system for the metrics defined in this config. and deploy it to my data
[11:41] bricks workspace. I have copied and paste it to my large language model in session and filled in the configurations. For this demo, I'm going to continue
[11:53] using the payment failure as the example metric and we will also have additional dimensions for root cause purpose. For example, I added region, payment method, and product line.
[12:08] And I'm going to hit run. Last time when I ran this, it took me it took about one hour to build and validate the entire system end to end. The agent also handles the deployment.
[12:22] So by the time it's down, the system is live in the workspace. There's no manual deploy step. For time sake, I'm going to show you the
[12:34] results I got from last time. The agent eventually creates these files in my datab bricks workspace. They are under this dot bundle directory
[12:49] meaning their datab bricks asset bundles that defines the radar product project. This is the synthetic data generated by the agent that contains payment failure
[13:02] events and additional dimensions and a metric bill is created on top of the raw data to compose the hourly total payment daily account.
[13:16] In the next step, the metric view is read by the anomaly detection job that runs hourly. What it does is that it applies a
[13:29] statistical anomaly detection method on a metric and save the detected anomalies into a table.
[13:41] This is the model serving endpoint of the anomaly detection model that the jobs uses and anomalies are saved in the table called detected anomalies. And next you want to alert on these
[13:55] anomalies. So here comes SQL alerts. It queries the detected anomalies table using this SQL query and send alerts to the recipient.
[14:07] I put my email here but it could also be linking to other incident management tools like page duty. It runs every hour but you could also
[14:19] make it run more frequently. So what's behind is that it's essentially just a SQL query that you can make it as complicated as you want to achieve anomaly filtering based on
[14:34] significance and duplication and you will send alerts when the query returns true. You could also customize the alert content. For example, add a URL of the
[14:49] investigation dashboard. In real use case, an enco engineer will receive an alert with a link to a dashboard that realize relevant anomalies
[15:01] like this one. So in this chart, it shows a number of payment failures per hour. The blue line is the original value and the yellow represents anomalies.
[15:16] It also shows the same data breaking down by different dimensions like region, human method and product line.
[15:28] Instead of manually checking these breakdowns, you could ask in human language what what caused the most the recent anomaly.
[15:51] You will cring the underlying data and find the data slides that contribute the most in the increase in pain failures that we just observed.
[16:04] Last time when I asked this, it take a few minutes to finish querying the data. So I'll show you what I got last time I asked.
[16:20] So last time I asked what caused the most recent anomaly in this dashboard and it precisely finding the cost. It's mainly happening in the US east region
[16:33] and it's driven by credit card usage and home is the product line that is most affected.
[16:45] Knowing that root cause engineering team can now quickly find the bug and fix it. And remember all these things that I just show you are configured through data bricks asset bundle with these
[16:59] config files. All these assets can be deployed directly. And that's everything I want to show you.
[17:13] Okay. So the video the audio is kind of mumbled. It's my trick so that you're paying attention to what's saying there. Uh this is my last slide. The demo has showed you how straightforward it is to build a radar system using just one
[17:26] prompt. And if you want to work out this room and thinking that you have something takeaway one, you can use the radar framework to catch great failures before they escalate.
[17:39] and radar stand for reliability anomaly detection alerting and root cause analysis and two you can use scaffold to build
[17:52] radar on data bricks and if you want to get started today everything is behind this QR code the scaffold the demo the code and the prompt all of it so scan it out and try it on your own ideally at
[18:06] the end of the world we don't have to let our customers tell us anything's wrong. We discover any incidents by ourselves. And thank you. To the time sake, I'm going to take questions
[18:18] downstairs.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.