Beyond Monitoring: Lakeflow Observability for Operational Health

Summary

Lakeflow Observability helps data engineering teams move from reactive monitoring to proactive operational health management, using system tables that serve 18 million monthly queries at P99 latency under 10 minutes.
Key capabilities include a job health score built from six signals including anomaly detection, a job impact score quantifying blast radius and downstream dependency count, and detailed execution breakdowns for cost and performance analysis.
This video demonstrates Genie Code diagnosing a failed job through natural language, a query profile walkthrough for performance regression analysis, and Genie ZeroOps automated incident grouping and auto-healing for proactive pipeline maintenance.

Beyond Monitoring: Lakeflow Observability for Operational Health

It is 3 AM. A dashboard is stale, an SLA slipped, and everyone is guessing which job, pipeline, or dependency broke first. That is when surface-level monitoring stops being useful. You need observability that shows the health of your workflows and helps you find the cause fast. In this video, we will show how Lakeflow Observability helps data engineering teams move from reactive monitoring to real operational health. You will see how to monitor runs, tasks, dependencies, alerts, and health metrics across jobs and pipelines, and how Databricks system tables provide the historical depth needed for root-cause analysis, performance analysis, cost tracking, and reliability reporting at scale. We will also cover practical patterns for tracking streaming backlog, freshness, SLA risk, and failure trends, and show how Genie Code can help teams troubleshoot faster using natural language. Your 3 AM self will thank you.
Talk By: Theresa Hammer, Product Manager, Databricks ;

Chapters

FAQs

What is Lakeflow Observability and how does it differ from basic job monitoring?

Lakeflow Observability goes beyond surface-level alerting by providing historical depth through system tables, enabling root-cause analysis, performance regression investigation, and reliability reporting rather than just failure notifications. It helps teams move from reactive monitoring to proactive operational health management.

What are Databricks system tables and what operational data do they contain?

System tables are built-in tables in the Databricks Data and AI platform that capture operational data about jobs, pipelines, and runs with historical depth suitable for trend analysis. This video covers job, pipeline, and run system tables specifically, which together serve 18 million monthly queries at a P99 latency under 10 minutes.

What is the job health score in Lakeflow Observability?

The job health score is a composite signal built from six indicators including anomaly detection, designed to give teams a single at-a-glance measure of whether a job's behavior is within normal bounds. It is complemented by a job impact score that quantifies the blast radius of a failure by counting downstream dependencies.

How does Genie Code help troubleshoot failed jobs in Lakeflow Observability?

Genie Code can be asked in natural language to diagnose a failed job, using the job's execution context, error logs, and query profile to identify the root cause and propose a fix. This video shows a live demo of Genie Code diagnosing a job failure and walking through a query profile for performance regression analysis.

Full transcript

[00:08] Hi everyone. I'm Teresa. I'm a product manager working on data engineering offering, which we call LakeFlow, and I focus on observability. Um and today we're going to look um at beyond monitoring. And basically what that means is we're going to take a look at a couple of things that
[00:21] But before that, um before I get into all the details, um please complete your surveys. This is super helpful for us to know what content you like, what you want to see more of next year. So just a quick reminder at this point.
[00:34] Also, as always, a little forward-looking statement. And with that, let's get into it. So, I'm hoping you're going to walk away with understanding the following things. A very quick intro, what is LakeFlow, what do we mean by LakeFlow
[00:46] observability? What are the data foundations that power this observability? What's the evolution that we are working towards, which is smarter observability and making your lives hopefully easier?
[00:59] And then I'm going to cover what's next. Since we only have limited time today, I won't be able to talk about all the great things the team is working on, but I want to give you a little bit of a teaser. Um also, all of the slides you're seeing today and a recording will be made
[01:11] available after the summit. So, you don't need to keep taking pictures. Everything you see here today, you'll get access to. So, what is LakeFlow? As I said, LakeFlow is our data
[01:23] engineering offering, which is made up of three components. We have data ingestion, that's our connect offering. So, that's the managed ingestion connectors for database, SaaS, and file ingestion.
[01:35] Then we have declarative pipelines or Spark declarative pipelines or SDP or previously called DLT, this is what we're talking about for your transformations and making that much easier.
[01:47] And then finally, we have jobs to put it all together and to orchestrate. So, to to make sure that all the data is fresh when your your business users need it.
[02:02] So, what is Lake Flow observability? Um, Lake Flow observability, there is a cost component, but just to call out, the focus on today's talk is not cost, but as I tell you said, operational health. So, think about performance, failure patterns, optimization, those
[02:15] are more of the things we're going to be looking into. And what do we mean by observability? We have monitoring, we have observability. So, I think all of you in this room have yourself or have colleagues that have experienced the the
[02:28] scenario where it's 3:00 a.m. in the morning and you get paged cuz you're on call. And a data pipeline broke. And now you need to figure out what went wrong, which upstream system, which dependency.
[02:40] And who else maybe do I need to involve to get this fixed? So, what we mean by observability is for those kind of moments when monitoring is letting you know something's wrong and you need to figure out what happened. Observability tells you basically what
[02:53] happened, why, and what you need to do to fix it. Why is this more important now than ever? I think with the advent of agents especially I would say in the last 6 months,
[03:05] the cost of writing code has never been this cheap. And we're empowering much more users to also contribute. But by addressing all of these use cases, somebody needs to maintain them and fix them when they fail because they're
[03:17] powering important use cases. And I want to come back to the number that you also heard in the keynote. Over 50% of the time is spent on maintaining and debugging existing data pipelines.
[03:29] So, now imagine we're living in this world where code is super cheap. And now you still have all of this maintenance. So, how are you going to deal with this? And the answer is you need help. And that's what we're hopefully we'll be able to do with some of the smart
[03:42] observability features that we're working on. The story so far, because of course you know smart observability is great, but I do want want make sure to give a quick kind of overview that at Databricks today we already have a large variety of
[03:55] tools for different areas that are available. I'm not going to spend too much time, but we do have tools for discoverability, for notifications out of the box that are customizable using SQL. We have a bunch of data quality um
[04:09] solutions whether that's expectations into creative directly in declarative pipelines um or the data quality monitoring solution which tracks completeness and freshness using anomaly detection or DQx which is a very popular
[04:22] Databricks Labs tool. Um but we also have a bunch of um of course logs, metrics, historical observability data, anything you need. But as I said, this is the story so far
[04:34] and today I really want to focus on where we are going. So there's a couple of components to observability. So we have detection, we need to know something went wrong and something requires my attention. We have prioritization. If there's many
[04:48] different issues or if I'm working on a central monitoring or DevOps team, how do I know what to deal with first? And then we need to of course root cause and fix according to the prioritizations that we've put together. And while we do that, there might be
[05:00] this backlog that is creeping up on us where maybe our jobs are slowing down, but it's not yet dramatic. But eventually that could become a problem and lead to you missing an SLA so you need to make sure that you don't forget about that backlog.
[05:12] And so far as I said, we have all of these tools, but the biggest challenge is that all of you had to stitch this together. Um so that means basically you have hand like manually defined telemetry that you
[05:25] need to set up for every single new use case that you're adding that needs to be maintained over time. For prioritization, maybe there were already some mechanisms in place or you had to manually figure out when something went wrong, who actually
[05:37] relies on this table, how many people are querying them every day, how many other automated workloads depend on them. Or for root cause and and fixing it, you know, look at the logs, at the traces, at the metrics, put everything together. Some of it might be in the UI, some
[05:50] things might be outside of the UI, right? So, we're really trying to kind of bundle these things up together. And some of the concepts we'll look into is a new job health score, which is coming at the end of the month into preview,
[06:03] a job impact score, which is going to tell you about blast radius, how Genie code can help you for root cause and fixing things. And then of course, um very briefly touch on Genie Delta Ops,
[06:15] so kind of takes ties all of this together and takes it to the next level. But before we go there, um one thing that's very powerful is the raw signals in the data that actually powers all of these observability use cases, because
[06:28] observability is only as good as the data that it powers. It is powered by. So, our operational data data layer is system tables. And system tables is actually something all of you have
[06:40] access to. Um they are essentially live in the system catalog and by default, um available to any admin or admins can grant access to users. They contain a lot of like operational
[06:52] metadata that can help you address use cases from cost to AI to performance to security. It's really a large variety, but they curate it on the same place and
[07:05] they aggregate it across workspaces in the same region. Um why does this matter? This matters because historically, as I said, you we've had all these different silos data available in different
[07:17] places. And now, with system tables, we unified this into one kind of queryable layer that you have access to and that can power your observability, whether it's for reporting, dashboarding, alerting, or any other use case.
[07:30] Even agenda use cases require a lot of good data. So, let's take a look at what system tables are for Lake Flow. So, for Lake Flow audit engineering offering, we have two types of system
[07:43] tables. We have system tables about the metadata. So, that means the job system table, the job task system table, the pipeline system table. So, this tells you about the IDs, the names, the
[07:56] creators, the configuration. And every single time you make a change, we will add a new row tracing also those changes, which can be very helpful when you're debugging and you need to understand, "Hey, what happened over time?"
[08:09] And then we have the runs tables. The runs tables is the job run, the job task run, and the pipeline updates. And what that tells you is about your executions. It will give you metrics and data around
[08:21] the start time, the end time, the duration, the final status, the termination reasons, how it was triggered. And these are emitted once an hour. So, that means even for your long-running
[08:33] workloads, you will get updates as they execute. But Lakeflow system tables, they're powerful on their own, but they're most powerful when combined with other system tables. So, a couple of very popular use
[08:45] cases are, for example, use it like joining them with our billing table for cost analysis, understanding which workloads are maybe repeatedly failing and costing me a lot of money. Or um which
[08:58] um which jobs are running and updating on unused data sets. Or which jobs are taking a long time but are powering critical tables, and maybe I need to basic to to review what those bottlenecks are.
[09:11] So, for that we would need things like maybe the lineage table, the compute table to look at cluster utilizations if we're using classic compute, etc. So, all of this, I hope, sounds great, but um let's look at the numbers. System
[09:24] tables are very popular. In the last month, over 18 million queries were executed against just the Lakeflow system tables. And in the last 6 months, we've seen a 300% growth of the total daily query
[09:37] volume. And so next to this like very popular um growth and adoption, we're also working hard on making it better, especially bringing down latency. So, I'm very excited to share that right now
[09:50] our observed P99 latency is less than 10 minutes. So, that means from historically mostly powering only historical observability use cases, we're slowly moving into a world where we want to power more real-time use cases.
[10:03] And there's a bunch of improvements from the system tables team as a platform where we're going to introduce a data latency SLOs and also more low latency system tables to make sure that you can rely on our
[10:15] raw signals to power your observability. There's also two basically new things or that have in the last few months you might not know about um that I want to talk about because I think it's very powerful. One is that the job system
[10:29] tables are now generally available. And with that announcement, we also introduced the schema extension which added very valuable data especially on schedules and triggers around health rules which basically
[10:41] means um do I have alerts set up? So, will I know when jobs fail? Will I know when they're taking too long and I risk lose um missing an SLA? Or do I have a timeout set because I don't want to have bad code lead to a runaway job costing
[10:55] me a lot of money in the end. One other exciting thing is that we've also um added an execution breakdown. So, now you don't only see the total duration, but you can start analyzing where am I spending my time?
[11:07] Is that the startup time? Is it the actual execution? Is it the library installation? So, we we're going way granular now. Um one other kind of quality of life enhancement is that we've improved our
[11:19] slicing. As I said, you know, we emit every hour. So, as a result, they're now easier to align um with the billing tables cuz we've changed to kind of like clock hour slicing. So, as I said, I think I touched on a
[11:31] couple of cases. So, why does this matter? It means we can optimize cost by, for example, looking at the schedules and which jobs are running and not really doing anything useful. We can make sure it's reliable with all
[11:43] the right alerts being configured. And we can also make sure we maintain like the correct hygiene, you know, like is everything tagged? Is it running on an up-to-date um DBR version, etc.
[11:58] If you haven't used system tables yet and it's starting to get started, I think there's a bunch of ways that can really help you jump-start it. One thing is we have a GitHub repo with a system tables dashboard that covers a few of the most popular use cases around cost
[12:10] and performance optimization. Um however, we're also working on releasing a system table a skill specifically for Lakehouse system tables into Genie codes, so to make that experience even better.
[12:22] The second announcement I have is that for those of you who use already Spark declarative pipelines today, we have a pipeline event system table coming end of month to preview. Finally, I know we mentioned this last year, but we're
[12:34] we're almost over the line. Um so, why does this matter? For Spark declarative pipelines, the event log is the primary observability surface. It goes way more granular, like table level
[12:46] observability, rich metrics around data quality, throughput, backlog, um any kind of warnings, etc. are captured here. So, it's a really powerful tool for monitoring and debugging.
[12:59] The problem is it was available before, but per pipeline and there were a lot of frictions with accessing it. So, now we're putting into one system table, you'll be able to have great observability at scale for your
[13:12] pipelines. Um a few other things to call out, as I said, so you have observability at scale, there's no manual configuration, get this out of the box, it's aggregated per region, it has total coverage, no more thinking
[13:25] about, oh maybe I missed a pipeline in my custom setup. And we also have improved the query ergonomics by introducing variance and making all the nested structure easier to query. We also want to release a dashboard
[13:39] that's going to be a fast follow to the release. So, as you can see here just some examples. It's very similar, you know, we will get into more granularity on, you know, the pipeline health review, error distributions, performance
[13:51] benchmarking. But really, I think this table and flow level granularity of metrics is something that will be very powerful. All right. So, now we've talked about the raw signals.
[14:03] So, we know what will power our observability. But how are we going to move to smart observability? So, with everything I was telling you, yes, it's easier with agents to derive value out of it, but we
[14:15] want to help you. We want to make it even easier. So, the building blocks for smart smart observability all the way from detection to optimization that I'm going to talk about are job health,
[14:27] job impact, and also um Genie code and intelligent insights to start us off with. So, what is job health? Job health is a single score that is
[14:40] made up of six different signals to tell you whether a job requires your attention. It has three different states, so it can be healthy, warning, or unhealthy. It is rolled up, meaning if any of the
[14:53] signal, like worst signal wins. So, if any of the signal is unhealthy, it will be unhealthy. Um and also it takes historical data into account. So, there's a few of the signals we're looking at that looks at
[15:05] the last 25 runs. So, what are the signals that are included in job health? So, obviously, like the last run status, you want to know when all of your retries were exhausted and your job run failed.
[15:17] We have the failure rate. So, if the job's being flaky, it's very regularly failing, maybe you need to investigate if there's a root cause that requires your attention. We have a both user-defined thresholds,
[15:30] so if you're already using the duration alerts, that will be factored in. But we also have anomaly detection. It's at this point a little bit conservative cuz we're trying to figure out what's going to be the best fit for all of you guys. Um so it's going to look at
[15:43] duration that is more than two standard deviation away from the average of the last 25 runs. And then we're also going to detect um runtime outdated runtime version. The reason is that there was a change that
[15:55] we started to end of life old runtimes. And what that means is that once your runtime reaches end of life, your job will start failing. So it's very important for you to know about it. And finally, and this is something I'm very excited about, we're also
[16:08] integrated with our data quality monitoring solution, which monitors freshness and completeness. And so you get all of those signals in one single score.
[16:22] So this is pulse detection. Now, how do you know which one to prioritize first? For that, we have the job impact score. But what is the job impact score? So it's a lineage-based roll-up. We look at your jobs, what are the downstream tables the job writes to,
[16:35] and how many people or like how many queries are being executed against it by your users or automated workloads. So this is possible because, you know, Databricks, if you have a data on and
[16:48] the lakehouse, we have this kind of intelligent usage data about you. And finally, this should hopefully help you to do efficient triage. In terms of how this is derived, we actually also show you an impact score
[17:00] on the table level within data quality monitoring. So if a table is high impact, or at least one table is high impact, the job will be high impact because we really want to make sure
[17:13] that if anything's critical, it's not flying under the radar. But both of those, the health and the impact, are very new concepts. And we're going to launch them in a preview at the end of month, so we'd love feedback from customers to try it out, so we can work
[17:25] together and figure out how we can make it even more powerful.
[17:37] Oh. Okay, so my video is not showing. Can't see it.
[17:49] Okay, I wanted to show a little demo.
[18:14] Yeah, but it's not I can't click play. >> Oh, it's playing. >> Oh, it's playing. Oh, cuz I can't see it. All right, so then I need to come up here so I can better see you. Sorry about that. But essentially what you see in the list view that you know for your jobs and pipelines, um you now have
[18:27] um a health column and an impact column. The health column will be available as long as there are recent runs. And impact will be available as long as we have lineage data. And that basically allows you, I hope you all followed the video. I apologize
[18:39] for that little mishap. Um you'll be able to filter on those, combine them together, and have basically have a list to work through. So this is kind of like how we're going to start with the the detection, like the tragedy and prioritization, as a
[18:52] starting point. Of course, you know, as like as we evolve this product, we definitely want to have alerts as another entry point that help you kind of like do exactly that. But before we go there, we'd love some feedback on making sure that we are not just sending
[19:05] you alerts that are not helpful, but that we are sending the right alerts that will help make a difference. All right, so once you are in a score, we have a health score summary in there. So basically this gives you a list of
[19:18] all the signals that were not green. So you kind of like can quickly scan what's been happening and have a little bit of a summary. Of course, with everything being agent these days, this is integrated with the Genie code and you can that click on the
[19:31] diagnose button to have Genie help you resolve those problems. Um yeah, before I show the demo, just let's very quickly um take a look at
[19:43] what Genie code is in case anybody missed it. I know everything's been about Genie um this summit um but basically what is Genie code? It's it's it's our agent um that can help you to diagnose and fix your failed runs. It
[19:55] has special skills that are very targeted towards um fixing job failures. You can even edit the jobs by conversation. So both the the root causing and the fixing can all
[20:07] happen within Genie code. And this is uh in public preview for jobs and pipelines. All right, let's take a look at the demo of debugging a failed job.
[20:38] Okay. Okay, I I found out how to play the videos, guys. Now we're we're resolved then. >>
[20:50] actually the majority of my last 25 runs failed. So I click on diagnose to launch Genie code to help me resolve that problem. You can see it invokes the right skill and what happens in the background is that Genie code gets the
[21:03] context of the job health. You know, to cut down some of the guessing time, we start guiding Genie code in the right direction of what to look at. And in this case, in the summary we see a little hint that there was an out of
[21:15] memory issue, but was this once? Was this kind of like a recurrent problem? I don't know. I'd have to check all of those individual runs and the error messages to figure it out. So, maybe
[21:27] better just have Genie code do that for me. So, as you can see, it gives me a little summary. It it off like the the different signals, the root cause it is an out of memory error. And hey, this is actually the same error that's been
[21:39] happening in the majority of those failed runs. So, now Genie code goes off and comes up with um a solution, which in my case was to migrate over to serverless. So, I can edit this directly
[21:52] in the chat. There are some suggestions. And now Genie code is is doing its work and fixing my job for me.
[22:06] All right, it's thinking. All right, it's figuring out what it has to do. I can go into the settings. I'm just double-checking here. Hey,
[22:18] look, serverless has been applied and I haven't even like clicked on anything in the UI or left the Genie code context. And now I can also directly run the job from here.
[22:32] I will skip that because it does run successfully and it only took me a couple of Genie code clicks, um but it takes a couple of minutes to run. All right, so we looked at an example of a job that failed, right?
[22:45] But what about performance regressions? Cuz arguably those are sometimes very hard to catch and to understand what's going on cuz you really have to look under the hood. And I want to focus specifically on serverless because I know for serverless
[22:58] some of the feedback we've received how about observability does a lot of the low-level information I no longer have access to? So, I want to demystify this. So, for that I need to take one step back. So, the foundation for performance
[23:12] optimization in serverless is query profile. Actually, at this point, can I get a show of hands of who's using serverless? Okay, awesome. It's great to see. And who has heard of you or used the query
[23:25] profile? That's exactly what I mean. So, there's a big gap between customers using serverless and customers being aware of query profiles and that it actually holds a wealth of information for debugging.
[23:37] So, what is query profile? It essentially it tells you about the execution DAG. It kind of gives you all it gives you all of the query stages and the operators. And it has a wealth of um
[23:49] a wealth of metrics as well around the the memory, the time, the the rows processed, and even tells you what are the operators it took the longest. And it has some great features also for collaboration.
[24:01] So, this is just a little summary view. The actual query profile view is even much deeper where you see the the DAG of the operators. Um however, this is, you know, kind of like we saw the system tables. Now we're seeing query profile. This is the
[24:13] foundation. This is the data, but you still need to stitch it together to make sense of what's happening. That's why we have query performance insights, which are basically high-confident insights that
[24:27] Databricks derives for you based on the queries that your job runs. And with those insights, they're not only available in the query profile, but we've worked on making them available directly in jobs. What that means is if
[24:40] you're looking if you're looking into your job, you'll not only be able to see a visual breakdown of the time every single query took, but you'll also be able to see these little lightbulb icons and I'm going to
[24:52] show it in a little demo as well um to know basically, "Hey, there's some inefficient code or some inefficient configuration happening. But beyond those insights, there's also
[25:04] two other things were added. One is we're going to show you information about data volume, because that is something that you had you see you see it per query, but if you have a big task a multi-task job or a job with many
[25:16] different queries, it's basically impossible to easily like aggregate that information and also compare it across runs. And finally, this is more future-looking. Um maybe some of you know, if you're interested in serverless
[25:29] cost controls, rate limits are coming. So, it's going to be very important that you know, maybe my job is not inefficient, but it has been rate limited. And maybe I need to talk to my admin.
[25:41] All right, so I'm going to look at three three scenarios with you. So, one is I've introduced inefficient code. One is I've increased data volume, which started exposing inefficient code. And finally, my data volume has increased
[25:53] and there's a problem with the rate limiting. All right, let's look at the first one. So, inefficient code slowing down your job. So, as you see, we're back in in our job
[26:05] and we're back seeing how our health indicator telling me, "Hey, our our anomaly detection is actually figured out that your job is running slower than we'd expect it to run."
[26:24] You can basically then also Yes, you can go and expand to details to see a little bit more details. And once you jump into the job, you see basically the task run output for single-task job. That's all we had. You had the little performance tab, you could kind of
[26:36] drill into it, but it was a very manual scattered process. So, now for a single-task job, we made the timeline view available, which gives you a visual view of all of your queries. And here again, as I as I just mentioned, you have these little light bulbs telling
[26:48] you where you have to optimize. And it doesn't stop there. So, when you kind of like look into it, you get like plain text telling you what it is. In my case, a very inefficient joint, which I very much did on purpose for this demo.
[27:01] Um but you can go to Genie code and optimize your query. And that would basically launch this in in this case in a query and suggest the code to fix the code changes to fix it.
[27:13] So, this is yet another way in which Genie code can help you optimize your your code, your jobs more efficiently. Then let's look at this second scenario. So, now we have high data high So, we had
[27:26] inefficient code, but it's been exposed because there's now higher data volumes being processed. So, now you see this job, it was always running at around 1 minute, and I have a user-defined threshold. I dig into the details and I see, "Wow,
[27:39] it's taking 5 minutes now. That's like That's That's a big change." Um again, I could probably I could just use diagnose and that would help me figure it out as well, but if I want to just have like a quick spot check without even
[27:51] launching any agent, it's as simple as hovering over the the task information, where you see both an execution breakdown, which tells you where the time was spent. So, you can see "Okay, in the previous one it was spent running, and this is the data volume.
[28:04] And now I double-check with my really long run what's been happening, and I see okay, it's um actually also spent most of the time in running. So, it's not like maybe there was I
[28:17] don't know a new library installation introduced that's taking long. But the data volume has increased quite significantly. So, in this case, we would go back to the scenario before. I'd go into it. I'd also see the inefficient code, but in this case, it just wasn't a problem
[28:29] before because it was exposed now with the increased data volumes. The third example is, as I said, coming soon future-looking once rate limiting is introduced, where you see the
[28:41] execution breakdown and where actually the job spent time. There will be a little like speedometer icon. So, you've been slowed down because you've been rate limited. And that'll be a great limit great way for you to understand what a baby you
[28:55] need to adjust it or maybe it's okay. Maybe you need to adjust the schedule otherwise and and start it running earlier. Yes, so these were some of my examples for the um performance optimization. And
[29:07] as I said, we're going to take it one step further with Genie Zero Ops. These were just some some building blocks which are available. Actually, I forgot to mention both the the insights icon
[29:20] and the data volume information is available in Yes, basically all these smart performances is available in beta. There's two betas that need to be enabled, but I've put the links um
[29:32] enable instructions on the slide, so that will be made available to you. So, this is something you can use today. But all right, as I said, let's move on. Um where are we going next? Um next we're going to go towards an
[29:44] auto-healing platform. Essentially, this is what Genie Zero Ops does Ops does. So, we have the different building blocks we're looking into which are integrating with Genie Zero Ops. But instead of kind of like
[29:56] using those manually, Genie Zero Ops take it takes it one step further by automatically doing this for you on autopilot. So, to recap for anybody who maybe missed it in the keynote or who you know wants to have it summarized again,
[30:08] basically the Zero Ops value proposition is that it helps you to distinguish the signal from the noise. It does So, when you have like transient failures, right? Something failed and then it fixed itself. When you have many many many
[30:22] jobs failing, but they are related with each other, you get one entry for an incident that you need to review. Uh and not only that, it takes you one step further and actually works on a fix for you that you then maybe just have to
[30:34] review and accept. And finally, and I think that's where it becomes extremely powerful that it has the sandboxing environment to actually test out the fix and confirm that it works.
[30:49] And zero zero ups is coming to private preview in July. Yeah, that's that's it what I had for where we're going, smart observability. And now I want to talk a little bit about some other road map items that are
[31:01] exciting. So, when we look at serverless, I think one of the big thing a lot of customers have asked for is we will support exporting application logs to easy volumes. So, that's coming in the in the next few months.
[31:13] And in terms of the lake flow launches, also some very cool things are that we have a data latency metrics launching for Spark declarative pipelines. What this means is there is an event creation time column and basically we're going to
[31:26] calculate against that event creation column how much your data latency is. Which is very very powerful. And also one more thing is that we're also have more streaming metrics coming
[31:38] to query profiles. So, as I said, query profile is a great place to debug performance optimization problems and we're working on improving this the metrics and the support in the streaming world.
[31:50] But there's also a bunch of cool things for the platform. So, in the beginning we talked about system tables. So, some of the powerful features coming there is configurable retention. What this means is right now it's 1 year, but you could
[32:03] basically configure up to 10 years of retention. Then as I said, the data latency SLOs. So, jobs or lake flow is going to be actually one of the first candidates to start introducing that. And then also low latency system tables.
[32:16] And actually a fun fact is low latency system tables are powered by zero bus, which maybe you've heard about. And our events pipeline event system table is actually one of the first of those.
[32:29] And one final thing is there's also like if you care about platform observability, there's also the governance hub, which also helps you especially with a lot of like data governance questions, um, but it's also expanding to other use cases over the
[32:42] coming months. Yeah, that's essentially what I had. I wanted to make sure there's some time for questions. Um, here's a QR code if you want to let us know what you find very exciting in terms of like the
[32:54] things that you've heard of heard about in this talk, maybe any previews you want to sign up for, or any other feedback you want to share with us.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.