Genie, Who Broke My Cluster? Using AI Functions, APIs and Genie Code to understand your workload

Summary

This video shows how to set up a Genie space on the Databricks Data and AI platform to answer real operational questions—such as which jobs are over-provisioned, which dashboards are actually used, and what caused a cluster failure—using system audit tables.
Five best practices for an operations Genie space include selecting the right source tables (jobs, clusters, warehouses, usage), flattening data with materialized views for faster responses, and scheduling an automated weekly refresh job to keep data current.
AI Functions and custom instructions extend Genie's analytical capability, enabling joins across audit tables and dashboard metadata that surface hidden costs like job retries and warehouse over-provisioning.

Genie, Who Broke My Cluster? Using AI Functions, APIs and Genie Code to understand your workload

Have you ever wondered what happened in your Databricks environment? Have you spent time looking through logs and audit tables? Join us for this live demo of using Genie code to gain insights into your system. If Genie is new to you, you'll learn how easy it is to get started. If you are already a Genie expert, you'll discover ways to improve and enhance your experience using APIs, joins, AI Functions, and more. Take the guesswork out of auditing by setting up your own Genie with the real-world, practical examples shown in this video.
Talk By: Shelley Shaffery, DSA, Databricks ;

Chapters

FAQs

What types of operational questions can Genie answer about a Databricks environment?

Genie can answer questions like why a cluster stopped running, what compute type is right for a given job, which dashboards are being actively used, and which jobs are retrying and generating unnecessary cost. These insights come from querying system tables covering jobs, clusters, warehouses, and audit logs on the Databricks Data and AI platform.

What are the five best practices for setting up an operations Genie space?

The five best practices are: select the right source tables (jobs, clusters, warehouses, and usage data), flatten complex nested structures using materialized views for faster Genie responses, set up an automated weekly refresh job to keep data current, add AI Functions for enhanced analytical capability, and provide custom instructions that guide Genie toward operational interpretations.

Why should Databricks compute configurations be reviewed every six months?

Job data volumes and processing requirements change over time—more data, schema changes, or updated business logic can make a previously correct cluster configuration over- or under-provisioned. As shown in this video, regular review prevents silent cost inflation and avoids performance degradation, especially for jobs not using serverless compute.

How do AI Functions extend what Genie can do with audit data?

AI Functions allow Genie to perform natural language analysis on top of structured audit data, such as summarizing error patterns across cluster logs or classifying job failure types. Combined with custom SQL instructions, they enable Genie to join audit tables with dashboard metadata and surface governance insights like who is accessing which SQL warehouses.

Full transcript

[00:09] Thank you everybody for coming. I know it's late. I'm the last session. You're tired. It's been a long day and I'm probably in between you and dinner and alcohol. So I get it. So I will make this as entertaining as possible as we go through.
[00:21] All right. Uh forward-looking statement, you've probably seen this like a dozen times. I'm going to tell you it doesn't apply here. At the end of this, you will have a QR code. You can have my code. You can have my slides. You can have everything.
[00:33] Go take it home, deploy it. It should take you less than a day. If it takes you longer than that, ping me. I would like to help you. Okay, and there's surveys. Please do the
[00:46] surveys. I don't actually know what happens, but please do them. Okay. So you are here for Genie, who broke my cluster, audit tables, APIs, and all the wonderful stuff.
[00:59] Me. Hi, I'm Shirley Sheffry. I am a delivery solutions architect in startup and emerging enterprise. What that means is my job is to make things go well for you. I want to help you get through hurdles,
[01:12] roadblocks, whatever, and actually deploy something. And the key point here, deploy something that you're not going to be angry at me later for. Something that you can manage and deal with day two and isn't going to give you
[01:24] pain. Because I was day two for a very long time and I do not want that for any of you. On a personal level, I'm an avid reader. I love books. I'm reading Discworld.
[01:36] There's 55 of them. I don't know if I will ever finish. If anybody wants to talk about any of that, come see me. Um married, teenagerish children. My husband's actually here as a customer, but not here at my session
[01:49] because he didn't book it and he didn't think he'd get a seat. So I'm going to yell at him later. Uh we have a rescue lab. Hits him up in the corner. Enough about me. We don't have a lot of time. So, these are all legitimate questions I
[02:01] get from my customers. Why did my cluster die? What's the right compute for this job? I cannot tell you how often I get what the right compute for this job is. And in case you haven't guessed it, it changes. It could be the right compute
[02:14] right now, and then 6 months from now your job could change. It's more data, it's less data, something has happened, it's not the right compute anymore. That should never be static unless you're using serverless. If you're using any other kind of compute, you should be
[02:27] checking up on it. Um what does dashboard usage look like? We built these beautiful dashboards, but Sarah in sales is still asking me questions. Is she using the dashboards? Are they doing it, or are they just coming to me
[02:39] cuz I'm the faster answer? So, I want you to be able to answer these questions. And I want you to be able to answer them when you go home. So, Genie spaces. I'm going to assume everybody knows a little bit about Genie
[02:51] spaces, but just in case, they're marketed as ways for us to talk natural language for our business teams to get answers out of their data. So,
[03:03] they can talk like a human being to this thing and it will give them answers about data. That's right, but I also want you to think about it for technical teams. We don't usually market it that way, we
[03:15] don't usually talk about it that way, but you can set up a Genie space to make your life easier, not just your business users. And that's what we're going to do today, and that's what we're going to demo, that's what you will have code
[03:27] for. Five major best practices, just so you level set that you're setting up a good Genie space. And there's a link at the bottom, it'll take you to the documentation. Again, you'll have my slides.
[03:40] Metadata enrichment, first one. Very simple. Your tables have descriptions, your columns have comments, you have tags, you have lineage. Some of that you get out of the box just by using Databricks. Some of it
[03:53] you have to curate for that content. Second, optimized queries. If you have a bad query that you're going to use, Genie's going to be slow. It's running
[04:05] that same bad query. So, if it's a giant join to answer a simple question, we need to make that better to start off. Instruction clarity. You're going to give Genie a set of text instructions.
[04:17] If you make that confusing, if you make that filled with a bunch of jargon it doesn't understand, it's not going to give you good answers. So, instruction clarity is important. Business semantics. This is the only
[04:29] thing we're really not going to get into here because we're not building a business Genie space, we're building one for you guys, a technical one. So, I don't have to care for that, but if you want to talk about it in detail, please come see me. High level, you want to be able to give
[04:43] a Genie space for your business teams things like what are my acronyms that I use? What's my calculation for sales commission? Things that are very specific to you. There's different ways we can do it,
[04:56] metric views, knowledge stores, glossaries. Again, if you want to talk about that later, come talk to me. I don't have a dinner date till 6:30. I have time. And then data conciseness. I cannot tell
[05:08] you how many bad Genie spaces I have seen where we just shove everything into it. Here's all my tables. Why are you slow giving me an answer? Because it's scanning all your tables. Try to make a good decision. Data conciseness is
[05:21] important. So, we're going to limit what you give to it and we're going to clean some of it a little bit differently so it's easier for Genie to use. All right.
[05:34] Core of what we're talking about here for our technical teams, audit and system tables. These come as part of your Databricks installation right now. I think there's like 39-ish system tables that you could use
[05:46] Last I checked. But these are read-only tables that the system uses. Audits, job runs, what your compute looks like, all that kind of data, including billing, so you can also get
[05:59] costing out of this. We will look at that when we set our demo up. So, super easy to use. Guess what? It already has all the metadata stuff you need. So, super easy. We're just
[06:13] going to plug and play this. But, our problem is system tables are again for the system. We want to talk in human language. I do not want you to give me an idea of a
[06:26] dashboard. If you give me an idea of a dashboard, my display has names. I have no idea what we're talking about. So, I need you to be able to tell me things in a human way I understand that makes it useful for me. So, that means we have to
[06:39] do two things. We're going to have to build some dimension tables using the API, super easy. We'll look at it in the demo. So, that we then we get actual names for certain things. And we are going to do a
[06:51] materialized view, so Genie isn't reprocessing these giant joins, and we're going to flatten some data, so it's easier for us. Cuz a fast Genie is one you will want to use. A slow Genie
[07:03] is going to make you annoyed. Demo time. Cuz I want you guys to see this, and I'd rather have you guys having fun with me.
[07:16] All right. And I'm just going to make this big, because if I don't, I feel like you can't see anything. Is that better? All right. That looks good. All right, so we're going to start off, and I'm
[07:28] just going to grab a question here, and this is a legitimate question from my system. Man, my gosh. All right. Cuz I want to put a question in live. And actually make that just
[07:40] I want to make this just a tiny bit smaller if you guys can still see cuz otherwise we can't see the whole screen. There we go. All right. We're going to ask a question. So, this is a legitimate question I get. Is my compute size properly for this
[07:53] job? If not, what should I change? I get asked this a lot. If you can answer it yourself then I don't have to, I can help you fix bigger problems. I like that. I like bigger problems. So, we're going to kick that off. While
[08:06] that's running, we're going to talk about how we built this and what it looks like. So, first thing, under our configuration we're going to drag this across. We're just going to let that run in the
[08:18] background. So, uh da da So, there's some basic stuff in here, common questions. I'm worried about that. We're going straight into the data. So, you will see these system tables
[08:31] here, quite a few. Not as many as you think. Jobs, pipelines, clusters, warehouses, and usage. This is the cost one. So, if you want to ask a question about cost, that's the one you add.
[08:45] And then we have these other ones we added, these dimension tables and our materialized view. So, let's just take a quick look. We're going to my catalog. Open a new link.
[08:58] All right. So, let's start here actually. So, you can see this is one of our system tables. Job run timeline. Here's our metadata. We've got description. I've got my comments. I don't have to care about any
[09:12] of this cuz it's already done for me. If I make the view just a tiny bit smaller just for a second, there we go. You can see it's got tags, joins, lineage. Metadata taken care of. Best practice
[09:25] step one done. Yay. Now, we also have to care about those other tables that we're building, right? Our dimensions and our materialized view. So, we did this super easy.
[09:39] I've got this weekly job that runs. Doesn't take very long, 3-4 minutes. It is currently set up. All right. It runs an individual
[09:52] notebook and we can see right here for each table that we did. So, we did a dimension for dashboards, notebooks, and users. Those were the three we cared about right now. So, there's three phases. Whether we do a seed data that initial
[10:05] run, which is going to take a little longer to get all those names of all the dashboards, all that extra data we want. But, then from then on, it's short. I know this is saying 30 seconds, but remember it's going to that run is for the notebooks, the users, and the
[10:17] dashboards. It's doing all three just back-to-back. So, that's why it's about 3 minutes. So, you can have this code, but essentially what it's doing is it's reaching out to the API. You'll see it
[10:29] down here. Reaches Sorry. I clicked too far. I clicked too far. All right. So, where we I'm in the wrong one now. There we go.
[10:41] So, it reaches out to the API. It's going to first check the event log. Is there a new ID in here? Yes, there is. Great. Let me check. Do I already have a dimension table? Yeah, I do. I don't
[10:53] need to look this up. Continue on. Oh, no, I don't. Let me go look it up through the API. Add it to dimension table. Done. This runs in the background once a week. You don't have to care about it once you set it up.
[11:05] And then you are solid. For our materialized view, same deal. This runs once and then it stays in sync. We gave it the it's a flattened view. So, the audit system table has
[11:18] embedded struct data. So, we just want to flatten that out. I want to be able to have a column that says dashboard ID instead of requested join more complex
[11:32] and it just makes the query a little slower. Remember, we want clean, quick data. So, by just doing this view, we're now making an individual column for all those ID numbers that we need to match on to be able to answer our questions.
[11:44] Super simple. You do it once, it's done. It stays in sync, you don't have to do anything else. Joys of materialized views. All right, going back to this guy.
[11:56] Instructions. This is all in the package I'm going to give you, so you can copy and paste the instructions right out. Very easy. It gives us an idea of what the data looks like, what our joins look like,
[12:09] what its job is to do. You're going to answer questions about this workspace for me. Uh how you're going to behave. There's a critical message here that says, "You're going to use the view
[12:22] I created, not the audit table." Because we didn't even give it the audit table in the data. So, we want it to be very clear that this is where you're going to go for this. Okay? Very simple.
[12:34] We've added the joins in already. That was one of our best practices, having the joins, having the easy queries. Nothing hard here. If we look at one up close, there's our view.
[12:47] We're going to match it to our dashboard on dashboard ID and dashboard ID. Like, none of this is super complex. And just so when you go home, and you've listened to a lot of these
[12:59] sessions, and a lot of these talks, and you're jet-lagged, and you don't remember how do I did did this, and this looks like magic now, it is all in this wonderful file right
[13:11] here that you're going to get. All the joins, everything. I really, really, really, really, really want you to go home and be able to set this up and run it for yourselves. I really, really want your jobs to be easier.
[13:24] Swear. It is my That is literally my job definition. I I make your life easier. Okay. So, these are our joins. And then last, SQL queries.
[13:36] So, essentially, you are teaching your Genie to say, "If somebody asked that question, what's the dashboard activity over the last 7 days?" This is the exact query I want you to run.
[13:49] Important things I'm going to say, Shelley-isms, you can ignore me or not on this. But, you'll see I have it filtered based on the workspace ID. This Genie space is for my production. I
[14:01] mean, it's my dev, but for my production workspace. I do not want to ask, "What are the top five dashboards?" and see stuff come back from QA. That doesn't help me. I want to limit it to this particular workspace. So, you'll see that in some of the queries.
[14:14] Take it or leave it if you don't want that piece. Other Shelley-ism, and I don't have it set up this way cuz it's dev, but you should do this. I don't know if it's in any best practice documentation, but you should
[14:27] do this or you will probably have pain later. Separate out those dimensions and the view tables. A lot of my customers will make a separate catalog. It's an admin catalog. If you can't make a catalog, you're
[14:39] close to your limit, make a separate schema at least, and put them there. There is no reason your business analysts need to go to your dimension tables. There's no reason for that data to be shared. So, you just want to be able to segment it easily, make your
[14:53] lives easier, do not be at a table level restricting access. These aren't that big of a deal, but I promise once you start building stuff like this, you'll find a new use cases, and you'll need a home for them. So, if
[15:06] you set it up the first time, you will thank yourself later. Future you is happy. I like future you to be happy. Okay. Let's go back over and see what Genie came back with.
[15:18] Mhm. Mhm. And I'm actually just going to close this. Okay. So, we asked it if this job was messed if had bad compute specifically. And you
[15:30] can see right here the thinking process it went through. That's a lot. Like it did a lot of thinking. It didn't just come back with some quick, you know, yes, no answer. And if we look, so it's looking at this
[15:42] is the cluster we're using. This is the driver. It's set to auto scale, but it starts at three nodes. And it runs hourly. So, if we look at it, we have a max CPU, average memory,
[15:56] average CPU. 4.31 is your average CPU, you're probably doing something wrong. So, if we look down here, you can see the little spikes in it. It gives us visualizations. We didn't even ask for
[16:09] that. But then it gives recommendations. So, right off the bat, you can go, "All right, this one's too large, so we're going to back it down. We don't need that big of an instance for what we're doing."
[16:22] And it's looking at this job I think it's been running for almost 2 weeks. So, it has a significant amount of data for it to be running about hourly for about 2 weeks. Uh changing our auto scaling from three
[16:35] to eight workers to two to four workers to make this better. Suggesting a spot incident. So, it gave us valid recommendations right off the bat. You do not need to be an expert in the compute. It will help you.
[16:49] If you give it the right tables in that data set up, it will help you. So, I generally recommend my customers are looking at any job that isn't serverless
[17:03] or any pipeline every 6 months to make sure it's sized appropriately. And that's both sides of the thing. General rule of thumb, if your if your average is somewhere between 40
[17:16] to 70, you're probably okay. If your average is above 70, I worry. If your average is below 40, you can probably drop it. These general suggestions here. But,
[17:28] this is looking at that same data set and making those decisions and trying to help you. So, super easy, super super fast. Now, I don't want to ask a thousand
[17:40] questions of Genie while I run here cuz it does spend a lot of time thinking about it, but I want you to see a few important things. Sources down here. So, we talk about this, but we don't
[17:52] ever actually click into it. If you click into it, it will show you what it had and the code it ran to get that answer. I point this out because if you're
[18:04] asking the same question of your Genie over and over again, steal that. Make a dashboard. Make it easy for yourself. I don't even have to run the Genie in some cases. I'll just have a dashboard for it to to check this stuff for me. Super easy.
[18:19] It's there, you just have to look for it. And you can download this. So, if you're like, "Oh, I want to run that query again sometime on my own just to see if I've made it better." Download it. Save it.
[18:31] So, super helpful. I love that they give you that. It also has this lovely history here. So, I ran a whole bunch of different questions on it over the last couple of
[18:44] days so you guys wouldn't have to sit here, but you could see some of the cool stuff and some of the answers it comes back with. So, we're just going to pull up an old one. I want to go back to the top. So, this first one,
[18:57] I asked it, "Please provide me a list of the most used dashboards in the last 30 days." So, if we look, we got some graphics, key findings, where it is. What I want to point out here, one
[19:10] source. And if we look at it, most used dashboards in the last 30 days. I would quiz you if you didn't have headsets on cuz I think that'd be fun,
[19:22] but I'll just tell you. If you come over to instructions and you look at the SQL queries, what it's referring to, right there. That one source, it knew from the question I asked that it had the exact
[19:36] query it needed to answer it. It didn't do a whole lot of thinking, a whole lot of time. It just went and grabbed my exact query. So, it can be really fast if you set it up right and you're getting repeated
[19:48] questions. So, that's a dashboard question, but there's a lot of questions we can ask. Notebook attachments. Who's attaching a notebook to my compute? I want to see. So, if I could see like a shared clusters being used a lot, I can
[20:01] actually ask it and it'll tell me Sam's attaching this notebook to that cluster and running it. I can even ask it to tell me how much money Sam spent. So, really cool cuz we added that usage
[20:13] billing. Just remember, if you add that one, you can ask it about costs. Uh I think I asked a different cost question later. Job failures in the last 4 days. Now,
[20:27] I know we made an announcement earlier about the zero ops and that is future and not something I'm going to talk about too much, but that's supposed to help you when a job completely fails, right? Job fails,
[20:40] figure out what's wrong, give recommendations. What I like about this, a lot of your jobs are set to retry. If a job runs and fails, it retries and it works, it's a green checkbox in your
[20:53] view. You don't see that failure unless you're looking for it or you put up an alert for it, which you could do, but most of my customers don't. They don't think to do that. They think to put an alert on if the job fails, not if it retries and passes. And that's
[21:06] different. So, you could ask that question up here. You could say, "Show me all the jobs that have had a retry. Show me all the tasks that have had a retry." You can even ask it, "How much did those retries cost me?" And it will
[21:19] tell you "You retried this job four times." Or maybe I have one customer every time there's a retry and then it's successful. So, 13 minutes, it fails, it
[21:31] retries, it works. Every single time. That job runs every half hour in their system. So, there's 13 minutes of processing every half hour that does nothing.
[21:44] And that's up. So, you can ask the Genie space and it will give you that information. Which I think is really cool if you look at the job runs. So, I think I asked Yeah, successful after a retry. You can
[21:57] ask that. You can ask the cost. So, I have a lab. My retries are very small. But, it's able to tell me based on the retries, that's how much it was costing
[22:09] me. 18 cents isn't a big deal. In your real world environments, that can be a lot higher. I have done an audit for a customer that their retries were costing them about $800 a month.
[22:22] So, depending on what it is, it can be significant. Okay. I just think that's one of those hidden things you don't see. You green check box and I'm not looking for a failure in
[22:35] that. Another one. Um you can ask about the compute. Why did it fail? So, our question here was, "Show me all the clusters that terminated unexpectedly and why?"
[22:52] And it'll give you the details of the error message. If you ask it, it will even tell you how to fix it. Which I think is really a cool thing. And the The that we're not giving Genie a lot of context on best practices and stuff in
[23:05] Databricks, but we've as Databricks have fed it that information, and that's how it's able to give you that that information. Like it's able to say, "Oh, that error message means this." It's able to say,
[23:17] "In my case, I purposely put a bad Spark version in." It's able to say, "That's not a valid version. These are the valid versions. That's why you're dying." So, it's very cool that it has all that context, and you don't have to get it.
[23:31] All right. I know we're almost at time. And I want to hop to one quick thing in the slides. Because you guys need to have a QR code.
[23:44] And I Wait. Come on, slideshow. Okay. QR code. That's my code. That's my slides. Steal it. Change it. Keep it. Do whatever you
[23:57] want with it. But seriously, if Yeah, I gave you my LinkedIn on the slides. If you try to use it and you have problems with it, honest to God, ping me. I will be happy to chat with you and help. Um and I just wanted to shout out some
[24:10] other ideas you could do for non-business Genie spaces. I have a customer that has a cost one. I just asked it cost questions. You know, can you compare to me what my most expensive user is?
[24:23] Is Sam spending all my money or is it Tina? Like, you could ask questions like that if you wanted to. You could set up one specifically on warehouse performance. A big problem I see with my customers is we have
[24:36] a warehouse that has too many jobs pointed at it, and we need to split them up. Right? Or we have unexpected things from dashboarding experiments hitting warehouses causing a lot of money, and
[24:49] they're not tracking that. Because somebody is playing in a little weird way. You can also do governance. If you point it at information schema, I can do things like take a look at who has permissions,
[25:01] who's touching a table, who's the last updated. So, I can do an audit like that if I wanted to. You can also do storage. So, if I wanted to look at storage, we didn't add any of those tables in this example, but you could. And then the last thing I like to point
[25:14] out cuz most people think this is far-fetched, integrate it with Slack and Teams. You can do it today. It's easy. Some of my customers do that. They'll have a um a data engineer ask a data engineer
[25:26] Slack thread. You put this in there, people ask a question, it will try to answer first. Even if it cuts down half of the questions you get, so you can focus on the fun part of your job instead of
[25:39] answering the same question again. Go ahead for that. One other thing I will say, in the instructions, if you do use dashboards a lot and you have users who are asking
[25:51] about dash like dashboard questions, "What's my sales commission? What what's my marketing output?" Like, if it's the same question over and over again, you can literally put it in the instructions to Genie to say, "If they ask a question
[26:03] about any of these topics, send them the link to the dashboard. Don't do anything else." And it will do that for you. I love that. It makes me very happy. I'm like, "You will adopt my dashboard
[26:16] whether you like it or not. I'm not answering this question again." So, really super helpful. Um if you haven't seen it, the only other thing I will mention is monitoring under Genie spaces shows you all the
[26:30] previous questions that were asked, who asked them, if they were good or not, and you can go back in and click any of them and see the source. So, you as Okay, I see I see some questioning
[26:42] looks. Hold on. There are questioning looks. So, if you go here on monitoring, I can see, once it loads, all the previous questions. I can see good, bad. I can see any failures, like this one
[26:55] legit failed. I honestly put in as many questions as I possibly could until I ran out of context. I didn't know if it was possible. I wanted to create a failure. Apparently, it is. So.
[27:07] But you can click on any one of these and I will see what the question was, what the answer was. I can click on the code. I can see the queries that were run. So.
[27:19] If you do deploy this and someone complains the answer is wrong, you can literally see what they asked and what the answer was and figure out if they're right. Was the answer wrong or was there something else?
[27:31] All right. So. I'm over. I'm sorry. It's supposed to be lightning. If you have questions, I'm here. Ask away. I'm happy to help. If you go home, try it and it's not
[27:43] working, again, talk to me. Happy to help. Have dinner. Have drinks. Have a great summit, everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.