Skip to main content

Over Architected with Nick & Holly: Databricks updates for May 2025

Summary

  • System tables GA: Databricks released system tables for compute as generally available, providing detailed data on cluster utilization, nodes and timeline for cost optimization.
  • Feature breadth: May 2025 updates include Llama model integrations, DLT private tables, AI builder specialization, Genie improvements and dashboard upgrades across the platform.
  • Insights at scale: New system tables enable teams to identify low-utilization clusters, expensive GPU nodes and anomalies, turning operational data into actionable insights for infrastructure optimization.

Over Architected with Nick & Holly: Databricks updates for May 2025

Watch: Over Architected with Nick & Holly: Databricks updates for May 2025
Databricks platform release notes: https://docs.databricks.com/aws/en/release-notes/product
SQL release notes: https://docs.databricks.com/aws/en/sql/release-notes/2025
AI/BI release notes: https://docs.databricks.com/aws/en/ai-bi/release-notes/2025

Chapters

FAQs

What are system tables for compute?

System tables are real-time data about your data and workspace usage. For compute, they provide detailed information about clusters, nodes, timelines and utilization patterns, now generally available.

What insights can system tables provide?

System tables enable teams to find low-utilization clusters, identify expensive GPU deployments, analyze cost trends and detect anomalies in cluster behavior for better resource management.

What other updates are included in May 2025?

Updates include Llama model integrations, DLT private tables for sensitive workflows, AI builder specialization, Genie improvements and enhanced dashboard capabilities.

What is the format of this content?

"Over Architected with Nick and Holly" is a podcast-style show where data engineering and AI experts discuss new Databricks features and explore how they fit together in real architectures.

Full transcript

0:00 Introduction to system tables

So this month we are the is this reasonable company where all we do is we ask are various sources of our data quote reasonable. I believe it's called anomaly detection. I think that's what the serious people call it but um we're not serious people. Hello. Hello and welcome to Overarch Architected with and where we're together experts in AI and data engineering but separately not so much. This is the podcast where we talk about new features in data bricks and then try to shoehorn them into a single architecture to see if it's remotely realistic.
Here's our favorite features from the last 30 days. And normally I ask to go first with his AI features, but I know I've got one more data engineering feature than him. So I'm going to sneak in first uh and say that a big thing that's just come out for data engineering is we have new system tables for compute that have just become generally available. So for all of you who have not used system tables before, these are the really useful kind of data about your data.
They're not even metadata. It's the data about how people are using your workspace and they are going to be automatically updated within Unity catalog uh to give you all of this information about your compute. So it has been private preview uh before but now it is generally available. So that is going to give you loads of information about the clusters about the nodes underneath um about the timeline so how they change over time.
Uh, and you can use those to go ahead and then do things like find out which of your clusters has low utilization or which are the ones that have got GPUs and potentially are very very expensive. So with that in mind, can I ask you to start drawing some architecture with some compute system tables? Compute system tables. So I'm part of the data bricks privilege class. So I wasn't even aware that this wasn't available to everybody. part of the team that doesn't have to look at utilization and figure out who's been spending too much of our budget each month.

1:37 Analyzing compute data

Yeah. All right. Confuse system tables. Uh well, I mean, absent anything else, I can just put this here. This will work with everything since it applies to basically the entire platform. Um is that too simple or can we go with that? Uh yeah. So maybe it's worth highlighting we've got three different tables in here. So we've got um the compute system tables have got a table called cluster. We've got a next one uh which is the node.
Uh and then we also have the node timeline as well. Wait, what's node and what's node timeline? So the node is things like how many cores does this particular node have? Are you again the GPU? So what is the memory? Um whereas the timeline is more things like your CPU usage. It's more like your memory swaps, that kind of stuff. So that's the stuff that happens over time rather than like the physical details about the node itself.
Real straightforward. Yeah. Go on, bamboozle me with an AI one then. Okay. Uh it feels like every month we talk there's a new model and this time is no exception. Um so in the last few weeks um Meta released the Llama 4 models. Uh model her. Well, they're calling it a herd. So three models. Uh yeah, Scout. Um Scout, Maverick, and Behemoth. Um I believe Behemoth is still in preview, and Scout and Maverick, um Scout is a smaller one.

2:55 Integrating Llama models

Maverick's the middle one. Um are both out and generally available. Specifically on Data Bricks, it's Maverick that's available. All right, then. Uh so I guess in terms of uh where we put this in our architecture, again, there's not very many options for us, is there? So, uh, we have our AI. Um, so we've got our agent. Um, lovely. And this is going to have uh Maverick. And you said it had um how was it? It was it foundational a foundation API?
Yes. So, two things. It supports paper token and it's part of the optimized um foundation model APIs. So, I like to just think of it as built into data bricks. Okay. Then so could I yes I'm trying to think how our agent could benefit from our system tables from compute um oh could I use them so does that mean I could use them as part of AI query yes absolutely all right then so I think I think I'm going to use these as part of AI query and not over complicate this uh too much so this is just going to be some SQL that I write uh where I'm going to query my data and I'm going to get it to return some things.
I think maybe uh what we want to do is uh is there any way we could do something about usage to be like do you think it's reasonable? Maybe maybe that's a good question. Yeah, sure. Like is this let's find out what the model thinks is reasonable. Yeah, there is another feature I'll tell you about in the next turn that will that uh I'm I'm seeing some ideas come out. So, but I will but I'll I'll I'll hold my tongue for now.
But is this reasonable works? Yeah, I I think that's sensible because if you have you could like aggregate over all of your node table and be like, hey, this one specific node. Maybe it's like an F-ser that's like I don't know really old, you know, is this uh is this realistic? and it's like, "No, this F- series has been out for 5 years and it's not very uh well used across your organization and it's kind of old and not that well optimized.
So maybe this is something that you want to change." So I I think that could be I think that could work. Not very well, but it might work. I'm not sure you'll save money, but you will get a response. Yeah, it's it's very much the diagnosing step, isn't it? Okay, so I have uh some DT features for you this week and I am coining them DT sneaky tables. Uh that is not quite what they are called. Uh they are DT private tables or um having temporary views within DT that are not shared wider.

5:42 DLT private tables

So the idea with a create private table is that you are either creating a either streaming table or a materialized view. Um, and this is something that means that you can make that table, but it's then not published to the catalog. So, if you have a bunch of intermediary steps that you want to materialize, that you want to persist, um, that you don't want to chain loads and loads and loads of things together as part of views, you can use a private table, which is now quite nice and a gives you a lot more in terms of stylistic choice and how you want to create your pipelines.
Um, in terms of the create temporary view, again, this is a view that is only defined within your DT pipeline as well. So, it's not global. No one else can access it. It's only accessible from within the um from DT as well. So, instead of having to publish literally all the inner guts to your catalog, you can instead be a lot more uh careful or specific in terms of what gets stored to a catalog, what could be queried by other people as well.
Okay, that sounds cool. I didn't quite understand. Um were there two things or is just one thing these private private streaming tables, private materialized views? Um but so that part I understood but um you compared it to a temporary view. Um no sorry you you can also have temporary views and you can also have temporary views. Yes, that's part of the sneaky tables. Yes. Okay. So there's like three things. So I do have a question. um then you it notably not published to the catalog.
So then published and referenced where and how so within that DT pipeline you can refer to those tables but once you're outside of the pipeline you can't. So, if let's say you're doing some stuff with some like PII anonymizing data um or maybe you've got a bunch of workspace ids and you don't want to see you don't want to show how you make the sausage basically and kind of like have all of that in the catalog um and all those intermediary steps because that's that's only useful to you as a developer to for performance reasons to make sure that you persist that after you've done something maybe quite complex and you want to have that as that kind of materialized because it's a big step rather than that as final useful data set for someone else to use.
Now, how would how would I use it though? I'm looking at our architecture and yeah, we don't have a lot of data sources this week, do we? No. No, we don't. But I mean, I can kind of do let's say I want to read this as part of a data source or this is going to be a source of data. The answer to this AI query is also going to be a source of data to my DLT pipeline. This is very this is uh my drawing skills are not improving month over month here.
Okay. And then we'll call this the DT but fine. And in here I will have a private join. I'm guessing this is something like this. So I'm going to make this some kind of intermediate join join the two sources do something what I don't know we have uh maybe I'll run another AI query that asks is the first one is the first suggestion reasonable yeah okay then after after the first suggestion so this join the two sources uh as a private private table and but you said it's a private streaming table or a material ized view.
Yes. So, if you wanted to constantly append something from the node timeline, that would be a good candidate for a streaming table. Uh, but if you wanted something that was kind of like a a normal table, but it's incrementally updated, that's where a materialized view would be good. So, that's kind of like a classic table that you'd overwrite bits and pieces for. That's like a non-streaming table. As part of this, we also ask this, is it reasonable?
And we publish our final. This is so this is the table that actually actually works, right? Okay. And published MV. I think this is tolerable. It doesn't make any sense, but oh, it's some kind of like data validation really, isn't it? Just to be like, hey, you know, my first my first Maverick thinks that it's this and then my second attempt says yes or no. Is itable? Part two. Great. Okay, we're off to a strong start.
Do you want to tell me about a new AI feature then? Last month we talked about AI builder um which in short again is kind of like a prompt based approach to developing an agent. So you describe your task, you give it sample data if you have it, even if you don't have it, it takes an attempt at creating a model. Um and then with those two pieces of information, it gives you a little agent and then you can query that agent as you would all other um agents in the data bricks platform.

10:42 AI builder specialization

So you could use it with like AI query and all this kind of stuff. So um the feature was released last month in in beta as you say and sorry for my accent. well I like it. So it released in beta and um now I totally trip myself up trying to make a joke. Maybe you're just not naturally funny. I hate making No, I'm definitely not. That's okay. Um so it was it was released. The first one was information extraction and um so information extraction read some document and pull it out um all the structured data that might be in there or rather output it as structured data.
And then this past month um a new type of tile or model or agent has been released and it's uh it's called model specialization and this allows you to um do a little bit more arbitrary textbased tasks. So classification, analysis of customer reviews, um things like this. So you would you just literally here's a bunch of data that is let's say a bunch of customer reviews. Here's the what I generally want you to do like how do you how do I how do I expect this agent to analyze these reviews.
You give it these two pieces of data and it spits out a little agent that will perform that task. So it's for domain specific uh tasks where you can actually give it some information about how you want it to do the task um and kind of arbitrary like classification summaries things like this. I I think I have an idea of how we can incorporate this. Could we take our AWS bills that are sent to us uh and then have those and then do the information extraction for those and then compare them to our data and amp up the reasonleness and be like hey with the context of the bills that you've got from yeah I do like that it's really silly or is is this so this month we are the is this reasonable company where all we do is we ask our various sources of our data quote reasonable.
I believe it's called anomaly detection. I think that's what the serious people call it, but um we're not serious people. So, we've got our AWS bills here, right? Let's move you over here. Uh and so I'm going to make uh a lovely agent. Uh we're just going to extract all of this. Uh so, what what is it called? Um AI builder. AI builder. And it's uh let me double check. Uh model specialization. Uh so AI builder is this still an agent?
Yes, the output the output of AI builder is an agent. So uh model specialization uh and I want us to extract um the uh bill per node. So I wanted to do some text extraction. Uh maybe I want it to um do like the final bill as well. the final amount as well. So, the final number uh maybe any kind of like aggressive text where we've not been paying our bill for the last 90 days uh about whether like this is a good way to spend our money or not.
So, how about that? And so, at the end of it, we then get an agent. I like how you're agreeing with me as if this is something totally that normal people would do. Uh and then from that, we get our AI agent. Um, and then that can also compare the data that's that's come from that. Is that is that kind of correct? No, I just love I love that final step where you take the box and you're like just make it around everything.
Like is this uh Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. This is fine. It's like the fire the house on fire. The house on fire meme. Yeah. This is good. This is good. This is great. I love it. What's there to like? was not to like. Okay. Okay. So, um I have another data engineering one for you that is Genie has had some massive improvements. Uh so in terms of the experience of using Genie, so if you've not used Genie before, Genie is like a chatbot for your data.

15:10 Genie improvements

So you can ask it questions about your data um and what you've got. But now when you ask questions um it can offer you example questions that you might want to ask. It will offer follow-up questions for you as well. Um, it'll also give you improved data access transparency as well, so you can't select tables to put in your Genie space that you don't have access to anymore. So, that's quite nice. Uh, we've also got customer managed key support.
Now, there's also a rev um revamped feedback flow as well. So, if something isn't quite right, um, you can get some options as to why it's not right and then when it tries to redo that query, um, it will then take that into account. So, for example, you might say the values look off, it's missing a column that you're expected if it made up a definition. Um, if it reruns it, um, it'll take those things into account and give you a better result.
So, yeah, a lot of genie improvements. So, I've already been kind of doing that. So, you're saying that it's it's it's improved at doing that. Yeah. So, the uh the follow-up questions, that one's new. Um, there have been example questions at the beginning where you set those up at the beginning, but now it will offer it will offer them up for you rather than you having to define them up front. Um, and the you've always been able to thumbs up, thumbs down whether you like the result, but now you've got that feedback loop where you can specify um from a predefined list um for faster feedback, but also it will take that feedback into account when you ask it to do it again.
So yeah, I'm just going to point the Genie Space at this new and improved. So you can ask another agent genie how reasonable it is. Okay. So we could now ask a genie space about what our bill data whether that's been extracted sensibly. We've got our compute tables that we're assessing in isolation to say is this reasonable. uh we're then doing a double check of that assessment with DT and now we're triple checking it with the build data.
So I feel like this is more valuable um with this and now we've got a genie space on top of that. So we can now ask more questions in plain English. This sounds like an excellent audit capability to be honest because you know if you just like select star from compute tables you know how can you justify that as like well it was a really thorough audit you know I did select star from a table but now you've checked in triplicate you know if you're a consultant you could charge loads for this we have three agents in the loop that'll be four times as expensive thank you invoices in the mind that system tables is pretty much free and sis select star is again near to free.
Um we are instead going to have all of these agents which are definitely not free. Uh all right then did you want to talk me through another oh my god another AI feature. Yeah this one we're just all over the map with these agents. Uh so the last one you're going to laugh at but it's more of the same. Uh, but this one, um, I'm excited about this one because I like using the assistant. And the assistant is now also new and improved.

18:15 Assistant and code execution

Um, two notable things. The first is it now follows up with actual references and follow-up links. So when you ask a question in the assistant pane um you also get like not just an answer but how this answer was actually arrived at. That's the word I'm looking for. How the answer was arrived at. Uh what documentation was used and what documentation you might want to read as a follow-up. So I think that's uh that's fairly like industry standard now but now it's included in AI assistant as well.
And the second piece is that AI assistant can now execute code itself and it does that using serverless. So the serverless context uh is provided to the AI assistant and so if it generates any code that needs to be run um it can also execute it. So that's pretty awesome. Excellent. So I also love our data bricks assistant. I've been doing a load of work recently and it is it allowed me to work so much faster. Um, there's so many features I didn't realize existed actually.
So, you've got uh fix, which is the obvious one. If you've got an error and you need to do fix it, you can do things like explain. You can do the inline stuff as well, which I didn't know. So, if it's like you've got a particular column and you're like, no, I need to cast it as a string and you can't quite remember the Python way of doing that, um, you can just like inline with the assistant and it will do it for you.
Um, I do like explain. Um, no offense. I've been reading some of your code and I'm like, I don't know what this does. So, I just go explain slash explain ask me what you've written. Oh, that's on purpose. Offiscation is is the goal. I'm sorry. That's not a good career strategy anymore. Like people are going to know what you've done. Um I don't really know how I can put AI into our architecture diagram to be honest. Um to say that it's helping me code.
Um put it on the put it on the side. say the AI assistant is what suggested that you do a private MV that then eventually was published. Okay. I don't think it's that stupid. Um I don't think we need to take No, no, no, no, no. We can't possibly take responsibility for what we've drawn. We need to blame the assistant. Right. Okay. Okay. Uh that is that is peak level. Right. Okay. Um the AI did it. Right. So, this DLC pipeline, I'm going to say uh that it was uh AI.
AI helped. AI made this pipeline. AI made me do it, right? Uh AI, is it called AI assistant? It's called data bricks assistant. I think it's called data bricks assistant, isn't it? Uh AI wrote this pipeline. Databicks AI assistant. Oh. Oh, we're both right. Uh no, I'm I'm just I'm just making things up. I really hope people aren't coming here for factual uh information. Oh, wait, no, no, they probably are. Sorry. It's probably where they get their updates.
Excellent. Okay. Um, if you've not had the chance to give it a go, I really, really do recommend it. Even if you consider yourself to be uh quite a good data bricks developer, um, the amount of times that it just does things for me which I can't be bothered to do, like if I can't be bothered to reformat something, I'm just like, "Hey, change this round so it's in a different structure." It just does it for me. It's wonderful.
It is. Sometimes I could select all and say make this better and it actually does. I I'm not embarrassed. This is this is what the tool is for. So you are an expert programmer. I will give you $200 if you make this even better and more concise. You don't even have to. And I just find it being so helpful. I still say please and thank you to it. Um I I feel like I don't have to admonish it that often. I actually I don't know if you I don't know if this is like fake fake news or whatever, but I saw something about OpenAI saying that people saying please and thank you, the token count of that is costing them millions of dollars.
I wonder if it is nicer to you though. I wonder if it goes out of its way to like think a little bit more. Yeah. No, probably not. Probably not. Okay. Well, I tried. Okay. Uh back in reality land of where things are reliable and the same thing in, same thing out each time. Uh let's talk about uh our last feature of this month and that is dashboards. So we have to start with some bad news. We're going to pull one out for legacy dashboards because they are now deprecated.
So they are not getting any changes. Um they are now deprecated. But the good news is um there are two ways that you can upgrade your dashboard. um you can right click on the little kebab menu on the top hand side um and you can just uh create as a AIBI dashboard and that will do it for you. There's also the API version as well. So if you're like gh I have 100 dashboards I don't want to go through 100 to upgrade them you can do this with the API as well.

22:46 Dashboard upgrades

So if you need to do it at scale that's an option for you. There is a bit of faf when it comes to the parameters. Um, so that will need a bit of manual intervention, but as of now they are deprecated. Um, but that's the sad news. The good news is we now have source control for AIBI dashboards as well. So if you want to be able to check these into GitHub, uh, you can do this. If you need to have them going across kind of dev test and prod because, you know, you want it to move with your data and kind of test it as it goes, this is fantastic.
Uh, and the big one that people are really excited about is custom calculations. So before custom calculations, if you had a table of sales, but you wanted to have kind of an aggregate table, which is kind of like, I don't know, aggregate over geography, uh, and then over maybe day of the week because Saturday you always do good business, you would have to create that table as a second table that aggregates the data.
Whereas with custom aggregations, you don't have to do that anymore. So, does that make sense? No. No. But continue. So, when you've got a dashboard, uh the first thing you do is you switch to the data tab and you do select star from my table that I've got. Um and if you like the shape of that table, that's nice. But if you want to start aggregating across it in different ways um you need to then create a second table or second tab um which has that aggregation logic in it um which is one kind of a pain.
Um it also means you have to hardcode a lot of things that maybe you didn't want to hard code and you can't reuse um those aggregations anywhere else either. With custom calculations it allows you to define how you want these things to be aggregated. So if you wanted to have it in a table or if you wanted to have it a bar chart or a pivot table um you absolutely can do so you don't have to have those duplicate things uh being queried now.
Uh you also get better performance for it as well. So there's some some secret source underneath that's also going to help those be um displayed a lot faster. Okay, got it. So I don't need to publish some kind of intermediate table just for the dashboard just to support the dashboard. The dashboard can do everything for me magically. Well, I think our dashboard has this as this and this and these and more of this. Don't take this very seriously at all.
Takes metrics from this. Should we talk maybe a little bit seriously about maybe how we could use our our new uh custom aggregations? So for example in the node timeline uh you might want to aggregate over a certain time of day. So for example during business hours instead of having to make that kind of custom table which is kind of got the flag of like case when it's within these hours then it's business else it's not business and then you can split all of the things in your dashboard by that.
Now with custom calculations um calculations the custom calculations um you can uh you can define those uh up front. Easy peasy. Uh all right so that is me for data engineering features this month. All right, it's time for rapid fire. You have the list in front of you. Um this time we're not doing just the features or we're not doing features at all. I'm going to have you list off as many columns and tables from the compute system tables as you possibly can in how many seconds do you want?

26:33 Rapid fire and wrap-up

I don't think I could do it in 30, but if I say that you're just going to give me 30 seconds, right? Okay. 30 seconds. Well, I mean, do you want to go for longer? This is I'm giving you a way out here. You could say 10 seconds. Then then it's over real fast. All right. 40 seconds. Okay. 40 seconds. All right. Um, in that case, I have my stopwatch and count me in. All right, take your marks. Cassette, go. In cluster tables, we've got the account ID, the workspace ID, the cluster ID, the cluster name, who's owned it, who owns it, who's owned it, but for goodness sake, the create time, the deleted time, the driver node type, the worker node type, the work account, the minimum for autoscalers, uh the maximum for autoscale workers, uh auto termination in minutes, whether you've got enable elastic dicks, you've got tags, cluster source, uh inner scripts, AWS, Azure or GCP attributes, the driver instance pool ID or the worker instance pool ID, the DBR version, the change time, the change date.
We've also got the node table which has got the account ID, the node type, the core count, the memory megabytes, uh and the GPU count. And then in the timeline table, you get the account ID, the workspace ID, the cluster ID, the instance, the start time, the end time, the driver, the CPU, user percent, CPU system percent, CPU weight%, memory use percent, memory swap, time, time. These are the interesting ones. The network sent uh uh discs free bite per mount point, and the no type as well.
Blatant disregard of the time. I feel like I feel like listening to you that do that was me doing it. You know, that's how stressed that was. Okay. I'm glad I was I was holding my breath while you were doing that. Okay. I'm sorry that you've had such a stressful day today. All right. Thank you for joining us this month. Uh what do you think about architecture? Do you think we did a good Oh, we didn't rate it, did we?
Should we rate it? Let's rate it. Yeah. If last week was what last whatever month was three or two, huh? That can't be much better, honestly. I mean, it did kind of fall apart at the end, but it is I mean, it would work. I think last week, last month, it wouldn't work. This would work, but is it excessive and a big waste of money to just super duper validate how much you're spending and whether it's a sensible thing to do?
Um, no. Please, please don't ever build this. If you build this, you'll never catch up to how much you're spending. That's because it'll be it'll be the most expensive thing that you do. You could save maybe $100 on cluster spend, but you would have spent a thousand on AI. Exa. Exactly. So, at a certain scale, this does make sense, but probably not at yours. All right. So, I'm giving it four. What are you giving? I'm going for four.
Okay. Um I um I I mean, oh, the excessive reasonleness is um I think is all over the place, but I do like this idea of taking your AWS bill and then comparing it to your system tables to be like, "Hey, does this make sense?" Um, what's going on here? Is there anything that I could like look at that's a bit weird? So, I don't hate it as a concept. I just hate the way that we've built it. So, I'm going to give it a five.
We got to start dividing this into like, you know, idea, creativity, execution. Like, u, no, I score my ideas each month. That sounds terrible. Difficulty, execution, creativity. Uh, it just sounds like a performance review. Really? All right, count us out. Let's get out of here. All right, thank you for joining us this month. Um, do you agree with us that this is a four or a five out of 10? Uh, let us know down below.
Check out the notes below. I will put all of the release notes in there as well or any links to any particular features that we've spoken about. Until then, Every month, see you next month when we'll do exactly the same outro. Bye. Love you. Bye. Excellent. I have a I have a feeling you're going to cut that into a short.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.