Skip to main content

What’s New with Databricks Assistant: From Exploration to Production

Summary

  • End-to-end assistance: Databricks Assistant simplifies workflows from data discovery through productionization, integrated in notebooks and the SQL editor.
  • Contextual intelligence: The Assistant leverages Unity Catalog metadata, documentation, lineage, and popularity signals to provide contextual, relevant recommendations.
  • Productivity gains: Customers report tasks that took hours can now take minutes with Assistant, accelerating development and reducing time-to-insight.

What’s New with Databricks Assistant: From Exploration to Production

Watch: What’s New with Databricks Assistant: From Exploration to Production
Talk by Gal Oshri, Sr. Staff Product Manager, Databricks ; Samantha Banchik, Product Manager, Databricks
Databricks Assistant helps you get from initial exploration all the way to production faster and easier than ever. In this video, we'll show you how Assistant simplifies and accelerates common workflows, boosting your productivity across notebooks and the SQL editor. You'll get practical tips, see end-to-end examples in action, and hear about the latest capabilities we're excited about. We'll also discuss how we're continually improving Assistant to make your development experience faster, more contextual and more customizable. Join us to discover how to get the most out of Databricks Assistant and empower your team to build better and faster.
Intelligent analytics for everyone: https://www.databricks.com/product/business-intelligence
BI meets AI: https://www.databricks.com/resources/ebook/business-intelligence-meets-ai
Explore the only unified platform for agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What can Databricks Assistant do?

Databricks Assistant helps with data discovery, code generation, code explanation, code fixes, and productionization across notebooks and the SQL editor.

How does metadata improve Assistant recommendations?

Assistant leverages Unity Catalog metadata, documentation, lineage, and popularity signals to provide contextual recommendations, enabling better results without repeating schema descriptions.

What productivity gains do customers see?

Customers report that tasks previously taking hours now take minutes with Databricks Assistant, significantly accelerating development and reducing time-to-insight.

How do you help Assistant provide better results?

Label your tables and columns in Unity Catalog to provide better context, which helps Assistant understand your data model and deliver more relevant and accurate recommendations.

Full transcript

0:00 Introduction to assistant

Good afternoon everyone. Uh. I'm a product manager at Data Bricks. And I'm also a product manager here at Data Bricks. And we're here to talk to you about the data bicks assistant and how it helps you throughout your data workflows. Thank you so much for joining us today. So maybe a quick show of hands. How many of you have used the assistant already? Awesome. Almost everyone. So the data bricks assistant can help you all the way from data discovery to productionization and it is especially helpful for coding.
Right? It can help you generate, explain and fix your code and it is integrated throughout data bricks surfaces especially the notebook and the SQL editor. But the magic is that it can leverage Unity catalog metadata and other context to provide better results. For example, you can easily use the assistant to generate data transformations for a particular table without describing the schema every time you have a question about it, which can really help you move things faster.
But that's just scratching the surface. There's a broad collection of context that the assistant can leverage beyond Unity catalog. For example, popularity signals, lineage, notebooks, documentation, and a lot more. And all of this comes together under what we call data intelligence. This enables the assistant to help you find the right assets for your task, to generate better responses, and all while respecting your governance and permissions.
And I want to emphasize just how impactful it is to be able to label your tables and columns to help the assistant provide better results for you. It's really worth investing in that. and we'll actually show you a couple of assistant features that also help you make that process easier. And customers absolutely love the assistant. We hear so often that they tell us they can't imagine using data bricks anymore without it.
And frequently they tell us that what used to take a few m a few hours could now take just a few minutes. And we think the best way to convey this to you is not through slides but through a demo. So I'll hand it over to now. Awesome. Thanks, Gal, so much for that overview. Today, I'm super excited to be showcasing how the assistant can make you more productive in your endto-end data development within data bricks. Along the way, I'll be showcasing some of our new features inspired by real end user feedback.

2:19 Data discovery demo

So, to start, I'm simply going to open up the assistant directly from the workspace homepage. The beginning of any data development process within data bricks involves actually finding the tables that you want to use. Historically, you might have done this by utilizing our top navigational search. You might have gone into the catalog explorer. But now, with the help of the assistant, you could leverage all these capabilities directly from the side panel without ever having to leave your active page.
In this case, the workspace homepage, learning about what's new. So, I can do this by simply typing in slashfind tables. And for the purposes of this demo, we'll be looking at some squirrel and park data specifically for New York City. So, let's say at first I want to find this park table. I'm going to simply type in park And within a matter of seconds, the assistant should give me a series of different tables it thinks I would want to look at for the purposes of my development process.
And we can see here that the first table it identified is actually the exact right table we'd want to be using for the rest of my workflows. The next step in this data discovery process often involves writing a couple of queries to further validate that this is the right table I want to be working with. Historically, in order to do this, you might have gone into the SQL editor or notebooks. But oftentimes, we found users would create these new queries for this initial validation and never end up using them again.
Thus, they often also cloud up your workspace. But now, with the help of the assistant, you don't have to generate or create any of these new queries and it can generate that code for you on demand and it can run it on your behalf irregardless of what page you're authoring on in data bricks. So, let's say for this particular example, we'll know this is the right data set if it exclusively contains data relevant to New York City.
I can simply do at park and tell it which table to look at specifically. And let's say we want to find all unique area names. And again, this will only be correct if it specifically contains data specific to New York City. And now from directly from the workspace homepage, I can seamlessly click the run cell button. And this is actually a new feature here available within data bricks. Historically, if you wanted to run code directly from within the assistant, you could only do this directly from within authoring pages where you already have a compute selected.
But now for workspaces with serverless enabled, you can seamlessly run code irregardless of what page you're on throughout data bricks. Those of you may have also noticed that I actually added this table which created this tile as context. When we took a look at assistant usage, we found that many users were fully copypasting fully qualified table names as a part of the prompt. But even so, sometimes we did we weren't able to pick up that relevant metadata as context.
So by directly adding the tables, you directly tell the assistant to bring in that metadata specifically helping it know exactly what tables to look for. And we'll see here that within a matter of seconds, thanks to the help of serverless, the assistant is seamlessly able to run this code for me directly from within the side panel without me ever having to leave my active page. And we can see here by looking directly at the data that this appears to be the correct data set.
Um, we can take questions maybe at the end of the presentation. Cool. Awesome. Now, the assistant isn't just great at helping you generate and run net new code from AC code from across your workspace. It can also be leveraged as a navigational tool. So you'll notice here how I directly added this table and it created this tile. Now seamlessly with the click of a button. I can hit on that tile and it'll take me directly to the catalog explorer page.

5:35 Catalog configuration

Here I can learn more about my table, see where it lives within my workspace. But you can see here this is starting to create quite a crowded experience. When we first launched the assistant, we specifically focused on integrating it within our authoring services first, namely the notebooks and the SQL editor, but we've since worked really hard on fully integrating it throughout the entire data bricks platform. With that being said, the luxury of screen real estate we had on the SQL editor and the notebooks is not necessarily the case everywhere, including on pages like the catalog explorer.
So, in order to make the assistant even more integrated into your flow, we decided to make the UX even more configurable. And now if you click on this button in the top right hand corner, you can dock the assistant to the side, making it a lot easier to work with across all these different pages. Not only that, but you can drag the assistant around. And for the next portions of this demo, you can also easily minimize it.
So the assistant isn't just a great configur configurable navigational tool. It can also help you follow Unity catalog best practices. For example, now whenever you upload any new t table into Unity catalog, you'll automatically be met with an AI generated table description that you can edit on top of or seamlessly accept. And the assistant can also help you bulk generate AI generated comments for all of your columns.
So I can again seamlessly click on this button and within a matter of seconds, it should give me a bunch of different AI generated comments that again I can accept or edit on top of. Now, the assistant is also really intelligence intelligent in the sense that it knows not to overwrite any of my existing column comments. So, you'll see here we have these two columns, one for start time and end time. And we can see those are strings, but the assistant may not know the exact format of these rows.
They could have minutes, seconds, AM, or PM. And so, in order to better direct the assistant in the future, we've actually given it a column comment here. We can tell it exactly what this rowle data looks like. And you'll see in a moment why this is so important, especially when it comes to dealing with these two columns and writing code on top of them. Awesome. So, as I mentioned, for those of you who've been around with us for a while, when we first launched the assistant, we specifically integrated it within our notebooks and SQL editor.

7:33 Coding and optimization

We've worked really hard on integrating it with all of our pages across data bricks, including the catalog explorer. So, let's say for the next steps in this demo, I want to edit on top of my co-worker's existing notebook and get it ready for productionizing it into a job. I can actually directly ask the assistant, what notebooks have used this table? And with pretty minimal prompting on my part, we can see here that the first notebook that it found is actually the exact one I'll be want to be using for the remainder or the bulk of the rest of this demo.
And again, utilizing the assistant as a navigational tool, I can seamlessly click on this tile and it'll take me directly to the notebook that we'll be using. And again, now that we have this extra space, I'm just going to pin it over to the side. So, let's begin to go over my co-workers notebook. Um, we can see here they start off start start off by importing a couple of different libraries. They load in a couple of tables they think will be useful for my squirrel analysis.
They do this really complicated join, which I appreciate. And we can see here that they ran into an error. Now, just by looking at this code initially, I'm not quite sure exactly what they wanted to do. And looking at the error doesn't help me much either. I can see that it's this long-winded Python exception, but I'm not quite sure exactly what to do with that information. When we took a look at assistant usage, we found that by and far one of the most popular use cases was around diagnosing code errors.
And so we wanted to further embed this directly within your flow. So now for complicated errors like this, if you seamlessly scroll down to the bottom, you'll be able to click on this diagnose error button, which will not only take in the context of the error, but the cell itself, and seamlessly give you a response with exactly how to fix the issue. And we'll see here and reiterate again why adding those column comments is so important.
You can see here that the error was actually related to the fact that it's using those start time and end time columns, but it didn't understand the exact format of those. So now the assistant knows that extra information because we added that extra metadata earlier and was able to subsequently fix my code accordingly. So from here, I'm simply going to click on this button to replace the active cell content. And hopefully if I rerun this cell, we should now be able to get the average squirrel watching session duration per park.
Awesome. And we can see here it was seamlessly able to fix that error. So as I mentioned, diagnosing code errors is by and far one of the most popular use cases uh for the assistant here at data bicks. When we took a look at this data a bit more closely, we found that more than 70% of these code errors were often really simple fixes. Misspelled column names, trailing commas, things that could often be solved in a single line, didn't necessarily require a long-winded explanation, and that we had high confidence could be fixed by an LLM.
So in order to further embed this directly into the authoring process, we introduce what we're calling assistant quick fix. So let's say in this particular example, I don't catch that this park column name doesn't actually exist. I can seamlessly run the code. It'll result in an unresolved column error. And within a matter of 1 to 3 seconds, the assistant should seamlessly give me a fix that I can accept and rerun without ever even having to leave my keyboard.
And we'll see now that within a matter of seconds, without me ever having to do any fixing myself, the assistant was seamlessly able to fix my code. So, let's continue to go over my co-workers notebook. We can see here they do some cool data analysis. They get um the number of squirrel sightings per park and they graph that in a bar graph. Tomkins Square Park has the most number of squirrel sightings. If you've ever been there, that makes a lot of sense.
Um we continue to look at them uh going through all this park data and they've gotten this really nice pattern of doing this initial data analysis and then plotting it in a really convenient visualization. Except when we go to this next cell, we can see that they did the analysis in code, but they didn't end up displaying the data in the bar graph, which would have been preferred. So let's say for the purposes of this demo, um, I want to keep my co-workers existing code, but rather than replacing it, I'll just simply add a bar graph below and I want to keep that within the same cell as that follows the pattern that was done previously.
So this time I can actually utilize the inline assistant to do this. I don't necessarily need an explanation on how to create this bar graph. I can clearly see how it was done before. So I can hit enter a few times here. And I'm going to utilize keyboard shortcuts here and hit command I. And here I can just say put this into a bar graph. And we'll see here within a matter of seconds it does exactly that. Again, not touching any of my co-workers pre-existing code.
I can seamlessly accept this code and rerun it. And if this worked correctly, we'll still have my co-worker's original display statement. But as we scroll down, we'll see here a really awesome bar graph that the assistant was able to generate for me on top of my co-orker's existing code. So, we can continue to go over my co-workers's work. We can see here that they also plotted the number of animals also seen with squirrels.
Dogs, humans, and pigeons are pretty common. That's not that surprising. And then we get to this query at the bottom where they're doing a large select over a fairly large table full of squirrel data. Let's say for the purposes of this demo, I want to optimize on top of this query, but I might be a novice data brick SQL e user. I'm not as familiar with this table. I'm not quite sure all the different partitions or filters I could add on top of this.
Now, with the help of the assistant, we actually leverage metadata from the query profiler to give you even more intelligent recommendations on how you can optimize data brick SQL queries. So, you'll see here I have this button in the bottom right hand corner for optimizing. I can seamlessly click it and the assistant will give me an awesome description of how I could potentially optimize my code. And if I want to learn more about where it was able to get this extra metadata, we actually site the query profiler directly from within the side panel.
So, I can click on this button number two. It'll open up my new query profiler and I can learn even more about the performance of my query. And again, it'll not only give me an authoring based suggestion how I could potentially optimize my query, it'll even give me follow-up query or follow-up questions I can ask to further optimize my code inspired by other queries within my workspace that leverage the same table. Now, some of you might be sitting here thinking that's great, but I much rather prefer Spark SQL over data brick SQL.
Does it also work for my Spark SQL queries? And the answer is yes. So, let's say we want to translate this to Spark SQL instead of uh data brick SQL. I'm going to change the cell language to Python. And this time I'm going to utilize the inline assistant to do this translation, but rather than generating net new code for me, I'm going to have it edit on top of my existing code. So I'm going to highlight this line, click on this button to edit the selection.
I'm going to say use Spark SQL and display the result. And we'll see here yet again, the assistant does a great job at doing this translation. I'm going to accept and rerun it. We should have essentially the exact same result below. And again, you'll still see this optimize button appears. Again, I can seamlessly click on it, and we'll get a pretty similar response to what we had previously, except this time using Spark SQL as opposed to data brick SQL.
And similarly, I can still go in and look at the query profiler. Awesome. So, let's say we're done doing an initial first pass. After this, we want to focus on productionizing this notebook into reusable pipeline or job. and my data engineering manager might have a number of different standards this needs to adhere to before it can be translated into a pipeline. Historically, in order to do this, I'd have to review every single cell individually, one by one.

14:13 Advanced edit mode features

But now, we have offered you a new capability with the assistant that we're calling assistant edit mode. So, I'm just going to open this up in a new thread. Edit mode essentially allows the assistant to make multiple simultaneous edits across your entire notebook within a single prompt, making it optimal for things like large-scale refactors, code migrations, or library migrations across multiple cells, or even actions that may not necessarily impact authoring across multiple cells, but take in the entire context of your notebook like generating unit tests.
So let's say for the purposes of this example, we can see here I added a bunch of imports at the top, but as you scroll through my notebook, imports become scattered. I also probably import plotly five or six times within a single notebook which is definitely not necessary. So I can actually ask the assistant to not only refactor these imports into the first cell but also remove duplicates. So I can simply say move all imports to the first cell.
Remove any duplicate imports. And we'll see here edit mode in action. It seamlessly identifies all the cells where we have all of these imports. And you can seamlessly review all of them within the side panel. Let's say I want to jump to each cell individually and review it close up. I can seamlessly click on the response, jump to the cell, and we can see here it did exactly what I asked for. It moved all of my imports to the top and only added code, not removing any of my pre-existing code.
And again, I can further validate that it worked in my other cells. We can see here it only removed those imports. It didn't impact any of my other existing code. And from here, I can accept these individually cell by cell or within the side panel, I can seamlessly accept all of these at once. And we can see here the response is updated accordingly. So let's try and do something a little bit more complicated. Let's say we want to take all of my existing pandas functionality in this notebook and translate it to Spark.
Historically, I'd not only have to identify which cells are using pandas, but I'd also have to look up Spark documentation on how to do these translations. But now with the help of the assistant, I can say translate all pandas in this notebook to Spark. And again, within a matter of seconds, the assistant is able to do exactly that. again identifying all the cells where we're utilizing pandas as of today and showing me how I could potentially do that translation myself.
And as it's streaming, I can actually actively review the responses. So I can look at cell 10 while it's still generating response for cell 12. And we can see here it does these translations pretty seamlessly utilizing Spark this time instead of pandas. And again, I'm just going to accept all of these directly from within the side panel. So now let's do an action involving summarizing the entire notebook uh in a notebook comment at the very end of my cell.
Again, this action doesn't necessarily require authoring across multiple cells, but it does require knowing the context of my entire notebook. Again, utilizing edit mode, I can simply say summarize this entire notebook in the last cell as a comment. And it'll know exactly which cell is the last cell too. Again, with many minimal prompting in this case, it's cell. And now I can seamlessly jump to that cell. And we can see here we have a pretty awesome summary of what exactly is going on within this notebook.
This time I'm going to accept directly from the cell itself. And we can see that it's reflected in the side panel as well. Finally, let's say my engineering manager wants me to document all of my cells and add comments accordingly. Again, historically, I'd have to go in and add all these comments manually myself. This can also be particularly challenging on a notebook like this where we've collaborated with a co-orker.
I might not know exactly what they've done historically. So, I can simply ask the assistant to add comments to all of my Python cells. And again, as we've seen before, the assistant in edit mode is able to do exactly that. And it knows also to skip over the SQL cells or anything that it doesn't doesn't need to be impacted by this work. And again, we can go through and look at how it added all the comments accordingly.

18:13 Productionizing workflows

So, I'm just going to accept everything from the side panel. Let's say for the purposes of this demo, I'm done. I'm ready to productionize this notebook into a job. I'm going to go ahead and close the side panel and go into the schedule tab. And we'll see here that my co-worker already attempted to schedule this job, but it resulted in an error. And my guess is that it's likely due to that same Python exception we saw earlier in development.
So to validate this, I'm going to click into the job, go into the broken job run, and we'll see here that my exact hunch was correct. We're running into that same Python exception we had previously. I can seamlessly go in and highlight the error. And for those of you that also saw it at the top, we still have that same diagnose error button, except this time directly within the broken jobs page as opposed to pulling it up in notebooks.
I can again seamlessly click on this and utilize the assistant directly from the broken jobs page rather than within my notebooks. Now, let's say I want to jump back into the notebook and potentially fix this error. Maybe we didn't actually fix it earlier. The assistant again isn't just great at helping you generate code. It's an awesome navigational tool. I can actually scroll up to the top of my thread and you'll see here we have these convenient page dividers that actually indicate where within the workspace you asked all of your questions.
I can click on that page divider. It'll take me back to that notebook. Scroll down to the bottom of my conversation and apply the result accordingly. Awesome. So that concludes the notebooks portion of this demo. I'm now going to briefly talk about how you can utilize the assistant in AIBI dashboards. So I'm going to go back to my workspace. I'm going to go into the recents tab. And for the purposes of this demo, I already started in creating just an empty dashboard and I took some of those queries from my notebook and put them into the data tab.

19:39 Dashboarding capabilities

The first step in creating a reusable dashboard within data bricks is adding a visualization. So I can simply click on this button in the bottom uh panel. Click into it and we'll see here that the assistant is actually able to give me a number of different examples for the types of visualizations I might want to be making based off of the squirrel analysis data set. So, let's say we pick on primary fur color proportions.
Within a matter of seconds, the assistant is not only able to generate me a visualization that follows all of our standard best practices with a title and key, but it also is able to choose for me the exact right format where this data should live. In this case, it should be in a circle graph as opposed to a bar graph. It's easier to read. I can accept or iterate on top of this. And just to show you how contextual and dynamic this is, let's try and have it generate a visualization for a different table.
And this time, we'll have it use the park data set. I'll select it from here and we'll see here that the responses are again dynamic. This time instead of using our squirrel data set, it's able to give us a number of different sample prompts for the park data set. I'm going to do squirrel sightings by park. And this time, rather than giving us circle graph, it's able to give us a bar graph, which is a far better representation for this data.
And finally, just to prove to you it doesn't just work on those sample prompts and that you can also give it natural language. I'm going to try and give it one more prompt. Uh this time I'm going to click directly into the prompt bar. And let's say we want to get total time per area, bringing it back to that initial query we had using exclusively natural language in the columns that were given. We can see here it's able to seamlessly give me an AI generated visualization and we're off to a great start.
Awesome. With that being said, I'm going to pass it back to who will go over some of our assistant quality improvements over the last year and summarize what we learned today. Awesome. Thanks for demoing that. Yeah. Yeah, that was an awesome demo. Watch out. All right, cool. So, maybe just a quick recap of some of the things we saw in the demo, right? Edit mode is coming soon and it enables broader changes much more easily in notebooks, right?

21:45 Assistant improvements and wrap

It's great for adding comments, refactoring, making bigger transformation updates throughout the notebook. We're excited for you to try. I also want to call out again that the assistant helped us add comments to the tables and columns in Unity Catalog, which then helps the assistant and creates that feedback loop that's so important for AI assistant tools. And finally, just a quick call out to the at tables capability.
So you don't have to describe the schema for your table every time you're asking a question about it or generating some code about it. But I also want to use that as an opportunity to talk about some of the quality improvements that we continuously make in the assistant that are less visible in these types of demos. So one example is that while you can use the at tables capability, we also saw that many people don't know about that functionality or don't remember to use it or they don't know the full table name that they want to mention.
So at the top here we see the inline assistant being used to generate a query on the assistant events system table which exists for admins and to be able to view the assistant events in the workspaces in the region. Now you see two problems in that first uh response from the assistant. The first is there's no user ID column. It's completely hallucinated. The second is that we don't have the fully qualified table name.
So this query is just going to fail. And while the assistant can help me with the diagnose error loop and all of that, we really want the assistant to be able to just provide you the right answer as quickly as possible. So we've made improvements to the retrieval systems behind the assistant so that even when you don't use the ad functionality in many cases, we can still find the right table and generate the right query based on that.
So the result at the bottom is what you see now in many cases when you use the assistant. Another improvement I want to call out is changes to code completion. We know you're in data bricks, right? And if you're in a notebook and you add a comment that says exit the notebook and return exiting, right? A standard off-the-shelf code completion model might just add the function or suggest the function exit. Right? But because we know you're in data bricks, we've actually fine-tuned the model to provide better and more contextual results.
So in this case, our model would return dbuttills. netebook. exit exit providing the right functionality. And this applies across many different use cases where we optimize which off-the-shelf model we use or how we fine-tune a model and what context is provided to get the best results. Maybe to give another example, we saw the awesome quick fix functionality where as soon as an error popped up, the assistant automatically suggested a fix for that error.
We've previously only looked at a certain class of errors that we believed are easy to resolve through that methodology. But now we have a model to help us decide which errors we should attempt to fix through quickfix which has enabled us to both expand you know the number of errors that we can resolve through quickfix while maintaining a very high quality bar. So to summarize, we've talked about how the assistant is used by many customers across the end-to-end data workflow and all these different capabilities in authoring and outside of authoring.
And we've also talked about how the assistant has gotten a lot better over time, both in terms of product capabilities as well as the quality of the assistant in in providing responses. And I want to encourage you to if you find something wrong with the assistant or if it could be better, it's very easy to reach us and give feedback straight from the product. If you're interested in learning more, here are a few links to documentation and some blog posts that we thought might be interesting.
And actually, I see a bunch of cameras. I'll I'll put that back. Um, and then the next slide is if you want to continue your learning journey. And also I just want to call out if you can fill in your survey about this uh session. We love the feedback and we want to make this better in the future events. So with that, thank you so much. We appreciate your time. Thank you for listening and enjoy the rest of the conference.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.