Skip to main content

What’s New in Databricks SQL: Latest Features and Live Demos

Summary

  • 200+ features: Databricks SQL has launched over 200 features in the past year, including new SQL editor, streaming tables, and AI-powered optimizations.
  • BI and cost management: New features support BI integrations, cost management, system tables, and observability for analytics at enterprise scale.
  • AI integration: AI-powered performance optimizations and features like multimodal AI in SQL demonstrate next-generation data warehouse capabilities.

What’s New in Databricks SQL: Latest Features and Live Demos

Watch: What’s New in Databricks SQL: Latest Features and Live Demos
Talk by Gaurav Saraf, Sr Manager, Product Management, Databricks ; Kent Marten, Staff Product Manager, Databricks
Databricks SQL has added significant features in the last year at a fast pace. this video will share the most impactful features and the customer use cases that inspired them. We will highlight the new SQL editor, SQL coding features, streaming tables and materialized views, BI integrations, cost management features, system tables and observability features, and more. We will also share AI-powered performance optimizations.
Here's more to explore:
The best data warehouse is a lakehouse: https://www.databricks.com/product/databricks-sql
How to migrate legacy data warehouses: https://www.databricks.com/resources/ebook/migrate-your-legacy-data-warehouse-databricks
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What are streaming tables in Databricks SQL?

Streaming tables enable real-time data ingestion and transformation in Databricks SQL, supporting modern analytics workflows. This video demonstrates streaming capabilities.

How does AI improve SQL performance?

AI-powered optimizations automatically tune queries, improve execution plans, and reduce costs. This video shows AI integration in the latest Databricks SQL features.

What new BI integration options exist?

Databricks SQL supports integration with leading BI tools while providing native AI/BI dashboards and Genie natural language interface. This video covers integration options.

How can organizations manage SQL warehouse costs?

New cost management features enable organizations to monitor usage, optimize queries, and control spending on Databricks Data and AI platform warehouse operations.

Full transcript

0:00 Introduction to DBSQL

Welcome everyone. Today we're going to talk about what's new in data brick SQL the latest features as well as some live demos from. Uh. I'm a product manager in the DBSQL team and I'm joined by my colleague also a product manager in the DBSQL team. All right. So just PSA uh we'll be making some forwardlooking statements. Obviously things can change road maps change. So if we say something that changes don't be surprised.
All right. So what is DBSQL? Uh DBSQL is our data warehouse. Uh we say that the best data warehouse is a lakehouse and DBSQL is that kind of warehouse on a lakehouse. operate on uh your cloud storage with data formats like Delta Lake and iceberg fully governed by Unity catalog and fully integrated with all the other apps uh of data bricks that you're familiar with including mosaic AI lake base aib lakeflow apps as well as marketplace.
So let's let's double click on that DBSQL box and let's see what's inside of it. So in DBS SQL you can ingest your data from all sorts of data sources get them into that data lake transform them to get them ready for some analysis. By the way to transform them and to do any analysis you need the most important language in the world SQL. And so there's a lot of SQL functionality that goes inside the DBSQL engine. After that you can use our data bricks products to query or visualize that data.
Or you can use thirdparty apps and use DBSQL as a serving engine. you can serve up whether it's your PowerBI apps, your custom apps or other tools you might have outside of data bricks. Just to give you a sense, I think more than half of our workloads actually come from tools outside of data bricks, right? All of this is sitting on top of a very powerful AI powered engine that makes sure performance and TC are great.
And again, like I said previously, unified and unity catalog, sorry, governed with Unity catalog uh operating on top of your Delta Lake and iceberg tables. So, we're going to cover everything that's new, but um, how many features have we launched since last year? Yeah. I mean, I I lose I've lost track of count, but it's over 200. 200? Yeah. And I have 37 minutes left. Okay. So, we're not going to cover all 200 features.
Uh what we are going to do is do kind of the big highlights since last year and some of it looking forward to the next couple quarters as well and focus more on the core DBSQL aspects uh in that feature space. uh and so you might have to go to other sessions to capture some of the other details and I'll tell tell you which sessions by the way uh to go to um so first off let's start with the AI powered engine now just a show of hands does anybody over here care about time and money okay just 50 70% I think the other 30% might be bots in the room maybe you won't tune correctly but okay no worries 70% of you care about a about time and money that's what this is about right so The AI powered engine what it does is it's taking in your queries your data that's coming in every single day passing it through the engine but we are building AI models behind the scenes to detect what those query patterns are.

2:38 AI-powered engine updates

We're trying to understand what are the data patterns, what are the query patterns, and how can we optimize two things. One is the storage, how your data is actually laid out in your data lake. Again, iceberg and delta, but also how do we optimize the engine so that the next time a similar query comes in, we can run it that much more faster, right? And what this results in is better performance out of the box and improved TCO for you over time.
Now, in this, we've got three things that are really new. One is automatic liquid crossing. This allows you to lay out your files correctly based on the query patterns which you're saying. Previously, we already had liquid by the way last year. You could already cluster them, but you had to go state what your columns would be that you want to cluster around, right? But the thing that's new that's G right now is that you don't need to do alter table by cluster by, you know, column 1, column 2 and figure out what those columns should be.
The AI is now smart enough to say, hey, I can I can actually figure out those columns for you automatically and change them dynamically if your query pattern change, right? So you can go today and change those tables to say alter table whatever the name is cluster by auto and your job is effectively done. It's going to automatically start clustering your tables behind the scenes correctly. The second one second one is automatic statistics.
So statistics are really important to help the query engine analyze your queries and run them efficiently. If we know the stats of each column, of each file in the data lake, it's just easier to skip files and detect which query patterns to run. Now, if you enable automate statistics, your performance could be up to 33% faster. The thing that's new this year, and by the way, is G now is that previously you had to run analyze by yourself manually.
Now, if you have predictive optimizations enabled, it is already running stats for you behind the scenes magically. Right now, you only get this for managed tables. And if you've enabled predator optimizations going forward, however, and I'm going to click one more slide. Uh yeah, going forward, predictive ops will be enabled by default on manage tables. If you're not using UC manage tables, however, uh you'll have to run these tools by yourself and it's not as efficient as just simply using manage tables and predictive optimizations.
The impact of all of this is that if you're using this stuff, you'll see performance improve by as much as 20x. This is like an up to 20x the way not an average as you might see an improvement in query time and as far as the storage goes because we're running back behind the scenes uh you will see a 50% improvement in the storage cost of all that data. uh what is the impact of all of this over time? Hopefully, you saw and Urslan talk about some of our benchmarks early at the keynote, but just to repeat it, uh we've been improving performance steadily over the last three to four years.
And you can see that over time your query performance has improved almost 80%. My marketing team will require me to say 5x uh if you do the math. And so it's 5x improvement over the last four or five years. By the way, this is not a benchmark, right? This is your queries running day in and day out. we only look at the queries that ran let's say 10 times in month one didn't change anything ran 10 times the next month so it's statistically significant and we say great only look at those queries that ran repeatedly and see what the month change was and we measure those with a lot of data science and we see that it's a 5x improvement over time the last thing we're announcing uh rolling out right now is there's a bunch of smaller software upgrades rolling out uh into the new version of DBS SQL in the coming weeks it's already started by the way but it's going to take all of June to get it out you're going to see a 25% improvement across most workloads running on DB running on serverless SQL to be specific in the next few weeks.
By the way, this is the same skew, same price, nothing is changing. You can keep sleeping and you'll get 25% better performance and cost. So, let's go to the next section now. Ingest and transform. Now, this is a part where uh again, if you're at the keynotes, you saw Bilal and talk about LakeFlow. And so, I'm not going to cover this in detail. You should go to the lakeflow sessions and get more information about lakeflow connect DT or job.

6:52 Ingestion and transformation

I saw people taking pictures. So if you want to take a picture you can take this picture. But this is primarily by the way how you connect to all the various data sources out there in the world pull data into data bricks use DT or jobs to transform them and get them ready to you know serve into analytics. After that I will double click on a couple things though. One is materialized views. So if you're writing uh if you're writing let's say a PowerBI um query it's pretty common for you to be writing them across let's say two or three tables right and those joins can become timeconuming and so a common pattern is you say great if these tables are often being joined together let's say my customers table and my orders table let me just create a materialized view that makes it easy to then run that same query but on the materialized view that makes that BI performance get better now the beauty of this is you don't have to figure out all the logic of how to like add rows as new data comes in the materialized view.
You just define the select star join these two table. This is the table output I want and it's going to do the rest of it behind the scenes automatically for you. Now this is cool obviously because you get better performance but every time there's new data added to your underlying orders and customers table somebody needs to refresh that materialize you. There's two ways to do that. One is we either have to do a full refresh or it's smart and it can do the incremental refresh of just the rows or things that were changed.
And so this has already existed by the way for more than a year now. The thing we're adding on top is just more use cases where we can be incrementally refreshing that same materialized view. Right? So on the right hand side you can see more operators like left outer joints, full outer joints, inner joints that previously required if they were in your query they required a full refresh and now with the screen blinking uh they don't require an inc they don't require full refresh.
There'll be an incremental refresh which basically means the cost of refreshing a materialized views will go down. performance will stay really good. This is by the way generally available right now. Um the second thing is dbt. Any any dbt users in the crowd? A few. Awesome. Including Juo. Uh so there's two version of dbt, right? There's dbt core which is the open source version and dbt cloud just is offered directly by uh the cloud company.
Is this screen fine? Should I continue? Keep going. All right. It sounds sounds scary. Um so what you can do is um you could already connect data bricks with dbt through partner connect but if you want to orchestrate those dbd jobs you can now create a dbd cloud task in jobs and orchestrate all the models that are running directly through jobs. What that really means is you can have let's say a regular data bricks file that does something in one task and then the next task is a dbd cloud task running right after that.
This is in private preview right now. Okay, with that I'm going to hand off to Thanks, Goff. Uh, all right, let's talk about SQL. If you're taking pictures, this is a pretty good slide to take a picture of. This is not an exhaustive list of all the things that we've built or that are new in SQL. Um, and some things here are in private preview, others are in public preview or GA already. But it shows you the three categories that we think are sort of most impactful across all of our customers.

9:57 New SQL language features

The first is enabling SQLbased ETL and migrations from legacy data warehouses. So you'll see features like scripting and stored procedures, recursive CTS, all things that were previously not supported that were sort of a gap when moving from legacy data warehouses and now they're here and available. The center column focuses on more complex uh data structures and unlocking use cases on top of enabling uh migrations as well.
So this is spatial SQL and support for geospatial data types uh working with variant as well. And the last column is AI functions. We continue to invest in uh all things AI and make it easier for the SQL analyst to get started. Let's double click on go just for a minute with spatial SQL. Previously in in in DBS SQL there was just no way to do kind of the most familiar spatial analysis with vector data. And now you have a set of 80 plus spatial SQL expressions, new geospatial data types, a geometry and geography and optimized performance which we already have really good uh customer testimony around.
Our spatial joins are super fast with this uh the state of the feature. So right now it's in private preview, but it will be in public preview coming to the DBSQL preview channel in July and fully rolled out in the current channel by August. So if you're working with spatial data, we got you covered. All right, let's dive into a few demos uh where I'll focus on some of the features that were in that SQLbased ETL column.

12:00 SQL and AI demos

Demo time if I can find the right way. Okay. So, let's imagine for a second that I work or I own a fictitious bakery, the bake house data set. In this data set, I'll start up here at the top. Uh I'll just run this and make sure that I'm using my uh catalog and schema. You can see I've got customer tables. I've got transactions for people who've bought cookies from my bakery. And I've got reviews associated with the satisfaction or lack of from my customers.
So I've got two tasks. First, I want to start a loyalty program. All right? I want to assign every customer into a new loyalty tier. And the second task is I want to understand the sentiment of these reviews. So let's go ahead and do that using some of the new features. So, first up, let me go ahead and I'm going to go ahead and assign one of my customers by ID to a loyalty tier. And I'm just going to call this stored procedure.
Now, you might be saying, stored procedures. Yes, this is a new feature now supported in data bricks in version uh the data bricks runtime version 170. It's in public preview which will roll out just after data data in AI summit into DBSQL with stored procedures. You now have a stored asset in Unity catalog. So you can easily reuse existing logic and stored procs are much more powerful than like userdefined functions right you can do complex uh DDL and DML multiple statements within a stored procedure all using the scripting capabilities that are available with proxs you can uh supply input parameters out parameters inout parameters all those things that you're used to doing if you're coming from a traditional or legacy data warehouse.
So stored procs are now available in data bricks SQL. Uh just to take a little walk through here. This procedure took just one customer ID and seeks to update that customer's loyalty tier. I can declare variables. I can write some uh some custom logic here to understand how much those customers have spent. Some conditional logic within there uh just to assign the tiers and then I can do an update at the end. So here I've already gone ahead.
I called that procedure uh before. Now I can go ahead and run this cell and make sure my customer has a loyalty tier. is in the silver tier. He likes cookies. That's good. Okay. So now we've used store procedures and scripting. Let's take a look. What if I had, you know, all of my customer base? In this particular case, I have 300 customers. Everyone needs to be assigned to a loyalty tier. I don't want to just go through and do it one at a time.
I've created a second procedure. So you can imagine after every customer makes a purchase or maybe on like a regular cadence like every month you might update your loyalty tiers. In this case I have a stored procedure which doesn't take any input parameters but it loops through the customer ids from my customers table and calls that procedure that I just ran. So it's a nested stored procedure and this works uh seamlessly.
So I ran that just before the demo to make sure we were going to be good here on time. Let's make sure my table now has a loyalty tier for every customer. You can see it stored procedures now available in DBSQL. Excited for folks to use it who are moving from legacy systems. So we've answered my first question. Can we establish a loyalty program? I think we've done it. All right. Now you can imagine a world in which maybe over time you want to you want to update the loyalty program and make sure um or or you might want to update it because maybe your qualifications for the loyalty program tiers are are too steep.
I was expecting people to spend $1,000 a month on cookies. Maybe that's too much. We need to update the tiers. I could go ahead and update the existing procedure. Or maybe I want to split out my tiers into new tables. Again, you have full flexibility to do this with SQL. In this case, if I wanted to go and split out each of the tiers into their own table, well, I can use a temporary table just as an intermediate data set to work from and extract the data.
So, what's the new thing here? Data bricks now supports temporary tables. These are not temporary views. These are real temp tables materialized in uh and managed by Unity catalog. So these temp tables are session scoped. At the end of this session, they get cleaned up automatically. So there's no more I need to create my own table, manage it, and you know, litter my workspace with uh with uh permanent tables. These are temporary tables.
Great for uh processing intermediate steps in ETL, but you can also use an ad hoc use cases. Um so yeah, temporary tables, stored procedures. Let's see what else. Let's go ahead and uh run this. I think I need some of these tables. So, I'm going to go ahead and create uh a number of tables for each of those tiers. Now, you can see I've actually enabled a table property here. This is because we have a a feature in preview which is multi-statement transactions.
So, if you want tables to participate in an atomic transaction block, uh you're going to need to enable this feature uh as part of the preview. I mean, you got to get enrolled in the preview and all that, but um this is the way you're going to do it. All right, let's scroll down here. So, yeah, I've created those separate tables. Let's say I want to revamp my loyalty program, maybe apply some new logic so it's uh not as hard for customers to get into uh these particular uh tiers.
I can go ahead and run this because if I start to put customers into new tables and remove them from tables, I want all of that to be handled as a single transaction. I don't want to update one table and then have a mistake or an error, which apparently I have here. What have I done? Oh, is not pinned. This is a new error and did not happen when I ran through this demo several times. So, there you go. I'm going to call this one part of the preview and just move along.
It is what it is. The demo gods sometimes they curse you. Um, I will say though with multi-statement transactions, it was brought up in the keynote. Um, the way data bricks does it is a little bit different. when looking at what our competitors do. So we have uh optimistic and rowle concurrency control. Um so this provides the best performance compared to our competitors. It's up to 5x the concurrency. Uh so it's really great performance with multi-statement transactions.
Okay, that feature is still in private preview. So if you're interested in uh multi-statement transactions or temp tables, please reach out to your account team and we can get you enrolled to try it out yourself. it will work for you. Okay. Um my second task I wanted to go and analyze the sentiment of customer reviews. So how do you do that? Sentiment analysis. It feels like a little bit more complex task, right? Um for anyone who's, you know, done data science or or been in that world, maybe you've actually done some training, you understand how to do it.
But in data bricks, we're trying to make AI really accessible and give you just a singular function to do a sophisticated piece of work. I'm going to go ahead and run this and you can see what is going on here. Cannot add column. It already exists. Okay, I've already gone through and added this uh this data set. , don't be mad at me. You know, I know what's happen. It's not a real demo unless something breaks. I know what's happening.
Let me go back up here. We only did this five times successfully, so you know. Yeah, the charm. We're going to run with this. Let's uh I forgot to clean up my demo state of all the things you're supposed to do. Um if it don't doesn't run a second time, I'll have a different excuse. All right, let's try it now. Waiting. Starting. So I wanted to call out this example working with scripting uh just because I think it might be pretty common.
In this case I'm actually looking at uh I'm using the information schema here but I'm actually using uh parameter markers for the schema. So you can see up here I'm using the data summit schema and I'm looking for every table within my information schema that includes a column that's called review. I'm going to go analyze the sentiment of that column. So you can see I've built up in this parameter a dynamic string which alters the table and adds that field and I can execute that right away.
And then I'm calling the key function here which is AI analyze sentiment to do that sentiment analysis. And then I can go down to my table and I can see here I've got some positive I got some mixed reviews which may be your reviews of my demo at the end of this. Um, but hey, that's where we're at. Um, so I've accomplished my two tasks. Uh, I know the sentiment for my analysis. I've created a loyalty program, scripting, store procedures, transaction support, um, temp tables.
So, all these features can help you do more with SQL. Look forward to you guys testing that out. All right, let's jump back to the deck for a second here and just talk about this AI part for a second. So, I said, hey, you might need to do some training. uh you might need to have a a you know Python experience um if you're going to do AI like you need to manage uh scalable compute and infrastructure it becomes kind of a big complex hurdle to get started with AI and as I showed with sentiment analysis it's now just a a singular function ready for a SQL analyst to take it and run with it right so that's awesome but what if you're working with more complex data what if you have unstructured data like PDFs so you You know, we're all at the conference this week.

22:16 Multimodal AI in SQL

You might be taking pictures of receipts and going and looking for reimbursement. Well, this is a it's a common thing in this uh in this situation, right? So, what if I want to look at PDFs and actually analyze the content of those PDFs or work with imagery? Well, now you can because in data bricks, we support multimodal AI functions. So you can use AI query and actually look at and inspect and analyze imagery or you can use AI parse document and look through some PDFs.
I'm going to do that right now and show you a quick demo. Fingers crossed. Garoff. All right, I got it. So here I have just a set of uh PDF receipts. You can see they're all varied in structure and style from different businesses. And my task is just to categorize the receipts. I want to know if they're travel or food or, you know, related to a specific category. So, in the SQL editor here, uh, I have this function up top.
It does a little bit more underneath, but you can see I have AI parse document, and I'm looking through my volume to look at all the PDFs within it, which are all those uh, PDFs that I just looked at. So I previously run this and if I want to look at the actual output result, make sure my my warehouse is running. Looks good. I can see it here. So now instead of that unstructured PDF content, I now can make sense of some of the text within those receipts.
U but of course if I'm just scanning this, you know, I need to sort of interpret it and analyze it. I have a another function here using AI gen which I can prompt or provide the prompt to actually go and categorize these receipts and I can say something smart like make sure there's just one word categorize it with things like food or transport like I mentioned. So let's go ahead and run this right now. All right, my receipts are summarized.
AI functions right at your fingertips. PDF processing. It's no longer away from you and hidden and difficult to work with. You can you can use it today using these functions. Let's talk about imagery. Okay, so here I am in a notebook. I've got this image which talks about data bricks performance and it's a chart, right? You and I can look at this chart and interpret it. U but what if I have charts, lots of charts, and maybe I want the machine uh to do some analysis.
And I also have some lovely photos here of a meadow ali when he was obviously younger. Um some polar bear cubs and the beetles. So I've gone ahead and uh set up a couple of prompts to look through these captions. So first thing is let's take these photos and say, "Hey, I want to post those on social media. Just provide me a caption." Right? I want to automate uh you know the creation of a social media caption. All right.
So in this case uh yeah with AI query so we're actually using data bricks hosted you know state-of-the-art models. This is llama 4 that I used and the prompt was uh yeah create a caption for a social media post. So here I can see with it says meet our team lead. You know this is social media. I already had some troubles with the demo. Let's go all the way. What would a provocative uh caption be? Let's see if we can walk the fine line.
All right, we're going to rerun that. I'm going to look at the first the first two here. Let's bring them up. This is the chart. Oh, here it's it's already done. The future is now and it's wearing a smile. Not as edgy as I wanted it to be, but nonetheless, I have access to these models. Uh, and I can do some pretty sophisticated analysis of imagery. I've got one more here that I want to run. In this particular query, uh, we're using the claude, right?
Again, data bricks hosted. And in this prompt, I'm asking for it to understand, make an inference about the situation, the intent, the relationship between things in the image. Like that's a pretty sophisticated question, right? I need uh the model to understand, you know, what were those polar bears doing in the den or, you know, what were the beetles doing when they were hanging out with each other. All right, so it's obviously testing the model.
And here we go. So, let's look at the polar bears. The image shows two young polar bears cuddling close together in what appears to be snowy and icy environment. It makes a summary about those things like, hey, it's good to stay warm and cuddle up, etc. , etc. Now, with AI query, you can do this kind of multimodal analysis and start to interpret imagery and hopefully that unlocks new use cases and interesting things for you.
Awesome. Okay, let's jump back to the deck and talk about how we integrate and serve out uh data into the ecosystem that you guys are working with. So this is the I think my main point which is that there is a broad ecosystem around data bricks. All of the tools that you probably use, all the BI tools, uh, PowerBI, Tableau, etc. , Sigma, um, all available through integrations, um, all the connectors, um, accessible and can connect the data bricks ready for you to build.

28:06 Ecosystem integration

Um, along with AI agents, more to come there in that category as well. So, hey, there's a ton of flexibility, but all the all the tools that you know and love are there. In the last year, there have been a ton of improvements in the space, but I want to call out a few related to BI. Uh, so on the PowerBI side, uh I think this was also mentioned in the keynote, but uh PowerBI tasks in jobs are available and allow you to sync semantic models.
Uh with Tableau, it's now a one-click integration allows you to open up Tableau online and explore data and build build visualizations in a seamless way. And then new improvements on the spreadsheet side. anyone anyone still work with spreadsheets? Are they still popular? Um, well, that's where half the world's data is, it seems. Um, it ends up there. And so, we're doing a ton in that space to make sure you're enabled and your business users are enabled to still work with data bricks even through those environments.
Let me show a couple of demos. Uh, okay. So, in this demo, make sure my warehouse is started. Yeah, in this demo I'm just using the New York taxi cab data set just to show um you know a brief example of working with data that I want to explore further um in outside tools. So previously if I wanted to use Tableau or Tableau online, I'd have to download like a Tableau data source file and upload that to Tableau online.
I'd have to do authentication and validate, you know, my credentials. you know, there's a little bit of a process to get it just to the BI tool that you end up wanting to work with, right? So, now I'm going to go ahead and I'm going to click on one of these views that I've created, uh, which is a filtered view over some of the taxi trips in New York City. And up here in the right, I can just hit explore in Tableau Cloud.
If I was working in PowerBI, you know, I have other options available, but this is the new part. I can open up directly in Tableau Cloud. Yeah, it's saying uh make sure your identity is valid here. And this is like seamless right away. I can start to build a visualization. So if I want to look at taxi pickup zones, I can do that. If I want to look at uh trip distance, I can add that to the label. I can add uh trip counts here uh to color.
I can zoom in a little bit. Let's actually change trip distance to average, which makes a little more sense. I can see lots of folks leaving from the airport here in New York City. Awesome. But I also notice, if you know New York, we're missing Brooklyn. Um, that's not what I want. Let's go back to my view. And I can see that it was filtering out some of the zip code regions. Let's just remove that. I'm going to rerun the view here uh in data bricks.
I can jump back to the workbook. I can go to my data source and hit refresh. And voila, they show up. So it's like a seamless connection. No credentials, no configuration. I can immediately explore my data in Tableau. Now for the business users out there who are still working with spreadsheets, um what can you do? How can you integrate with data bricks? Let me show you. There is a new connector for Google Sheets and one coming uh for Excel.
What does it allow you to do? Well, it's pretty easy. Once I've added uh this add-on uh to my profile, I can go ahead and launch the sidebar. And there's a warehouse. I could say select star from main. deis why did I make my schema name so long uh taxi trips where let's look for expensive rides fair amount greater than 50 okay so now from sheets connected to the warehouse no configuration here as well I can execute these queries and from here I can jump right off into doing further analysis is directly from this interface.
Okay, I think that's all I'm going to show you today on SQL language serving and other things. I'm going to hand things back over to and it's all yours. Thank you. Thank you. All right, so you got six and a half minutes left. I might go a little faster because the demo gods weren't with us today. Uh so showed you a bunch of AI functions, right? He showed you the llama one, he showed you the cloud one, he showed you the multimodal one, the parse one.
What we allow in DBS SQL is through that AI query function and some built-in functions. You can use any LLM model out there whether it's hosted in data bricks or hosted somewhere else. You just have to register them with our AI gateway and you can call that model in SQL directly right and this is very easy because again you're just writing SQL there's no setup required is all entirely serverless. The cool thing is that since last year we actually been working in performance a lot and so our throughput which is really important right in SQL you're used to processing many rows of data and so our throughput has increased a lot and now we are almost 3x the closest competitor in the industry as far as how many rows you can process uh per second.
Next up though is for our you know our AI agents in the room who you were not interested in time and money. So what we showed previously was using LLMs inside SQL. What about the other way around? What if you were in an LLM tool or you were an LLM agent, but you wanted access to DBSQL's data, right? The other way around. And so what we're launching this quarter is an MCP server on this is called model context protocol by the way, something new from cloud that the industry has started to adopt.

34:10 AI agent interoperability

And what it allows you to do is tell any agent to connect to data bricks and it basically can send any query it wants to from let's say chat GPT or the cloud app and you can ask questions like hey what's the revenue in Japan and the tool itself will figure it out write the right query to DBSQL and DBSQL will fetch the data and return it back to the LM. This way you can basically allow agents to query direct data directly from DBSQL.
This should be in private preview uh by end of Q2. Okay, we skip one box in the middle. This is our tools inside data bricks that help you query and visualize. Uh the four primary tools are the SQL editor, AIBI dashboards, Genie and alerts. Now, you already saw a lot of this at the keynote with and. So, I'm going to skip uh the AIBI parts and the Genie parts. Please go check them out if you didn't already see the keynote.
There should be other sessions. There's also a really good session done on how to use uh SQL in the notebooks and a SQL editor. So I won't cover those here, but there's one new thing we launched recently called alerts v2. So what is alerts v2? We have a v1 version uh and we're releasing a new one. Uh you've got all this data in your lakehouse, right? Your business metrics, maybe your system tables for your usage of data bricks.

35:35 Monitoring and alerting

What alerts allows you to do is write any simple SQL query, set up a condition. Let's say did my revenue go up by more than 10% in this region? And if that event happens at the schedule, you kind of ask it to run the alert. It'll send a notification to you whether it's by email, Slack, or any web hook that you want. We already had a V1 version, but we got really good feedback and we've built a V2 version which is now in beta.
The differences are that you can now write the query and set up the alert configuration side by side. You can see that in the image. So, it's like a unified UI. We got a unified API. So, you just need a single API call to create, delete, or update alerts. By the way, fully supported Terraform SDKs and DABs, data brick asset bundles. Uh we have a new alert history page. You can actually see when the alert was historically triggered versus not.
And lastly, we have support for git sync. And yeah, this is beta now. You can go use it in the product immediately. Okay, so we covered all these aspects on the top row. We covered AI powered engine, but we missed one thing, which is as you're doing this stuff, somebody needs to go monitor and manage all these warehouses that actually does all this work for you. Right? You saw going and clicking on start warehouses a couple times there.
And so I'm going to add one more section which is warehouse manage and monitoring. So what is this? Um what happens is that on day one you might have one account on data bricks, one workspace, one warehouse, one team, one notebook. Pretty easy to manage stuff hopefully. But pretty soon it's going to look like this. Multiple workspaces, multiple warehouses, multiple teams, multiple tools. It gets pretty gnarly to go manage all of this.

37:03 Warehouse management and cost

And so what we want to do constantly is make sure that we are giving you the tools so you can easily govern uh the cost uh across these assets and kind of focus on creating value to your organization instead. So we see this in two different sections. One is monitoring what's going on. The other one is optimizing on the right hand side of what is actually going on. So if you want to take a screenshot or a picture, this is the right slide to take it.
All the green stuff are the new features and some of the features will be rolling out later on in Q3. I'm going to pick on three or four of these really quickly. Uh so the first one is cost observability. So we charge you for warehouse uptime, right? If you charge like using a medium warehouse, how long that medium warehouse is up for, we will charge you accordingly and you can see that cost in the billing system table.
However, what if you wanted to know what was the cost for those queries just ran, right? What is the cost of all the queries coming from Google Sheets? what is the cost of perhaps uh all the queries coming from a single AIBI dashboard on data bricks. So what we've done now is we've taken the billing systems table as well as the query history system table that has this metadata about each query and kind of just calculated what the attributed cost per query would be.
So we've done the calculation for you. We've published this in a materialized view that's available for you to download right now in data bricks labs and we've also built a dashboard on top of that for you that you can just download by the way. It'll take you to clicks. This way you can easily see views like cost per user, right? So you can roll up all those queries that came from and say great, is costing me too much today because he's running some demos that didn't always work.
Uh and then secondly, you can also see the cost for at the statement level, right? If you want to go diagnose, hey, what was the cost for this specific query, you can go see that. Just to be clear, we don't charge you by the query. We're just doing the math for you. So you can have some attributed cost uh per query. As you're doing this and trying to understand cost, one of the drivers of cost is actually workloads that run all the time, i. e.
I. e. a user may have gone and put a AIB dashboard to refresh every hour when you actually want it to run every day and obviously the cost for that will be different. So pretty soon we'll release a system table that tells you the schedules runs every day runs every hour that kind of metadata for all the data bricks assets right we don't know the schedules for PowerBI or Google sheets but we do know the schedules what's happening in data bricks and we can publish that for you in a system table.
Uh the next one is default warehouses. Uh again, you saw clicking on the warehouse dropdown a few times and selecting his warehouse. So what happens sometimes is you'll go run a dashboard and the author of the dashboard has set it to run an extra large warehouse. Fantastic. It's going to run fast. I see somebody giving a thumbs up already. He knows where this is going. And so then they come to a SQL editor and they just want to write a random query, but we just picked the last warehouse you used, which happens to be extra large when in fact the small warehouse is probably fine.
And so default warehouses solves that problem. What the admin can do is say across this workspace the default warehouse is X whatever size what name that you want. This way when users go to it go to the SQL editor they don't go snap back to the extra large warehouse. They get small by default. They don't have to do the drop down stuff that was doing. And by the way the users themselves will be able to say hey I actually want a different one because I'm a advanced user.
I want to use a large warehouse occasionally. So they can go and change that uh on their personal settings as well. This will be in private preview in u Q2. I'm out of time so I'm going to go a little faster. Query labels. So again, if you're running all these uh queries from let's say PowerBI, Tableau, you might not know where which dashboard or which you you know the user, but you won't know the dashboard or the metadata uh of where those workloads are coming from.
So we're working on something called query labels that allow you to tag each query execution, right? You can set a label perhaps at the Tableau dashboard level or if you've got a custom app, you can embed query labels inside those things. This way when you get the query history in the system tables you'll know that each query came from a certain label whether it's your you know project ID or dashboard ID or some team name and you can roll up the costs uh from that cost query dashboard that I showed you at the label level.
Okay, warehouse timeouts. So a really important way to manage costs is to make sure runaway queries get killed. Right? Right? If you're running a PowerBI dashboard, you know that the query should take 3 minutes and if a query takes 24 hours, you are spending too much money and that customer is probably waiting for too long. So, we have this thing already called a statement timeout uh that's apply the workspace levels, right?
You can set it to be two hours for example. But very often you'll have different types of workloads running in the same workspace whether it's a BI workload or an ad hoc workload or an ETL workload. They need different timeouts. So in Q2 we will release warehouse level statement timeouts allow you to set let's say 10 minutes for the BI warehouse 30 minutes for an ad hoc warehouse and maybe two hours for the ETL warehouse.
This allows you to control the costs a little better. Okay, I think we're at time so I'm going to wrap right over there and I'll point to a quick slide which had a bunch of other features that we did not have time to announce today. Thank you so much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.