Skip to main content

Introduction to Databricks SQL

Summary

  • Lakehouse architecture: Databricks SQL combines data warehouse capabilities with lakehouse flexibility, operating on Delta Lake and governed by Unity Catalog.
  • Ease of use: Streamlined interfaces enable analysts, developers, admins, and business users to create warehouses, query data, and build dashboards in minutes.
  • Integrated AI: AI-powered engine optimizes performance and costs while supporting BI integrations and next-gen capabilities like AI/BI dashboards.

Introduction to Databricks SQL

Watch: Introduction to Databricks SQL
Talk by Himanshu Raja, Director, Product Management, Databricks, Databricks ; Pearl Ubaru, Technical Product Marketing Engineer, Databricks
If you are brand new to Databricks SQL and want to get a lightning tour of this intelligent data warehouse, this video is for you. Learn about the architecture of Databricks SQL. Then show how simple, streamlined interfaces are making it easier for analysts, developers, admins and business users to get their jobs done and questions answered. We’ll show how easy it is to create a warehouse, get data, transform it and build queries and dashboards. By the end of the session, you’ll be able to build a Databricks SQL warehouse in 5 minutes.
Here's more to explore:
The best data warehouse is a lakehouse: https://www.databricks.com/product/databricks-sql
How to migrate legacy data warehouses: https://www.databricks.com/resources/ebook/migrate-your-legacy-data-warehouse-databricks
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What is Databricks SQL?

Databricks SQL is an intelligent data warehouse built on the lakehouse architecture, operating on Delta Lake with Unity Catalog governance. This video introduces the platform's capabilities and architecture.

Can I set up a Databricks SQL warehouse quickly?

Yes, this video demonstrates how to create a warehouse, load data, transform it, and build queries and dashboards in approximately 5 minutes.

How can users query data without writing SQL?

AI/BI Genie within Databricks SQL enables business users to ask natural language questions about data, automatically generating optimized queries and returning results without requiring SQL knowledge. The interface understands data relationships and business terminology to interpret questions accurately and provide relevant insights. This capability democratizes data access across organizations, allowing analysts, managers, and executives to independently explore data for decision-making.

What are the best data integration platforms for OLTP and warehousing?

Databricks SQL demonstrates how modern data platforms combine OLTP capabilities for operational systems with data warehousing for analytics on the same infrastructure. Integration platforms should support diverse source systems like Salesforce, SAP, and databases while maintaining data quality and enabling real-time synchronization. The lakehouse approach eliminates the need for separate specialized systems, reducing complexity and cost.

What are the defining features of a modern lakehouse architecture?

Databricks SQL demonstrates core lakehouse capabilities including unified governance through Unity Catalog, seamless BI tool integration, AI-powered natural language querying through Genie, and real-time analytics on operational data. The architecture combines transaction support and data quality controls of traditional warehouses with the flexibility and scale of data lakes. These features enable organizations to consolidate multiple specialized systems into a single platform for analytics, data science, and AI applications.

Full transcript

0:00 Introduction to DBSQL

Good morning everyone. Uh thank you so much for being here. Really appreciate it. I know it's like 9:00 on Tuesday morning and it took a lot to be here. So thanks a lot. Um this is introduction to databicks SQL. I am a product manager in databicks uh leading DBS SQL. Hi everyone. Uh I am part of uh product marketing at data bricks and I'm happy to be here. Exciting. Uh so we'll talk about uh DBSQL data bricks SQL. Uh and the intention of this this session is it's a it's a very introductory session.
The intention of the session is for us to give you a quick tour of what data bricks SQL is. Um and uh I have some slides I'll talk about what introduce various capabilities of DBSQL and then will uh do we give you a tour like a real actual live demo uh pretty risky but we will we'll try our best uh and uh the intention is after this session you are actually familiar with data bricks SQL and you can just go and kick off tires and try out the product yourself.
Uh so we we have the survey so please definitely rank us uh give us uh give us feedback we look at it after the submit every year uh to see how we can make sessions better and lot of the time the feedback actually helps us to make the sessions better. Um so quickly moving to what is data brick SQL um so I just want to start with our our uh datab bricks platform and place you in in where what we are talking about today.

1:35 Platform and architecture

Uh so if you see the data bricks lakehouse uh platform you know we have the storage layer the delta lake or iceberg then we have unity catalog and then you have the products on top of these uh the product that I'm talking about today is the data warehousing offering from datab bricks and it's called data bricksql in short we call it dbsql so this is what the internals of like the data bricksql looks like this is how we think about it when we are building our road map and we are building functionalities and if you start it's a very it's it's nothing uh you know revolutionary but if you the the pieces that do make it unique uh I'm going to talk about some of those pieces and that's really towards how we think about a data warehousing in the lakehouse architecture uh so starting from the left and this this is a mental map for rest of our session of how we're going to uh discuss and talk about DBSQL uh so there is injection transformation information you have SQL capabilities then you can visualize your queries and then you can serve those results um and then these these can be accessed through outside uh with any of the tools uh so I'll talk about all of those capabilities and how data bricks SQL has a complete suit of capabilities for all of these things um we have had we launched data brick SQL about three years ago I think three three three years ago uh and since then we have had a tremendous adoption and usage of the product and today around 11,000 customers, organizations are using data bricks SQL as a database offering.
Uh so very exciting in terms of the feedback we get and how quickly we are able to iterate. The three key things I want you all to take away from this session and this is how we're going to we think about how when we build features and capabilities into data bricks SQL. The first is we want data bricks SQL to be the fully featured enterprise ready data warehouse. Uh and the key piece is that it's open by design. I'm going to talk a lot about how we do data warehousing but in an open architecture in open format.
Uh we continue to focus on ETL, BI and AI. All three use cases. uh for performance for TCO it remains the focus and I'll show you some some uh really cool graphs that actually prove that uh data bricks SQL is the best data warehouse for ETL BI and AI and then lastly uh I'm not going to talk about this uh migrations in today's session because this is an introductory session but there are many other sessions where we're going to talk about how we are making migrations super easy especially using uh LLMs just to convert code and cannot take like months or years to migrate to data bricks SQL.
So quickly talking about open by design that's a philosophy that you know like data bricks uh throughout the years we have really focused on how we have open architecture and no locket and let the best agent win. Uh so the first thing is is 100% on open formats. Uh you're going to hear uh big announcements in the keynotes. Uh one of the announcements you will hear about is around our eyes offering. Uh so now data bricks SQL and all the other datab bricks products uh run both on delta and ice uh it's built on open catalog UC is the is the fundamental piece of what makes data brick SQL uh uh really impactful data uh data warehouse uh and I'm going to talk about some of the capabilities that it enables but the key thing here is thanks to UC data bricks SQL is can be in the center of your data platform and it enables uh unified governance uh and access to across your data estate uh without going anywhere else.
And lastly, thanks to the the open APIs and the open drivers, we just have this ecosystem of partners uh and uh tools that you can connect your to data SQL. So, it's fully open and interoperable. uh in terms of you know when you start with data warehousing you're like hey the first thing I need to do is to ingest uh from any sources uh and we have a really big suit of products from using which you can you can do ingest starting from five tren to any other the data sources or applications uh we also have lakeflow connect which is the our in-house uh product that allows you to easily move data in um uh and get started um but I I didn't want to spend too much time on ingestion Because you know like injection is like hopefully if we are doing our job right injection should not be a problem and you just get started and it's like really simple.

5:55 Ingestion and BI integration

So when you try out the product you will see that how easy it is to get data in and get uh get started. Uh and that's also the beauty of the open architecture that when you have data present in your data lake uh you know you can just start quering it. So I'm going to focus more on the serving layer of the piece where you how do you serve the data? Uh so talking about the open APIs uh so the all data bricks SQL has the SQL API that you can use to integrate with your applications to get data.
Um also our J drivers the JDBC and ADBC driver ADBC is for PowerBI and JDBC is for all the other BI tools. uh they are also uh open sourced and that really helps us to improve performance and iterate very very quickly um uh in the community. So this is how you know you can query data from DBSQL. There is a whole suite of products uh from Google Sheets to uh Tableau, PowerBI uh you know your usual suspects and then we also have integrations with all the all the uh uh AI agents and apps that you can use uh to get data.
Uh spreadsheet connectivity is something that we we uh had uh launched very recently and you don't need to it's a plug-in. You don't need to connect anymore through the drivers and you can just like query data from Google Sheets. Pretty awesome, pretty easy. Um apart from PowerBI and Tableau, we have the AIBI in in the product itself and it's a really good way to get started with DBSQL. uh you know you can interact with your data, you can uh query the data and use our AI assistant to uh even if you're not familiar and have sophisticated SQL experience, you can just start playing with the data and this is what will demo some of it today.

8:32 AI/BI dashboards and Genie

All right, thanks. So here I've already created a dashboard. I did this prior to coming. Um, but I wanted to kind of share with you the background on this data set is just some sales metrics, some customer metrics about um, our bakery shop. And so really quick here, the great thing about AIBI dashboards is that everything is powered obviously by SQL warehouse, but with our data sets and these data sets are all registered in Unity Catalog.
And I'll share a little bit about Unity catalog a bit later. So I've added all of these data sets. You can add them from your source which would be uh from your from the catalog explorer or unity catalog. I've already added all the tables I needed and then you can also add from SQL. So if you wanted to create your own SQL code so in the canvas here once I have all my data sets added I can simply select down here which is my viz and uh place it here.
And the great thing is I don't even have to run any code. I could literally just type what I want and it will give me the viz. So in this case, I'm just going to say total uh product sales. Awesome. I even spelled quantity wrong, but I still understood what I was doing. Um, so as you can see, I didn't have to run any code, and it generates a lovely bar chart. I love it. So, I'm going to go ahead and accept that. Um, but I think what's equally as as cool is here I have a line graph of just adjusted amounts across our products um, according to date.
And so what I can do is I can forecast what these amounts could be really simply. I don't have to run any code either. I literally scroll down here and I click forecast. I just clone with AI forecast and boom, they are forecasting across next year and they give you the upper and lower limits and potentially uh I mean and the um the amount. So here again we try to make it super easy for you and your organization to get insights from BI without having to do much.
Awesome. Uh, so this is not a AIBI session. This session is really DBSQL. We did not bait and switch. But the intention is to just like help you understand that if you use DBSQL, a very easy way to get started with the analyst and just playing around with data is AIBI. Of course, there is integration with PowerBI and Tableau uh that most often people use with DBSQL. Um, the next very cool thing with AIBI is that when you actually use this is just the exploratory space. you know, you have the genie where you can just like ask questions in English.
Um, and it's governed and that's the key thing is it's governed and secured because of Unity catalog. Uh, so you can just safely deploy Genie uh on your data that understands your own data and semantics uh to help you start querying. And this is what will show next. All right, so back to our dashboard. Um, right now this is a draft version of my dashboard. So, I'm just going to go ahead and publish that. So, when I publish it, I'm able to embed my credentials.

11:44 Advanced Genie capabilities

That just means that I can share this with someone. They can see what's on my dashboard. They don't have to have access to the underlying data. And then, additionally, I'm going to enable Genie. So, that's going to allow me to ask questions using Genie on my dashboard. So, let's go ahead and hit publish. Awesome. So, as you can see, the Ask Genie app, Ask Genie button appears. I'm going to clear this. Sorry. And um all you need to do is click on a viz, any viz that you want, and then you can ask a question.
So, I could say, what is on this one, what is the average total sales of all my products? Yeah. So, you can see the key piece. What it's doing here is fetching the metadata to understand the context of what sales and products mean right and you can see the underlying code of how it's also um achieving this which is really great all right it is correct yes and again it's leveraging this vis visualization but what if we want to get more insight right this is fine but you know my dashboard doesn't have all the data sets I need for analysis and so what you can do is you can open up your own genie space and once you do that you're brought to a page I've already created the jin space because of time and um this is this took the all the tables that you saw in um the dashboard in that data tab it took all of them here and then I added some as well so that I can do some uh some analysis so I also changed some things here so here I can go ahead and just ask questions in detail so one of the questions I have is Can you tell me which franchise in Japan So, Genie is trying to figure out what I mean.
I don't have any information. It's saying, "What do you mean?" And so, what's really great is I can tell a genie what I need. I can say, "Champion stores are franchises that generate $600 So now Genie is looking through my metadata and all the tables that I have in my uh in the Genie space and it's trying to figure out which ones in Japan are my champion stores. So it's giving me the franchise ID and the name of the stores.
It's kind of knazawa confections, chibatus, and matsumia morsels. What's really cool is if I like it, I can of course say yes, but I can also say this as an instruction. So next time that I actually decide to or someone in my organization asks about Champion Stores, it's register. And so Genie now knows what Champion Stores is, they can ask any questions regarding Champion Stores. Additionally, I want to share that with um Genie um you do have to make sure that your data is um um organized but also documented.
So as you can see all of my tables have descriptions um and it also Genie registers any uh data modeling. So any primary and foreign key uh constraints that you have it does support that as well. And then sharing your genie space is super simple. You just grab your link and share it with some with anyone in your organization or you can add their name here and you can give them privileges to manage the actual genie space.
They can edit it. They can just run it or they can view it and then you can give them no um no abilities. So okay, back to you. Awesome. Thanks both. Uh cool. So now talking about uh unity catalog and data bricks SQL how you can have open governance uh with data brick SQL. So, so we totally understand that when you're using data bricks SQL or any other product, you still have all your data present in different paths, right?

16:16 Unity Catalog and federation

There might be data on prem, there might be data in the legacy data warehouses, there might be data in OLTP. Uh, and what you don't want is like doing these jobs to keep on duplicating the data and run it and collect it in the same place even before you have figured out whether that data is useful or not. Or sometimes it's just not possible, right? If it's like uh the data is literally being generated as we speak uh you don't have the time to bring it to your to your data lake.
Uh so DBSQL makes it really really easy with federation. There are two types of federation. This the lakehouse federation that we had for time that just simply goes to other data sources other warehouses or databases through the JDBC or DBC connectors and then queries the data uh in place. Uh so that's what we had uh for quite some time we saw we we are seeing very uh high usage of it just because it makes it so easy to bring all your data in one place.
Uh but what we also now have is catalog federation. And one really cool thing that has happened lately with the with the emergence of open table formats delta and iceberg is that the communities have evolved that has figured out that hey if we actually have data in these table formats we can just go and query it. There is no need for two engines to run at the same time to just interpret the data. Right? So data brick SQL can go to any iceberg table or any delta table or any park table for that matter and just query it uh as long as you have a catalog u unity catalog can go to another catalog to talk to understand the permissions uh and enforce that governance.
Uh so we have lighthouse we have catalog federation now and using catalog federation you can query uh snowflake you can query any data in AWS glue in hive meta store uh and multiple other uh integrations are coming uh very soon uh including uh iceberg rest catalog uh so this really opens up a whole ecosystem of uh of uh engines that you can query from or data sources you can query uh in database uh in DBSQL. Uh the cool piece of uh lakehouse federation with unity catalog is that thanks to unity catalog you can have a single governance model across all those data sources.
You don't have to worry that with DBCQL you have given access to your analyst to some uh sensitive data uh that you don't that you know they're not allowed to. Once you define your policies in Unity catalog, that's all you need to do. The governance is applied across the entire data estate. Uh I'm going to quickly touch upon it little bit later. But the very nice piece of uh lakehouse federation is you can even define data pipelines uh ETL uh using materialized views without actually knowing and you know like having any experience of creating a complicated pipeline.
We do that for you automatically. All you need to do is like create MV from source and the source could be anything. Uh and then we just create a pipeline data pipeline that we keep up to date under the hood uh uh for you. Uh data warehousing, data modeling is very very common uh use case. Uh you need to have some understanding of semantics of how to define relationships. Uh so DBSQL supports primary and foreign key constraints that you can define in the schema.
We have identity columns just to keep track of the row ids and then we also have the enforce check constraint that you can define for the data quality. Uh so all of this together makes it really easy for you to do any kind of data modeling in DBSQL. Uh so lastly this is a new functionality of uh Unity catalog the UC metric views uh which basically allows you to define your business metrics in Unity catalog. Uh so then Unity catalog can now act as a complete semantic layer for your entire data house.
Um uh which uh which allows you to then understand the relationships and it it you can push those uh to uh they're automatically available in AIBI but you can also sync them with PowerBI for example. Uh now let's see a quick demo. All right. So here I'm in the catalog explorer. This is the unification area of data bricks. And so as you can see all these um sorry these are cataloges and these are feeding from the cloud your cloud object storage by default.

20:58 Catalog and source demo

Um but what I wanted to share with you is how you can um have a foreign catalog which is basically lakehouse federation federating into a different source right but if I show you um for example this is a catalog called bake house and within bake house I have several schemas and then I can click on sales and then within the sales schema you can see all of my tables. So uh with Unity catalog we have a threelevel hierarchy catalog schema tables but we also have views that can be within um within um schemas as well.
We can also have volumes and models. Volumes are a way to store non-tabular data. So like images um video and all that fun stuff. Um, and then of course models which is really cool and I'll show you that in in a little bit how you can leverage a model that your ML team might have created in DBS SQL. Um, so with all of these uh the great thing is that you can set permissioning. I don't know if you saw but when I clicked on bake house you can set permissions at the catalog level.
Uh you can give uh privileges to users or groups of users to use your catalog and give them all sorts of um privileges here which is super important. I've had someone overwrite my catalog before and it broke my heart. So then you can also do uh within the catalog I mean within the catalog there's schemas like I said you can also provide permissioning here. Same kind of thing grant and revoke and then you can do similarly with uh tables, views, functions, models all of that can be governed right here in one place.
So um I was talking about uh uh not uh federated sources right? So, with a federated source, you need to create a foreign catalog. But before you even do that, you need to have a connection, right? And so, what you would need to do is go up here and you click create a connection. And you can give it a name, but I I've already created. I just want to show you how you would do it. Um, and we have several connection types that you can connect to.
This is really really awesome. This is a really quick way to just start working with your data within DBSQL. I'm going to do Snowflake. So, as soon as you select Snowflake, you have the ability to authenticate in with a username, ooth, any of that. I've already actually done that. So, I'll show you my connection here, Snowflake demo, and I'm going to show you how you can go ahead and take this Snowflake demo. Uh, you can also set permission permissions on your connection as well to let people use it if they need to.
Um, and then you create a catalog and then that catalog will be the will house the data from your from your snowflake instance within data bricks. And I'll show you what that looks like as a complete as a completed uh function. So I created this one called the snowflake demo catalog. And as you can see they have my snowflake demo connection here and it's saying that it is a foreign catalog. I have my public schema and then I have my performance table and I can still see uh the descriptions and the types of data I have within Snowflake just from DBSQL I mean just from data bricks sorry cool and this looks correct I can go to my Snowflake instance and make sure that looks good I can go to my data tab HR data public go to my tables which is the performance table columns look the same and the data looks the same so we're mirroring what we have in Snowflake right in uh right in data bricks and I can query this too and I'll show you that in a in a little bit but for now we'll go back to awesome live demos always risky.
I'm glad it worked. Um, yeah, because you know, you're querying two systems. You're pairing data bricks SQL and uh Snowflake. Uh, so yeah, it's it's pretty awesome that you don't have to leave data bricks SQL and you can just bring all these sources together. Uh, cool. Uh, just zooming out a little bit. So, how do you get started with data bricks SQL? Right? You you launch a cluster and it's like really simple. We have made it super easy uh to understand how to launch a cluster.

25:28 Compute and SQL editing

You just like spin up a t-shirt size, small, medium, large, excel, whatever you want. Uh you define max and min, number of clusters for concurrency, how many concurrent users you expect and off you go. Uh it automatically uh scales up and down based on the workloads. Uh and we all automatically have concur concurrency concurrency and query routing to route queries to the clusters to get maximum utilization. Uh so now quickly talking about the SQL experiences uh what you will do once you have the data warehouse uh just in terms of the capabilities these are just like a quick snapshot of the capabilities that uh we have um it's a complete enterprise data warehouse a bunch of new capabilities are being announced uh in the summit uh the ones I want to highlight is around multi-table transactions uh temporary tables through procedures uh various and data support is all available in in data bricks SQL.
Uh now uh I kind of touched upon this. This is a really easy way to get started with building data pipelines without even knowing how to build data pipelines which is just you create a materialized view uh for your any data source including delta iceberg or uh uh federated data sources. Uh and the beauty of this this using an MD is we will make sure that the uh the data is refreshed automatically um and it's incremental.
So we we we incrementalize the refreshes so that we are not doing full table scans and it's so it's cheap, effective and fast. Uh cool. So once you have your data, you have your DBSQL cluster running uh you want to interact with uh with your data. We talked about AIBI which is the visualization piece of the data but you know when you want to interact with the data you want notebooks we have full data bricks always had an amazing notebook capability and now you can use those notebooks uh with data bricks SQL as well uh they are multi- language they are collaborative they are Jupiter compatible they are idle for exploration they're adaptable they're reproducible and they just get you to production faster uh if you are using if you are familiar amilar with that kind of environment the data engineering space and if you're not if you're just used to uh the SQL editing experience and you want a simple interface we have that as well uh will demo that a little bit later but you know the very cool piece of integrating LLMs into into uh this whole data brick space is that you have the AI assistant that can help you to write code or help you to fix mistakes as you are going along uh and I think instead of me talking about it we will just like to see that from.

28:41 Query editor live demo

Awesome. So before we get started, um I just wanted to quickly touch on the SQL warehouses that um Hamanchu touched on in the slides. So creating SQL warehouses is super simple. I've already created two of them um or leveraging two of them. This is a serverless one. Um, this is the elastic compute. So, it's super easy to spin up and spin down and automatically manages um your your your warehouse, but with Pro, it's it's a step down.
I'm not going to lie. It's a step down. Um, it's still very powerful. Um, and in this demo, I will probably use the serverless one just to show you the extent of what it can do. So, I just wanted to show you the area where your SQL warehouses are. Okay. So, um this is the SQL editor and um here I have my um I want to select all from my Snowflake instance just for you to take a look and see that you can actually query uh data from Snowflake here.
Awesome. So, it is working great. And so I actually want to go ahead and create a view from this because I don't want to keep um you know hitting this table. So I can go ahead and just add create or replace material view uh what is it? Snowflake record view as okay let's run that. Okay so it looks like I have an error. And this is where the assistant comes in. And it's literally telling me, "Hey, you made a mistake in the red."
And here's the correct way to write this. Something as simple as a spelling error um can obviously mess up your code. And it's telling me that I should just replace it with the correct answer and just run it. This is just one of the small ways the assistant works. You can also in natural language ask it to do something for you. You can literally say, "Join these two tables and create another table for me called this."
And it can do that. And I think that's really powerful especially for someone that may not be super super skilled in um SQL. Um a really important thing here uh this view that is being run uh we are creating it from our snowflake table but I want you to know that we are writing it to a um to one of our cataloges here in um on data bricks. So we cannot write tables back to snowflake or views. So it's just mirroring whatever that's a snowflake into data bricks.
Yeah. Great question. It's quing. Yeah, it's quing directly snowflake and then when the materialized view gets created then it's going to process the data in Yeah. Yeah. It's in a in in actually delta lake format and you can do that in iceberg as well. Okay. Awesome. So my um view has been created and I can verify here in my catalog. I'm going to go to my bake house HR and then my schema HR data just to make sure. Open that up.
And it looks like I do have my snowflake perfume. We are good to go there. All right, back to you. Awesome. Uh, cool. So, I'm just going to rush through a couple of things. Uh, mainly because I want to give you a sample of what data bricks SQL can do instead of going super deep into these topics. Uh but the key thing of with DBSQL we've been focusing how do we help you integrate AI into DBSQL and just make it a very easy way to get started.

32:20 AI integration and functions

I understand there is like so much noise around AI these days and there are very useful applications that are coming on like what told you know like with interacting with data uh and using natural language to interact with data. So with DBSQL our intention has been how do we give you a very easy way to interact with AI. So now you can uh degrade any uh any of the models that you want including your own custom model that have been approved by your by your company inside your company and in hook it up with DBSQL uh to query your data.
Uh the querying using AI is also very easy. So you can have you can query within DB DBSQL and call your model that you have and it just simply like select AI query and you can literally talk in English saying summarize the pros and cons. Uh and then that just creates the calls the model and gives you the results. Uh we've been very focused on performance. I can understand we understand that what happens very quickly is that the performance goes out of whack as you are querying because it's very very easy to keep on quering these these uh systems.
Uh so so in terms of batch inference throughputs uh data brick SQL has extremely high throughput when you are calling these models. Uh so this is a complete suit of capabilities. I'm not going to go into details but the two things I want to highlight is the fact that you can call your custom model is like very very impactful. Uh the AI parts can be used to parse documents and unstructured data and put it in structured data which is very useful capability.
And then the AI query that I already talked about that allows you to just query uh models uh in a natural SQL uh format. Uh awesome. will show some of these capabilities. All right. So, let's say um you have a bakery like I shared earlier and a lot of people are leaving tons of reviews on Yelp or Google and while there are several reviews, you want to create responses to those reviews as quickly as possible and um you don't really have a lot of you know AI or you know ML background or skills. um but you do know that your ML team has created a um a model and so what we've done with in data brick SQL is given you the ability to query um not just a custom model as you can see here but other foundational models um like chat I mean uh open AAI and claude and stuff like that but I want to show you here how you can do it with a custom model and all you need to do with just a few lines of code is uh call the AI query uh function.
And these are built in. So, I'm just going to hit run. Uh oh, I know what's wrong. Okay, let's see. Do you want to say what happened? All right. The assistant is rescuing and coming to save the day. Let's see. So, what was missing? I'm not sure. Nothing. I I didn't really change anything. Um, but as you can see on the left here, I have my review that was from Google. And then it has automatically generated a response to those reviews. uh leveraging that custom model and so this was just to show you a small example that's why I limit it to one but if you wanted to do this at scale you can just increase that to a thousand or however much that you might need and then additionally um like Hamu shared um all of these are built in so if I remove query you can kind of see all the different functions that we have we have analyze sentiment we have classify I use AI mask a lot mask some certain um data and so we make it really really seamless was for you to leverage AI within DBS SQL.

37:12 Monitoring and performance

Awesome. Uh just quickly rounding it out, uh once you have DBCSQL, you are using it, you know, your admins or whoever is responsible for paying the bill wants to or the permissions and looking at the entire data estate is now wants to start monitoring it. So we have built lot of capabilities in DBSQL to help you monitor the how people are using it where the opportunities of optimizations lie uh in a very seamless and easy way.
Uh our first philosophy is we should just give you the data and you can integrate it in any tool that you want. That's our uh building block. So all of this information is available as system tables uh uh to you uh that you can just build on top of it. uh we have multiple system tables that can be used but then we also have dashboards. We have these cost observability dashboards that can clear that can help you see where majority of the usage is going, who's quering, what is the cost per query is.
Uh so all those capabilities are available. Uh and then will just quickly do a tour of these capabilities. Sorry. All right. So remember our SQL warehouses. Um I'm going to click into the serverless one here. And within the warehouse, if you do have admin privileges, you can monitor your warehouse. This is where a lot of our customers spend time just seeing how their queries are running. Um it gives you really quick um and simple um insight in a really nice layout.
Another really great thing I like is this query history portion here. Um you can see what obviously what queries you're running, but if you click into it, you can kind of see how much time is being spent scheduling it, optimizing it. And if you click this uh query profile, you're able to see in a treelike format, which is really great. So, if you ever want to debug or see what's going, maybe your uh warehouse is running slow or maybe you're not running uh you're not optimizing your queries as much, this will be a great guide to help you figure out where that is.
Um and then of course you can leverage the assistant for help on how to do that. Um so yeah, that's it. Awesome. And uh quickly wanted to end on the uh just in terms of performance remains our big focus. I just wanted to show this one graph which is the actual customer queries that we measure every year, every quarter and what the performance improvements are on your workloads on repeated queries uh automatically without you doing anything.
Uh and we we focus across the stack. We look at BI queries, we look at interactive data, we look at ETL queries and this is some of the just performance improvements that have gone gone out in the last one and a half years uh without customers doing anything. Uh I wish I had time to talk about this but the key thing is our goal is to replace the knobs using AI. So we have this predictive optimization engine that is running is using unity catalogs data to understand how read and writes are being done to automatically optimize the data.
So using DBSQL uh you don't have to worry about the data layouts. You don't have to worry about the workload management. All of that stuff is taken care of. So just wrapping it up three things to remember about DBSQL. Uh it's a fully featured enterprise data warehouse. It's open by design. It's unified by the architecture and it has the lowest TCO for AI, BI uh for ETL for interactive data analysis. And we're making migration super easy uh so that you can e migrate to DBSQL in a very short time.
You can play with the cookies data set uh that just showed uh using this QR code. Um and you can you should I suggest if you are interested in learning a little bit more I really encourage you to attend any of these three sessions especially which is what's new in DBSQL cooking with SQL uh if you want to do a little bit more deep dive and then uh data warehouse migrations to DBSQL if you're interested in DB. Can I plug mine too much please?
Um, tomorrow I have two sessions. Um, it's a 90minute deep dive and I'll literally take you end to end on all of this. So, you'll have a hands-on experience. You'll have your own workspace. So, if you are interested in that, I think it's almost full. I think we have about 180 people registered. So, we only have 20 room for 20 more, I think, in each one. So, if you're interested in getting some hands-on experience with data bricks SQL and having an opportunity to a workspace for about a week to play with it, um, come check out my session tomorrow at 150.
If you look up, you should see that. Yes. Awesome. Thanks a lot, folks. I hope this was cool. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.