Skip to main content

Why the Best Data Warehouse Is a Lakehouse: Databricks SQL, Serverless and Lakebridge

Summary

  • Unified platform: A lakehouse provides all BI, analytics and AI capabilities on a single open, governed platform, eliminating the need for separate data warehouse and data lake stacks.
  • Performance optimization: Predictive optimization automatically handles statistics, file sizing and liquid clustering, while serverless SQL warehouses start in seconds instead of four to five minutes.
  • Automated migration: Lakebridge provides open-source migration tooling with agentic code conversion to lift-and-shift warehouse stored procedures to the lakehouse, enabling incremental migration without full rewrites.

Why the Best Data Warehouse Is a Lakehouse: Databricks SQL, Serverless and Lakebridge

Watch: Why the Best Data Warehouse Is a Lakehouse: Databricks SQL, Serverless and Lakebridge
Talk by Esha Deshpande, Sr. Solutions Architect, Databricks; Prashant Upadhyay, Sr. Solutions Architect, Databricks
Lakehouse vs data warehouse comes down to one question: do you run BI, analytics and AI on a single open, governed platform, or maintain two separate stacks with data copied between them? This deep dive makes the case that a lakehouse is the better data warehouse, and shows how to modernize without rewriting everything: predictive optimization that automates statistics, file sizing and liquid clustering; serverless SQL warehouses that start in seconds rather than the four to five minutes a classic warehouse can take; automated lineage in Unity Catalog; and lifting and shifting warehouse stored procedures onto the lakehouse.
Learn how predictive optimization automates statistics, file sizing and liquid clustering, how serverless SQL warehouses start in seconds instead of the four to five minutes a classic warehouse can take, and how Unity Catalog captures automated lineage across tables, dashboards and workflows. The talk covers Lakeflow connectors and ZeroBus for ingestion, dbt and query federation, stored procedures for lift-and-shift, AI functions and Genie for natural-language analytics, and Lakebridge, the open-sourced migration tooling with agentic code conversion. A live demo walks through lineage, serverless compute, federation and AI functions.

Chapters

FAQs

Why do companies maintain separate data warehouse and data lake stacks?

Historically, data warehouses were optimized for BI and structured analytics, while data lakes were used for AI exploration and predictive analytics. Organizations maintain separate stacks because these different use cases required different tools, leading to data duplication between systems.

How does predictive optimization improve lakehouse performance?

Predictive optimization automatically handles statistics, file sizing and liquid clustering without manual intervention. This automation reduces query performance tuning work and helps the lakehouse automatically adapt to your data patterns and query workloads.

What are the benefits of serverless SQL warehouses?

Serverless SQL warehouses start in seconds rather than the four to five minutes required by classic warehouses, providing instant query performance. They eliminate the need to manually manage warehouse clusters and scale automatically based on demand.

What is Lakebridge and how does it help with migration?

Lakebridge is open-source migration tooling that uses agentic code conversion to automatically lift-and-shift warehouse stored procedures to the lakehouse. This enables organizations to migrate legacy warehouse workloads without requiring complete rewrites of existing code.

Full transcript

00:00 Introduction

Hello, everyone. Good afternoon. Thanks for coming to the session. I know what time it is after lunch, right? Warm lights, heavy eyelids. So don't worry, we will be here for 90 minutes to sleep or learn, depending on your choice. But yeah, I hope it is mostly learning. I am Prashant Upadhyay, Senior Solution Architect with Databricks. She's my colleague, Isha. Hello, everyone. Isha Deshpande here. I hope you can hear me clearly.
Senior Solutions Architect based out of the Bay Area over here. Nice to see you all. Today, we'll be talking a little bit about why the best data warehouse is the Lakehouse. In the back half of today's session, we'll also be having a demo that Prashant will be showing. And I'll get you all started. it. So buckle up. So the picture that you're seeing on my screen over here right now is the one picture that explains the whole problem.
As companies are climbing up the data and AI maturity curve, from understanding what happened and generating reports for decision making to hitting a wall right in the center before they get to answers of what will happen in the future. If you look closely, there's a split. The data warehouse is more for BI, clean data, reports, and more ad hoc queries that help us answer what happened in the past. And the data lake on the right-hand side is more for AI exploration and predictive analytics.

01:47 Why warehouses and data lakes split into two stacks

If you look closely, there are two separate stacks. Two separate stacks means two separate copies of data, two separate governance models, and also two separate bills that you need to pay. The most valuable use cases that you are solving for are going to be predictive, prescriptive, something that drives automated decisions. And these need both types of data. Data that is coming from the data warehouse as well as data that is residing in a data lake.
Now since you need both these engines to make your decision and there is a wall in between, you're copying from one place to the other. I want to get a gut check of how many people in the room over here are maintaining a pipeline that copies data from their data warehouse to their data lake and back and forth. Is there anyone in the crowd? OK, there are a couple of hands. So hopefully after the session today, we're going to get rid of those pipelines.
Now, just to understand, why does this split persist? So for more than 40 years now, the data warehouse has carried the same original sins, the three original sins of being closed, standalone, and being expensive. Closed because most data warehouses are on proprietary formats. So proprietary formats, you have standalone warehouses, they drive their own compute, which is separate, and then they end up being more expensive.
So if you're trying to get out of that situation where a data warehouse is just acting like a gym that you are paying for, but you're not really using as much as you'd like. With the data lake house, what we are doing is we are making it open by design. So hence, it's not closed anymore. It's open with respect to where your data sits, which format you choose to put your data in, and it ends up being part of an open ecosystem with 200-plus integration with different types of technology.

04:27 One lakehouse for BI, analytics and AI

So you can ingest data from anywhere. It resides in one single place for you to do both your what happened analysis as well as what will happen in the future with predictive analytics. Now, let's try and unpack what is powering the data lake house under the hood. And that is what we call Databricks SQL or DB SQL. So data warehousing under one platform, one copy of data, unified governance for AI that understands the enterprise context sitting on all the same tables.
So then there's no question about one team arguing with the other because one of them is pulling data from the data warehouse and the other team is pulling it from the data lake. The wall that you remember from the earlier slide of the data and AI maturity curve, it's completely gone. It's not federated. It's just gone. It vanishes. If you remember Ali's keynote where he said there are all these separate islands of data lakes and data warehouses, DB SQL is what brings those two islands together of data warehousing as well as data lakes.
And now with the latest announcement with Raiden computing, Lakehouse real-time has been introduced, which essentially gives you the advantages of a lakehouse, but at a much more performant way. So with real-time lakehouse, what you are doing is you are able to bring in real-time analytics to the same data lakehouse infrastructure. Today, in the session, we'll be trying to touch upon most of the boxes that you're seeing here today.
And in the back half, we will be leaving room for more questions. So bring your questions to us, and we'll try to answer them as best as possible. Now, this slide, why is this important? This is important because this is not an experiment. 60% and more of the Fortune 500 companies are using the Lakehouse today in production. And we're going to dig deeper into this by understanding how these companies are delivering value with the Data Lakehouse.
So to understand how they're delivering value, it all starts with this unified ETL approach, where more than 14,000 customers are already using Databricks' Lakeflow to deliver high-quality data. Today, teams with SQL-based knowledge are also able to build extremely sophisticated data engineering pipelines that in the past were only something that data engineers with Python or R knowledge were able to build. So what we have done is, by bringing DB SQL and making it a more natural language for the users that are comfortable and proficient in SQL query, are able to not just ingest, but transform their data and also manage the orchestration of those pipelines natively in one single platform.
It's not a separate ETL process which is bolt on top with a separate tool. It's all managed in this one single unified platform. And what is important is that 14,000 number that I quoted earlier, which shows that this has already been battle tested by so many different companies out there, and they have, in fact, vouched for how efficient the data lake house is. Now, I'm diving a little deeper into how the architecture and the process looks on the Lakehouse platform.
It first starts with the, on the edge if you see there are different data sources. Those are external tools that you're bringing data from. They could be coming from a Kafka streaming pipeline. They could be a message bus. It could be data coming from SharePoint, Workday. It just doesn't matter. All of that data can be ingested easily and brought onto the Lakehouse platform. It can be transformed using multiple different ways.
I will get into it in our unified architecture slide later. Then query it, visualize it, and serve it on any of the applications of your choice. Now, a lot of people would say, if I'm getting started with my journey, this is extremely simple to understand. But a lot of enterprises out there, they already have acquired a lot of tech debt where they have applications and tools that are already in their ecosystem. So this is a very rosy picture of what the end state would look like.
But in order to get there, we also play extremely well with various tools and ecosystems. So where you have the Databricks tools that help you connect, build pipelines, and manage jobs, if there's something like a DBT or a 5Tran that you already have and you want to leverage, we have the option of doing that. Similarly, if you have Databricks tools like AIBI, Genie, there were a lot of announcements about Genie ontology that if unified at one single place will deliver a lot of value.
Though that is true, we understand that the tech debt that you all have want you to leverage Power BI, Excel, as well as Tableau and Lookers of the World. So we have integration with those softwares and those tools as well as part of the ecosystem. And ultimately, yes, we are proud of Genie. We are proud of Omnigent that we are open sourcing. But along with that, we also play very well with other LLM ecosystem out there.
Once you centralize all of your data in the lake house and make it your new data warehouse, playing with all these other tools is going to be extremely simple. I touched briefly on a lot of things that you can do when you centralize your data on this lake house. But all of that you want to do along with an added layer of governance on top. Unity Catalog is that one place where you can put governance across all of your data.
Again, referring to the first slide that I had shown, where you had data warehouse in one place and a data lake in another, what happens is you're managing two separate governance infrastructures. And because there are two separate infrastructures, the overhead of managing them is a lot. When you combine it, everything is managed through the Lakehouse, through Unity catalog that is slapped on top of the Lakehouse. So it makes things much more convenient.
Now, to dive deeper into how Lakehouse exceeds all expectations, I would like to pass it on to Prashant to dig deeper. Thank you, Isha. So moving on. So we saw the overview. At least now you understand what Lakehouse is and what the positioning is. So let's dive deep into how it can meet your expectation. We will build it on top three pillars, which is like AI-driven optimizations, intelligent experience, as well as what the unified architecture looks like.
So coming to the first pillar is AI-driven optimization, right? And in this pillar, we'll be focusing on how it improves your data layout, as an example, automatically without even doing anything, right? How it will help you with data engine optimization. So let's get started. So right from intelligence perspective, right, for data management, these are the challenges if you see in the center. That's where basically, you know, if just by raise of hand, like if you can tell me how many have done, you know, vacuuming or partitioning just to improve the performance.
Right, yeah, so many of them, right, yeah. So like, you know, you have to think of partition, you have to think of, you know, which column has to be clustered as an example, what the update stats looks like so that a query plan can look good. Should I add index or not? What about small files, right? Because, again, a lot of small files create a problem, too. Then you have to have a compaction process to merge the small files together to get the performance.
And then how do you make a query faster, right? So all of these optimizations which you have to do today, Lakehouse solves that for you, right? So that's the intelligence part of the platform, is basically it is intelligent enough to basically drive all of these manual work or tuning which we have to do into an AI-driven optimization. So you don't have to worry about any of these questions. And believe me, this is true, right?
So the platform does that. And then I will go in for the slides to show you how. But again, right, you don't have, what we are trying to do is basically not improve the knob. We are trying to just remove the knob completely. So how it solves the problem in the backend, right? Again, from the platform perspective, like what it does, it understands the query. So if you see at the top, right, there's a query pattern. So the platform keeps intelligently understanding what a query is, what an access pattern looks like.
It passes on to an AI-driven model. Again, right, there is no cost to customer. It is part of the platform where it does all those intelligence and automation for you in the backend. And that evaluation and training data is passed with the predictive optimization where it does automatic statistics for you, auto file sizing for you, so no more compaction, no more small size problem, auto liquid clustering, so you don't have to worry about which key to cluster with for performance improvement, an example, and auto predictive I.

14:46 Predictive optimization: auto liquid clustering and file sizing

U. All of this predictive optimization inputs are passed to the data layout from storage perspective, which can either be uniform or delta, so iceberg or delta, right? We cater to both. And then also the engine optimizations with respect to query plan as an example of what files to prune, what to look into. And this cycle keeps repeating. Basically, some data engineer might have developed some query, say, two years back or one year back, right?
What we are saying is you don't have to worry about maintaining that going forward. In case even your access pattern changes, the platform is intelligent enough to auto-cluster those data in the back end for you, and then make sure you get the best performance out of it, right? So what we have seen in the real field is the statistics on the right side here is 20x faster query time performance and 50% lower cost, right?
Again, with respect to data optimization and the data layout, we save a lot of storage cost as well. So there might be a question on, like, what is liquid clustering, and then what is automatic liquid clustering, right? So again, liquid clustering is one of our ways to improve performance. So you don't have to worry about partitioning or you don't have to worry about Z-ordering or indexes. It takes care for you, right?
So it completely eliminates the data layout decision which you have to make, full autonomous optimization and powered by predictive optimization which I was showing you before, right? So it has the telemetry of your workloads. As part of the platform, it understands what kind of queries you are asking. what query plans are being generated, and then it optimizes in the backend for you. But then how you enable this automatic optimization, automatic liquid clustering.
To do that, you have to make sure your data are in managed table. It's not external, but managed. And what I mean by that is basically we are not saying when you do a managed table, we will remove your data from an existing S3 and move it to some property data format. That's not what we do. By managed tables, we mean it will still be your S3 location, wherever you have put your data in, but the path will be managed by Databricks engine to make sure it is much more optimized.
So all the data layout which it has to change, it has to make sure it has access to do those changes. That's where managed plays an important role. And this is a simple command. In your existing external table, you can just alter the table and then set it at managed and it will become a managed table for you. You don't even have to worry. So once you have done the managed table, that's where all the auto predictive optimization, auto liquid clustering comes into play.
So from auto liquid clustering perspective, you can, instead of, so earlier, it was like manual liquid clustering where you had to understand your access pattern and define the columns which you needed to do for liquid clustering. Now all you have to do is alter table and cluster by auto. You don't have to worry about column changes. Again, in future, if data access pattern changes, you have to rename the column or change the column or add more column.
You don't have to worry about any of those. So if you see the benchmarks which we have put here, right from workload perspective, we have benchmarks for ATL, we have benchmarks for BI and analytics, and we have benchmarks for AI. So from TCO perspective, if you see on the left side, right, We have TPC-DS data, which is kind of industry standard. And if you see that as the data size keeps growing, the number lower is better, right?
So the red one is basically our Lakehouse engine, right? And as soon as the data grows, the performance gets much more better and the cost of ownership reduces on Lakehouse platform. And over the past two to three years, we have made a lot of performance improvement internally without a customer having to do anything, and the performance have increased 5x, starting from, you know, 22 to 25. And even from throughput perspective, we have now 3x higher performance as compared to any of the completer's benchmark.
And this is, again, the latest announcements with respect to Raiden, where we are saying DBSQL or Lakehouse was basically giving you second-level query performances. Now this dives, Lakehouse Raiden, or real-time dive-sleep into a millisecond latency, right? So if you have serving needs where you have to serve some applications, example from, you know, customer segmentation perspective or even from if there is an external internal dashboard which you are maintaining, right, or operational dashboards you are maintaining.
So from those perspectives where you have, like, very high latency requirements, this will, low latency, high serving requirements, Raiden is the key, right? So you don't have to worry about, you know, just run the report, go take a TN, come back. You don't have to do that anymore. With Lakehouse real-time, we are saying you will get millisecond performance. So these are, again, like the test results which we did with different vendors.
And if you see, as the data size starts growing from small to x-large, some of the vendors were not even able to run the queries, right? Whereas Raiden was giving consistent performance. Coming to the second pillar, which is intelligent experience. So let's dive deep into what does that mean and how it can help you. So Genie 1, right? Again, there are a lot of announcements with respect to Genie. You all might be knowing by now, like, or maybe attending session to understand what Genie is, right?
And Genie, as a family or as an umbrella itself, has a lot of multiple different items inside it, one of them being like agent or chat. You can do like, so SQL is the, English is the new SQL basically, right? So you can start typing chat in English language and then it will generate DB SQL queries in the backend for you and then it will run reports. So everything that Gini does is backed by DB SQL, right? So it is powered by DB SQL.
And if you see here, Gini gives you a web end. It is also, we have launched a mobile app. we have just made an announcement on that also. So, executives can have genie spaces on their mobile phone and they can start asking questions. So, basically, that means you can start doing self-service. The executives can also do self-serve. And for any problems they see in the dashboard, as an example, there will be data to blame, not the engineers, right?
So, and then we have integrations with Slack, MCP apps, right? All of this is backed by DB SQL Lakehouse. So, as I was saying, talking to data, you can start asking questions in natural language to Gini, and there are two modes it can answer your question. One is agent mode where it can dive deep into it, or it can be a faster version in chat mode. So, you can not only ask what is going on, but you can also ask why it happened, right?

22:13 Genie: ask what happened and why in natural language

So, that why question where it has to dive deep and understand more and do the relations, right? you can do both of those in Genie. But not only natural language questions, there are also use cases where you have to do visualization, you have to have dashboards. So we are also doing a lot of innovations in dashboards. So AIBI dashboard, you can create a lot of different charts. You can, again, this is backed by DB SQL, Lakehouse as well, wherein if you want to do some certain analysis, as an example, what my sales looks like over the last three years, as an example.
And not only that, we take it a step further where you can start asking natural language questions to the dashboard itself. So you can ask for a given chart why the sales is down for the last three months, as an example. And it will do its analysis, and it will show you the results. Also, the best part, this dashboard does not have any licensing cost. So it is like fully functional enterprise-ready dashboard, but at no license fees to you.
All it charges is the consumption in the back end with respect to DB SQL, based on the query which you will be running. And for the people who are hard-coded SQL developer, who still love SQL editors as an example, we have made this, like we have made CGS investment in SQL editor as well. So it is AI ready now. We have AI assistant embedded into SQL editor where you can actually start asking natural language commands and generate SQL for you from the SQL itself, right?
So, you know, you can also have the SQLs as part of your CI CD pipeline. So with that, like in case any dashboard breaks, you will know who to blame for, right? You don't have to worry about doing an analysis, root cause analysis, who missed the data or what. So all the SQL code you will be generating can be backed in the Git itself. And then we keep saying, like, AI directly in SQL, right? How far does it go? This is the slide which is showing, like, within from SQL query, you can call any LLMs, right?
So on the shelf LLMs from the lake house, right, AI, Open, Meta, and not only those, even agents can query those Lakehouse or SQL, right? So you can have external agents, you can have agent breaks, and from Gini perspective, I was already talking about like Gini1, GiniMCP, they all can directly call to Lakehouse, and Lakehouse where you have already built your relations, right? You have normalized data or denormalized data, all semantics which you have built, all those contexts will be passed to those agents just from that single Lakehouse platform, right?
You don't have to worry about building a lot of different platforms for different applications. And also, like, throughput has increased a lot, right? If you see here, so from a computer which, say, gives you a 20-row-per-second output, ours will be 65-second-row-per-output. One of the other important factors coming to LLM is AI functions. So what we have done here is basically provided you most curated AI functions so you can directly call them and start working on unstructured data, say PDFs, image files.

25:34 AI functions for classification, sentiment and document parsing

So if you see here, before and after picture, wherein before if you have to apply AI on your data, you have to hire an ML engineer, you have to train the custom models, You have to do all these different things just to be able to run AI on top of your existing data, which you have. And then that existing data could be structured or unstructured. So there has to be a lot of other Python frameworks which have to be built to do the unstructured analysis.
But with this, it is just like one SQL line. So select text and summarize that text. That text can be a column in your data warehouse as in text format, or you can also have pointed to some external files as well. And in the demo, I will also show you how we can do some of these. And not only to few limited functions. This is a continuous improving platform where we are adding more functions every few months. So we started from a set of few functions.
Now it has grown to much more. You can do text classification, sentiment analysis, text similarity, or vector search, batch inference, document parsing. So all of the PDFs which you have, you can start parsing them, right? Call logs, all that can be parsed. Again, your data can come, say, in a structured format. You can keep it in volumes and then point that volume to your SQL query with this function, and it will analyze that data for you, and it will generate a structure for you. so this is one of the example wherein AI is directly integrated in the system and not only for text but you can do multi-modal analysis right so as I was talking like if you have image as an example you see here like from the set of images you can send an AI query and within AI query which is an inbuilt function you can you know pass any different models as well so suppose you want to use Maverick 4, or you want to use some GPD latest model, which works very good on images.
You can parse those model, set it, give it some prompt as well, and tell it what the output should be generated. So in this example, we are showing, like, filter out the catalog of the images, and then show me only the white shoes. So it will filter out all the images with white shoes and show it to you. Document intelligence from AI parse document function, wherein, again, you can start not only parsing the PDF file, you can start parsing any images of the document, write the tickets, or basically, if you have to do some reimbursement, when you click the pictures, all those can be parsed.
So the companies can actually start looking into what people are spending, as an example, or even from old PDF, from contract perspective, which no one was looking into before, AI makes them accessible, so that you can start looking into what contract for each different customer was made and start querying. See, as an example, when does, you know, our warranty expires on some of the products which we have built, right?
So using Databricks helps all of those, and then we have one customer example of how these AI queries as part of the lake house have helped them, right? Passed through, like, millions of files. So your PDF can finally read the... Sorry, your lake house can finally read the PDFs, right? Which no one wanted to look before. And with that said, we will dive into the third pillar, which is unified architecture, and I will invite Isha to continue.
Thank you, Prashant. Thank you for going over the different components that the Lakehouse is offering in order to help us achieve the unified architecture that I'm going to be talking about right now. So there are three jobs that the Lakehouse platform is doing, which is one platform is providing ingest, transform, and querying capabilities. Every company wants a single unified stack to do their data and AI initiatives, especially for BI that everyone in the organization can use.
So, the challenge has always been doing all these things without using three separate tools by three separate teams, because that ends up having argument between all these separate teams. So, let's walk through each of these parameters. So, the first thing that we spoke about was ingestion. Now, Databricks pulls from virtually any source, as I spoke about. Enterprise apps that include Salesforce, Workday, NetSuite, Oracle databases.
And via CDC, query-based connectors, and file sources, you can also bring data from message buses like Kafka. The methods range from various Lakeflow connectors to direct writes using APIs. guys. And if any of you all have been to the ZeroBus sessions, you know that Databricks also offers an offering called ZeroBus that will help you ingest streaming data without even having to maintain something like a Kafka instance in between.

31:15 Ingestion with Lakeflow connectors and ZeroBus

So, the key word over here that I want to emphasize on is managed. All of these things are easily managed by Databricks because most of the times you're not hand coding anything. you're not babysitting any connectors. All you're doing is you're just configuring the connection and you're running the pipelines. So all you need to do is connect using your Lakeflow connectors and configure it. That's it. And the ingestion pipelines keep running on their own.
And I also wanted to highlight that this is not just for complete ingestion, but it is also accounting for incremental ingestion of those pipelines. Now, the next pillar was the transform pillar amongst the three jobs that I spoke about. So for transform, here's something that genuinely is differentiated, which is there are three more ways to write your SQL ETL, SQL-based ETL, and all of them are, again, governed on this one platform.
So you're sharing the same tables and the same governance operations. If you're someone who is more procedural-oriented, you want to do a lift and shift of whatever you have on your data warehouse to the lake house, you have the option of doing that because you have stored procedures, multi-statement control flows, temporary tables and transactions that you can also bring in. So all of those engineers in the house that are familiar with data warehousing concepts, the procedural approach might resonate with you a lot.

32:37 Lift and shift warehouse stored procedures to the lakehouse

The second approach that is there over here highlighted is the declarative pipelines approach. So, if you have been to any of the declarative pipeline sessions, you know that someone who has SQL knowledge, it makes it easier for them to build those pipelines using declarative pipelines. But also we have the third approach for more citizen developers who are used to doing their data preparation in tools like Alteryx, where there is a more visual tool set, a visual UI that helps you drag and drop tables, make joins, and use the UI interface to build your pipelines.
Now, that being said, what I want to highlight is we're not forcing all of your teams to just follow one format. They can choose whatever they wish or whatever methodology they are comfortable in to build all of their pipelines. So from an execution standpoint, it's the same idea. Different personas that use SQL for different types of roles get the right type of experience without having to split the stack. So data warehousing engineers who want to modernize without rewriting their existing lines of code, they can just do a lift and shift and follow the procedural way of modernizing code and bringing it to the lake house directly.
The analytical engineers who rely more on building pipelines every day, They have the declarative approach that they can lean on. And then ultimately, the analysts who are not really used to building pipelines, they can also be part of this journey by building pipelines using the visual interface. And Prashant is going to walk you through a couple of these examples as well. Now, what happens under the hood? Many of you might be familiar with this, but just to reiterate for those who are not familiar, what is happening under the hood is a classic Medallion architecture that we are following.
The Medallion pipeline consists of raw bronze layer where you're getting all your streaming tables. The second part is your silver layer where you have your single source of truth data. And then ultimately the gold layer, which is consumption-ready aggregated data sets that you're building for different types of applications to ingest. So the magic over here is intelligent ETL optimization. Prashant did touch upon that a little bit, but it's incremental, it's more cost-effective, the kind of updates that it makes, and the engine processes only the changes that are being brought in.
So your CDC is completely taken care of, and you're able to incrementally maintain these pipelines easily. Now, remember in my previous leg today, I did speak about if you already have certain technology stack that you're using of the likes of DBT, we don't really mind. You can straight up go ahead, continue using DBT, and simultaneously leverage the power of a lake house where all of your data is unified, and you are still continuing to use the tooling that you are most comfortable in.
The second thing, one other thing I want to highlight is the orchestration is very native. So generally what happens is once your pipelines are built, you have to use another set of tooling for the orchestration part of things. If you are under the lake house architecture, you don't need to use a separate tooling because we do offer a built-in equivalent of your Airflow clusters that you don't need to babysit. What you can do is schedule and run those SQL queries, build dashboards, alerts, and all of this is under a fully managed infrastructure that is being managed by Databricks.
So you can orchestrate multi-step workflows, have dependencies, do built-in monitoring and observability, and all these dashboards are available out of the box for you and all of your team members that can be shared with each other with the right type of governance. And the number that really gets people's attention is 30 to 50 percent cost savings for just replacing something like an Airflow that you might have as an incumbent solution that you have been using for a very long time.
So just by replacing that and with the latest announcement of Genie ZeroOps that analyzes the failing pipelines and then sends you a response that actually tells you how to maintain those pipelines is game-changing. Now, I would like you all to just take a quick minute to orient yourself towards this grid. If you see, there are so many data warehousing features that are required for you to feel comfortable to move from your data warehouse to a lakehouse. and all of those are either available GA or in previews.
So you don't need to even second guess whether you're making a right choice or not moving from your traditional data warehouse to a Lakehouse platform. Now, the legacy ETL runs on native Lakehouse. So the migration headline that de-risks everything is legacy ETL that runs natively on the Lakehouse platform. The Lakehouse enables a true lift and shift of your existing SQL and ETL patterns across all clouds, all other platforms as is.
And you are able to have granular level of governance. and also this granular level of governance is not just on dedicated clusters, but is also provided on serverless warehouses that you spin up. So there's no big bang rewrite of anything. Another important thing that really helps people make the transition or make the move is spatial analysis. So spatial analysis really is supercharged. native geometry and geography types of data sets that are stored right in your tables that make it easier for you to visualize maps in AIBI dashboards that we provide.
And 90-plus spatial functions are there, high-performance spatial joints, with more than 15 times gain since it has been previewed. So every time we keep adding more and more functions and make it more and more performant. And again, it's open, so you have Delta sharing available. It's available on Iceberg v3, and it has contributed back to Spark. Now, to dive deeper into how you end up visualizing everything that you've built, like I said, we do have AIBI dashboards.
There is Genie inherently available on those dashboards without any extra cost. But if you have teams that are more comfortable in Tableau reports or Power BI reports and want to continue leveraging those types of integrations, those are also available. In fact, when Prashant shows you the demo, you could also see that on the cataloging site, if you are exploring different data sets, you can directly open those data sets as part of your Power BI or Tableau dashboards if you have those connections or leverage the inherently available AI BI.
I also want to reiterate that all of this is under one governance model. So you're not having to maintain a separate governance model for Power BI, Tableau, your data warehouse, data lake. It's all under one Unity catalog. Now, like I said, automated lineage is something for all workloads. So we auto-capture the runtime lineage or any cluster or SQL warehouse across tables, columns, dashboards, workflows, you name it.

41:34 Automated lineage and Unity Catalog governance

Any kind of objects that you have in Databricks, we auto-capture all the lineage. and we provide you end-to-end visibility of how the data flows and how it is being consumed. So there is complete auditability. And why it matters, like I said, when an auditor or an exec asks, where did this number come from? Who has touched this data? There is complete transparency in who has access to what data assets. Now, this is a very nice slide which talks about Lakehouse Federation.
Now, like I said, if you had to start everything from beginning from scratch with no tech debt, yes, you would have the Lakehouse infrastructure built. But since you already have a lot of data on your existing infrastructure, and this is not going to be a big bang approach where you lift and shift everything overnight, there is going to be a requirement of Lakehouse Federation, which is federating data from different data sources that you already have in your ecosystem.
So let's say you have some data in Snowflake and some data in your Microsoft SQL warehouse, and you have some data in Databricks already. And you want business users to combine all of this data and leverage the Unity Catalog governance structure, you can leverage Lakehouse Federation to do that. And that, again, is extremely powerful because you're not having to manage governance at separate location. And while you are in this journey of migrating and consolidating everything on one lake house, you are following the same governance strategy that helps across all these different platforms.
Now, something newer and something that's more powerful is our unified OLTP using LakeBase. So, this is zero ETL between your analytics as well as your applications. Now, as you all know, with the advent of AI applications that are being created, everyone wants their data in a Postgres-esque database so that this data is available on applications on demand. So LakeBase is that managed Postgres offering that we have. So picture it.
Your Postgres application tables and your LakeHouse analytics tables are completely in sync. So there is no brittle pipeline in between. operational and analytical worlds are finally sharing the same source of truth. So this is the next island from Ali's presentation that is coming together under one lake house. We spoke about data lakes, we spoke about data warehouses, and now this is the quintessential transactional OLTP databases that are the third island that are also coming together under this lake house.
Now, none of this will matter if it's not secure. So we do offer, we take pride in offering best-in-class security on the Lake House, a serverless data warehouse that you can completely trust. Three pillars, simplify your security responsibly, industry-leading practices applied by staff who's the best. So we really take pride in the kind of security that we are providing. We're simplifying. It's column-based, A-back control, R-back control, governed tags that are managed at the account level.
So like I said, none of it would matter if it wasn't governed, and we are offering that governance to you. Now, let me close my little chapter by the built-in features that we have to offer for SQL ETL workflows. This inherits with no extra tools to manage. It has built-in orchestration with Lakeflow jobs, like I said. The data quality and change data capture that we have under unified governance via Unity Catalog. You can have batch to real-time mode all in one framework. serverless, zero-tuning, compute, everything with AI-driven optimization, and everything is open and interoperable.
So no matter if you are using Antibase SQL, you have open table formats, you have stored procedures that you want to bring in, it all can be brought in under one roof. So with that, I would like to pass it on to Prashant to go over the demo. Thank you, Isha. I hope this is going good so far. We are more than half into the session. Nobody's sleepy. So, okay, we made it, right? So maybe let's take, you know, 10 seconds, just rotate our shoulders, and, yeah, get warm-up again for the rest half.

46:38 Live demo: lineage, dbt jobs and federation

Yeah, so you want to do with me? Maybe, you know, just some warm-up? Okay, I think we are good. We know we have the graveyard slot, So thank you for bearing with us. So yeah, let's get into demo now of whatever we have seen and then how it can help you, right? So let me, so I have to be glued here because my laptop is here and it has to be tied to the network. So I have to demo from here, but let me check. Okay, one second.
Respect to mirroring, choose content. Here we go. Yep. Cool. So even getting started to your lake house and starting querying the data directly, right? As the unified architecture which Isha was showing, you have to ingest the data first, right? So we have a lot of connectors inbuilt already as part of the platform where you can start ingesting the data. So what this picture is showing is basically, you know, if you go to on the left-hand side, you see data ingestion.
If you click on it, you will see this page wherein actually, you know, you can start ingesting from a lot of inbuilt connectors. Or, you know, you can also build your own pipeline using custom connectors as well, right? And we also have partners like 5Tran, right, GitHub, et cetera, as well, to start help you with the ingestion of the data. Once the ingestion is done, right, let me take you to the catalog of where the data gets stored.
So this is example as a Unity catalog. Most of you might be familiar with it. If not, this is the Unity catalog, which we keep talking about, which provides you the enterprise level governance, security, and stores different types of metadata. It is not only about table. It also stores materialized views. It stores streaming tables. It stores functions. It stores models. And then we also have something called as volumes, which is for unstructured data.
So you can attach your object store, as an example, S3 or ADLS as part of volume, and then it becomes accessible to the platform directly. So if I go to agent, as an example, if you see here, I have three different tables. I have another schema, pipeline bike, as an example, wherein I have this as a materialized view. So if you see here, this symbol tells me this is materialized view. We also have streaming tables. If you see here, it tells me this is a streaming table here.
I can go into the details and look what that streaming table means. Another option which we have, which I think can also help you visualize the power is something called as lineage. So if I go into lineage, I can see all the different assets which this table is being referred into, and those assets can be a different table, a different pipeline, even the notebooks. So all of those, if you see here, drop down all the different assets where this has been referred to will show up here.
And not only that, you can also see end-to-end lineage graph, right? So if I click on see lineage graph, and it gives you a nice visual of how the things are moving from source to target, including all, you know, what pipelines getting referred and what tables are getting modified as well. I can just click on top of this, right? It is telling me this is part of the pipeline here. And not only at table level, I can also go at column level to see how it is flowing.
So I can click on one specific column as an example. It can tell me bike ID. So how is bike ID getting referred? Whether it is getting modified? So you can see all this end-to-end lineage as part of the platform without you doing anything. The platform is intelligent enough to create that lineage for you based on the references it has in the queries. Another thing I would like to show you is basically jobs, right? What each I was mentioning is like automation of jobs as an example.
So, including your DBTs, right? So, you can create a specific jobs. Sorry. Just a second. It's a live demo. I'm brave enough to do a live demo. It may not. Let me, sorry. Go to the right location. Yeah. So, here. This is where basically if you have built a pipeline, you can have your scripts, jar files, wheel files. How do you automate or orchestrate that? That's where jobs comes into play. You can have a job of dbt type.
I can create another job, and you can see all the different types of jobs which you can do. Ingestion pipelines, ETLs. And example, if I go to dbt, which some of you might already be having, you can define that as one of the type. and then you can define the project directory where it resides and then you start ingesting and running those. Another thing I wanted to show you is basically coming to compute now. So we have ingestion, we have seen how you can orchestrate what the jobs looks like, what Unity Catalog is, what lineage is, what compute looks like.
So for Lakehouse, we have something called a SQL Warehouse. So you can create a SQL Warehouse from here and it comes in different options. You have a serverless options which we generally recommend to give you a lot of different benefits inbuilt. As an example, for classic, it may take four to five minutes for that warehouse to come up. With serverless, it is few seconds, which changes the game altogether. And we will have a new type here, which is Lakehouse RT, which was announced by Ali yesterday.

52:32 Serverless SQL warehouses start in seconds

So Lakehouse Real-Time or Lakehouse RT. It is in beta right now, so get yourself enrolled in the beta version, and once it goes in public preview or GA, you we'll also see that option here. So within that warehouse, again, as Isha was mentioning, you can directly integrate a lot of different visualization tools as part of the platform. If you see here, we have Tableau, Power BI, all those integration. As an example, if I click on Power BI, I can download the connection file directly from here and then connect to this lake house as a backend for your Power BI dashboards or Tableau dashboards.
Another option, again, is like, you know, I can also go to my catalog and a specific table, say Billings table, and then open it in the dashboard. The dashboard can be AI, BI dashboard, native dashboard, or it can be Tableau or Power BI dashboards as well. With that said, right, so we have ingestion, we have compute, we have seen what DBT is. We can also do federation. What federation does is basically you can connect to external data sources which are not in Databricks, say PostgreSQL database, right, or MySQL database outside.
Just create a connection as part of Unity Catalog and create a foreign catalog inside Databricks, inside Unity Catalog, and then that foreign catalog becomes your federation where you can start federating queries from your external sources as well. Let me go back to the catalog. and let me go to the SQL editor now. So with all this premise set, right, we can now start actually seeing how the SQL query works and what SQL query editor provides you, right?
So new SQL editor, of course. So as an example, what I can do, and we were seeing all the different AI functions, right? So I will show some demos with respect to text data as well as with respect to image data. So suppose, you know, if this is a sentence, I'm not sure if you can see what the mistake is already. If not, let me run and see whether it can help me. This AIFix grammar is a function, inbuilt function. So, this sentence have some mistake.
Let's see if my English is correct or how it fixes it. And I'm just passing a text here, but you can also actually pass a column of a table. So, it does not have some mistake. It should be has some mistake. So, thank you for correcting my English. AI, and then suppose if I want to classify something, right? Suppose if a password is lead, if that is part of some of the log file as an example, right? I don't want to analyze it.
So this AI function can actually classify whether it is an urgent matter or not, and then you can have alerts based on that, right? So AI classify whether it is an urgent or not urgent. It can tell you that. Urgent, definitely, it has to be. AI translate, right? So I can also translate, say, one of the column I mean, one of the languages you want an output in a different language, and then send it based on the different regions you serve in.
So suppose in this case, I want to translate this amazing function into French. So once I run this, so it has translated into... Sorry, I cannot read French. I'm not sure if anybody here would volunteer to read it. Is this correct? I see head nods over there, so it's correct. Okay, thank you. Yeah. So this is like the basic function examples, but how you can actually start asking questions to a specific model which you want to use.
As an example, GPT, or which are outside of Databricks, or some custom model which you have built. So that's where we have something called as AI query. So in AI query, you can send the model name, what you want to use, you can send the prompt which you want to use, and you can also send the response format in which you want to access the data. So I've already created this function using GPT-5 as an example, which is ask LLM model GPT.
And now I want to start asking to that. So if I ask that model to generate a short product review for a blue dress. So luckily I have a blue coat and she's also in blue. So I just selected blue. I had selected blue before, but it turns out we are in blue anyway. So it turns out good. So it has automatically generated a fake review for us, right, based on customer is very happy. So it has generated, like, I'm very thrilled with new blue dress.
The moment I put on it, I knew yada, yada, yada, right? My world changed, as an example. So, yeah. So this is one example, like, of how the power of LLM can be, right, where you can generate reviews. It reviews, one example, you can generate text out of any of the LLM models that you want to and store that as part of the columns. You can also basically analyze the reviews which you have got from customer and then also generate a response which you want to send them, right?
All through Lakehouse. You don't have to worry about any other tooling or modeling you have to use. So here I'm asking what kind of review this is, right? So I've sent a certain review and I'm telling whether it is a good review, bad review, what are they discussing here? So it runs and then it says some of the reviews which you did, like, I loved Kellogg's Corn Flakes, then Sleek Coffee Mate, Creamer is awful. I'm not sure if someone relates to it.
But then now I want to generate a response, right? So for some of the reviews, what the response should I generate? So here is a, you know, response which is generated for you, right, which you can send it as an email to the customer for positive or negative or in some social media website, if you have to reply back, right, you can do all of those. So I have already generated some of the responses for the reviews which we have seen, right?
So if you see here, this was my input table, right, where I had all these reviews from the customer, customer name, et cetera. Then I can understand, you know, what the follow-up we need to do from the customer and what a draft response should look like, right? All of those, again, you don't need to have a specialized AI or ML engineer to do all of this, right? Our regular data engineers, you can do it, right? And coming back to other powerful multimodal capabilities which we have, right?
That is where, again, AI query shines, where suppose you have some models which are very powerful in images and not text, right? You can use those image models and then read data from not your structured data or structured column, but read data from the files in a volume, right? It can be in the object store and again, right, volumes is our way of connecting to an object store and getting data in. So for unstructured files or you know, JPEG or PDFs or text, right?
So here, I have a model Lama for Maverick, which was supposed to be good a few days back. It might have changed now already. I don't know. And what I can do is I can start, you know, analyzing Let's see what the image looks like first. So if I go to my catalog, queries schema, I have this volume, I have kind of images. So I have some machine generated images and I also have some like real images. So let me show some, like I have some cows.
I'm just picking some random, right? And then here, just a big field. And I also have some kids playing, some studying. Very good for you. Yeah, this is not a kid here. So again, this is again very hard to find, cow, right? So I have all different kind of images I have put in for our experiment. And I have another set of data as well, wherein images are generated by machine, right? So if you see here, this still looks like a cow in a farm and cows grazing, right?
And then let's see what kids are doing here and what kids look like, actually. So this is AI which generated all the images, right? So kids playing with balloon. So I have two different type of images, one real, one generated by machine, and let's see how this function performs, right? So for real images, let's go back. What I'm trying to tell here is I'm giving an input to AI query. I'm giving it a prompt, like describe this image in 10 words or less, right?
And I'm giving it a model. This is the model you have to use, and this is the path where my files are. And the files can be any binary file of any of these specific types. And when I run this... I hope... It runs and not fails in the live demo. Fingers crossed. Okay. Oh, it ran. Yes, it completed. Right? So it did all the analysis. And now it is telling me the result, right? Young girl playing with hay in a festive setting, as an example.
Kid number nine. Let's see what it looks like. So if I go my real images. kid9. Oh, yeah, a girl playing with the hay, right? Very good description. You know, suppose I want, I also have some image as an example. Let's see kid14. I can't even see him. Next. Kid21. Okay, a lot of kids sitting around, right? Or kid, I'm just trying to figure out a good image. Kid23, right? So let's see what, you know, AI query generated for this image, right?
So I can do, again, I don't have to filter out based on query, I can just do a natural language here. So I can say path with file kids underscore 23. jpg, right? So it will filter this out for you. So again, filter in natural language. You can also generate queries in the natural language. Sorry, wrong prompt. Having kids 23G. Right, so it found that and it says children building and with wooden block together, right?
That's exactly what it was. with wooden blocks, they're building things. So pretty good, right? And it does similar analysis on your data, which is generated by a machine as well, even though it is not as clear, but it understands it is still a cow. So if I run the same query on the images which are generated by machine. Take a few seconds. Sorry, it has a filter here. There is no kit 23 file there, yeah. So once I remove that filter, it is saying cows in the field, right?
Cows in the field with floating balloon. So for having fun 02, let's check that image out. So machine generated image, cows grazing, cows having fun 02. Right, so cows are playing with balloon, yeah. So this did a pretty good job of analyzing the image and also giving you an output, understanding the context and then giving you what the text can be. So it can help you segregate, right? If you have a lot of images coming out of your sensor data as an example, right?
A lot of images coming out of your machines, you can find the defect, all again, everything getting automated. And last, after you have done the query, right? After you have done all the hard work, you have ingested data, you have ran your query, you want to send the report to end users, right? You can do that in two ways now. One is you can start asking a natural language question to your data, or you can do the visualization to the data, right?
So we have something called as GiniSpace. Let me go to one of the GiniSpaces for supply chain optimization example, right? And I can start asking question, describe the important KPIs for this data set, right? And then there are two modes it works into. One is an agent mode, and one is a chat mode. Chat mode is, like if I skip to the faster answer, that's where actually it will give you the chat mode. It's still thinking, right?
And you can look like, you know, what it is thinking into what tables it is referring to. And then in an agent mode, it actually does a dive deep and tries to understand, you know, why it happened and gives you a good briefing of, you know, what the output should look like. While it is doing that, let me open the dashboard in parallel to show you what the visualization looks like, right? So this is a visualization layer wherein, you know, I can start looking at the chart, again, based on the data which we have already built, right?
And as an example, I can start asking questions in natural, like, explain this chart. What does this chart mean? Right? So, and we can publish this dashboard to the end users. You know, you can have different tabs added to this dashboard as well as part of the edit mode. So, if you see here, it already analyzed my data, right? It tells me what the different KPIs I should be looking into, what the data structure is, what the raw material supply chain looks like.
That is an important KPI for me to look into. So it goes pretty deep. So having this natural language capability, you may not even have to build dashboards. We also have customers where we have seen proliferation of the dashboards, where you have thousands and thousands of dashboards, Tableau or Power BI dashboards, which are even becoming difficult to maintain. So with this capability, all users can do is start asking questions in natural language.
You don't have to even go and look into the fancy dashboards. We can reserve that for executives. But now when executives are getting fancier, so when they want natural language capability, all can be given to them through the single platform. With that said, I think we are at the last part of after the demo is like where to go next. That's the clicker. Thank you. So how do we now modernize your legacy warehouse into Lakehouse, right?
Given all the awesome features you have seen, I am sure you would want to modernize to the Lakehouse, right? So how we do that, right? The rocky road of migration. Given, like, all the assets which you have built already, which can be so much complex, the migration itself can be a big task or a big project, right? Some migration goes for months and even years, and, you know, you have to have a budget just for migration.

1:08:12 Lakebridge: migrating legacy warehouses with agentic code conversion

How do we solve that problem? We have something called as Lake Bridge. This was announced last year, but now we have open sourced it, and then we have also done a lot of optimization and innovation on top of it. Now it is AI-powered. And the two words to take here is basically free and AI-powered. So we are giving this to you so you don't have to invest in anything else and then start getting your migration going. right?
So what the path looks like for LakeBridge. What it will do, it will do the analysis, it will do a deep run, deep scan on your existing assets. It will tell you what the scope of assessment looks like, right? You know how you can convert your PLSQL or BTEQ or TSQL queries. Then there's another step which is called a code converter, wherein based on the analysis done, you know, you can do the conversion of those codes, then do the migration.
So once the code is done, how you can start running those code and do the migration of the data. And the final one is the data validation, right? So there has to be someone, when the migration completes, does the validation, right? There is a reconciliation which needs to be done as well. And that's where trust gets built, right? So first three may be easier, but the fourth part is the major part. And what additional LakeBridge has to offer is this agentic code conversion, right?
So for all of these different sources and more sources will get added in the future as well. You start converting your code iteratively. It fixes error, then converts the code again, all backed by automation and AI, and then you will have the converted code. So basically, you can bring all the assets in a specific folder, ask it to convert each of those. You don't have to go one by one, but you can do it at folder level as well, or directory level.
So basically, that 20-year-old SQL, right, which someone asked your data engineer to update, and they left the company fearing, you know, what can happen, and then they changed their phone number as well, they don't have to worry about it, right? Automation and AI will take care of it now for you. and the formula for success which we have seen as part of our experience in the field, right, is basically following this pattern for the migration is aligning with the partner, having a partner, again, is not mandatory to have, but it is recommended, so wherein they have done this migration already for multiple different customers and they can bring all their learning and tooling to you as well right, you have to also define your payback is like, you know, if I'm investing so much time and so much money in migration what the payback looks like, what the timeline looks like, and also the scope, right, and tooling, what kind of scope you are, again, you have to not boil the ocean, but take things slowly, right, take one specific project at a time, see the success out of it, show the success to the business users or end users, and then take it from there, right?
So that is the key to successful migrations. And again, right, these are the examples of the customers who have done successful migration using our tooling and the recommended path, Walgreens, HP, Alliance, so millions of dollars saved, performance improvements, reduced cost, and those are the top three things which every customer wants to do. And we are giving them tools to do that themselves or engage a partner to do that as well.
With that said, we are at the end of our presentation where we have covered all the three different pillars, and I'm sure by now you understand how we have removed that wall of data warehousing and Lakehouse and AI tools into one platform. Yeah, thank you so much for bearing with us, and hope it was helpful.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.