Summit Live: Best Practices for Data Warehouse Migrations

Summary

• Fastest Growing DW Platform: Databricks SQL serves over 10,000 organizations as the fastest-growing data warehouse solution.
• Migration Support Tools: Lakebridge migration tool and partner ecosystems help enterprises move legacy data warehouses to Databricks.
• Enterprise Validation: Amadeus and other enterprises are successfully migrating from legacy systems like Oracle to Databricks SQL.

Summit Live: Best Practices for Data Warehouse Migrations

Databricks SQL is the fastest-growing data warehouse on the market, with over 10k organizations thanks to its price performance and AI innovations. See the best practices and common architectural challenges of migrating your legacy DW, including reference architectures. Learn how to easily migrate per the recently acquired the Lakebridge migration tool, and through our partners.

Chapters

• 0:00 Intro
• 1:35 Introducing Databrick
• 2:59 Introducing Chris Ernst
• 4:04 Databricks Architecture
• 10:13 Migration Methodology
• 11:25 Recap

FAQs

How many organizations use Databricks SQL?

Databricks SQL serves over 10,000 organizations and is the fastest-growing data warehouse solution on the market, chosen for its price/performance and AI innovations.

What migration tools does Databricks provide?

Databricks recently acquired the Lakebridge migration tool and works with partners to provide comprehensive migration support. Reference architectures and best practices are available to guide enterprises through the process.

What are best practices for data warehouse migrations?

Follow a phased approach starting with less-critical workloads, validate data quality at each stage, automate testing procedures, and maintain parallel systems during transition. This video demonstrates proven migration patterns on the Databricks Data and AI platform that reduce risk and enable organizations to complete complex warehouse transformations.

What data warehouse migration solutions exist?

The Databricks Data and AI platform provides Lakebridge for migration automation, pre-built connectors for multiple source systems, and migration accelerators developed through experience with thousands of organizations. These tools handle schema conversion, performance optimization, and data validation to streamline transitions from legacy warehouses.

What is a data warehouse?

A data warehouse is a centralized repository designed for analytical queries and business intelligence on structured, historical data. Modern data warehouses like those built on the Databricks Data and AI platform combine warehouse performance with lake flexibility for analytics and AI workloads.

Full transcript

0:00 Intro

Hey, Ari Kaplan, head of evangelism at Data Bricks, and we are back live at Summit Live here at the Moscone Center in San Francisco. Love seeing everyone here at the virtual summit all over the world. I hear 140 countries. It's now over 60,000 people uh have registered for the virtual conference. So, thank you from all over the world. And I am here with Lauran and speaking from all over the world from uh France. Yeah.
And France. Yeah, indeed. And we we're great to have a customer here as well. Yeah. Thanks. Glad to be here. Yeah. So, we are going to be talking about everything data brick SQL. A lot a lot of people know about it and have been using it since of the performance and cost and serverless, but also because of the intelligent data warehouse aspect of it. But I travel around the world and people like I kind of heard about it but never don't really know much about it and some people like oh data bricks has a DBS SQL but it actually has been our fastest growing product uh out there.
It's been remarkable because of all of those reasons. So with that I wanted uh to have you two introduce yourselves. Yeah thank you. So uh as you mentioned I'm based out of France uh working as a data warehouse migration specialist uh and covering EMA for this uh questions and uh yeah for what you you mentioned data brick SQL is really the fastest growing uh solution we have um in terms of data warehouse um and if we look back uh imagine uh data brick circuit has been g in 2021 on and now it's really mature to support all of these uh data warehouse workloads uh supporting SQL and driving performance at the very uh best cost performance ratios that we could have and uh yeah the with lots of innovation in this uh product uh we have of course this serverless uh infrastructure we have uh now new features coming from the AI world to support and to connect the descriptive and the productive analytics worlds.

1:35 Introducing Databrick

So very good solution u that have a a bright future uh in the data warehouse uh uh ecosm and today I'm I'm very happy to to welcome because uh a couple of uh years or months ago we started a an an Oracle ex data migration correct uh uh project. So maybe you can quickly introduce yourself and then we will dig into the yeah the details. So I'm I'm a data architect with uh and uh is a travel IT organization. We've got uh dozens of products that travel or that follow uh a guest through hotel stays or through bookings etc.

2:59 Introducing Chris Ernst

And also for uh travelers and air and uh with airlines either with um reservations or bookings uh costs sorry uh yeah uh searches etc. Right. Um and uh we've got several different commercial products that are analytics driven and uh one of the data warehouses that we uh that we support is uh going through the migration as as mentioned earlier and uh so this is kind of the uh we're we're we're finally starting to see some of the some of the improvements and some of the some of the results from the testing and things like that we've been doing with our migration.
So it's really exciting. So maybe you can uh have a look at a quick slides uh just to show you to the to the to the people uh what was the architecture and how you drive this migration. Yeah, absolutely. Okay. So yeah. So the so Oh yeah, sorry. So uh on the on the uh the top here we've got our our uh very simplified view of our old architecture um where uh the data flows from right to left where we've got uh files and streaming sources that go through some preparation and separate compute and then through our data warehouse uh to a uh a a platinum level basically with our with our data set uh that sits in Druid that has uh extremely rapid query um query results that we're that we're looking for to drive our UI.

4:04 Databricks Architecture

We have a custom UI rather than using something uh commercial off the product or sorry, commercial off the shelf. And really the intent here is just to emphasize the fact that this is um we we don't have anything that's super bespoke that's super, you know, outlandish versus anything that uh that might be present in other products or in even uh uh relatively complex analytic spaces otherwise. Right? Whenever you break it down, these are the core pieces.
Um what we were excited about is uh not just with data bricks SQL but with other uh pieces and features within data bricks we were able to consolidate a lot of our resources so that instead of needing to do as much preparation before our exit data tile um we we could uh use data bricks to do uh our additional prep and then move it through our data warehouse and then use DBSQL to do the serving layer with our custom applications.
Okay. And so regarding the migration that you already uh did or processed uh what are the level of performance did you get in terms of if you compare to what you had in Oracle? Yeah. Um so it was actually pretty great. So the this chart uh emphasizes the uh the differences in performance that we saw in some of our query times. So this is a really extreme example because we've got uh a a pretty large set of records that we were trying to find where we're looking at uh 25 trillion records uh for about 6 months or so of search.
Right? So these searches these are uh coming from the Amadeas GDS. So if a travel agent or an OTAA like uh Expedia or even Google flights um if you're if you're searching for a flight from a city to a city for a given set of uh uh dates and a number of passengers all those sorts of things each of those searches would be a record with this data set and uh the uh the queries that we were looking for came from our custom application uh the the UI the query that goes that goes against uh to drive our dashboards.
So, we had uh from our result set, we took uh a six-month range of uh search results for um two specific uh destinations from any city to two specific destinations for two specific travel dates. And um the chart that you're seeing here really emphasizes the uh the mostly the timeout that we set for the for the for the testing, but it it also shows um how how much better performance we were seeing with data brick SQL versus uh some of our other spaces.
In this test, we actually loaded the data into four different technologies. We've got the data in a um uh we we had the data the data in our Oracle exit data space. We also loaded it into Druid alongside a bunch of our other uh data that's serving our applications. And then uh we've got a uh another warehouse that's in Redshift that uh has some uh custom software on top of it. And we also uh check the performance against that space.
And um we in in data brick SQL it's important to point out that we had uh uh liquid clustering enabled on this table and there wasn't really a lot of other prep that we did otherwise. it was uh load the load the data a little bit of configuration and move on. Um versus with uh you know with Oracle Exodata we after this test we did a little bit more uh tuning and things like that to find to see if we could improve the performance just a little bit to not hit our timeout right um but uh the performance here is pretty pretty spot on with what we saw.
Yeah, it's really impressive the performance gains that you had for sure and did you have any reason about of course data brink secret is very performance but the difference is really huge so yeah so we're we're able to separate the compute and the storage of course with uh with the lakehouse and also our our Oracle dags data space is never as quiet as a single data bricks warehouse cluster would be and so um all the time where we're doing these tests we're also loading our our data through our through our standard loading processes and we just have the one set of lanes that go in and out through uh through our MPP space, right?
So the fact that we can um spin up additional compute to do things like um uh the separate the ingestion from the streaming processes and from the files and things like that completely independent from what's being queried from our applications. That's really the uh the emphasis here. Yeah, that just looking at this that's super impressive. Fun fact, I used to be president of the worldwide Oracle user group. Uh Thomas Curran, uh Ellison here in Moscone, but Exodata back in the day like 20 years ago was like the fastest thing on the market.
So this is incredible and I love your point like the serverless I know a lot of people like just to provision a new uh you know data warehouse takes a lot of time often and when I do demos like I actually do my own demos it's all live and so on sometimes I give a demo and you know like the I forgot to start up a a server you know I did it like a week ago and it just starts up in like it seems like 3 to 5 seconds for me um and that is just incredible.
You just it works is a phrase that I hear, right? And something that is also very important is the to have a a clear migration methodology to to migrate. And so at data bricks, we are usually uh using two different uh methodologies, the injust first and the BI first. Uh what is the the one that you use for your migration? Yeah, we're we're following the ingest first, right? So we we uh we want to make sure that we have uh uh enough time with our migration to uh to adjust all of the ETL processes because we're changing technologies and languages and all of those sorts of things.

10:13 Migration Methodology

Um and our our serving layer is pretty consistent. And so for us it's uh it's it's the bigger work is to get the data moved. So we started there. Okay. So very very interesting uh to know that our customers are following the the methodologies that we uh we recommend. Yeah. Um so thank you. Thank you. Um I like to uh conclude this. Okay. Well, we're going to we're going to conclude but yeah it's so great. You're actually our first customer.
We have many more coming up and we have a great schedule coming on. Um wanted to thank you from Dallas and from France. Thank you. Thank you for coming on and sharing your story. Thank you. All right. Thanks. And now we're going to toss the keynote recap video. Super excited. 4 3 2 1 What impressed me the most was the agentic AI and the presentation that we had where we had a live demo of uh how you could build those AI agents all you know out of the box through a data bricks bringing this concept of OLTP to the lakehouse so that's probably I'm interested to see what we can do there.

11:25 Recap

He talked about leaders in organizations, how they should be open-minded and make sure that how you lead the people is always with keeping in mind that everyone is different, but you need to listen to them and also listen to your clients. Agent Bricks is very very cool. Yeah, that announcement with Neon is probably my favorite. Asian Bricks seems like it's going to be the new up and coming and I think it's going to be awesome.

Learn more about the Databricks Data and AI platform.