How to Migrate From Snowflake to Databricks SQL
Summary
- Snowflake to Databricks SQL migration: This video presents comprehensive migration methodology, technical approaches and automation tools for moving from Snowflake to the Databricks Data and AI platform.
- Code and data conversion strategies: Learn product and feature mapping, technical approaches for handling Snowflake-specific constructs and best practices from real-world customer migrations on the Databricks Data and AI platform.
- Migration planning and execution: Discover why careful planning is essential when migrating between cloud platforms and how to choose between ETL-first and BI-first migration strategies on the Databricks Data and AI platform.
How to Migrate From Snowflake to Databricks SQL
Talk by Koundinya Srinivasarao, DBSQL Adoption Lead, Databricks ; Matt Holzapfel, Sr Product Adoption Architect, Databricks
Migrating your Snowflake data warehouse to the Databricks Data and AI platform can accelerate your data modernization journey. Though a cloud platform-to-cloud platform migration should be relatively easy, the breadth of the Databricks Platform provides flexibility and hence requires careful planning and execution. In this video, we present the migration methodology, technical approaches, automation tools, product/feature mapping, a technical demo and best practices using real-world case studies for migrating data, ELT pipelines and warehouses from Snowflake to Databricks.
Here's more to explore:
The best data warehouse is a lakehouse: https://www.databricks.com/product/databricks-sql
How to migrate legacy data warehouses: https://www.databricks.com/resources/ebook/migrate-your-legacy-data-warehouse-databricks
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
Chapters
0:00Introduction and motivation3:05Why migrate to Databricks8:40Migration planning phases12:37Lakebridge migration tooling15:16Assessment and architecture21:01Migration strategies26:03Code and data conversion32:23Data validation techniques35:54Governance, performance, and wrap-up
FAQs
What is the best approach for migrating from Snowflake?
This video presents migration methodologies including ETL-first versus BI-first approaches depending on your business drivers. Learn how to choose the right strategy and leverage automation tools on the Databricks Data and AI platform for smooth migration.
How do I handle Snowflake-specific SQL differences?
This video covers product and feature mapping to help you convert Snowflake-specific SQL to Databricks SQL. You'll learn common conversion challenges and best practices for ensuring code compatibility on the Databricks Data and AI platform.
How do I validate data accuracy during migration?
Learn data validation techniques including reconciliation best practices to ensure data integrity throughout the migration process on the Databricks Data and AI platform. The video includes real-world case studies demonstrating validation strategies.
What benefits will I gain from migration?
This video showcases benefits including improved scalability, real-time analytics capabilities, unified data and AI platform and cost optimization. Customer case studies demonstrate the competitive advantages of migrating to the Databricks Data and AI platform.
Full transcript
0:00 Introduction and motivation
Okay, we're going to get started here. We have KD and who are going to talk us through how to migrate from Snowflake to data brick SQL. Please don't forget at the end of the session to fill out a survey that helps us inform us as to which sessions you want to hear more of and more about. And without further ado, I present my friends and KD. Thanks, Licker. Uh as Siko introduced us uh I'm uh Cinia people call me KD and I'm apple so we will be talking today about uh how you migrate of uh snowflake and when we decided to schedule this call we chose hey this is the time and we just wanted to make sure the people who are here really really do want to migrate off snowflake I know you're you know keeping you all away from your happy hour and if I had my way I would have had you know brought the bar here.
Uh but here we are. Uh some usual sort of um standard operating procedure for us. Some of the information we'll share today it's forward-looking road map. Uh so this is the standard disclaimer around that and as called out we'd appreciate any feedback on the session the content uh what we had to share here as well. We both are part of the product adoption group. I specifically focus on data bricks SQL and a big part of um my job is around making sure customers as they adopt data brick SQL whatever uh their friction their experiences are optimizing that and migrations happen to be a big source of how um you know customers get uh onboarded and adopt our data bricks you want to introduce yeah and I specifically focus on helping our customers who are looking to interoperate with snowflake or migrate workloads from Snowflake over to data bricks.
So between the two of us, we see a lot of customers kind of going through this this journey and working with them at different parts uh of it. We'll very briefly touch on why uh customers are looking to migrate off uh Snowflake and the this is plays pretty significantly into your strategy around what approach you take for migrating. uh for example when we get into strategy one of the things will talk about is this ETL first versus BI first kind of an approach and that knowing that why grounding your strategy and why you're actually moving is super important uh we'll also go through the approach itself how do you incorporate these different processes tooling uh the methodology quite often when it comes to migrations people think of conversion as code conversion as being the most critical piece of the migration it is to some extent but there are more important equally important you know phases of the migration around data validation and we will talk about approaches related to that too all right so um to start let's let's talk about why why why migrate um and before we we get into that I'm interested to understand where you all are at in in your journey and so how many people by a show of hands are actively migrating from Snowflake to data bricks.
3:05 Why migrate to Databricks
You're in you're in flight right now. Okay. How many are convinced you are going to migrate and so you want to get some ideas and best practices for how to do it. All right. And then last group on the fence, maybe migrate. You want to see what what it's like and then you'll decide from there. All right. Great. Great. Great. So, we I mean we have a little bit of everything and we're we're going to touch on I mean really kind of the full endto-end journey and so try to hit on as as much of that as as possible.
But for the folks who are in the middle of the migration um please share your experiences with your fellow uh uh kind of data bricks community members because that's um how I think you know really these these go well is just by understanding and hearing a lot of perspectives. Um and so why migrate um off of uh Snowflake and um on to data bricks? It it's really three three reasons. Cost, governance, a IML. Um and so cost kind of in in particular what a lot of customers see on Snowflake is as the data volume scales up, as the scale of their ETL jobs scales up, the cost starts to increase exponentially.
People love very predictable, low cost on on ETL and that's one of the real advantages and strengths of data bricks. And so that is probably the most common reason that people migrate from Snowflake onto to data bricks. Um but some of the other reasons um that we see kind of in particular for customers who've migrated one or two workloads over from Snowflake to data bicks seen some cost savings they'll decide to make data bricks the whole platform um uh partially because um the governance suite is is so strong with Unity catalog where you're able to have this unified governance layer across all formats all all use cases and so that's another big reason um for people to kind of shift more to uh to data bricks and then finally, um, for those who were at the keynote this this morning, you saw all the awesome stuff we rolled out, um, around Gen AI, around MLflow 3, um, we continue to invest very heavily in in AI ML.
We want to be able to serve both people who are looking for very low code um, type experiences and then people who want a lot of configuration in that full MLOps tool chain. Um, so that's the other reason for migrating. Um, and so that's really kind of tied to the key benefits of migrating is just the inverse. um of those challenges is on data bricks you see lower TCO better governance and then you're able to drive that innovation um agenda.
Um one of the things that we're super proud of that was just released um last week was that we were ranked number one in both vision and execution on Gartner's um 2025 data science and um machine learning magic quadrant. um first time we were ranked number one on both dimensions over the hyperscalers um and others with within the within the space and I think you know the kind of TCO story um is is well well known and for people who are just now getting started in a IML or kind of starting to embrace it more and more um I think you'll see a lot of um a lot of stuff there as you saw um earlier um today um and so just to uh kind of summarize um you know why customers migrate And the key benefits I think this this quote ties it up nicely which you know the headlines we can now get the data quickly and easily.
Um I think one of the key use cases that customers will migrate from Snowflake to um data bricks is often when it's uh there's a streaming use case and there's very high um and kind of diverse um data that needs to be ingested. Um, and so the platform is really really strong um at being able to give you more of that kind of real-time experience that um a lot of your internal stakeholders are be are starting to expect and some of our customers like like have have really seen the benefits after doing the migration.
Um and so finally for why migrating it's never been easier. Um, and so one of the great advantages of first um, you know, what um, uh, kind of we we talked about this this morning of everyone's embracing Lakehouse. Everyone's starting to embrace open formats. Everyone here gets to benefit from that. All of us in the community get to benefit because now that makes it easier for you to interoperate between platforms. a lot of the things that Snowflake um released last last week around interoperability just kind of further advanced this this story making it easier for you to keep data in an open format interoperate between the two platforms and then migrate the workloads where um you see kind of strong advantages from shifting those over to to data bricks.
Um and then finally um this is just a a triedand-rue motion. We saw how many people in the room are going through a migration. there are even more people in the other rooms and at the happy hour who've completed a migration. And so strongly encourage um everyone to work with your your account teams, work with the folks at data bricks to help find other customers um who who've gone through the migration if you're looking for more stories and more um insights because um this is has become a very kind of common practice for us.
Okay. Um there's one important um aspect of um understanding why you migrate right the this is something as talks about the approaches he'll he'll get he'll tie it back to this why right that why is going to influence are you going to move the serving layer first are you going to move the ETL layer first and that is actually going to have a very um major impact in you know how you actually start to see the value now before we get far ahead I'm going to spend the next maybe 10- 15 minutes uh talking about how some of our customers how we recommend they start approaching migrations right uh so there is a tried and tested uh approach that we recommend from our point of view migration tooling is often where a lot of the a lot of practitioners say hey think that is where they need to focus but it comes back to standard practices around hey how you approach these migrations right we have identified by five different phases.
8:40 Migration planning phases
Number one, what where you actually understand and assess your uh your landscape, your estate that you're going to move. Uh very often when you look at say a data warehouse instance, you're just looking at some aggregate number. Hey, I have so many terabytes. I have so many tables. I have so many databases that I need to move. But you don't actually dive deeper into say some lineage based tooling to understand hey do I need to actually move all this or should I prioritize certain portions of my estate right?
So that is where assessment is super critical from once you've assessed your current sort of footprint the current architecture you then start getting into defining the target state. and defining the target state again there's a couple of things that you need to uh make a decision around uh hey do I actually try to modernize very often as customers adopted Snowflake they more than likely transitioned migrated from some other legacy data warehouse and as they were making that sort of uh change you probably carried on some of the tech deed from that legacy warehouse too do I actually pause take a break and say hey this is my opportunity to modernize and retire I remove some of that tech date or no I my business needs are such that I do need to continue to carry of that some of that over uh into the next stage and then eventually somewhere down the line I'm going to modernize.
So that's where you're defining the target platform and hey do I modernize do I do something closer to lift and shift kind of you land in this middle space which is what I would call as minimum viable sort of replatforming that is a delicate balance that you need to figure out right around that stage. Uh number three, you get into this. Hey, do I take care of moving the injection ETL first or do I do BI first? Right?
There are a few considerations will go over about that. The area which u number four is what most customers, most practitioners tend to pay less attention to is the validation, right? Hey, can we actually rather than going with this big bang approach, it is quite beneficial to do like an MVP kind of a horizontal tracer bullet where you identify three or four different patterns. Take a few pipelines for each one of those patterns and then actually get that migrated end to end.
That is a approach that you can helps you figure out hey what are the what are the landmines that I need to navigate before you actually go for a full scale migration. And number five, this is where you actually start getting into implementation, right? This is where your tooling uh conversion tooling comes into picture. This is where um quite often the data governance and modeling is uh something you customers tend to approach last.
Starting early with actually looking at governance, performance, monitoring is super critical. Now migrations is it's not all just process. there are some set of hard problems to solve and this is where tomorrow you'll hear about this uh in the keynote and this is just a pre-announcement so to speak uh this is where we are introducing lakebridge uh lake bridge is a conglomerate it's a collection of different migration tooling that uh some of it is homegrown some of it comes from an acquisition we did earlier in the year with blade bridge and the different phases of the migration it helps you actually understand when you're in that assessment phase.
12:37 Lakebridge migration tooling
This is where we have the profiler that actually can look at not just your code but also the deployment of your snowflake instances, all your different virtual warehouses and help understand hey what is the level of utilization and usage and then tie that together with the code that our analyzer runs and scans and understands what's the complexity. So you it kind of combines the usage and the complexity of the code that you're trying to migrate.
The next much much later in you know closer to your implementation stage this is where the code converter comes in. Uh the initial phase of uh our release does not yet include u a geni based solution but our goal is to basically leverage generative AI use large language models and you hear much more details about this uh definitely um you know check it out at the keynote tomorrow. Once you have converted your code and not just the code but also your DDL then you get to actually moving the data and lakeflow connect comes into picture here.
In certain cases you rely on lakeflow connect in other cases you rely on other uh native tooling or other tooling sometimes customers already have it right. The last part in my opinion one of the most important ones is the validation itself. Hey you have your source and target systems uh running in like production parallel. How do you know the actual transformation that's been done by either the LLM or our rules or uh AS based uh engines?
Are they accurate? Right? This is where the validation comes in. It'll help you compare this and it'll help you figure out what's the source of this disparity when you actually see a problem. And we support this for um many different sources. And this is historically something there's been a ton of experience and expertise and knowledge that we picked up from the blade bridge acquisition. So a lot of the legacy warehouses terod data Oracle exodata um and snowflake itself we have the ability to help migrate from.
Now, kind of fitting this back into the process, the five-step process that I was talking about, it starts with um like once you've done that assessment and that assessment is as I mentioned before, not just looking at your code, but also looking at utilization and usage information, you could have a 100 terabyte or a pabyte, you know, sized uh storage. But if your pipelines are basically populating this data uh tables which are not being utilized as much that's a you know that's an opportunity for you to do some rationalization right and once you've gotten past that and you get to the architecture you now start to understand hey what are all the different dependencies it's not all just about moving the code but all the different tools that touch this hey you have more than likely you're probably using dbt you're probably using airflow for orchestration what what do you actually need to do to basically repoint those?
15:16 Assessment and architecture
That's an important um point of assessment. There are plenty of tools available for getting this u sort of assessment understanding your current state. Now we've taken a lot of that baked it into lake bridge. The lake bridge not only includes a profiler but an analyzer as well. So it can tie this u sort of that utilization information as well as your um your code estate and DDL estate right and generally when we see customers approaching migrations they are typically thinking about everything that is the second uh you know what's in the funnel the business case is not something they're thinking about right one of the that's that's where my call to action would be to start thinking in terms of business case identifying hey what for a particular domain for a particular business case what are the databases users the objects and what are the um different sort of uh interactions that I need to actually take care of rather than going at I'm going to go take on my entire snowflake estate break it down into business cases and you migrate business case by business case uh based on priorities uh the profiler itself actually helps go gather a lot of that metadata Right.
So going back one whether it's the DDL whether it's the actual all the different users the roles their access control we have the ability to go pull all that using using this automated tooling. Right? Now, this also helps you kind of understand, hey, what is your footprint on data bricks going to actually look like, right? Hey, if this is your current utilization in um on Snowflake, you can actually start modeling, hey, what my utilization is going to look like and what my cost is going to look like on data bricks.
Once you have assessed that, you now have to start getting into the target state. Uh this is where understanding your data architecture, your data model, starting to map that to um data bricks which really should be uh pretty much uh carrying it forward as it is. You just have to take care of certain best practices around data modeling which I'll touch on in just a moment. Now one thing to remember here is one sizefits-all doesn't generally apply.
There's always nuances. there's always uh some bit in your architecture or maybe there's some type of an integration that you kind of have to uh look at differently. So this is an opportunity to record that document that and again if this varies by business use case this is the opportunity to define that. uh I mentioned this earlier right as part of that defining the architecture uh you should be looking at your data modeling early on right so a lot of uh so generally if you're coming from um the data warehousing sort of word all the you have multiple data modeling techniques you have data vault you have star schema snowflake schema and then you often hear in the context of the lake lake house the medallion architecture how do this kind of u melt together this visual kind of neatly puts them into uh sort of this bronze, silver, gold category.
So your on the left you have your bronze and silver. This is typically your landing and injection and on the right you have your presentation layer you know all the various data marks your that basically go end up in the gold uh layer and everything in between which is the integration you know whatever you want to call it basically lands in your silver. Now you can basically take that same data model and carry it over here. all the there are a whole bunch of best practices around hey the primary key foreign key definitions you need to bring those as well as part of your DDL so that unity catalog understands those relationships when unity catalog understands those relationships and you go to data brick SQL and make run a query we have the ability to actually use hints and then um optimize those uh those queries and those joints uh the phase decommissioning is something u that is critical I mentioned this briefly, right?
Everyone thinks of, hey, I'm just going to take this big behemoth and I'm going to move it over. Instead, it's very um pragmatic and this is also where you start to see your businesses start to see value sooner. You take this approach where you figure out, hey, all these different business cases I have, what is their priority, right? And then sequence your migration based on that priority. This helps you basically maximize your ROI. you see value, your business sees value sooner and minimize risk by taking on this more important use cases.
Uh first, all right. Um and in terms of methodologies and strategies, um there are really two primary ones that we recommend. ETL first, which I would say is the most common, um but not necessarily always the best fit. Um, this is great if TCO is the primary driver of your migration. You're you're looking to get down the total cost of ownership of ETL as quickly as possible. So, you start there, you get that relief um and then um you're able to ultimately kind of in the end push to the serving layer.
21:01 Migration strategies
Alternatively, you can work backwards. And so a new pattern that we're starting to see a bit a bit more and also advocate for um a bit more is around um BI first. Another way to think of this is gold layer first. I want to shift my gold layer from my legacy to data bricks right away so that I can start serving that to end users. the business can start benefiting from um data bricks as as quickly as as possible and specifically in the context of advanced analytic and AI use cases.
And so which strategy is is right for you depends on what your your driver is, but um certainly both um worth considering um as you go through the migration. Um we'll we'll start with ingestion and ETL first. This is perhaps um the more um intuitive um of the two processes or the two strategies. Um and step one here is first shifting the pipes or really kind of adding the pipes so that you're also pushing the operational data stores into the bronze layer um within um data bricks or delta lake or iceberg.
Um, and then from there now we want to marry those with the snapshot um, of the silver layer from our legacy um, data warehouse. Specifically, we want to make sure we maintain that history. And so we push that over into the bronze layer with all of the kind of continuous feeds that are coming from our operational data sources. Um, marry that data to together within the silver layer and data bricks push that into our data marts in in the gold layer. we start to um have both the legacy data warehouse snowflake and um data bricks um serving and then this is where we're able to go through that validation um step that um KD highlighted previously um that's so important we've set up the full endto-end pipeline we could validate that what's coming out the other the other end um works we get the performance that we're expecting users are accepting it and then from there then we could do the cut over and shift to a full migration where now um our our pipeline in our warehouse um is purely um within within data bricks.
Alternatively, we can take a a BI first approach. So that first approach we're basically kind of building up step step by step focused on ETL first and then getting to serving. Alternatively, if we want to give our end consumers access to that data within their BI tools, we want to start doing serving early. We want to um enable users to take advantage of MLflow 3 as as quickly as possible um and all of the other great um innovations that have been coming out.
Um what we could do is um we could push the gold layer um uh to um data bricks so that now your gold layer is uh in your serving layer is um in data bricks. People could start um using it um there. We're we're still running the pipeline um that we had um with uh with Snowflake um in parallel. um but we're incrementally now starting to move more and more of the source data um into data bricks. What's important um here is you know not only um are we taking what's in the serving layer in Snowflake making that available to data bricks but also now we're going to start joining in and augmenting with more sources.
And so this this next trunch of work from step two on is really about how do we start marrying the new sources and then that's what's going to drive ultimately um the cut over where kind of very uh similar to what we saw in the ETL um first migration. We're building out that pipe from um bronze to silver to gold, running the tune in parallel, doing our our validation. Once we are confident in what we're seeing from price performance as well as the actual data being validated, then we're able to do that cut over.
And so both kind of ultimately lead to the same destination. But here we started with gold layer, we're serving right away. ETL. We're getting to ETL cost savings more quickly. We're shifting that over more more quickly and then doing the cut over. Um, and now to dig into kind of some of those details, we'll look at the kind of code conversion piece on how we actually do that. Cool. A quick uh question, show of hands.
26:03 Code and data conversion
How many of you are evaluating Genie right now? Okay, cool. One person. Yeah. So the BI first approach is definitely one that I would consider right if you're looking to unlock Genie so that you don't have to wait for all the ETL pipelines set up the CDC and start seeing start evaluating Genie to see hey is this something that my business can get value right and that's definitely one of the cases where our customers are uh looking at the BI first kind of an approach right so once you have estab figured out which approach you're going to uh code conversion is uh hard is one of is the is the critical problem to solve for right so here we have um lake bridge lake bridge itself comes from kind of two places uh we used to have an internal uh internally homegrown tool called remorf we acquired blade bridge as well we kind of dropped both together uh remmorf kind of uh is where a lot of the snowflake intelligence kind of we carried over into lake bridge there's in fact a great blog by Door Dash.
Uh happy to distribute that link. So the with lake bridge we basically take your DDL right so you all before you actually move the code you have to move the uh the metadata about all these different tables the schemas etc right so lake bridge helps you take all this snowflake SQL and basically turn that into I wouldn't call it data brick SQL we basically convert it into an SQL right because datab bricks SQL adheres to the ANC uh standard so any sort of uh code any sort assets that you migrate over you and this is another open standard that we are basically embracing here to we this is where you get tons of automation.
Now if you do have a bunch of stored procedures that you have running on your uh snowflake that is something uh we are looking to add and incorporate into lake bridge itself. So your stored procedure itself can come over. Uh data bricks SQL now also supports uh stored procedures and multi-statement transactions. U very often in my in our experience the um a common area place where customers trip up is around this uh different data types. uh and we have uh a pretty uh strong practice of doing these migrations from within our professional services as well as our partner ecosystem where we have this well- definfined recipes on hey if you have these types of snowflake data types this is the what the conversion should look like and a lot of those will be basically translated u by converted by lake bridge once you have done your conversion of your DDLs as well as the code you then get into the data.
How do I actually move the data itself? Now, historically the third option here, right, the snowflake connector where we just go to the native source of the uh the native formats um that snowflake has the proprietary uh native format. We basically uh pull from there and then write it into the lakehouse. Right? So that's historically we've used a connector to do that. Now the downside of that is you are basically even just for moving the data there's double compute. you need to spin up this warehouse uh the snowflake virtual warehouse to basically pull that data and then write it into or at least offload it to a staging area and then you have to kind of pull it into data bricks.
The approach that we are gravitating more and more towards is the iceberg table approach. Uh so we are going to be announcing also support for managed iceberg. Uh so quite a few previews that I'm um so quite a few announcements I'm giving you a preview about here in this talk now with managed iceberg support within unity catalog the fact that snowflake also has a managed iceberg approach kind of actually helps us uh with migration too.
So from snowflake from the native table you basically offload the data into managed iceberg table and that means that is all data that's still within the same cloud region. So there really should be no cost for you to go from snowflake native to the managed iceberg right there no um egress cost at least that you're paying for and once that data is there in iceberg we now have the ability to read that from data bricks right so unity catalog can federate into snowflake and start reading that uh data now once you're able to read that data and start doing migrating some of your pipelines you uh you will probably likely not write back into the production table you'll probably have like a shadow table that's where you'll be doing the ETL and writing your data.
That's where you'll be doing the validation. And then eventually when you are ready to um actually cut over, we are looking to add the ability to take a snowflake managed iceberg table and flip it into a UC managed iceberg table. Right? So what that means is what used to be like a readonly source for us now also becomes writable for us. Right? So this is uh the approach that we expect increasingly our customers will take.
Now meanwhile there are cases where even as you're going through this approach hey even if you're not yet ready not at a place where you are uh ready to adopt uh iceberg right federation is quite handy but generally federation lakehouse federation this is where you take a snowflake catalog bring it into um data bricks as a as a foreign catalog and here the format is still native format that I'm talking about and then you can start doing some exploratory analysis is this is helpful for ad hoc exploration but generally when it comes to data movement depending on the data size you can pull it off but the as you start getting into larger data volume this is not something we would recommend there are uh you know different considerations right cost is one of the consideration you have this compute performance is another consideration so generally more and more we believe the iceberg um sort of pattern is what our customers would take and just on the iceberg piece the as he alluded to, we're adding more capabilities tomorrow that we'll be announcing.
32:23 Data validation techniques
Snowflakes released a lot of new capabilities. It can be hard to stay up to date on all of those changes. And so if you're implementing the migration yourself um or your your team is and you have questions about how do the different cataloges work together, how does UC work with Horizon catalog and you want guidance and support, please reach out. like we we understand that this is a very fluid um motion and it's a very kind of new uh capability that's being brought um to market and so um you know please don't hesitate to uh to ask for help and support if you have any questions on how that interoperability pattern should should work.
Yeah and uh we'll make sure we have some sufficient time for Q&A. We our MC himself is actually a iceberg expert u so please do uh reach out to us about that. So as I mentioned previously the validation is one of the most critical parts of a migration journey that you are going to be on and this is also the uh phase of the journey that most people underestimate the level of effort that goes into uh and very often the there going to be this minute differences that you will see.
Hey there's some due to some data type or precision mismatch some something is not working out or ordering of the data is off or case is off or collation is u not enabled that's the reason these differences are coming about uh these are the kinds of issues the data validation tool so within lake bridge one of the components is the recon reconciler this helps you figure out hey what is the diff between my source and uh my um and my target.
Uh it uses different techniques such as hashing and uses aggregates. First compass the aggregates before it breaks down to the more granular sort of a view, right? I mentioned this earlier, right? So governance and uh data modeling and security is quite often thought of lost, but this should be uh something you take on early on in your migration planning. Right? Now by governance, governance itself is a heavily loaded term.
It you know it could be interpreted many different ways. Starts with hey what is my catalog design going to be? Um what is it going to look like? Largely it should be the same as it is in snowflake but it also this migration is also an opportunity for you to do some cleanup and do some restructuring if needed. Right? Uh access control is another important aspect. Uh very often you are coming from a regulated industry kind of a setup. you have uh very fine grain access control, row based access, column based access, uh all those policies you need to bring forward and attribute- based access control is uh also in public preview.
There are new capabilities that we have introduced there. You know uh the this policy based sort of access control should actually help simplify a lot of that. And lastly, this is uh something that is usually turned on last, you know, the very last phase of a migration project, but something you should start early, which is around PHOPS and budgeting and tagging, setting up the right tagging policies, setting up the right budget policies by the different business units.
So if you have a certain PHOPS model in Snowflake, hey, how does that actually translate into uh data bricks, right? start early with performance. So typically I would say uh as you've once you have done the code conversion pretty much as you start some of the data validation that is also the time where you start do doing some of this uh performance uh sort of um um monitoring testing load testing etc. So we have u you know this is something we'll share tomorrow as well over the last 3 years just even looking at the more the most commonly run queries if you look at that the performance of those have increased uh improved almost 77% um in in just that period and this is not benchmarks this is actual queries that our customers run and one of the most important takeaways in getting that performance optimization is manage tables as long as you pick manage manage tables.
35:54 Governance, performance, and wrap-up
There's a whole bunch of performance benefits that follow through. We will automatically uh do predictive optimization. So the way the data is laid out on the storage, we will automatically optimize that. We'll vacuum the data. We will uh pick the right clustering keys. So we have auto liquid clustering. So just by picking making that one simple choice a lot of those benefits come come through. Also important is the iceberg right.
So the managed iceberg is something we are able to actually offer for managed tables. Um technical details I can get into much more but that also gives you that um gives you that ability to integrate with a much larger ecosystem. Uh say you want to actually hook up your TRO or hook up your uh snowflake this becomes significantly simpler. I'm a technologist at heart and this is something I'm guilty of. So I'm just want to make sure right it's not all just about the technology it's important to capture throughout your journey hey what is the value that is being driven and this is important so that your relationship with your business uh you know deepens you are actually focusing on the high value use cases as opposed to uh just you know more somewhat blindly going after uh you know taking on this big estate and moving that over right it's not about the infrastructure it's about the business and this just helps you basically get into a more healthy organization where that you can uh grow.
Uh so the key takeaways here number one tooling and accelerators are key and we have a whole bunch of offerings here. Uh lake bridge is uh something we're going to announce tomorrow. Uh we also have partners who have their own tooling and accelerators. It's not so much about hey our tool is better than any other in the ecosystem. Um from when it comes to tooling our whole point of view is that we want the ecosystem to benefit.
So that is something we are looking to provide for free to the ecosystem. These are important but equally important or if not more is methodology right methodology is super important tying that methodology back to the reason you're actually migrating. Hey if ETL is uh TCO is why you're looking to migrate yeah go ahead migrate the ETL first. But if you're actually looking to unblock some AI use cases or genie use cases, the BI first migration is the um is an important consideration.
Iceberg is also not only critical from a open sort of adopting the open uh ecosystem but also from a migration standpoint, it's an important tool that can help you accelerate migrations, right? And lastly, but uh probably one of the first things that you should do is uh address governance, data modeling, and performance. Uh with that, we have about just 48 seconds, I guess, for Q&A if people don't mind. Um we can we'll be around for a few minutes also, so feel feel free to swing by and ask questions if you have more.
Thank you.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.