Migrating from SQL Server and Synapse to Azure Databricks

Summary

This video covers a phased migration approach from SQL Server and Azure Synapse to Azure Databricks, starting with Lakehouse Federation for immediate read-through access and adding CDC-based replication and side-by-side query validation before final cutover.
Lakebridge provides AI-assisted conversion of legacy T-SQL DDL/DML, views, and stored procedures into Databricks Lakehouse and Spark workloads, with Genie Code skills used to define custom SQL formatting rules and migration conventions.
Post-migration performance is addressed through DBSQL Serverless for fast BI, Lakebase Realtime for low-latency operational workloads, and Genie Ontology as an AI-ready semantic layer that preserves business context after the migration.

Migrating from SQL Server and Synapse to Azure Databricks

Learn proven best practices to modernize from SQL Server and Azure Synapse to the Databricks Lakehouse. The session will include Lakebridge's AI‑assisted migration to analyze and convert legacy SQL (T‑SQL DDL/DML, views, stored procedures) and Synapse scripts into Databricks Lakehouse and Spark‑based workloads. Adopt unified governance and privilege mapping in Unity Catalog, and follow a phased cutover: begin with Lakehouse Federation for read‑through access to SQL Server and Synapse, add CDC‑based replication, and run side‑by‑side reconciliation and query validation before switching over. Optionally leverage Lakeflow Connect for managed SQL Server ingestion under Unity Catalog governance. Finally, serve high‑performance BI at scale with SQL Serverless to deliver fast, reliable dashboards in Power BI and beyond. Learn how to de‑risk migrations, improve price/performance, and accelerate delivery.
Talk By: Mehran Golestaneh, SSA, Databricks ; Will Girten, SSA Practice Leader - FinServ, Databricks

Chapters

FAQs

What is the recommended phased approach for migrating from SQL Server to Databricks?

The recommended approach starts with Lakehouse Federation for read-through access to SQL Server and Synapse, letting teams begin working in Databricks without a full migration. CDC-based replication is added next for live data synchronization, followed by side-by-side reconciliation and query validation to verify results before the final cutover.

How does Lakebridge accelerate SQL Server and Synapse code migration?

Lakebridge uses AI-assisted analysis to inventory database objects, assess code complexity, and convert T-SQL DDL/DML, views, and stored procedures into Databricks-compatible Spark workloads. Genie Code skills can be defined to enforce specific SQL formatting conventions and migration rules, standardizing the output across a large codebase.

What is Genie Ontology and how does it support post-migration analytics?

Genie Ontology is an AI-ready semantic layer that allows teams to define business terms, metric definitions, and relationships that Genie uses when answering natural-language questions. After a migration, it preserves the business context embedded in legacy views and stored procedures, making the new Databricks environment immediately usable for self-serve analytics.

When should Lakebase Realtime be used after migrating from SQL Server?

Lakebase Realtime is recommended for operational and low-latency workloads requiring sub-second response times, such as application backends and transactional read patterns that exceed what DBSQL Serverless is optimized for. DBSQL Serverless is presented as the right choice for BI and analytical workloads that prioritize throughput and cost efficiency.

Full transcript

[00:10] Hi everyone. Uh thank you for being here. here. I know the conference is winding down and uh lunchtime and uh not as as easy of a task to go through this subject, but uh we're really excited to be here to talk about migrating uh from
[00:23] SQL Server and Synapse uh to Azure Data Bricks specifically with all the capabilities that you've been hearing about through Genieode and and uh optimizations and everything that I'll
[00:36] go over uh and I do have a demo for. My name is Maran Golastana. I'm a specialist solutions architect uh along with uh my partner and hello everyone.
[00:48] My name is Will Girtton. I'm also an SSA and for the last seven years I've helped customers migrate their workloads to data bricks and also make their BI lightning fast. Yeah, same same with me over 20 years of uh migrations and uh
[01:02] working on data warehouses, implementing data warehouses in and out of different platforms and uh previous engagement I was with uh terod data for many years and in fact uh I did have a session on
[01:15] how to migrate off of terod data using the batch inference using the AI that is uh this is a little bit of a different flavor with all the capabilities and everything we have. So getting into the
[01:28] why, there should not be any convincing here because we're all sitting here, but all of the capabilities that data bricks has when it comes down to AI, ML, and streaming
[01:41] uh and the rigid scaling that we don't have and uh with all the serverless capabilities and we're going to touch upon you know the migration aspect of it on the demo as well and the uh you know
[01:56] everything being in one place one go to shop. You don't have to uh use different software, different platforms, duplicate your storage and uh with a much much
[02:09] better TCO and uh one thing that per my past experience uh is that uh customers or companies are
[02:22] not really wanting to do the migration. And I kind of put together things that have come to me over the past uh you know engagements with different companies was that the last line of
[02:34] defense was it's so hard to migrate. It's too expensive, time consuming and uh there's so much technical debt sitting in one platform that it's really not feasible to to uh go through it,
[02:49] define the lineage, understand how objects are connect to each other and a lot of bandating that have gone into the place that is really hard to figure out and nobody wants to sponsor that. uh
[03:02] which could end up being a very lengthy migration and time consuming and with a really high cost. Uh so I've been on the other side and uh you know typically 8
[03:14] months to a year and a half of migration over a million dollar cost depending on the size of the platform that you're migrating from and the the pipelines could potentially be very complex. So
[03:26] taking those ETL pipelines, converting it into data bricks equivalent and understanding what they do. Uh and I really do believe uh with this agentic
[03:38] approach we uh we do have a great solution in place and that could really bring about the uh time to value and and and what it takes to do the migration.
[03:50] So one thing that uh you know you probably have heard from partners and there are so many tools out there uh but uh the focus here is about Genieode and how to use Genie code uh all available
[04:04] to you at no cost of course the cost of compute but but not as a separate licensed software that you have to implement uh and uh a very basic way to go about it which does produce something
[04:17] that is very desirable in my opinion and we've been deploying this uh with different customer environments uh building the scaffolding that you could
[04:29] DIY build upon or uh have the partners implement it the similar way and uh it really works uh so very simple uh I I'm not sure how how much you spent time on creating skills
[04:44] within the genie code uh so you have the foundational model with all the learnings or everything that's already been trained on but uh you could assign your own skills to it. So those are the
[04:56] uh you should say like how you want your system to plan things and uh beyond just prompting. So uh those are the definitions of you know the the skills
[05:09] coming from Claude uh of course and then use things in agentic mode. all of those standards uh whether it's a naming convention, SQL formatting, uh
[05:22] SQL writing, you know, like using CTE, using uh merge statements, things that you would want to kind of do the optimization or mig modernization on top of. Those are all the things that you
[05:35] could define within your skills and it would it would do it for you and you could make that better and better along the way. As soon as you go through a few objects of migration and and you realize
[05:47] it's missing something and you want to add to it, that capability is there. And the good thing is that you use Genie code to build upon that. So if you see something that is not up to par with the
[06:00] output, you have your uh genie code by writing that go ahead update it based on this lessons learned or based on this error that you got and skills have a lot of to-dos and not todos uh few shot
[06:14] learning. So uh it's it's it's very useful things to be aware of. I just put that as a a few bullet items uh that uh we do have a what we call a lake bridge
[06:27] synapse profiler. So it does establish the catalog federation and goes through all the objects that you potentially have and assign a level of difficulty complexity
[06:39] and kind of build out this inventory of the systems that you have. So that is it was something that was being run as a separate uh software but
[06:51] right now it's all integrated within Genie code. It's in a beta version. So something to look into that you would want to just bring in all the inventory of the systems that you have uh with Genieode. Again, what I'm going to get
[07:04] into as part of the demo is uh applying skills. Uh but you could assign an MCP server so that there's no problem building your own MCP server, custom MCP
[07:18] server that really works for you when it comes down to migrations with all the tools and capabilities that you want and then assign it to Genie Code as part of the setup. And uh from that point on
[07:30] everybody within that workspace could use the same set of skills and same set of tools within the MCP servers. Uh I this is a something that's
[07:42] new. Uh we do have an optimize command. So let's say you do do a conversion of a SQL statement. there's always that option of looking at the uh the query history and looking at the performance
[07:55] and and let the system come back with some sort of a recommendations of what would be most optimal to get into. It was announced uh at the summit genie zero ops it's an AI agent that does the
[08:09] job monitoring for you. So let's say you have a classic job or a serverless job or whatever job that you end up building through the Genie code. uh you could have the system continuously
[08:22] review the performance and be proactive about alerting if anything is is not performing well and and it to me this is uh the I built agents to do this now as
[08:35] part of the product that basically looks at the u you know spark logs and come back with recommendations so having this is really powerful and again this is a converting powerbi or tableau Hello uh
[08:48] dashboards we do have through Genie uh an import BI type of you know whether it's JSON XML or whatever the format is uh to import it in and convert it to an
[09:00] AIBI dashboard which is an equivalent to what PowerBI offers. So this is very powerful all of these things to to be aware of beyond what I'm going to get into as part of the demo. So let me just
[09:14] uh jump right into the demo. uh basically just to make things very simple I used u uh you know AI based tool to to
[09:27] basically create a uh retail demo it's nothing no with bunch of tables bunch of fake records in it uh and uh the idea is
[09:40] to take a ETL store procedure on that side of the fence without having to really move everything over uh build the skills necessary to build the tables on
[09:53] data brick side build the uh equivalent to the store procedure um on the data brick side what we call uh uh spark declarative pipeline I'm not sure if if you've had a chance to play
[10:06] with it it's a it's a very powerful tool that you don't have to do uh a lot that you had to do from an audit mechanism from restartability from uh error handling everything you know
[10:19] from data quality aspects all integrated with spark declarative pipelines through a CDC fashion of moving data and incremental load into the target state. So that was the idea to basically uh use
[10:33] the latest and greatest being the modern way of ETL and data engineering and take your objects and move them over and uh use a catalog federation in this case.
[10:45] Uh that's what I'm I'm doing. So it's it's basically uh this is a workspace uh a demo workspace uh within the environment. And just to realize
[10:59] what we have and I've already created a catalog federation that that doesn't copy the data. It just basically makes sure that uh you know looks at the data and establishes the connection. Things
[11:11] are slow because we're getting to another system to get information back. Just to show you that um this is uh there's no per se uh data copy here.
[11:25] So it comes back it shows how many gold tables I have, how many bronze table equivalent and the uh synapsite and uh you know and and uh running a few checks
[11:37] to see how things are. So the the gist of uh demo is that what I was explaining as far as the uh skills are concerned.
[11:50] Every work user directory or workspace directory has a an assistant uh in it. It basically uh this is where your skills go. Uh let's say you build
[12:02] your skills through a different tool and you want to import it in just do a copy over for the entire directory of skills. So in this case it took me like uh probably half an hour to build four
[12:16] different skills here. And uh I'll show you how everyone you know work differently. So the idea is that uh we do have the catalog federation uh on our
[12:29] end and the workspace has already established the unity catalog you know uh catalog and schema. uh it basically creating the skills that does uh uh
[12:41] converts the tables uh creating the skills that uh uh builds a pipeline which is in this case I've chosen STP pipeline it could be a classic job it could be but STP
[12:54] being the modernized way the whole idea of migrate and uh modernize and uh the final one I was just I just built it this this morning is a skill
[13:06] that uh what if I wanted to create a dashboard uh for data validation. So run the data validation for every job that you have and then uh show it on a dashboard
[13:19] visually that for every run that I've had this many failed this many is not comparing and basically comparing the result or the output table on two ends right synapse side and and uh data brick
[13:31] side and go through a bunch of rules and those rules could be I just put what are the common rules of data validation and reconciliation you could have your own set of rules
[13:43] Uh and the interesting part of this and I've deployed for my customer was interested in over 30 40 different rules to be applied. Uh and it wasn't just the
[13:56] ones that I picked six because it was very specific about every job. They wanted to have different types of rules, common rules and specific rules. So basically what I did was ask Jeie to
[14:09] look at the workspace directory, look at this Excel file, look at this all these rules defined, go ahead uh turn it into a a uh uh basically code that runs every
[14:25] time that you want and if they modify the Excel spreadsheet, it would remodify the notebook and then you rerun it again. then look at it visually through the dashboard everything is on par or not. So the good thing is that that
[14:38] interactive way of observing and deciphering files and being able to do that conversion on the fly or building out a notebook that does all of those things and every time you run it you
[14:52] could ask the skill to refresh it before running it. So with that said, uh I'll just jump into how uh these skills are and just just for you
[15:04] to be uh kind of aware of you know what we're converting here is the uh we have this uh
[15:16] in this case I just for visual purposes not that this thing is is going to use I put the store proc that synapse is is using I wanted to kind of So they're they're not that complex but I said
[15:29] medium level complexity go ahead create something that uh builds a gold table in this case and in this case uh happens to be a silver table. So they all relate to each other it could have been different
[15:43] workloads or you know any which way that we wanted to convert all at the same time or separately. So just to show you this is this is the equivalent of what is on the uh on the synapse site and uh
[15:58] to kind of observe uh what these for example um skills look like. Again this is not written by hand. This is been built. The scaffolding has been built
[16:11] and as I go through the process of conversion I ask Jeie to go ahead and update the skills fully aware of where the skills are and I'll show you uh how it gets added to the system. So
[16:23] basically uh through the settings you uh you know settings of Genie you have uh lots of MCP servers that are pre-built that you
[16:35] could apply to this whether it's Gmail or Google calendar and everything else. But the interesting part for me is uh the fact that you could assign all of these skills and look at the the skills
[16:50] folder. So it it knows where the skills folders are and then it gets invoked as you go through the conversion process and again uh you know the the for other
[17:05] customers also I've defined like an MCP server which you could uh also assign your own MCP server here too. So it's it's very flex flexible when it comes down to uh uh what you want to do and uh
[17:20] so going to uh the whole uh invoking invocation of a skill and what it's doing. I just wanted to quickly show you. So this is going to be a basic
[17:34] prompt. Sorry. in this case is going to be a very basic prompt that I would say
[17:55] and then using at and then this is where it's going to be uh pointing you to the skill that you have. So in this case I have everything under synapse. Uh
[18:11] so you see that that scale of validation is here. Uh there's a you know the code conversion is here. Anything that we want to choose. So in this case uh synapse to STP pipeline.
[18:44] So upon selecting that skill, I'm not going to run it. It takes like for this uh store because of the catalog federation it's doing it takes a probably a good 8 to 10 minutes of running but I'll show you the output uh
[18:56] the idea is the same. So basically in this uh uh kind of to as a demo I I I'm calling the skills uh uh you know to create the
[19:09] pipeline and then uh valid data validation skill as we talked about and the final skill is to build that AIBI dashboard for that visual summary. So
[19:21] those three I run um you don't always have to do it sequentially. You could have one skill that wires the rest of the skills once you're comfortable with the output. So it could be one command
[19:35] uh as long as you put the ad skill and it runs everything. So taking that store procedure uh and uh understanding the way things uh are with the lineage it
[19:48] basically creates uh a STP pipeline. So this STP pipeline is taking all of those SQL statements that you saw within the store procedure,
[20:01] separating them out, and then creating a workflow depending on the dependencies and the lineage of, you know, going all the way from silver to bronze and uh uh
[20:13] bronze to silver and gold and uh and basically it creates this. Again, it takes about 10 minutes. You guys can practice with it. Very simple to do. And then once you run the pipeline, it goes through and uh it it basically does uh
[20:30] what STP does from from a workflow standpoint along with the fact that it knows exactly what was the latest uh data set that was processed last whether you want to do files or the records
[20:44] through a change data feed that we have. So it knows exactly the last time stamp of observed records and it records the data finish not that many records here and then moving along with the with the
[20:57] load. So that once that's done I uh again I've I've run the validation skill that I showed you guys. uh it basically uh creates uh the output as a table
[21:12] summary table that has how number of records failed and rules applied and everything else. just to show it in a visual way. Um we're uh
[21:27] this is a migration dashboard that the cut over something that whatever that uh and basically uh it would be just one second. I did it create a better one.
[21:43] Yeah, this is a better dashboard that so it basically I on the skill I don't know if you saw the writing part of it. I said go ahead create me a validation dashboard with the first tab being the summary of the exported or the the runs
[21:56] that you have and upon selecting each run go ahead give me a secondary tab that has all of the statistics for that run. So the good thing is it did that and uh it was in one shot it was able to
[22:11] build this out. So you really don't have to know how to build out these you know all the links and everything else. So it goes through it shows uh you know for a specific run this is for everything but
[22:23] if you wanted to select a single run here you could and it shows you everything else. So on the summary side, uh this is very basic uh but uh you
[22:35] might want something that's more complicated, something that's more alert you to where the error is. So so show some sample records that failed. All of those things are part of the
[22:47] possibility. But right off the bat, it just created this, which is pretty good. It would take time for somebody to do this. This was less than five minutes of generating the AIBI dashboard.
[22:59] So, uh with that said, I uh that was my presentation. Just bear in mind that we are specialist solutions architect uh within data bricks uh offered at no cost
[23:14] to customers. So in case you do want to build something like this and you want us to kind of come in do a proof of concept pilot uh talk to your account team and get us engaged and definitely
[23:27] we will be there to support. So I'm going to pass it on to to Will for the rest of the presentation. All right, cool. Thank you. Okay, I need everybody's help in this next section.
[23:40] Uh I get it. It's after lunch. It's the last day of the summit. Everybody's tired. I get it. The the seats at a little bit of a recline. We're going to be doing a lot of this. Okay. So, by show of hands, who here currently uses
[23:53] SQL Server today? Wow, a lot of people. And by show of hands, thank you very much. Uh, by show of hands, who here uses uh, Synapse in any of its forms?
[24:05] Okay, still quite a few. Okay. And then it wouldn't be a BI talk if we didn't talk about PowerBI. Who here by show of hands uses PowerBI? Okay, awesome. A lot of people. Perfect. Okay. Um, so thank you Morand. Um, so
[24:19] Moran talked to talked to you about how easy it is to convert your workloads to data bricks using Geniode. Uh, he went through his demo and his voice was very soothing and uh, it's time to wake up. Okay, so in this section I want to talk
[24:31] a little bit about the high performance. How do you squeeze out lightning fast BI at scale? Um, so I want to talk to you about a couple things. So if you tuned into the keynote on Tuesday, you probably heard a little bit about this something called Raiden Engine, right?
[24:43] Everybody hear about it? It caused a lot of buzz. Um, and if you've used data bicks, you probably know something about data bricks SQL, DBSQL for short, which uses the the photon engine. Uh, now you probably know you it can handle those
[24:56] large massive analytical queries on the order of terabytes or even pabytes at scale. And it's great if you want to pop open the the SQL editor, maybe do some ad hoc querying, exploration, things like that. Maybe you want to use a SQL
[25:08] warehouse to do your ETL, refresh your materialized views, uh refresh that goal layer, things like that, or maybe you're experimenting building out a dashboard. The thing is, it only really scaled up to about hundreds of concurrent queries
[25:21] at a time. H but that all changed on Tuesday, right? We heard about this thing called Raiden or Lakehouse real time. What's the difference? Well, Raiden is the engine. Lakehouse real time is the the SQL warehouse behind it. Um and we saw in in that demo that it
[25:35] could handle thousands of concurrent connections at a time and it'll all it all executed uh in very quick uh in just the order of a few milliseconds. Now this is opens the door for a serving
[25:47] layer, right? So uh if we can do quick uh low latency queries and have thousands of connections at a time, why not use Raiden for everything, right? It's a big sledgehammer. Can't we use it
[25:59] for everything? Uh it should work, right? So in this next section I want to give you uh three things to take away. One is the information to be able to choose which engine to choose and why. Uh two I
[26:12] want to give you my top five tuning techniques for making your BI reports super super fast. And then third hopefully by the end of the section I want to convince you that you only need a single copy of your your storage
[26:24] layer. You don't need to copy out the data and build out something like a serving layer or a platinum layer sometimes it's called. So, let's dig into this lakehouse real-time thing. Um, so what's different? How do you use it? Um, it's
[26:37] still in private preview, so you can't use it today. You have to connect to your uh data bricks account team and have them enable it. But imagine that you're creating a warehouse today. It's just going to show up as a radio button. That's really the only difference. You're still going to go out to your
[26:49] create warehouse UI and you're going to select next to serless or classic. There's going to be a new option there for for real time and that's the rated engine. The hardest part about spinning up a warehouse is two things. One is
[27:02] being able to to accurately pick the t-shirt size of your warehouse and two picking a nice name. Uh that's it. Everything is is handled for you automatically behind the scenes. It's a reduced UI. It's versionless. It's
[27:14] runtime timeless. Autoscaling already happens for you uh uh out of the box. You don't have to think about it. Uh, and like I said, it's it handles thousands of concurrent queries and handles that those really tight SLAs for
[27:27] the millisecond responses. If you tuned into that that demo on Tuesday, you saw Reynold uh refresh 8,000 dashboards and each dashboard had multiple queries. The do you remember what the median
[27:39] execution time was? 31 milliseconds, 32 millc, something like that. It takes about 150 milliseconds for you to feel a beasting or feel that the the the the oven's hot.
[27:51] Like that is insanely fast. Um so when would you use uh lakehouse real time? Well, it's purpose-built for serving. That means if you have an ETL or ELT use case, uh this is not the right engine
[28:03] for it. This is this is really built for those tight SLAs, millisecond serving, thousands of concurrent queries, uh that are read only. And you might have a use case for uh thousands of users
[28:15] refreshing a a BI dashboard. Maybe you're and you have a use case like uh monitoring the the IoT sensors of your gas pipeline or detecting fraud within your your pipeline or maybe it's uh intrusion detection within your SIM sim
[28:28] pipeline. Uh if you need that low latency uh interactive real-time responses then RAIN engine is the right right engine for you. The best part about this, if you're using PowerBI
[28:40] today and data bricks, you're probably making a copy of the final data, creating this thing called a platinum layer into one lake and then that is how you get your your import like speeds in your PowerBI dashboards. With Raiden,
[28:52] that changes everything. You no longer need to make a copy of the data. When as soon as you copy the data, you have to spin up a pipeline to refresh the data, keep it in sync. That's double the storage, double the compute. You don't want to do that. Uh so when to use
[29:04] lakehouse real time good rule of thumb is if you have a read only query today that executes on less than 10 seconds on DBSQL serverless then that's probably a good candidate for Raiden.
[29:16] So key T takeaway here uh Raiden Lakehouse real time is really simple to spin up. Two which which engine to choose when choose Raiden if your query is 10 seconds or less on DBS
[29:29] SQL and three you don't need to make a copy of that data. Now when would you use data bricks SQL serverless? Well, I kind of hinted at it, but uh really this is kind of the workhouse, the power horse. If you have a massive query that's that's doing
[29:42] something like a scan on terabytes of data, doing deep joins or maybe applying like really massive aggregations like window functions uh on terabytes of scale, then DBSQL is is kind of the the
[29:54] engine that you want to go after. I love DBSQL serverless because it spins up within seconds. It automatically autoscales for you. It aggressively scales down for you and it has the power of the the photon engine as well. So you
[30:06] can still use it for those large complex analytics that are queries that that last longer than 10 seconds or aggregate data across terabytes or pabytes of scale or if you're refreshing that goal layer. Maybe you have it within a STP
[30:18] pipeline refreshing the the final serving layer or maybe you're just building out an ad hoc uh dashboard or exploring your your data sets. This is the go-to engine for you. Now, on the far right, these are kind of like the my
[30:31] two favorite goodies here. Uh, intelligent worklet load management and deep learning scan optimization. One more time, by show of hands, who here has heard of predictive IO?
[30:46] You don't count, Ben. Ben's from data bricks,
[30:59] and it can predict past your metadata. It can predict past your statistics. It can look at your query patterns and predict within the file where it should search for an answer to your query. And that only happens uh with data bricks
[31:12] DBSQL serverless. Okay, now I got to go kind of fast here and I'm sorry because I'm running out of time. I want to leave you here with my five top tips for tuning fastbi. So kind
[31:24] of the obvious one, use raen. If you got a query that's 10 seconds or less, use raen. That will literally take your dashboard from seconds down to to milliseconds. Two, this is kind of a little bit more subtle. Again, this is uh kind a similar
[31:37] name, kind of different consequence, but there behind the scenes. If you have a UC managed delta table, there's predictive uh optimizations that happen in the background that you don't even have to think about. They run
[31:49] automatically. Nobody thinks about them. They kick in and it's not until a nerd like me comes up here and it tells you that that it's there that you think about it. So what it does is it will automatically run things like the maintenance operations like optimize or
[32:01] analyze the collect statistics. And this helps when your query is planning out uh the query plan how it can optimize it. Three, uh, liquid clustering. I'm sure everybody has heard of liquid clustering. We announced it a couple of
[32:14] summits ago. Uh, I want to leave you here with a tip. Uh, you should you should always liquid cluster your tables. However, which keys should you you cluster on? Well, does it really matter? Do you really want to do that?
[32:26] If you turn on cluster by auto, data bricks will automatically look at your query patterns. It will look at things like your join keys, your aggregate keys, uh your filter keys, things like that. It will detect which key it should automatically lay out the data to. so it
[32:39] can answer those queries a lot faster. And then the the automatic predictive optimization will apply those uh the optimizations under the hood. And then last but not least, I kind of hinted at the statistics there how that can help
[32:51] kick in the the query engine and kind of optimize your query plan. But just keep in mind Delta there's a subtle configuration there. Well, by default, it will only collect the first 32 columns uh statistics on the first 32 columns. So if you have a really wide
[33:04] table with the the join keys or keys, filter keys, things like that that fall outside of that, you're going to miss out on those optimizations. So collect table statistics on all of the columns or use column mapping within Delta to
[33:17] bring those those columns forward. And then number five in um sorry everybody's taking a picture here. It threw me off. Last but not least, the the metadata
[33:29] layer is very very important. However, with Raiden, we're talking about a different scale here. We're talking about milliseconds. Milliseconds matter. So, when you're reading the metadata, yeah, sure, it's in milliseconds, but can you shave that off? Over time, your
[33:41] deletion vectors will accumulate. You need to purge those and reorganize the data to make the the read a lot faster. So, these are my five top tuning tips. Sorry, I have to go a little fast. Uh,
[33:53] now hopefully I've convinced you that a couple things. One, rating can give you those really, really sub-second uh fast queries. Two, you don't need to make a copy of your data into one lake and create a semantic layer there. I want to
[34:06] convince you that you can create that semantic layer down in data bricks in uni catalog and unlock a a lot more potential and still support tools like PowerBI. So why would you why would you do this?
[34:18] Well, if you don't have to make a copy of that final data, yes, you don't have to double the compute. You don't have to double the storage. You don't have to maintain that separate copy. But another thing happens. You only have to apply your governance once. So you define your policies once. They're enforced everywhere. And then when you're using
[34:32] the Genie code to experiment with your migration, a lot of the things already happen behind the scenes. So things like bringing over your entry ID users and groups or your service accounts, they happen automatically at the account
[34:44] level. Those identities appear immediately. They're enforced across all of your workspaces. You don't really have to think about them. And then things like modifying your grants or selecting your your schemas and cataloges, they map seamlessly. Or even things like row filters or column mass.
[34:57] So if you bring if you point Braden at your queries, you can have your serving layer, you have your import like speeds, you don't have to duplicate your governance, you don't have to duplicate your data for a serving layer. The you have a single unified location, but
[35:10] something other what I what I call magical happens, and that's this geneontology. They went kind of fast during the demo, but I want to break it down to uh why it's important. If you have that semantic layer down in data
[35:22] bricks uni catalog and it's unified there, you can support your BI tools. But another thing also happens too. We're the data and AI company. We in the background Genie ontology will kick in and it's a continuously updated
[35:35] knowledge graph that you're not you don't even see. You don't have to think about but all Genie always has the context of your latest uh data your business how that relates to your questions that you ask. So things like your measures it already knows what
[35:48] you're going to be asking and how that relates to your overall business. It can predict the types of things that you're looking for and suggest other things that you should search for. And this is only happens if you have that semantic layer down a unique catalog. So it kind
[36:00] of opens the door for your agents to ask questions, answer questions that you can trust and then provide the lineage of how it got to that that that decision. Okay. So went super fast. I'm sorry. uh
[36:13] at a time but I want to end here on uh how I would migrate to data bricks if I'm coming from something like a SQL server or synapse. So if I were to do that today I would do it in a couple
[36:25] phases. One is don't copy any of the data today. So leave your data as it sits but use the power of uni catalog. So there is a lakeflow federation what allows you to create a connection to either your SQL server or your dedicated
[36:39] SQL pool and synapse and create what is known as a foreign catalog. So no data movement you have a connection straight to the source you can query that all of your tables show up and then all of the optimizations like pushing down your
[36:51] filters uh skip data skipping pruning uh columns things like that already get pushed down the source. can still see performant queries today and you can experiment with things like Moranush with the the genie code. Uh and then you
[37:05] get on top of that you get uni governance at start. So you can define your policies and then have that for the next phase. So the next phase once you start trusting that hey uh this migration is not as complex or as risk
[37:17] as I thought you can start to establish a CDC pipeline. So with the power of Lake Connect, you can you can do a historical load and then immediately cut over to a CDC, land your silver, clean
[37:29] up your data, apply your business measures, uh curate your gold layer, and then have a sideby-side comparison there where you can fall back on that first phase to do your data validation. And that helps with your migration validation, making sure that things are
[37:41] coming over, converted properly, your measures are being calculated correctly, etc. And then last but not least, once you have that confidence uh in the migration, now you can start beginning to phase out your SQL server and your
[37:53] Synapse pools, things like that, and you start realizing the the cost, the the cloud cost savings uh that we talked about. Okay, so in summary, sorry going super fast again. One, modernize to the
[38:05] lakehouse. Try today. Use the skills that Moran talked about. Use Lakef Connect. Use Genie. It's awesome. Don't migrate by hand. AI is everywhere. I literally use AI to do everything. I use
[38:17] AI to figure out what restaurant I should go to. Things like that. So, use Genie Code to help you do things uh that used to take months or even years. You could do in a couple days. Maybe you want to experiment uh this weekend. Last
[38:29] but not least, use those five tips. They're awesome. A lot of it is just letting data bicks platform kick in and handle the optimizations for you. I highly recommend that. I know it's tempting to drop down in the configs. It's, you know, you kind of feel like a
[38:41] nerd. It feels awesome. But use data bricks platform. It's there. There's a lot of built-in intelligence there. And then last but not least, bring that semantic layer down in data bricks. Leave it there. It's going to open the door for things like ontology, tracing
[38:54] your lineage, your auditing. It already happens automatically. Your governance applies at once. And then last but not least, drisk your migration. Break it up into phases. Don't trust me. Go try it out today. Uh, you know, create that
[39:06] connection. have one of your admins create a connection to your your SQL server or your Synapse and start experimenting today with a migration. It's really awesome. And then my call to action here, like Moran's saying, uh SSA
[39:18] are are here to help. So if you're no matter what journey you're on, everybody has a different migration journey. Pull in an SSA. We're here to help you. We're totally free. We have a whole SSA offerings that curated to migrations. So
[39:30] no matter your journey, if you want some help, just maybe a sounding board, we're always here. talk to your account team and we're here to help. So perfect. Yeah. Thanks everyone. I think I went over about 10 minutes. This this light won't stop flashing.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.