Migrate to Lakeflow Jobs: Modernizing Data Orchestration on Databricks

Summary

This video covers practical migration patterns for moving from legacy orchestrators to Lakeflow Jobs, Databricks' native orchestrator for data and AI workloads, including how common legacy DAG patterns map to Lakeflow Jobs equivalents.
Genie Code provides AI-assisted job migration and can accelerate bulk conversion of legacy pipelines, while Databricks Asset Bundles (DABs) enable source-controlled, multi-environment deployment to reduce risk and simplify rollout.
Real customer migration experiences are discussed, including the tradeoffs between the familiarity of existing infrastructure and the benefits of consolidating orchestration natively onto the Databricks Data and AI platform.

Migrate to Lakeflow Jobs: Modernizing Data Orchestration on Databricks

Still running orchestration outside Databricks? That often means more tools to manage, less context about your data, and more overhead connecting pipelines, schedules, and dependencies across systems. Join this video to learn how to migrate to Lakeflow Jobs, Databricks’ native orchestrator for data and AI workloads. We will cover practical migration patterns, the tooling available to accelerate the move, and the tips, techniques, and best practices that help teams reduce risk and avoid common pitfalls. You will learn: Why teams are consolidating orchestration onto Databricks How common migration patterns from legacy orchestrators map to Lakeflow Jobs What tooling, tips, and techniques can speed up migration and simplify rollout The best practices that help teams move faster with less operational overhead What real customer migrations look like in practice, including key challenges, tradeoffs, and outcomes
Talk By: Greg Hansen, Specialist Solutions Architect, Databricks ; Saad Ansari, Product Manager, Databricks ;

Chapters

FAQs

Why should teams migrate to Lakeflow Jobs from their existing orchestrator?

Lakeflow Jobs is Databricks' native orchestrator, which means it has deeper context about your data, eliminates the overhead of connecting external schedulers to pipelines, and is fully managed to reduce operational burden. Consolidating onto Lakeflow Jobs also unlocks native integration with Genie Code, Asset Bundles, and the full Databricks Data and AI platform.

How do common legacy DAG patterns map to Lakeflow Jobs?

This video walks through how standard DAG constructs such as dynamic parameter references and dependency chains translate to equivalent Lakeflow Jobs configurations. Genie Code can assist with this translation by interpreting existing DAG logic and generating the corresponding Lakeflow Jobs structure.

What are Databricks Asset Bundles and how do they help with migration?

Databricks Asset Bundles (DABs) express the entire Databricks project as source code, enabling source control, multi-environment deployment, and reproducible rollouts across development, staging, and production. During migration, DABs allow teams to deploy the same job definition consistently across environments with reduced risk of configuration drift.

What is Genie Code skill invocation and how does it accelerate migration?

Genie Code skill invocation applies a predefined migration skill to many jobs at once, allowing bulk conversion of legacy pipelines rather than manual one-by-one rewriting. This video shows how this capability can transform what would otherwise be a multi-week manual migration into an accelerated, agent-assisted process.

Full transcript

[00:08] All right, perfect. Thank you very much. Hey folks, um come on over. There's a lot of space up front. Uh so, thank you for taking the time to come to our talk. Um we're among friends. Uh we'll make sure
[00:20] that what we're showing you here is real. Um but, uh you know, we have to put the the legal disclosure up here, but um I'm Saad. I'm a product manager at Databricks. I work on jobs and um I used
[00:34] to work uh in my own company making Fitbit for cows, uh which is another story. Um and then I was a developer before, and uh I used to study here in the Bay Area as well. Uh Greg? >> I'm I'm Greg. I'm a specialist solutions
[00:47] architect at Databricks helping customers with data engineering problems. Um an open source contributor to Apache Spark and a number of other projects, and I studied at Georgia Tech. >> Awesome. Uh all right, so let's get
[00:59] started. I'm going to zoom in my speaker notes here just so I stay on track. All right, next. Okay, so I'm assuming people are here because of migration. So, could I see a
[01:12] show of hands of how many people have done a migration like in the last 6 months? Okay, a few hands going up. Okay, that's good. Um so, the rest I'm assuming you're you're in this talk to get
[01:25] prepared for uh for this. So, there's a version of this talk where I talk about how uh jobs is uh the best uh orchestrator ever. Uh this is not that talk. Um we're going to try to talk
[01:37] There is a talk about that, by the way. It's a little bit later today. But, uh we're going to talk about uh the reality of what um migrations are actually like. And um you
[01:49] know, like you have the thing that you're working with today. It's um it saves you money. It, you know, gets the gets the job done. status quo is good enough. And um then comes the new thing and it's
[02:01] uh cheaper, faster, um and of course there's uh agents. We have to say that. Uh we're legally bound to say agents uh 50 times in this presentation. That's a joke. Uh uh but no, agents are great. You'll
[02:14] you'll learn a little bit more about that today as well. And so this is usually where the pitch stops and we talk about uh how uh you know, you're going to get this uh value out of this new thing that we built. But you know,
[02:26] the the reality of it and I think folks in this room would agree that you've got uh you know, on your existing infrastructure you've invested processes, tools, uh training and um all your planning has gone into
[02:39] it. So there's a this investment and this familiarity on uh on the existing tools uh that you have. And um uh Oh, sorry. I've got some password
[02:51] prompts here. Okay, there we go. Uh and then the other thing that comes up often is uh there's a lot of hype. And uh as engineers we're all skeptical of that uh and as we should be. Um and
[03:04] then there is this hidden cost or it's not really that hidden, but the switch cost of uh using the new thing. You you have to retool, you have to rewrite your DAGs, rewrite your logic, test it. So um
[03:16] the actual value uh in in your minds might actually be lower. Um and yet we see people migrate. And um what happens in our experience and we've both Greg and I have seen a number of
[03:28] these migrations first hand is there's either some sort of a scale uh change that's happening. You've gone from 10x to 100x or uh perf or uh cost TCO becomes a a conversation. And uh
[03:42] there might be organization edicts, you know, like uh someone is saying you have to move over. Um and you know, we have to kind of follow through. There's some top-down decisions. Then obviously AI is kind of disrupting everybody's job now.
[03:55] Um and so like these uh what I call push factors kind of convince you to kind of move away from the status quo. And then there's these pull factors which uh you know, one of them is all the nice
[04:07] features that we'll talk about a little bit today as well as in the other talks that we have scheduled. Um uh and then this talk which is about how migrations are becoming easier and we'll take you through some of the frameworks
[04:19] that uh we've built uh on this. So just to make sure uh okay, and clearly the internet gods are not with me. So there was a nice image of a DAG uh up there. Um but um just to make sure we're all in
[04:32] the same room, we're talking about Lake Formation Jobs. And jobs are not just uh scheduler. Um it's you know, the simplest way to build, to trigger, to run reliably your
[04:44] data pipelines. And uh you know, you define your unit of work, whether it's a notebook or uh SQL script or what have you, pipeline. Um and you stitch them together using
[04:56] these building blocks. So that's loops, conditions, dependencies, retries, um and and and so on. And then we have triggers where you can uh you can do more than just cron.
[05:09] Um you can react to file arrivals or uh data conditions or table updates. And you should run when your data is actually ready, not uh hope when it's ready. And then
[05:21] um as as jobs you we kind of ship the orchestrator built into the lakehouse. So you delegate the infrastructure to us. Uh so there's no cost of running the orchestrator. We actually have a 395
[05:33] SLA. Um and uh you pay with the compute of course, but you get that for um out of the box and including observability, logging, and monitoring, and a bunch of stuff that you heard
[05:45] about uh yesterday and today around uh zero ops and a lot of things around Genie. And we'll go through some of that um later. Um and when it comes to these migrations, over the last couple of years, we've seen these customers move.
[05:58] And I it, you know, just just to give you a little bit of a why uh this uh does happen um in in our experience. So, part of the like I talked about these push costs or these push uh um motivations. So, customers want to
[06:12] optimize their compute, they want to be able to run jobs compute instead instead of all-purpose or go to serverless, um or be able to bin pack tasks together, um or uh you you might also have
[06:24] infrastructure issues. We carry the pager, as I mentioned, uh on your behalf. Uh or you're dealing with some sort of a version upgrade. Like Airflow uh version 3 is out and we actually have a lot of customers who are on older
[06:36] versions today on the platform. And then, you know, you might have uh orchestrators that might be uh either deprecated or no features are being added. And so, so you've got this push factor on one side and pull factor uh on the
[06:49] other. And let me just keep an eye on the time. All right. Um And uh we are shipping a lot of good stuff, uh which uh this talk is not about that one, uh but we will we can
[07:01] talk about that uh later. Um and then, just to kind of reiterate uh what we do, it is a sort of enterprise grade orchestrator. Uh 1.7 billion, this was a few months
[07:13] ago. The data is actually much higher now. Um but that's a lot of jobs that we're running on our customers' behalf. Uh 395s, as I said. And then, obviously, uh the uh unit economics are good
[07:26] uh because uh you don't have to pay for this. Um Now, this is a bit of an eyesore of a chart to read, but I was trying to get Claude to write all the feature uh comparisons of uh what we have uh today
[07:39] and what is coming up. But I just want to highlight a few things. So, on jobs, um this is the the right-most column is where we will be at the end of this conference and we these are the the
[07:51] updates that we're going to be talking about. Uh so, on jobs now, we will be able to support multiple triggers. Um and uh we are building um context awareness within the scheduler. So, we'll be able to recommend a
[08:03] schedule for you, which is optimizing uh your compute, which is uh basically we we know what your um jobs are doing, so we can recommend that for you. So, for instance, lineage, what you're
[08:15] reading from, you're better off running a table update versus a cron trigger. Um we're building in support for uh sensors and customized um sort of dependencies outside of Databricks.
[08:27] Uh and uh that's a pretty exciting uh set of investments that we've built. Uh we're have like about 50 or so that we're going to be publishing in GitHub. Again, that's the second talk.
[08:39] Um And then, uh if anybody's used apps, I used to be the apps PM before, so I'm pretty excited about that. Now, we support uh visual authoring, so you can just go on to the jobs canvas, drag and
[08:52] drop, and everything is source-controlled for you. You don't have to go worry about YAML if you don't want to. Um and then obviously, we've got all these AI skills. Um Speaking of AI skills, um Genie code is
[09:04] uh getting more and more powerful. You saw probably at yesterday's keynote that more than 50% of pipelines are being uh managed by Genie code today on on Databricks. And we'll actually showcase
[09:16] some of that uh today as well. And um uh Zero Ops is another great uh product uh that we're launching. Uh there's a number of talks uh we'll have a
[09:28] a list of talks that you can also go to in addition to this one. And then, we're also introducing some agentic task types for Agent Breaks, Genie code, Genie, all within the jobs canvas. Um and then with that I will hand it over to to Greg so he can
[09:41] take us through some of these uh uh migration actual migrations. >> Yeah, thanks Odd. Um, so hopefully we've given you enough motivation to migrate to Lake Oro jobs or maybe you've brought
[09:53] your own to the session. Um the next obvious question is what does a migration look like? Um, you may have hundreds or thousands of pipelines that you're managing in your business and maintaining a a manual migration is sort
[10:06] of untenable for sort of three key reasons. The first is that there's considerable complexity that you need to understand. Your jobs are dependent on one another and understanding those complexities at scale is very complex.
[10:18] The second is that the pure effort of processing all the configuration in those jobs is often something that an organization just simply can't tolerate. And the third is that uh manually doing this process
[10:30] introduces the risk of human error and sometimes those human errors, even if you have very strict validation processes, can go undetected and cause a failure three months down the road after your migration.
[10:47] So, you probably don't want to try this, but if you do, you will realize that not only do you want automation, but you need automation. Uh automating this process, however, is another complex technical challenge
[11:00] because each orchestrator defines uh core concepts differently. They each use their own domain-specific language and those domain-specific languages are as different as natural languages that we use for communication. Uh there's rarely
[11:13] a one-to-one mapping. They often have differences that span across three different categories. The first would be structural differences in the syntax and grammar um of how they express things. For example, in Azure Data Factory, you
[11:26] may be familiar with pipelines and activities, whereas in Lake Oro jobs, we have jobs and tasks. The second is in semantics and how the underlying syntactics represent some
[11:38] kind of intent in the target system. So, for example, in Lake Flow jobs, you can define a Lake Flow connect manage ingestion pipeline. That's something that you might have to do more procedurally in something like
[11:50] ADF or Airflow. And the third are pragmatic differences. And these are differences in how you want to use the tool to get the most out of it. How you can optimize your performance, maintainability,
[12:02] observability, and sort of all the big underlying concerns that you have as an organization. And a good example of this would be if you're migrating to Lake Flow, you may prefer to adopt more declarative style paradigms to use something like
[12:16] Spark declarative pipelines. So, the sum total of all this is that intent is the most important part of the translation process.
[12:28] And if you convert too literally using something like a rules-based transpiler, you end up with something that is quite brittle, often fails when cases don't map cleanly, and is
[12:40] often sort of over-engineered and better architected for your legacy system than for the system that you're targeting. You could try to solve this by just throwing everything into an LLM, but as most of us who have used LLMs knows,
[12:54] this is very hard to govern and standardize. And so, you would like to steer the LLM with the best practices of your target system and also the best practices that you'd like to enforce in your org.
[13:06] So, the key is to use both. You have LLMs that are handling ambiguity. They're delegating different edge cases to sort of a human in the loop who's guiding the process. But then you offload things to deterministic
[13:19] transpilers that are handling the non-ambiguous cases and providing a lot of automation.
[13:32] So, this is the exact that we're taking. Our migration tooling is architected as a AI layer and a deterministic layer. The AI layer is invoked through Genie code or any agent harness that you have available to you.
[13:44] It includes a set of skills that codify the migration steps, handle customization, delegate requests back to the user, identify patterns, even generate code. Uh they're also some bridge to some MCP
[13:57] tooling. And that MCP tooling is invoking in the deterministic layer a set of parsers, translators, and preparers.
[14:09] And the sum result of this is that we get a declarative automation bundle that expresses the sort of semantics along with any customization in flight that you'd like to do during the migration process.
[14:25] So, here's an example of what that customization in flight looks like. You can see in this case the transpiler detected that we have a dynamic parameter reference coming from the legacy system.
[14:37] It's associated with the retry policies. So, it's now asking the user via prompting in the session if the user wants to maintain that dynamic parameter reference or move to a hard-coded value. The user has chosen to maintain the
[14:49] dynamic parameter reference. So, in this case we're handing things back off to that transpiler that's effectively parsing that out and creating something that's supported in Lake Flow.
[15:09] So, uh this is what the migration journey looks like using a tool like this. We start from the legacy definitions in some type of local or workspace backed repository.
[15:21] Uh we'll first run some discovery on what you have in your legacy system to create an inventory of what needs to be migrated. Then we'll pass that inventory through our converters, map all the activities to something that's
[15:33] equivalent in Lake Flow. And the final step will be to package everything and create a deployable declarative automation bundle. You can then verify this to confirm that the intent was preserved because
[15:45] everything is configuration files and code that we're generating for you. And then once you're ready, all you have to do is deploy that bundle either using your command line interface or directly from the workspace.
[15:57] And the idea with this is that by using agentic tooling we can speed up the time to value, reduce the complexity, risk, and the effort involved in the end-on migration
[16:09] process.
[16:21] So, we're going to do a live demo of this tooling in action. The first thing we're going to start with is this deterministic migration of an Azure Data Factory pipeline with sort of three Databricks notebook activities and just show you what the tooling looks like.
[16:40] Let me switch over to my workspace. And so in my workspace, I've actually created a Git folder. >> Actually, can everybody at the back see? Should we zoom in? Yeah, is this
[16:52] visible? Okay, I see thumbs up, so we can proceed. Yeah. >> So, I've created a Git folder and in this Git back folder, I have all of my configuration that defines my orchestration layer.
[17:05] I'm going to open my Genie code window. >> Zoom. Oh, can you zoom in just a little bit? >> Sure. Yeah. I'm going to open my Genie code window and I'm going to
[17:18] invoke a skill that's going to handle the end-to-end migration of the simple pipeline.
[17:32] So, the first thing that's going to happen is Genie codes obviously going to read the skills, load them into context so it understands exactly what it needs to execute as a set of steps in the migration. You can see that it's going to read the JSON files from that folder in my
[17:44] workspace. Now, it's going to hand off to the MCP tool.
[17:57] And this MCP tool is actually going to handle all the conversion on the server side.
[18:10] And then once it returns, it's going to summarize everything that it's done. It's going to estimate the complexity of the migration and show me the overall coverage.
[18:22] And then it's going to instruct me to review all the artifacts that it generated in the metadata. And so, if I look at my target directory, you'll see now I have this
[18:34] complete bundle. It contains metadata containing the inventory of all the activities that we translated along with a profiling report in CSV. In this case, it's just a single row
[18:46] that shows me the complexity and the types of activities that were migrated. I also have a resources file that defines the exact job configuration in YAML of this job. So, you can see I have
[19:00] a high-level pipeline with three tasks in it. I've pulled the notebooks that are run as part of this into the bundle directory so that this bundle is clean and can be deployed immediately across
[19:13] different environments. And if I wanted to do that deployment, uh it's fairly straightforward. So, I'm going to go back to my pipeline simple. Going to open this in my bundle editor.
[19:31] And I'm just going to click deploy to dev. And it's literally that easy. So, I've gone from
[19:44] a pipeline defined in Azure Data Factory to this in a matter of minutes. And beyond that, all of the dynamic references that were
[19:58] sort of codified in specific ADF syntax are now being expressed and pushed down directly into the parameterization of the Lake Flow tasks.
[20:14] So, um this is nice, but uh your pipelines probably don't look like this. Most likely, uh
[20:26] they look like this. They're quite complex. They involve activities that are being run outside of Databricks, deep control flow, uh lots of complex logic, maybe even end-user notifications via API calls or posts.
[20:39] So, the question would be, can we migrate that? And um I'd like to show you. So, I'm going to switch back.
[20:55] And just to give you an idea, here's my ADF pipeline. You can see I have I think about eight or nine different activities. It's quite nested. And there's quite a lot to manage here.
[21:08] I'm actually looping over some ingestion from a SQL server. And that ingestion is driven by a lookup from a metadata table. So, I'm actually pulling information from SQL that drives
[21:20] a metadata driven pipeline for ingesting from SQL. So, let's go back to the workspace. I'm going to open up my Genie code session one more time. And I'm going to start
[21:33] with the same skill invocation. This time, I'm going to invoke it on the complex pipeline.
[21:45] And so, we'll see the same series of upfront steps where we're going to read the input files and then hand off to the MCP tool.
[22:00] But in this case, the MCP tool is actually returning early. And it's going to ask me as the user for input on some things that it observed throughout the pipeline that I might want to either modernize, consolidate, um or guide.
[22:12] So, you can see that uh once we get through the the prompting generation, we're going to have a number of different options to customize this migration in flight. The first is that uh it's
[22:25] detected that we have some type of metadata driven ingestion pattern. And so, it's offering me the opportunity to modernize that to something like Lake Flow Connect. The second is that it's detected activities that are running outside of
[22:37] Databricks for which we're generating code. And it's offering me the choice to customize that compute to either serverless or use a classic compute cluster to run those.
[22:50] There's also some metadata driven copying options and an option to customize how the notifications are happening so that we can use the built-in notifications capabilities of Lake Flow jobs. So,
[23:03] in prompting, I'm going to instruct my agent to use Lake Formation Connect,
[23:21] conso- keep the default, and then here, oops, I'm I'm going to instruct it to consolidate and not keep the default. And then I'm going to instruct it to use email notifications.
[23:37] And so outside of the MCP server, the agent is handling all of this back and forth between the user, and it's understood that it needs to actually ask more questions to guide further guide this migration.
[23:52] So here you can see it's asking me how it should actually get to that metadata. Uh I have two options. I can either provide it as a file or I can give it direct access to query where that metadata lives. And it's also asking me how uh I would like to notify where where uh those
[24:05] those notifications should go. So I'm going to
[24:38] this export that I took from that SQL server, paste it into my agent session, and let's see what it does. Um So what it's going to inspect is that CSV file.
[24:50] It's going to understand what metadata it's iterating over, and then it's going to drive that metadata into uh manage ingestion pipeline definition that can just be run without the metadata driven pattern.
[25:45] The contents of the CSV is the metadata that I'm loading in ADF to drive the ingestion using iteration. So in this case we're we're going to load that and drive it into the definition of the manage ingestion pipeline so that ingestion pipeline just runs one time and you're
[25:57] not reusing compute over and over again to iterate over different tables.
[26:19] Looks like Genie is quite slow today. Sure. So the So the the source documentation all the pipeline definitions in JSON are actually coming from a workspace folder.
[26:31] The workspace folder has the RM JSON exported to GitHub and then I just create a Git folder. These are new skills. Sorry, can you repeat the question so that folks can
[26:44] Sure. Sure. So the question was if the skills are sort of built into Genie code out of the box or if they need to be installed. The answer is they they do need to be installed and we're working on a way to package them and and bring them to you today.
[26:58] So, let's stop this and see if we can resume it.
[27:15] It's reading them directly from the ARM JSON. Yeah. So, we pull in the data set ARM JSON, linked service ARM JSON, uh activity ARM JSON, and uh trigger ARM JSON and we use all those inputs to
[27:27] create a converted pipeline. Yes. >> I have a very quick one. So, if we have a situation like if any of the
[27:40] connector is not available which we have used in the ADF, will they going to find the another one for this particular question? >> Yeah, great question. So, the question was if there's a connector in ADF that
[27:54] you're using for ingestion and that connector is not available, what will we do? How will we handle the situation? The answer is today, we'll create some type of placeholder notebook that you can then
[28:06] customize manually and we'll notify you that that's sort of a downstream step that you need to take before you deploy.
[28:33] Let's try this one more time. >> While you're doing that, um to I just recalled you were asking about what happens if the connector or let's say there's some sort of integration that doesn't exist. >> I have a simple example like we had this issue with the SharePoint sites.
[28:45] >> Yeah. >> Because it was not supported earlier. So, we used the idea for the SharePoint connector. >> I see. I see. So, um the the person here is talking about SharePoint um as an example here. And uh so,
[28:58] obviously, if we have connectors available, we would use them. But, one of the things that we're also adding is a framework to add these integrations into jobs. And again, I keep punting those uh conversations to the jobs talk
[29:11] that's going to happen a little bit later today. And we are going to put skills with Engine E so that you could custom write your own integration if you need to.
[29:27] >> Okay. Uh it took us two prompts. Uh that's never happened before. But, uh we have the final output for this complex pipeline. Let me show you what that looks like.
[29:39] So, if I navigate back, you can now see I have this pipeline complex in my output directory. You can see I have that same metadata folder. I have a root-level databricks.yaml file that's controlling bundle deployment across
[29:52] different targets. And in my resources folder, you'll see that I not only created YAML for the for the ingestion pipeline. Let me close Engine E. I not only created a YAML for
[30:04] the job um that defines the the migrated pipeline complex, but I actually created YAML that defines a managed ingestion pipeline with Lake Flow Connect. And you can see now that I've driven
[30:16] through that CSV, the source schema and source table mapping that came from my metadata table.
[30:31] So, in order to deploy this, we're going to have to make a few small changes. Mostly, we'll have to customize the target catalog and target schema cuz that's default. So, I'm just going to come in, copy my
[30:43] catalog, paste it here. And the second thing we'll have to do, and you you'll probably
[30:55] notice that in the source, we also created a setup folder. And that setup folder has generated notebooks that will create any prerequisites required to run things like manage ingestion pipelines. In this
[31:07] case, we require a Unity catalog connection. So, this notebook is automatically generated by our code generator, and it creates that connection for me. So, I'm just going to quickly run this manually.
[31:26] I created my connection, and now I'm ready to deploy. So, let's go back to this guy, open this in the bundle editor.
[31:57] It's going to validate my bundle, and then once it's ready, we can just deploy it.
[32:09] Yes. Yeah, so uh the advantage to dabs is a few things. First of all, um you can version control these, and you can deploy them to a number of different environments as they mature. The second is that uh you have manual control to
[32:21] edit all the configuration. You can also run things like, you know, Genie over the configuration files to identify any gaps or anything like that. >> There was a question as well uh somewhere in the back. Yeah.
[32:52] Yeah, so the question is, you know, ADF obviously has uh sensitive information in it, credentials, things like that. They may be stored in a key vault, they may be backed by uh some type of secrets manager. And the
[33:04] question is, how do we get information out of those secrets managers? The answer is, we don't. Uh we sort of rely on you maintaining uh secrets in your preferred secrets manager, but I will say um
[33:16] in addition to the setup script that created the connection, we also have a setup script for creating a secret backed uh scope in Databricks, where you can store all that credentials and and access it directly from your workspace.
[33:37] Looks like my >> need to deploy? >> Yes, I need to deploy.
[33:55] So, now I've deployed and I've deployed both a managed ingestion pipeline that ingest data directly from SQL Server uh using all the metadata from my metadata table.
[34:08] And I've deployed a job. And the nice thing is that job definition is actually quite simple now. So,
[34:20] we will run the managed ingestion pipeline, then we'll run the one downstream or few downstream activities, and you can even see that in the configuration of these
[34:37] >> Way down here. I've consolidated the notifications into something that's supported directly in the Lake Formation Jobs product. So, we no longer need to call web activities or you know, execute post requests or anything like that. We can just directly notify via email as
[34:49] part of the task definition.
[35:08] So, once you have performed a migration with this tooling, the next step is to validate the results. We have a few different mechanisms that you can use to get started with validation. Um the first is we have this report.csv
[35:21] that shows the overall coverage of the tool and the estimated complexity of the pipeline that you've just migrated. The second is that we have a setup file that documents exactly which prerequisites need to be satisfied before you can deploy this, validate it,
[35:35] and migrate it to production. Now, there was an earlier question about whether this is sort of uh default skill
[35:48] that's available in Genie code. The answer to that is that the preferred path to, you know, making this available to you will be to push it into our Lake Bridge product. Uh we will expose some
[36:00] Lake Bridge skills for Genie code, and those skills will help you do end-to-end migrations not only of orchestration, but also of code and and sequel.
[36:20] So, this talk is just about migration of jobs, but there are a number of other talks related to both jobs and other Lake Flow topics and migrations happening at at Dice this week. So, we encourage you to check out all of these talks
[36:32] um as you have availability and time. And please get in touch with us. You can scan the QR code. There's an interest form for this tooling and for migrations. We would like to engage with
[36:46] you, get your feedback, and help understand your migration use cases.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.