Classic to Serverless: An Agentic Migration Playbook for Python and Scala

Summary

An agentic migration tool inspects code, classifies incompatibilities, and proposes fixes to automate the migration of notebooks, jobs, and pipelines from classic to serverless compute on Databricks.
Serverless compute eliminates idle costs, provides auto-scaling, and removes the need to manage Spark configurations and init scripts, allowing teams to focus on shipping rather than managing infrastructure.
Concrete migration playbooks for both Python and Scala/JAR workloads are demonstrated, including A/B testing validation steps and fleet-wide tools for platform administrators managing large job inventories.

Classic to Serverless: An Agentic Migration Playbook for Python and Scala

Moving real workloads from classic to serverless compute can mean dozens of small changes: Spark configs that no longer apply, init scripts that need rethinking, Scala and JAR jobs, and pipelines tied to specific runtime versions. Doing this by hand across hundreds of Jobs and Notebooks does not scale. this video is a hands-on practitioner's guide to migrating Notebooks, Jobs, and Pipelines to serverless, with a focus on the new agentic tooling that automates most of the work. We walk through concrete examples covering Python and Scala/JAR workloads, show how the migration agent inspects code, classifies incompatibilities, and proposes fixes, and demonstrate how to validate the result before promoting the change. Attendees leave with a repeatable migration playbook, a clear view of what is fully automated today versus where a human stays in the loop, and a sense of which workloads are ready to move now.
Talk By: Achille Negrier, Product Manager, Databricks ; Prashanth Babu Velanati Venkata, Product Specialist, Databricks ;

Chapters

FAQs

What is the Databricks agentic migration tool for serverless compute?

The agentic migration tool is an automated workflow where an AI agent inspects existing notebooks and jobs, identifies incompatibilities with serverless compute, and proposes specific code fixes. This reduces the manual effort that has historically prevented teams from moving off classic compute despite the clear cost and operational benefits.

What are the main advantages of serverless over classic compute on Databricks?

Serverless compute eliminates idle costs by only charging for actual usage, provides automatic scaling, and handles infrastructure concerns like Spark configuration management. Teams can focus on shipping data products rather than managing cluster lifecycles and init scripts.

Does the migration agent handle Scala and Java jobs?

Yes — this video demonstrates the migration agent handling both Python notebooks and Scala/Java JAR-based jobs, classifying incompatibilities specific to each language. The agent proposes targeted fixes and the workflow includes validation steps before any change is promoted to production.

How do I validate a serverless migration before going to production?

The migration playbook recommends using A/B testing to validate migrated workloads against their classic equivalents before promoting changes to production. Fleet-wide migration tools are also available for platform administrators who need to assess and manage large numbers of jobs simultaneously.

Full transcript

[00:08] Thanks for joining this session. Uh this is a talk on classic to serless like how you can use agentic migration for for migrating from classic to serless for both Python and Scala. So we'll be doing a bit of live demos also but before that
[00:21] I'm Prashant uh product specialist at data bricks. I'm u we have Ashel also here. >> I'm a product manager. >> Yeah. Uh so this is a uh QR code if you scan it you'll find all the summit
[00:33] sessions on serless uh serless job serless workflow serless uh pipelines etc. So and also it has a set of private preview features and upcoming features which are going to go live like df.casheach and migration agent and a
[00:47] lot of number of lot number of uh private preview features. Please please if you're interested please look take the uh take a look at the form Google form and probably uh fill it and send it
[00:59] to us. Thank you. Thank you very much. Uh moving forward uh this is a brief very high level brief agenda like first we'll talk about why serless second why are migrations hard and third we'll be
[01:12] doing uh a migration agent demo that will be done by me and Ashel will be covering uh Scala and Java. Basically those are the two live demos we have and then the last step in this agenda is about road map like basically what's
[01:26] coming next for agentic migration. So uh without much further ado I will move forward. Um this is a very famous meme like there are two sets of people on the
[01:39] the person on the right is having is seeing wonderful view like it's a sunset and the person on the right is probably not able to see much and if you correlate this I'm I'm correlating this
[01:51] saying the person on the left is on serless has been utilizing and getting the value out of serless but the person on on right is still on classic and the reason why he's on classic is not
[02:03] because he doesn't understand sless. It's more like the effort to migrate from classic to sless. That's the major pain point he or she has. So just talk explaining a bit more it's exactly the
[02:15] same uh road but very different view for two sets of people there and uh just digging digging more shipping and not thinking about compute is the person on
[02:27] the on the left and unfortunately the person on the right is still wants to be on serless but can't justify the migration effort. Um it's not just uh the person on the left but there are lots of customers also who are on
[02:40] serverless job serless pipelines and serless notebooks. Um we had had multiple sessions like uh Delot did a session yesterday and supercell also did a session yesterday and CL also was doing session. So if you are looking for
[02:53] some sessions on serless from customers our customers please uh take a look at them. Um just to highlight this point what keeps you on classic isn't serless but it's the effort to migrate from
[03:06] classic to serless. So that's about I'm setting a session context. Now um why serless will be taken over by Ashel. >> Um yeah thanks a lot pashant. what is
[03:19] the kind of feedback we hear from customers using um classic compute like yeah serverless sounds good but why why do I actually need that I mean these are some of the points that we hear from customers often like hey uh I love data
[03:33] bricks but my clusters are really slow to start so every time I want to do something or test something it takes me five minutes to actually do my work or I actually end up paying a lot of money for idle compute like I leave a cluster
[03:46] running I mean I do this all the time in our staging environment ments or I'm testing a job and I'm not quite able to size it the right way. I pick the wrong executor size, the wrong driver memory, things like that. And then you end up
[03:58] paying a lot for resources you don't actually use. Also, if you kind of think about classic compute, you need to manage a lot of things like the sizing of the VMs, the incripts, the uh kind of cloud provider
[04:11] codas for the instances you use. there's like a lot of undifferiated work that is not actually your own business logic that you're doing to be able to just run your Spark jobs. Also on on classic
[04:24] compute um you might have noticed there's a tab that's DBR versions and we ship a lot of our performance improvements and new features through DBRs. But often moving from an older DBR to a newer DBR requires a lot of work
[04:38] like it's something you need to put on your road map. is something you need to plan in your sprints and it's kind of unclear what value you get from some of these. Um some of our customers tell us like hey 20% of my time is spent on
[04:52] upgrading DBRs like uh it's a lot of kind of work just to keep the platform up to date. Um yeah so this is why we built serverless compute the kind of core
[05:05] principle of it is it should be simple performant maintenancefree. So on classic you need to tune infernops to achieve a certain cost and performance profile. On serverless the idea is you just tell us like hey I really want this
[05:19] job to be fast then you run it in performance mode or hey this job is not quite um so timesensitive and I care more about having a lower cost profile in which case you can use standard mode.
[05:32] In terms of performance, um on classic you have four to six minute startup times and you often have to pay or you have to pay the cloud provider for the instances. On serverless we have uh the
[05:44] concept of a warm pool. So we have this uh pool of compute that's pre-warmed and then if you need it for a job or notebook task we can just um allocate you for that and that's kind of startup
[05:57] times or more in the seconds
[06:10] allocate compute for your jobs. Also, serverless is built around Photon. So, in our usage data, uh we see that customers uh job complete about twice as fast. Um in terms of it being
[06:24] maintenancefree, the infra selection on classic is is very much manual, right? You pick the clusters in serverless is fully automatic. So we pick the right driver executor size. We pick the the
[06:36] right number of nodes for your job to complete. And actually it's our most stable compute product. So one is I don't need to manage it and the fact that I don't need to manage it. I don't make mistakes whilst managing it. So it
[06:48] helps um our customers not get paged in the middle of the night if a job is failing. Uh in terms of DBR upgrades, uh Prashant will also talk a little bit about why the architecture of serverless
[07:01] enables us to upgrade the servers in the back end without breaking your code. There's often a lot of questions about this, but basically on on serverless, we're able to ship server side changes so your client code stays the same, but
[07:14] on the server, we can keep giving you uh performance improvements. Um yeah, so serverless is kind of auto optimizing and and it us having that control over the underlying
[07:26] infrastructure enables us to pick the right compute for a job. Um in terms of a picture, this is our old cluster creation page. I appreciate the font size. It's really small. Um I the knobs,
[07:40] there's so many buttons. I mean, you kind of need a PhD to be able to get it right. And I mean if you're running thousands of jobs, it's very likely your team is not at all efficiently configuring this page. Um on serverless
[07:54] you just have oh that little button and then you can this is for jobs. You can tell us um please run this fast with the kind of worm pool or please uh don't I I prefer it to be cheap.
[08:08] Um often when we speak about serverless some customers get a bit worried like hey are you going to deprecate my classic clusters like uh are these uh here to stick around? They're not going anywhere. Uh classic clusters are our
[08:22] most kind of compatible compute product. Um we have a lot of customers have some customizations to the spark engines and things that we cannot currently support on a shared infra or are using a lot of like legacy
[08:36] code that we is not supported on serverless. Um for those use cases classic is here to stay although what we see is more often than not there's not
[08:48] um hard blockers to use serverless. It's more like a lot of little changes that you need to make um that make migrations kind of painful. Uh and with that I will
[09:01] let Pashant elaborate a bit. >> Yeah. So um Ashil gave a gave a small brief intro on what serless is and uh extending the same thing like why are migrations hard? That's what I'm going
[09:14] to touch upon like and how big is this wall and how can we break that's about uh this particular session. So uh Ashel was mentioning about what is happening in Sless like why do we need to make the changes to to code um on the right
[09:31] yeah on the right is uh classic spark architecture on the left is the serless architecture in the classic architecture you see everything is on in driver like a driver is completely overloaded with a lot of Python code packages and
[09:44] everything so basically everything is on uh driver that makes it impossible for us to update the DBS. Now if you move to uh the new world serless world, it's completely decoupled because of spark
[09:56] connect and all the code I mean all the code is still on ripple. Now the cluster is completely decoupled and that allows us to automatically update the DBR without customers doing anything on
[10:10] their ripple code or application code. So that's what is the beauty of serless and this is the reason why we have to make some code changes uh some code changes which are not compatible. Um so so I will be digging more into u
[10:25] migration agent now. So there are two ways of uh approaching this problem. One within the platform you have uh geni code. So that's what we are going to discuss now in the in this session now about migrating notebooks and jobs in
[10:38] the context. So basically you are in the notebook in in the platform and you do /mp compute I I will showcase that you do /mp compute in the gen panel and it automatically updates that particular notebook's code to be compatible with
[10:50] serless. So and it will also explain what are the features what are the uh what are the what are the snippets which is which uh genode is rewriting to be compatible with serless at the same time
[11:03] there is a I mean you are on java or uh java scala scala code which you need to deploy as jars for that also we we have covered like uh there is a uh a skill
[11:15] which could run on codeex cloud code or any any of your uh llm agent so that will help you to migrate uh logic in your in your uh environment. So those are the two things like and to
[11:28] extend this topic like uh there are four stages of uh migrating. The first one is uh detect like identify what are the jobs and uh uh how many jobs are there
[11:41] what are the jobs which could be easily compatible to serless or what are the changes which are required to be uh compatible to serless that's what diagnose will look at understand what changes need to be done and so on and so
[11:53] forth and finally what exactly the changes which uh Genie code can do is about uh refactor and when I say genie code it it also is applicable uh data brickskll. So basically uh genu/ uh g
[12:07] the skill will rewrite the code and wait for your approval. So that's where the approval part. So you are still in control completely in control of the code changes what either geni code is
[12:19] doing or uh or the skill is ultimately you decide whether you want to move forward or you want to uh you want you don't want to move forward. Um all four stages basically are compatible gene
[12:31] genie code compatible and genic code driven and you just decide whether to move move over or not. Um to do a small demo I'm going to introduce a use case. U meet Marcus a data engineer who is
[12:45] taking care of a particular uh use case and a particular maintenance of a particular use case. Now he has a uh notebook which does daily refresh of uh of their products. Now today he got a
[12:58] requirement to make a small modification, a change requirement, a small modification to one of the notebooks. He just needs to do a small very small edit to the notebook. But the problem is he's still waiting for the
[13:11] for his classic cluster to start. So that's we all have been there like we all have waited for classic cluster to come up for at least five five minutes a minimum of five minutes. Um so why not
[13:24] use serless notebooks uh serless interactive but he doesn't know if his notebook is serless compatible. Now that's what we'll do uh the demo live demo on serless notebooks. Um so let me
[13:39] jump to a notebook but before that I would like to uh start with the migration. So there is a notebook and I created a clone of it just to showcase uh what is happening and let me change
[13:52] it to 17.3 which is my cluster and I will clear everything here. So what you do is I will start this so
[14:04] that we can move forward and uh continue. So I I'm just doing the slash command slashmp compute and then migrate to serless
[14:19] uh because the because geni code understands the context it has the notebook and it understands what to do. So I'm just pressing uh migrators that's what that's all I did in this notebook. We have a ready made ready notebook
[14:31] which has some catalog some information etc. I will walk through that. uh but before that I just wanted to start this so that we can get on with it. So to show more about this particular job uh
[14:43] particular work workload so I wanted to show some more things like this is nonphoton cluster and a 17.3 LTS with a single node uh single node as in uh just
[14:56] driver node. That's what is this environment and also couple of other things like it is in a single user uh mode which is dedicated mode and it also has init scripts like basically whatever you can think of a regular uh one of our
[15:10] regular customer workloads. So yeah and uh this notebook what it does is it it installs a library in this case it is tablet and it also has a lot of other
[15:24] libraries which are part of init script like human eyes and so on and so forth. So there are so many uh libraries and also uh an init script uh etc. So this is a typical notebook and then uh I do
[15:38] some some code some code here. So there is a lot of code here. Uh just to showcase some uh there is normally as always a lot of our customers use debuts and also copying files and then there is
[15:52] uh yeah there is uh some other some other changes etc etc. So we have all this and also there is sea.ro broadcast which is not supported by uh by standard standard
[16:04] access mode and a lot more other things technically are not supported by notebooks uh by by uh serless. So that's what this particular tool geni code is going to migrate now to to show you that
[16:17] like you can see here it started looking at the entire notebook try to understand what changes changes it has to make and it will also emit continuously what it is trying to do for for that notebook
[16:30] and at the end of it it will showcase completely visually what changes it has made and what were the previous changes etc etc so so all that is uh it is trying to show But before we go there,
[16:42] there are some more things like for example in this code there is RDDs also are used. Uh if I move to uh if I move to RDD, yeah, RDD is again uh is an
[16:54] age-old code which logically we shouldn't be using anymore. Probably move forward to data frames if possible. So all the all this code is technically not compatible to be run on uh uh standard mode and and spark spark
[17:08] connect. So what uh the genie code is trying to do here is uh it will try to run the uh modify this code and showcase what all changes it is trying to do and
[17:21] why it is making the change also it will showcase now uh let it complete. Okay. Yeah, it is trying to uh create like as I said it looks it looked at uh the environment and then yeah so it it
[17:35] looked at the environment and identified there is an init script and there is it found the DBI version of of the previous uh of the notebook and all the information like uh there is spark conf which are which is there in the cluster
[17:48] and there is uh I mean there are no cluster libraries but there is init script anyways. Yeah. So all those things it is able to understand and then it will start writing uh updating this notebook in place the particular
[18:01] notebook which I cloned in just in front of you and uh modify and showcase all the changes including removing this pip install instead moving towards uh uh environment libraries environment
[18:14] version. So that's what uh this is trying to do. Yeah, you can continuously see what it is trying to do uh right here uh in the in the in the console. Yeah. So without moving from the platform the entire notebook is being
[18:28] modified by uh Geni code here um yeah there is RD code it identified that and it is continuously looking at the all the code and modifying and trying to
[18:40] understand which lines need to be modified uh to be made to be for this notebook to be compatible with serless. Uh probably it might take a probably less than a minute to complete this. We
[18:53] are almost at the end I guess. Yeah. Yep. Yeah. So it found uh the the best compatible version environment version for this particular notebook is uh v4 and it also migrated all the init script
[19:07] related libraries to a requirements txt file which will be bundled into environments which it is going to tell me also anyways. Yeah, it did some networking related uh checks as well as some sparkcoms and then code changes.
[19:20] Yeah, you can see all the code changes like you can skim through uh jump to any line look at what the code changes are and why those code changes are made and all those things it is trying to tell
[19:32] you right here like it deleted pip install I mean it is showing here uh deleted pip install uh tablet because that's not required it should ideally go sit in the environments panel so all
[19:44] those things it is able to identify understand and ultimately you are still the owner of all the changes which uh we are trying to make here. For example, I can click on apply all. All these changes will be automatically applied
[19:57] into the notebook. Like I can click on this and I'm done. As before, as I said, we had an init script. Now that init script will have uh some libraries. So that's what this should go as ideally as
[20:11] a base environment. Now if I click on this, it it also explained that as well like there is a base environment file. There is an environment. ML file which uh Genie created. It even it is even
[20:24] telling you the instructions like open environments side panel go to the base environments up select the environments uh ML file and apply it. So if I go here
[20:36] click on custom and go here bundle files. Yeah.
[20:51] environment. I click on okay and then apply for tablet as I was saying was in the as the first cell in this notebook. Now it moved to the dependencies. So each each
[21:03] user will have their own libraries uh which they would be installing in the notebook. Instead of installing in the notebook, what we are saying is use it in the environments panel. So they are available for caching and etc. So that's what uh this is doing. Like one last
[21:16] time I will show uh the environment yl file here. Yeah. So this will be the uh the requirements.txt file where all the uh
[21:28] information for those libraries will be there. So and I will click on run all the moment this uh this is ready. So that's the end to end demo of migrating from classic to serverless of a notebook
[21:43] which you have right in the platform. So we are doing everything in the platform without leaving the platform and modifying the code in place. Uh like you can see pip install has been removed and so to se um SC.rocast and RDD and all
[21:58] those changes have all been modified by uh Genie here. So this is and also sparkcoms and so on and so forth like a lot of changes which it has done are all listed here.
[22:11] So that's about the demo. Uh let me run it run all probably it will run all that should be okay. Yeah. So we are done with the with the demo and just to uh
[22:23] give the summary of what we saw. Um slashmp compute is a slash command right in the platform in gen code and you trigger it for the notebook or for the
[22:35] job or whatever whatever for the job and there is no separate tool required no wizard nothing so basically you're doing just class compute and everything is taken care uh taken care by looking at the context gen is looking at the context etc and it it is we saw it tried
[22:49] to reason across a lot of surfaces like environments uh init scripts and also libraries and spark counts and uh sit or broadcast and basically all the code which is not compatible with the spark
[23:01] spark connect and serless standard mode. Um at the end of it you still are in control completely in control each and every change are all the changes you can see you can review and finalize only
[23:14] then you can apply basically I you can see you saw me applying all so that's where uh it is genie is helping you to migrate seamlessly from classic to serless uh that's about my demo I will
[23:27] pass it on to Ashel now to do the scholar demo >> yeah thanks pashant So obviously this looks great like hey I have a Python notebook genie can I can do a slash
[23:39] command in the in the sidebar that seems obvious but what if I use cloth code to write my jobs like I use cursor um I use any other things outside the platform
[23:51] what if my code is actually not a Python notebook but I have very large production jobs that are written in Scala and Java um obviously rewriting those into Python notebooks is not something my team would want or I mean
[24:04] there's many reasons why uh you might not want to do your migration from a notebook or you might want to not migrate a Scala and Java workload um to Python. So uh this year we recently
[24:19] launched u Scala and Java jars on serverless compute. So part of the making migrations easier is also expanding the API surface area with support. So Scala and Java was we heard
[24:31] was a big pain point. I will be very brief on this but you basically get the same experience you have for Python is simple performance maintenance-free. You don't need to provision compute. You get a fast startup times the versionless
[24:43] upgrades. If you have new workloads um we have some starter templates uh to make it really easy to get started. So data bricks bundle in it uh it has everything set up so you can work from
[24:55] your ID. Um in terms of requirements I will be really brief on this but uh serverless is like a big change uh so it's Scala
[25:08] and Java on serverless built around spark 4 um which brings a lot of things like Scala version changes to 213 and JDK17 and many of the things Pashon showed like RDDDs that are not Spark
[25:22] connect compatible APIs and are supported also incripts etc. So, let me switch into my laptop.
[25:50] So, um here I have a Spark job that's running on a schedule. Um, if I look at the metrics of this job, it's actually kind of frustrating because the CPUization is super low. Uh, I just picked a random cluster size. I couldn't
[26:02] really be bothered with a really long form to figure it out. Um, so about I'm paying 70% of this cluster is underutilized. So I'm paying the cloud provider for an EC2 instance that I don't really need. Um, the memory, same
[26:17] story, really overprovisioned. Um, and it's running on a schedule. So I'm actually paying I'm wasting these resources every single day. Also this job um if I look at the time
[26:29] it takes it's a relatively short job like it only takes 8 minutes to run to do these aggregations but of these eight minutes I'm paying it takes what this morning I ran it uh it
[26:42] took six minutes to boot up. So twothirds of the time is spent uh paying the cloud provider for an instance that I'm not um even using and then my code runs for 2 minutes and then it shuts
[26:55] down. So this made uh made it a really great candidate to actually migrate it to performance optimize where hey this should just start up and go. Um but so
[27:08] actually let's take a little look at what the project looks like. Um, my team likes to productionize things using data bricks automation bundles. Um, this is
[27:21] not a requirement by the way, but I just don't like to click around in the UI. What automation bundles do, they allow you to define your job as a YAML file. So here, my job is called big house
[27:33] revenue ETL. Uh, if it fails, please send me an email. It has a couple tags. Um it has a main class that is called from my drawer and then it has a cluster
[27:46] config configured. So uh it's a fixed number of workers. It's running an old DBR. So it's basically a year behind on the latest performance improvements. Um
[27:58] so again wasting a lot of money on using a slower slower software and it's also using any script. If we look at this job, um the business logic is really simple. Uh it takes two
[28:12] tables, the transactions table, the franchises table, it uh joins them together, does a little data quality check that someone else wrote a while ago that for some reason uses an RDD.
[28:25] uh it does some aggregations, looks at revenue, uh the order count, and then ranks each country every day like, hey, what were the countries uh that perform best?
[28:38] I'm going to be really brief on this because you might not all have Scala workloads, but basically I have a build file uh that's compiled against the wrong Scala version, Java version, and a
[28:51] set of dependencies someone else has picked in the past. um which are not kind of up to date with the latest on serverless. So how can I easily migrate this over with
[29:11] sorry writing with one hand tools. Okay. So what we did was uh if you have the data CLI installed, you can use data
[29:23] bricks AI tools which is a set of pre-created skills. They're also available in a repo so you can just read them and modify them. Um but it's a really nice way to get started. We have some testing internally to make sure
[29:36] that they work well with the platform. So they understand like what Scala version uh serverless requires. What are the dependencies we give how you should actually test different things like
[29:49] don't just uh switch over a job in production but actually test that the data and things are equal. you you get to decide essentially where you want to deploy the uh the scale like do I only
[30:02] want it for this one repo or do I want it available globally across all of my coding agents currently we support cloud code cursor codeex I mean most of the major ones uh I've already done the
[30:16] install so I will not do that so here I have um my terminal we're running cloud code but obviously it could be any other coding agent And I will ask it can
[30:31] you
[30:47] by the way this is a live demo. So and AI agents are very undeterministic. So let's hopefully it goes well. But basically uh it picks up the AI skill and the AI skill gives the agent like a playbook of how do you actually do a
[31:00] migration. The first similarity to Genie it starts by understanding what the project looks like what does the job look like is it uh in this case it will recognize hey it's a scala job
[31:12] it reads the the YAML file it reads the incripts and it quickly picks up on the incompatibilities I mentioned like the code incompatibilities also the data
[31:25] access incompatibilities like hey I'm using this old RDD code Um, and I'm writing like a kind of log file to DBFS. Both of these are not no longer
[31:37] supported on serverless because they're legacy concept. And I'm using an incript to install dependencies. So what the agent then does is it kind of makes the changes it thinks that are
[31:49] right. um like the from Spark context to data set APIs it updates also the build file so it knows the skill tells the agent like
[32:03] what Scala version you need to use etc and the nice thing about this is without the skill what you'd notice is agent would just keep looping over it will like try hey I will pick a random
[32:15] version I will deploy it fails I will read the error message and it will just keep doing this over and over again and it actually takes a long time. What a skill does it can follow that like exact right instructions.
[32:28] Uh and for example, this is a good example here. Uh this is a login library I added. It's the wrong version. Sometimes if you bundle the wrong things um your code breaks in production. So in
[32:40] this case the agent knows oh you should bundle against 210 which is what we provide. It updates also the build file uh to to use the right versions. And off it goes. It deletes all of these
[32:53] like cluster configurations because in serverless we set it automatically. What is it doing now? Ah, and this is where it's interesting.
[33:05] So it read the table um that serverless that the classic cluster was producing to kind of snapshot it of like hey what is the output of this table? like yeah you're
[33:17] making all these code changes but how do I know that the job will produce the same data right like my maybe serverless just deletes half my transformations and it passes but or sorry the agent deletes half my transformation and the job runs
[33:31] but it no longer produces the same result. So it kind of take took a snapshot of the data on classic uh and now it's deploying and running the job on like a kind of cloned copy. So it's
[33:45] able to then compare an AB test.
[34:00] ah you can see my job already completed. Uh it took a minute so it actually ran a lot faster. Um I don't took a couple seconds for it to boot up. Um I don't have to think at all about what the cluster config is here. So I'm not
[34:12] wasting that 70 or 80% of CPU and memory usage. Uh and what it does it then runs using the
[34:25] skill uh a kind of command of hey how many rows were produced uh by the job or the rows and the schema equivalent is there any difference and this kind of gives me the confidence of like hey um
[34:39] my job brand is producing the same data and then you can audit the changes and then manually decide of whether or not you want to promote this uh further
[34:59] yeah, so what are the changes the agents did here? It changed the build and config file. It did the API rewrites and the data access kind of changes from DBFS to volume. Um I'll hand over to
[35:11] Pashant to talk a bit about what's coming next. >> Thank you. Thanks a lot for awesome demo. Um what next I mean what's a road map of uh what we have what we have discussed is uh what we are going to
[35:23] talk now for in few minutes. Um what we saw is one in in the platform genie in the panel/computee uh if you're in the platform and if you're using your local environment id
[35:35] environment or uh vs code or anything for that matter uh you use data bricks I tools installed. So this is for the practitioners. Now imagine there are hundreds and thousands of jobs for example and that implies you're doing a
[35:48] fleetwide changes and implies admins will come into the picture where admins will look at workspacewide readiness and what workloads could be easily migrated from classic to serless and also what
[36:02] are the bulk migrations and etc etc. So that requires a lot of coordination between job owners, the uh job authors, job developers and the admins. Um what we are going to do uh in the near future
[36:15] is provide in the platform tool basically which tells you like there are some some uh some titles some headers here which say serless ready serless
[36:27] what is the current situation and a lot of other things like if you see here uh serless ready is eligible now for example and also if you hover on the on the eligible now for example in this
[36:40] case uh whatever We just migrated it looked at initcript. It converted the init script in place and then switched to standard access mode. The job the workload notebook workload we were seeing was actually a single user if you
[36:54] remember the dedicated mode but it migrated genic code migrated from uh single user mode to standard access mode and also it rewrote the code which was incompatible for like spark and non-spark compatible and a lot of other
[37:08] code changes. So all that it is helping geni code is helping you to understand which code changes had to be done and it is also highlighting those code changes right in the platform. Um ultimately
[37:20] this is the goal like if you have the assume there's a job like in the previous tab I was talking about uh so nightly KPI plan KPI pipeline now this is exactly the same pipeline. Now what
[37:33] happens if you click on run uh and look at the migration. So it does a lot of checks all those checks and also spark conf etc etc and then you you are in control you say click continue analysis
[37:45] and simulate success path. So that's what uh Ashel was showing success path as in you do AB testing and so on and so forth. So all that is you're doing right in the platform and there is a 30-day roll back also. So basically you can do
[37:59] you can migrate a job guided handholded by the agent and once you are comfortable that everything is fine you move forward you move you migrate the code to uh sess
[38:11] um that's that's all about our talk uh I just wanted to highlight this slide again um these are all the talks on sless uh at at summit this year uh this year like we are here right after this
[38:23] there is another talk by one of the customers on serless like they migrate 6,000 pipelines from EMR to classic EMR to serless uh notebooks and jobs and pipelines and there are some more some
[38:35] more like uh other a lot of other sessions like again this QR code has all these summit sessions please please photograph this uh screenshot this this slide and more importantly uh that QR
[38:49] code also has all the private upcoming private preview features and if you're interested please give leave your email id we will reach out to you when the features go for private preview or public preview etc. Um basically the
[39:01] three takeaways from this let serverless do the job. Let let serverless do all the heavy lifting like whatever you were doing before migr uh configuring the cluster looking at right sizing the
[39:14] cluster config uh instance type and maintaining efficiency uh sufficient efficiency and etc etc all those let uh data bricks take care of it. Um it helps
[39:26] you to also optimize the infrastructure. Uh second if you are on notebooks use Genie I mean notebooks or jobs use Genie. Let uh Genie do the uh hard bits diagnose detect etc. And finally if
[39:40] you're using your your coding agents uh for utilize uh AI skills which Ashel was explaining. So basically let agents do all the work. Um I will leave with this.
[39:53] Spark is amazing, has been amazing for what now 12 years now already. Uh but managing it is hard. So let data bricks do it for you. And when I say let data bricks do it for you, it's not just on
[40:05] the cluster configs uh cluster configuration, cluster man cluster uh management and efficiency management etc etc. Even the code also like let Genie do the migration from classic to serless
[40:19] make the code Spark connect and standard access mode compatible. let data bridge do it for you. So that's our uh session. Thank you. Thank you very much for joining us today. Uh we are here for any
[40:31] any questions. Thank

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.