A Practitioner’s Guide to Databricks Serverless

Summary

• Serverless compute simplified: Databricks Serverless eliminates infrastructure management, enabling data engineers to focus purely on building robust pipelines with automatic scaling and cost optimization.
• Workflow orchestration: Serverless workflows simplify orchestration of diverse tasks—ingestion, transformation, dashboards—while integrating with the broader Databricks Data and AI platform ecosystem.
• Practitioner benefits: Learn how serverless environments accelerate time-to-insight, boost productivity and reduce operational overhead for teams managing data pipelines at any scale.

A Practitioner’s Guide to Databricks Serverless

Talk by Prashanth Babu Velanati Venkata, Product Specialist, Databricks
Databricks Serverless revolutionizes data engineering and analytics by eliminating the complexities of infrastructure management. This talk will provide an overview of this powerful serverless compute option, highlighting how it enables practitioners to focus solely on building robust data pipelines. We'll explore the core benefits, including automatic scaling, cost optimization and seamless integration with the Databricks ecosystem. Learn how serverless workflows simplify the orchestration of various data tasks, from ingestion to dashboards, ultimately accelerating time-to-insight and boosting productivity. this video is ideal for data engineers, data scientists and analysts looking to leverage the agility and efficiency of serverless computing in their data workflows.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Session introduction
• 2:08 Classic cluster setup
• 4:46 Serverless notebooks
• 6:58 Workflow enhancements
• 10:30 Budget policies demo
• 13:08 Versionless compute
• 15:06 Environment management
• 17:31 Serverless architecture
• 19:33 Workflow orchestration
• 21:42 Performance comparison
• 23:07 Cost governance
• 26:05 Databricks asset bundles
• 29:48 Azure Data Factory
• 32:28 Advanced workflow tasks
• 35:47 Advantages of serverless
• 37:47 Auto-optimizing infrastructure

FAQs

What is Databricks Serverless?

Serverless is a compute option that abstracts infrastructure management, automatically scaling resources up and down based on workload needs. This video shows how it streamlines data pipelines on the Databricks Data and AI platform.

How does Serverless reduce costs?

Serverless automatically scales resources to match demand, eliminating idle compute and over-provisioning. You pay only for the compute you actually use.

Can I orchestrate complex workflows with Serverless?

Yes, Databricks Serverless Workflows supports orchestrating notebooks, jobs, and other tasks with automatic dependencies and error handling for end-to-end data pipeline automation.

Is Serverless suitable for production workloads?

Absolutely. The video demonstrates how serverless handles production pipelines with automatic scaling, cost governance and integration with Unity Catalog for governance.

Full transcript

0:00 Session introduction

Hello. Hi. Hello everyone. Uh good evening and uh thank you very much for joining this session. I know it's very late in the evening and probably I'm the last I'm the person keeping you away going from going from here. So uh we'll try to be as short as possible like I might be taking around 30 to 40 minutes the usual regular time. uh but essentially that's that's what it will be like and if you have any questions maybe we'll keep it at the end and if you are not able to put some time for that probably I'll be around and you can ask uh you can come to me anytime so uh this session is on uh is basically explanation or a deep dive into what a practitioner can do with database serless and when I say serless this session will be more talking about serless workflows in general which This is where you orchestrate all your jobs uh pipelines, notebooks etc etc.
A lot of lot of things like PowerBI also more very recently. So this session is more about workflows and what you can do in workflows and more so what you can do with data bricks serverless essentially. So that's a uh high level overview of what the session is about and very brief intro about me. Um I'm a product specialist b of uh IMAI London to be very precise and I've been with data bricks for the past 7 years now and uh my main role is about uh on serverless I a product specialist on serless workflows to be very precise so um for a change I do not have any slides this is the only slide I have for today so what I will be doing is I will be opening up the platform database platform to showcase what you can do in database for platform with database serless workflows and also I will run through a brief uh overview on uh what you can do in serless notebooks also and then provision and then transition to serless workflows uh in in a seamless manner.
So as I said this is my only slide for today. So now this is database environment and uh this is what you see when you create when you launch database platform and probably create a notebook. So let me start creating uh a simple notebook our in fact we can take this simple notebook itself. Now this is a notebook which has a lot of cells and uh I'm not doing anything uh crazy here. I'm just simply listing a particular location.

2:08 Classic cluster setup

This is available in any any and every workspace uh directly coming from datab bridge data sets and I read that data sets uh the the data which is there in that folder uh with the in the perk format and then do some write it in delta format and then just display it. That's what I'm doing in this notebook. I mean this is the end of it. So there is limit. So this is a simple notebook and I have a uh companion notebook for this where I do some reporting and uh filtering.
So we'll start with this first. So as I said this notebook doesn't do anything uh too much. So I run through a couple of steps and then move forward with uh this is basically injection if you if you see it from that angle. The next one is for uh uh reporting and filtering. Now if you are in um the uh classic workflows. So let me go create a simple cluster. So this is how uh you go create a cluster create compute and then you have to select a whole list of things here like right from compute name to more importantly the DBR version and if you require Photon or not and also the worker node type and driver node type.
I mean driver node type is auto which picks it up from worker node and a lot of other things like uh even more advanced where you might have to specify what mode do you want like usually it is always standard mode in this case I'm choosing standard mode which is which was recently renamed to standard mode it was previously called shared security access mode so these are the things we need to do uh for creating a simple cluster now we create a cluster and we might have to wait for depending on the instance type and the number of instances and the cloud provider probably you're looking at waiting around 3 to 7 minutesishes approximately.
So that's what it will take for creating a new compute. Now that's in the classic world. Now if I go to the serverless world I right now I'm connected to a serless probably I will just terminate this. Now the only thing you can see here is it is simple uh notebook and it's not connected to any cluster right now at the moment. Now you have two options. One choose serverless or two you can create your own resource. So in this case I will choose serverless.

4:46 Serverless notebooks

Now look at the uh I mean you might have gotten used to uh classic clusters. So that's where if you see how seamlessly and how easily you can create a serless serverless cluster and you can connect to serless cluster in how much time that's what is the magic here like when I click on connect now you can see it is already starting that's because the VMs are coming from the worm poolool not we are not going all the way to the cloud provider and get getting the VMs provisioned by the cloud provider now if you see I'm already connected and I can start start coding with my uh start with my development.
So that's how that's that's so easily we are moving ahead and we can also uh continue with everything like you can see my my code already is running. So in this case I will just do clear everything so that I can show that it is really running right now and uh I will do a run all in this case. So let me do a run all uh run all below. Yeah. So let it run. So it it will start immediately as I said uh okay there is one cell which is already running.
So this is how fast it will be like if you are used to classic this is a completely different paradigm and it is much much more efficient because you're not waiting for the compute and you are not selecting thousand odd things like if I go to the if you have attended roland session just now it finished in the other room so this is what it was before uh on the left hand pane on the left hand side is uh is what you have to select in a classic cluster environment There are a whole lot of things as I mentioned like DBI version foot on on or off or instance type driver node worker node instance type and access mode and so on and so forth.
Now in the classic world there is nothing by default it is uh compute of serverless. So that's the only thing you need to do and I'm talking about uh notebooks here. So there is as extension we can um we can transition this pipeline directly to uh workflows also. So let me take one more step here to showcase what exactly we have in serverless. So in the right pane you can see a couple of options here. Um what I'm interested is in environments.

6:58 Workflow enhancements

This is a new concept in serverless. So this is only available in serless. If you see why it is I will be explaining now. So there is this accelerator which is more for if you are interested in some GPU processing uh GPU required processing probably you will choose A10. In my case, I do not need GPU, so I'm not choosing. And then there is memory. I mean the way uh serless works, it's it works on the concept of spark connect.
So it's not the same old classic spark architecture where is there is a driver and there are executor nodes. So now in the new world it is spark connect will be the default between what you execute and what is there in the driver and executor. So it's a completely different paradigm now. So that is the reason uh the concept is different uh to say so and assuming that you have uh pure python workload for example you're doing a lot of pandas related uh processing in that case probably I mean in the classic world you would have started a cluster say assume five to 10 nodes uh five as the minimum and 10 as the maximum worker nodes in that case you would be provisioning the entire cluster like five to 10 working worker nodes uh for that particular ular job which doesn't need p the cluster at all because it's all running on pandas.
Now in the uh serless world we will not create a driver and worker node at the entire cluster until unless you run a spark uh command. So if assuming I mean in this uh example I do have a lot of spark code but if if it was the case that I didn't have anything spark but it was only pure pandas we would we do not even we will not even create uh the cluster and uh by default we specify uh the standard uh memory as 16 GB.
So the repel which is executing this code through spark connect is having a uh memory of 16 GB. But if it is pandas obviously people will do uh dot to pandas I mean uh dot collect or whatever it's going to be huge. So in all those cases probably you might want to choose uh high memory which is 32GB. Uh so that's about memory and there is a new concept of budget policies. If you are used to uh classic world you might have seen cluster policies where you can specify mandate the specific DBR to be used foot on or off number of VMs etc. instance type and a whole lot of things which are possible in cluster policies.
So that's about uh classic world. Now if you move to serless world logically there are no clusters because clusters are running it's more on cores rather than clusters in serless world. So you will not see any clusters. The only thing you will see here is a simple one. Here it is. That's the only thing you can see. There is no cluster ID. There is nothing you can see here. Uh so for that we have a new concept called uh budget policies.
Now budget policies are uh pretty simple. Now if I if I go to my settings and uh go to compute uh compute and then I should jump to budget policies. So this is where you create budget policies. Now in this case I created a budget policy called budget policies demo. Um I can specify tags. So which is what you see in the UI also. So I can I created a budget policy demo and I created a project tag and its value of data engineering.

10:30 Budget policies demo

So this is how a budget policy uh is created. Um I mean I can edit it right now uh saying probably demo at dash and I will save it. So this is how I created a simple budget policy with a tag. Now if I go to my notebook, I can see what budget policy. So um let me refresh this because I created the budget policy just now. Um I should be able to see what budget policy I have and also what budget policy tags I might have uh I might be getting.
So you can see here I do have uh I did get the u demo of dice also as budget policy tag here. Now the importance of this uh is that this is useful for attribution like how many DPUs a particular user or a particular group of users have have consumed. All that information could be easily available in uh system tables. I will run through that simple uh query also and show that. So essentially this is how you run uh budget policies or run notebooks with a budget policy.
The other thing to note here is I might I can have like five or 10 budget policies assigned to me and at that point I can select whichever budget policy is applicable to me for that particular workload and run this workload. So and again please note that you can have any number of budget policies and you might have to choose which one is most adaptable for you are there is a probability of choosing default budget policy as well for uh serless clouds.
And now the concept of uh base environment. Now in the uh classic world there used to be init scripts where you specify the jar files, the environment variables and a whole lot of things. Coming back to serless again in serless there are no clusters. So there should be an intelligent way to do this. that intelligent way is uh environment and I can specify the environments here like whatever version for example um in the case of uh classic you'd be upgrading the DBRs constantly like um if you attended uh the session with uh he did talk about one specific thing uh on uh serverless uh yeah this is the slide I was looking for he talked about versionless where uh a lot of our customers are running a lot of are spending a lot of their time in manually upgrading DBRs and uh up to 20% of their time is spent only on upgrading.

13:08 Versionless compute

So I mean the SMMES highly knowledgeable skilled spark developers and machine learning engineers instead of spending time on productionizing their use cases they rather are spending their time in upgrading DBRs from 14. x to 15. x to 16. x pretty soon 17. x. So that's a pain and most of our customers uh said they're spending up to 20% of time and also it's usually months behind uh latex performance. So if if we move to serless here um in the last one year and one and a half years to be very precise we have done 18 uh deb upgrades and in that we moved from 14. x uh step by step to right now at 16.

4. So a a number of workloads which you upgraded as part of this one and a half years is 1. 3 billion workloads like each workload each notebook each job uh you see how many of our jobs we have right now there's the total we have upgraded is 1. 3 billion and the other thing to note here is uh the success rate uh the way versionless works I mean there is a wonderful talk by P uh tomorrow uh on serverless how versionless ensures serverless Please take a look at that session.

Please join that session if possible. Uh he's going to go into the weeds of how serless is versionless. Um and also there is one important key point here. Though we had done 1. 3 billion workload upgrades, the overall success rate is almost 100%, it's just 0002 is where we failed and there is better performance. Now I said a lot of things there. So to show what exactly it is uh let me open what is the release notes now this is the release notes you can see 15 14.

15:06 Environment management

3 all the way to 16. 4 so these many versions we upgraded in the uh last one and a half years and I'm I was talking about versionless uh and environments now my dependency for whatever reason I have dependency on Python 3. 10 10 for example in that case I would use environment version one now if I click on system environments you can see it is 310 so you choose environment version based on your requirements it doesn't need to be that data bricks is mandating DBR upgrade so you are you are being pushed to upgrade DBRs that's not the case anymore it's more like you are choosing your own environment version because you need uh something very specific now if I go to version two the Python version is different.
It is 310 311. So that's the difference like you upgrade your environment version based on your requirements. If for whatever reason you have a library which which depends on Python 3. 0, you'd be uh you'd be using environment version uh one. If it is version two, you move to uh version two. So uh going back again. So this is how you define versions here. Now there are multiple ways of uh installing libraries. You can do with pip install, percentage pip install that works the same way in uh serless as well.
But the better and the most uh important and efficient way of doing it is creating dependencies right in this notebook here. Like uh let me install say faker here and the only thing I need to do is this apply implies whatever I changed here they will all apply and the library is getting installed right now in this uh uh cluster right now in the in the serverless environment. The moment it is installed, it's available for me for use uh to use uh in this notebook and also when I upgrade when I push this pipeline to workflow even there it's available.
Um so this is all about uh how you can use environment. So this is the very important concept in the entire uh serverless versionless. Now if I if I go back to again uh slides from there is a section which talks about this. So this is what I was talking about the ripple VM. Uh so in the traditional classic spark architecture which is on the uh left hand side of this uh slide um everything is running in the driver node like your Python code your Python version all the libraries which you have installed everything is running in the driver.

17:31 Serverless architecture

So driver becomes a bottleneck and and the executors also are there. So in that case to upgrade the DBRs you'd have to manually update your application code and also any Python libraries which are uh bundled in that particular DBR. So you need to take care of all that. Now if you move to serverless the clearcut uh the difference here is you can see there is a ripple VM which is running standalone and it connects to the driver and executors through spark connect.
Now that's where the difference comes here. So because the way uh the Python libraries are are running in a ripple VM and not on the on the cluster so it makes it easy for us for data bricks to upgrade the DBR seamlessly. So that's how we have been doing uh a 1. 3 billion workload upgrades. So that's a lot about what we have been discussing. Now I did um install Faker. So Faker is already available. Now to show it is really available I will just download uh this IPython notebook.
I mean this is IPython version. So I will download this and uh let me open the IPython notebook. So here is where uh I started my yeah you can see I my dependency is already available in IPython notebook. So the moment we transition this pro this notebook to workflows it's available again there in the workflow. So to show that all that I will just move to uh create a schedule for this. Uh I just ran a notebook for this uh a job for this but I will just show probably live demo of uh serverless.

19:33 Workflow orchestration

So it's already uh choosing serverless. Um there is a subtle difference the way you run uh a notebook in serverless workflows. The difference being uh there is a performance mode which is called as performance mode implies I want to run my job now. So that's where you choose performance optimized. Now this uh tool tip talks about it. Performance optimized implies it will run as fast as possible and immediately. And there is another mode called standard mode which which has a startup delay of approximately 5 to 6 minutes and after that the job starts and uh it is cost beneficial to run it in standard mode.
I mean depending on your use case you might choose uh performance optimized mode or standard mode. Now in this case I will just choose performance optimize mode. Um and then I will go to the job. So if you have if you have already used serless you might you might be familiar with this. So essentially I have chosen performance optimized mode here. Uh let me go to the task tab and uh I have the normal injection task and then uh a I will be creating another task by name probably filtering or something like that filtering and choose the notebook.
So I'm done. So this is same way as what you've been doing with classic uh workflows as well. Now when I click on create task and because this is performance optimized mode the job should run immediately. Uh let me start start the run. The run will start immediately within around 30 seconds the job should start. Uh we let's let it run. So meanwhile I will showcase what I've done uh today afternoon. Uh so um so I had I had a job which was running uh let me go to the job.

21:42 Performance comparison

So this is a job I ran this job uh multiple times earlier today. So you can see there is I ran with parameters where um you can see serless compute uh parameter as serless and mode as standard and the other one is compute as uh serless and mode as performance optimized and the third one opt uh third option being um computers classic and uh there is no mode in classic so it's normal. So you see the duration of each of these jobs.
It's it's exactly the same job. The only difference being the way it is triggered. Uh when I say the way it is triggered, I meant uh is it performance optimized or is it standard mode or is it serverless or non-s serverless? I mean classic. So that's the only difference. Now you see the difference. Um standard mode is approximately exactly the same time. The duration of the job is exactly the same time as uh classic.
So that's the difference. Um though we are running in serverless we are being costefficient and at the same time uh the job is taking exactly the same time. Now performance optimized as I said the job starts immediately and try will try to run within no time and there are some limits but still it is much much better much faster option. I did a comparative analysis as well of uh these runs. Again, um if you go to the uh if you have access to system tables, there is a whole page of information about uh system tables.
Uh let me go to the most important uh billing dot usage. So this is the table which has all the information of which user has used what workflow has caused how many DBUs and how and a whole lot of information. Now I'm showing here a bit of information by joining two tables. One system. billing billing do usage uh here it is and the other being uh job runtime uh job run timeline so by adding by joining these two tables I get a view of the number of DPUs of a particular job and also the elapse time of that particular job now here it is the same jobs which we were discussing just now um performance optimized it's approximately 181 seconds and the number of DBC is 4454 four.

23:07 Cost governance

5 for example and uh standard mode is almost the same I mean if you see the uh elapse time it's almost the same as classic 527 to 531 and the number of DBUs now the difference here is in the classic world you would get a bill from datab bricks as well as your cloud provider in the case of serless you'll get one single bill that's from data bricks now I mean just try to imagine what exactly we are seeing here. Now the DBUS here is the whole bill like whatever is your uh rate on the cloud provider and also the region which you are running in.
If you multiply the value of uh data brick serverless there that's that will be your overall cost and in the case of uh classic probably you might have to uh multiply this with the region value I mean dollar DB value and then add the cloud provider cost also into this. That's how the total cost will be coming to you. So essentially I'm just trying to showcase the elapse time and the number of DBUs for a particular job with exactly the same code everything the same.
The only difference being the way it they are being triggered. So this is about how you would use uh system tables for running uh for trying to ensure and do the attribution. I mean there is a lot of information in system. billing billing do usage like for example let me run a simple code uh system. billing billing usage and uh yeah that's okay. Um so the moment I run this I will have the entire information of my partic of what who did what in this.
I mean basically I need to do order by and other things but essentially you get a whole lot of information from this and from this you also can look at each of the jobs uh attribution who did what when and what they were doing and a lot of other information like if you see the record ids Q names I mean this is all purpose all purpose as in uh the notebooks and there if someone adds some custom tags they will be visible here or else you will be having a lot of other information like uh usage metadata and usage quantity and so on and so forth.

26:05 Databricks asset bundles

So this is how you would be working with uh with serverless in the if you're trying to do any uh attribution this the system tables and a whole lot of dashboards and other things also are are available like if I uh just go to the usage dashboards this is the usage dashboard uh which you can import into a work into a account and you can choose there is an option to choose all the workspaces or you you might want to choose a specific workspace it to list all the DBUs and who and the number of DBUs per each product type product line.
Uh so this is one way of doing it. The other way of doing it is monitoring which is what we discussed like uh we run through system uh billing do usage table and find the exact attribution of each of those jobs/notebook/ users etc all that information directly from the notebook. Uh so that's about this. I just want to quickly showcase one other thing which is I I mean most of us would like to develop in IDE. I mean it could be PyCharm or it could be VS code or it could be cursor.
Um in this case I have um cursor and I have a probably a simple enough uh job. Um in this I have injection task and that injection task I mean there is an injection task and there is uh I do some analysis in Python and I do some analysis in SQL. So that's the entire uh DAG here. Now um you might be using VS code already and the way you deploy this job to data bricks is uh using DABs uh data bricks asset bundles. Now DABs is purely ML based.
So you can see a lot of ML here like the job config itself is in ML and there is datab bricks um ML file also which you showcase where you want to deploy in my case I'm deploying to a specific workspace and uh I I'm specifying a root path also where I want to push my uh notebooks so this is the entire thing I mean there is uh code uh there is main. py Pi where I'm trying to loop through and find what is the number of uh number of records of a per a specific uh table which I'm passing through uh again job parameters.
So I do try to find uh spark uh spark. t and just find the df uh count. Yeah, here is where df count I'm doing. And then there is query a simple query again select star and there is uh a notebook also which does a lot of things uh here I I do inest some data. So this is the entire thing and I can deploy directly with this. So I do derbix bundle sync if I changed any code it will reflect in the in the uh platform and also deploy as in it deploys the code and also runs all of that in a single step.
And so this code whatever I have on my lake uh on my notebook on my uh laptop it's essentially pushing everything into the platform and also running it. So with the bundle run I'm doing a run of that particular uh so it's already available. So um I have only one job here. So I will just run run that. Now this will start because I chose uh let me go back to my job config. Because I chose performance target as performance optimize the job will start immediately uh to show you the difference between them.

29:48 Azure Data Factory

So yeah, so I have create job create a new job. In this I can directly jump to performance of performance target. Yeah, performance target. So this is the performance target which we have been talking about. Uh that performance target of performance optimized will trigger the job immediately. Standard mode will there will be a startup delay of 4 to 7 minutes. So uh to go back again performance target of performance optimizer I chose and ran the job.
So this is running right now and if I can copy the URL I can directly jump to the notebook platform and see where my notebook is uh where my job is running. Um yeah so can see the job is already running yeah you can see 4 4. 57 seconds my a lot of code is there here but essentially uh my job is running my just to show the DAG uh this is the DAG there is an injection task and there is analysis task in Python which is a simple Python script and there is a SQL notebook again so this is the complete DAG And my job is right now running I mean my injection task is complete analysis uh tasks are running both of them are running in parallel in serless environment.
So this is another way of running uh your code if you want to use serless and the final bit I wanted to show was if you are used to ADF um triggering the pipelines that's been a bit of a pain uh because um there are lots of new features which are landing in uh data bricks workflows for a simple example being for each uh recently introduced powerbi task type a lot of other new features also uh run job for example. So u at the same time um ADF orchestration has been a bit has been lagging a bit.
So that's where recently ADF launched a new new task type um let me go to the orchestration. So if I go to datab bricks previously there was notebooks notebook jar and python. Now we have job also. So you can trigger you can create a uh workflow in datab bricks and just use the job ID of that particular job into directly in ADF. So for example in this case uh I already have triggered uh I already created a job. So what you need to do is you just need to create a link service.
I mean in this case I will just test the connection. Uh it should be successful. Once it's done I will show uh what I did for this job. Yeah connection is successful. And here is where I did a uh job ID. I I created uh jobs here. So let me go to the job. So ADF PF optimized mode and ADF and standard mode. So I have similar jobs here. ADF perform optimized and uh standard mode. Uh let me go to performance optimized. And it is one.

32:28 Advanced workflow tasks

Let me copy from here. That's better. ADF and per optimized. It's exactly the same job. and I will run through the job. Okay. What is it? I need to test connection. Yeah. And I do have the parameters also already defined here. Uh just to showcase that I'm running the job. Uh let me trigger the run without any trigger right now. So let me trigger the run and you will see within no time this page getting refreshed. Uh this page should get the same job.
It already has come. Yeah, you can see the job already triggered. So this is another way. I mean if you are use if you are more on ADF because of probably the data is somewhere and you need to load that data into uh ADF uh gen two probably I mean ads gen two probably and you would like to use ADF only as an orchestrator. This is another way of looking at things like you create the workflow with the entire DAC. I mean it could right now I only have two tasks here but I could probably create any number of uh d any number of tasks with this uh number of uh task types we have.
I mean very recently we added powerbi as you can see here and then there is ai dashboard and a whole lot of other options also visible here and the number of tasks right now recently was increased to thousand I mean you can create thousand task in a single uh job. So um running such a workload through ADF would have been a pain because you would need to um each task will go into one pipeline. So each task will run uh in a single job the single job cluster job cluster will needs to shut down and then again one more new task new cluster etc etc all that is a pain.
Now if you move to completely through uh create the job as a workflow directly in database environment with a with a UI or with the API or any which way you create the d you create the job get the job ID and just put that into the ADF and you're done like this entire the process of uh reusing clusters and all that will be possible because of this. So that's another way of orchestrating your workflows. Uh so to to uh just briefly run through we r we discussed uh quite a lot like these are some of the important um advantages of serless.
So the number of things someone I mean the teams have to spend or like upgrading DBRs that's the most important and very painful work like the most of the SMMES uh who are highly scaled instead of them spending on productionizing use cases are are putting some velocity into the business value rather they are spending their time on upgrading DBRs. with serless you don't need to bother about that and there is a lot of uh right sizing the cluster type clusters and finding correct instance types and a lot of other information a lot of other very um I mean unwanted things probably you would have you would be doing uh if you move to serless again that's not going to be a problem uh I will run through these three things here simple we saw I don't need to create anything I just need to select uh serless from the drop down and you are done and same with the with the workflow as well.

35:47 Advantages of serverless

Um so business goals instead of in front ops like selecting thousand things you don't need to do that there is strong cost governance I have shown how we can do uh system use uh system tables to do to create beautiful dashboards to understand who is costing who is causing what and how much cost etc all the job runs and so on and so forth and we saw the startup time is very very fast I mean we saw in the notebooks and we saw in the performance mode also performance optimized mode also uh environment caching I discussed briefly like do not use percentage pip in serverless please please use uh environments uh which is on the in the right pane and we all we can also use base environments like that's a complete topic probably please join tomorrow's session again uh maintenance free don't need to upgrade DBRs you don't need to upgrade any versions so database will do the version upgrades behind the scenes and you didn't need to uh worry about that automatic infraction so this is a very uh important topic like um you might be choosing say uh some graviton for example you might be choosing graviton for a specific set of use cases where you would like to use um photon for that uh workload now that is again you would have to understand you need to spend time to understand which workload is it storage optimized is compute optimized or what exactly is happening etc all that you need to understand your workload and its corresponding instances instance types number of instances and a lot of things now data bricks is doing all for you.

37:47 Auto-optimizing infrastructure

You don't need to worry about that. Rather you spend time on uh writing code and just pushing the pipelines to production etc. Uh one more thing here is automatic infra selection. Um I mean you might have chosen a specific instance type for the for brevity probably i3. We'll take i3 as an example. Um you chose A3 as a instance type for your workload for one fine day. It so happened that there are no i3s available in in your particular region.
Now what that entails is you would have to spend time to identifying the nearest family type uh instance type which would work for your workloads and you need to modify it for all your workflows. Now this is from my experience. I worked with a very big uh uh retail and CPG customer from IMIA. I'm based out of London so uh from Imia and uh during COVID time early COVID uh the cloud provider said they can't give enough VMs because the VMs are being allocated to u very tricky uh tricky uh industries.
So I mean tricky and strategic industries because they're very much required. So at that point all their workflows did not start because they were using a specific instance type that instance type is not available. Now we had to sit and modify each of the workflow to find another instance type and run the pipeline. So that's how painful it was. Now if you're using serless data bricks will do that hard work and if it so happens that we can't find an instance type which we have been using will automatically fail over to your nearest family instance type automatically for you by us.
So you don't need to worry about that. So this is the reason we are saying serless is autooptimizing and uh yeah as we discussed if if you want to run the pipeline immediately use performance mode optimize mode if not use uh standard mode. So u there is a lot of other documentation as well. Please please look at uh the docs. So if I if I go to the release notes. So this is the release notes which has a lot of information like um each of the versions what it entails.
Now performance mode it uh landed uh recently uh mid of April so almost two months now. So this is GA now uh talked about that as well in uh in session today uh now G standard mode is G. So please take a look at it and please use uh serless and uh let us know what you think. Uh that's a brief of what I wanted to discuss in the practitioners guide to serless. Uh if you have any questions I'm available. I will I will be uh looking forward to talk to you.
Thank you. Thank you very much.

Learn more about the Databricks Data and AI platform.