First‑Class Containers Across Databricks: A Unified, Portable Dependency Management Strategy

Summary

This video explains how Databricks is bringing first-class Docker container support across the platform to replace brittle init scripts and at-startup package resolution with portable, governed, faster-starting environments built on Spark Connect and environment versioning.
Live demos show two practical use cases enabled by the new container support: OCR PDF parsing using specialized image processing libraries and geospatial user-defined functions with third-party libraries—both previously requiring complex, fragile init scripts.
Upcoming capabilities on the roadmap include Docker support for serverless GPU workloads in early preview and a managed artifact registry within Unity Catalog, making container images governed and discoverable alongside other data assets.

First‑Class Containers Across Databricks: A Unified, Portable Dependency Management Strategy

Databricks is bringing first‑class Docker support everywhere. this video shows how we’re making containers a core part of the Databricks developer experience, delivering faster startup, tighter governance, and clearer observability. We’ll walk through the product architecture, the UX investments that reduce time‑to‑first‑result, and how we approach compatibility and safe rollout for critical workloads at scale.
Talk By: Justin Breese, Staff Product Manager, Databricks ; Xiao Chen, Software Engineer, Databricks ;

Chapters

FAQs

What problems do init scripts create for Databricks dependency management?

Init scripts run at cluster startup and typically resolve and install packages from the internet each time, meaning any outage or change in a package repository can break all clusters relying on that script. They also conflate build-time dependency resolution with runtime execution, adding significant latency to cluster startup. This video explains how replacing init scripts with Docker containers eliminates these failure modes by baking dependencies into immutable, pre-built images.

How does first-class Docker support improve security for Databricks workloads?

By using pinned Docker images rather than unpinned pip install commands, teams avoid security vulnerabilities that arise when packages update automatically to versions containing exploits. The video references a real incident caused by unpinned packages in customer environments as an example of the risk. Storing governed images in a registry also creates an auditable record of exactly what software was running in any given workload, supporting compliance requirements.

What are the architecture differences between the original Docker support and the new container support v2?

The original Docker support, introduced as a hackathon project in 2018, had significant limitations in which cluster types it supported and how images were managed. The new container support v2 is built on Spark Connect and introduces environment versioning, making containers a first-class part of the Databricks developer experience for standard compute. This video walks through how images are built, stored in a registry, and configured on clusters as part of the updated architecture.

What use cases benefit most from Docker containers on Databricks?

This video demonstrates two practical use cases: OCR PDF parsing that requires specialized image processing libraries not in the standard Databricks runtime, and geospatial user-defined functions that depend on third-party geospatial libraries. These use cases previously required complex init scripts that were fragile and slow to start. Docker containers enable teams to build these dependencies into reproducible images that start faster and behave consistently across development, staging, and production environments.

Full transcript

[00:08] I'm Justin, product manager here at Data Bricks. With me is my esteemed colleague. Who are you? >> Oh, it's Shiao. >> Yeah, Shiao. Nice meeting you. >> All right. So, who am I? Well, I like to
[00:21] open it up. You get to know me. I would love to get to know every one of you. Um, I'm a product manager. Been here for a long time. the Elder Statesman. I work on runtime and dependency management because it's awesome. That's why um I
[00:35] live in Topanka, California, so LA area, but up in the mountains near the water. And uh I have my wife and two small tyrants. Uh they're two and four, and they tell me what to do. Uh you can
[00:50] always send me send me random questions, musings, jaydatrics.com. I took that alias, by the way. I saw it was open. So that's mine. Um, also on Reddit, uh, you can troll me at Justin
[01:02] Databicks and I'll do it back. So be careful. Jiao, who are you? Who are you? >> Yeah, so I'm a software engineer in data bricks. I work on dependencies,
[01:15] container support. Basically, I work for him. >> Yeah, that's me. >> All right. A man of few words. Okay. So, why are we here? We're going
[01:28] to talk about some container stuff at data bricks.
[01:41] challenges of dependency management. So, especially on data bricks, brittle init scripts. Basically a bunch of customers like to put these brittle in it scripts that start every time a cluster starts
[01:54] up and they say, "Oh, this is fine. This is great." And actually what ends up happening is conflating kind of buildtime things with runtime things and we get a lot of tickets. And when I say we, I mean Shiao and team, they get a lot of subzer couple times a year
[02:08] something random like yubuntu's appget repository is down. Pi pie is down. So it breaks all these init scripts and it happens over and over and people open up tickets and they're like hey what's going on and we're like internet um
[02:23] instability. So as we talked about in that other uh other example is uh resolving packages at startup. It also adds latency. So not only do things break because internet goes internet
[02:35] things it adds latency. So um yeah not fun. Uh security. Oh wow. This has been a fun one too over the past few months. The trivi exploit that I'm sure you all
[02:47] got to got to know that ruined weeks of our lives. Um, so basically customers will have unpin packages whether they're cluster libraries or you know what could go wrong just pip install fu and just it
[02:59] can just update whenever it wants. Uh so that's that's not that fun. And the the other thing is slow startup times. So repeated package installs every time on every launch just adds latency and and
[03:13] also not to mention things like requirements. It could be regulatory uh requirements for like uh immutability wanting performance and also being platform agnostic. So these are all the
[03:26] problems and that kind of led us to way back way way back 2018. This is Evan. Evan still works at data bicks. Evan loves skiing. That's a fact. Uh, he
[03:38] likes skiing and he likes Docker. That's all Evan lives for. Um, he had this idea in a hackathon was like, "Whoa, Docker is cool. I'm cool. What if we had a hackathon to put a data bricks uh
[03:51] cluster that starts as a container image?" Uh, so about two days later, Evan wrote about 450 conditional statements into our cluster launcher, and they're still there today. So, it's great. U members of the team curse him
[04:03] daily. they they uh they yeah they cast spells upon him for that. Um but that basically was our first foray into container support and this fell short on many many areas.
[04:18] So as I talked about before we have poor reproducibility. Um there's system binaries and there specific ones like pispark delta. These are actually injected from source at runtime. So even though you provide us
[04:31] with a docker image we're like ah you don't want that pispark. you on a different one. Uh, right. No.
[04:43] pulling the image each and every time. So, it ends up being really really slow. There's a great timeout that some of you might see. You know, 15 minutes, just a hard time out. Oh, it's taking 15 minutes. Bye. Um, another one, um, fine
[04:56] grain access control because when this first came out, like things like Unity catalog didn't exist. It was just old school hive meta store. So governance really wasn't the thing. It was I don't know it was like the wild wild west of the data lake. This is pre-lake house.
[05:10] And then multi-user compute like doing multi having multiple users using a same cluster on this offering ended up being very very difficult. So no and then the last thing is for the
[05:23] R&D teams the engineering teams are lazy. There I said it. You're lazy. Um so lazy meaning every time we had to do feature development you'd have to develop it for dedicated clusters you'd have to and when I say you I mean Xiao
[05:36] and team and runtime team they would have to do it for dedicated clusters standard clusters oh yeah and Docker clusters >> so naturally engineering teams being the devious people that they are they're
[05:48] like no no Docker after G. But then Xiao and everybody else is on a different project and they forget about it. What this ends up being is hundreds of little tiny features just not there. So more rough edges for customers. Now I've
[06:01] really I've really sold you on this. It's a really good offering. Everyone loves it. But well the reality though is customers do and and did and still do. So it's something like um it's it was
[06:16] really hard due to kind of all the intricacies that we talked about. It was really hard to figure out, but once people got up to the the top of the mountain, they're like, "Cool, I'm staying here." And it ends up being like something like 98% of all the DVUs
[06:29] is on this thing. So, it's it's weird. It's like this I don't know, people have a glutton for punishment, but it's there. They like Docker, and that's okay. So, rough edges and all, they want more of it. So, and the verdict ended up
[06:42] being like, okay, this product has been pretty good for ETL, painful for for interactive, but there's definitely something there. And it's one of those super opinionated things that people love,
[06:55] which led us to think about, okay, how could we how could we do this different in a more sustainable way that's kind of more forward compatible with what data bricks wants to do? And so that led us to container support for standard compute and uh also you could say like
[07:08] the artist formerly known as shared clusters. So shared standard they're one and the same because we're bad at naming things. Uh I'll take some responsibility for that but not all of it. So the the
[07:20] goals for for Docker on standard compute was let's have the ability to do fine grain access. So we want many users to use the same cluster um and have fine grain access control via UD catalog 100%
[07:34] reproducibility. So whatever your container images, we're not going to inject any fun things like it all has to come from PI. Figure you all don't mind about that. No more surprises and then lower cost to build on. So we want
[07:46] features just to show up for free. We didn't want additional line items for each and every engineering team because it's exhausting for me to try to ping every team and say but but Docker please. So those were our goals. That's
[07:58] what we set out. And so how we arrived onto that was you know how or how we got there was uh standard compute architecture which underneath the hood is spark connect. It's a client server
[08:11] model. So basically rip out all the dependencies from the server put them put them into the client. Spark Connect is built into OSS. It's language agnostic, backwards compatible.
[08:23] Um, the whole goal of the client, all it has to do is create an unresolved logical plan and send it over to the server. I think I have to do a talk tomorrow on Spark Connect. So, if you want to learn more and geek out more,
[08:35] I'm your huckleberry tomorrow. Um, so the idea though is like what we had to do is think about, okay, so here's the client, here's the server, what what should be in the client, and where is
[08:47] that thing going to come from? Ziao being the enterprising man that he is, he's like, wait a second, we're already doing this and in serverless underneath the hood, we have this thing called an environment version. All that is
[09:00] implementation behind the scenes is a Docker image. So what what an environment version is you know if you double click into what that docker image is is there's an Ubuntu version so it's fixed a fixed Ubuntu version fixed Python version fixed series of packages
[09:14] and we version them we have one two three four five maybe eventually we'll have six I'm just saying I don't want to you know don't want to spill that or spoil that for you um so the idea is
[09:26] that we we have this version bundle of packages and we maintain it we support it for three years just like you know traditional DB VR model and um there think of it as these packages are fixed upon release. So uh we're we're not
[09:40] going to we're not going to mess with it. It's it's uh you know it's it's in your control. We create the base image, you build on top of it. So um the idea is that and due to the backwards compatibility of Spark
[09:52] Connect, it's whatever whatever you ship that environment version, we just make sure the server can run on that. um due to kind of the core like decoupling that Spark Connect does.
[10:04] So that's exactly what we did. So use the serverless environment image and underneath the hood for uh kind of classic compute for standard this is exactly what it is. So on the left hand
[10:16] side you can see effectively it's a it's a client sandbox and inside you've got uh many sessions. Each one the the underlying environment of that session is the Docker image and it uses via gRPC
[10:30] under the hood goes across to the server and in the event of a UDF we actually instantiate that same Docker image inside of a sandbox in a UDF. So if you
[10:42] need your packages for UDFs like many people do, we just handle that for you automatically. You don't have to think about like oh make sure make sure they show up in the executive when needed. It's like nope, ready to go. Super super simple. Um, and this is an example like
[10:57] uh for environment version five, which you know that's the YUbuntu version, that's the Python version, etc. So, all right. And it's super simple. So,
[11:10] for example, how do you do it? Um, really simple. Just build the image. Um, oh, and by the way, I think I don't know if this is in a slide. Maybe it was a little back. I didn't call it out, but so uh we've been in preview on this for
[11:22] like
[11:36] there's a new base image. So this base image is on DockerHub. Um databicks runtime environment and then v5 standard. And the example here is just like, hey, go ahead if you want to do a run command
[11:50] to install stuff, you know, from like a requirements.txt, that would be an example. Um, and add your code and assets to it if you want. Uh, so key considerations here is, um, make sure
[12:02] you target that specific slashdatabicks Python 3 interpreter specifically and not the system Python. Um, and then of course like regular Docker things like keep layers small in order to to
[12:15] optimize going forward. Um, and then yes, please pin dependencies. Please, please, please, please, please. Like we I try I put it in our docs. I try to put it in all the UI and UX things. Like please, friends don't let friends not
[12:27] pin packages. That should be a t-shirt. Save yourself and me. Okay. And then so now that so there you build your image. Um you got to put it
[12:40] somewhere. So you can put it in any place you want as long as these three. Um so right now um ECR, ACR or or DockerHub. Um so right now we support
[12:52] those three. Um seamless authentication. It's it's the same authentication method as the current um you know existing uh container support that we've had. And we'll kind of talk about in a little bit
[13:04] about where where we're going after, but super simple. And the third step, configure the cluster.
[13:18] you're creating a cluster, there's a Docker field and then put the URI in there and then hit start and then profit. Uh so pretty simple. Uh, and it
[13:31] just works and runs. So, well, and Jiao was going to do a demo because I wanted to throw him on the spot like and I said, "No, it must be a live demo. It can't be pre-recorded because the demo gods always smile upon us." But before
[13:44] we get into that, like, let's compare. So, I know so basically recap a little bit. We talked about V1, which was the story of Evan loving skiing and writing a bunch of conditional statements uh and
[13:56] creating the the first version. And then we talked about v2 which is like okay just for standard architecture but how we compare the two is instead of spark being injected at runtime spark is baked
[14:08] at build time like there are no more no more secrets no more surprises. Uh I've been yelled at by many customers over the years for that. So if you've yelled at me I deserved it but no more please. And um and then in terms of uh the I
[14:23] would say the third thing that I like to call out here which is the important one is traditionally with uh container support on data bricks we enabled customers to do init scripts libraries and all sorts of fun layers but we're no longer allowing uh customers to do that
[14:37] because it's like just dropping weight on your foot. It just hurts like there's there's no other way around it. And so going forward for container support, we don't want to enable customers to use a docker image and do init scripts and do
[14:50] cluster libraries because I mean one of the main reasons is if you want performance and you want an immutable artifact, well layering things on top of it doesn't really get you that. So it's kind of antithetical to what goals are.
[15:02] Um and also much better for for support and you don't have to get paged as much. Um, and now, and the other cool thing too is, so the base image, everything that's in the base image is all resolvable from public pi. No more
[15:16] source, no more anything crazy. And ultimately, you get exactly what you want, exactly what you've built. That's what shows up. Whereas before it was just this nasty little dev loop. You're
[15:28] like, well, what if I try this? That wasn't what I expected. Let's reverse engineer it and figure it out. So much more straightforward. With that said, Xiao, you do some demos and then I'll talk for a few more
[15:40] minutes after we're we're done. >> Sounds good. Thank you. >> Um, so here we look at our first demo, which is like probably one of your
[15:52] favorite jobs, which is to do uh optical character recognition, aka parsing PDF. Um so I just showcased the docker file here. Uh so as you can see it's based
[16:05] off our public uh available base images we five standard. And here in this example we do uh here we actually set up our uh pip index URL but this is like
[16:19] you know just pi.org for but you can put whatever private uh index URL your company owns like could be a very private one serving only private packages and after that we have
[16:32] abt get update and install these are all of uh native packages that usually in the runtime you're not able to install but you can install in the docker
[16:44] oops and after that is the python packages we installed uh I just commented what they are, but actually they're actually put into a requirements.txt. We copy into the package into the Docker file and then we
[16:58] install them and that's all. It's quite simple. So let's take a look at the actual notebook. Uh I'm jumping out of the slides. So this is the notebook. But
[17:10] before that, let's take a look at the cluster just to reinforce a little bit how simple it is to configure it uh in our standard compute. So this is our standard compute cluster. So uh for beta
[17:24] as we said this feature is already beta released. Uh we have to use DBR 18 beta or above. I think later on a few weeks later this beta will be gone. So it will be 18 or plus
[17:38] and has been standard mode. uh here since I'm pulling the image from ECR so I have to configure instance profile which give me access to the ECR ripple
[17:51] and here's the docker if I'm pulling from ECR I'm using instant profile the authentication type is just default and nothing else and this is my image and let's take a look at what what this
[18:04] notebook does is simulating like you have like a a inventory of uh invoices is they're all like PDFs files. Um so first of all I just confirm that the
[18:17] um the PDF parsing library which is called tessoract is available which of course it is because um this notebook uh the
[18:29] python process is running in the sandbox with the customer image I just presented
[18:43] and now I just created a list of examples le uh invulses metadata and now I just you know convert that to a image and just this is totally made up
[18:55] don't take it seriously then I define a UDF uh so this is the the contract of uh the UDF it's basically telling like the the
[19:08] data format and schema Yeah, we don't have to read into that. But uh uh basically this UDF does is take the metadata data frame, convert
[19:21] that into a PDF image and then calling this native PDF parser function from the docker container and parse that into a final data frame. Uh this is the final
[19:35] data frame that was parsed. And as you can see like here the docker image is not only affect your python notebook runs actually is would be the
[19:49] same image that powers your udf execution. So in UDF we launch this container or this uh your image as a sandbox. So all the dependencies are
[20:01] aligned with with your notebook. So the UDF actually can parse everything here. So, I can hit run all just uh because I don't want to I want to make sure everything really runs because like this is my last run a
[20:15] few hours ago.
[20:27] >> The demo gods are pleased. >> Yeah, please. Uh don't take me. All right. There we go. >> Yay. And that's the demo.
[20:39] All right, I have a another demo. Thank you. >> Yay, we did a UDF. Yay. >> This is another demo. I don't want to dive into the details, but it achieves something similar, but this time this is
[20:52] more like a geospatial uh processing in the PI Spark UDF. So we can see I actually use a third party library called GTO uh which is a geospatial processing libraries and yeah
[21:06] >> yeah and to be clear like we actually stole this use case from a customer in the preview. So if I see you all in the room yeah I was like oh that's cool we'll use that for data summit but thank you. >> Yeah I would go get back to Justin. All
[21:19] right. So, so that's like, all right, we so we've talked about initial foray. We talked about this current thing. Now, what else? What else do you have in your pockets for me, Justin? Well, we'll talk
[21:31] about this. Uh, so as of last week, we started a preview for Docker support for serverless GPU. Yeah, fun stuff. Um, so this it's all
[21:45] like we're in the early stages on this, but really how it works is there's a local CLI command called sgc. So, sgli run a YAML file and then if we look what is the cat of that that YAML file for
[21:58] example and it you can just supply a docker image again from I think right now this is just supporting docker hub because I mean we just started like we call it like friends and family preview last week uh so you know we got to start
[22:10] small and uh so this enables uh users to reference an arbitrary docker image inside of YAML config and launch a serverless GPU workload um you know so from your laptop into the cloud and and
[22:22] beyond. And um you can pick the GPU types that we have available for serverless and it enables you to wire everything straight into MLflow and Git. So uh this is our
[22:34] initial foray into that and we're going to get super aggressive into this uh specific scenario in the coming quarters even this quarter. But what's next next? Like what's even more next than next? Great question. So
[22:49] um this one we're working on a lot right now which is a managed artifact registry. So what I want to do this is the world where you can just push arbitrary artifacts to Unity catalog. So
[23:02] um containers, Python packages, npm packages, etc. They're going to live inside of a schema for example. You can have many of these artifact types hanging out in the same schema. There's no reason like you don't need a schema
[23:14] just for Docker and a schema for Pi. No, you're going to have it all there. And that's going to be super super fun because we can geek out on like performance things and the reason why we want this in UC. So there's a few things. Uh so if there's a container
[23:27] that's now in UC, we get lineage for free on it. So you have a registry that knows exactly where that uh given artifact is being used likely or and then also what we can do is we can convert that into a lazy load format and
[23:40] make startup go from say 15 minutes on a like a fresh poll down to like 10 seconds. Uh so super duper awesome in terms of performance. Um and then ales because again since it lives in uni
[23:52] catalog you just get that for free as well. Um so that's something um you know I can't give timelines per se but like we're we're getting there super super close. Um but the idea is yeah like
[24:05] there's there's a world where um a schema or folders for example will just be you know a folder for tables. It can be a folder for models. They can also have many different artifact types. Um,
[24:17] so look for like some SQL syntax for example or like a grant permission for like add artifact. So that is all coming. Um, and that basically that will enable a much I would call it like what
[24:30] you all deserve instead of like a slow docker experience. It's going to be super super fast. Um, but first like yes, we're going to be starting with containers. Um so yeah fewish months then the next thing will
[24:43] be serverless notebooks and jobs. So serverless uh serverless notebooks and jobs the regular CPU serverless offering will be bringing container support for that. Now the really cool thing is what we spent a lot of time talking about in
[24:56] this presentation which is um standard support for or container support for standard clusters. It's the same base image because remember it's the base image from serverless that we brought down to classic. So if you all get
[25:09] started on the beta that we have for uh uh classic compute with uh standard clusters once we have support for serverless you can just use your same image that you use there. So it's one of those things to be forward forward
[25:21] compatible. That's a good thing. Um yeah and and basically this will this will basically round out um it will pretty much have container support everywhere and I'm sure model serving I think
[25:34] people want container support for model serving. So, we'll eventually get there as well. All right. So, with that said, I wanted to ensure we had time for Q&A because sometimes people want to talk about dependency management and that's
[25:46] totally cool. Um, you know, if you don't want to ask questions in front of a big big crowd, again, troll me on Reddit or email me, uh, you can email Xiao as well. Um, but, uh, but yeah, that's all
[25:59] we have for today.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.