Build Better & Faster on Databricks with the AI Dev Kit

Summary

The AI Dev Kit is an open-source framework that gives AI code-generation tools like Claude, Cursor, and Gemini native integration with the Databricks Data and AI platform, enabling them to generate platform-aware code that follows best practices for pipelines, jobs, and Unity Catalog.
The framework uses two primitives: Skills—concise Markdown context files that route the LLM toward correct Databricks patterns—and Tools—Python guardrails that constrain and accelerate agent actions to prevent common mistakes.
A customer demo shows Alpura building a production milk-freshness application in approximately three minutes using the AI Dev Kit, and the repository surpassed 100 GitHub stars within one week of its January 2025 launch.

Build Better & Faster on Databricks with the AI Dev Kit

AI-assisted development tools have transformed how software is built, yet lack native integration with enterprise data platforms. The AI Dev Kit is an open-source framework enabling AI code-generation services—Claude, Cursor, Gemini, and others—to build and deploy production workloads directly on Databricks. It provides platform-aware tooling allowing AI assistants to generate Databricks-native code adhering to best practices across pipelines, jobs, and Unity Catalog. Critically, it enables non-technical analysts to build robust Databricks workflows through natural language, dramatically lowering the barrier to entry. Since launch in January 2025, the project surpassed 100 GitHub stars in one week with accelerating organic adoption. this video covers architecture, real-world use cases, and implications of AI-assisted development for the Databricks ecosystem.
Talk By: Cal Reynolds, Demo Product Manager, Databricks ;

Chapters

FAQs

What is the Databricks AI Dev Kit?

The AI Dev Kit is an open-source framework that integrates AI code-generation services—including Claude, Cursor, and Gemini—directly with the Databricks Data and AI platform. It provides platform-aware tooling so these AI assistants can generate Databricks-native code for pipelines, jobs, and Unity Catalog resources while adhering to best practices.

What are skills in the AI Dev Kit and why do they matter for LLM performance?

Skills are concise Markdown context files that provide an LLM with the specific Databricks knowledge it needs to generate accurate code for a given task. A key lesson from this video is to keep skills short and human-readable: overly long or complex context files degrade LLM routing performance and increase the risk of generating incorrect code.

How does the AI Dev Kit lower the barrier for non-technical users building on Databricks?

By giving AI assistants the right platform context and guardrails, the AI Dev Kit allows non-technical analysts to describe what they want in natural language and receive governed, production-ready Databricks code without needing to understand underlying infrastructure. This dramatically reduces time from idea to deployed workflow.

What real-world result did a customer achieve with the AI Dev Kit?

Alpura used the AI Dev Kit to build a production milk-freshness application on Databricks in approximately three minutes. This customer story illustrates how the framework compresses the time from business requirement to deployed application, making it practical for organizations with tight delivery timelines.

Full transcript

[00:07] Hi everyone. Welcome. This is your 20-minute lightning talk on how Databricks is going to use Databricks to make Databricks more money on Databricks. It's a lot of Databricks-es. And I know all of you, it's Tuesday, you've gone
[00:20] through a lot of sessions, I'm sure, a lot of conversations. Let's keep this fun, simple, straightforward, and uh let's just get this thing started. My name is Cal Reynolds. I'm a field
[00:32] product manager at Databricks, meaning I live within our sales organization, and I build products for our sales organization that makes us more efficient and impactful with customers and partners. Basically, I build stuff
[00:45] that reduces the amount of time we got to spend doing logs and boring stuff, and it enables us to have more time to spend with you learning about your use cases and solving your data and AI problems.
[00:57] Okay. Before I even get started, the AI Dev Kit repository is right here. It's also going to flash up at the end of the session, so I'll give you about 20 seconds to pull out your phone, and if you want to scan, you can go ahead. I know like a lot of sessions, people
[01:09] get frustrated cuz they don't have the code. A lot of people are visual, they want to take a look as I talk. That totally works for you. I've got 60 slides of visuals, so we're going to start rapid-firing pretty soon.
[01:23] I'm taking a sip of this bougie coconut water, and you guys can continue to scan. Delicious. All right. Who here has ever received a software demo
[01:37] from a tech company? Been shown software before? Okay, raise them high, raise them high. I mean, you know, let's see. Okay. Thank you. Okay, so it looks like half of you, I bet more than half of you, maybe you
[01:49] know a cousin who recently learned how to cloud code and wants to pitch you, you know, their new idea. Uh any of that is considered a demo. All right. So, if America runs on Dunkin's, I can't use Duncan cuz that's copyrighted obviously,
[02:02] then tech sales as an industry including Databricks runs on demos. That's an obvious fact. We need to show you our platform so that you can make sure that you get value from it. Oh, that's pretty cool. You see my closed captions up there? All right.
[02:15] So, uh what does that mean? I focus on demos at Databricks. I want to make us really good at building stuff that resonates with you, our customers, and partners. And what led to the development of this AI dev kit that I'm talking about? Well,
[02:28] at Databricks, uh solution architects, as in technical sales people who were talking to customers, were spending days on demos building assets custom to their customers. That includes synthetic data generation, dashboards, agents, Spark
[02:42] declarative pipelines, the whole darn gamut. And with the rise of tools like Claude and uh Cursor and all of these wonderful things, we wanted to bring that time down to a matter of minutes.
[02:54] Issue number two. All right. Basically, LLMs as of last year, if you all remember, couldn't do stuff with Databricks. They couldn't directly interface. If you went to general-purpose ChatGPT and you said I want a dashboard, it's going to say,
[03:06] "Great. Let me plan that for you." But not actually do anything. That's infuriating and ridiculous. So, my mission uh the start of this year was build a toolkit that enabled Databricks specifically to build stronger demos faster. And why
[03:19] does that matter to our business? Because the less time that we're spending building demos, provided that their quality and resonating with you, our customers and partners, the presumably more money at the end of the day we're making. Again, Databricks is a
[03:32] for-profit company. Uh we need to pay the bills at the end of the day, and demos are again the tip of the spear to make that happen. All right. So, you're probably wondering to yourself, "He's talking a lot about this AI dev
[03:45] kit thing. What does it actually do?" Here's some example questions. Can I get a RFM analysis, is a retail thing, uh dashboard on top of my retail data. Sure. AI Dev Kit can do that. Simple query. Can you build me a
[03:57] lakehouse reverse ETL Python data app that catalogs everything my developers are doing? Oh, I also want to No problem. That's That's fine. AI Dev Kit can handle that. How about it? Yeah. That's fine, right? And uh Right? I'm not sure if you're saying my name or if
[04:09] you're saying can. But yes, the answer is yes. If it's a Databricks resource, the AI Dev Kit can build it, and that's how we've designed the system. All right. So, I'm giving you examples. I'm talking up this darn thing, this this Dev Kit. But
[04:22] what is this thing really when you look under the covers? And how could you implement something similar for your organization? It's really darn simple here, folks. That's the bottom line that I want to drill is that especially nowadays in
[04:36] 2026, everyone talks in a very ethereal manner talking about how complex their agentic systems are, harnesses, all this kind of stuff. I am going to argue to you today that a good set of context
[04:48] that's concise and well-constructed, as well as some good tools, can pretty much That's That's what can automate these busy work processes in your business. When we talk about workforce automation, really uh it it
[05:01] starts with the good context. So, AI Dev Kit is really a collection of skills that informs an LLM of how to do stuff on Databricks with good documentation and a set of tools that guardrails how an LLM actually actions on that. Okay?
[05:15] Skills, tools, context, tools. I'm not going to say it again. You guys got it, right? Nods? Okay. Nice. Beautiful. What is a skill?
[05:27] Oop. Okay, my clicker was lagging there. All right. Uh basically, if you haven't seen it already, I'll just go through it very quickly. Skill, markdown file established by Anthropic this year. It's pretty darn cool. What is it really?
[05:39] Okay, I just said it. I I I spoiled it. But it's a markdown file, okay? This is natural language. It's really simple stuff. Think of it like a brain. A general purpose model can do a lot of stuff, but
[05:51] for enterprises, you have very specific needs. You need to supercharge these hosted models to do stuff that's actually relevant and scoped to your business. Cuz yet again, even in 2026, a general purpose Claude code or any of
[06:04] these coding agents that exist, all through AI Gateway, by the way, got to plug that. They can only do so much and they don't fully understand your data. That's why Databricks is so damn powerful. But, when it comes to skill construction, what are these things actually made of
[06:17] so you can learn and kind of bring this back? It's really really really simple. At the top of a skill, it's the same construction every time. There's a description. That's how an LLM, you can think of it like a brain, a router, is finding that skill. So, this
[06:31] is the means for how we can effectively create a brain, a corpus of knowledge that an LLM can act on for your business, is having really concise descriptions of what each skill does for
[06:43] your business that it can then retrieve easily. These should be human-readable to any reader super, you know, simply, yet again. What else is this? This is just the context, the documentation, the examples, whatever else you want your LLM to actually action on, okay? All
[06:57] right, that's the first one, skills. That was skills, okay? All right. Tools, F Bot Builder. F Bot Builder fans in here? Yeah, okay. Okay, I see you, there we go, yeah. All right. Um we we can build it. Uh yeah, so for
[07:11] the AI Dev Kit, we built tools in the form of Python. And why does this matter for us building demos? Well, we wanted our LLMs to be able to do things that we knew were crucial to our customers and to our
[07:23] demos uh that that were going to reduce the amount of steps that the LLM was taking otherwise with with a traditional CLI interface. Put really simply, basically, if we know that we're often doing like parallelized SQL queries,
[07:36] let's put that into a tool that an LLM can leverage instead of having to reconstruct it every single time or even having that exist as context, which it then has to reason on. Tools are a great way to guardrail what
[07:48] your LLM can do, and reduce the amount of tokens that you require to get from the input to your destination, the thing the darn thing that you want to build, right? Okay. All right, that's a blank slide. Yeah,
[08:01] that was the talk track. Okay, so uh after we did this for the Databricks uh field, we wanted to bring this to customers. Uh SpaceX IPO, in the same way AI Dev Kit
[08:13] kind of IPO'd, you could say, in February of this year. And it came at a good time for us generally in terms of adoption, because as you can see here, the interest in the term vibe coding has gone up exponentially. And
[08:25] this graph was as of October of last year. Since then, you can imagine agentic coding and develop the development is literally everything. You'll hear it in every single session that you go to at this conference. Okay? What else happened? AI Dev Kit started
[08:38] spreading organically among customers. We have done zero blogs on this whatsoever. Uh we knew that Genie code was coming. We wanted to give customers a way to interface with Databricks from outside of the platform with the tools that they like. So, that was pretty
[08:50] cool. It uh ended up having such an extreme slope, you know, you could even imagine Tony Hawk uh doing doing a nice halfpipe in there. Okay? Well, anyway, I got I got a couple smiles out of that, but All right. Let's uh continue. How about a customer
[09:03] use case? How are real organizations using the AI Dev Kit? And shout out to Alpura, Timery here is Latin American uh you might know them. They are one of the main milk distributors in Mexico.
[09:16] And their use case was really interesting. They had an app that they called Embajadores, or ambassadors, where basically they're monitoring the distribution of all of their dairy products across different uh retail
[09:28] stores in Mexico. And for anyone who doesn't know, dairy is actually a very complex supply chain, because if you mess up any step along the way, that could be spoiled milk, right? I mean, everyone's had that. Go to the grocery
[09:40] store, you get your milk and it's already disgusting. Ugh, horrible. But, Alpura is using data and AI, and ultimately the AI Dev Kit to make this happen. Where previously, they were spending days and had a full
[09:52] development team building out an application to uh service this embajadores use case for milk freshness in the supply chain visibility. It's a lot of words, but basically just looking at their milk, right? Uh they took the
[10:05] AI Dev Kit. They married it with their Gemini CLI that they were already using for vibe coding. And in 3 minutes, they were able to build what eventually became their production embajadores app from the
[10:17] ground up. That's their testimonial. It still drives me nuts in how crazy it is. Uh but, you know, it's a picture and a quote are worth a thousand words. How about I reconstruct it for you very quickly, okay? All right. Live demo.
[10:32] So, I took the use case that Alpura uh gave us the wonderful rights to talk about and to use, so thank you for them. I went into the AI Dev Kit. I described it very simply in my terminal, as you
[10:44] saw at the start of this presentation when I flashed it up. Literally going into a terminal, and hopefully it shows. There it is. Asking for things in your natural language, and it'll just happen. And just to show you, cuz I don't have
[10:56] the time to show you the uh execution here. This is a one-shot, okay? So, I gave the use case. It did synthetic uh a data generation that supported our
[11:08] story, which is great for demos at Databricks, but potentially also for customers that want a digital twin. It built a dashboard, as you can see, with interesting spikes there. I wonder if this thing has like a pointer on it.
[11:20] No. Okay. Well, whatever. You can see them there. Bar chart, line chart, pretty cool. Also, how about a geospatial showing the uh relative health of all the different Alpura stories. And again, this is all synthetic, mocked, but it just goes to
[11:33] show the validity of their use case. Oops, I tried to click next, but this is a a URL. Okay, how about Genie? Same thing. AI Dev Kit will generate it, right? Then, the time to go from uh you know,
[11:47] learning Databricks, uh it it it for for customers like Alpura, it's going from days or even weeks and training sessions down to simply downloading a tool like the AI Dev Kit and asking questions of your data in plain business terms. And
[12:02] just to plug Genie code, look at how cool this is. I asked, "What trends should I care most about?" And holy cannoli, it's giving me a full analysis in in agent mode here. They're going to talk about that over there actually, kind of interestingly. Uh
[12:14] highlighting to me that the Central region is the one that I got to worry about the most. Okay? So, Alpura, this is their use case. They did it with the AI Dev Kit. It's pretty magical how quickly uh these things can happen. But,
[12:27] let me just tell you a bit more about what we learned and maybe why it's not so magical. Okay, okay, okay. Right back to the slides. Okay.
[12:39] So, Alpura is one of 3,000 customers at Databricks. Uh and maybe some of you already installed this. I should have probably started with that, but who have been using the AI Dev Kit in the past 4 months. So, this thing has spread organically pretty
[12:52] crazily. We've had a lot of customers come to us and saying that they are displacing their production use cases uh with projects that are just as good built with the AI Dev Kit. Now, sounds ridiculous. We're getting real customers
[13:04] doing this. We're getting thousands of customers doing this. But, again, what we should take home from today is that this stuff is not rocket science, right? Good context paired with a couple good skills
[13:17] can make your life as a Databricks developer immensely more easy. So, TLDR, curate your darn context. All right? Let's talk about context. I got about 6 minutes left.
[13:31] Because all of you in some kind of way are thinking about vibe coding, agentic development, and therefore you're involved in a act of context curation by necessity.
[13:43] You got to make sure your information is palatable and interesting and easy to understand for an LLM that it's actually meaningful. That's Socrates with the DataBricks hat. I'm sure if he existed today he'd be a big DataBricks fan. All right. So, when I talk to customers they
[13:56] have context like this, okay? This is crazy, all right? Like whether it's a single prompt or it's your skills, there's so much stuff. A lot of context is also AI generated, which produces a
[14:08] big problem actually, because AI models are immensely verbose. Everyone's heard it, there's a big AI slop problem in our industry right now. Everyone knows their cousin, nephew, daughter, whoever it
[14:21] happens to be, who has gotten access to these agentic coding tools and now coins themselves a app developer or a context curator, right? Right. Okay. Well, the simple lesson
[14:33] here oh Okay, my slides are skipping a little bit, but just keep it simple, okay? Like your context should be as stripped down as possible and it should truly be something that you can glance over and understand immediately, okay? I can't
[14:46] overstate this enough. Keep it simple. For the folks in the back, I see you guys at the table there, you know? Someone on a laptop, right? Phones, okay. And how about in purple? People who like purple, there's a lot of purple lights
[14:58] coming down here. Keep it simple, okay? Keep it simple. Everyone, everyone agree? All right. All right. Thanks. Keep it simple, it'll make your life a lot better. Let's see an example. All right, these are two different skill
[15:10] files. One on the left, that's AI generated slop. You can see pretty clearly here, I'll just show you, okay? Long lists that don't really mean anything, right? That's not good for a model to understand. The more context that a
[15:23] model takes in, the more more degraded your performance is going to be. And business nowadays is won on very small margins because everyone has these LLM tools. So again, curating this context is really darn important.
[15:37] Right hand side, you see a good skill. It's pretty darn simple, okay? It's saying generate a demo outline for a use case. That's really easy to route to. It's easy for you as the user to pallet and roll around your brain. And
[15:51] generally a lot less verbose. Okay? If it's readable for you, it's readable for an LLM. So keep it simple, okay? That's it. That's the That's the thing. Take that away. All right.
[16:03] All right. What's lesson number two? Cuz I can't come to a presentation with just one. I I got to spoil it right there, but it's anyone can do anything now provided you have the right skills and the right tools. Meaning a lot of you
[16:16] are traditionally people who are technical. People who can do stuff on Databricks, right? A lot of times I talk to customers and you know, people who are more technical are spending a lot of time servicing people
[16:29] who are less technical. Think analysts and people who have never touched code in their entire life. With good context and good tool curation, those analysts can now directly build dashboards, apps, agents.
[16:44] And that shouldn't be an existential concern. It should mean that you, the more technical person, are able to spend time on things that really drive value for your business. Okay? All right.
[16:56] Here's an example. In the AI Dev Kit repo, we package an app so you can do this with your own customers. We have customers who have adopted this already, reskinned the entire UI, and basically they deploy this on Databricks and it's
[17:08] a hosted version of Claude code and the AI Dev Kit. TLDR, any non-technical person can come into this darn app, ask for a dashboard, and then get it. And then they can iterate from there.
[17:20] Beautiful thing, too, is that everything in this app is mapping back to MLflow and AI gateway. So, it's using the platform in the best way possible. It's traceable. It's understandable. Something that you can go back and analyze. Okay?
[17:32] So, that's available today. It'll come back. I see folks coming in now. Welcome. Uh okay. Summary, I'm not going to go through all of this. Running out of time here, but basically, like, if you have analysts or
[17:45] you're trying to service more people uh to build stuff with AI, uh good tools and good context, again, are the way to do so. And this is a real value unlock for our customers. It means, again, less
[17:57] time servicing these ad hoc requests, more time on valuable stuff, okay? I'm saying these things over and over again just to make sure. I know it's the end of the day. Homework, download the AI Dev Kit, try
[18:09] it out, okay? That's number one. Try it out. Number two, within your business, because I did this for Databricks with demos and with our sales team, okay? For you, think about what are the things
[18:22] in your business? They could be Databricks, they couldn't be as well. Dashboards, presentations, when your kids dance recital is, right? Doesn't necessarily matter. Think about those things.
[18:35] Turn them into skills. Describe them in plain natural language, and an LLM will be able to action on those tasks for you and automate them for you in your life. Wait, wait, wait, wait, wait. How about
[18:48] this Genie code thing? Who's heard of that? People here? Yeah, Genie code? Okay, that's really exciting. It's available in our platform. It's awesome for building with Databricks inside the platform. Basically, uh the AI Dev Kit is for outside of the platform. If
[19:01] you're using Claude, Cursor, Gemini CLI, Copilot, then the AI Dev Kit today is our tier one way to do so. Genie code will eventually be available from outside of the workspace, in which case, everything will go there. But by using the AI Dev Kit today, we will route you
[19:14] in there for you. Okay. Yeah, as you can see there, that that's what's going to happen. You won't even know if you use the AI Dev Kit, it'll be great. Uh questions, I actually am going to stick around cuz we're running out of time. And uh if you want the repository
[19:27] or my LinkedIn, they are right there. The last thing I'm going to tell you cuz it skipped over the slide before you clap is that for Databricks, our field on a mass survey of 3,000 people on average reported being greater
[19:40] than 10 times more efficient in building demos. So for us, having good context and good tools has been the difference of a lot a lot of money that I can't touch on cuz that's illegal. Okay, thank
[19:53] you everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.