LakeFoundry: The Databricks-Native Delivery Platform for Spec-Driven Autonomous Software Building

Summary

LakeFoundry is Slalom's autonomous, agent-driven delivery platform that transforms plain-language ideas into production-grade Databricks solutions through a three-plane architecture: a Control Plane for specification, an Execution Plane for governed code generation, and a Continuous Improvement Plane powered by MLflow trace analysis.
The platform integrates natively with Slack, Jira, GitHub, Teams, and CLI, uses deterministic patterns with a Databricks-specific AI DevKit to generate consistent governed output, and addresses the finding that only 10% of enterprises have successfully scaled agentic AI value.
LakeFoundry uses Lakebase-backed state and conversation management, job isolation with scoped permissions for secure agent execution, and supports plugins for Codex and Claude Code—enabling both developers and non-technical users to deliver Databricks solutions through natural language.

LakeFoundry: The Databricks-Native Delivery Platform for Spec-Driven Autonomous Software Building

Most enterprises don't struggle to write code — they struggle to consistently deliver production-grade Databricks solutions that are clearly scoped, governed, and built to last. LakeFoundry is Slalom's autonomous, agent-driven delivery platform that transforms ideas, tickets, chats, and draft requirements into a repeatable, auditable path from concept to production — entirely within Databricks.
At its core, LakeFoundry operates across three planes: a Control Plane that enforces structured specification and stakeholder alignment before a single line of code is written; an Execution Plane where autonomous subagents generate, verify, and deploy production-grade code with Unity Catalog governance, policy-as-code guardrails, and replayable LLM audit trails; and a Continuous Improvement Plane where an embedded Improvement Agent analyzes MLflow traces, detects performance drift, and proposes human-reviewed PRs — keeping systems from decaying post-launch.
LakeFoundry integrates natively with Slack, Jira, Teams, GitHub, and CLI — meeting teams where work already happens — while eliminating the ambiguity, context rot, and governance gaps that cause autonomous development to fail at enterprise scale.
The result: spec clarity before execution, test-driven autonomous generation, and governance embedded by design — not bolted on after. For Databricks customers, LakeFoundry provides a native path to operationalize agentic software delivery on the same trusted platform that already governs their data, workloads, and environments.
Talk By: Cameron Shotwell, Databricks GTM Lead, Slalom ;

Chapters

FAQs

What is LakeFoundry and what problem does it solve for Databricks customers?

LakeFoundry is Slalom's autonomous, agent-driven delivery platform that converts ideas, tickets, chats, and draft requirements into production-grade Databricks solutions through a structured, auditable process. It addresses the reality that while two-thirds of enterprises have experimented with AI agents, only 10% have scaled to find value—largely due to unclear specifications and governance gaps.

How is LakeFoundry architected?

LakeFoundry operates across three planes: a Control Plane that enforces structured specification and stakeholder alignment before code is written; an Execution Plane where autonomous subagents generate, verify, and deploy governed code with Unity Catalog guardrails and LLM audit trails; and a Continuous Improvement Plane where an embedded agent analyzes MLflow traces, detects performance drift, and proposes human-reviewed PRs.

How does LakeFoundry handle governance and security during agent execution?

LakeFoundry uses Lakebase-backed state and conversation management, job isolation with explicit permissions scoping, policy-as-code guardrails, and replayable LLM audit trails. Unity Catalog governance is embedded at the Execution Plane—not bolted on afterward—ensuring every generated artifact is governed by design.

Who can use LakeFoundry and what tools does it integrate with?

LakeFoundry is designed to serve both developers and non-technical users including product owners and data analysts, meeting teams where work already happens through integrations with Slack, Jira, GitHub, Teams, and CLI. It supports plugins for Codex and Claude Code, and uses a Databricks-specific AI DevKit for context-aware code generation.

Full transcript

[00:08] Hey everyone, can you hear me? Good. Great. Um, my name is Cameron Shotwell. I'm one of the uh data bricks leaders here at Soom Consulting. Um, and I lead our West region, all our pre-sale solutions and solutioning. I'm also the
[00:22] product owner and sole developer of Lake Foundry and I'll be talking to you about that today. So quick shout out to Soomn. Uh we are top 10 global gsis consulting for data
[00:34] bricks and a large consulting firm in general. Um we have won two amazing awards this year. The global data platform of the year and the global insurance partner of the year. So um huge shout out. Yep. Thank you.
[00:53] So, as most of you have probably seen in the announcements this week, in the world that we live in today, twothirds of enterprises have experimented with agents. Uh, but the truth of it is that
[01:05] only 10% have uh scaled to find value. And um if you're like me, it begs the question, why is that? and and why are certain companies winning in the 10%
[01:17] versus which companies are losing and so I naturally go and look at the leaders in the space which is open AI and anthropic if you start looking at their socials if you look at their strategies
[01:29] it becomes very clear quickly that coding with agentic tooling is the future in the sense of for developers but also for non-developers and you can
[01:42] actually see that in A lot of the product directions if you play pay close enough attention you have cloud co-work you have these other things that are using aentic coding under the hood but are still for those end users who are
[01:54] not developers. So now that we know what the best companies or the top companies the model providers are doing uh let's go through a little scenario of maybe what it looks like today in the real world when it
[02:07] comes to how this works. So here I have an idea owner's problem. So here we're talking about product owners, but this could be a a business user. This could be um a a data analyst. I just want you
[02:20] to open your mind. We're going to use a product owner here because it fits very well. But this is what happens today. A product owner has an idea. That idea needs to be vetted through uh
[02:34] UI UX designers, through tech leads. There's a lot of conversations that happen. Those conversations happen in meetings over Slack, during PRD documents, things like that where
[02:46] they're assessing requirements. They're assessing feasibility. And let's say after two to four weeks of conversation, you then go ahead and put it in the backlog. You give it to engineering. And
[02:59] then if you're a product owner, you wait and you wait and you wait until eventually your feature is in some type of prototype and eventually makes it to production. And now you can realize your idea to bring the value to your
[03:12] customers. So your first instinct like most of us is to get a coding agent. do you go and get your uh enterprise contract with OpenAI and you say okay product owners
[03:26] idea owners in this case you now have a coding agent great so now you can expect that um we're going to be able to speed up our work a 100fold
[03:40] the reality is that it does not work that simply there are too many things that you as a user need to own and do to make a coding agent like that do what you want it to do in any way, shape, or form.
[03:52] For example, to get high quality output, your agent needs to know these things, especially as it relates to data bricks, your enterprise context, your enterprise architecture patterns,
[04:05] feedback from UI and UX, your agent skills, plugins, hooks, these things that if people have used these, you would know the complexity of what different skills you have to use to get
[04:17] what results and how that works. different data bricks knowledge in general. These agents come with a cuto off date right of the models and how much they know in data bricks depends. So you use skills to inject that
[04:30] context. If you just go download cloud code it does not come with that by default. And the last thing which is really important here is the concept of specdriven development to give the TLDDR there. What spec driven development
[04:44] really is is the concept of well if I idiate with a model and through an agent harness I can work with the with the model to basically create a specification or create a requirements
[04:56] document in a product owner's world it would probably be called a PRD product requirements doc so you specify that and then you give it to a different model and say go execute it and that is what I would say is the leading industry
[05:09] standard to doing agentic development today. Now, as a developer, I know this, but as a product owner, in my opinion, they should not need to know this
[05:21] because it's too complicated. It's it's too much to learn. The the the what you need to learn to be successful in that world of agentic engineering is very high. And so, let's go back to our example
[05:33] here. So, I open you up to the question, what if you didn't need to learn that stuff, right? someone in your organization, someone in your engineering organization has gone through these trials and tribulations I
[05:46] have and they know these things. So now the question is how do you scale that to the rest of the organization without requiring a large amount of upskilling throughout your entire or
[05:58] that is where lake foundry comes in. Lake foundry is the thing that distills down your common engineering patterns, your common engineering skills and
[06:10] everything else that is required for aentic engineering, but it abstracts it away for the end user. I'll show a demo in a second and what that looks like. But I just want to pause here and I want you to think about more of the concept
[06:24] of how do I as an engineering leader abstract away the complexity of aic engineering for my business so that my business can accelerate our data
[06:37] products. So our business can accelerate the work they need to do without needing to understand the complexities of what's going on under the hood.
[06:52] with Lake Foundry. That same process of communication between users of um doing specri development in general is now fully encased in Lake Foundry
[07:07] for an engineering team. The benefit here is actually the latter half as well of as you can see Lake Foundry builds a prototype
[07:19] and that prototype will follow the engineering team's best practices. So if you think about it this way, most engineering's teams have a mono repo, a repository where they keep all their ETL
[07:32] pipelines. Lake Foundry will use that as a template to do a few things. It reduces the cost. You're not recreating from scratch every single time. You're just adding a configuration to an
[07:44] existing pattern you already have. It also creates a much more consistent output. These models, these harnesses are an undeterministic thing. And so what Lake Foundry is trying to do with
[07:58] the help of your engineering team is make that as deterministic as possible in an agentic way. So what lake foundry will provide is
[08:10] managed outcomes for your business for your product owners so that engineering manages the workflow they manage the templates they manage that experience
[08:25] for the end users so that you can enable aentic AI for the rest of the business and scale without creating slop everywhere throughout your organization.
[08:37] Just to cover this right, Lake Foundry will have access to your enterprise context through managed MCP or through Geneontology in the future. Your enterprise patterns for engineering are
[08:50] provided. You provide those patterns to Lake Foundry. You can collaborate with teams via Slack on ideas. I'll show that demo in a second. Lake Foundry houses all of the agent skills, plugins, hooks, things that an
[09:03] engineer needs to know. But to reiterate, with Lake Foundry, a user does not need to understand those things. They get the benefit of them without needing to understand what they are to call them to do them.
[09:15] It uses the AI devkit. That's how we get our data bricks knowledge outside of just doing data bricks tooling. And then last but not least, it handles the spec driven development for you.
[09:32] So where else does Lake Foundry fit? And this is kind of the three main ways that I see Agentic AI being covered throughout the organizations and the organizations I speak to. There's level
[09:44] one of using the CLI and this is more of a level ABC. Maybe don't think of it as one, two, three in terms of which one's better or not. There's different use cases where you would want to use others. Level one is just using the CLI
[09:57] like codeex cloud code where you can iterate extremely quickly, right? You're just chatting with your agent very fast in the CLI doing whatever you need to. Level two are the agent workflows. So
[10:10] the idea here is using skills. You know what the best practices are within your organization or how to do agentic development. And so you're able to tell the agents, hey, this is how we build
[10:22] things in the organization. This is still through some type of managed service. So codeex for example has a codeex app or an application and that is a little bit more user centric compared to the CLI. And then the last thing at
[10:35] level three is fully managed. So when a user is in a chatbot for example or in a chat-like interface building things agentically what it is is fully managed. And so where lake foundry is really being put
[10:49] in is those last two levels. So we have the lake foundry plugin which is essentially a codeex plugin that allows engineers and the developers on the team the ability to use an inter interop
[11:04] between the actual managed service behind the hood. And then the one that we've been talking about mostly in this talk is actually the last one the lake foundry app or application.
[11:19] Let's talk about a little bit of the technical architecture. There's some interesting things in here that I think for anyone who's trying to build a AI within data bricks and within the organization I think can be very interesting. So as we go from left to
[11:32] right there's there's something so a user today with Lake Foundry we have a front end that I will show off in the demo that uses fast API. It's a data bricks app. It uses the foundational hosted models to do a lot of the
[11:45] conversations with the LLMs, but you can also use we have integrations into teams in Slack so that you can do that same interface, the same chatting of building products, but in a UI that you're
[11:59] already familiar with and that you understand or your users understand. In the middle here is really the core engine of everything in the sense of we use GitHub and we use specs to store
[12:11] information to track history like most developers do and then lakebase is really the core memory of the agent history. So every message that's being sent and and recorded is all being
[12:24] stored in lakebase which allows us to do some very interesting things not a part of this talk of how do you create an improvement system on top of all of those traces to uh fix user behavior or
[12:36] to fix certain skills. Anyways now the last thing here is the execution. So um in a lot of skills today in a lot of agentic engineering uh there's these concepts of uh ideiation
[12:50] brainstorming is one that's very popular where you talk with the agent to make you think of your ideas to come out with your requirements and then you immediately go into building. Well, in a
[13:02] world that I'm building here of trying to bring Agentic Engineering to the business, we actually want to split that entirely because we want to make sure users are collaborating with their
[13:14] co-workers on the ideas and not immediately building things that maybe didn't catch the right requirement. They didn't have the right conversations. And Lake Foundry accelerates those conversations today. But the other thing
[13:27] that's extremely important for the execution here I've put per user lakeflow jobs as the main thing as you can imagine as soon as we start talking about security peripherals when it comes
[13:39] to aentic AI you may think well I'm an admin in my data bricks account so can my user build anything in data bricks do they have access to all data can they do
[13:51] anything like that the answer is no because each job that the user user uses to execute the agents in real time are all user scoped to their permissions. So
[14:04] when a user goes through this flow, they'll never be able to see data they weren't originally able to see in Unity Catalog. They won't be able to access data. The agents that they spin up will not be able to access data. It's all managed through Unity Catalog in the
[14:17] same way that they use data bricks today. And then the last thing there is lake foundry output. It's very small here because quite honestly it can build
[14:29] anything in data bricks. It has access to do whatever product you want in there assuming the user underneath has the proper access to build it. If if if that makes sense. Let's do the demo. I think the thing
[14:42] most people are probably waiting for. So this is the Lake Foundry UI. This is a a short video. Um, if we have enough time, I'll also just pull up the live UI
[14:54] so we can we can look through it. Here we go. So, here are the three things. You describe your outcome, you refine your spec, and you build it. This is sped up, but this is me as a user
[15:07] building out an ETL pipeline, for example. Right? Maybe for a end user, they wouldn't want to necessarily build an ETL pipeline. They could build a data bricks app. But the idea here is anyone who's used aentic engineering should see
[15:19] and notice that I'm not calling skills. I'm not calling plugins. I'm not doing anything aentic engineeringy in this UI. I'm as the developer have
[15:31] abstracted that away. They don't realize what's happening under the hood. I understand it. So the right side here is all of the user specifications. So this is the specdriven development of it. So
[15:44] you have all these tabs. We have judges to make sure your spec is good. And what you can see here is this is as we go into the execution plane, the group of agents fully building out the specification that you put. It runs as a
[15:58] data brick serverless job as I said per scope to the user. And this is it actually finished. So there's a couple of key things here that I'm actually going to pause for this. Let me see.
[16:18] One of the main things right now in aentic engineering is actually the concept that agents are always correct. Whatever they build is always right. When they write tests, the tests always pass. And when the tests always pass,
[16:30] that tells themselves that they've done a good job. But in reality, they haven't actually done anything. So what lake foundry does is we have a separate process that goes out and checks what
[16:42] assets did you actually create. In this case I said hey create a pipeline an ETL pipeline. So there I have live links to the ETL pipelines it created and then on top of that we have separate agents that are actually going out to those assets
[16:54] and that's the acceptance criteria on the right actually validating did that job run does that job produce an output is that output the right schema things like that. So we're we're no longer
[17:07] doing unit testing in that same way. We've elevated it more to what kind of testing would I do as a developer, as a user of well, if I think a pipeline ran, I would just go check it, right? I go into the UI and I check did that run,
[17:20] how does the output look and automating that process to really validate that the artifacts that are created in this process are validated and truly getting to the outcome we want um as as a user
[17:32] and as a developer.
[17:45] So this what I'm showing down here is really all the different logs of of the run. But here is actually the ETL job that was created and a pretty simple example, but um Lake Foundry can do a lot of different things that I'll show off in a second. Here's an example of uh
[17:59] us. You know, this is me playing around, but basically the whole process that you saw uh previously in that UI can be completely done uh in Slack in your own threads because even though there was a
[18:12] fancy UI in front of it is fully driven by the agent in that chat. The agent controls everything in that UI which means it controls the way that it talks and you can pass it requirements, you can update your specs, you can do it in
[18:25] the interface that the business users already understand. And just to drive it home, your engineering team is the thing that owns the engineering prowess under the
[18:37] hood. The engine that's actually driving this to make sure that the things they create match the specifications that you expect as the user, as the developer.
[18:59] So here's Lake Foundry. um as I um this is a different example of building an ETL pipeline, but just to show some more goodies of how this works. Um here there's different context layers that we can we can cover here and and that video
[19:11] had to go fast for for the presentation today, but really what I wanted to show is um imagine this is gene in the future. um the different specs that you can create and and and go through in terms of the different assumptions that
[19:24] are provided. Something that didn't cover in the video uh too quickly was the idea of phases. One one failure mode that um I witnessed really frequently while developing this
[19:37] was actually um if you give the agent too much scope, regardless if you write an implementation plan, it will fail to finish that plan. And so we try to break that up in time because
[19:51] uh again if if you give it too much it will fail. So I have 7 seconds left. If there's anything that I want you to leave from this talk I want you to really think about if you're an engineering leader
[20:04] think about how you can manage aic engineering for your business to bring them to speed. And if you're a business leader, I want you to go knock down the door of your engineer engineering leader and tell them, "Why aren't you doing this for me?" Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.