Databricks Lakebase: Database Branching in One Second

Summary

Databricks Lakebase enables teams to create isolated production database branches in one second at a cost of only cents per branch, ending the bottleneck that forced developers to work against mocks or shared staging environments.
ThoughtWorks engineers Cameron Casher and Chanelle Fernando demonstrate three patterns: database-per-developer for feature branches, integration tests against real production branches with zero mocking overhead, and safe destructive testing on isolated branches.
The copy-on-write architecture that enables instant database branching extends to AI agents, allowing each agent task to receive its own isolated database environment for safe and concurrent execution.

Databricks Lakebase: Database Branching in One Second

Engineers spend weeks testing database migrations on mocks or shared staging environments because production databases have been too expensive to fork. With Databricks Lakebase, teams now create isolated production database branches in one second, costing only cents per branch, governed natively in Unity Catalog.
Learn how Cameron Casher and Chanelle Fernando from ThoughtWorks demonstrate three patterns: database-per-developer for feature branches, integration tests against production branches with zero mocking overhead, and safe destructive testing on isolated branches. Discover how copy-on-write architecture enables instant database branching and how this capability extends to AI agents needing isolation per task.
🤝

Chapters

FAQs

How does Lakebase enable database branching in one second?

Lakebase uses a copy-on-write architecture that separates storage and compute, allowing new database branches to be created instantly without duplicating all the underlying data. This makes branching cost only cents per branch rather than requiring the expensive process of provisioning a full database copy.

What developer workflows does Lakebase database branching enable?

Lakebase enables three key patterns: database-per-developer so each feature branch has its own isolated environment, integration tests run against real production branches instead of mocks, and safe destructive testing on isolated branches without risk to production. These patterns replace workarounds like shared staging environments and brittle synthetic test data.

Why were traditional database testing approaches a bottleneck for teams?

According to the ThoughtWorks engineers, 57% of teams were waiting for another team to finish before they could use a shared environment. Teams built a culture of workarounds — mocks, shared staging, and synthetic data — and called it best practice, developing against fiction rather than real production state.

How does Lakebase database branching support AI agents?

The same instant branching capability that helps developers also extends to AI agents needing execution isolation per task. Each agent workflow can receive its own isolated database branch, enabling safe and concurrent agent operations without interference between tasks.

Full transcript

[00:09] Hello and thanks for being here. We've got 20 minutes, three demos, and one question we want to leave you with. What would your team do differently if you could fork your production database in just one second? I'm Cam Casher. This is Chenille Fernando and we are both engineers at ThoughtWorks and we've
[00:26] spent the last several months working with the data bricks team on what happens to an engineering org when one specific bottleneck goes away. So just the forward-looking statement for you here reminder to complete surveys
[00:42] and let's get into it. So every other artifact in our stack already branches in seconds. code branches, container images branch, infrastructure templates, config, all of it. The database is the one thing that held us in waterfall. So we built a
[00:59] culture of workarounds around the constraint mocks, shared staging, brittle synthetic data. We learned to develop against fiction and we called it best practice. So Chanel, do you want to walk us through some of what the actual tax looks like in numbers?
[01:15] Thank you, Cam. As uh Cam mentioned um we have different type of challenges when we are uh doing our development. One thing is that it is difficult to uh fork a database or it is expensive. So what do we do? We find a
[01:33] workound about around it and with the time this workaround becomes our best practices for our teams. I would like to discuss three of these uh workarounds and discuss what are the challenges we are facing with each of these
[01:48] workarounds. The first thing I would like to uh point out is that a shared environment we develop our features and the functionality and we want to test it before we go into production. So what do we do? We uh deploy our changes to uh
[02:05] stage environment and trying to do the testing but there might be another team waiting for us or we have to wait for another team or we want if we want another
[02:21] audio. You are not getting the audio. No no I just want to make sure because I just I couldn't see. I just want to make sure. Okay. You guys can hear me correct? Um, so I believe all of these uh
[02:36] challenges you have faced. So first thing is that a shared environment you develop the your feature and you want to test it out but the problem is that you have to wait for other team to finishing in the their environment or you have to create a ticket and wait your infrastructure team to give you a isolated environment.
[02:54] What we see what we have shown is that 57% of the teams or streams waiting for another team to get their environment to test it out their features. Similarly, I want to discuss about the second uh workaround tax mocks.
[03:10] Most of the time 20 to 30% of our code base just sitting there for our mocks. Now the what is the problem with the mocks? Mocks are fictional. those are not the real uh production database or they are constraints and also they drift
[03:27] from our actual production environment. It happens for all of us. The third uh workaround tax I want to discuss is the Friday afternoon migrations. We made our feature. We uh made all the
[03:42] preparation for production but we don't have a environment to test it out our uh migrations with the actual production data. Sometimes we are lucky we will get our environment we will run it or we just
[03:59] keep that part we go to production environment just run our migration directly and what will happen? What is the blast radius? it is the production environment. People get fired for these type of uh errors. And we found that through the uh Google
[04:14] Survey 70% of outrages is caused by direct uh changes to the database in the production environment. We would like to discuss how can we overcome these uh workaround tax and Cam going to uh take us that through and
[04:32] back to Cam. Thanks Chanel. So, we've been living with these workarounds for a long time, long enough to have them show up in onboarding docs as a way you're supposed to do things. And here's the other side of each column
[04:48] that Chanel just showed you. Essentially, what flips when the database forks in just one second. The shared staging environment, it becomes a database per developer, per poll request, per hypothesis. It's ephemeral. It's isolated. and it dies when the poll
[05:05] request dies. The mocks, they become integration tests against a real branch of production. Same schema, same constraints, same data semantics, all just pennies per branch. And the Friday afternoon migration, it becomes
[05:21] destructive testing on just a Tuesday afternoon. You drop the table, you watch what breaks, throw the branch away, and production never has to know. Each of these things used to be impossible or expensive or just terrifying and now
[05:36] each is cheap cheap enough to be routine. So these three patterns reflect the three demos that we're going to run through showing an endto-end branching experience. So let's start with pattern one. A
[05:52] database per developer per poll request per hypothesis. And a quick mental model before we go into your into the terminal. Think about how your team already handles environments today. Main is production. You've got staging, UAT, test, and then
[06:10] these are the longunning tiers that everyone shares. Then you also have every developer cutting feature branches off the bottom that come and go today. The green lines, the code branch in just a second. The red lines, the data
[06:26] usually don't branch at all. And what you're about to see is what happens when the red lines branch like the green lines do. One database, one branch per environment. Same workflow you already know.
[06:43] All right, into the text editor. So to quickly frame the demo, what you're looking at is a real codebase using Vendur, an open-source e-commerce backend. And think of it kind of like an enterprise system any of you might be working with having real Postgress
[06:59] uh real schema and real product catalog. And here's the actual store admin running against production with our current live production list. So notice or note the five products here. This is our working application. For the rest of
[07:15] the talk, imagine this is your team system. So now I want to show you uh this plugin. This is the lakebase SCM extension. SCM being source control management. It's a VS code or cursor
[07:32] extension built by Kevin Hartman and his team at data bricks. It helps engineers keep code branches and database branches in sync. So it helps automate the workflow and it also provides a nice visual for our demos here. And so right
[07:48] now I'm on the main branch and as you can see productions tables I'll expand them here. This is our data the enterprise application is running against. And I'm going to run a command here in the terminal GP command just to
[08:03] take a peek at my m file. So you can see this lakebased branch ID currently set to production. So that when I run another command here to start up PG CLI, now I'm going to run an SQL command to
[08:19] select count from product. So remember the five products shown earlier, these are now reflected here in production. So I'll exit here. And before I show you the command to branch, I just want to show you what's going on under the hood real quick. So
[08:35] I'll do a a cat to see this script this postch checkout shell script. And I just want to point this out because when we create a git branch the shell hook fires and calls the data bricks uh postgress create branch which is the CLI primitive
[08:52] for lakebase. And so when we run this it pairs the git branch and the database branch together in one motion. And really with this primitive, we can build what's best for your team, whether it's using it through an extension like the one I'm showing or
[09:09] even an MCP server if you've got agents in the loop. So now let's do the git checkout command. And I'll create a new branch just called uh lakebase. Now as I do this, I want you to watch the sidebar here to the left as we work
[09:27] with the internet connection. Here we're waiting for the branch. And now you'll see my current branch is reflected to show the new database. And if I expand this, you'll see that now our feature branch has all of the same
[09:45] tables and same schema that it inherited from the production branch. And it happened about I mean depending on your internet, usually it takes about a second, but um we'll see this load here in a second. And um there we go. Now I
[10:01] just want to show you that same GP command I used. You'll notice that same shell hook updated my M file end file. So now it's pointing to this uh lakebased branch ID. Same M file just updated. So now if I go back into uh the
[10:18] PG CLI just to complete the demo one I'll I'll run that same count and you'll see same five products just a branch of production totally separated. So just want to call out this isn't some black
[10:34] box. This is all visible in the data bricks console alongside what you're used to and queryable from your lakehouse. So that is demo one. So just switching back to the slide deck
[10:51] and here we go. So just a quick recap. Uh what we just showed you isn't a faster version of provisioning. It's the moment provisioning stops being a a thing your team waits on. And the branch is
[11:10] actually just a metadata pointer at production storage. So no bytes are actually copied until you write. And that's actually the architectural unlock here that everything else depends on. So now Chanel, do you want to take the audience through the next demo once that
[11:28] branch exists? Yeah, thank you Cam. Take that. So I'm going to talk uh discuss about the second uh tax workaround tax I
[11:43] discussed in the second one we were talking about uh mocking and how we can get rid of the mocks from the our codebase and I also want to show how we evolve through this process.
[12:00] Let me show you what I have here. Here you can see we have three different type of variations of uh testing. I'm going to uh trigger this one testing just to make sure save some time.
[12:16] So the first one uh what we have is the uh SQL uh JS. Uh we are running through the SQL light database and I also want to show what it is is running here. You can see there are five scenarios including uh four of them are uh reads
[12:32] and one is uh creation as well. So we have the SQL light. So in the SQL light we don't test again the actual pos database and also it is not against the uh actual constraints or the posgress
[12:48] functionality. So this is a an experience I I myself faced uh in my uh current project. So what I did to overcome this uh limitation is that introducing a test containers. You if you are familiar
[13:04] about test containers in the your local environment you can easily bring up a postbase instance and you can run your local uh test against the test containers against the actual postgress instance. That is our second evolution. How
[13:21] again that posgress instance is runs against a new schema, new data that we mock or we uh seed it to that instance. Can we do better? The next evolution is that can we run our test against the
[13:38] production database that we just branched. That is what you are seeing in the uh this scenario. So this is test is running against the uh production branch itself. So same three uh test here tests are no
[13:57] difference also we are uh mutating the production data without being worry about what will happen to production environment data because it is isolated. Same use cases we are running against production database and verifying that our feature works.
[14:25] I want to recap what we just did. We had SQL light, we have postbrace but both of them are not uh testing against the reality. The only option in this scenario and only option that gives actual confidence for us that our migrations or the uh functionality will work in the
[14:40] production is the lakebased branch solution. With this one I would like to go to the next scary one. As I mentioned the previous uh when I'm running the test I am doing mutations on the production environment.
[14:56] It it was not possible before because we cannot mutate the production environment. But now with the branch we can do mutation into the production environment even we can actually uh do the destructions. So let me uh show you.
[15:21] So here what we have is that as you can see uh five uh counts are here. I'm going to drop the table.
[15:41] and it dropped the uh table itself to make sure that uh it is actually dropped the table. You can see uh it is now in the uh no table exist. What? How? Just to make sure that I did not uh drop
[15:57] the the main branch. We I did the destruction on the uh our feature branch. And if I switch it to the main branch
[16:29] source yeah connect to PGCLI and if I check the uh count again as you can see in the production uh branch we still have the data as it is same is applying to the application as well. In the application, nothing changes. This is our production application
[16:45] with this one. I would like to uh recap what we have done. So what stays in intact? So production environment untouched even though we can uh did a destructive action. We were
[17:02] testing our uh mut mutable uh actions against the production again production uh branch unintact. We can drop our tables. We can do anything we want in our branch without impacting the production environment.
[17:18] With that one I would like to pass on to the cam to discuss about what can you do or what can you learn from this and applying next Monday.
[17:37] Uh before we get to what you can do with all this on Monday morning, one thing I think that's worth saying about where this is headed, uh everything you just watched, we built for humans. A developer's loop gets tighter, a tester life gets cheaper, a migration becomes safer. Well, now we have agents and
[17:55] they're already working together with us and they need exactly the same thing a developer needs, which is isolation per task. And that's exactly what a branch can offer. So that spec file that Chanel shared in demo 2 uh that had five
[18:11] scenarios in plain English, that's the artifact that agents can read and write to. And the specs are just nicer test format and it's how agents and humans can actually share intent. So um the interface is the next wave of agentic
[18:29] contributors to get work done. So that's where we think this is going. So given all that, let's talk action items. So we've had three demos, three patterns, and three things you can do on Monday. First branch your next
[18:47] migration. Whatever's queued up for the next sprint, try running it against a real branch with real production data before it goes anywhere near production. And the change failure rate uh metric in Dora. It exists because the highest cost
[19:04] bug in any system is actually the one that you find at deploy time. And now you can just completely eliminate that. Second, pick one mock heavy test suite, take note of how much mock setup there is in that file, and then rewrite assertions
[19:20] against a lakebased branch, and the lead time actually drops because the mock maintenance t tax disappears. And this is also the workflow that makes uh real test-driven development against production shape economical for the
[19:35] first time. Uh third, reproduce a bug. a production bug on a branch. So right now when something breaks in production, you try to write a fix that matches the symptoms of production. Um so now I want everyone
[19:52] to try to go to a branch at the database at the moment of failure. You can use the point in time recovery. Um try and reproduce the exact bug, validate the fix on the branch and then you can ship it actually knowing it fixes the thing.
[20:09] So this means meanantime to recovery dorometric can drop because you have the real data and your business is pleased as well. So here's a short list of resources and links for your reference after this talk
[20:25] and yeah Chanel. Yeah. So data data lake uh branching actually help us in our real life in our real projects. It help us to improve our productivity, improve
[20:40] our uh developer experience and most importantly it help us to improve our confidence to go to production. I hope you all can uh experience the same by adopting the same uh strategy. Thank you very much. Uh if you have any questions,
[20:57] we at the thought works uh booth there. Please uh come and meet us. Yeah, visit us at the thoughtworks booth. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.