Lakebase: Building AI Apps with Serverless Postgres

Summary

Lakebase is a serverless Postgres database built on Databricks that is 100% Postgres compatible, using a separated storage and compute architecture to enable instant branching, sub-second database creation, automatic scaling, and pay-for-what-you-use pricing.
PG Vector support enables semantic search and agent memory natively within Lakebase, and the branching capability gives AI agents isolated execution environments per task for safe concurrent workflows.
By combining OLTP and analytics on the same Databricks platform, Lakebase eliminates the custom ETL pipelines that would otherwise be required to synchronize separate operational and analytical databases.

Lakebase: Building AI Apps with Serverless Postgres

Lakebase is a serverless Postgres database built on Databricks, combining Postgres compatibility with cloud architecture. Separating storage and compute enables instant branching, sub-second creation, automatic scaling, and pay-for-what-you-use pricing.
Learn how developers use branching for safer development, how AI agents leverage PG Vector for semantic search and isolated execution, and how SaaS companies build multi-tenant apps with automatic cost scaling. See why combining OLTP and analytics eliminates custom ETL complexity, and how Replit and Superhuman use Lakebase to simplify app operations.
📂 GitHub repository: https://developers.databricks.com
🤝

Chapters

FAQs

What is Lakebase and how is it different from regular Postgres?

Lakebase is a serverless Postgres database built on Databricks that is fully Postgres compatible, meaning developers use the same connection strings and extensions they already know. The key difference is a separated storage and compute architecture that enables instant branching, sub-second database creation, and automatic scaling to zero — capabilities traditional Postgres cannot offer.

How does Lakebase support AI agent development?

Lakebase supports PG Vector natively, enabling semantic search and agent memory within the same database developers use for their application. Agents can also use instant branching to get isolated execution environments per task, making it safer to run concurrent agent workflows without interference.

Who is using Lakebase in production?

Replit and Superhuman are among the companies using Lakebase to simplify their application operations. Lakebase is designed for SaaS builders who need multi-tenant database patterns with automatic cost scaling as their user base grows.

How does Lakebase eliminate ETL complexity?

By combining OLTP and analytics on the same Databricks platform, Lakebase removes the need for separate operational and analytical databases that require custom ETL pipelines to stay synchronized. Teams can run both transactional and analytical workloads against the same data source.

Full transcript

[00:20] All right, we're back. This time Maria and I are joined by Andre, who's joining us from formerly the Neon team but now the Lakehouse team and Lakehouse is if you haven't heard about it yet at Summit, it's one of the most important products we've been working on at Databricks and it's something we're super excited about. So, Andre, would you mind
[00:37] introducing yourself a little bit and like what your role at Databricks is and and how you've been working with Lakehouse? Yeah, absolutely. So, I joined Neon a few months before the acquisition as a developer advocate and through the acquisition I joined the advocate advocacy team here at Databricks and I've been mostly focused on the
[00:53] developer audience or anybody who's building on the Databricks platform. That's who we try to like create content for. And then yeah, so at last Summit I think Lakehouse was really like introduced and now it's the second Summit. In the first Summit I have to work for Databricks. Last time we were just almost still guests, right?
[01:10] I think you find that side of that. Yeah. It's more work. But yeah, it's like incredible the scale here obviously like the scale of the keynote and everything is still mind-blowing. Wow. Yeah. Excellent. Then can you introduce a bit Lakehouse then and why I guess Postgres on Databricks versus no?
[01:25] Yeah, absolutely. So, the good the best thing about Lakehouse is that it's just Postgres. So, if you're building an application, you would still just use that database connection string, plug it into your application. It speaks Postgres. It is 100% Postgres compatible and any
[01:41] of the extensions that you like from Postgres like PG vector, you can just use them with with Lakehouse as well. So, that's a great thing for the application. The problem is like with Postgres is though it's a very old platform and it's it's very popular and what's great about it is it's open-source. The problem is
[01:56] though it's not really set up for success in a like modern dev environment just because it's like a long-running server, it's very heavy, and so what what we did is we split uh storage and compute and create like a serverless architecture. If you're using it, you don't really
[02:11] have to care about it, right? In the end of the day, it's the Postgres, but behind the scenes, we really changed how it works. Um and that enabled us to do some really great things, most importantly, auto scaling and branching, which really like enables this amazing developer experience. Yeah, so how was um if you're a developer, how was your experience or
[02:27] how was your your workflow changed? Yes. Cuz now you have the separation of compute and storage. What can you do today that maybe you couldn't have done before? Yeah, absolutely. So, historically, what you would do when you when you're developing against the database, right? Like you would probably set up Docker locally, run your Postgres
[02:42] database locally, and then you run into sync issues, right? Like the the seed data doesn't work anymore because someone changed it, and then like uh Sure, today coding agents can you know, fix that real quick and massage it. Um but it's way cooler if your platform just like has those primitives for you
[02:58] and you can just leverage them. And the the biggest primitive that we offer there is branching. So, that is just a way for you to instantly copy So, branching like I would with like Git or something like that? Absolutely. So, it's exactly like Git. Um the problem is though, you can't merge back, so you still like have to
[03:13] apply the same migrations to your production branch after the fact. But yes, every time so in my agents or in D file, my agents my coding agents I tell them to first create a Git branch, and every time you create a feature branch, also create a Neon database branch or Lake database branch. Oh, excellent. And the amazing thing about that is that
[03:29] it is both copying the schema and the data of your database. Nice. But it's copy on write, so even if you have a very big database, it still still does it almost instantly. And then why this branching is actually really relevant on this agentic era where I guess agents create
[03:46] Yes. Yes. database as well. So, initially, we built branching for developers, right? Like we just wanted to make that development workflow much easier and nicer, but turns out if you have a great developer experience, that usually translates into a really good agent experience as well. So, if you give each agent an isolated database
[04:02] branch, right? Like they can't mess up the production database, they can experiment stuff, and then afterwards you review the changes, and then you decide if you want to promote it, right? Like same way you would do that in Git as well. Oh, excellent. And then I mean, one of the things that's novel is the the separation between storage.
[04:18] Yeah. But the other thing that I thought was really interesting whenever whenever we first uh started working with you was um there's the sheer speed at which you can do pretty much any operation on Lake Base. I mean, it's like almost instantaneous, right? Yes. Yeah, no. Yeah, if you used to other Postgres platforms and you create
[04:35] a new instance, it may take minutes until that is like done. Um which if you like create your first project, sure you can go grab a coffee, but let's say like something bad happens, right? Like you have an outage, you want to test, you need to like quickly spin up a new environment to like see what happened, then 5 minutes
[04:50] that feels like a very long time. minutes or longer, right? Um and what's really great about the Neon platform or the Lake Base platform is um that the compute and storage works separately from each other. So, if you spin up, you already have compute running, and then if you want to assign a new uh Postgres
[05:05] instance to that, right? Like it takes less than a second to spin that up. Is uh so if it takes less than a second to spin it up, that means could you uh for auto scaling, does that mean it's it's equally fast? And like if if you can you scale all the way down to zero? Yeah, yeah, yeah. Yeah, so absolutely. So, auto scaling and then scaling to
[05:22] zero is like kind of an edge case of auto scaling. Um so, we detect how much traffic there is on your database. Let's say you run PG Vector and you do semantic search, right? And like it's just a little bit more compute intensive. We detect that and we scale up your compute. But let's say you have a dev branch or
[05:37] you have a branch for your agent, and the agent is just not working anymore, uh then you don't really need that compute to be running, right? And that's exactly where scale to zero would come in. Uh we always say like scaling to zero is easy, but scaling back up is the hard part, right? Uh so, yeah, scaling is uh to zero is great for uh
[05:54] cost performance, but then uh once you come back and you want to work, you also don't want to wait 5 minutes, right? You want to wait less than a second. So, creation of a database and spinning up or like waking up a database is almost the same uh action for us, so it's like both both of the times less than a second.
[06:10] Wow, that's amazing. Yeah, so we spoke about how we can use actually branching when we build with agent, right? Yes, but how can we use Lake Base for memory for agent? Yes, great question. So, there's like two types of agent use cases for Lake
[06:26] Base, I would say. The first one is you have a coding agent that I use to build my application, and that agent builds my app, and then provisions my database for me, manages the database for me, right? And we have those code gen platforms like Replit where that is also happening somewhere in the cloud, and they're
[06:42] using Lake Base behind the scenes. But then uh there's this other use case where you want to use Postgres or Lake Base as the database that your agent uses for its own state and memory. And there, obviously, PG Vector is very important, right? You want to be able to use semantic search, so you don't have
[06:58] to do like a full-text search through your data. So, that's really important. And then generally, Postgres has just become kind of the default for agent memory in a lot of places. Again, since Lake Base is just Postgres, anything you would do with any other Lake Base definitely also works. Cuz PG Vector is like like a Postgres plugin, right?
[07:13] Yes, it's an extension, yeah. So, you can do any extension on Postgres? Yeah, so yes and no. So, uh because Lake Base is compatible with Postgres, we can make every extension work, and we already support like a vast majority of them, but every extension we add uh is something we have to make
[07:29] maintain and make available in the platform, so you can like install it, right? So, like for us, it's always like a choice, are people like do people want this? But like the most common ones, you know. Yes. Those are the obvious ones. Or if you're a customer, you ask us, right? Like we have to evaluate and then Yeah. Excellent. you are out there in the community a lot talking to a lot of developers and this
[07:46] is actually the person we are trying to target as well. So, what are the most funny, popular misconceptions you heard about Lake Base as a product for them and then for AI? Yeah. So, like serverless Postgres, which is also how we call Lake Base, right? It's like definitely a new concept and
[08:03] I mean, in the Databricks world, there's a lot of things that are serverless. We just announced serverless GPUs, right? So, I don't know if that's things here. But like outside of the Databricks world for app developers, the only thing that is really serverless is like a Lambda functions, right? So, like and they behave very differently from a
[08:18] serverless database. So, there's a lot of misconceptions there, right? Like we don't run like parallel databases. It's still one compute, right? Talking to a multi-tenant system of storage. That's like one thing that people kind of get wrong. And then the other thing is, yes, we support scale to zero and yes, that
[08:34] is very great for dev environments, but it doesn't mean you have to scale to zero. So, it's a configuration setting that you can disable. Like for a lot of production use cases and scenarios, you actually want to have like an instant connection even if there was like a little bit of a, you know, no time without a connection for like bursty use
[08:51] cases or like an e-commerce store, right? You don't want to scale that to zero at all. Yes, yeah. Yeah. And yeah? Go ahead. So, I just want to ask and what is the message we need to learn to all of the developers who are building apps and other systems to bring them to Lake Base, you know? Because people who are
[09:07] listening to us maybe are more data practitioners than dev practitioners and etc. This is a new group of people. Yeah. What's the message we need to transfer to them? Yeah, absolutely. So, building has never been easier, right? Like it's insane how like I'm about to build an internal tool to
[09:23] manage my internal tools because I have so many internal tools. And like each of them needs their own database, right? So, it has never been easier to build apps, but apps and even agentic, specifically agentic apps, they're really not that powerful without access to data, right? So, the main problem is
[09:39] still like how do I get all my company data or the one that I need into my application? And that is really hard if you have different separate systems. And it gets so much easier if there's just a button or one CLI command that you can call to sync the data from one space to another,
[09:54] right? And we So, in the keynote we saw a lot of exciting announcement even with how we can make that split even more seamless. Seamless, yeah. Uh so, that is like the the core message is like if you want to build apps and agents, they need access to data. So, it makes a ton of sense for those apps and agents to be in the same
[10:09] environment and platform then as your data. You bring up a good point though because a lot of times you're using a database like Postgres as the backing store for like an application, but you want to do like a downstream analytics from that and you know, historically you're building these ETLs to move the data from the the database to something like
[10:26] Delta Lake so that way you can run the analytics on it, but you're not like trying trying to run those same analytics yeah, which is a heavy load on the database itself. Yeah. And so, Lakehouse has kind of simplified this whole process, right? 100% yeah. You get rid of your custom syncs, right? So, like one great example for that is our customer Superhuman who
[10:43] did exactly that. So, they got rid of all their custom sync stuff and replaced it with a Lakehouse and Databricks asset up. What is the How much complexity that it does this does this actually solve, right? say a ton. Right, yeah. Like
[10:59] What is this What do people say about like that what's hard in computer science? It's like caching, naming things. I would add syncing. Like syncing stuff like consistently without issues is definitely in the top three, top four problems for sure. Bravo. Nice. Excellent. What Yeah, what are what are the some of the
[11:14] things that we haven't talked about that you feel like, you know, the one thing that people shouldn't shouldn't shouldn't miss when they're leaving this talk? Yeah, so I feel like historically people especially in like data teams, right? Like they look at notebooks as a solution or a dashboard as as to
[11:30] represent their data. Yeah. I would make the pitch now that you have like cloud code and we we just announced a Genie app builder, right? Like you have all of these great tools to build applications. We have a partnership with Replit. It has never been easier to just build an app instead of a dashboard, right?
[11:45] Yeah. And what it what an app allows you to do is just like the second you want to give that to a different team, it's just a URL. Mhm. And then the second they want to do something with it like you don't just look at the data, you can write to the data as well, right? And then LakeBase comes in again as the the data store that's powers that behind the scenes for
[12:01] the like fraud overrate operations. And then from there the data team can use that data again, sync it back into the lake, use it for another set of analytics or another pipeline. And it's just like it closes the gap for me because analyzing data is fun, but you really want to operationalize it, right? And
[12:17] and that's where LakeBase comes in. For me it's not really relevant to LakeBase, but one of the most fun applications I've seen from business users actually slides building. Oh, yes. Really? Yeah, yeah. Oh, okay. Right? Is so instead of having your slide, you go on a Databricks app and
[12:32] you just have your HTML page like Yeah, I've done that before, too. Yeah. Looks so much better. Yep, yep. I understand. I work a lot with built-on customers and some of them are experimenting with like basically creating a separate database for every customer tenant. Yeah. And then that's beneficial because I mean
[12:49] one it just sits it's an easy way to isolate things, but but two, you know, you're never you rarely have all of your customers using your application that all the time. And so for some there's going to be a lot of lumpy lumpy access. And so the ones that aren't using anything they can scale to zero and the other ones can scale up. And because
[13:05] you've isolated the data to one database, you don't have to do big scans every time you run a query. So Yeah, I understand. Yeah, so like multi-tenant scenario or like use case like that are super exciting to us. Um and I think they get really enabled by the uh auto scaling architecture that we
[13:20] created, right? Because um so basically anything that you can do in the LakeBase UI, you can also do it our APIs. So you can programmatically spin up new databases for your customers and then like run your seed migrations on top of it, like create the schema, and delete them again, transfer them
[13:36] from a one account to another account. Like all of that works really like autonomously. Um or you can do it through the APIs, right? And that's really important for for these kind of use cases. Yeah, excellent. Cool. Yeah, last question from my side is, can you explain a little bit the
[13:52] devrel role? What does this actually mean? And What do you mean get devrel? Yeah. Yeah, I feel like you could go to any company and ask a devrel what they do and they would all give you very different answers, right? Like, "Oh yeah, I'm a content creator." Like, "No, I write docs." "No, I'm I am responsible
[14:09] for writing and authoring the SDKs." Right? Like any devrel would tell you something differently. And that's the exciting thing about the job, right? Like it's very diverse. Um but yeah, so here at at Databricks, uh it's we say it's like all about the community, right? Like where someone who's in field would probably talk to
[14:24] one customer one-on-one, we try to broadcast. Right? And that can mean a lot of things. Uh some people, like Holly and Nick, are really great on our team and write a build doing like podcasts and YouTube videos. And then I am located here in the Bay Area, so I do a lot of meetups. I go to conferences. I speak to
[14:40] developers and give like meetup talks and that kind of stuff. Um and yeah, and yeah, we we create examples and anything the community needs for us. Amazing. Excellent. Well, we got like about 1 minute left. So, if you just if you had to have any parting words for the audience today, like what would you what what would you want to leave them with
[14:55] for like this? absolutely. Yeah. Uh shameless plugs allowed? Yeah, of course. Uh we just launched developers.databricks.com Nice. site for builders. If you want to build on Databricks with Lakehouse, with Databricks apps, it's like the one stop shop for you. Uh if you go on the
[15:12] landing page, the first thing you see is like a copy prompt button. Yep. Click that, paste it into your coding agent of choice, and from there it will take care of building your first app for you. App and Lakehouse instance behind the scenes? Absolutely. Excellent. Yeah. Like base by default. Yep. Yep. Yep. I love to hear that. By default, yeah.
[15:27] All right. Well, thank you so much, Andre, for joining us. Um I really appreciate it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.