Lakebase: PostgreSQL OLTP and Zero-ETL for AI-Native Development

Summary

Lakebase is a fully managed PostgreSQL service that came to Databricks through the Neon acquisition, built on a separation of storage and compute architecture that integrates directly with the Databricks Lakehouse through zero-ETL.
The LTAP announcement enables Lakebase to sync PostgreSQL data continuously to Delta or Iceberg tables in the lakehouse, giving AI agents and analytical workloads access to production data without requiring separate ETL pipelines.
Database branching creates instant, full-fidelity isolated PostgreSQL environments in roughly one second with zero initial storage overhead, enabling safe experimentation, agent memory testing, and CI/CD workflows.

Lakebase: PostgreSQL OLTP and Zero-ETL for AI-Native Development

Lakebase is a fully managed PostgreSQL OLTP database engine built for AI-native development and lakehouse-integrated workflows. Unlike traditional databases, Lakebase separates storage and compute and integrates directly with the Databricks Lakehouse through zero-ETL, eliminating the need for complex pipelines. Hear from Bryan Clark, Lakebase product lead, on how this architecture radically simplifies enterprise data management with lower TCO and open-source flexibility.
Learn how Lakebase enables AI agent architectures through database branching, allowing instant creation of isolated environments for testing, experimentation, and CI/CD workflows. The LTAP announcement enables seamless syncing of PostgreSQL data to Delta and Iceberg tables in your lakehouse, giving agents access to high-fidelity production data while maintaining governance and isolation.
🤝

Chapters

FAQs

What is Lakebase and how does it relate to the Databricks platform?

Lakebase is a fully managed PostgreSQL database service that came to Databricks through the Neon acquisition, built on the same storage and compute separation principle that underlies the Databricks Lakehouse. It runs standard, unchanged PostgreSQL while integrating with the Databricks platform to give developers a database that works seamlessly alongside the lakehouse.

What is LTAP and what does it enable for data teams?

LTAP enables Lakebase to take the separated storage layer of a PostgreSQL database and sync it continuously to the lakehouse as Delta or Iceberg tables. This means AI agents and analytical workloads can access high-fidelity production data from PostgreSQL without the complexity and delay of building separate ETL pipelines.

How does database branching in Lakebase benefit AI agent development?

Lakebase's copy-on-write branching creates instant, full-fidelity database clones in roughly one second with zero initial storage cost, giving each developer or AI agent its own isolated environment. This enables safe experimentation, testing of agent memory changes, and CI/CD workflows without risking the production database.

Why does the separation of storage and compute make Lakebase a good fit for AI agents?

Storage and compute separation is what makes both instant branching and lakehouse integration possible, allowing Lakebase to create clones by referencing shared storage rather than copying data physically. AI agents accelerate and amplify what developers do, so a database architecture that can keep pace with agentic workflows is a natural fit for building AI-native applications.

Full transcript

[00:20] All right, welcome back to Summit Live. I'm for those just tuning in, I'm Ari Kaplan, uh, head of evangelism at Data Bricks, and we have Maria and Brian Clark. Yeah. So, welcome. Welcome, Brian. Let's start with what is Lakebase and what is Zero ETL? Yeah. Uh, so Lakebase is a Postgress
[00:37] database service, right? Uh it's part of the Neon acquisition and that's where I came from. I am one year into my data bricks journey right now. Welcome. Yeah. Yeah. Because I was a VP of product at Neon. Uh and uh yeah, we we set out to build a service that would be the best
[00:53] database for developers. Uh and then just happened to find out that that database that branches and has storage and compute separated in order to make that happen is also the best database for AI agents because they just accelerate what developers do. Uh so you
[01:09] know it has to be fast. Uh and this separation of storage and compute is also kind of what makes it a real good fit with data bricks, right? It's what data bricks did with the lakehouse and it's it's why they looked at Neon and said like that's the right platform for us. I love all of those topics. I love geeking out. Um but one question that
[01:26] comes to mind is like what does Lakebase do that enables you know with a workflow that just a standalone Postgress database doesn't? Like how does data bricks ecosystem help? Yeah. Uh so lakebase is just pure Postgress. One of the things we set out
[01:42] to do was not change that because if you go to a service that has what we call like a Frankenstein Postgress where parts of it are there but not developers often hit one of those things that's missing and then they're like oh we chose this database so we're kind of stuck with it now but these pieces don't
[01:57] work. Uh so we have core Postgress no like unchanged from the Postgress you would download and then we've built all this tooling around it. Uh the lakebased platform is basically a really custom file system for Postgress and that's what enables branching and then we also integrate
[02:13] with the data uh data bricks platform for lakehouse uh data sharing. Uh if you saw the LTAP announcement loved it really cool right that's uh the ability for us to take that separated storage piece and then sync that to the lakehouse. So we just uh transfer all of
[02:30] your Postgress data and we turn it into Delta or iceberg data. Uh, and those are the pieces that's like we're integrating with everything that data bicks can offer to make a better database. It's the core reason why data bricks has this is that if you have a database that exists elsewhere, you end up with all
[02:46] these pipelines to try and kind of connect it to your environment, but we're able to just sync it directly with everything. So me personally, I'm coming from an AI world. So I want to understand a little bit more why lake based for agents and maybe agent memory even. Yeah. Oh yeah, I'm doing a deep dive on
[03:03] memory. So, we're going to have a lot to talk about. Sweet. Uh, and the AI, we only have about 2 minutes left. We only have 2 minutes. That's fine. Uh, but yeah, AI agents, like I said, with uh developers, uh, being able to branch the database was a an effort to make them integrated with their CI/CD and
[03:18] allow them to work like git. So, they have like an isolated environment. And then we were uh, you know, Neon and Lakebase is the backing agent uh, database for Replet and their agent system, right? And that's because they fan out agents and they give them each like an isolated copy of the database
[03:35] and you know so six agents all work together to try and solve the problem. One supervising agent will choose the winner and then they toss all the remaining work. And if you try to do that in a traditional environment like databases take 10 20 minutes to spin up sometimes. Totally. You wouldn't want
[03:50] like yeah six agents waiting 20 minutes each and then copying all the data and all these things. So yeah, it's the instant branching and these isolated environments. That's what makes it really powerful for AI agents. And then the same for um agent memory. Very similar. We have a demo where we kind of
[04:06] poison the memory of an agent. So you can really like run tests and try and figure out what if I change the memory, what if I alter the embeddings or the indexes and what would that do to my agent and run the eval on that environment. So the branching there is really fascinating to kind of mess with
[04:21] people's memory and also try not to think about that in terms of humans. Very interesting. I put it off. They're giving us the hook. But thank you so much. You are amazing. Lake Base is amazing.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.