Introduction to Lakebase: OLTP for Data Apps and AI Agents

Summary

• Transactional Data Layer: Lakebase brings fully-managed serverless Postgres to power data apps and AI agents, filling the gap between OLAP and OLTP workloads.
• Unified Data Storage: Data stays in one Databricks account with synchronized tables between Lakebase and the lakehouse, eliminating cross-system data movement and custom pipelines.
• Production-Ready Apps: Build internal tools, customer apps, and AI agent backends on Lakebase with the reliability and scalability needed for mission-critical systems.

Introduction to Lakebase: OLTP for Data Apps and AI Agents

In this video, you will learn how Lakebase brings fully-managed serverless Postgres to the Databricks platform to power data apps and AI agents, fully integrated with the lakehouse.
🎯 What You'll Learn:
• What Lakebase is and how it fits into the Databricks ecosystem
• Why Lakebase uses Serverless Postgres and what that enables
• How to use Lakebase for data apps, internal tools, and AI agents
⏱️ CHAPTERS:
0.58 OLTP vs OLAP: The Gap
📚 Resources:
• Caspers Kitchen GitHub repo: https://github.com/databricks-solutions/caspers-kitchens
• Blog: https://www.databricks.com/blog/how-use-lakebase-transactional-data-layer-databricks-apps

Chapters

• 0:00 Why Lakebase Matters
• 0:40 What Is Lakebase
• 1:28 How Lakebase Bridges the Gap
• 1:43 Lakebase Architecture
• 1:56 Why Postgres
• 2:33 Serverless Postgres Explained
• 2:58 When to Use Lakebase
• 3:41 Demo: Casper’s Kitchen Walkthrough
• 6:45 Reverse ETL with Lakebase
• 9:35 Wrap-Up & Resources

FAQs

What is the difference between OLAP and OLTP, and why is Lakebase needed?

OLAP systems (like data lakes) optimize for analytical queries, while OLTP systems (like databases) optimize for transactional consistency. Traditionally, data had to cross system boundaries to sync between them. Lakebase bridges this gap within one unified Databricks account.

How does Lakebase maintain data consistency between the lakehouse and the database?

Synchronized tables allow you to decide what data is synced between Lakebase and your Databricks lakehouse. Data stays in one unified system with Unity Catalog governing access, eliminating custom synchronization mechanisms.

What is an OLTP database?

This video introduces Lakebase on the Databricks Data and AI platform as an OLTP (Online Transaction Processing) database. Unlike data warehouses optimized for analytics, OLTP systems prioritize fast read/write performance, ACID transactions, and consistency. Lakebase combines OLTP capabilities with lakehouse features like branching, versioning, and integration with Databricks analytics and AI tools.

What database to use for AI apps that need real-time data?

This video demonstrates Lakebase as the ideal database for AI applications requiring real-time data access on the Databricks Data and AI platform. Lakebase provides the transactional consistency and low-latency access that AI agents need while maintaining integration with the broader Databricks ecosystem for model training and analytics. The serverless architecture scales automatically with demand.

OLTP vs OLAP?

This video contrasts OLTP (Online Transaction Processing) with OLAP (Online Analytical Processing) on the Databricks Data and AI platform. OLTP systems like Lakebase handle frequent, small transactions with consistency requirements; OLAP systems handle analytical queries on large datasets. Lakebase bridges both worlds, enabling operational data management while remaining accessible to analytics and AI workloads through the unified lakehouse.

Full transcript

0:00 Why Lakebase Matters

Agentic systems are top of mind right now to automate processes. However, for that your agents need access to your company data and usually in real time. Likewise, you also still want to have a human in the loop to make the final decision. And for that, your human operators also need access to your company data, usually through internal tools where they can read data to gain insights, but also make updates to the system.
And that's where Lakebase comes in. Lakebase is your transactional data storage for your data and AI apps. and you can build some really cool stuff with it. I'm a developer advocate at data bricks and today I want to give you a complete walk through of lakebase what it is when you would want to use it and how it looks like to use lakebased today. Let's get started. First things first, lakebase is actually two things because why not?

0:40 What Is Lakebase

First of all, it is a new architecture for an OLTP database. But then it is also a new service on data bricks that builds on that architecture. What makes the lakebased architecture special is the deep integration with the lakehouse. Traditionally, there's a gap between your OLAP system and your OOLTP systems where data has to cross system boundaries, network boundaries, and sometimes even cloud boundaries to get moved from one system to another.
And that's just complicated to get right. You need custom pipelines and synchronization mechanisms. Have to figure out security. And there's no surprise that most companies report that their employees do not have access to all the data they need. It's difficult. Lakebase is meant to be another tool in your data bricks tool set to address this issue. With lakebase data stays in one unified system, your data bricks account.

1:28 How Lakebase Bridges the Gap

With Unity catalog, you can decide who has access. And with synchronized tables, you can decide what data is synced between your lakehouse and your lake bases. Next, let's talk a little bit deeper about the architecture behind Lakebase. It's not built on a new database management system. Instead, it's built on Postgress and more specifically a serverless Postgress architecture. So, let's see what that exactly means. If you're an application developer, you're already familiar with Postgress.

1:56 Why Postgres

It's a battle tested, open-source, and very popular OLTP database with a thriving community of contributors, users, and extensions. Take PG Vector for example. With PG vector you can add vector search capabilities to your Postgres database just by adding an extension and with that you can do semantic search over your data which is perfect for use cases like rag. Postgress is quickly becoming the most popular OLTP database just like git for version control.
Postgress will be the default for building applications. But lakebase is not just Postgress it's serverless Postgress. One of the most fundamental principles of the lakehouse is the separation of storage and compute and serverless Postgress is bringing that to Postgress databases. With the separation of storage and compute and serverless Postgress, we can enable some really awesome features for an amazing developer experience like database branching, point in time recovery, and even time travel queries.

2:33 Serverless Postgres Explained

We'll have more to share about these features very soon. All right, so when would you want to use lakebase? Lakebase is ideal anytime you want to build a transactional system on top of your data and AI stack and there's three use cases that come top of mind for me. The first one is if you want to build internal tools, right? You want to make data accessible to your co-workers throughout your company and you want to give them access to a system where they can actually make changes as well.

2:58 When to Use Lakebase

That's when you would add lakebase on top of the lakehouse. But you can also use it for agent memory and state if you want to give your agents a low latency data storage to access during their workflows. And finally for real time feature serving for your ML models. Let's say you want your models to have access to real-time data then you can also consider using lakebase. All right let's walk through an example how using lakebase looks like today on data bricks.

3:41 Demo: Casper’s Kitchen Walkthrough

I have a fake company here called Casper's Kitchen. And Casper's Kitchen is a ghost kitchen company invented by my co-orker. And what the setup is is that we have data coming in from our order system where people make orders and then we have drivers pick up the deliveries and drive them to our customers. And in data bricks, that data is then organized and then channeled through a refund recommener agent. So that agent looks through all the order details and decides if we should do a partial or full refund of the order.
But we don't want the agent to make the final decision here. We want to have a human in the loop. That's why kitchen also comes with an internal tool, a databicks app where we display all the recommendations by the agent and then a human can take action and either apply or deny those uh recommendations. Let's see how kitchen could use lakebase to build an internal tool for refund management. All right. So now we are in the kitchen workspace.
And the first thing I want to show you is this model serving endpoint we have here. Caspers refund agent. So this agent we can send it event data and then the agent decides if there should be a partial or full refund for a fulfilled order. Uh so imagine an order arrives very late to our customer. then the agent may decide that it's better to give a partial refund to keep the customer happy. So when we're using this caspus refund agent endpoint in one of our pipelines.
So if we go to our pipelines, we can see there's a refund recommener stream. And if we go to the tasks, we can see there's just one task. And if we look at the source code for this one, we can see that we are here defining a get chat completion lambda function where we make a request to that model. And then down below, we can see that we are piping all incoming events through the stream here. Um, but we filter out only the events that have been already delivered because there shouldn't be a refund uh decision made on orders that are still pending.
Maybe the kitchen is still working on them, right? So only for delivered events, we're then calling our get chat completion function. And the agent response is then stored in the refund recommendations uh table. So let's look at that table next. So if we go to our catalog and look for Caspers, we can see here we have a recommener schema with two tables and one of them is the refund recommendations. And if you look at the sample data, this is how it looks like.
For each incoming delivered order, the agent will at some point make a decision. And you can see here that the past few decisions have been made to not give any refund which is very good for our business because it means all orders have been within reasonable delivery times. At least that's what the agent decided to recommend. And you can also see the agent gives a reason here uh why the decision was made that way. So there should not be a refund for this order ID because the order was not late.
That looks great to me. What's really exciting is this PG recommendations table over here. Um, if you over over this here over this label, you can see it says it's a synced table. And the source of the sync table is the refund recommendations table that we just took a look at. And it's then saving it in the caspers refund manager. And that's a lakebased database instance. So if you click on this one, we end up in the compute tab under database instances.

6:45 Reverse ETL with Lakebase

And this is our lakebased instance. Here we can see our configuration. Uh we also have access to the connection details if we want to and can review the catalog. So if we jump to the catalog, we're back at the catalog. So if I look for Caspers again, you can see we have Caspers and then the Caspers refund manager and that's a catalog for the Postgress database here. And PG recommendations pops up here as well. So this is the synchronization what we call reverse ETL from the lakehouse to the lakebase instance.
And here uh if you look at the sample data that's exactly the same data we also see in the um in the refunds recommendations table. So here we see all the agent responses again but now we have access to this in our postcus instance for low latency queries and to make quick updates to that data. All right. You can also see here is a refunds decision table and this one is just a traditional Postgress table um where we save the decisions that have been made by the human operator.
So the human operator has access to a tool where they can review the agent recommendations and then act on them and anytime a human makes a decision the decisions end up in this table. So next let's take a look at that application. So if we go into compute, you can see here lakebased Postgress. This is where we saw the lakebased instance that we reviewed earlier. But we also have an apps tab. And here we have a refund manager application.
And if you click on this link, this is what the application looks like. And you can see here uh I listed all the suggested uh refund recommendations of the agent. And you can see we already took actions on a few of them, but there's one left where we haven't taken an action just yet. So, what we can do here is click apply uh review uh the amount we want to refund, decide if this is a partial or full refund. Here we can just follow the recommendation from the agent and say uh this decision was made by me and apply.
And just like that, we read data from the lakebase and then wrote and updated data back to the system and persisted the data in Postgress. Awesome. Just like that, we reviewed how data is moving through the data bricks platform until we are synchronizing it from the lakehouse into a lakebase instance from which we then have access from internal tools to read and write that data. Thanks for joining me in learning about lakebase.
If you want to learn more and try it out yourself, check out the guide, how to use lakebase as a transactional data layer for your data bricks apps.

Learn more about the Databricks Data and AI platform.