Introducing LTAP (Lake Transactional/Analytical Processing): a new data processing architecture

Summary

• LTAP architecture: LTAP combines operational and analytical workloads in a single system using Lakebase and the lakehouse, eliminating fragile CDC pipelines.
• Single data copy: Unified platform unifies transactional data from OLTP systems with analytics, replacing error-prone change data capture with a governed architecture.
• Serverless Postgres: Lakebase provides serverless PostgreSQL on open object storage, simplifying deployment and scaling for hybrid workloads.

Introducing LTAP (Lake Transactional/Analytical Processing): a new data processing architecture

Introducing LTAP: the first Lake Transactional/Analytical Processing architecture.
OLTP unifies operational, analytical, and streaming data without the need for ETL, replicas, or pipelines. The new data processing architecture combines OLAP and OLTP to create a single copy of data, all goverened under one model. Underpinned by Lakebase, LTAP is the first platform to combine serverless Postgres on open object storage with the Lakehouse. Learn more: https://www.databricks.com/company/newsroom/press-releases/databricks-launches-ltap-first-lake-transactionalanalytical
06:16 — Coming soon: Open Source LTAP Writer Library
07:06 — Lakebase + LTAP demo with Holly Smith, Developer Relations at Databricks
Presenters:
Reynold Xin, Co–founder and Chief Architect, Databricks
Holly Smith, Developer Relations, Databricks

Chapters

• 0:00 Reynold Xin, Co–founder and Chief Architect at Databricks, introduces the challenge with legacy solutions
• 2:35 Bringing Lakebase and the Lakehouse together
• 5:13 Introducing LTAP
• 6:16 Coming soon: Open Source LTAP Writer Library
• 7:06 Lakebase + LTAP demo with Holly Smith, Developer Relations at Databricks

FAQs

What problem does LTAP solve that HTAP and CDC pipelines don't?

LTAP eliminates fragile CDC (change data capture) pipelines that often cause data corruption and maintenance headaches. Unlike HTAP systems that typically compromise on both performance and proprietary ecosystems, LTAP uses Lakebase to provide a single governed copy of data for both transactional and analytical workloads.

How does Lakebase enable serverless operations?

Lakebase provides serverless PostgreSQL deployed on open object storage, allowing users to build transactional applications without managing infrastructure. This integration with the Databricks Data and AI platform creates a true lakehouse that handles both operational and analytical data.

What is LTAP and how does it change data architecture?

LTAP stands for Lake Transactional/Analytical Processing and represents a new data processing architecture that the Databricks Data and AI platform introduces. LTAP unifies transactional consistency with analytical performance by combining ACID guarantees on Delta Lake with optimizations for complex queries. This architecture eliminates the need for separate operational and analytical databases, reducing data duplication, simplifying pipelines, and enabling real-time analytics on the same data store as production transactions.

How does LTAP improve query performance?

LTAP on the Databricks Data and AI platform combines the benefits of lakes and data warehouses by providing ACID transactions with performance optimizations for analytic workloads. This video explains how LTAP handles both operational queries and complex analytical scans on the same data without requiring replication. The architecture achieves this through intelligent data layout, indexing strategies, and query optimization that automatically adapt to your workload patterns.

When should I use LTAP instead of separate transactional and analytical databases?

LTAP on the Databricks Data and AI platform is ideal when you want to eliminate data duplication, reduce operational complexity, and enable real-time analytics. This video demonstrates that LTAP works well for organizations currently managing separate transactional systems and analytical warehouses. By consolidating onto a single LTAP system on Databricks, organizations reduce ETL complexity, enable immediate data availability for analytics, and simplify compliance and audit trails.

Full transcript

0:00 Reynold Xin, Co–founder and Chief Architect at Databricks, introduces the challenge with legacy solutions

You heard from Bilal, simplified data engineering from myself about how we're going to unify analytics and from Modernizer OLTP. But honestly, one of the biggest problems there's still one big thing remain, which is we have the giant continent OLTP databases where applications start putting their data in first. And if your application becomes successful enough, you want to reason on that data and analyze on your data.
And so we end up actually building CDC pipelines that ship the data from OLTP databases over to your analytics continent. And how many Can I get a show of hands? How many of you love your CDC pipelines? All right. Oh, there's some. Like I would love to learn from you how you actually maintain them. All right. CDC pipelines are basically change data capture pipelines. It reads the bin log of OLTP databases and it gets all the delta little delta changes and ships the delta over to analytics systems and reconstruct the state of OLTP databases in them.
They're super fragile. They're very annoying to maintain. Many data engineers been woken up at 3:00 a. m. actually because it caused a data corruption. As a matter of fact, some of people at Databricks actually joke about CDC doesn't stand for change data capture. It really stands for continuous data corruption. All right. And to solve this problem, the industry started introducing zero CDC, zero ETL, mirroring. And all of this are just fancy terms of let's do a managed CDC by hiding the pipelines that automatically happens.
But in practice, it's still a pipeline under the hood. It suffers from the exactly the same issue, CDC pest. So some of you might have heard of another term called HTAP, hybrid transactional analytical system or processing. And HTAP is sort of the holy grail of database engineering. The idea is you create a single database system that's capable of handling both your OLTP workloads and your analytics workloads in one system.
But HTAP has I think most of you probably haven't actually used HTAP system in practice and the reason is HTAP as a category had largely failed. There's very little adoption of it and all of the HTAP systems out there are kind of very proprietary. They don't have a big ecosystem and by building a single system, they started compromising on both the performance for OLTP and the performance for analytics. So, the question is how do we truly unify the two?
Right? If HTAP doesn't work and CDC is very annoying and causes corruption all the time, how do we do it? And the solution to this actually goes back to the very fundamental problem or dichotomy of row-oriented storage and column-oriented storage. As Ali and both alluded to earlier, um OLTP systems need row-oriented storage because it needs to do a needle in a haystack lookup. It needs to actually be doing uh updates on the rows rows very quickly.

2:35 Bringing Lakebase and the Lakehouse together

Whereas analytics system does a lot of scans of large amount of data and it really benefit from column-oriented storage. If you look at the architectural diagram that already showed you, where we have safe keepers and page servers that write data in an actual row-oriented format to the data lakes, um the solution actually lies right there. When we looked at and profiled the systems of safe keeper and page server, those are storage services, it turned out they're very underutilized from a CPU perspective but very I/O bound because all they're doing is local disk reads, local disk writes, and network reads and network writes.
So, it actually gave us an opportunity to leverage those idle CPUs on those storage services to do a transcoding at the very moment to turn the data from row-oriented format to a column-oriented format. All right. And it turned out once you convert the data from row-oriented format to column-oriented format, it doesn't actually increase the CPU utilization that much but actually dramatically shrink the data volume because column-oriented format has a better compression ratio.
It's often somewhere between 10 to 1 to 100 to 1. And because of all the services IO bound, by having a better compression ratio, we can now actually benefit from this transcoding instead of hurting the performance. And with that, we can actually write the OLTP data into the data lake in column oriented format. And once they're in the column oriented format, Delta or Iceberg, you can actually apply any analytic compute directly against that data.
So, how does this work in practice? Imagine inserting a new row into the lake base. And the lake base compute eventually sends that row to the underlying storage system. And the storage system will do the transcoding right there, and actually write that rows directly into the columnar format in Delta or Iceberg. And now you can actually fire up a query against your lake house, and the lake house will read directly the Delta or Iceberg format, and return your query on the freshest copy of the data.
So, we think this is actually revolutionary. It's probably as at least as big as the way that Tog was talking about. And the whole concept of lake house plus lake base, we can actually have unified storage for both of them. And we're giving this term the storage technology a name called Altab. And Altab stands for lake transactional analytical processing. We think Altab is HTAP done right, and accomplish the goals of HTAP without actually having a single query engine for it, but we're able to actually unify the storage, which is by far the most important part.

5:13 Introducing LTAP

And now you have one copy of data for your OLTP databases, and one copy of data to govern. So, >> >> Altab can truly unify your data infrastructure. You start with the OLTP database for applications, but you don't have to copy your data. There's no pipelines to maintain, and most importantly, there's actually no compromise in performance at all for both your OLTP systems and your analytic systems. And all these are built on open interoperable formats, Postgres, Delta Lake, Iceberg.
So, with Altap, we can finally actually combine the separate continents of OLTP and analytics into a single giant continent. But, Wonderland we don't want to do this alone. So, we'll be open sourcing actually a very fundamental library that allows converting Postgres data directly into columnar format in parquet. The logo is actually it's kind of a cute logo that has a parquet elephant to indicate what they do. And this should be coming very soon.

6:16 Coming soon: Open Source LTAP Writer Library

So, >> start using Lake Base. We'll be rolling out the Altap capability in coming weeks. And once it's rolled out, every single one of your Lake Base table will automatically appear in your Lake House. There's nothing you need to do. And we can do only do that because there's no pipeline to maintain under the hood. It's available for every single table. Right? And in order to sort of demonstrate to you, this is the last demo of the day, we invite onto stage to actually show you a demo of Lake Base combined with Altap. >> I'm going to give you a tour of both Lake Base and Altap and why you need to combine your analytical and operational data.

7:06 Lakebase + LTAP demo with Holly Smith, Developer Relations at Databricks

I have business banks across the Americas creating billions of transactions a day, and I want an agent that spots potential VIP customers and matches them to an advisor based on their qualifications and local regulations all in time for teller to tell them that information before you've left the door. Previously, this would have been impossible. Uh the delay would have been too slow and the intervention would have come too late, but not anymore.
So, let me show you Databricks and let's go ahead and start with Lake Base. So, Lake Base is open source Postgres fully managed by Databricks with separated storage and compute and it's really easy to get started. I'm going to create a new project. I'm going to name it and I'm going to stick with the defaults and that's it. It's made. The compute size isn't fixed, so I can scale that up and down within the UI and set a minimum and maximum size.
And on the left, you can see we've got branches here and I can click in it and I can go and create a new branch. We can name it. We can have it auto delete. We can make a branch of production. We can make a branch from a point back in time, but we can also branch from a branch. And it's created almost instantly because there's no need to copy data. Speaking of back in time, we also have backup and restore. This is configured to retain 7 days of history by default, but you can do more or less if you like and pick any time window to restore to.
Or if you wanted, you could restore from a snapshot which is scheduled on a regular basis. Now, all of this is empty. Here's what I set up earlier that's processing all of our data. And we can go into the tables and I can see all of my transactional data here and I'm going to need a query to scan across all of these historic balances to calculate who is an outlier and potentially a VIP. And for those queries, we care about some metrics.
So, first is TPS. So, this is the line chart in green that is we're going to care about on the right-hand side. If this number drops, our business is impacted and if it stops, our branch staff are going to have an outage. The second metric we care about with our queries is how long our analytical queries take. And this is going to be the top left number in blue here. And finally, how fresh or stale our data is. And this is measured in seconds, and it will be the top right number in purple.
Defines how up-to-date our data is. If I'm querying in real time, I better get real time answers. So, earlier said there were three ways of running a query like this. And the first is what you might an agent might try by default, and that's to submit the complex analytics query to existing lake base compute. I'm running this from the Databricks SQL editor. It's possible to connect to lake base from there. And when I'm running this query, I can see that this is not quick to come back.
This is not good of a one-off analytics query, but it's terrible for our agents who are going to need answers a lot faster. And we can also see that it's starting to impact our TPS. We can see that this green line is going down, and it shows a bit of a worrying trend because realistically, it's not just going to be one dip here. It's going to be thousands of them. You can barely do analytics like this, let alone flag in real time who would be a VIP customer.
Okay. Um Well, you will know my name. Looks like we have time for everyone to go around the room and introduce themselves. We'll start at the front maybe. And Oh, it's done. Okay, finally. Maybe we'll do that later. And in blue, we can see that this took nearly over a minute to complete. We can say that the staleness is also quite bad as well. And we risk taking down production. Okay, so that clearly didn't work. So, what if instead we took a copy of this data using CDC?
I'm using the lake base change data feed feature, which is meant to be for audit workloads, not up-to-date analytics. Um and with this, I know that there's going to be a bit of staleness with at least 15 to 30 seconds of data, but this is within the Databricks ecosystem. If I was using a different Postgres offering um or a different CDF tool, this could be into multiple minutes, not to mention the additional bill and the additional team to maintain it.
We can see that the TPS is flat, um but it's finished running and it took 13 seconds to run, but it was out of date by 25 seconds. So, finally, the ELT app way of doing this. Um with ELT app, these tables are available in Unity Catalog, and here it is looking like a managed table, but we can see from the icons that yes, it's a lake base table, but it's also available to both Delta and Iceberg readers. I didn't have to set up any additional pipelines to make this happen.
I just specified a catalog and it appeared. And you might notice a slight difference in our UI here, and that's that we have an S3 path. And if I go to this S3 path, um I should be able to see all of my tables in here. This is great for openness. I can query this with open source Spark if I wanted to. Uh that's a bit of a tangent. Uh let's head back to ELT app. So, here is our query again, and I'll be querying it with a new type of warehouse, a real-time warehouse.
And so, now I click run and it's done. It ran in milliseconds. Um we can see that this is very real-time data without any staleness and it had no impact on our TPS. So, we can throw as many agents at it as we like. And because this ran almost instantaneously, it also means it's significantly cheaper in compute cost, too. It is also cheaper to maintain because there were no intermediary tools to manage. For all of us, this is huge.
We have a single system of record where your data can be used for both your operational and analytical systems without having to compromise on performance. All of this happened automatically. I didn't have to set up any steps in the background. And finally, we no longer have a divided view of the world between operations and analytics. We no longer have to choose. Thank you.

Learn more about the Databricks Data and AI platform.