Open-Source DLT Pipelines: Vision, Roadmap, and How to Get Started

Summary

Lakebase is a generally available, serverless Postgres service on the Databricks Data and AI platform that runs on lake storage, enabling elastic scaling, instant provisioning, and scale-to-zero without manually tuning database parameters.
Lakebase introduces copy-on-write branching for safe development and experimentation, cross-cloud disaster recovery, and pgVector for hybrid full-text and vector search workloads.
LTAP (Lake Transaction Analytical Processing) enables real-time analytics to run directly on live Postgres operational data at up to 12,000 QPS and single-digit millisecond latency, eliminating the need for a separate ETL pipeline.

Open-Source DLT Pipelines: Vision, Roadmap, and How to Get Started

Spark Declarative Pipelines is a new open source standard for defining and executing data pipelines. Since its release, Spark’s declarative approach to data transformation has helped data engineers focus on business logic, instead of details of how to execute their queries. But, up to this point, this declarative model has only applied to individual queries. Data engineers building pipelines that keep multiple datasets up to date have needed to resort to imperative programming, which is time-consuming, error-prone, and difficult to make performant. Databricks’ DLT project, initially released in 2020, extended Spark’s declarative approach to entire data pipelines, dramatically simplifying the task of building data pipelines using Spark. As of X date, DLT has become part of Apache Spark, as Spark Declarative Pipelines. This will enable the ecosystem to converge on a single standard for ETL with Spark. In this video, we’ll explain Spark Declarative Pipelines: their core concepts, how to get started, and where the project is going.
Talk By: Ippokratis Pandis, Distinguished Engineer, Databricks ; Jonathan Katz, Sr. Staff Product Manager, Databricks ;

Chapters

FAQs

What is Lakebase on Databricks?

Lakebase is a generally available, serverless PostgreSQL service on the Databricks Data and AI platform that runs Postgres on lake (object) storage instead of traditional disk. It maintains full Postgres compatibility while enabling elastic scaling, instant provisioning, and scale-to-zero compute.

What is LTAP and how does it work in Lakebase?

LTAP stands for Lake Transaction Analytical Processing—a capability that lets analytical queries run directly on live Postgres operational data without a separate ETL pipeline or read replica. Benchmarks show up to 12,000 QPS at single-digit millisecond latency, competitive with dedicated analytics-optimized vendors.

What is branching in Lakebase?

Lakebase supports copy-on-write branching, letting developers and AI agents create isolated database branches for testing and troubleshooting without affecting production. Branches can be created and discarded instantly, making iterative development and safe experimentation much faster.

How does Lakebase support vector and hybrid search?

Lakebase includes pgVector support, enabling hybrid full-text and vector search directly within the Postgres database. This makes it suitable for AI agent memory, RAG retrieval, and other workloads that require both traditional relational queries and semantic vector similarity search.

Full transcript

[00:08] My name is Jonathan Katz. Uh I'm a senior staff PM on the lakebased team. I'm on the core team of the open source project. Um in spite of how I'm dressed, I'm a product manager. We'll also hear from Hypocratus Pandis who is a distinguished engineer. Hey Bo, come up on stage. Are you sure? Say hi. And
[00:21] we're talk about how we're going to do real-time analytics on live Postgress data. And first we want to talk a little bit about Postgress. Like why Postgress like why is like based on Postgress if you're at the keynote you heard a lot of these
[00:33] points but part of it goes back to how Postgress was originally designed in the 1980s which was to be extensible and because of this extensibility you can kind of do a little bit of everything with Postgress it could be you know vector workloads document workloads oh
[00:46] and like boring relational database workloads but this has allowed it to be very adaptable even as data patterns have changed and especially now that we're in the agentic age we see that
[00:58] this whole open source ecosystem built around Postgress is just very adaptable as we're building new agentic applications and regular applications and it's helped to keep it very popular you know throughout you know throughout
[01:10] the years now you know be because of that you know because of a lot of these changes we thought about like how do we bring both Postgress into the lakehouse and into the lake because Postgress when
[01:24] it was originally designed it was in the spinning disc error and effectively What we've done throughout the years is like make discs faster. But being faster doesn't necessarily help you to scale. So because Postgress is extensible, we
[01:37] said all right well let's create a whole new storage engine for Postgress. We keep full Postgress compatibility at the top of lakebase but we can just do much more with storage. We make it faster. We can make it more scalable. And a key
[01:50] thing is we made it serverless and elastic. Now I've been around the Postgress community for around two decades, maybe more than two decades now. I've lost track. I've you know I've used Postgress as an app developer.
[02:02] That's where I started. I've you know come to the vendor side and if you ask me to tune your Postgress workload even after running them for decades I'm like I don't want to like tune all these parameters. I want things to be able to scale up and down automatically. And
[02:15] we're able to do that with the lakebased architecture. Now, scalability and performance is not, you know, just enough. We need to be able to govern this. And personally, when I went into like the enterprise space, I've been
[02:27] told many times by customers, we don't care about the performance numbers. We don't care about scale. We care about who has access to the data. So, our next step was to make sure Lakebase worked with Unity catalog and we built that
[02:39] integration. And finally, you know, we jade this back, you know, February of 2026. Since we launched Lakebase last year, we've seen overwhelming adoption that there's already 3,500 customers today.
[02:52] And this is the number I was allowed to say, you know, it's it's a little bit higher than that. And we've already seen a lot of success stories, you know, both running, you know, mission critical and production workloads. And this is just the beginning of all of the of the
[03:05] roadmap that we've been delivering on, some of which we'll talk about today. It's allowing more and more workloads to come to Lakebase from net new applications and you know especially agentic applications that we've seen things like feature stores and agentic
[03:17] memory to just traditional applications as customers modernize their infrastructure to leverage lakebased you know the other you know technology within the data bricks platform. Doing this also requires making sure
[03:30] that lakebase works for traditional workloads and those traditional features that might come up in architectural review boards or you know the requirements that takes to bring uh you know a workload to op you know operating. Over the past year, while
[03:43] we've been delivering a lot of the exciting features around performance and scale with Lakebase, we've also focused on those features that might be a little bit more on the boring side, but are very important around resiliency, high availability, and more. Um, you know,
[03:58] you know, one of the key areas though is, you know, improving performance. Um, I have a slide I'll touch upon that in a second about how we've been making, you know, Postgress really, you know, hum within within the lake. But another key
[04:10] thing within Lakebase is this concept of branching. How many of you have tried the branching feature in Lakebase to date? Oh, you're missing out. Even if you don't use it for like the branching was
[04:23] originally invented to help with development. So basically you could have a copy of your production data create this branch and which basically does a copy on write and then you get this isolated compute with a you know this you know exact replic well well not even
[04:35] a replica this view of your production data that you can experiment on. But it also extends to production where you might have a problem like there's a rogue query or you know there's a bug in the system you can create a branch and
[04:47] troubleshoot that on that branch without impacting your production workload. That's going to be the theme of today's talk which is how we can do things with lakebase that don't impact our production workloads. Now we still want to make sure lakebase
[05:01] performs well and we've invested a lot you know in the overall performance. This chart in a nutshell shows how you know for a very common workload we use the a TPCC- like benchmark to show how
[05:13] we can continue to increase the throughput of uh lakebase without impacting the tail latency because all these systems you can continue to increase throughput like you might eventually hit a bottleneck but like latency will start to spike with
[05:25] lakebase and like how the lakebased architecture is designed you know that spike comes much later and this is just the beginning like we have you know we have more to come um you know check out the the Lake Base blog over the next few
[05:37] weeks. But we're able to do this because we have that separation of storage and compute and operations that normally a compute layer has to do, we can push down to the storage layer. Now, resiliency is actually one of my
[05:50] favorite topics. Um my my manager says it's like eating your veggies, but like you need this to make sure that your operational workload is, you know, is available. And we've we're trying to take this a step further. How many
[06:02] people have workloads where your operational requirement is I need you know cross region disaster recovery. Cool. Did you know that to do cross region disaster recovery like the next
[06:15] logical step is to go crosscloud which is especially important in areas where you have data sovereignty requirements but you have to you know you you typically don't have like a managed service provider having two regions in
[06:27] that area. Well, now with Lakebase, we're introducing introducing, you know, the capability to be able to do that, you know, leveraging, you know, our crosscloud technology. And where it's unique is that it's fully managed. Um,
[06:39] this is, you know, in many ways, this is awesome. And it also is going to extend to being able to move your data around to bring your data closer to your, you know, you know, let's say you're using a foundation model and one managed service provider you want to ship your data around. This is where, you know,
[06:52] crosscloud DR can help. And additionally, you know, going to like Postgress to do a little bit of everything. Postgress can do search. Um, we're also introducing lakebased search
[07:04] which takes the open source PG vector library and adds you know that adds an additional indexing type on it that you know is designed for you know the lake workloads. Uh I've actually spent a lot
[07:16] of time working on PG vector personally and one of our goals of this project was to make sure like this is still PG vector that if you're going to move your data around to other Postgress systems it's you know totally PG vector but for
[07:28] the search aspect you're able to get this like very efficient very you know very accurate we use the term recall and vector search you know searches that keep your data much more compressed it's much more suitable for the lake and then
[07:40] we layer in full text search you know on top of it if you want to be able to to do hybrid search. So we've been doing a lot to you know with lakebased to make workloads both more resilient and take advantage of the flexibility for Postgress
[07:53] but how do you run analytics in Postgress that's why you're all here today right you're like John this talk is about real-time analytics in Postgress well why you've been talking about all these features in lakebase it all kind of ties together but it's been
[08:06] very hard to run analytics in Postgress and I'll take a classic example so here's a database this is like an operation you know this is a postgress you're having operational workload typically it's what we call point lookup
[08:18] so you're returning like one row or two rows or 10 rows and suddenly this like you know this count statement comes in and it's this huge aggregate it queries all the data and suddenly like your nice
[08:30] operational workload goes away because there's all this resource contention for memory and postgr you know a classic uh commercial feature is oh you do workload governance you say like hey this user can only you know query you know they
[08:42] only get this many resources to your database Postcrist doesn't have that. We can't you know I can go into the reasons as to why if need be but the you know this is challenging I'm sure you know I'm sure a lot of you have dealt with this in production but wait postgress supports
[08:55] replication in fact we just talked about that we have like you know this crosscloud DR feature and you know the classic architecture pattern was okay well push all your analytical queries in Postgress to a replica and that could
[09:07] work until like a gigantic update comes into your primary and then we get into some challenges where Postgress is waiting for all those changes to replicate over it holds up certain
[09:19] processes. So for example, you can end up saying like well you know don't block on these changes just like time out while the while you know while all the changes are replicating over but it's kind of rough because you basically cause your analytical queries to fail.
[09:33] Well, the other thing is you could say like well wait you know make sure all those changes I've replicated over don't remove them on the primary but this blocks an important postgress process called vacuum like vacuum helps to clean everything up and make sure that the
[09:46] data you know the data is okay but if you have a lot of a lot of analytical queries coming in it can block vacuum indefinitely which actually causes your primary database to catch on fire and your replica database to catch on fire
[09:58] and it's just a complete mess. So the way that Postgress has been used for analytics is you know HTAP in the degree that people try to make Postgress faster either through uh upstream
[10:10] changes or you know bringing extensions into it but you're still consuming resources in your Postgress instance and that doesn't you know that doesn't get us to a better place. So typically the most common pattern has been just to
[10:23] move the data out of Postgress into other systems. But then you run into a challenge where you have issues with freshness or you know being able to get the latest data and you don't have a fully consistent view of it. And this gets into the problem like
[10:36] moving the data out of Postgress for analytics is most likely the best approach. But we get into this trade-off where we don't see our most up-to-date data. Um it's not integrated into your full analytical system. you know, you've basically, you know, broken a catalog
[10:49] boundary, and you're just trading off between freshness, workload isolation, and what the truth of your data is. So, I called up IPO. Um, IPO and I have worked together, you know, for for a
[11:01] while, and I was like, hey, can we do better? Like, can we get to real time, you know, analytics in Postgress? So, I'll let explain, you know, what he's been cooking.
[11:14] >> Thank you, Jonathan. >> There's stairs over there. >> Oh my god. All right. Hi. Uh, hi everybody. My name is Cipo. Uh, I'm here to present. I'm an engineer at datab bricks joined a little over a year ago. I'm here to present some of the work
[11:26] that uh the real engineers have been doing. Christo is sitting on the first row here. He has done most of the work. Vadim Sriram uh uh but it is an exciting time right now in in the lake. Uh and
[11:39] let's for a second go back to forget transaction transactions and databases. Let's talk about analytics. Uh for we have been working on analytics for decades now and the traditional wisdom
[11:52] was you were loading your data in a data warehouse and you were running your analytics there. The architecture of these systems was very monolithic. The storage and the compute were tightly coupled and you were not able to scale
[12:04] uh one or the other independently. Instead of that uh datab bricks was has been the leading vendor of providing a lakehouse architecture where we have separated the storage and compute and we are able to scale and do beautiful
[12:17] things both on the storage side as well as at the compute side and you folks have uh voted us and probably the main reason you have been here is that uh you have been able to
[12:29] run amazing analytics use cases on on on data bricks and uh we have been very uh confident to say that uh the best data warehouse, the best analytics systems are the lakehouses. Uh since we
[12:42] introduced the lakehouse in 20 five years ago, we have seen tremendous growth. Uh 60% of the fortune50 companies right now are running on data bricks and lake house. Uh they were the
[12:54] leaders in in this in this market and every day there are many of exabytes that being uh uh processed uh by by our systems the lakehouse systems.
[13:09] And over the past years uh the team led by Sam here uh has been doing some very hard work in order to improve the performance. Uh you can see that it has been a continuous uh uh continuous improvement on on the performance of our lakehouse and that it is coming we pass
[13:23] it to you by you seeing better performance without having to pay more. But at some point over the period of time we start hitting a wall the let's say call it the one second wall where
[13:36] many of the systems including uh our own uh start hitting a wall when you wanted to go below say in the uh one second when you want to power realtime applications or you know uh have uh uh
[13:49] you know real time applications uh for your for uh for your application. Um and in order to get past this wall, uh
[14:01] the gener the wisdom has been that you were ending up building a separate stack. uh uh you would take your warehouse, you would get to your lakehouse, you would take a sliver of your data, you would load it to a
[14:13] specialized system uh call it uh I don't know from vertika pack to pino to druid to click house or what have you in order to have this separate thing uh real time
[14:25] layer that would answer your queries in say milliseconds. Now, but now you have a challenge. You have you have a problem because uh on the one side your data uh lives in the lakehouse. Uh you have a bunch of use
[14:38] cases. You have your ETL processes there. You have your governance your catalog there. And at the same time now you have to make a copy. You have to the data is no longer fresh. Uh it that's complexity because now you have a
[14:50] pipeline that you need to load the data and your governance is a mess because now you have multiple readers that have different authentications and whatnot. And that was not as an acceptable solution for us and our we believe that was not a good solution for our
[15:02] customers. So we say can we get the have the cake and eat it too? Can we have real time performance uh low latency on analytics at the same time have one copy on the lake
[15:15] and when I joined one year ago the uh datab bricks had already assembled an amazing team uh on of engineers that were working on this very hard problem and the decision before even uh joining
[15:27] was to build a new engine uh uh from scratch. uh Reynold earlier today uh talked went a little more into the details and a b a bunch of uh uh sessions even today that they go dive a
[15:41] little more into the details of what we did with Raiden. Uh but it suffers to say that the resulting edging uh has pretty much it's is overcoming all the challenges I talked about earlier. Uh you can run all the queries both small
[15:54] and large. It operates on a single copy of the data in open file formats. It can handle complex joints, complex queries. It autoscales and it is still govern it doesn't and so it bypasses all the
[16:08] challenges we you used to have in with having a two separate stacks of your data. And uh today we announced that we are introducing we are opening up uh lakehouse RT we call it lakehouse RT as
[16:20] a offering where you can go into your workspace today and you can have an option where you can go and start running lakehouse RT. uh we and you get millisecond performance, massive concurrency and uh direct operations on
[16:34] your governed data lake. Uh uh as of uh this weekend, we were able to push up to 12,000 QPS uh on a single uh uh lakehouse RT endpoint and achieve
[16:48] latency of singledigit milliseconds. So this thing is it works uh very well and we are excited. is probably the most important uh release uh uh by datab bricks over the past five years
[17:02] and uh this thing works uh we have been working with a bunch of preview customers on that and in this graph what I'm uh what I'm showing and what reol showed earlier it was uh the performance
[17:14] of uh uh lakehouse RT compared to uh other vendors uh oh I'm sorry the graphic replaced But uh there is a one well-known vendor out there that it is
[17:27] trying to get into this space and he has making multiple attempts putting multiple generations of the systems offering an interactive version of that that it is cap to answering queries up to only 5 seconds. There is another
[17:40] vendor that it is well known for operating on oper on on denormalized well uh data that have been denormalized so that you can do very fast scans and aggregations and all of them the
[17:52] performance of them as you increase the concurrency starts the latency of your query starts going up uh the one vendor starts crashing and what we have seen is that with lakehouse real time we're able to push several thousands of concurrent
[18:05] queries per second at very low latency sub millisecond uh single digit millisecond latency. And the interesting thing with this engine is that not only it can serve the
[18:17] easy denormalized scan and aggregation type of workloads that you would normally run operational analytics like you will power your graphana dashboard or something like that. But it can also answer more complex analytical queries.
[18:30] For example, in this graph, what I'm comparing, what we're comparing is uh uh we're using a well-known uh database B analytics workloads benchmark called TPCH. And we took the same vendors and
[18:42] we compared uh the performance of uh these systems as we increase the size of the data. And you can see that compared to the traditional vectors uh the real time uh lakehouse real time
[18:54] is up to four times faster especially on the low on the smaller data. And as the data volume grows, uh the the other optimized engine is starts going getting worse and worse in performance and at
[19:06] some point it starts crashing, failing to run and the interactive version of the other vendor start timing out and not being able to use it. You have to have a fragmented uh uh application. Things get more even more interesting if
[19:19] you start even more complex analytical workloads such as the TPCDS workload. uh DPCDS benchmark is a more complex version of the TPCH that has many many more joints and more complex stuff. H
[19:31] and in this uh in this one you can see that uh lakehouse real time is achieving up to 15 times higher performance than the operational analytics optimized vendor. H at some point it gets 25 times
[19:44] better and at some point the anal operation analytics optimized vendor starts crashing whereas Raiden Lakehouse real time we call it RA then internally uh uh starts crashing.
[19:57] So we have achieved this on the analytic side we have achieved a best-in-class realtime analytics performance very low latency. uh Reynold earlier this morning he did a live a live demo where he was
[20:10] able to show you that he we can answer query in single digit milliseconds 7 8 milliseconds uh over the weekend uh as Ream can attest that we are able to achieve up to 12,000 uh queries per
[20:22] second uh in terms of concurrency from one endpoint we have been working with many preview customers design partners and uh tens of them and we can see we can actually uh confirm that uh there.
[20:36] What we see in the lab is actually what they our customers, our design partners see in real life. And it can do not only can address not only the easy scan and aggregation type of workloads, but also
[20:48] it can do more complex analytical workloads and and deal with them equally well. So everything is nice and pitchy on the analytic side. But you had the transactional data like Jonathan started
[21:02] talking about how beautiful posgress is and how important your transactional data your money application your system of record it still lives in Postgress. You have to go so far and you have been forced to create a very complex weird
[21:15] brittle expensive data pipeline to move your data from your transactional store to the analytical store or your lakehouse in order to perform analytics on them. Can we do that without compromises?
[21:29] Again, a little of a recap. Postgres vanilla you had your database and you can actually say that not only this is not the architecture not only of Postgress but pretty much any other transactional database out there
[21:42] including Oracle or DB2 uh you know all this this paleontology if we're talking about these systems right now that they used to be one monolithic system where the storage the computer and
[21:54] transactions the indexes all used to be in one box uh many many moons ago I used to work for IBM where we used to go to the to the boxes and we will do you know debugging this type of things and it was
[22:07] not a place where I want to go back again. Uh with lake base we were able to separate the storage and compute. We have a very interesting uh you know cloud native architecture where you still get your
[22:19] postcress compute at the same time. We have a a a a dedicated storage layer that uh that has a combination of persisting the the persisting the wall the right ahead log into safekeepers and
[22:31] having page servers to be able to verify with low latency answer uh uh to retrieve the pages the data pages from the storage and all the data stored down on S3 on the object store. And once we
[22:44] have this thing, we can now apply the main big idea da is that we can now have analytical compute being able to operate directly on the storage layer of lake base. That's that's the whole reason we
[22:57] are here and today I'm very excited as part of the uh the announcements we're making today to announce that lakehouse real time is able to in preview mode in in as
[23:09] in preview to be able to access the live lakebased data and you can run and perform complex analytics on your live postgress data on your live uh analytic lakebased data. So you can have use cases where you have
[23:23] your lake base, your postgress endpoint. You can uh read, write, do complex transaction processing in there, multi-statement transactions, all the nice things you used to have to do uh and love in in Lakebase. And at the same
[23:38] time you can also run very efficient low latency very low latency operation and analytics using the lakehouse real time on the same one copy of the data.
[23:50] How does it work? Crystal just left. He had to go and meet a customer. But the the main idea is that we have the the lakehouse real time uh uh conducting and asking the live post live posress or
[24:02] lakebased instance give me the latest uh the LSN like the the latest uh number where I can start pulling data from S3 or from the storage layer and be
[24:14] transactionally consistent and have the latest view of the catalog. It gets back an act and from that point on it goes down to the storage layer and it is able to access the data pages the data in very high uh perform with very low
[24:26] latency high performance and we have been working on that for a while and actually it does work. uh so in this graph uh what I'm comparing is the following uh we took again this TPCH
[24:38] uh uh benchmark u scaling factor 10 meaning 10 gigabytes of data which is pretty much equal or bigger than uh most of your databases out there if you go and look at your operational databases
[24:51] now many of them will be probably below this 10 gigabytes uh size and we run this complex benchmark TPCH which has a bunch of joins and stuff in it and we did spend spend a fair amount fair
[25:04] amount of time trying to tune uh Postgress and lake base in order to get the best performance out of Postgress on this analytical queries. We built indexes, we increase the buffer push and we did all sorts of stuff. Still the
[25:16] best performance we could get was something less than 100 seconds to run the 22 TPCH queries on this on this database after a fair amount of tuning. I repeat on the other side on the on the
[25:29] analytics optimized side of the world where we had lakehouse real time operating on delta of on on columnar delta data out of s3 the same workload the same benchmark runs in a little over
[25:42] 3 around 3.4 4 seconds alto together and with uh now with lakehouse real time accessing the live lakebased data we are able to achieve something like 9 seconds we so we are getting about over 10x
[25:56] better performance than your traditional progress query optimize pro optimize process quering its own data you can achieve 10x better performance and you are in the same ballpark as the
[26:08] performance of uh lakehouse RT accessing the highly optimiz ize delta data. So you have the cake here you eat it too. But we have not stopped there. We can we
[26:23] believe we can do even better. Uh in this graph you can see there is a small gap there right? There is a a 3x gap. I mean this is amazing if you're coming from the posgress world. If you are coming from the analytics world you say
[26:36] oh I'm leaving something on the table. It's not like that because you have the tremendous value of now you are able to run analytics on your live posress data by the way in complete isolation with
[26:48] your main production database. That means how many of you have woken up at night because your production database has been down. I hope there's more than one person. Yes, like like you just the developer
[27:02] they write they tend to write some simple query in your main production database or your control plane where you say okay join this and that that becomes a very bad join and the production is down because your postcrist database is
[27:14] trying to deal with nested loop joins and consuming all the CPU and bricks your production down while you just want to have a read and write application on that. Uh so uh you can do that now. You
[27:26] can run very uh uh complex and powerful analytics on your live production data without with full isolation of your main production database.
[27:39] Well, there are some physics or some facts, right? Uh facts uh on the transaction side, in order to get good performance, traditionally you have been storing data in a row major order, right? Every every record was a row and
[27:52] you were storing these records serially. On the other side on the analytic side on the OLAP side we have been traditionally since the past 10 to 15 years we have been storing the data in a column oriented major so that uh when
[28:05] you are accessing data that you don't you you don't have to bring the entire record you just bring the column at a time but uh it's very very hard for the both of them to coexist. It's something that in I I I have a PhD from Kangi
[28:19] Melon on databases. This is kind of the holy grail of of data management for for decades and and we have been trying to get a crack at it for a while. Uh but it is very very hard. I can tell you that.
[28:32] Uh but then then we have an opportunity now with this beautiful lakebased architecture since the storage layer is disagregated from the running postcrist
[28:44] and we have these guys these safekeepers and paid servers that are acting as a buffers for the requests from the database to the storage layer. We have a place where instead of when we are writing to the object store instead of
[28:56] writing the data in a row major we have the opportunity to transcode them and now store them in a column in columnar major. In fact we can write them in any of the open data
[29:09] formats that we have out there. We have delta, we have iceberg and we love them. And by di doing that now you have one copy of the data in open file format managed by the unity catalog as managed
[29:23] tables but now you can run transactions on them and you can run analytics on them and that's what we call laptop lake or okay and obviously we can apply lakehouse compute on that you can use any table reader to read and write to
[29:36] read this data right now it's for reads we eventually probably we will get into the rights and the data that has been transpired. It is a few minutes old. If
[29:48] you're using lakehouse real time, you can access the live version of this data. And by doing that, we're going to close even this small gap that I show you earlier. This is where we are.
[30:01] Uh we call this new paradigm. It comes from lake transaction and analytical processing. We unify the data infrastructure. You don't have to have duplicated data. You don't have to separate your operational data from your
[30:13] analytical data especially because these things are the computes are very are isolated. You one has read only access the other is modifying and the the analytical side the trolls cannot bring
[30:26] production down on the other side. You don't have to have pipelines. Uh things are pretty interesting. And next I will ask Jon Jonathan to come up and show you a demo that this thing is actually working. And we are we are
[30:39] in preview right now. >> So thank you Ipo. Uh first I want to make one correction. It's not vanilla Postgress. It's pistachio Postgress. Um
[30:51] so let's demo and here's we have an application. Well it's not really an application. It's it's you know it's an insight into you know you know what we're building and similar to the demo we had this morning.
[31:03] It's we're processing bank transactions. This is actually based on the TPCB benchmark uh which was for you know for processing and it looks like a regular operational workload. You have you know reads and writes coming in data is
[31:16] constantly changing and it moves at a pretty good a pretty good pace albeit I'm going to speed it up a little bit. And what we're going to test is a common operation that occurs when you're a bank which is you might need to perform an
[31:28] aggregate. You might be summing up a customer's you know all all the transactions in their system or doing things to detect fraud. And can you tell when we actually ran that large aggregate across all of our data? Yes.
[31:41] Our you can see like everything goes down the performance dips and you know it's going to take some time for both the analytical query to finish but also for uh our system to get back to its you know standard performance. And here the
[31:55] query hasn't finished yet. Now it's finished and we can see that you know it took a long time to run but there's really there's the challenge like not only was our analytical queries slow it took a while you know it it tanked our oltp performance and it still hasn't
[32:09] fully recovered from you know from thereafter. So the point of you know LTAP and the point of putting lakehouse RT on uh on lakebase is that we can get that
[32:21] workload isolation without impacting our operational workload and we're able to get that data in real time. So look only after like you know a minute in this simulation you know it actually took a
[32:33] little bit longer did our workload get back to you know that peak performance. So let's try running those analytical queries now and see if we can do better and see if we can do faster. So at this point we're going to click we're going
[32:46] to you know we switch over to you know lakehouse real time and we ran the query and first off it's way faster but our production operational workload did not dip. So like let's run it a few more times and like see how we do
[32:58] performance-wise and we can see that everything works. uh we're able to get you know very fast analytical queries and our production workload stays good and to me you know on the operational side you know I've always been about you
[33:10] know how fast can we go how fast can our operational workload can go and the fact that we can get you know both data freshness and workload is isolation and transactional truth meaning like you know we have one source of data is
[33:24] really powerful and it's really cool that we have a new way of being able to handle processing where transactional workloads and analytical workloads can defy the laws of physics as Zippo was describing you know and you know part of
[33:37] the magic was you know finding the right way to encode Postgress data within the lake we're able to do this because you know Postgress is an open format like it's an open standard and you know we know how we can map Postgress data
[33:51] losslessly into you know you know an open format like parquet and this is part of you know that open source library that will be released soon you know does is that we're able to take all all the complexity of the Postgress
[34:03] types and safely encode them in the data. Um the other big part is it's not just about writing, it's reading. We can take the data that's stored in these open types like any other open table format and have them read into you know
[34:16] you know Postgress systems that are able to you know understand that format. So available in private preview uh if you have this need if you want to be able to query your Postgress or your
[34:29] lakebased data in real time uh talk to myself talk to IPO uh we're able you know this is available in private preview and like we're looking for customers to you know try everything out
[34:41] I see one question we we're going to have time for one question by the way we're running a little hot um I don't thank

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.