Lakeflow Declarative Pipelines Integrations and Interoperability: Get Data From — and to — Anywhere

Summary

DLT Integrations: Expand Declarative Pipelines (DLT) with new integrations enabling data ingestion and delivery to virtually any external system or format.
Interoperability Vision: DLT Sinks API and Python Data Source support now enable reverse ETL, Kafka streaming, JDBC connections, MongoDB, and more.
Ecosystem Parity: This video demonstrates how DLT achieves Apache Spark ecosystem parity with hundreds of connectors and integrations for unified data flow.

Lakeflow Declarative Pipelines Integrations and Interoperability: Get Data From — and to — Anywhere

Talk by Ryan Nienhuis, Sr. Staff Product Manager, Databricks
In this video, you will learn how to integrate DLT with external systems in order to ingest and send data virtually anywhere. DLT is most often used in ingestion and ETL into the Lakehouse. New DLT capabilities like the DLT Sinks API and added support for Python Data Source and ForEachBatch have opened up DLT to support almost any integration. This includes popular Apache Spark™ integrations like JDBC, Kafka, External and managed Delta tables, Azure CosmosDB, MongoDB and more.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What new integrations does DLT support?

This video details DLT capabilities including Kafka streaming integration, JDBC connections, Azure CosmosDB, MongoDB, ForEachBatch support, and custom Python Data Sources. These enable DLT to handle not just ingestion but also reverse ETL to send data to external systems, expanding beyond traditional lakehouse-only pipelines.

What is the DLT Sinks API?

This video explains that the DLT Sinks API enables writing data from DLT to virtually any destination using declarative, easy-to-use syntax. This addresses customer feedback about needing to move data out of Databricks for reverse ETL and other use cases beyond traditional analytics.

Structured Streaming versus Kafka Streams?

Structured Streaming on Databricks uses a declarative approach to define streaming pipelines that integrate seamlessly with Delta Lake's ACID guarantees and Unity Catalog governance, making it the preferred choice for most enterprise streaming workloads. Kafka Streams is a lower-level library requiring custom code for each pipeline, while Structured Streaming abstracts away infrastructure concerns and provides better integration with the Databricks ecosystem. For teams running on Databricks, Structured Streaming provides superior development velocity and operational simplicity compared to Kafka Streams.

How does DLT compare to Apache Spark?

This video notes that DLT aims for feature parity with Apache Spark, which integrates hundreds of different connectors and partners worldwide. By expanding DLT integrations, Databricks enables the same ecosystem flexibility for declarative pipelines as traditional Spark batch and structured streaming.

Full transcript

`0:00` Introduction to DLT

Good afternoon everyone. Uh. I'm a product manager on the data bricks streaming engine team. I also work a little bit with DT and open source spark. And today we're going to be talking about one of my favorite topics, uh, expanding DT such that we could can read and write data from virtually anywhere. Um, so we're going to go through, uh, kind of where we're at today. I'm going to walk through a lot of different code examples of what's possible right now and then I'm going to give you some sneak peek with some code examples of what's going to be possible in the near future.
So let's get started. So why does interoperability matter? Uh you know traditional ETL pipelines they typically work on ingesting data from a you know variety of sources uh and then go through various steps like using the medallion architecture in um getting to your gold tables then serving your data. However, uh and this is where DT historically started at, right? We, you know, ingest data into maybe a raw table, write data to a series of streaming tables, and maybe end in a materialized view.

`1:10` Challenges in interoperability

However, one of the biggest pieces of feedback that we've heard over the last couple years with DT is that, you know, I have additional use cases that don't follow that simple architecture. I do a lot of reverse ETL. I need to get data into and out of uh data bras. and that the your world your data engineering world is not all the lakehouse. So how can I do that more easily declarative easytouse service? So in this talk we're going to talk about how we fixed uh that the majority of that problem and how we're going to complete uh our vision of making that having uh DT having basically the same uh parody of ecosystem and maybe perhaps even better than uh spark batch and spark structured streaming.
So I think you all know Apache Spark is integrated everywhere. I mean there's hundreds of different connectors, hundreds of different partners that are built on top of Spark. If you want to process data with Spark, there's the world is your oyster. There's plethora of things to do. Um there's over the last even four or five years, you know, we've modernized a lot of the data sources and syncs. uh there a lot of the you know uh most popular integrations like the JDBC integration have been uh redone uh to be both simpler and more featurerich.
You know we support across all spark is supported very well across all the cloud providers but there are some limitations you know as we grew there was not a lot of like uniformity across the connectors. So different connectors were supported in different languages. Uh depending on your environment, you may or may not be able to use a connector. Uh for streaming specifically, streaming really really relied heavily on for each batch and using that construct with batch connectors to spread out the ecosystem and work for supports which was a very very it's a very powerful operator and very flexible but it's not that easy to use and there are some limitations on latency for it.
Um, so how do we get bring parity to this? How do we bring and kind of modernize our connector ecosystem with Spark as well as make it simple to use on data bricks and within DT. So I want to go through kind of a history of where we started um and where we're going to end up. So when we launched structured streaming we came we first launched just the foundational integration. So like file system streaming uh support for Kafka uh initial support for Kinesis as data sources and they were really really focused on ingest.

`3:38` Evolution of integrations

All right. So all of the message buses support were really supported on data sources and then you'd write to your lakehouse uh your file system object storage so on and so forth. Most of these connectors were first built in using data source v1. Data source v1 was like the initial primitive for most of uh uh spark's integrations. Some of them have been rebuilt in data source v2 which was shipped within the you know several years ago but it's taken a while for those connectors to modernize.
After that we started getting wide as spark took off we started getting widespread community support. So Cassandra uh Azure built a Cosmos DB connector uh MongoDB built a streaming uh native connector and you saw that ecosystem expand and depending on when they implemented it uh they might have feature X or feature Y. So this led to like disparity of feature set of security and performance and so on and so forth. About four or five years ago we started project it says 2023 but it was around probably 2021 2022 we started project light speeded which is basically a marketing term of we're reinvesting a whole lot into structured streaming.
As part of that one of the focuses was on interoperability and integrations. So we built uh basically message bus sources for almost all of the message buses. So pulsar uh pub sub we redid our kinesis connector upgraded our kafka connector integrated with schema registries. We built the Kafka sync uh and re upgraded the Kafka sync. We made sure that the you know right stream and reream from delta was highly performant and that's kind of where we are today.
And uh today for like the core use cases on data bricks as well as in the OS ecosystem we do pretty well. All right. But like I mentioned on the first slide not everything is lakehouse native. Not everyone only has ETL and inestion use cases. So if I need to write to my Postgress database, how do I do that? You know, there's three different ways right now in the in open source and on data bricks to do so with different JDBC drivers or Postgress specific drivers, so on and so forth.
And then depending on what product you're using that it becomes even a little bit more confusing in the fact that some products like uh classic compute uh DBR and jobs supports a set of connectors and DLT supports a different set of connectors. So how do we make this simple both in the open source community as well as on data bricks? So the next couple slides are going to talk about some uhatives that uh we've built over the last couple years to help simplify our story.
One of them uh you may be familiar is Apache Spark Connect. Apache Spark Connect provides uh secure infrastructure and connections to a lot of the integrations I mentioned. And uh when we shipped this there was limited support for um like the a lot of the streaming connectors and over time we've added more and more. But the goal here is that you can have very easy security things with like credential rotation and stuff like that as well as governance over your integrations within data bricks.
The data bricks manages the connectors to like tables and files. It manages external uh connections such that when a user wants to read from a particular Kafka topic they simply use an external connection within UC. This is all that Apache Spark connect unlocks. Great. What we're showing here kind of demonstrates the uh the evolution of the ecosystem uh within spark. So this is uh a Postgress SQL driver connecting using a the data source v1 API.

`7:30` Product parity roadmap

It's still widely used across data bricks. Uh it's one of the most popular database for you to integrate with is Postgress. And even though we uh shipped like a brand new JDBC connector in open source as well on data bricks in March that went GA all right a lot of customers are going to take a while to uh you know adopt that JDBC driver and switch all of their queries all their batch jobs all of their structured streaming jobs to it.
So, you know, with this API, like how do we make sure that customers can use the products they want to on data bricks using the older connectors while they upgrade to the newer features and functionality, you know, to know and this is definitely something you want to do. You want to be able to upgrade because the new integrations, first of all, they're built on data source v2. A lot of them have like advanced performance features like uh push down filters and things like that you can take advantage of.
They're are more stable and scalable, but it still might take you some time. So, the next two slides are going to paint a pretty I just want to take this paint a pretty picture of kind of where we are today and then where we're going to be in like the very near future. And you can kind of see where we're I'm bringing up like depending on whether you're using serverless uh standard or dedicated clusters or DBSQL you have a different integration story.
Uh for all intents and purposes that standard slide is DT effectively because we DLT runs on standard clusters. So we support reads and not writes for serverless on JDBC connection on standard clusters you for you can use certain drivers for my SQL databases and JDBC but you have to give any file permissions so like there's a question mark there and there's a lot of nos across the board here notably the last line the Python data source which is a brand new uh uh which I'm going to go through a code example later in this presentation so it's uh shipped last year and we're going into GA in the next month or two.
Uh but notably we that's very recent and as such it has yes across the board. But how do we bring everybody else and all of your legacy implementations up to speed such that you can use DT such that you can use serverless? How do we support a much broader set of use cases on DT more than just ingestion and ETL to gold tables and DBS SQL? How do we support operational use cases like real-time analytics uh reverse ETL?

`10:24` DLT declarative connectors

So this slide reflects where we are likely to land at the end of the year. Um so almost all of the bundled and unbundled connectors will be supported on serverless standard and VT as well as dedicated. you'll have a consistent experience where you can use the same features across these product lines. So if you have a structured streaming query running on jobs and DBR, the connection experience is going to be very similar to what you experience at DT.
It won't be this like non-parity feature set that we have that we have today. Uh in addition to that, there'll be very few limitations. Even the DBSQL, I have the two asterisks on there. Um, we think we're going to be able to support read only, but we're working also on write as well. So, DTarative declarative connectors. I went through this uh kind of hinted at it in the earlier slides, but we started the focus of DT was to make the most common data engineering patterns as simple as possible.
So ingest into a streaming table which abstracts a lot of the complexity of managing your own cataloges and schema and stuff like that. It's defined declaratively within the pipeline go through a series of steps. The gold tables um in we did start out ingesting from virtually every uh structured streaming source. Uh but this was primarily like the native built streaming connectors that we have built in. So if you wanted a custom connector, we didn't support that.
So it really because of the lack of interoperability and integrations, it really only focused on the ingestion and ETL use cases into your data lakeink. So where we are today with the features that we have in public preview that are production ready is we have a lot more robust for data sources including custom data sources that you can build using the pi pispark data source API. So native Python custom connectors. We have UC service credentials working for all of our messes bushes.
That means that you can share and I'll go through a code example of this. You it means that you can very very easily manage your connections in DT to your Kafka topics to your Kinesis streams to pub to pulsar so on and so forth. You don't have to hardcore any credentials. We handle credential rotation and everything like that out of the box. And it's very easy for you to start up and share those credentials across your team using permissioning and governance through UC.
In April or May, we shipped uh public preview for a new API called the sync API. The sync API, the purpose of that API is to give full parity of all syncs in structured streaming and spark to DT. So right now in public preview we support Kafka as a sync UC manage tables and UC external tables. In private preview coming to preview in the next month we have for each batch. All right which for I'll go through the use cases if those aren't familiar with it.
And we support in public preview going to G very soon Python uh data source. So for custom syncs uh a lot of our essays have built some incredible things using the Python data source and written some blogs. at the call to action at the end of the slide. I'll link you some, but custom uh you know syncs to Dynamo DB to Reddist to so on and so forth whatever you want to write to it's not it's a you know very simple implementation using the PISA API to write to it.
So virtually getting data anywhere so what's possible today so I kind of gave an overview and next part of the talk is just like heavy code examples. All right, we're going to walk through like uh what what these new APIs look like, what's available, what's exactly available today, what's coming in the near future, and the advantages of each of these approaches. So, the examples I'm going to go for are using UC credentials with message buses and how simple that is.

`14:02` Secure streaming integration

I'll go through the overview of the DLT sync API and using that API to write to any of the delta tables and I'll go through the advantages and disadvantages of each. So the advantages of using a streaming table, what you might want to use there versus if you want to write to an external or a managed table. Why would you want to do that? Uh I'll go through a it says customer pispark sync my mistake but it I'll go through a custom pi spark sync writing to plunk and uh we'll go through writing to custom integrations particularly a fanout use case using for each batch.
So writing to multiple destinations from the same flow. So you see creds with your message process. This is perhaps uh so we just launched this like two weeks ago ahead of DS like we're still updating the documentation so it's really fresh. Um it's in public preview. This is perhaps one of the uh features that I'm most excited for on DT because it really simplifies security and governance. All right, walking through the code on the right hand side, we see we have a table named for Kafka ingest.
All right, and I have a read stream with my like normal Kafka options set up. And then if you go to the last options you see I'm picking I'm using a data brick service credential with a UC cred name and in there you can put uh you know confluent credentials msk credentials GC like if uh not relevant for Kafka but any of your cloud credentials basically can be used that we will rotate them integrate with AM for you out of the box and then when you want another developer to go build another use case this isn't go set up an IM M roll again get it to work with data bricks make sure it's loaded or hardcode credentials in your code or anything like that it's simply give them access to the credential and they'll have access to the external connection via UC so it drastically simplifies that connectivity process of getting data from your message buses our goal here is to support these credentials wherever it makes sense across the board so we made it uh these work and some of the other connectors uh we'll work on Next, DT sync API.

`16:37` Using the DLT Sync API

This is an API we've worked on uh like we had a private preview um starting in December of last year and we went public preview I mentioned say like around April. Um and as I mentioned before this is the mechanism in which we're going to provide declarative syncs and integrations to you across and get get par with structured streaming and um uh Apache Spark the API is very very simple you create a sync that sync can be used across your pipeline so it's not it's isolated to the pipeline we don't reuse syncs uh um across pipelines or across workspaces.
You define a sync here but you can reuse the connections in UC as I mentioned and then you basically attach an append flow on it very simple code of you know your name and then your target on the whatever sync you created and your normal DT code all right so it's I would consider syncs this is like a new first class uh concept in DT all right uh similar to streaming tables materialized views and views. This is another option to construct your pipelines.
So you can do basically everything here. I mentioned um we'll cover some other examples, but you can put JDB syncs in here, Dynamo DB syncs, so on and so forth using the PISpark uh Python data source API. So let's go through a specific limitation of the DTSync API. This is writing to the slide says writing to any delta table, but this allows you to choose based off of your use case whether to write to a streaming table, whether to write to an external UC uh UC manage UC external table.
So, first before I go through the code, why would you want to do this? All right, like why wouldn't you just use streaming tables? There are significant advantages to scoping and defining your data with your codebase and versioning the data with your codebase. All right, it's easier to manage especially if it's scoped down to just your pipeline. So the if I were to simplify the recommendation of when to use a streaming table, it would be if your data processing can be confined within the pipeline, use streaming tables.
That is your best bet. You'll get incremental processing. You also get advanced features like expectations, apply changes into, so on and so forth. However, if you need to work across many different workspaces or pipelines or you need lots of external connectivity or you want to manage your catalog and schema independent from your code, streaming tables prevent that. The DT sync API allows you to have depending on how you've set up your data platform within your organization.
It allows you to if you want to manage the schema and catalog outside of DT, go ahead. If you want to do some of things in streaming table and some things using uh delta manage tables, you can. The great thing here is you can go back and forth too. So you can have a pipeline with a flow to an ST, a flow to an MV, an MV to a external table, an external table back to a streaming table. And this is especially important for external readers and things like that.
It also is working across organizations depending on where they are on their UC journey where they're on the their uh adoption of DT it might make sense to use exclusively you know managed tables or exclusively external tables. But the point is it gives you the flexibility. So going through the code it's as you can see it's very I'm using a fully qualified name uh to mention the table which is the best practice and it's a very simple uh uh append flow into with a simple expression.
So previously like one of the most common reasons customers like pain points customers had is how do I get data out of streaming tables? All right so what they would do is they'd run a structured streaming job to basically read from a streaming table and then write to an external table as a copy. It's like wasted processing and you'd have to use a completely different tool. So there's a product cliff there. You no longer have to do that.

`20:58` Custom Python sink examples

All right. Custom PIS spark syncs. So this is an open-source um API that was designed specifically for Python developers to build custom integrations. All right. So we have I mentioned two other APIs and interfaces. Data source v1 and data source v2. These were you know they're very very fullfeatured interfaces to build custom integrations but they're very like scholar Java you know heavy. So if you want you know to write a simple Python integration you're you were stuck using a primitive you couldn't do something reusable this new functionality allows you to uh get away from that.
So for example, you build this uh pi spark s once to say some rest endpoint you're writing to as one simple example. All right. Uh we have splunk here. All right. You build it once and that code can be reused like through your CI/CD and code repositories across all of your pipelines uh very very simply and you do it once with a shared codebase to help enable interoperability across your organization. So this was uh available in open source for a while but now it's uh official released in Spark 4.
0 and DBR 15. 4. You can use it for both data sources and data syncs. So you can build custom integrations on data sources as well as syncs. Um it not not necessarily that relevant for DT but you can run batch uh you can also build batch integrations with it. The example I have is a very very simple push to splunk. So imagine you have a DT pipeline. Your use case is I would like to force perform some uh analytics some massaging of my telemetry data.
All right before I forward the data to Splunk namely because I want to filter and aggregate data so I don't spend however much on my Splunk bill. Right? Good use case to get telemetry for security data and everything like that. now is definitely is very in couple lines of code very simple to implement on DLT and this is a very you know very different from the use cases that we started with on DT the injection in ETL so the point of me showing Splunk specifically is that DT with these changes including this one and the other uh things I'm presenting today is now uh viable for a much more broader set of data engineering challenges and Well, uh, another one of my favorite um, actually just out of curiosity, a show of hands, who knows what for each batch is andor has implemented before?

`23:28` Advanced ForEachBatch usage

So, not a huge portion of the room. All right, so for each batch is a primitive that is um, a very very powerful primitive. All right, that has been used in structure streaming for a while to support a number of use cases. When we did research uh last year to figure out what the most popular use cases for each batch were, they were the following. So custom merges into delta. All right, so like things like fanning out reading from a uh data source that has multiple different row types or event types in it and fanning out to n different delta tables.
All right. uh doing a custom merge based off of your own logic um uh writing to multiple destinations. So uh one customer I know work writes to Delta use it and also writes to um uh Cosmos DB in the same connector. The other and final use case that it was most popular for was basically it gave you access to every single batch integration. All right. So if there was a batch connector that you wanted to reuse, you could now use it in the streaming context.
So you'd have your streaming pipeline and this for each batch sync is effectively like and then we're going to run this mini batch job on the data you just processed. And you could reuse any connector within the much broader set of batch uh uh connectors uh within a streaming pipeline. But there's a catch with this. So it's not like this be all gets all thing. It does solve. It's a very flexible powerful uh sync but issue is is that it is there's limited housekeeping and state management.
You control that on your own. Uh the it's a lot of custom code to write. You don't get expectations with it in DT and there's very limited telemetry. So you're going to have to publish your custom metrics like how many records you wrote out or things like that. So very very powerful. Um gives you access to a lot of different functionality but there are def definitely some trade-offs. It's not a perfect solution, but the best part about this for DT users is that we apply changes into is a very very very powerful piece of functionality.
All right, if it works for your use case, it's cost effective. All right, it's fast. Uh it does incremental processing and it's great. But what happens if that doesn't work? All right, and there's a set of use cases and reasons why apply changes into and we're always proving wouldn't work. the product cliff would be you'd have to go use structured streaming or copy the data and get do something else maybe run a batch job to do the merge after it.
Now with an introduction this functionality you can perform things like fan out and merges within DT without switching products. The final thing I want to mention is with this connector when we go to public preview uh we and actually in the private preview now we support full refresh support. So one of the other reasons why people like to do custom merges with for each batch is because they like to man manage their own item potent merges.
So like if they have to reprocess data or anything like that they want to manage how that occurs. All right. This allows you to do that uh with this operator and it's fully supported. So if you do a full refresh on a DT pipeline that uses for each batch as a sync it works seamlessly. So, I went through what's available for you to try out like today. All right. Um, I'm now I'm going to go through where are we going and I'm going to be pretty explicit about uh timelines on things that we're going to be supporting.

`27:03` Update flows and custom merges

One of them is the merge use case that's uh upcoming when we go to public preview. So, I mentioned the um for each batch and custom merges. Here's a good example what that would look like. So you kind of get a uh what I the way I like to think with the changes we're making here with DT is we are declarative first but we allow you to bump out into imperative if you need to. All right. So, we're declarative and simple first, but if it doesn't work for you and you need fine grain control over how things happen, here's an example of uh how you're able to so this is a for each batch sync.
Um you can see that I am uh let me sorry, let me read this. You can see that um I'm doing a merge within that sync with specific conditions on that merge. All right. And what's going to be coming soon is what I'm very excited about which will simplify this use case significantly is DT will support update flows. All right, meaning that instead of just being able to append data to it, we will support the update output mode in structured streaming.
This is very important for um especially stateful stream processing. And I'll walk through a use case of why. So imagine from silver to gold you're doing an aggregation. All right. And in that aggregation, let's just say it's a sum or minmax or whatever. All right, you're performing that aggregation. All right, and then writing it out to the uh destination table. All right, with append, you set your watermark and you'll get a single result and if any data arrives late, it gets dropped.
You don't get updates, especially so you're forced to make this uh tradeoff on latency or completeness that's very like bipolar, right? You can choose a short watermark and prioritize on latency to append the data with that staple aggregation as quickly as possible. But then any data that arrives late, you'll drop. Or you can choose a very long watermark in an append flow. And when you choose that very long watermark, what'll end up happening is the data uh say let's just say you say an hour or two hours or six hours or eight hours, a you're going to use a lot more resources in state and memory to do that computation.
All right? and B, uh, you're only going to get that output once and it's going to be however long the watermark is delayed, right? Append flows are very, very powerful and simple, but they don't fit all the use cases and they don't provide a good trade-off on latency and completeness. Update flows solve that problem. So update flows, instead of getting a singular result at the end of the watermark, we will give you an intermediate results up until the end of the watermark.
So you have an aggregation. The first time we have uh this aggregation, we send it downstream. If we receive new data, we update the computation and send like kind of like an amendment downstream. Right? So we do this continuously until the end of your watermark. So the watermark effectively becomes almost like an allowed lateness value where you can tra still get the latency but can be have a lot more complete results. why this is so relevant to merge use cases.
Uh if it's not self-evident, let me just uh spell it out a little bit more clear. It means that if you design with like a for each batch for your merge to be like item potent on this, you can get continuously updated fresh results with a very little code just by setting it to update and then just doing upserts into uh as based off of your merge based off of a key. And the powerful thing this won't work with just delta.
This will work with Postgress databases, MySQL databases, so on and so forth. This the green slide that I showed up that grid. You can do this type of processing with low latency with completeness across a vast array of destinations using DT. Now, so this is coming um this is available in the preview channel today. I I want to caveat that uh because I we recently rolled back the preview channel from 16. 4 4 to 16. 1.
I still think it's available, but I'm not but it'll be available soon. If it's not available in the preview, it's been available for a couple months now in the preview channel. It'll be available in public preview sometime in half too. Uh hopefully Q3, but uh we will see run low latency operational use cases. This is a um brand new piece of functionality that I think will uh um announce on the stage tomorrow that we're open sourcing it, but the SPIP is already out there in the dev and user thing, but it's a new trigger type within structured streaming.

`31:39` Operational real-time usage

Uh for those more familiar with DT, you kind of have two trigger types. You've got uh continuous and like scheduled or triggered, right? Those the underlying uh structure streaming primitives that those use are two trigger types called available now which processes all the data that's available now and then a processing time trigger which will run the uh the query continuously based off of some fixed interval be it like 0 milliseconds or 500 milliseconds it'll just keep on processing based off that interval.
This is a new trigger type where you specify as you can see real time. All right. And what it provides is continuous processing over um as d and enables data to seamlessly flow through the app and send results out immediately. All right. So unlike uh a traditional structured streaming job where you schedule your job and one stage occurs and then another stage occurs reading from memory and then it writes to memory and then another stage occurs so on and so forth.
What happens is data flows seamlessly through the tasks at low latency. So uh we shared last year in early stages PC and we have other talks this where we're sharing where we're at with the latencies but we have now have customer vetted latencies on this that they're hitting median latencies in the tens of milliseconds. All right and P99 latencies for the very complex multi-operator use cases in the hundreds of milliseconds.
So why does this matter? Again, it with DT, we started with primarily support of injection and ETL with support for this. All right, we will be you'll be able to run basically very low latency operational use cases. So think Kafka to Kafka, not hitting delta, Kafka to a database, uh Kinesis to Reddus. So if you have use cases where like for example you want to uh read clickstream data in near real time all right process it and then give a live recommendation back to your customer you can do that at a point in sale if your customers in your store shopping around and they're using your app you can send them a notification for a discount code right that opens up a brand new set of use cases and potential business value with this trigger and this will be available in DT as you can see with the update floats sometime in the second half of the This feature will first be available in classic um uh and standard clusters but DT will come very soon.
Public preview for uh classic on DBR is going to be right around August. All right. And the final point about this uh this trigger type what's very very cool is similarly to mon going from a triggered to a continuous job very simply you'll be able to go to this real-time trigger mode with like kind of a flip of a switch with no changes in code. So if you have super efficient triggered streaming job pipeline that is running in DT that's running once an hour and then your business comes to you and says we want that to be real time.
We need it faster. we need it faster. You're talking about like one or two config changes and you're moving from once an hour to subsecond processing. All right, with no other changes. So very very powerful at the same on the other flip side. If you you know implement an operational use case and you're like you know this is valuable but we really could use the efficiencies of just doing this once an hour. You can go the other direction.
So the power of moving with this like latency kind of bar on where you need to be is with simple config changes is what this unlocks. So we're coming up to the end of my presentation. Um this is the I think the point I'm trying to convey here is that with the we've made already a vast majority of a vastly new use cases are now implementable on DT especially reverse ETL use cases different types of merges and things like that and within the next couple months you'll be able to implement a much wider variety of use cases basically to serve almost all of your data engineering needs using DT our goal is to have full parody with structured streaming on DT There's other things we're doing around monitoring around and things like that uh around like loading custom libraries.

`35:28` Future outlook and wrap-up

Other things like that we're we also are doing with this but this is probably the most important is making sure that interoperability and integrations are the same regardless of what product you use on data bricks uh regardless of uh you know what type of use case you want to implement. The other area we're going is we're going to start building more native integrations. Uh we're still figuring out what we're prioritizing.
By far the most popular customers have asked for are native JDBC syncs. So a native streaming sync that's a low latency right to say your postgrace database or any JDBC or lakebase any of your JDBC integrations. Um no SQL databases come up a lot. And then the final one is the right optimized delta sync. So our DevTools stick is really hyper um uh it's really really configured and set up to make sure you get very good read performance and we save on save on object storage costs right however there are a lot of customers that you know my business keeps on asking me for lower latency I can deal with the smaller files the increased object storage cost or I'm using some version of object storage that's cheaper that I can get high high throughput writes on I want to write as fast as possible that's another native streaming integration that the customers have asked for.
So I I'll go through this slide a little bit more quickly, but governance and security everywhere. The we did this for message buses and we've done this for a number of batch connectors, but our goal is for you to be able to manage these integrations using UC, right? So you're you don't have to ask a data scientist or uh an engineer to file a ticket to go you know to your security team to get a new AM role and to give that person permissions or that workspace permissions every single time.
You just manage it through UC and through secure infrastructure using uh Spark connect as well as uh us handling the uh rotation the credential rotations and everything like that. So call to action. I borrowed really heavily some very smart people for some of these slides. They're uh who is the tech lead on the Python data source. She has a awesome examples on that GitHub repo for it. Uh Alex has two great blogs that were written in the last couple months that cover the 4HP stuff Python data sources and how to use it within DT.
So, step-by-step code examples, walkthroughs, and uh we'll uh we saw a lot of this stuff is coming in hot and fast with the high velocity. We'll uh in the process of updating our own documentation, doing as many examples as we have. And finally, and I'll end with one minute of Q&A, please fill out your surveys. It helps us out. Let us know what talks to prioritize um next year. So, really appreciate uh all your time.
And for right now, I'll just take some questions for the last minute.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.