Lakeflow Connect: Easy, Efficient Ingestion From Databases

Summary

• Database Ingestion Simplification: Lakeflow Connect streamlines incremental data ingestion from popular databases like SQL Server and PostgreSQL, simplifying the ETL stack on Databricks.
• Best Practices Framework: This video reviews practical guidance on networking, security, minimizing database load, monitoring, and common industry scenarios for database integrations.
• Unified Data Engineering: Lakeflow Connect is part of Lakeflow, the unified data engineering solution combining ingestion, transformation through declarative pipelines, and orchestration.

Lakeflow Connect: Easy, Efficient Ingestion From Databases

Talk by Bret Grantham, Member of Technical Staff, Databricks ; Peter Pogorski, Senior Product Manager, Databricks
Lakeflow Connect streamlines the ingestion of incremental data from popular databases like SQL Server and PostgreSQL. In this video, we’ll review best practices for networking, security, minimizing database load, monitoring and more — tailored to common industry scenarios. Join us to gain practical insights into Lakeflow Connect's functionality so that you’re ready to build your own pipelines. Whether you're looking to optimize data ingestion or enhance your database integrations, this video will provide you with a deep understanding of how Lakeflow Connect works with databases.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to Lakeflow
• 1:23 Lakeflow core vision
• 2:51 Data ingestion challenges
• 5:45 Lakeflow Connect capabilities
• 7:11 Supported data sources
• 8:17 Connector architecture
• 12:04 SQL Server demo
• 28:30 SQL Server connector roadmap
• 30:37 Postgres and MySQL updates
• 33:47 Pricing and conclusion

FAQs

What databases does Lakeflow Connect support?

Lakeflow Connect supports popular databases including SQL Server, PostgreSQL, MySQL, and Oracle. The managed connectors enable easy and efficient incremental data ingestion from these sources with point-and-click simplicity or powerful options for advanced users.

What best practices does this video cover for database ingestion?

This video reviews best practices tailored to common industry scenarios, including networking setup, security configuration, minimizing database load, monitoring strategies, and database connector functionality. These practices help you build reliable database integration pipelines.

Lakeflow Connect versus Fivetran for ingestion?

This video presents Lakeflow Connect as Databricks native database ingestion solution, supporting SQL Server, PostgreSQL, MySQL, and Oracle through a unified experience. Unlike external tools, Lakeflow Connect integrates directly with the Databricks Data and AI platform for consolidated governance via Unity Catalog and orchestration via Lakeflow jobs. It handles schema evolution, automated retries, and data quality monitoring, reducing maintenance overhead compared to point-to-point connectors.

Lakeflow Connect versus Fivetran for enterprise CDC and SaaS ingestion?

This video demonstrates how Lakeflow Connect provides native change data capture (CDC) and snapshot capabilities for enterprise databases, with built-in support for schema evolution and delete handling. By integrating with the Databricks Data and AI platform, organizations get unified lineage, governance, and quality monitoring across all ingestion pipelines. The platform-native approach eliminates the operational complexity of managing external CDC tools.

How do native connectors handle schema evolution, deletes, retries, and data quality?

This video details how Lakeflow Connect automatically detects schema changes without manual intervention and adapts incrementally using primary key and cursor column logic. Delete and truncation events are handled transparently, while built-in data quality alerts flag issues early. Unity Catalog integration provides automatic lineage tracking and compliance auditing across all data transformations.

Full transcript

0:00 Introduction to Lakeflow

So I think we have a few more people trickling in but we'll go again ahead and get started today. Um I'll start off like all sessions with a forwardlooking statement. So, it's important we show this slide to mention this presentation is forformational purposes only and it may contain forward-looking statements as we continue to develop this product for the future. And let's start off with a couple of introductions. I'm a product manager on the ingestion team at data bricks.
And I'll be joined by an engineer on the ingestion team. and he'll be showcasing a super exciting demo using one of our database connectors today. And in our session, we're gonna discuss the manage connectors that anyone can use to incrementally ingest data from popular database sources such as SQL Server, Postgress, and MySQL. We'll start off by setting some context around LakeFlow. And then we'll deep dive into the database connectors.
From there, we'll demo the product, discuss the road map, and then share some of the best practices for database ingestion using LakeFlow Connect. So, let's start off with some critical context. LakeFlow is our unified data engineering solution for ingestion, transformation, and orchestration. It's essentially our vision for what data engineering should be in the evolving world of data and AI and what makes LakeFlow so great.

1:23 Lakeflow core vision

To start off, LakeFlow will provide an endto-end experience across all of those steps. So, it really simplifies the ETL stack on data bricks. It's also easy to use for those who want that simple point-and-click offering while also being high-powered for users who are fluent in things like PiSpark or have needs like subsecond streaming. And finally, it's going to produce reliable, fresh, and accurate data that downstream users can trust.
For example, LakeFlow has integrated and proactive quality quality monitoring and alerting. And in terms of the actual products, there's three key pillars that make up LakeFlow. The first being LakeFlow Connect, which is for ingestion. The second being LakeFlow declarative pipelines, which are for transformation. And then finally, LakeFlow jobs for orchestration. And today we're going to dive into LakeFlow Connect, specifically focusing on the database connectors.
But first, why did we actually build Lakeflow Connect? You likely have a lot of different databases. This might include SQL Server, Postgress, MySQL, maybe an Oracle database or two. And it can be really difficult to get that data into your lakehouse so that you can actually do meaningful things with it. To start, database ingestion is hard to ramp up on. For example, every database is going to have its own protocol, authentication method, metadata API, and even within the same database.

2:51 Data ingestion challenges

This behavior can vary quite significantly across the different database versions. And then change data capture mechanisms, they're going to be different across every single database. Some support logical replication, others use transaction logs, change tracking, or maybe even triggers. And many database APIs are often undocumented, unstable, or they're not very performant. And source databases can have proprietary or extended data types that are hard to map cleanly to targets like parquet or JSON.
And running CDC and snapshot queries can affect the production performance of the source database. So CDC ingestion really needs to kind of strike a balance between throughput versus the impact on the source. And then moreover once you've actually set up your connector, you're going to need to maintain it. So how are you going to handle schema evolution? How are you going to actually monitor both the pipeline health and the data quality that's coming in?
And how are you going to actually recover if the database or the extraction infrastructure actually goes down? And then when you take that and multiply it across all of your different database sources, you're going to have a lot of complexity. So it's no wonder that organizations spend a lot of time not only building but also maintaining their ingestion pipeline. And given all this complexity, that work often lands on a specialized data engineer who has a bunch of other important things to do.
But in spite of all that, you might not want to even democratize that data because you have a patchwork of different pipelines which are very hard to govern effectively. For example, across all those users, who's actually tracking lineage to make sure we're not duplicating all of the tables or who's implementing data governance and best practices on these pipelines. And then we also want to make sure that nobody's blowing up the company's data engineering bill and that everyone's actually respecting data retention and PII policies.
So at the end of the day, it's hard to get the full value from your data. You're probably running into pipeline outages and you might even have some security risks with that data. So that's why we built LakeFlow Connect. In short, LakeFlow Connect aims to solve these challenges by enabling simple and efficient data ingestion into your lakehouse. And there are three key things that we're most excited about. The first is the simplicity and ease of use.

5:45 Lakeflow Connect capabilities

You can set up a production ready pipeline in just a few steps. And once you've set a connector up, it's also fully managed with schema evolution, observability, alerts, automated retries, and even more. So this is going to lower your overall maintenance costs, but more generally, it's going to mean you can spend less time moving data and more time actually getting value from that data within the lakehouse. And lakeflow connect is also unified with your existing tools and governance. uh for example the pipelines and the tables you ingest are governed by Unity catalog and it's all orchestrated via lakeflow jobs.
Finally, LakeFlow connect is efficient with incremental reads and writes wherever possible. And moreover, since ingestion is just the first step, we actually write the data to support incremental processing downstream as it works its way through the medallion architecture. And along the way, it has a simple cost-sefficient pricing model that's simply based on the compute which is used by your pipeline. So what sources are actually supported with LakeFlow connect?
Over the past year, we've been focusing on expanding out our connector breath and we now have 14 connectors across various release st uh various release states and eight of those connectors are now generally available. And then in addition to our native CDC connectors, we're also announcing a new set of connectors for databases and data warehouses which are going to enable you to ingest data without making any source modification such as having to enable CDC or creating a replication slot on the database.

7:11 Supported data sources

With these connectors, we're able to incrementally ingest data by leveraging a primary key and a cursor column. And if these columns aren't available on a table, we're still able to ingest the data, but we'll just do so using a batchbased method. And these connectors will be coming very soon and we'll support all of the sources which are currently available via lakehouse federation. So that's a highle overview of LakeFlow Connect.
Let's now lift the hood on these connectors. We'll start off by talking about the architecture of a database connector. Each database connector has the following components. The first is going to be a connection which is a Unity catalog securable object that stores authentication details for the database. The second component is an ingestion gateway. So this is going to be a pipeline that takes metadata snapshots and change logs from the database and it's going to stage them in a Unity catalog volume either backed by S3 or ADLS.

8:17 Connector architecture

And the gateway runs on classic compute in order to simplify the network connectivity to your privately secured database and it will run continuously to keep up with the log retention on the source database. We then have an ingestion pipeline which picks up the data that was staged in Unity catalog and applies the changes into the destination tables. This pipeline runs on serverless compute and can be scheduled to run uh with whatever custom schedule you want.
And finally, the destination tables where the ingestion pipeline is actually writing the data. These are going to be streaming tables which are delta tables with extra support for incremental data processing. And now you might wonder how we actually establish that connection to your source. For cloud databases, we can connect via private link or if a workspace has appeared v-net or VPC, we can deploy the gateway right inside of it.
And then for onrem databases, we're able to connect to your database directly from your workspace using services like express route, direct connect, or a VPN. And with LakeFlow Connect, we're able to offer efficient data extraction through some key capabilities. First, we've optimized native CDC for each specific data source. So, this means we can capture changes while minimizing the over overall impact to your system performance.
Second, we offer flexible ingestion options so that you can start extracting data immediately. So without any complex source setup being required and this combination means that you can begin seeing value from your data in minutes and not weeks while maintaining the reliability and efficiency that your business demands. Second, LakeFlow Connect is built for speed while being efficient through intelligent parallelization.
We run multiple parallel processes both between tables and within individual tables, reducing your initial data load times. But we don't stop there. We're also going to parallelize your ongoing CDC loads alongside historical snapshots, so you don't have to wait for one to finish before the other starts. And finally, one of the biggest pain points with data replication is handling failures and restarts. LakeFlow Connect will eliminate this headache with automatic checkpointing.
If your connection drops or you have a process which fails, you don't have to start over from scratch. Our connectors resume both CDC streams and snapshot operations from the last successful checkpoint, saving you hours or even days of reprocessing time. And this built-in resilience means your data pipelines are truly enterprise ready by rec recovering gracefully and keeping your data flowing. So I'm now going to hand it over to demo one of our database connectors.
All right. Thank you. Check. Check. Okay, good. Um, hi everyone. As Peter mentioned. Uh I am a developer and the subject matter expert for SQL server at datab bricks and I have the privilege of working on lakeflow connect. Um I'm based out of Los Angeles now but I grew up in southern Louisiana and uh living in California you are spoiled for choice with 24 professional sports teams. But back in uh Louisiana there was only one team that mattered.

12:04 SQL Server demo

Um, well, there's a basketball team now, but whatever. Uh, so and the uh the fan loyalty there is just absolutely legendary. So, um, today we'll be working with that Louisiana team that for the for the purposes of this demo, we'll call the Crescent City Crusaders. Okay, this team is terrible, folks. Look at last year's record. Five and 12. That's laughingtock territory. The fans wear paper bags over their heads at the games.
That's true. This team's in trouble and uh with their salary cap issues. It's not going to come. The help's not going to come in the form of better players. Enter Bayou Analytics. I love that logo. Let's look at their mission statement. Empowering the Crescent City Crusaders to achieve championship excellence through innovative data analytics and actionable insights that drive strategic decisions on and off the field.
It's powerful stuff. Okay, meet. not doing so well and that's because he is a huge fan of the Crescent City Crusaders and they're terrible. Also, he has exported 47,000 CSV files manually this year, and he's got more database connections and social connections. I hear he's even being seen down on Bourbon Street on week nights. He's a mess. But luckily, there is hope for because he has recently learned that data bricks now supports ingestion from SQL Server databases.
So, uh, after trying and failing with other products, could this finally be the answer? Okay. Um, so this is my demo video. In this video, we'll follow from his own one yard line where his data is trapped in SQL Server to the end zone with clean delta tables in Unity Catalog. We'll watch him drive down the field by setting up his ingestion gateway, configuring continuous ingestion from the Bayou Analytics database on Azure to our staging volume, avoiding fumbles with proper change tracking, and scoring a touchdown with a complete pipeline.
So, let's head over to and take a look at the data in our bio analytics database. Okay, here we are in our Azure uh database. I mean our bio analytics database. We have four tables. Uh kickers, quarterbacks, offensive linemen, and wide receivers. First, our kickers table. Yeah, this tracks our special teams roster, a small specialized group that's critical to game outcomes. With only five players, roster changes here are infrequent but impactful.
Each uh signing decision affects our salary cap and field goal reliability. This table manages our uh most expensive, most valuable position group. We track contract details, salary cap implications, and player descriptions for our quarterback depth chart. These are moderate frequency updates driven by injuries, performance, and contract restructures. And next we'll see our offensive lineman table. And just like an offensive lineman, it is much larger than kickers and quarterbacks.
This table captures our offensive line performance data, tracking blocking assignments, pressures allowed, and grades for our five starting linemen plus backups. With 2,00 performance records, this represents substantial game-by-ame analysis that helps our coaches evaluate performance and pro protection schemes. And finally, our largest, highest volume data set with 12,000 records, the wide receivers table. The wide receivers table is the widest table we have.
This table captures the most granular receiver analytics. We track every target route, reception, and performance metric across practices, games, and training sessions. Wide receivers have the most diverse skill sets and situations to analyze. generating extensive data for our offensive coordinators to optimize play calling and personnel packages. Okay, so those are our tables and before we set them up for ingestion, let's understand our options.
SQL server offers uh two mechanisms for checking data changes. Uh one is called change tracking and that is the more lightweight process. It just it just tracks whether a column has changed, gives you the latest state. Um CDC on the other hand, change data capture is a a little heavier process, but it captures all of the before images and after images of all your data. So this is good for if you uh need complete audit history um with your with your dragging or if you just need synchronization.
Um that's for CT. So, we'll go ahead and set up this necessary CT and CDC configurations on our tables. And I'll walk you through this process step by step. First, we're going to enable CT with the first command. That's going to enable it for the whole database. And we'll enable it with a reasonable retention period. This this configures our database to retain change tracking information for three days with automatic cleanup to prevent excessive storage use.
And uh our offense um our kickers and quarterbacks table are uh perfect for change tracking because they don't have pime they have primary keys and we just need to know what changed. So we don't need the history. So, so let's enable CT on our kicker table and our quarterback's table. All right. And now for CDC, let's enable CDC on uh the database level. This command activates the CDC framework for our entire database, creating the required system tables and capture and cleanup jobs.
Our offensive lineman and wide receivers tables are ideal for CDC. They have no primary or unique keys and we just need and we need to complete change history. So we'll enable CDC on our offensive lineman table All right. So now uh we just have one more step before the bio analytics database is fully set up for ingestion. We need to install the necessary DDL support objects in the database. This is these objects allow lakeflow connect to keep track of schema changes.
Uh this is a one-time setup step and datab bricks provides a convenient setup script for this purpose. And we'll go uh to the data data bricks documentation site which has so cleverly bookmarked and download the script. Now we're going to uh run this script against our Bayou Analytics database. Okay. So, first we need to provide some information in the script. That'll be up. There we go. You'll see options for CT, CDC, both, and none.
None will remove the objects from the database. Um, we're going to use both since we're using CT and CDC. And we'll also supply our username here. Supplying a username directs the script to enable the privileges for using these objects. These objects are for CT are a trigger and a tracking table. And for CDC, there's a few sort of procedures for managing capture instances. And that's it. That's the uh we're set up for ingestion.
So now that our source is prepared for ingestion, we'll head over to um the datab bricks web UI and we'll select new add or upload data. Here we'll select SQL server. We'll create our ingestion gateway. I'll create I'll call it by gateway. We'll select this location for our staging data. I'll pick my catalog and I'll create a schema. We'll call it crusaders. And we'll leave a nice little descriptive comment so no one is confused about which crusaders we're talking about.
And all right. And now we'll go to the next step and in and set up our uh ingest our our pipeline which will read from staging and write to the delta tables. We'll call it by your pipeline. We can see our destination catalog there. Um we'll create a connection to our source database by connection. Copy in our host domain URL. There's our username bio analytics user our super secret password and port 1433 for SQL server.
We'll click create connection. Here you can see it already selected. We'll click create pipeline and continue. we'll select the tables we want to ingest. So in about 8 seconds we'll be doing that or yeah we're just fetching all the schema and metadata. So here we see our source database and our uh schema. We'll just select the whole schema. We hit next. Select our Crusader schema for the destination. And now we'll click create pipeline. we will head to our fifth and final step.
We have just to uh set some settings, some optional settings. There we go. So you can create a schedule that will schedule the pip the applier pipeline to write read from staging and write to your delta tables. Um I'm just going to leave the default of to read and write every every hour. You can also add an alert. Um I'm just going to leave the default of the current user to receive failure messages. And we'll click save pipeline and continue.
Here we go. All right, now that is it and it is spinning up and uh so it's we'll just give it a moment to come up. But while that's starting, I'll talk briefly about our schema change tracking. Um if if the schema change is additive, like you've added a column, we will require a full refresh of the table. But if it is not additive, say you drop a column, we'll just keep on trucking. No, no refresh required. You will see the value stop getting updated but you'll keep the historical data.
So all right initializing. Oh you can pick any any time, right? Anything. Yeah. Yeah. It could be seconds, it could be hours, minutes, days, or you could use table triggers. Okay. So here are our tables. You can see a nice little graphical representation. We can drill down into the tables. We'll take a look at the offensive lineman table. Here you can see the data coming in. We'll go back to all tables. And we'll see that our data has arrived.
If you look under upserted records, you see all our rows there. and we'll go. Okay, we'll go to our catalog just for sanity sake and make sure all the data is there. So, here's our catalog. There's our schema crusaders. Here you can see our tables. And I've also created a query that we're going to run just to be doubly sure all our data is there. So, we're connected to our catalog now. will connect to the Crusaders schema.
Untick that. Okay. Kickers. All 2,000 rows of our offensive lineman performance data. and 12,000 rows for our wide receivers. And there you go. So, and finally, I'll show you where to find the pipeline logs. Um, just let it show you the rows there. All right. So, logging, that's the last thing I want to cover here. Uh, we'll go to jobs and pipelines. There we go. Here you can see our pipelines. Here we can we can look at recent runs of the pipeline.
We'll just we'll dig in here and we can view any issues. We'll click view issues. And this is you can get to the event log. View issues event log. And here we can see anyformational me messages, warnings or errors. And if if you need to drill down deeper, you can actually go to the compute logs here and you can see standard error, standard out, log for JL logs. And with that, we have reached the end zone. And that's what we call a touchdown.

28:30 SQL Server connector roadmap

Yeah. Look how happy he is now. I don't know about that green drink. I think that must have came from Bourbon Street. But, uh, thank you very much. And with that we'll uh I'll talk about some more um SQL server connector um let's see with our SQL server connector we we s provide support for most SQL server environments running SQL server 2012 and above. This includes Azure SQL Amazon RDS SQL Server onrem and SQL server uh on VMs.
We also s provide support for all SQL server data types including some of the more complex ones like geometry, SQL vari variant and XML. We provide the ability to select between CT CDC and CT and we will optimize the selection based on table properties if both are enabled on a table. Okay. In addition to the features that are available today, we will be adding support for auto refresh when table uh cleanups occur. This could be because of some schema evolution operation or uh um truncate or some automated cleanup.
We can uh automatically refresh. We're we're also going to be greatly simplifying the setup of the native of CDC on the source database. You saw the process we went through. It was just a few steps, but we're going to make it even easier. We're pro we're going to automate it on our platform. So um and finally we're also going to expand the authentication options in addition to username and password and Microsoft intra ID arbback which are available today by adding support for micros uh Microsoft intra ID service principle based authentication.
And with that I'll hand it back over to talk about some other database connectors. Thank you. Awesome. Thank you. I really hope the Crescent City Crusaders do better now that you have their SQL server data in the lakehouse. Thank you. And I'm very quickly just going to go through some of our other upcoming connectors. So, let's start off with the Postgress SQL connector which is now available in private preview. If you do want to participate in the Postgress connector preview, reach out to your account team and we can get you on boarded.

30:37 Postgres and MySQL updates

And with our Postgress connector, we're going to support all major variations of Postgress. This is going to include RDS, and Azure Postgress. And our connector is built natively on PG output, which means you're not going to need to install any third party library on your database. and we focus on robustness. So we're going to provide full support for toast columns. We're going to support partition tables and the automatic handling of schema changes.
We're also going to include replication slot management. And looking ahead, we're looking to support replication from read replicas on databases running Postgress 17 and above. And we all know that with Postgress uh the replication slots that are critical for ch change data capture if left unmanaged they can fill up and they can bring the entire database down. So traditionally teams have to manually create and monitor those replication slots which is pretty operationally expensive and with lakeflow connect we've automated that entire life cycle.
So slots are going to be created as needed and then safely cleaned up. So we're going to reduce your risk of an outage and simplify the ingestion operation altogether. And lastly on Postgress schema changes are very difficult to replicate because in the Postgress write ahead log there's no tracking of schema changes um like alter table. So a lot of CDC solutions will either mislaim or require manual manual handling anytime a schema change happens.
And this is going to create a lot of fragility when the source schema evolves. With LakeFlow, we're going to handle all those schema changes automatically, whether it's a column being added, modified, or removed. So your pipelines are going to stay up to date without any manual intervention. And it's fully automated and built to support dynamic real world environments. And finally, we'll talk about our MySQL connector, which will be entering private preview later this year.
Our MySQL connector is going to be built directly on the native bin log format, meaning that no third party libraries are required. Once again, it's going to support change data capture from both the primary and the replica instance and it includes the full handling of binary data types and most most importantly it's also automatically going to adapt to schema changes like the column additions or the type modifications.
So once again, no manual intervention is needed. And now all of these connectors are powered by a simple, efficient, and entirely compute-based pricing model. You're simply going to pay for the compute that's needed to run your pipelines. And pipelines for SAS sources like Salesforce and Workday, they're going to run exclusively on serverless infrastructure and they're just going to incur serverless DBU charges. For database sources like SQL Server, the ingestion pipeline is going to run on serverless.

33:47 Pricing and conclusion

However, the ingestion gateway could run on classic compute or serverless depending on the source. So for database connectors, you may receive a combination of classic and serverless DT DBU charges. So that wraps up our chat about database connectors. If there's one thing to take away, it's that we're offering simple, unified, and efficient connectors that are intended for any data practitioner. To learn more, you can visit our product page or please reach out to your account team to participate in some of our previews or provide suggestions for our product.
And with that, and I will be up at the front to take some questions. But thanks everyone for joining.

Learn more about the Databricks Data and AI platform.