Lakeflow Connect: Cutting Enterprise Ingestion Costs by up to 90% on Databricks

Summary

Traditional CRM-to-lakehouse pipelines built on tools like Informatica, Talend, and Azure Data Factory require teams to manage compute resources, scheduling, monitoring, alerting, upgrades, and failure retries—creating high operational overhead and stale batch data that fails to meet near-real-time SLA requirements.
Lakeflow Connect on the Databricks Data and AI platform replaces this complexity with built-in checkpoint management, out-of-order event handling, and near-real-time CDC ingestion, reducing enterprise ingestion costs by up to 90% and cutting setup time from weeks to hours for a Fortune 500 client.
This video delivers a live 7-step walkthrough covering connector selection, OAuth setup, DLT pipeline configuration, object selection, Unity Catalog destination mapping, CDC monitoring, and production scheduling, along with a cost justification framework demonstrating multi-year TCO savings.

Lakeflow Connect: Cutting Enterprise Ingestion Costs by up to 90% on Databricks

Enterprise CRM-to-lakehouse pipelines built on traditional third-party ingestion tools can be expensive, time-consuming to implement, and often deliver stale data batches. We replaced this with Databricks Lakeflow Connect for a Fortune 500 client — significantly reducing monthly costs, cutting setup time from weeks to hours, and achieving near-real-time CDC ingestion. This talk delivers a live 7-step walkthrough: connector selection, OAuth setup, DLT pipeline config, object selection, Unity Catalog destination mapping, CDC monitoring, and production scheduling. Takeaways include a cost justification framework demonstrating significant multi-year TCO savings, a migration blueprint from legacy and cloud-native ingestion tools, and governance best practices with Unity Catalog lineage.
Talk By: Rishabh Pandey, Senior Consultant | Databricks MVP , Celebal Technologies ;

Chapters

FAQs

What are the problems with traditional CRM-to-lakehouse ingestion pipelines?

Legacy ingestion tools like Informatica, Talend, Azure Data Factory, and custom Python applications require teams to manually manage compute resources, VMs, clusters, connectors, authentication, scheduling, monitoring, alerting, and patching. This creates high operational overhead, risk of failures, and stale batch data that fails to meet near-real-time SLA requirements.

What is Lakeflow Connect and what does it replace?

Lakeflow Connect is the Databricks Data and AI platform's built-in ingestion solution with built-in checkpoint management, out-of-order event handling, and near-real-time CDC ingestion capabilities. It replaces third-party ingestion tools by handling operational complexity automatically, eliminating the need for custom logic and manual intervention while reducing costs by up to 90%.

What does the 7-step implementation walkthrough cover in this video?

The walkthrough covers connector selection, OAuth setup for the Salesforce connector, DLT pipeline configuration, object selection, Unity Catalog destination mapping, CDC monitoring dashboard setup, and production scheduling. It is designed to show a complete end-to-end implementation path from scratch.

How does Lakeflow Connect integrate with Unity Catalog for governance?

Lakeflow Connect writes data to Unity Catalog as governed tables, providing full lineage from source to destination and enabling auditability across the ingestion pipeline. All ingested data inherits access controls and lineage tracking automatically through Unity Catalog, without requiring separate governance tooling.

Full transcript

[00:09] Good morning to all of you here. Uh I'm Brisha Pande working as a senior consultant at celible tech. I'm a databix champion also an MVP from last two years. I'm also going to be next one as well. So this session is going to be
[00:21] a very uh interesting for you guys because databix has one launched a new feature that we name as lakeflow connect right where like we have to talk about like connect the cutting enterprise
[00:33] injection cost by up to 90% on data bricks because we do have lot of injection tools right it's cloud basics like some other like cloud bases right where we have to uh we are using currently as a traditional approach but
[00:45] now databix gives you that feasibility to use lakeflow connect which can give you a lot of exposure in terms of injection from external sources and you don't have to like create some custom
[00:57] logics right regularly and then it can like reduce your manual intervention and then you only have to get into how you move your data from one place to another without relying on the custom logics and
[01:09] then manual intervention right so that you can use one aentic models and all of the thing in this one uniform solution that is data bricks right so uh I'll start with what basically a traditional
[01:22] CRM to lakehouse pipelines are broken because we have a legacy injection problem previously which solve is by data bricks right now right so what exact the legacy injection flow
[01:34] previously we were having when talk about uh we have a source right I'm just taking an example of SQL and Salesforce right so and then we have this kind of flow right first of all we need to get data from Salesforce then we have
[01:46] infrastructure execution software We have structure streaming and then custom notebook and then next to delta ri right. So what does infrastructure with extraction software is mean right? What do you know about it right? So let me
[01:58] take some examples for you like you have 55ran Informatica talent Azure data factory AWS glue and also you have some custom Python applications running on VMs kubernetics and airflow scheduulers
[02:12] right in this particular flow right what teams have to manage like compute resources VMs clusters connectors and authentication right they have to manage
[02:24] their own scheduling method they have to manage monitoring and alerting Right? And they have to also manage some upgrades and patching and failures and retries. Right? So that is why the legacy injection flow is there which
[02:36] takes a lot of time to move data from one place to another. So now the legacy injection problem like traditionally injection pipelines were never designed were never designed for today's scale
[02:49] right we had source system like SQL Salesforce feeding into extraction tools custom streaming frameworks and hundred of notebooks maintained by engineering teams right so every new sources must
[03:01] mean more infrastructure more code more monitoring and more people required to support it right schema changes pro jobs incremental logic had to be handcoded. Troubleshooting become delay activity.
[03:14] Instead of focusing on analytics and AI initiative and you know that currently we have to lot of AI initiative right the databics is doing agentic models we have to like acquire those things. So we have to shift our focus from manual
[03:28] intervention to the AI shift right that is why so this growing operational burden is exactly why organizations are moving towards manage injection solutions like clickflow connect okay so
[03:40] this is a solution basically uh traditional CDC uh if you talk about like it's a kind of a datab bricks native injestion built into datab bricks it's a serverless governed multicloud so
[03:53] this is a kind of architecture that we have. So you can see like a lakeflow connect and it can take data from multiple external sources right now. Uh this basically yesterday Ali also introduced 100 plus connectors right
[04:06] from lakeflow connect. So you don't have to worry about manual intervention you just have to go choose the connector and give some connection strings and then you do it right. So why so traditional CDC was there previously right? So
[04:20] earlier if you wanted CDC change recapture you had to write code like this right like read source changes maintain watermarkings handle insert updates deletes write merge statements
[04:33] manage checkpoints handle out of order uh events like this and all of this had to be maintained by developers engineers right and architects right so now solution lakeflow connect is having a
[04:46] native CDC where they Wix automatically handles insert updates, deletes and like ordering of events, checkpointing, stage management and recovery from failures.
[04:59] So uh what is connector? I already told about it's a kind of management based connector that gives you feasibility of uh unit catalog enable system at the back end. It's DT pipeline. Now its name
[05:12] has been changed to is spark declarative pipelines and we also have workflow DAC for observability. So you don't have to uh go for another external third party tool to make your observability proven
[05:24] right. So we have unit catalog delta lake. So next slide I want to show you some cost estification and pricing model basically for lakeflow connect that I've been working for one of the fortune 500
[05:36] client customer where we have did some kind of assumptions on their budgets and how exactly their difference between from lakeflow connect to other tools right um so let's say uh let's talk
[05:50] about the elephant in the room right when it comes to modern data in the m tax you know what is It's a monthly active row right uh row right so your business grow your bill grow
[06:02] exponentially and with lakeflow connect database is fundamentally changing the economics of data ingestion up to 90% you can say cost reduction it depends on use case to use case like um it can be
[06:16] it could be like uh 40% it could be 50% for use if you have data a lot of amount of chunk of data right lot of connections so it can be but it will reduce your cost by up to 90% person for sure this is proven by data bricks right
[06:28] so because in this particular stuff we uh we charge based on actual serverless compute DB use databicks unit right that not not on row count so we routinely we
[06:41] routinely see uh set up to 90% reduction in tool cost of ownership right ECO compared to legacy MR based SAS platforms also we have like 25 uh 25x faster pipelines set up to production
[06:55] where uh Uh but cost isn't just about software licensing right it's about engineering uh overhead instead of spending weeks right writing custom scripts configuring connectors or uh
[07:08] managing infrastructure lakeflow connect takes minutes to hours to set up and deploy to production and you can go at right okay then let's look at how this plays out in real scenario of a team moving up 10 millions of rows per day so
[07:21] I think legacy Matt tool attend row a standard M tool will set you back between $4,000 to $15,000. It could be five six depending on the use cases, the
[07:33] connection that you have, the data that you are dealing with, right? And why? Because you are paying a premium license fee, license fee for every single row, the updates you are doing there regardless of how simple how simple the
[07:46] data structure is, right? And when talk about cloud data pipelines there you have I mean many other options like you can just there you could try using native cloud ETL tools but they charge
[07:59] per activity run and data indication units like that still runs you around $1,500 to uh this one right okay next so uh here I want to show you the seven
[08:13] steps end to end I mean walk through I have some screenshots attached that I get from my like systems like customers and then how exactly we are doing there. I cannot show you everything but I can show you some of the screenshot that I
[08:25] took from there. Let me show uh first of all uh what you have to do. I mean uh I have one question. I mean how can it possibly I mean you guys you guys have must have a question that how can it be
[08:37] possibly that that at le that cheap right dollar 0 to uh five $500 right how can be so uh cheap right so
[08:49] there basically why it's so cheap because there's no catch it's a structure business companies whose entire revenues model is built on margin heavy licensing of injection uh data
[09:01] bricks on the other hand is a unified data platform. uh we want you to bring other uh we want you to bring uh uh your data in the lakehouse easily and cost effectively right because the real value
[09:13] uh for both of us is uh uh when you run your analytics right and machine learning algorithms and agentic models and AI workloads on top of your data we don't need to uh put a heavy markup exchange on the gateway right like other
[09:27] M based tools are like because they are getting revenue from that particular stuff so even if you change any single uh visual in your data it will charge you but in data bricks you don't have to charge about it it will basically uh cost you how the number of DBUS you are
[09:41] using so there's a very uh good news for you all that datab is like giving you uh 100 uh DBUS per day per workspace as free to use lakeflow connect so at last
[09:54] the lakeflow connect is charging you nothing right if you go beyond that it will charge you some some sort of DBs but at least 100 DBs per day is free for you guys. I mean, so first step is to uh
[10:07] connector selection. There you have to choose the the method add data. You have to choose the Salesforce and any other connector. Then you have to go for connection setup. Right? There you have to choose like what uh connection string
[10:19] you have to give to Salesforce user password and then it will automatically take the data the connection string from Salesforce to this particular database right there. And then you will see like we have some sort of connection I have already create right created right. So
[10:32] then the next step is about injection setup right there where you have to choose like uh what exactly yeah you have to choose the default behavior like catalog where you want your where you want to store your metadata right in the
[10:45] default and then you can choose some other catalog as well. Then the next setup is about uh tables right. So here is a main step that you can check how many tables you have present in Salesforce right from there you can
[10:58] click on all those check box of the tables and then you can take data from particular tables also you can change the name as well if you want to rename any particular table name right from the cat from the source to target let's say
[11:11] you have one table name as u account right and if you want to make it as account underscore let's say sales and then and production that you want to take it there you can change the name of
[11:23] the table as well and then you can choose the there is one method as history tracking right in history tracking basically it's off by default and if you want to keep the sedd2 you can choose that method as true right so
[11:35] if you make it to true it will keep your table as sedd2 otherwise it will take you for sedd 1 right then you see destination there you have to like choose the connection method and then
[11:47] there you have to give the table where you want to stored in the UC enabled workspace there you can manage the permission fine grain access control as well everything from row to row column to column that's it right next is about
[12:00] CDC injection monitoring right there basically it's a kind of inbuilt data observability framework kind of thing that in the datab itself where you don't have to uh choose any external third party tool there where basically you
[12:13] have to give some tool for data obsibility that can make your lineage that can make you like logs that you need to check let's say if any pipelines got failed any day then what you have to do you have to go for transaction logs
[12:26] to check why it got failed but here it's a native CDC then you can come to this particular lakeflow connect and then you can see okay all the tables all the observability cat log logs and everything event stream everything query
[12:39] to query table to table you can see here and then you can check okay why my pipeline got failed why my pipel behaving at the back end right last step is about production scheduling. You can also schedule your jobs and this
[12:52] pipelines in this particular uh by giving chron syntax. You can also if you have any specific time window where you cannot have the UI method then you can give the crown syntax regarding that right also if you have some let's say
[13:05] you want to run the every four four hours you have to run this job right you can give the crown syntax otherwise you have to go to simple and then you can have advanc option as well where you can just choose the day time and the time zone as well if you are US you can
[13:18] choose PD you have to use I mean Asia kakata all of the stuff are present here right then Uh yeah so when I talk about governance because uh this table is uh this particular lakeflow connect is also
[13:30] giving you governance as well right so the problem in legacy systems once you ingest data you have to spend weeks setting up separate governance right tools and catalog schemas and third
[13:42] party like govern like kibra all right so writing data quality and configuring access control lakeflow connect solution is there uh because lakeflow connect is natively integrated with unic catalog the second data And it is fully governed
[13:56] linenous track and secured automatically. Why it's linenous? Because it can autoregister your streaming table uh created by data fle data bricks lakeflow connect automatically register itself by unity catalog inside the unity catalog right.
[14:09] So instant lineage also you will be able to see like because it will automatically draws the lineage graph showing exactly where the data came from like Salesforce like which downstream
[14:21] dashboards you are going to use what is the source table for your downstream table. So you can all you can see all the lineage in the lakeflow connect itself and then you can have the granular security u like row to row
[14:33] column to cable where you want to uh uh mask the PI data right email phone numbers and everything you can do that as well here right which we are not able to do previously and we doing that is like a encryption method and you're
[14:46] using some custom logics there so you are just do not have to do that like that we can do all the things in the lakeflow connect itself also we have here a built in DAG metrics you can see like which can connect uh back to the
[14:59] previous monitoring dashboards latency and record counts are tracked dynamically as well also we have I mean here data quality expectations right uh which can mentions about the current bad
[15:12] record because you know that when you are migrating from somewhere I mean due to some mistakes some data got corrupted some data got like missed from the source system and then you have to check you have to go to transaction log you
[15:24] have to make end to end solution how what record might go which record might got dropped right but here they connect that gives you that feasibility to give you the performance okay this record particularly got current quarantine
[15:37] because it this record got corrupted due to any this particular stuff right this will make that particular record to that current bad record table and then you can go and check what records are not following by criterias if you whatever
[15:49] criterias and constants you are setting up that if there constraints that record will get to the content table as well, right? Uh okay. Okay. So here basically this is for my fortune 500 client case study
[16:02] where like uh they were using a legacy marb base right. So the assumptions were about 500 million per records data ingested and 15 25 enterprise so systems
[16:14] they have they have CDC enabled injection also and also right now they are having 10 to 20 active workspaces right which can includes operational and maintenance effort as well. So if you
[16:26] use legacy madbased system what is happening is there is I mean I'm just talking about my experience they're having multiple workspaces like staging development UC prod right and if you have if you're using legacy marb system
[16:40] there you have to create a connection separately for every workspace like UAT development and then because if you're migrating from some source right and and then the target would be data wicks right but itself you are connect you are
[16:52] creating some kind of connectors again and again right for staging different development different production different and for MA based control they are charging you on a basis of a connector's usage and every connector
[17:05] means every uh uh active rows and update rows per handling there and that is how they are getting the revenue right so because it's a margin on the particular connectors and the how much you are using but in here in data bricks they're
[17:18] not uh charging you on the basis of your connectors they're charging on the basis of the number of DBUs of cost the database unit you're using consuming a day and right now databix is giving you
[17:30] 100 DBs per day as a free that you can use while migrating the stuffs from any source system to uh lake house so I think for 100 DBs I mean it's like u 100
[17:43] million records you can process a day as at free so how much work I mean h how uh data you have currently which can go beyond this particular 100 million records right because I have handled I
[17:55] have worked for more than 10 plus customers right there they have like realtime data as well and there that particular it's coming into the budget itself so that is going to be a free for me right now I mean maybe in the future database charges about that particular
[18:08] and they will remove that free condition but right now it's free for all of you guys so you can just go there and then just uh do some PC and then you can also have the system for your customer as well the key the key takeaways uh what
[18:22] you take back to this to your team from this session because LakeFlow connect isn't just another injection tool. It's a paradigm shift. We are moving from high overhead, high uh text legacy pipeline to a zero admin serverless
[18:36] utility model that integrates directly uh with security and governance. Right? We have toss and speed. The power of the free tier, right? Reminds them that 100 DBs a day of free tier compute can
[18:48] process roughly 100 millions record at zero cost. I mean zero dollars, right? For many projects, they can pilot or even run light production pipelines without spending a time. Then we have setup and CDC right speed to value. So
[19:02] highlight the fivestep vizard moving CDC built on serverless. We have like it can takes you around something 30 seconds data freshness without complex constant text and we have the governance. I
[19:14] already told you about the auditing uh pilot the four step I already told about the unity catalog that is going to be uh taken for you. Apart from all of this, if you have any uh like uh question, doubt and you want to understand about
[19:27] your customers use cases, I have a booth basically at 218 Moscon P South Cable Tech. You can come over there and like I'll be meeting you guys and then we can discuss over about your use as well. So
[19:39] you can just do some PC and then we can have a connect could connect there as well. So yeah and also I have some reference which I took to create this pipeline. Basically u we have a database
[19:51] lakeflow connect pipeline uh pricing model we have a fivestand pricing as well I've just considered and I also consider one lakeflow connect free tier blog post right which is officially right which can tell you about $35 per
[20:04] day is free which is uh 0.3500 right dollars. Yeah. Thank you guys.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.