Chaos to Clarity: Secure, Scalable, and Governed SaaS Ingestion through Lakeflow Connect and more
Summary
- Ingestion platformization: Databricks evolved from scrappy scripts to a unified SaaS ingestion platform that standardizes how all third-party data flows into the lakehouse with compliance and governance built-in.
- Governance by design: The modern platform handles compliance challenges (PII, secrets, SLAs), access control through Unity Catalog, and ensures every ingestion meets organizational standards without manual oversight.
- Self-service with safety: Teams can independently ingest third-party data without data engineers manually managing each integration, while policy controls ensure compliance and prevent accidental exposure of sensitive information.
Chaos to Clarity: Secure, Scalable, and Governed SaaS Ingestion through Lakeflow Connect and more

Talk by Krishna Bhupatiraju, Sr. Engineering Manager, Databricks ; Prashant Gupta, Software Engineer, Databricks
Ingesting data from SaaS systems sounds straightforward—until you hit API limits, miss SLAs, or accidentally ingest PII. Sound familiar? In this talk, we’ll share how Databricks evolved from scrappy ingestion scripts to a unified, secure, and scalable ingestion platform. Along the way, we’ll highlight the hard lessons, the surprising pitfalls, and the tools that helped us level up. Whether you’re just starting to wrangle third-party data or looking to scale while handling governance and compliance, this video will help you think beyond pipelines and toward platform thinking.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
Chapters
0:00Introduction and challenges4:44The stone age of ingestion8:43The bronze age and compliance12:15The iron age and access control17:20The modern age platform19:22Architecture and configuration27:33Future outlook and conclusions
FAQs
What SaaS ingestion challenges did Databricks solve?
The platform addresses API rate limits, missed SLAs, accidental PII ingestion, data fragmentation, policy confusion across teams, and the need for standardized, governed ingestion processes.
How does Lakeflow Connect help with SaaS data ingestion?
Lakeflow Connect provides a unified platform for ingesting third-party SaaS data with built-in compliance checks, PII detection, SLA management, and access controls through Unity Catalog.
What are the ages of ingestion evolution?
Stone Age (ad-hoc scripts), Bronze Age (compliance awareness), Iron Age (access control), and Modern Age (platform thinking) where ingestion is secure, scalable, and governed by default.
How does the modern platform prevent data governance violations?
Fine-grained policies prevent accidental PII ingestion, enforce SLA monitoring, route failures to Dead Letter Queues, and use Unity Catalog to control who can access ingested data.
Full transcript
0:00 Introduction and challenges
Welcome to the data and AS network folks. It's nice to see you all. This is our forward-looking statement. We really appreciate your feedback. So, at the end of the talk, please take a few minutes to give us feedback. We really appreciate it. And with that, let's get into third party ingestion. So we'll talk about ingesting data from a third party system into the lakehouse. But the lot of the learnings from this talk will be applicable generally towards all injection.
We'll talk about some of the tools that we used at data bricks to do injection right. But we'll also talk about platformization. How to do it such that every injection at your company happens the right way. Okay. So a quick show of hands. How many of you have worked on any form of injection? Like almost all of us, right? It's a very common thing. Now let's kick it up a notch. How many of you had customers complain that the data is not available on time?
Yeah, quite a few of us. And how many of you were the last to find out that the data was delayed? Right. These are times when you're supposed to go out for dinner with your significant other, you get pinged. It's not fun. This is the worst of the worst. The last one. How many of you accidentally ingested PII or worse even secrets into the lakehouse? And bonus points if it triggered a security incident. The myth is data injection is all about moving data from some source system into a destination like a data lake.
But that is farther from the truth. Right? So let's take a scenario. Say a VP of sales at your company comes to you and says I have this treasure trove of data in Salesforce. Please bring it into the lakehouse. I want to analyze it. You would say ah this is very simple. I can write a script maybe in Python. I can bring the data in. takes me two days tops, right? Really easy. So, you bring the data in and then what happens?
The salesperson likes you now. They come to you after two weeks. They want another table and you tell them, "Oh yeah, it's a script. It's a list somewhere. I'll add it to the YAML file. It'll flow in." Great. It flows in. On a Friday evening, they ping you. Data is not available. We have a weekly business review on Monday and because of you, we can't do the weekly business review. The data is not here yet. You didn't sign up for this, right?
All you thought was I'm I'm doing this guy a favor. Turns out the whole company's weekly business review now runs on your data. Happens all the time. It starts very simple. The v 0ero is always easy to build. It's much much harder to build something that actually scales. And this has been our journey at data bricks. We made the same mistake. Got the data in, wrote some scripts, things broke after a while. So, we'll quickly introduce ourselves.
I'm a senior manager at data bricks on the data platform team. I've been at some big tech firms like Airbnb and Amazon. Uh, I've spent around seven or eight years there. And most of my colleagues tell me that I have a seven-year age because I tend to leave after seven years. And with that, I'll call on Prashant to quickly introduce himself. Thank you. Hey hey hey folks myself Prashant I am a senior developer at datab bricks and what I mostly focus on is bringing third party data into our lakehouse and my focus has been try to do it in a way that it it acts as a platform or I at least try to do this in my past life I've worked at Google in data platform there as well where I was trying to unify a lot of massive data sets and try to drive AIdriven insights on top of it so yeah thank you thanks so both of us are on data platform team at data bricks and our team is responsible for building the data intelligent platform.
So we are a very cross functional org. We work with a bunch of business teams marketing sales HR engineering product and our responsibility is to bring in data ingested into the lakehouse. We build a bunch of tools like metric store to actually process the data and we built a lot of processing layers after that all in the service of actually driving insights into our business. Now let's get into it. So let's start at the very beginning.
4:44 The stone age of ingestion
We call this the stone age. We didn't have a lot of tools during this time. During this time when injection was done, we would just actually write a script, get the data into the lakehouse. We had a few issues during this time or rather a few features of this time. The first is that of rudimentary tools. We would write scripts in notebooks. we would check it into git and that would actually and schedule it and then that would get the data into the lakehouse.
The thing that happened was there was a complete lack of standards. Let's take an example. So how would we manage passwords or tokens? Let's say if want to connect to a SAS tool, you need a token. Where do you save it? Sometimes we would put it in notebooks. Sometimes even check it into git, right? Very very few of us actually use data brick secrets, right? This is a very common thing when you start off you tend to do these kind of things. you're in a hurry, you tend to do these things, right?
And multiply this across teams at data bricks like every team was doing this and we were all in silos. We were this was before unity catalog. So we did we did not have a discovery layer. So what that meant is multiple teams were bringing in data multiple times and the learnings that like let's say one team had about injection, it was never transferred over to other teams. So there were a lot of bad practices and even like it was very hard to like arrive at good practices and arrive at consensus across multiple groups.
So the first issue that we had during this time was that of quality and I'll drive home the point through an example. So Salesforce data was one of the popular data sets and with this we had multiple teams actually asking for tokens for to Salesforce. Multiple teams were bringing the data in and what happens when you do that you start hitting API limits right? we were constantly hitting API elements. The second thing with Salesforce, if some of you have worked on it, has this thing called SOQL fields or formula fields.
These fields actually do not refresh when you do an incremental refresh. And all of us are told only do incremental refreshes. But with Salesforce, if you do an incremental refresh, the data is not refreshed for those formula fields. So you would get things wrong all the time. The third one is that of secret management. As I said, some of them in notebook, some of them in Git. Hardly anybody used data brick secrets. So quality was a real problem.
The second issue was that of observability. Some teams had pag duty set up. Some teams had SLA set up. Most of the teams did not even have anything. It was all Friday night pings, right? My data is not available. The third was that of discoverability. This was pre-unityity catalog which basically meant we did not know what was in the lakehouse. So a lot of times people would get same copies of the data multiple times right.
So we had three issues and how did we solve it? The first one we moved to unity catalog. If you haven't used unity catalog it basically is a tool that data bricks provides. It gives you centralized access control. It gives you auditing. It gives you a lot of lineage across your data assets and it provides data discovery. So you can actually search for things in your catalog. With Unity catalog, the issue that we had with discoverability was like taken care of, right?
The second tool that we used was LakeFlow Connect. Lakeflow Connect is a UI based tool through which you can set up connectors. It already has connectors to a lot of the popular SAS tools like Salesforce for example or Netswuite and it also has connectors to things like S3 and stuff like that. So with this you did not have to write scripts anymore. So the issue that we had with quality on the first slide when I explained that was not there anymore.
So once we adopted these two tools, what happened was our engineers didn't have to be neo from the matrix dodging bullets, right? Things were much simpler. They would go to the UI, set things up. They could discover things through Unity catalog and things worked pretty well. So on the right hand side, you'll see a screenshot of how Lake Connect looks like. This brought us into the bronze age. Things were slightly better.
We now had some tools. We had some trading going on. The tools in this case are Unity catalog, right? And lakeflow connect. So folks could go to these tools, get the data in. Things were much better. So we had built-in connectors, we had efficient injection, things were looking good. So we asked ourselves, are we done? Right? Injection seems to be done, right? We're moving data in. It's it's super efficient. Unfortunately, the answer is no.
8:43 The bronze age and compliance
The thing that happened was our legal team was getting constantly bombarded by requests to review the data because for us legal team has to approve before the data can show up in the lakehouse. So they were getting questions about oh does my data have PII does it have phone numbers does it have emails does it have secrets? We made injection so easy that the legal team was getting overwhelmed. We had to find a solution.
So we had two issues during this two new issues that came up. The first one was that of PII leaks. Let's take an example. Let's say your friendly VP comes back to you now. Right? He says, "Oh, now you have this legflow connect. Things are much easier. Get me case commands." Right? In Salesforce, case commands is the data set that has all of your cases and the comments from those cases. That could have PII. It could have emails.
It could have passwords. it could have a bunch of stuff. So this was an issue. How do we detect that there's PI in our data? The second one was that of sensitive data leakage. So let's take another example. So this often happens when you try to get data sets like Jira. Let's say you use your internal Jiraa system to actually set up tokens for like different systems. Usually those are added to commands. Bring the data in the lakehouse.
You kind of lead secrets now, right? So you've solved the injection problem. Now you got the compliance problem to solve. We did two things to solve this. The first one is we use data classification. Data classification is a is an offering by data bricks that you could set on top of your catalog and it automatically scans the data in in the in the tables in that within that catalog and it tags them with one of the 12 categories that you see on the right.
I have a subset of them but there are a few more. The reason this is very powerful is because it doesn't use regex like a lot of other tools. It goes to Unity catalog, reads through the documentation, figures out the context, and then uses an LLM in the background to categorize data. It's actually pretty accurate. The next issue, if you remember, was that of secrets. For this, we actually worked with our trust and safety engineering team to develop a tool that actually can detect secrets.
And so, you'll see an example here. This is actually a case command where the first one that's not a real token by the way. The first one is a token. The second one is when it gets redacted. Right? So those were the two tools that we used uh the data classification and the secret redaction to handle the compliance problem. Here's a screenshot on your left of the data that came in without any tags. The one on the right you'll see the tags across some of the columns.
Right? It auto does that. Now you can make a decision as a company what you want to do with it. Right? Right? You'll have to you'll have a way to deal with it. Will you redact it? Will you do something else? Will you completely drop the field? Right? You can take actions based on that. And with this, we went to the next stage. We call this the Iron Age, right? We had defenses around our data sets. We knew what was coming in.
12:15 The iron age and access control
Our legal team was slightly happier, right? That we were compliant. And just and to walk us through the Iron Age and what came after and how we actually platformized the whole thing, I'll call on Prashant. All right, thanks a lot. So folks, you have not now gone through all the ages till now. I'll walk you through what happens after this. So this was the age where we started to get a lot more serious. We were no longer just ingesting data.
We were securing it, organizing it and effectively laying down the groundwork to scale safely. After this when we reached Iron Age one of the questions that we asked ourself was hey is this where we want to be? Is this the final place? And there were three things that we considered at this point of time. First we had a single source of truth. No longer five different teams ingesting the same data five different ways.
You now had one owner. You now had one pipeline and effectively one trusted version. Second, your injection was by default observable and reliable. With lakehouse monitoring and lake flowing picture, you had alerts, you had lineage, you had the whole package. And after that, compliance was baked in with data classification library and secreted action in place. We were no longer just hoping that secrets don't creep in.
You were actively catching and controlling it. So, this did feel like a good package, right? And what that means for you is, hey, is this talk over? Sadly, no. We still had some struggles because the truth is that just having clean and classified data isn't really enough. What you also want to make sure is that not everyone should have access to all of it. So this is where we wanted to solve the visibility problem. And how we did it, we basically used something called as attribute-based access control.
For those of you who are not familiar with what ABACK is, I'll try to give a very very quick overview. Abback or attribute-based access control is a private preview product of data bricks that allows data governance administrator to define broad data policies that are is still very flexible and precise. So you no longer have to go to each table and define your policies and define your filters. You could define a policy once and then use govern tags to apply that policy over a schema or even over entire catalog and then Aback would make sure that these policies are applied correctly on the tables and columns of your choosing.
As soon as you turn Aback on, all operations on your data assets are logged immediately and are available in real time into datab bricks audit log. So how did Aback work? There are two things that we needed for Aback to work. The first one is user attributes. What exactly are user attributes? Think of user attributes as characteristics of your user. Not just who the user is but also what a user in in a very specific context.
For example, what their role is, what the region they are working in, what their clearance level is. Once you have user attribute, the second thing that we need to worry about is role level filters. This is effectively saying that you don't want to show all of your table to everyone but only a subset of rows. And once you have user attributes combined with role level filters, what you get is Aback. Aback combines user characteristics like their role, their region, their level with the role level filters that you want to define using SQL based functions and then can make sure that folks only see the data that they are supposed to not the global data set.
Now I would like to remind that for those of you who are keen on using Aback, Aback works best when it is tied with Unity catalog because Aback works on top of govern tags which you get out of Unity catalog. So for Aback to work seamlessly, you need to be on Unity catalog. So once we had this in place, this is where we were able to get to. On the left hand side you have the opportunity table where you see the data for all the regions.
This was the world before Aback. Once you applied the Aback policy what the one you see on the right hand side on the top there what you're doing is you're defining a function that restricts your data to only Europe region. As soon as you do that, what happens after that is your employees when they try to query this opportunity data, they would only see the data for the region that they are in. And when we did that, we reached our next stage.
We call it the modern age of data injection. This is where we are privacy first. We have lightning fast data injection. And we are also ready for any scale without breaking a sweat. So what made modern age truly modern? There were three things. Do you remember the initial struggle in our stone age? We had the issue of quality. We had issue of observability and discoverability. All of that was solved through lakeflow connect.
17:20 The modern age platform
The backbone of our entire injection. It's no longer just about pulling data. It's about doing it in a way that is secure, scalable, and you no longer have to write endless code to do it. Next in the bronze age we had issue of compliance. We solved that through data classification and secret reduction library. This dual approach made sure that secrets don't creep into your lakehouse and the compliance was always front and center.
After that in iron age we had these two issues resolved but we had the issue of data visibility. Everyone could see all the data that we had. So to solve this, what we needed was attribute- based access control on top of Unity catalog. This made sure that our teams only see the data that they're supposed to do their job efficiently. No more, no less. And now we are here. You've gone through the our entire journey where we were and where we are now.
And now I would like to kick it up a notch. Let's talk about creating platforms. I'm about to show you how our datab bricks data platform team used all of these tools to create a unified platform that was secure, scalable, and ready for anything. And trust me, this is where the magic happens. This is where all of your complexities turn into simplicity and your toughest challenges into your biggest wins. Introducing thirdparty injection platform, a no code platform that we developed right here at datab bricks for an internal use case to bring all of the data bricks tool that you already love under one umbrella.
19:22 Architecture and configuration
So effectively third party injection platform is an abstraction on top of all the three things that you saw in modern age. It effectively bundles everything together to make sure that you have a platform that all of your teams can reuse. Now you're here. I'll walk you through the entire end to- end user journey of how third party injection works. It starts with injection config. Think of this as a blueprint of your injection.
You define three things. First, your upstream. Where do you want to get your data from? Salesforce. Second, your downstream. Where do you want to get your data to? Your data lake. And finally, your consumers who should be seeing all of this data. As soon as you write a blueprint, we move to the next stage. Third party injection would kick off an automated legal review. We would scan your data and classify it and send it for compliance review so that you know that you're compliant before you move forward.
Once you are done with compliance, we would automatically validate your config, make sure everything is in the right place. Do calary testing to catch any issues early on and automatically deploy your injection at a frequency of your own choosing. Once your changes are deployed, automated orchestration takes over. This is where we run our upstream sync using LakeFlow connect. We run data quality checks, scan your data for PI and secret and finally run ACL and Aback on top of it to make sure that folks see the right data.
All of this combined together is what creates third party injection. A tool that is widely used across all the businesses in data bricks including marketing, go to market sales, IT data, even engineering. And this allows them to collaborate seamlessly. And since you've seen what the end to end user journey looks like, I'll give you a quick peak under the hood on how the architecture works. I would explain this in three very simple steps.
As soon as the 3PI job begins, it would trigger the lakeflow connect pipeline so that your SAS data can get into your data lake and it would wait for that pipeline to complete. Once the injection completes, the data lands into an intermediate catalog. And this part is really, really important. Let me walk you through why. Consider this as the staging area where we do all the heavy lifting. We would run data quality checks to make sure your data is exactly what you want to be.
We would filter data by doing row filterings. We would do column transformations over here. More importantly, one of the things that we do at this stage is we tag all of our data through a taxonomy that is standard across data bricks. And why is that important? Because attribute-based access control is only as good as the data that it runs on, the characteristics of data that it runs on. So as long as you can make sure that you have a narrow funnel through which you can tag all of your data, you know that if someone defines a policy that would work correctly.
Once you have this clean classified data in intermediate catalog, we now promote it to main catalog so that folks can read this data. While doing that we would run PIA checks on it and do seeker reduction so so that you have the right data and you know that it doesn't have any PII that it should not have after that you would simply apply aback rules and then have write ACL checks on top of it. So now you know this you remember when I said that hey this is supposed to be a no code platform you should not have to write a single line of code to ingest data from any source system.
I'll show you how that works. I'll take you through the guts of a typical 3PI config. I'll divide this into three parts. First up connector config. This is where we specify how your data gets sent in. For example, on line number one, you see that we're ingesting data from Salesforce. After that, we specify where exactly your data should go into. For example, here we want to land data into main. sfdc bronze cutler. You can define your own schedule.
You could choose to run it at every hour, every day, or anything as long as that is a chron expression. Remember, the goal of a platform is to make sure that it is easier for folks to do the right thing. If you see line number six, folks can very easily specify how they want to ingest data. In this case, as soon as they write their sync type to be an incremental sync, we would make sure that we are not being greedy and ingesting all the data every time.
We only get the new shining bits. And it's not really about folks can't do it. It's about we give them the tools so that they can do it easily. After that on line number nine, you see the lakeflow correct pipeline ID. This is how third party injection knows which lakeflow connect pipeline to trigger and wait for it to complete. The next two ones are interesting. This is for the folks who raised their hands when was asking, hey, were you the last one to know about it?
This is where we ask folks to specify owners and alert recipients because if something goes wrong, we need to find the folks who are responsible for this. So now this is the highle config. The second is how we define attribute-based access control. This is a very simple attribute based access control policy that we define on Salesforce data. We give it a nice name first of all and then on line number five you see a tag that we give to this policy.
In this case it's called SFDC back. Do remember this tag. I'll come back to this in next slide. On next three lines we have the policy SQL. This one is a pretty straightforward one. What it is doing is it is a simple geoence that make sure that only your employees in Europe region can see Europe data. So let's say you are an employee working from a beach in Hawaii and you try to query this data. It would be like no not today.
So now you know how AB works. This is your final configuration. This configuration specifies how we bring each data set in. In this case, this is the opportunity table from Salesforce. You could do a lot of different things over here. You could define the annotations that you want to apply on each table. Do you see that line number four? It's the same tag as you saw on the last slide. As soon as you have this tag over here, this table is automatically enrolled in Aback and only your Europe employees would be able to see the data in Europe region.
Then we could specify things like row filters because you don't really want folks to see the deleted data, right? Also, you can specify a lot of different data checks. These could be as simple as, hey, my table should not really be empty or these could go a little bit sophisticated. You could say things like this column should be unique or you could go one step up or you could you could basically write your own SQL checks.
After that we can define our own columns that you want to ingest. You could you could specify what the data type is whether or not it is a partition and you can also define your own definition the one that you see in the last line if you want to have a custom reduction rule. So this was about how the third party injection works by simply writing configs. As soon as we did that we started going towards the next stage.
We call this the space age of data injection. We are inching closer to this but we are not there yet because the truth is the future of third party injection might not even be just third party systems. We are talking about ingesting data from data shares S3 buckets OLTP databases and so on. We would want to have better access control through arbback, resource-based access control and peback, purpose-based access control.
27:33 Future outlook and conclusions
If you want to learn more about what peback is, there is another talk from data platform team that dives just into this topic. Next up, we want to make change management as easy as possible. Think about AI agents who would do this for you. Whether it is adding new config, adding a new column, a new table or even a new injection altogether. And finally, real time injection is also on the horizon. We want to do lightning fast data injection by using declarative pipelines which is a product offered by data bricks.
So now you've gone through it all. I'll give you a very quick recap on what all ages we have been through. Stone age where we didn't really had a lot of tools. We were stitching things together, writing brittle scripts and trying to make sure that we have the good data. We added lakeflow connect and unity catalog on top of it to make sure that when we are ingesting data we were doing it the right way. That led us to the bronze age where we had good data, we had easier ingestion but that created the PI problem.
You could now easily get in PI and secrets into your lakehouse. We use data classification library and secret reduction library built by our trust and safety team to make sure that your secrets never land in your lakehouse. That brought us to the iron age. This is where we had good data. We had clean data. We had classified data. But our visibility problem was not solved. Everyone could see everything. That's where we used attribute based access control to ingest data and still make sure that folks see only what they're supposed to.
And that's what led us to the modern age of data injection. When we were at modern age, we thought we have all of these tools. Can we actually make it easier for folks to use these? And that's where we created the platform called third party injection. As soon as we did that, we reached the space age which is uh age beyond SAS injection. So with this I would like to wrap up my talk and but before I do that I would like to leave you with some takeaways.
Think beyond just moving data from point A to point B or from your SAS system to your data lake. When it comes to third party injection, injection just isn't just about bringing data in. It's about doing it in a way that is secure, scalable, but more important than that, it is reusable across your entire organization. Ask yourself this question, how can you build tools that would make the right way the default way or they would make it very very difficult to do the wrong thing.
How can you leverage the primitives that databicks already gives you? Unity catalog, LakeFlow connect, data classification, attribute based access control and use all of that to create an abstraction so that your organization can move fast and still trust your data that I would like to thank you all for attending this talk. If you liked our talk, our data platform team is giving additional talks. I would highly recommend checking those out.
And with that we have some time left. If you want to discuss more on third party injection we would just be right outside. Please come and catch us. Thank you.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.