Sponsored: LakeFusion | When MDM Meets Ontology: A New Architecture for Enterprise Data Intelligence

Summary

LakeFusion is an MDM, PIM, and graph analytics platform built natively inside the Databricks Data and AI platform using Databricks Apps, enabling organizations to deploy entity resolution, streaming data quality, and AI-powered stewardship without any data leaving the lakehouse.
The talk introduces the concept of an operational ontology — a data architecture that not only describes what data should look like but actively enforces those definitions at scale in real time — as a more effective alternative to traditional MDM systems and top-down knowledge graphs.
Production deployments have consolidated over 100 million records across dozens of source systems in weeks rather than months, demonstrating that lakehouse-native entity resolution can achieve at speed what external SaaS MDM tools have struggled to deliver.

Chapters

FAQs

What is an operational ontology and how does it differ from traditional MDM?

An operational ontology is a data architecture that both describes what data should look like and enforces those definitions at scale in real time within the lakehouse. Traditional MDM systems typically define desired data structures without continuously enforcing them in production, and top-down knowledge graphs require extensive upfront modeling that slows deployment and creates brittle dependencies.

Why is third-party SaaS data processing becoming technical debt for enterprises?

Processing data outside the lakehouse boundary means it operates outside the governance, security, and compute environment that manages everything else. LakeFusion's chief architect argues that enterprises whose AI agents must call external SaaS systems introduce operational risk, latency, and governance gaps that will increasingly block agentic transformation over the next several years.

How does LakeFusion integrate with the Databricks Data and AI platform?

LakeFusion deploys natively inside a customer's Databricks environment using Databricks Apps, which means customer data never leaves the governed lakehouse. It is also available on the Databricks Marketplace, allowing organizations to install it directly alongside their existing Unity Catalog tables and analytical workflows.

What scale has LakeFusion achieved in production deployments?

According to this video, LakeFusion production deployments have consolidated over 100 million records across dozens of source systems in weeks — versus the months typically required by traditional MDM implementations. This speed advantage comes from running entity resolution natively inside the lakehouse rather than copying data to and from an external system.

Full transcript

[00:07] Hi, I am Roz King. I'm the chief architect at Lake Fusion. Lake Fusion is a master data management company, a product information ma management company and a graph analytics company. CEO sitting here in the front row and
[00:20] he's going to be pissed off to hear that I'm not here to sell you lake fusion. Who has seen ah he can't hear anything. That's good. I can talk poorly about him. Who's seen our coffee cups? With our with our logos on our coffee cups.
[00:34] Yeah, we're everywhere. We're excited. Um, so with our MDM, our PIM, and our lake graph solutions, we're competing with some enterprise data management applications like Informatica and Relio.
[00:46] Uh, we're competing with PIM tools like SIBO. and we are competing with graph analytics uh tools as diverse as Neo forj and puppy graph and timber and some other some really cool stuff that's getting built. What differentiates us is
[01:00] that we are built on top of data bricks. We're built inside of data bricks. You can download us and deploy us inside of your data bricks environment uh using data bricks apps which is pretty cool right your data is never going to leave
[01:12] your environment. But as I said I'm not here to sell lake fusion. I'm here to talk about something a little bit differently. If I can get this to change.
[01:32] There we go. All right. I want to talk about the fact that thirdparty data processors, third party software as a service sitting outside your lakehouse is becoming essentially technical debt. It is operational risk. um the enterprises
[01:46] that figure out how to remove their dependence on software as a service and start bringing all of those workloads and all of those applications inside of the lakehouse over the next few years are going to be the ones that pull off a
[01:58] truly agentic transformation. I want you to rethink what your enterprise architecture needs to look like over the next five years. Question the assumption that data should ever be processed
[02:10] outside of the boundary that governs it. Okay, I want to talk a little bit about why I'm saying that because that's a pretty bold statement. The reason I'm saying that is one of histo history. So
[02:23] let's talk about how we got to the current kind of modern state of enterprise architecture. Back in the 80s, we started with this idea of data warehousing. Ralph Kimble, Bill Inman,
[02:35] and that kind of generation of thinkers said, "Hey, in order to analyze our business, we need to move the data outside of the like outside of our database." Now, we had to do that because of compute constraints, right?
[02:47] Like we couldn't start running analytical queries on top of our transactional data. Our DBAs would tear us to shreds. I'm sure a lot of you in the audience have like PTSD from DBAs sending you angry emails, and it is a
[03:00] nightmare. So we moved all that compute outside of the internal database. After that happened, ETL became really common, right? We all remember when ETL was first like a big buzzword and then it became common. It became accepted. We
[03:13] started building out cloud. We started building out big data computing across multiple machines and we ended up with software as a service is kind of a default, right? Like every everything that we're doing has software as a
[03:25] service showing up across the enterprise. is I think I heard some metric earlier this week. An average of like 300 software as a service tools for the big modern enterprise. That's insane. That's a lot of logic. That's a
[03:38] lot of core data that is locked away in other thirdparty data systems. We got to this point because of an initial compute constraint back in the 80s. If that consu compute constraint hadn't existed,
[03:50] we would not have built their current state of enterprise architecture. My point, the whole point of the conversation is to point out that data bricks as it stands right now has closed that gap that initially initially sent
[04:02] us on this trajectory. Because that gap is now closed, we can start rethinking what enterprise architecture should look like. Okay, there are a lot of arguments
[04:15] against this bold statement that I'm pointing out to you. A lot of third-party data cataloges are going to call that are kind of rebranding as context layers are going to call out that we can actually lower the walls of these silos by building out context
[04:29] layers. Metadata management on top of all of your data silos. Now, there is some really cool capabilities that are showing up here and there's a lot of folks out in this room that are building those tools and I agree those tools are really cool, but it is really difficult
[04:43] to enforce actually enforce governance across those silos. And there's some other technical reasons why those silos are causing problems. On top of that, just being able to describe all of your
[04:55] data cataloges, just being able to tag those data cataloges does not necessarily mean you can unify the data across those cataloges, right? Like I can say, hey, this person over here is the same as this person over here. I
[05:07] still need that data sitting somewhere on a single compute cluster in order to draw analytics about that single person. Now thirdparty MDM tools like Informatica will tell you that actually I can just take the ID and as long as I
[05:20] syndicate our canonical ID across all of our enterprise architecture that canonical ID allows me to unify my analytics and plus if I have a canonical ID and if I have a single unified
[05:33] architecture or single unified governance platform across all of that then I essentially have busted the walls of my silo. I would argue that that is not correct and there are a lot of technical reasons why that is not
[05:45] collect correct. Let's look at some of the reasons. So when you have data silos, even if you've lowered the barriers of those data silos, even if you made it easier to move data across them, you still run into issues related
[05:58] to latency. Every time you try to move data from one data system to another, you're dealing with latency. You're dealing with delayed source of truth. You're dealing with the fact that, hey, my folks in my CRM, my folks on the sales side have a different idea of the
[06:12] source of truth than the folks downstream who are consuming it, which at best causes operational risk and at worst actually means the data is truly different and they're going to make incorrect decisions. Now, it causes slower iteration cycles and it causes
[06:25] broken lineage. Slower iteration cycles is every time you need to get another team to build something, you have to go ask and get into their backlog and their backlog is going to take forever. we don't know when they're going to meet your need. So just because they have the
[06:37] ID does not ne necessarily mean their API surface area is going to make it able for you to build whatever you need to build across their barrier. Broken lineage is another issue, right? Data bricks unique catalog is incredible because you can track lineage across
[06:50] your entire data system. If I have a data product I'm consuming in data bricks and my SLA breaks, I can see that that SLA broke. I can know why it broke and we can go and talk to the people who need to fix it within minutes. When you
[07:02] start bridging those across data silos, you don't necessarily know why or how your SLA was violated. You need to go talk to a team. We don't know when that's going to get fixed. Risk. We also have a lot of risk. You're held hostage
[07:14] by their API. You have forked governance policies. Trying to enforce governance across logical barriers is incredibly difficult. And describing is not enforcing. A lot of the great data catalogs are getting better at enforcing governance across boundaries, but they
[07:28] are not perfect. Now this is a lot of technical reasons why it's not great to have data silos but at the end of the day technical reasons can be solved with technical technical problems can be solved with technical solutions but we
[07:42] have a much larger problem when we think about why do we need to bust these data silos there's this guy Melvin Conway who came up with this idea of Conway's law the
[07:55] idea is essentially your data architecture is an isomorphic map of your underlying communication structures and your organizational structure. That means all of the software as a service
[08:07] industries didn't start creating data silos. They simply sold into data silos that already existed. If we need to bust data silos, we also have to enact organizational change. We cannot simply
[08:19] solve all the technical problems I pointed out in the previous slide through technical solutions. We need organizational change to bust those silos, but we also need technical change, right? It's kind of a chicken in
[08:31] the egg problem. So, if we cannot bust our silos without organizational change, and we cannot achieve organizational change without busting silos, how do we solve the problem?
[08:46] All right, I told you earlier that I wasn't going to try to sell Lake Fusion, and that was a bold-faced lie. MDM is a fantastic tip of the spear for busting data silos. So the reason it's
[08:59] fantastic is a couple different reasons. One, MDM exists as a concept because we understand from an enterprise level, we have the same version of customer, the same entities across many different
[09:11] source data systems. We need to be able to resolve those differences and figure out who is that single source of truth. So MDM is where most enterprises feel the software as a service sprawl pain
[09:24] first and foremost. The other reason MDM is a fantastic tip of the spear for enacting the organizational change you need to bust data silos is that when you do MDM initiatives you need to bring the
[09:36] IT and the business together. You're bringing data engineers. You're bringing product engineers. You are bringing your governance folks. You are bringing all the business folks that are working daytoday on the actual data and are dealing with the sprawl of MDM or sprawl
[09:50] of SAS, you know, into the same room to talk about, hey, how do we build MDM initiatives? How do we centralize our source of truth across all of the sprawl? That is why MDM becomes an incredible tool for busting silos.
[10:04] I want to talk a little bit in practice about how this works because we've done this, right? We're an MDM company. We're doing this every day. Every customer we work on is working with us because they want to bust these silos and centralize their operations, centralize the running
[10:17] of the business, not just the analyzing of the business on data bricks. So, one of our current customers is a global apparel and footwear footwear brand. They have a bunch of fragmented customer data across four different sources of
[10:31] truth and they were really struggling to do market segmentation. They were struggling to understand, hey, how do I sell into this specific customer when I don't actually know that we have a single source of truth for that customer? They'd built a homegrown
[10:44] entity resolution system in data bricks and it was not scaling. They still had duplicates leaking in. They had no ability to build an iterative platform that allow them to improve data quality over time. This is what MDM is really
[10:56] good at. So we worked with them. We deployed our application. We mastered 23 million custom records and now they have a single source of truth for customer, which is great, but it's also not really that cool because we can do a hell of a
[11:09] lot better. We haven't busted any silos. We just essentially created a new silo. Now it's sitting in data bricks. What we really want to do is build all move all of the operations, move their entire business into data bricks. So the next
[11:21] step Oh, that's right. I have this slide. That's a screenshot of the tool. Looks pretty sweet, doesn't it? All right, the next step is we want to start mastering products inside of our
[11:34] Lake Fusion PIM solution. So, because they have a lot of customers, they also have a lot of customers they want to sell to and they want to be able to manage all that sales data. A ton of SKUs running in a bunch of different t
[11:47] product taxonomies. We want to center that into a single source of truth as well. So, we use our PIM tool to master all of that data. That's another really cool story. Right now, we've got their customer data and their PIM data
[12:00] mastered in data bricks with Lake Fusion. Again, great story, but it can get a hell of a lot cooler. So, the next step is we start thinking about
[12:12] what is the relationship between a product and a customer. That is a transaction. That is a transaction across e-commerce. that is a transaction in physical stores. If we stream all of those transactions into Lake Fusion,
[12:25] those are now the bridge between customer and product. That's really cool. Most traditional MDMs will never do this because they operate at the at the individual entity level. They're not set up to handle trillions of
[12:37] transactions streaming into your lakehouse. We know that data bricks can handle that with its eyes closed. So, our relationship management feature allows you to build and manage and analyze that transaction. I can open up
[12:50] a single customer in our Lake Fusion instance and identify every single product they've ever purchased across every single uh physical store, every uh virtual store
[13:02] that they bought it at. And same with product. I can open up product, see every customer who purchased this. That is also a really cool story, but it can get a hell of a lot cooler because we can start mastering a lot of different
[13:14] entities. We can start mastering suppliers. We can start mastering the physical materials that make up the bill of materials that builds out the actual products that they are selling. All of this starts to look like a very common
[13:28] word we've heard today, which is ontology. People are really excited about ontologies and building ontologies on top of data bricks. When we start building those ontologies, we can now analyze them in our lake graph tool.
[13:41] Lake graph is really sweet. It automatically pulls in the ontology that is constructed and managed and mastered in lake fusion and allows you to add some very advanced graph analytics on
[13:53] top of it. We know that page rank is available day one on uh lakebased on on genieontology. We have 15 different algorithms sitting in lakebase that are available today to start analyzing your
[14:05] complex graphical data. Now we work really great with Genieontology because all of this data is sitting in data bricks. This is a table sitting in data bricks. You can analyze with Genie. We are ready to go with Genie day one. But
[14:17] our lakebased our data bricks native graph database is a truly fantastic tool for starting to ask some really cool questions and we can ask questions of this data that be are
[14:31] impossible in a traditional MDM. I can start saying hey if I've got a customer and I have a supply chain issue what are my customer segments are going to be affected by this? I can say if I've got
[14:43] a customer segment over here and my this particular part in my bill of materials goes offline, can I dynamically find another supplier to fill that so that I don't lose this amount of revenue that is tied to that amount of customers?
[14:56] That question might take a month of a team of data scientist time to answer in a traditional database. But with graph analytics, that is simply a chat away and I can answer that question in real time, which is really freaking cool. as
[15:10] long as I've done the work to build the trusted ontology in lake fusion. All right, so last piece here. SAS created data silos can and should be
[15:23] torn down. Everyone is saying that SAS is dead. And I agree, but most people are saying that SAS is dead is because AI has lowered the bar when it comes to actually building software as a service.
[15:35] I don't think that's right. I think AI is the demand signal, right? AI needs good context. AI needs access to all of your data in a single graph structure often, but definitely needs access to it
[15:48] without having to bridge barriers between your software as a service. The lakehouse is what kills SAS because it has removed the technical constraint that caused SAS to exist in the first place. We're just getting to the point now we can finally kill it off for good
[16:01] because of data bricks. So it's blame them if you're a SAS company or if you're not rebuild your SAS on top of data bricks because that's where the future is. That's where things really start changing. All right, hold up.
[16:13] There we go. This is us. This is Lake Fusion. We are live on the data bricks marketplace, which means if you want to start mastering your data, if you want to master product data, if you want to work with graph analytics native to
[16:26] Lakehouse, native to data bricks, data never leaves your environment. You have total control over it. total integration with all the incredible AI capabilities that data bicks has made available. Lake Fusion is the way to go. That QR code
[16:38] will take us take you to the data bicks marketplace where you can install our app and start mastering all of your data, mastering your product data, and doing graph analytics tomorrow. Just so
[16:51] long as you get a nod from our sales team first. All right, that is all I have for you. Thank you all for coming.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.