Skip to main content

Transforming Data at Rheem: From Silos to Scalable Data Lakehouse With Databricks and Unity Catalog

Summary

  • Rapid data product development: This video shows how Rheem evolved from siloed reporting to 30+ certified data products in just 1.5 years, integrated with 20+ source systems using Databricks.
  • Enterprise data platform: Learn how Databricks became the cornerstone for rapid product development and efficient data sharing, enabling sales, procurement, service, and operations teams to make better-informed decisions.
  • Unity Catalog migration: Discover best practices for migrating from Hive Metastore to Unity Catalog, including governance enhancement, developer experience improvements, and tips for successful enablement across the organization.

Transforming Data at Rheem: From Silos to Scalable Data Lakehouse With Databricks and Unity Catalog

Watch: Transforming Data at Rheem: From Silos to Scalable Data Lakehouse With Databricks and Unity Catalog
Talk by Elise Haskins, Senior Manager, AI & Data, EY ; Joseph Palomba, Vice President, Enterprise Applications, Rheem
Rheem's journey from a fragmented data landscape to a robust, scalable data platform powered by Databricks showcases the power of data modernization. In just 1.5 years, Rheem evolved from siloed reporting to 30+ certified data products, integrated with 20+ source systems, including MDM. This transformation has unlocked significant business value across sales, procurement, service and operations, enhancing decision-making and operational efficiency. this video will delve into Rheem's implementation of Databricks, highlighting how it has become the cornerstone of rapid data product development and efficient data sharing across the organization. We will also explore the upcoming enhancements with Unity Catalog, including the full migration from HMS to UC. Attendees will gain insights into best practices for building a centralized data platform, enhancing developer experience, improving governance capabilities as well as tips and tricks for a successful UC migration and enablement.
Here’s more to explore:
Economist Impact: Unlocking enterprise AI: https://www.databricks.com/resources/webinar/unlocking-enterprise-ai
The State of Data + AI: https://www.databricks.com/resources/ebook/state-of-data-ai
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How did Rheem accomplish rapid data product development?

This video demonstrates how Databricks' unified platform and rapid development tools enabled Rheem to create 30+ certified data products in just 1.5 years. The platform's ability to integrate 20+ source systems, including Master Data Management (MDM), provided the foundation for fast iteration.

What business impact did the data platform deliver?

The video highlights measurable improvements across sales, procurement, service, and operations teams. By democratizing access to reliable data products, Rheem improved decision-making speed and quality, enabling the business to respond faster to market opportunities and operational challenges.

Why migrate from Hive Metastore to Unity Catalog?

Unity Catalog provides unified governance, finer-grained access control, and better discoverability compared to HMS. This video covers the migration strategy that enables Rheem to scale data products safely while maintaining strong governance and improving the developer experience.

What are the key lessons from Rheem's transformation?

The video emphasizes building a centralized data platform, enhancing developer experience through better tooling, improving governance capabilities, and planning carefully for UC migration. These practices enabled Rheem to maintain momentum during modernization while supporting 30+ concurrent data products.

Full transcript

All right, we're going to officially start uh this session. So, thank you guys for joining uh transforming data at RE. Uh I thought think we'd just quickly start with uh introductions. Palumba, vice president of enterprise applications at RE manufacturing company. And we have Elise Haskins. Haskins. I had to make sure that I used her new married name. So Ely Haskins, senior manager AI and data uh at EY. All right. So I am super excited to present data and analytics from a REM manufacturing perspective.
Partnered with uh Ernston Young and Data Bricks. we've done, I think, an amazing job in a short amount of time and really excited to tell the story today. So, just real quickly about the agenda, a little bit about REM, who we are, what we do in the industry. We're going to talk about REM's data journey. Um along the way um you'll hear me talk a little bit about value business value streams which really drove the the transformation and the journey.
Uh then we'll get into Unity Unity catalog what that upgrade process looked like uh and the value that it brought to RE. A little bit about RE. Show of hands. Who has heard REM? Does anyone know what we do in the industry? Wow, that's pretty good. It's pretty good. So, maybe some facts that you didn't know about REM. We are a privately held company, roughly around 7 billion. Um, just this year, we hit our 100redy year anniversary.
So if those of you didn't know that uh 100 years in the industry roughly around 14,000 employees 23 factories in various stages um we are an air and water company. So that you you might not know that but we sell water products and air products and one of the only companies in the industry that sell both air and water and we believe that product portfolio really differentiates us in the marketplace and gives us a competitive advantage especially um as air and water products are coming together.
Uh lastly we are mostly a residential um residential uh type market but also growing in the commercial space. You can see uh we have uh numerous uh uh number one market leading positions uh across the globe our journey. So I want to I want to first talk about what has happened at RE before the data space occurred. So RE very quickly a few years ago entered into a digital transformation. Customer experience, supply chain, connected products.
We took all of this on at the same time. Data was part of it. But I will tell you in February of 23 when we launched our transformation budget, the first thing that got cut was data. How do you do a transformation without data? Well, guess what? It didn't work that way. In April of 23, we clawed back budget. We we clawed back money from the executives and we built the foundation of an enterprise data platform. Um, so you see April, April was really when we launched the data bricks platform and from April through September, this team Elen team built an amazing platform uh that I think we did in maybe a year's time, three years worth of work.
So kudos to the data bricks team and the eny team for pulling that off. Um shortly after that success we wanted to investigate more efficiency, more productivity, potential uh cost savings as well. Unity catalog uh was then brought to my attention, the team's attention. We entered that journey in in September um and really kind of evaluated um and finished the upgrade in a short amount of time. At least we'll talk a little bit about that uh in a few slides coming up.
So again, in a short amount of time, not to drain all of the success, but for those of you who are entering the data brick space, entering the data platform world, um a lot of great things can be done in a short amount of time, adding a tremendous amount of value to the business. Okay. Um business value delivered across the manufacturing value chain. So a couple of things here. Our secret sauce was not technical. Our secret sauce was building value stream teams led by business supported by technology.
Really really important thing. We now have the business value streams poking on us, pulling us, fighting for our time to create data products. very foundational thing. I'm going to take you through a couple of successes along the way. Sourcing tariffs top of mind for everyone in the industry. We used the datab bricks the datab bricks platform. We brought together all of our sources of supply, all of our suppliers into a single source of truth in data bricks.
We analyzed uh tariffs uh for imported material from China on our direct material spend. Within minutes, we received an answer. Funny story. I'm from Boston, so I have to tell a funny story. While we were demoing the power of Genie and the power of the datab bricks platform and showing the quick response to tariffs, our FPNA woman who in parallel was trying to find out the same answer had to along the way get a new laptop, order new hardware, new memory, and three weeks later she got the same answer we did within 10 minutes.
So she was not too happy. Uh second use case customer. Uh we went after the customer um data set if you will. Uh we created a product called distributor 360. a method by which to create uh more stickiness with our distributors by using their data, their point of sales data alongside our inventory data to create a collaborative 360 view of our collective data. Uh really really well received in the industry and while we were doing that we said hey we might as well take a crack at uh master data consolidation.
I think 3. 6 six uh distributor records consolidated to 26,000 in a very short amount of time. Very proud of that. Being at a conference like this, we have to talk about architecture. Not not my forte. So quickly um what you see here is I think a traditional medallion architecture, right? Bronze, silver, gold. You can kind of see uh the layers that would um impact a medallion layer. Uh you see data sources there. Uh Oracle and Microsoft.
That's just to name a few. Those are a few of our uh larger enterprise applications going through the medallion architecture. I will say that prophecy part of this architecture is very much part of a lot of the value that our organization has seen. You heard me mention 3. 6 million distributor records consolidated to 26,000. We've had similar success with suppliers. Um, one might think that's a very very difficult thing to do, but this team has done it very very quickly.
Okay, at this point I'm going to turn it. She's going to talk about the Unity catalog platform and the success we've made while upgrading. Thanks. So the decision to move to Unity catalog was driven by three key objectives. One, as RE enters the era of their AI transformation, we want to be sure that their data is AI ready. So RE is making significant investments in data governance in people, process, and technology.
And so through this, we wanted to make sure we're taking advantage of those governance capabilities embedded in UC like row and column level security lineage audit features so we can be critical to this mission. Second, as RE focuses on growth and efficiency, we keep we are constantly being asked what can you do data team to accelerate delivery to meet the business goals. And so we wanted to make sure we were taking advantage of Genie, the embedded AI features within UC.
So then we can increase our productivity with our data with our data and dev teams. And then last, as you know, we saw on the previous slide from, RE is very complex. There's a lot of business units, a lot of supporting functions, global, right? And so as we've transformed RE into more of a pull model from the business, we have compounding data and AI requests coming to us. I want to see this. I want to see that. I only want to see it for water.
I only want to see it for air. So we really have to be intentional and scalable with how we're going to share our data. So let's talk about our approach. We decided to divide the engagement up into two phases. First, a three-week PC where we migrated a low-risk solution. And then second, the project phase where we successfully migrated over 600 tables and views and we made code changes through automated scripts to over 200 notebooks and over 300 ADF pipelines.
So the goal of the P first, we wanted to validate the recommendations provided by our data bricks essays. Thank you. and we wanted to see what would work in Re's environment. From this, we learned that we actually wanted to do all of the migration at once. We wanted to do a big bang approach. And so, we actually increased our scope by 60% by reducing our timeline by four weeks. It's easier to just do it at once. We didn't want to break it up into higher priority solutions in the beginning and lower priority solutions later.
We were considering that maybe because of cost, but let's just do it all at once. Then when we moved into the project phase, we spent time up in the beginning doing that inventory and analysis. So we reviewed our environments, reviewed all of the solutions in scope, looked at all the artifacts we wanted to migrate, used UCX to do this, and we wanted to assess what really needed to be migrated, right? What were backups?
What was just testing? What was something that should just stay in dev, right? Developers were playing with it kind of like their sandbox. And so that helped us clean up our environment. We assessed all the environments for consistency. And we also wanted to review what was UC code compliant. And so what you see here is we actually spent a decent amount of time or decent amount of effort in the beginning of the beginning of the phase of our UC migration.
And by doing that, that helps with a seamless code migration and testing and stabilization phase later down the line. So, some might call us a little crazy um but we actually decided to go live with a brand new solution for a business unit that was also going live on Oracle at the same time um that had acquired uh at the same time that we went live with UC in production. So, we actually ran that in parallel and that would not have been possible if we did not focus that effort up front in our planning, prep, inventory and phases.
And so we were really coordinated and we actually and made sure that we you know could actually execute seamlessly and efficiently down the line. Okay. So some tips and tricks for a successful migration. I will start with the dos. So you're going to want to create a new Unity catalog enable cluster with that default catalog set because that will help with what that will help with limit your code changes. It'll help save time with that. you can just point to the new cluster.
You don't have to go update all your queries. So definitely do that. That was a great tip. Second is you're going to want to have a consistent IT point of contact from your infra team. There's a lot of coordination um in that PC phase, right? It was also really helpful. We did all of our UC setup. Then we set up the metas store cluster SCIM catalogs groups permissions. So you're going to want to make sure you're always doing that with that one point of contact. nothing.
You don't want anything lost in communication. And then also, you're going to want to make sure that your dev teams and your technical solution owners have the bandwidth to be able to validate those solutions post migration, right? Is everything working properly? Is everything functioning? Are you getting the expected results? Are pipelines running? Okay, because our, you know, the migration team might not know all the nooks and crannies and the details that you're going to have to know.
Now, on to the don'ts. So, who here has uh seen, you know, some developers maybe skipping a step? Uh, oh, I see a quick fix. Let me just put that into test. I need to get something out right now for this meeting or for this big demo. Let me just put into prod. Right? So, you're really going to want to make sure your environments are cleaned up and consistent before you're actually executing that code migration. So, whatever you have in prod, make sure you have in test and make sure you have in dev.
And then you don't want to wait until migration to perform perform those code changes. So sometimes you have to have some bug fixes. There were some deprecated libraries that weren't supported by Unity Catalog anymore. You're going to want to make sure you do all those code changes, make all those updates up front before you actually do that code migration. And then last, do not pick and choose artifacts to migrate separately. migrate everything all at once.
Whether it's 10 tables or a thousand tables, it does not impact your ef your effort or your timeline that much. You will thank yourself later. Less of a headache down the line. Okay, so now on to the value. We've only had UC enabled for a couple months and we're already seeing tons of value. We've seen a 20 to 50% reduction in notebook development time within our with for our developers. Yes, there was an AI assistant before you see but now with Genie it's better.
Before it was pretty basic could join a couple tables um but you know but now you can add context to that prompt. You can ask for those expected results. you can actually and it's taking the power of UC whatever you have cataloged and it's adding to that prompt and it's adding to that and adding to that code generation looking at primary keys table names and so you're going to get a better output second for that supplier use case that was talking about we actually integrated with Dun and Bradstreet to pull company information and Dun's numbers so we could create a supplier hierarchy and um Tarun over here actually in the front row he had to coordinate with our uh Dun and Bradstreet team or with the Dun and Bradstreet vendor um to configure the API, troubleshoot testing, right?
Making sure it was working. Um I'm pretty sure they're even trying to go have us update it now, right? Um so a little frustrating, hard to maintain. Um but now that data is just available in the Delta Sharing Marketplace. We can just procure that table, bring that table into the workspace. Now we don't have to go configure that API with D&B. ready to go. Up to 90% reduction integration development time. Predictive optimization.
Who here uses vacuum operations, right? So now that's automated. We're saving time with our developers. We're seeing a reduced compute and reduced storage cost within Azure. All right. So also with uh with the complex beus that ream has we're talking about right water only wants to see water and air only wants to see air what we can now apply those access controls upstream in UC govern that there and then now that's applied downstream to all of the reports that are accessing that data.
So we're seeing a significant reduction in that implementation of BU level security because instead of having to replicate that rowle security column level security 10 15 20 times for all of those 20 reports that we have live right we're going to do it once and there we go and then as RE is making those investments for their data governance tooling right we're able to jumpstart that with integrating with Unity catalog and having that lineage already brought into Microsoft Purview we're doing some PC's right now figuring out the best best places to start within the organization to use purview and so this really helps and then last uh you know I'm sure all of us here have experienced IT teams or infrastructure teams that are very busy have a lot of competing priorities uh you know sometimes it's it's difficult uh to you know get on their calendars uh and so we you know now we've eliminated that overhead uh with those from expiring credentials that's automated now those credentials with ADF, PowerBI, Azure, right?
So that's all automated now. We've eliminated that overhead and now we don't waste time, waste development time, productivity with the our dev team and with the IT team. And so if you've been contemplating on migrating to Unity Catalog, you're a little concerned on do I really want to do this now? Do I want to wait till later? Is this something that's going to take a lot of effort? Is it really going to halt my development time?
Right? I'm I I I really do encourage you to, you know, make the change, do the upgrade. You know, you'll already see your ROI very quickly. Um, and if you have any questions, um, you can come talk to us after the session. And then lastly, I just want to thank our amazing UC migration team. We have a few every a few of them up here in the room. You can stand up. We give everyone a round of applause. This would not have been possible without them.
So, thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.