Sponsored by: Confluent | How Grainger Powers Real-time Supplier Compliance

Summary

Grainger moved from batch pipelines and manual supplier scorecarding to a real-time compliance platform by using Confluent Tableflow to bridge streaming Kafka data directly into Delta tables within Databricks Unity Catalog, eliminating brittle ETL.
Confluent Tableflow handles schema evolution and compaction automatically, enabling domain teams to self-service onboard supplier data streams with low cognitive load and without custom ETL engineering work.
The platform supports machine learning and AI workflows on top of live supplier data in Databricks, giving Grainger's procurement teams real-time insights across a supplier network of approximately 5,000 partners serving 35 million products.

Chapters

FAQs

What is Confluent Tableflow and what problem does it solve at Grainger?

Confluent Tableflow bridges streaming Kafka data into Delta tables within Databricks Unity Catalog, eliminating the custom ETL code that organizations typically write to move data from Kafka into a lakehouse. At Grainger, this removed brittle pipeline engineering and allowed domain teams to onboard new supplier data streams through self-service, without requiring platform engineering involvement.

How does Grainger's supplier compliance platform work?

Grainger's platform ingests real-time data from approximately 5,000 suppliers through Kafka and Confluent, landing it into Delta tables in Databricks where ML and AI models analyze supplier behavior and compliance status continuously. The result is that procurement teams get live insights rather than periodic batch reports, enabling faster and more confident decision-making.

How does Confluent Tableflow handle schema evolution in streaming pipelines?

Tableflow manages open table format maintenance automatically, including schema evolution and compaction, so engineering teams do not need to write custom logic to handle changes in incoming data structures. This significantly reduces the operational overhead of maintaining streaming pipelines over time as supplier data formats change.

What were Grainger's guiding principles for building their data platform?

Grainger's platform engineering team operates on three directional beacons: self-service access, low cognitive load for engineers, and making it harder to make mistakes. These principles shaped the decision to adopt Confluent Tableflow for supplier data, since it allows the 70-plus internal engineering teams Grainger supports to access streaming data without needing deep expertise in pipeline management.

Full transcript

[00:08] Uh hello. Welcome everyone to our session. Good afternoon. Um my name is Paul Capelluzzo. I am the principal data architect uh for uh Grainger. Um my mandate is to increase data maturity across all of Grainger. And for the
[00:22] first uh 2 years I've been working on bringing data mesh and data products to Grainger. I'm part of the architecture team. And uh I work very closely with the platform engineering team, including Pradeep. Uh
[00:35] and the data engineering organization, few people here representing. Um Uh yeah, that's me. Go ahead, Pradeep. >> Cool. Good afternoon, everyone. Welcome.
[00:47] My name is Pradeep, and I lead uh I'm a senior manager leading platform engineering a data streaming org. Uh essentially, we support 70-odd engineering teams which are internal facing.
[00:59] So, we support them essentially with the three beacons: self-service, less cognitive load on the engineers, and uh making it harder to make the mistakes. That's Those are our three-directional
[01:11] beacons that we operate on to facilitate data streaming across the org. And um yeah, uh we can go on. >> Okay. >> For the folks who are not familiar with Grainger, um
[01:24] we uh we are a 100-year-old company, close to 100-year-old by next year. And uh we have close to 35 million customers, uh sorry, 35 mi- million products that we offer to the 5
[01:37] million customers. Um that's that's insane amount of products and SKUs that we carry, uh which we procure using our uh vast network of suppliers, close to 5,000.
[01:50] So, uh with that said, uh before diving deep into the supplier flow, how are we solving the problem? Let's take a step back and uh talk about like how did we evolve our data strategy and
[02:04] architecturally? Paul, you want to take it over? >> All right. Okay, so we've been using Granger for a long time, mostly for machine learning. It's our data repository for machine learning. We also have some transformations and
[02:17] some ingress of data that gets executed in Databricks as well. There was an existing data lake that was part of Databricks, but as part of evolving the data platform, we decided to create an
[02:30] independent data lake so that any component could connect to it to get the data that it needs. So that Granger data lake now serves as the system of record for all
[02:42] data that's emitted from source systems that people want to get out and make available to Granger and for data product data. So we're using Databricks as our batch
[02:54] data processor. So it reads the source system data, applies transformations, filters, and then outputs the data product data back to the Granger data lake. So it never gets persisted inside of
[03:07] Databricks unless there's a need to kind of stage some data. The system of record for data product data is actually the Granger data lake.
[03:19] Okay. So one gap that we saw was that we have a bunch of data that's sitting in Kafka and we want to make it available available for non-streaming use cases. So we want to get that data potentially into Databricks, potentially
[03:31] into Snowflake or data warehouse. And so we didn't really have a standardized, well-governed, centrally supported way to do that. So domain teams were kind of left on their own to basically build
[03:43] their own. And that's the gap that we that we fixed here. Okay, so prior architecture, this is prior to the solution. If you look across the top, that's kind of a
[03:56] non-streaming path uh for data from source systems um to our different data platforms like Databricks and Snowflake. Uh so but we're focusing on that streaming path, which is across the
[04:08] bottom. Um so as you can see, the data first enters kind of the Confluent realm at Grainger. We use Kafka connectors to bring the data to actually pull it from source systems persist it in Kafka, and
[04:22] then we have connectors on the uh opposite side of that to then push data to different destinations. So prior to our solution, we had connectors that was that were pushing
[04:34] data directly to Databricks, directly to Snowflake, directly to the Grainger data lake. Um and then there was a bunch of So that was custom work, and then we had custom work in terms of pipelines to actually
[04:48] then replicate that data inside of Databricks and Snowflake. So we had kind of a multi-component multi-hop, multi-persistence kind of a path leading from Kafka to um
[05:02] places where we wanted to serve up that data for different use cases. Um and so the burden, if you look off on the right, the burden is mainly on domain teams to make this work. So platform is supporting Kafka, supporting
[05:16] the Connect clusters, but domain teams have to build their own their own code, their own pipelines both in the Kafka connector and then in the pipelines um that pull the data from the data lake.
[05:28] So they're responsible not only for the topics and schemas, but then for creating the connectors, the S3 buckets, those pipelines, and also managing all of the schemas within the data lake,
[05:40] within Databricks, and within Snowflake, cuz the data is being persisted in all three. So that was a big problem for us. Um
[05:53] Okay. So if we look at some of the challenges that we're facing, that leads to what are the requirements for the solution. So the first challenge for us was enterprise scale. Uh our domain teams have varying uh sets of members, varying skills, knowledge,
[06:07] um and then competing priorities and workloads. So it's really hard to scale that kind of model because we're relying on domain teams that are not dedicated to this process.
[06:19] So in terms of requirements, sorry about that. In terms of requirements, what we wanted to do was set up a centrally supported automated system for getting data out of Kafka and getting it into the data lake.
[06:33] Um and we wanted fast onboarding for domain teams. Um and we wanted self-service. In terms of reliability, because each domain team's creating
[06:45] their own kind of path out of Kafka and into Databricks and Snowflake, um they're creating these multi-component, multi-hop pipelines that are bespoke, um and obviously it's pretty brittle, um
[07:00] because there has to be runtime coordination of all those different pieces to make everything work properly. So that leads to a requirement of having a standardized, reusable pattern um that
[07:13] domain teams can easily implement so they can design, implement, and then manage uh these new pipelines. In terms of latency, of course we've got batch and microbatch kinds of latencies,
[07:25] um which limits the ability of downstream consumers to meet certain use cases. Certain use cases require near real time, um and we weren't able to meet those use cases with the current architecture. So
[07:38] there was a gap in our data usage across Granger. Um so then of course the the solve is let's get it to near real time and now we can enable all the use cases that we
[07:51] can with that data. And that's a big unlock for Granger. Um and then the final piece is governance. So if you're maintaining these schemas and trying to keep them all in sync across Kafka, the Granger
[08:04] data lake, Databricks, and Snowflake, uh that's a manual process. Um and every time there's an evolution upstream, you need to evolve everything that's downstream. So that leads to a requirement of how
[08:17] can we have kind of unified automated governance as part of uh this new solution. Okay. So uh since uh since Confluent is
[08:29] part of our presentation here, uh it's pretty easy to guess that Confluent was part of the solution. But before we get to the solution, let's take a look at some of the alternatives that we did not choose. >> All right. Good stuff. >> Um
[08:41] directionally we are aligned that we want some centralized governance and we want a standard pattern across our engineering teams to implement the data pipelines, right? So how do we achieve it? Yeah, you know, one either do a
[08:55] custom build, uh provide the software and then maintain it, and it is going to take some time to get it out of the market. The other option we have is let the teams build their bespoke implementations to have this data
[09:08] pipelines enabled on demand. And it is going to be a compounded effect on the engineering teams. Every team had to go through and build through the same things over and over again. And there is no observability in a common place where
[09:21] we can go ahead and track which team is implementing best practices versus which is not. And there is a lot of engineering hours that would be spent by doing this uh implementation bespoke way from each of our internal engineering teams.
[09:33] That's We have close to 100 engineering teams and out of those 70 are using Kafka or even streaming. So, we are saving tons of engineering hours if we are able to obfuscate all of that
[09:46] in you know, implementation details out of them and give them a process where they can you know, implement it in repetitive way with all the sensible defaults and the observability baked into it and
[09:59] essentially you know, with no cognitive load at all. So, obviously we went with buying it off the shelf for the reasons you know, we can go we cannot achieve the speed to market
[10:12] much more aggressively compared to building it on our own. So, yeah, we'll go dig into the architecture how we implemented it in the next slide bar. Okay, so this is our new architecture and again focusing on that
[10:25] streaming part on the bottom. What we've added into the mix is Tableflow. So, that's been our solution for this and for those who don't know, Tableflow is a Confluent managed service that essentially syncs data from Kafka to a
[10:39] data lake using either Iceberg or Delta Lake as the open table format. So, this was the unlock for us. This is something you can set up very easily for a given topic.
[10:51] Tableflow manages the schema for the destination open table format table in the data lake. So, it will evolve certain schema changes that happen upstream, not all of
[11:04] them. Especially breaking ones, it's it's not going to be able to to manage that downstream. But so, that is now the pipeline and it's the only piece in the pipeline
[11:16] because what we added on top of Tableflow was the implementation of external table definitions in Databricks and Snowflake. So, this allows the data to live in the data lake, but then be surfaced without
[11:28] replication in both Databricks and Snowflake. So, this enables us to bring the data to where people are working at Granger. Um So, this supports uh the machine
[11:40] learning that's happening in Data Bricks, uh AI that's happening on top of Data Bricks, AI and reporting that's happening on top of Snowflake. So, the shift now for domain teams is
[11:52] that they still own topics and schemas. They still own creating the external table definitions, but those definitions don't require an explicit schema. It's just a pass-through kind of an object.
[12:04] So, the schema gets inherited from the Delta Lake tables in the data lake. So, now there's again propagation of schema evolution into Data Bricks and Snowflake
[12:17] not for every change that happens upstream, um but for a lot of them. So, you get automatic syncing of schemas along the whole pipeline. Um and we no longer have to replicate data into Data Bricks or Snowflake.
[12:31] So, that Granger data lake becomes the true system of record for data product data. It's just being uh kind of served up in Data Bricks and Snowflake. So, for the platform team, they still
[12:43] own Kafka. They still um own the S3 buckets, but they're also just managing Table Flow now. So, we've shifted a bunch of responsibility off of domain teams and added a little bit of responsibility to
[12:57] the platform team cuz Table Flow is a Confluent managed service. Okay. So, the next slide um is uh it's kind of a uh
[13:09] What is Table Flow? Kind of a deeper dive into what is Table Flow. So, looking on the left, those are all of the features that you get with Table Flow. So, when it says convert to Parquet, it means using Parquet as the data format
[13:23] and we're using data lake or Iceberg metadata on top of that. Schema evolution we talked about. It automatically creates the schema in the data lake, so it's doing that type mapping from your Avro schemas, let's
[13:36] say in Kafka, and your open table format tables in the data lake. Um it's also handling compaction and maintenance automatically on those uh open table format tables in your data
[13:49] lake, which is nice. Um and and you're getting materialization of CDC. So, you get a ton of benefits out of this. It's completely automated. Um and now those Delta Lake tables, like we talked about,
[14:01] can be surfaced in Databricks automatically. So, this is how we're getting near real-time latency in moving data from Kafka into Databricks and Snowflake.
[14:15] Okay, so now we're going to take a look at just a very short demo of how easy it is to actually set up Tableflow on an existing topic. So, it's a video. Uh there's no talking. There's just going to be you'll see like some cursor
[14:27] movement, and it'll basically walk you through the steps um that it takes. This is right out of the box. So, this is not with any customizations or any abstractions on top.
[14:47] So, you can see it's just, you know, within this this GUI here. You basically set up your storage, your your destination storage. Set up your settings.
[15:03] And now you get all of this observability, monitoring, reporting built-in. And so, here you can select uh the open table format essentially. Destination format.
[15:26] >> Okay, and that's it. That's literally all it takes to do. Thank you. Okay, so that's the overall architecture and solution from kind of a high level, and now Pradeep is going to walk us
[15:38] through an actual use case where we implemented this at Granger. >> So, um now that we saw how architecturally we're aligning, um how do we use that to
[15:50] activate our contracts or supplier compliance across our organization, right? So, um as I mentioned earlier, we have close to 35 million products, and those are procured from close to 5,000 plus
[16:02] suppliers across the globe. And uh for our supplier team, it's uh most important that they may monitor the compliance issues of the suppliers, uh whether they're able to provide the
[16:15] specific item in time, because that defines what time are we going to submit you know, send it back to the customer. This defines our line of how much time is it going to take for the shipment, uh when is the product is
[16:28] going to available. All of this information is dependent upon the supplier. And we have a diverse set of customers which has compliance requirements like federal government should have only specific
[16:40] rules around which supplier should we procure from on sensitive uh equipment and such things. So, uh how do we do that? Uh the if we go back to the uh next one. Uh
[16:56] Oh, sorry. Yeah. Um yeah, so we have a app that's called a supplier uh con- supplier app or supplier portal where suppliers would fill in the information. Historically, they fill in the information, it's a batchly jobs.
[17:09] The by the time it reaches to the operational or the orchestrator flow of the our purchase order, it's a stale data. It takes probably batch data after a day or so, and that means we lost the
[17:22] accurate information from the past 24 hours. versus the newer one where we essentially even source all of our events out of the supplier application itself and write it to Kafka. And our downstream
[17:35] process pick these events and to do their operational processing processing as usual. So, when we talked about the architecturally how are we solving or bringing the operational data into a
[17:47] business insights, this is where the simplicity of usage. So, we reached out to this team like, "Hey, you guys have the data in Kafka, why not you just, you know, enable it for our
[17:59] analytical use cases?" And uh Uh if we go with the traditional path, it would have been probably 3 weeks of engineering time for them to set up the connectors and then probably, you know,
[18:11] set up the schema and the S3 buckets and all of it. Now, they can just go and click a button and the data pipeline is enabled. So, we saved a lot of engineering hours by enabling this, and all of a sudden we
[18:23] have our operational data liberated into analytical use cases without much of a bespoke implementation. So, this uh the adoption of this had grown
[18:35] significantly once we launched it because it's easier for for our internal engineering teams. They don't have to reinvent the wheel for every data pipeline enablement. It's rather just like they go and click through few of
[18:47] the internal portal buttons and that's pretty much it, and they have their data available for analytical use cases and providing the business insights that are needed. Uh
[19:00] So, what we achieved is we actually monitor our all of our suppliers in near real time, and we have one source of truth for both analytical and operational use cases. And
[19:12] our engineering hours freed, which means they can do much more valuable work that's unlocking our business use cases. Um no more custom integrations. Uh I think we launched it close to 4 months
[19:25] back and our adoption rate has been pretty aggressive in a way in a good way. Uh right now, 10 different domain teams are using the table flow. Um that has been our and we are doubling
[19:38] down on scaling it and building more abstractions and observability towards this. All right. So, it hasn't been a smooth sail for everything. So, we We We We had to figure out a lot of
[19:52] things. Um before we uh move on to next steps, we would want to go through and, you know, talk about how should we approach if we had to do it now. We would start with a specific business use case,
[20:06] not not the not pick the technology. That's what we did anyway. But, uh that's the directional point we would say. Start with the use case that you want to target and then uh choose the technology later on.
[20:19] Uh how are we solving it? And start with the simple use case. Uh data governance does matter here. Um like which uh the schema because the quality of the analytical
[20:31] quality of the event is defined by the which schema uh how are you defining the schema? Uh what's your compaction? How are you How much time are you retaining this All of these decisions matter. Uh and these need to be thought through
[20:44] before enabling the table flow itself. Uh Uh you know, go through the end-to-end flow. Do the value stream mapping of like enabling this. How does it going to unlock my business use case? And such things. Um
[20:56] Cross-team collaboration is very much needed. Uh now that you're working with your analytical users and your operational users, uh you know, they need to think through how are we enhancing the quality of the
[21:08] event that we'll be producing. Essentially, thinking towards considering your events as a data products. How can you do that? Yeah, and self-service and simplicity accelerates
[21:20] that option. You know, teams doesn't have to spend weeks means that they can just go through and bring it to production same day. And you know, all of it we internally we
[21:32] backed it with the infrastructure as a code. Everything that you see in a UI we we have an abstractions that are built for APIs. That's how we make sure it is repeatable and scalable and observable down the line.
[21:49] Cool. Yeah, we can move on. >> Okay. Okay, so that's kind of the end of the the meat of our presentation here. We highly recommend Confluent and TableFlow. It obviously is working really well for us at Grainger. So, if
[22:02] you want to learn more about it, definitely check out Confluent at booth 510. Um and I got to say actually going back to this slide, leveraging Confluent's Kafka expertise,
[22:14] we've worked very closely with the Confluent folks. And whenever we have a problem or something that we might want as a feature request, they're very receptive. We work with them. We have early access
[22:26] to features and we're pretty early adopters of the tooling that Confluent is putting out. So, that's actually a big plus of I think of working with Confluent. They're really
[22:38] you know, more of a partner for us really than just a vendor. So, that's a big selling point for us. Definitely go check them out. They're doing a lot of great stuff. And of course, if you want to come work
[22:51] at Grainger, we're also doing a lot of good stuff. Not just in the data maturity space, but in overall technical maturity. There are some interesting issues around scale, but also it's it's such an interdependent ecosystem that uh the
[23:05] integration points are very important. They either enable or prevent us from doing certain things. So, there's a lot of good stuff to work on at Granger and a lot of good people.
[23:18] >> Come work with us. >> Come work with us.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.