Sponsored by: Perficient | From Second Gear to Full Throttle

Summary

Perficient helped a large construction distributor with over one million customers activate a broader set of Databricks capabilities—beginning with environment tuning, FinOps, CI/CD, and Unity Catalog governance before building production features on top.
A Customer 360 integration connecting Salesforce Marketing Cloud to Databricks was delivered in six weeks, replacing batch reports with sub-100ms queryable APIs for real-time customer data access across sales, marketing, and operations.
Two million plumbing product SKUs locked in heterogeneous PDF spec sheets were extracted using an agentic pipeline—an intelligent router plus LLM critic agent writing results to Lakebase—delivered to production in four weeks with 2.5 FTEs and designed to be reusable across all business divisions.

Chapters

FAQs

What does it mean for a Databricks customer to be in second gear?

Being in second gear means having access to 12 or more Databricks capabilities but actively using only two or three of them. According to this video, the barrier is not adoption but activation—customers often try to use the Databricks Data and AI platform the same way they used older data tools, missing the unique capabilities that make it powerful for modern workloads.

What two workstreams did Perficient execute for the construction distributor?

The first workstream built a real-time Customer 360 by integrating Salesforce Marketing Cloud with Databricks, delivering a unified customer view with sub-100ms queryable APIs completed in six weeks. The second extracted product specifications from two million plumbing SKUs in PDF files using an agentic pipeline with an intelligent router and LLM critic agent, completed in four weeks with 2.5 FTEs.

How does Unity Catalog govern agentic workflows in this architecture?

Unity Catalog manages access controls and governance for both traditional data assets and AI agents through the Unity AI Gateway. This ensures that agent actions are subject to the same permissions and audit logging as conventional data queries, maintaining compliance even as autonomous agents take on production work on the Databricks Data and AI platform.

What made the agentic document extraction pipeline reusable across business divisions?

The pipeline was built with pattern-based components—an intelligent router, an LLM critic agent for quality validation, and a Lakebase output store—rather than bespoke engineering for a single document type. This pattern-based approach means the same pipeline can be applied to spec sheets from different product categories or business divisions without rebuilding the core extraction logic.

Full transcript

[00:09] Good afternoon, everyone. I'm too loud? Man, when they when they say you're too loud, I always say you're too old, but my kids say I'm it's so loud because I'm old. Is that any better?
[00:21] Or is it still too loud? You have to do it on the headphones,
[00:33] folks. If it's too loud, there's a volume button on the headphones. My apologies. A lot of people say I don't even need a microphone.
[00:46] How many of you are running more than seven Databricks capabilities in production today? Not that many hands up, and that's what I hear from a lot of customers.
[00:58] My name's Michael Patterson. I'm the vice president of AI data platforms at Proficient. And over the next 18 minutes, I'm going to talk to you about how we helped a customer take their Databricks environment from second gear
[01:11] to full throttle. So, that doesn't look much like a Databricks environment, but that's okay. I'm a car guy, and I have been my whole
[01:24] life. And when I'm not chasing kids or playing with hairy data, I like to race cars. Most of my racing career has been in a Mazda Miata. And for those of you who
[01:36] know anything about cars, it's a fun car, it's great handling, it's not that fast, and it doesn't have that much horsepower. I like to say it's a little limited on its capabilities.
[01:49] So, about 15 years ago, I was afforded the opportunity to race a Porsche 911 at the Autobahn Country Club in Joliet, Illinois. Dream car,
[02:01] tons of power, was super excited for it. Suited up, put my helmet on, went out on the track, started running laps, and was instantly disappointed with myself because I couldn't run that car
[02:14] around the track any faster than I could my Miata. I spent the whole morning trying to figure out why I was going slow.
[02:30] I've driven for years, I knew the track, this car was incredible. I knew it had capabilities, but I just couldn't figure out how to get it to work right. I went into lunch, sat down with the team I was with, and I
[02:42] was saying, "Guys, I don't know what I'm doing wrong. I'm I'm doing everything I always do, but I'm slower. It's just not there." So, one of my teammates offered to come out and ride with me for the afternoon
[02:55] and coach me. I really thought about it when I think about what we're doing here with Databricks is this is where we see a lot of customers. They have incredible capabilities in this platform,
[03:08] but they're trying to use the platform the way they've traditionally used other platforms. They're at the starting line. They've, you know, often they put Databricks in a number of years ago for
[03:22] a data science or a machine learning use case, right? So, they got the environment stood up. They've got some data flowing through Lake Flow. They've built their models. They've They've shown the value. They've got it
[03:35] in production. And And for the most part, and I always hope they do, they've got some security and governance in place. And it works, and it's good. And that's all they're doing.
[03:47] But think about that. That's like being stuck in second gear in a supercar. You're using two to three capabilities of a tremendous platform.
[03:59] Now, I did these slides about 3 weeks ago, so they're out of date because the moment you put a metric down, it's out of date. About 3 weeks ago when I did this, I looked at the total capabilities
[04:12] in the platform. I pulled out the infrastructure capabilities, compute, storage, that it's table stakes. You don't have a platform without it. And there were about 15 major capabilities depending on how you classified things.
[04:26] And then I got here Monday. And this is woefully out of date. Right? OmniGen Genie apps, Unity AI Gateway, Rydin,
[04:38] like if you're not jazzed up after this week, I don't know what to give you. The The coffee out here isn't going to be enough. But, what it does mean is that there is even more idle horsepower sitting in that
[04:52] supercar in your environment. So, let's shift gears. I want to talk to you a little bit about how we helped the customer who was in this very situation
[05:04] go from second gear to full throttle with their Databricks environment. Now, this customer's a large distributor of construction-type products, right?
[05:16] They They distribute the products that are used in building hotels, building municipal projects, right? Big big distributor. Tens of billions a year in revenue, over 30,000 associates, over
[05:29] 1,500 stores, and over 35,000 suppliers, right? Over 1 million customers. So, this is not a small enterprise, but
[05:42] they were in the same situation as most of our customers where they were at the starting line. They called Proficient and said, "Hey, we need your help with a marketing
[05:54] problem. We have over a million customers, but we don't have a single view of who they are." This is not a traditional data problem, but it turned into one.
[06:06] They were trying to get off of a legacy marketing platform. They worked with our team leveraging our Envision framework to help them choose a new marketing platform. They chose Salesforce Marketing Cloud. It was a
[06:19] very good choice. We're premier Salesforce partner. Great. But, they had a data integration issue. Their sales teams had one view of the customer. Marketing had another.
[06:31] Operations and customer support had yet a third. So, as they wanted to bring this all together, they knew they needed to of course have that traditional data integration layer.
[06:43] They said, "Hey, let's use Databricks for this." We thought that was a fantastic idea. Came and looked at the environment, and we noticed the car wasn't ready for the race.
[06:55] They had started their Databricks environment about 5 years ago, and they were using it primarily for some machine learning workloads. But, what they hadn't done is thought about how they would use that platform
[07:08] for other use cases going forward. They didn't have a lot of the basics in place. So, they were using standard compute, not job clusters. They just had
[07:20] everything on and running, even though the majority of time it was sitting there idle. That was a huge FinOps opportunity, right? We were able to guide them to a better way of running the environment, save money right out of the
[07:34] gate. The second thing is we looked at how they were ingesting data. They were just using JDBC connections, not even pipelines. So, we're able to look at the data sources they needed to bring in
[07:46] for this customer 360 and do proper pipelines, which led us to another revelation. They had a big monolithic repo for their CICD. Every time they made any change, they
[07:59] were redeploying everything. So, hundreds of jobs were getting deployed for the smallest change. Ironically, that adds more risk than anyone thinks because sometimes not everything in the repo is ready to go, but they just
[08:12] didn't have the controls. I know our our team gets mad when I say like good mechanics, we were able to tune the car and get it ready. And then finally, one of the biggest issues we ran into was their governance
[08:25] in their Unity Catalog. They had cataloged the assets in Unity Catalog, so that is good. They had seat belts, they had an airbag, they weren't using them.
[08:37] They had no roles in two groups, admin and non-admin. And when you're just doing machine learning, that's well, I'm never going to say that's fine. So, I'm not going to say that's
[08:49] fine. But we see it a lot, right? It's not the end of the world. That's what we're there. We're that coach. We're able to help them get the right security, get the right groups in.
[09:03] Once we had the car tuned, now we were race ready. Okay, so now we have four sources of data to bring in, get everything mastered, and get it into Salesforce Marketing Cloud for them from Databricks
[09:16] using sharing. Honestly, that was the easy part because we were able to do this entire project for them in 6 weeks with four FTEs. That's coming
[09:28] in, evaluating the entire environment, going and acquiring the data sources, getting everything remediated, and put into marketing cloud for them. They now
[09:40] can do full omni-channel marketing. It doesn't matter if it's sales, it doesn't matter if it's operations, everybody billing, everybody has the same view of the customer, holistic view of data,
[09:53] they can ask all the questions, and it's because in Databricks they know they have a single source of the truth that is secure and governed. That's a win. That's a huge win. And again, this was
[10:07] not a monumental effort. This was simply unlocking capabilities that were in the car. So, I think at this point, I'd like to say they were third gear getting ready to get into fourth gear.
[10:23] But now we had a bigger problem, or I should say a bigger opportunity. They had a line of business. I mentioned this is a large construction distributor, right? They supply components and parts to everyone. They needed to have some help and some
[10:35] work done on their their product information management, their PIM system. So, they have um a a plumbing division that has almost 2 million SKUs, and they have a PIM. This is not an immature customer.
[10:48] Remember, they do tons of billions of dollars a year. I don't want anyone to think that they are not a sophisticated customer. But like a lot of people, they had siloed systems. So, their PIM system had for for plumbing almost 2
[11:02] million SKUs in it, and they knew who the vendor was or the supplier, they had the basic information of, you know, the description, and that's about it. Now, the challenge is in their industry, when someone's going
[11:14] to build something like the Moscone Center, or maybe a large municipal project, they have to answer requests for proposals. And it's funny because when they were saying to us, "Okay, here's the problem
[11:26] sales have. They get these RFPs in, they have a really short time frame by which they have to answer them. They have to have the exact right requirements and it has to meet the specs and I was like, "I don't know what that's like at all."
[11:40] "We do that every day. But, what's the challenge?" "The data the sellers needed to configure those quotes was not in the product information management system. It's in a bunch of spec sheets that are
[11:53] PDFs." "Okay." So, we asked them, "Where are those PDFs?" And they said, "Well, they're in Databricks." And I'm like, "Score! This is great. You're going This is fantastic. Why can't we use them?"
[12:05] "Well, they're just there. We don't really know how to get the data out of them." "Okay, talk to me. Why is Just use some document extraction. What's the challenge?" "It's not quite that easy."
[12:19] I'm not a plumber. I didn't stay at a Holiday Inn Express last night, but I learned a lot. So, each spec sheet, one would say, "Okay, there's a spec sheet for each product." We've got about 2 million, that's a lot,
[12:31] but we can build We can do that. Oh, different suppliers. How many? Tens of thousands? What format? They're all different.
[12:43] Got it. So, think about this. Some of the spec sheets were one spec sheet for one product. Some had all their measurements in metric.
[12:55] Some used terms like millimeter, centimeter. Some used terms like mm and cm. Other suppliers had the exact same thing, but they used imperial units.
[13:07] But, instead of just saying inches or in, sometimes they displayed everything as a fraction of an inch and sometimes they play did it in decimal notation. Okay, that's still not too challenging. What else you got?
[13:20] Well, sometimes we get a spec sheet that's not for one product, it's for a product family. So, it'll have an architectural diagram up top, a CAD drawing, and then it will have a set of tables,
[13:32] one or more sets of tables, that might have two SKUs, 20 SKUs, and then the associated specs for each SKU in that table. That's why they couldn't use the standard Databricks document extraction
[13:45] workflow. Okay. Not a problem. We'll build an agentic workflow for you. We put an intelligent document router up front. Gets the documents to come in.
[13:58] If it's a one-to-one, and it's a standard, we just use the native Databricks workflow extractor. Works great. In the case of a multi-variant, we went out and used the frontier model.
[14:10] In both cases, we route the next step to a critic agent, LLM as a judge, to ensure that what we pulled out of the document in fact makes sense. If it passes the agent, it goes in the
[14:22] lake base in a standardized format. If it doesn't, it's rerouted until we get it right. At the end of this, we had a fully agentic workflow that could take millions of disparate
[14:35] spec sheets, standardize extract and standardize the data, and get it into lake base. Awesome. So, now, how hard was that? Well, the
[14:47] reality is I had to take my lanyard off for the microphone. I was going to point at a button. I have an I Dream of Jeannie button that one of my friends at Databricks gave me. Anyone saw the the Jeannie announcements?
[15:01] We did this 6 months ago, but we did it with the origins and the original Jeannie. We were able to do this entire process, this entire agentic workflow, start to
[15:13] end two and a half FTEs in 4 weeks. From ideation through iteration through production ready in 4 weeks.
[15:31] So, when you think about it, they had the information, but they couldn't access it. It was dark data. It wasn't available. They had the capabilities, they didn't know how to use it.
[15:43] By coaching them and helping them understand how to leverage the platform, they were good to go. But, here's the best part about it. That entire Agentic workflow, we did it for their plumbing division.
[16:00] They have the same problem in their heating and air conditioning, in their electrical, in every other line of business, they're able to pick up and use the exact same Agentic workflow. Earlier, we talked about them using
[16:13] Unity Catalog. Well, this was a great opportunity for us to show them the power of how they could use Unity Catalog different than they were thinking about it.
[16:25] So, when we think about the the capabilities we leveraged here, the one I'm most proud of is that we're using Unity Catalog not just to govern the assets, because that's table stakes.
[16:38] We now have Unity Catalog governing the agents. This is where things are going. You saw the announcement about Unity AI Gateway. As we move into the Agentic frontier, if
[16:52] we don't have governance on our agents, we're going to have chaos. But, that's okay. Databricks gives us that capability natively. We just need to take advantage
[17:04] of it. So, in this case, five additional capabilities were unlocked. We could not have done this in 4 weeks if we didn't use the power of Genie.
[17:19] We would have normally iterated with Python several times through doing all this workflow. You know, a week at a time. It was a day at a time with Genie.
[17:31] So, if you're not using those capabilities, you're leaving horsepower on the table. You're just You're not leveraging what you could do. So, what are the key takeaways?
[17:43] Agent Bricks is the orchestrator we should all be using and really Omnigen. I'm going to actually do something you don't normally do when presenting, but one of my colleagues was coming in on
[17:55] Monday from Charlotte and at 30,000 ft on an airplane used Omnigen to build the new orchestration for a workflow on an airplane.
[18:07] That's the days we're living in now. 30,000 ft on an airplane, not the airline I like, but I'll leave that out. Built an entire agentic workflow using
[18:19] Omnigen and Genie and Agent Bricks. Genie is what accelerates the build. What we would take to do in sprints and weeks, we now do in days.
[18:32] We ask the what if question even if we know it might not be the right one just to see. And then lake base. If you can't serve the information back,
[18:48] you're just a data science platform. Lake base gives us that transactional capability, allows us to get the information back to the customers in milliseconds. So, again, it's about leveraging the
[19:00] capabilities you have. The platform didn't change. Realizing the capabilities changed. So, let me bring it full circle. I told you about racing that 911. I went
[19:14] out in the afternoon with the coach. Help me with my transitions, help me with my lines, help me reset my braking markers. I was racing a different car than I was used to.
[19:27] But I was driving a performance car the way I had always driven my other race car. I was braking at the same point, I was turning at the same point, I was trying to do the exact same thing I've always done.
[19:40] But with a different vehicle that was designed to do so much more. With the right coaching, I was able to have a wonderful afternoon. My laps got faster. I'm sorry, my my laps got
[19:52] faster, my speed got faster, and I had a lot more fun. That's exactly what we do with customers. We help you take your Databricks environment and go from second gear to full throttle.
[20:04] My name's Michael Patterson. I'm with Perficient. We're at booth 434, and if you want to know how to race your Databricks environment, come find me because we're here to help you. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.