From Web to Warehouse: Building a Real-Time Competitive Intelligence Agent on Databricks

Summary

This video shows how to build a competitive intelligence agent entirely on Databricks by combining LangGraph, LLMs, and live web data from Bright Data, with all results persisted in Unity Catalog for historical analysis.
Bright Data solves common agent failures—including 403 errors, rate limits, and hallucinated data—through browser fingerprinting and geo-specific content delivery, serving over 20,000 customers and 70% of AI labs worldwide.
The architecture uses specialist subagents for tasks such as scraping Amazon pricing and social mentions, with a fetch-once, cache-in-Databricks model that reduces costs and enables reuse across analytics workflows.

From Web to Warehouse: Building a Real-Time Competitive Intelligence Agent on Databricks

Your competitor just dropped their price. How long until you know? In this video, see how we built an e-commerce agent natively on Databricks - combining LangGraph, LLMs, and live web data from Bright Data, all persisted in Unity Catalog for historical analysis. We'll walk through the architecture, deploy a working agent live, and show how retailers can turn the open web into a queryable table.
Talk By: Swetha Kolli, Senior Director Solutions Architecture, Bright Data ;

Chapters

FAQs

How does the competitive intelligence agent built in this video work?

The agent uses LangGraph to orchestrate specialist subagents that scrape live pricing and social mention data from the web via Bright Data's APIs. Results are persisted in Unity Catalog on Databricks for historical analysis, enabling retailers to query competitive pricing trends over time.

Why use Bright Data for web data in AI agents?

Bright Data addresses common agent failures including 403 errors, rate limits, and hallucinated data through browser fingerprinting and geo-specific content delivery. With 14 years of infrastructure and over 90 petabytes of data collected, Bright Data serves 70% of AI labs worldwide as a compliant, ethically-sourced public web data layer.

What is the architecture of the competitive pricing agent?

The architecture uses Bright Data as the data discovery and collection layer, Databricks as the processing and storage brain, LangGraph for agent orchestration, and MLflow tracing for observability. Specialist agents handle tasks like scraping Amazon pricing and social mentions, with results cached in Unity Catalog to avoid redundant fetches.

How can retailers deploy this competitive intelligence agent?

The agent is available in Databricks Marketplace and can be deployed to Databricks Apps with a single YAML configuration. This one-click deployment model means retailers can get the agent running on their own Databricks environment without building the underlying scraping infrastructure from scratch.

Full transcript

[00:07] Morning everyone. Thank you for sticking through day three of information overload. For those of you are coming to San Francisco for the first time, you probably heard if you went around, you probably heard a lot of people talking
[00:19] only about two things, tech and tech. That's all we have going on in our lives here in San Francisco. So, we're boring, but that's what's going on with us. Anyway,
[00:31] let's get started with our session. If you are building agents, then you know the most common problems you get challenged with are not coming from what model you're using,
[00:44] not coming from the prompt you're using, but it's actually the context. Where is context coming from? The better the data that is fed into the context, the better the model, the better the
[00:57] outputs, the better the agent, right? So, garbage in, garbage out thing. That's precisely what I want to emphasize talking about today. The challenges that we at Bright Data are seeing, accessing that data on the
[01:11] web, how Bright Data and Databricks solves this problem together, and then towards the end I'll show you a live demo of a competitive intelligence agent that we built entirely on Databricks leveraging
[01:26] Bright Data's APIs. Um, so before we dive deep into the agent, let's zoom out a little bit and I want to talk about
[01:39] where the internet is right now. The internet as we know at the moment is consuming worldwide web data
[01:51] by both humans and AI, right? And Bright Data sits right in the middle as that discovery layer helping you discover the web data that's pulling from the web.
[02:05] Unlocking that data, collecting that data into a structured format, and sometimes even caching that data so you can train your models. Right? That's where Brightdata sits. Brightdata, by the way, is not a
[02:18] startup. We've been around for 14 years. We serve over 20,000 plus customers. And 70% of the AI labs worldwide leverage our data.
[02:30] Our moat, essentially, is um getting agents and getting all businesses reliable data at scale,
[02:42] also compliant and ethically sourced from the World Wide Web. So, what we focus on is getting you public web data. We don't cover data that is behind login
[02:55] walls and behind paywalls. So, everything a human can see, an agent can see. That is our moat.
[03:07] Before we get to talking about the challenges, let's talk about scale. At Brightdata, our scale comes from all of the the infrastructure that we've built over the last decade. And we collect over 90 petabytes of data
[03:21] serving all the labs I was talking about. And a fascinating number which I have recently found out is we process and serve over serve three times over what Google processes today.
[03:35] That's what we do at Brightdata. I'm the head of solutions architecture or the pre-sales group at Brightdata. And I have the privilege of actually talking to a number of customers and prospects who come up with various AI
[03:48] data challenges. Accessing that data at scale seems to be the most common problem everyone is solving in that AI life cycle. So, talking about AI life cycle, we put
[04:00] it in four quadrants, right? Data problem can exist in any of these four quadrants. So, when we start training our models or our agents, we need scale. We need
[04:14] labeled data. We need compliant data. When we start building the knowledge base, think you're an e-commerce agent. You'll need a knowledge base of e-commerce data fed into your agents,
[04:26] which means you need domain-specific knowledge. When we start um needing inference or rag layer built on top of it, then we need to access real-time web data.
[04:38] Otherwise, your agents are sitting on stale data, and we'll talk more about it in just a little bit. Now, once we start building an agentic layer on top of it, we'll need to evaluate all the blocks,
[04:51] get the structured inputs and outputs fed into the agentic layer. So, those pieces seem to be the most common challenges our customers are coming and our partners are coming with, and
[05:04] if we double-click into that, think about your agents, right? Your agents are accessing web data. The most common problems your agents will start facing and your engineers
[05:16] will start facing is they're either seeing 403s, forbidden pages. They're hitting rate limits, 429s. Or they're just the agents are getting
[05:28] fed fake information, what we call the AI labyrinth, right? When one or all of these start happening to the agents, what happens is agents will are just consuming stale data or
[05:40] just no data at all. And what happens when that hap- when um agents start consuming that fake data or not being able to access the right amount of data on the on production, right? It hallucinates, your context
[05:54] goes into chaos or confusion, right? Those are the precise challenges or the technical challenges Bright Data focuses on solving, right? We uh we have our infrastructure
[06:07] evading those blocks, trying to process the browser fingerprint and giving you geo-specific content, and bypassing all of the need for fake information by giving you access to the
[06:20] real-time data. This is together with Bright um together Bright Data and Databricks try to solve that very problem. What we're trying to
[06:32] do is a one platform giving you access into how to acquire data, and then how to leverage all the tool sets that Databricks is providing you under one platform
[06:44] to be able to leverage and process that data. What I call it is it's a composition that changes the agent economics.
[06:56] Here's how I see the synergy work. If Bright Data is really the eye, which is our vision to what is out in the public web data, Databricks becomes the memory and the storage. So, the reasoning and the storage behind it. Reasoning and the
[07:10] memory behind it. When these platforms start talking together, the web really becomes an extension of your lakehouse. I'll get more deeper into this.
[07:23] This is the architecture we've used for the agent, but before I talk about this architecture, let's talk about the agent we actually built on Databricks. It's the competitive pricing intelligence agent.
[07:36] What it at a very high level what it does is it's an e-commerce agent, goes out to the web, looks at competitive prices, looks at the social signals, gets some social sentiment on that product, and then once it gets all of
[07:49] that content, dumps it into the Unity catalog and all the tables, and then we have um agents running on top of the historical data sitting in the Unity catalog to build insights and analysis
[08:01] for you. And then we have the um monitoring agent that uses Lake Flow jobs to trigger monitoring price monitoring on top of it. This is just a simple agent that we built on
[08:14] Databricks, but the potential is huge. You'll know what I'm talking about as we go through the um workflows. Now, we're using LangGraph supervisor pattern. Most of you are familiar if you're building agents with this
[08:26] pattern. Um this There The four agents you're seeing on the screen here are the four specialist agents. One, the data acquisition, goes out to the web, fetches the content.
[08:38] The reviews agent again goes out to goes out and takes care of all social signals, bringing in social mentions of a product. That's what it does. History agent is um
[08:51] looks at the data on the Unity catalog, on your tables, and every pricing information it gathered so far, and then builds insights and analysis on top of it. The monitoring agent then um
[09:04] uses ML um Sorry, the Lake Flow jobs to build alerts based on the thresholds that are set on the jobs. Right? Those are the four specialist agents.
[09:20] A really cool visual of what the architecture looks like. Right? Um I come from a world where I used to build these flow diagrams in Lucidchart. We don't have to do that anymore. Everything gets built magically. Um
[09:34] So, you'll see on the on the top, it's a basic react agent on the UI. The chat prompt goes into the react agent and then we use the ML flow for
[09:46] invoking the supervisor agent and delegating the tasks into the um specialist agents. The specialist agents are the four agents that we just talked about and they each of them have access to their
[10:00] own tool sets. Like the scraping and the reviews agent uses bright data's um skills and bright data's tool sets. They also have access to the unity catalog tables and lake flow ML jobs
[10:13] history and monitoring um connects to lake flow ML. On the rightmost side, we have the foundation models. As you know, data bricks host their own foundation models.
[10:26] We're using cloud sonnet for this agent, but you'll be able to use other models as you get to it. I think with data bricks hosting the models, we don't have to deal with bringing an external API key plugging
[10:39] into your configuration. So, that's the um best part of using the foundation models underneath um what bright data sorry, what data bricks provides. And then we have lake base, which we're leveraging as a service to be able to
[10:52] plug in all of the chat history that we're um doing on the agent.
[11:04] What I'll show you next is um the meat of today's talk, which is the agent that we built on the uh data bricks and leveraging the bright data API. This is live in the marketplace. You will be able to access it and towards
[11:16] the um very end of our talk, I'll get I'll um I'll show you a QR code. You'll be able to scan it and get to the app. And I'll also give you a little bit specifics on how bright data uh connection works.
[11:28] So, this is a video. Let me play the video cuz it's uh easier to play the video than uh a demo and rely on demo gods these days. Um I'll play the video and I'll walk
[11:41] through some of the things it does. So, you'll see I put in a prompt here to compare some prices and go to social media and fetch some mentions. What it's doing here is it's calling the
[11:56] specialist agents. You'll see it called the reviews specialist. It's running. It went to Reddit. It's looking for the sentiment of the RTX 4090 laptops.
[12:12] It pulled the social mentions in a nice JSON format that will get fed into the React app. And then we'll scroll up in just a couple seconds to see the pro- progress
[12:25] of our scraping agents. You'll see the tools scrape search results. Those are the tools the specialist agent has access to. If I open it up, it's looking for that laptop on Amazon.
[12:38] Gets the output. And then the other result you'll see here is it also looks up. Those are all the tools it has access to, the specialist agent has access to, and they are built on top of
[12:50] Bright Data's APIs. It's comparing the products, nice JSON analysis right there in MD format. And then putting all the result in a JSON. Everything gets fed back into
[13:04] the React app. And look at the nice output. Tabular format. I asked for it to compare two products only. Gives you the
[13:16] price difference. Really nice sentiment pulled from Reddit.
[13:54] Right? So, you'll see here what the agents did is purely going out to the web, fetching live information that your agents need to be able to ground the context, to be able to get
[14:06] the right data at the right time. If the right data at the right time is not fed into the context, then your agent is just sitting on stale data, not so fresh data, and then it does it cannot really behave
[14:19] the way it's supposed to behave. Um and it ha- we call it hallucination, but um the fact behind these agents not getting
[14:31] fresh data is they keep serving on a very old trained model data, which is useless if your agent needs to look at external reality, right? So, I'm going to wrap this up by talking
[14:46] about the agent economics, right? When you combine a data acquisition platform to a data intelligence platform and everything happening in an all-in-one
[14:58] platform, what we do is in the end of the day, we reduce the engineering efforts, we reduce the effort needed to go out, acquire that data, plug that data, so
[15:10] you extract once, you fetch the data once, you put it in the cache, you leverage it for historical analysis. That's how we reduce the cost that is going out of your pockets to acquire the
[15:23] data, process the data, and leverage it for further needs, right? And then the real-time intelligence, all sitting in your data intelligence platform. When you use Lake Flow jobs on
[15:36] top of the data, you're actually triggering just the pricing changes using when the price changes in the market. So, what used to take days
[15:48] for the business to understand pricing changed by the competition, now only takes hours or maybe even minutes because you have the agent running quickly. And then the third piece of it is enterprise agility where you're able
[16:01] to deploy your Databricks native app in one-click model with a YAML file, right? That is how put together you'll be able to impact the unit economics of your
[16:13] agent. These days, as you all know, it's all about reducing costs. We have enough intelligence built into the models, enough tools set in, but the real problem is really reducing our spend.
[16:28] Taking actions like this and using Databricks and Bright Data APIs that leverage that allow you to leverage the right data at the right time allows you to manage your agent economics.
[16:45] That said, I hope you all find this very useful. That's the QR code that takes you right to the marketplace agent. At some point, when you start using the agent, you'll have to sign up with the Bright Data. But with Bright Data, you
[16:57] get 5,000 free tier credits every month that you'll be able to use across um a number of our API calls that'll get get you at least used to um
[17:09] trying to get the real-time web data into your agent. That uh ends our talk today, but I'll be around if anyone wants to dig deeper and have any questions. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.