Duct Tape to Databricks: 3 Years of Brickbot at Data & AI Summit

Summary

Brickbot, the AI conference assistant for Databricks Data and AI Summit, has been completely rebuilt for 2026 on native Databricks infrastructure using Agent Bricks, Vector Search, Databricks Apps, Lakebase, UC Functions as MCP tools, and Lakeflow — replacing all custom infrastructure.
The architecture demonstrates production patterns including Unity Catalog governance for function definitions and permissions, AI Gateway model routing to Claude Haiku with rate limits, and MLflow tracing for local development and production experiment tracking.
The three-year journey from a DuckDB-in-memory prototype on Heroku to a fully native Databricks platform shows how the Databricks Data and AI platform has evolved to support production-grade AI applications without any custom server management.

Duct Tape to Databricks: 3 Years of Brickbot at Data & AI Summit

Brickbot started as a simple chatbot hacked together for DAIS 2024. DuckDB in memory, parquet files in git and a prayer that Heroku wouldn't go down. At DAIS 2025 we improved it with tool calling and direct integration with RainFocus, but it was still mostly cobbled together. Now, for DAIS 2026, we've rebuilt it entirely on Databricks with no custom infrastructure, using Agent Bricks, Vector Search, Databricks Apps, Lakebase, UC Functions as MCP tools and Lakeflow. This talk traces that journey, the lessons learned shipping an AI app to 30,000 conference attendees, and how the Databricks' platform evolved alongside it.
Talk By: Nick Karpov, Developer Relations, Databricks ;

Chapters

FAQs

What is Brickbot and what does it do?

Brickbot is the AI conference assistant for the Databricks Data and AI Summit, deployed to approximately 30,000 attendees to answer questions about sessions, schedules, and speakers. This video traces its three-year evolution from a simple DuckDB and Parquet prototype to a production application built entirely on the Databricks Data and AI platform.

How is the 2026 version of Brickbot different from earlier versions?

The 2026 version was rebuilt from scratch on the Databricks Data and AI platform, replacing all custom infrastructure with native components: Agent Bricks for the agent layer, Vector Search for retrieval, Databricks Apps for hosting, Lakebase for persistent state, UC Functions as MCP tools, and Lakeflow for data ingestion from the RainFocus event management API.

How does AI Gateway work in the Brickbot architecture?

AI Gateway handles model routing in Brickbot, directing queries to Claude Haiku with configured rate limits to manage costs and latency at conference scale. This allows the team to control which model handles which type of query and enforce usage policies without building custom routing logic into the application.

What governance model does Brickbot use through Unity Catalog?

Brickbot defines its agent tools as UC Functions in Unity Catalog and manages access through Unity Catalog's native permission model. This means the team controls exactly which data sources and capabilities the agent can access without maintaining separate access control systems outside the Databricks Data and AI platform.

Full transcript

[00:08] Uh my name's Nick Karpov. I am on the developer relations team at Databricks. Um and I am one of the people responsible for Brickbot this year. Um when we originally kind of envisioned this talk, we thought we would go through all the previous years and what
[00:20] we did uh and then what it looks like today as a comparison, but the truth is what we did in previous years kind of doesn't really matter anymore uh since the way we built this this year is completely different. If you're if it's
[00:33] not working, you speak to speak to somebody else. Sorry. I'm not in control. >>
[00:45] actually just going to show you Brickbot live, completely live, and what it looks like in the Databricks environment uh so that you can see for yourselves uh how it works uh end-to-end from uh basically from the ingestion path all the way to
[00:58] the point where you're serving. Now, I will be showing kind of live data. So, if you want and you have some questions, uh you can like uh use Brickbot do like 1 2 3 uh and I'll I can filter by 1 2 3 to
[01:11] make sure I see your message. And then maybe if you want to communicate or ask a question, go for it. Um all right. So, this is Brickbot. It's the exact same thing that you guys see on your phones. Um I'm going to start by actually just
[01:23] saying how does Brickbot work and watching this trace come through. Now, as you can see, we do a little tool a little tool call searching sessions. We get 10 results and I'm just going to use this as the foundation uh for the
[01:35] talk. So, the entire talk track is basically here. Um now, as you can see, the foundation begins with ingestion into our lakehouse and what that actually looks like, I'm going to show you now by coming over
[01:47] here. You guys can It's a good enough size? Okay, awesome. Um So, in order to ingest data, we actually use uh our partner RainFocus. So,
[02:00] RainFocus is an event management software and they do all the all the talk submissions come through RainFocus. All the registrations, everything is done through through Focus. And they've exposed an API for us and that API has a
[02:12] lot of different things. But, one of the things is sessions, right? So, unfortunately, we don't get live data from them. We can only pull it every 15 minutes. And you'll see how that works. But,
[02:24] essentially, this starts with an API key. And what this used to be in Databricks is an API key that was just hardcoded or like in a config file in some single Python server on Heroku. But, this year
[02:38] what we did is we integrated the whole thing into Databricks. So, it starts with the connection string from RainFocus coming directly into Unity Catalog. So, this is a view in Unity Catalog. And if I go over here to the Brickbot schema and do settings and
[02:52] connections, and I'm going to filter by Brickbot or RainFocus, you can see that I've got my connection. This is the exact connection string that comes from that RainFocus page. So,
[03:04] we're actually governing the connection string right from the root directly in Unity Catalog. So, if anybody else needs permissions to this to be able to use this connection string, I can actually just go to permissions and open it up
[03:16] for them. So, I happen to be the owner. So, of course, I can I can use it as I wish. And we have an app that is allowed to use this connection as well. So, this is the beginning. But, this is only the connection string.
[03:28] From here, we actually need to go and do something with it and actually get the data. So, where the data actually comes from is also registered in Unity Catalog. So, I come back to catalog catalog, go to Brickbot, and you can see
[03:41] here RainFocus raw. These are the tables that are created by a job. And the way that they're created is through these functions right here. Get exhibitors, get sessions, and get speakers.
[03:53] And if I click into one of them, for instance, get sessions, you can see that the description of this function, right, is literally just the HTTP request. I'm selecting that I want that rain focus connection, so that's that same connection that's defined in the Unity
[04:06] Catalog. And then I have a specific URI sub URI to get the sessions, and then some parameters that come with that. So, I'm just getting as many sessions as I'm allowed to per the API spec. Now, this is kind of the second layer,
[04:19] right? So, we define the connection, then we have some Unity Catalog permissions on that. Anybody who is allowed to can use that connection. And then we have this function definition as well. Get sessions, get exhibitors, get speakers, which also has
[04:31] its own set of permissions. All right, so at each stage we can gate the access to this API. Now, what does get exhibitors, get sessions, get speakers do? Or rather, where is it used? It's used in a very
[04:44] simple pipeline uh that we run, like I said, every 15 minutes. So, if you've used Brick Bot, you saw like a discrepancy between the timings between like your schedule and or the session availability, um the reason for that is unfortunately
[04:57] our upstream data. I'm only allowed to pull it every 15 minutes, so, you know, too bad for me. So, we have a simple job that runs, and this job has a few tasks, and hopefully this is going to be visible.
[05:10] Uh I'll do my best to zoom in here. Um we do essentially those three calls, right, as parallel tasks, and this is the job definition in Databricks. Um at the bronze layer, we're just calling the
[05:22] API. So, if I do bronze session, and then I click into the SQL file, you can see that I'm actually doing that select get session. So, that is that
[05:35] HTTP call that was defined in the Unity Catalog directly. So, you can see this is the catalog, the schema, and the function call. Um it has those default parameters, so I don't need to give it anything else. And then I'm getting this
[05:48] We We pulled it already, so we know the schema, and then we prescribe the schema. We don't just do it on the fly, of course. And we get all this data, and it comes in, and it's stored directly in Delta, which is also
[06:01] governed by Unity Catalog. And so, if I come back to this catalog, come over here, view Rainfocus, Rob. We have our sessions table here, and that's that the SQL definition that you just saw writes this table. Now, this table
[06:15] has a lot more information in it than we can expose. So, like I said, Rainfocus is the lifeblood of the whole thing. So, it has like sessions that were not accepted, private said like it has all
[06:27] this information in it that we shouldn't expose. So, to filter that through, we have the second stage of the pipeline. And that second stage of the pipeline that is again defined over here,
[06:39] writes to a silver table. And that once again is back in the catalog, coming back here, written out to Rainfocus schema. And the schema, you can see
[06:52] there's these three tables. These are I call them our silver tables cuz it's only the second transformation, but it's You can think of it as the gold table. It's the final tables that we use. And you'll notice that there are index tables here as well, or index
[07:05] entries here. So, what are these? We don't actually, of course, query these tables directly like willy-nilly with a select star. What we do is we actually create an
[07:17] index so that we can do a vector search and a BM25 information metrics search. And so, that the way that we do that directly in here, I'm going to go to compute, AI search, and you can see that I've
[07:29] already created this Brickbot endpoint. And in this endpoint, you can see our three indices, right? And each one of these indices, for instance, if I grab again the sessions index, you can see that it's continuously being
[07:41] synced. So, every time that we get new data, right, the index is being updated and synchronized. And so, when we actually do the query, we just we get the most recent data. And the like I mentioned
[07:54] actually in previous years, we've had vector search for a while, but we only supported actual vector search. So, like a cosine similarity between your input and whatever is in the vector search. But this year we've by now we've
[08:07] implemented a hybrid version which does BM25 and the vector search. And so, we've been able to leverage this directly. What we found is that vector search is very good for semantic stuff, but if you actually want like
[08:19] a more complicated query that's not just like matching like that, BM25 does a lot better. So, kind of the old school way still prevail. So, that's kind of that runs through the
[08:31] foundation, right? You saw the tables that we have. The governance as you saw everything that we just did is in Unity Catalog. And this is this is live. So, this this is exactly as you can see, we the last commit that was
[08:44] processed was just 9 minutes ago. Um and all the governance run it through Unity Catalog. For the search, we're using a hybrid approach BM25 and the vector search. And that's kind of
[08:57] base case, right? But if you wanted to do a more complicated query like for instance right now it's 2:00 and I say, "How many sessions are in between 1:00 and 3:00 p.m. today?"
[09:11] Right? So, this is a type of query that can't easily be or at all really be solved by just a simple vector search or or a select query that filters everything through. So, what we do is we
[09:23] actually leverage a Genie space in the background. So, we have a tool called that goes and calls Genie, right? And I'll I'll show you what it looks like in Genie in a second. And we leverage Genie's power to
[09:35] actually translate this into a reasonable SQL query that queries those same tables that you saw. All right, so those sessions tables are plugged into the Genie space and that Genie space can then return you something more complex. So, you can see
[09:49] that actually wrote a pretty decent query. Today is Wednesday. It was scheduled true and the start time, these are in UTC, so between 1 Actually no, this is PST, but between 24 hours. So, between
[10:02] 1 and 3 o'clock. And where this Genie space lives is also, again, in here. So, if I go into the Genie space and you'll see DAS 2026 speaker sessions and exhibitors. And this is the Genie space
[10:14] that actually powers that that same question. All right, so you can see in this Genie space that we've added some more general instructions to help it out with dealing with timestamps, different kinds of
[10:26] filtering. These are just things that we discover as we go. All right, people are asking questions and sometimes the results are wrong or not good in our testing. So, we continuously updating these prompts to make sure that we capture more and more
[10:38] edge cases. Um in addition to this, we actually also have metric views. So, some of the things that are being asked like counts and things like this, there's no use in going and
[10:51] running, you know, select count star from this table over and over and over again. And unless you add some more information here, that's what the Genie space is going to do. And so, for that we have something called metric views.
[11:03] Um and those metric views allow us to Here, I'm going to go to edit and actually let's go back to the catalog. Let's go here. Let's go to
[11:16] Where are our metric views? You'll give me a second. Metrics. Exhibitors and sessions. Let's do sessions. So, you can see that also governed by Unity catalog, I have a
[11:29] schema called metrics. All right, probably not the best name. It's these metric views definitions, and they themselves are also defined in SQL. Uh I won't go into it cuz it's it's going to take too long, but essentially it's a set of pre-baked measures, which
[11:42] are constantly refreshed for us uh every time the data changes, so that throughout uh throughout the queries we're not just querying these tables again and again stupidly regenerating the same metrics over and over and over again. So, it's actually backed by a
[11:54] materialized view, and all this happens kind of automatically under the hood. All we have to go is to Unity Catalog, creating a metric view, and defining that metric view here. So, that's the metric views, and they
[12:07] also help power everything. So, if you've asked how many sessions are today, all it's going to do is just return this measure, which was most recently calculated on the order of every 15 minutes, since that's how often we get the data.
[12:26] Okay. So, that's for the analytical queries, right? So, we've got we have the basic search stuff going on. Uh vector search BM25. We leverage Genie for a more complicated queries, um and that's kind of that's basically just a freebie, otherwise we would have to
[12:38] write our own natural language to SQL layer. Um and that kind of brings us to the uh final piece, which is the AI Gateway, which actually um uh instruments the LLM calls uh and
[12:52] gives us all our metrics. So, if I come here back to Databricks and go to the AI Gateway, and I will search for Haiku. This is the model that actually if you've asked, I wonder if you've been able to get it to admit what model it
[13:06] is. Um this is the model running live, and this is our kind of data. Let's do the last 30 minutes. You can see here that the queries per minute that we're getting, we can see the tokens per
[13:18] minute that we're processing, and it gets kind of high. You can actually see um it's uh to the order of like 600,000 tokens per minute, which is pretty solid. Um and so you can see all the metrics
[13:30] here. Um end-to-end latency, all this kind of stuff. You can actually I'll show you something cool. This is uh just in the last couple of days what we ran into. You can actually see a point where the errors spiked. And the reason this
[13:43] happened, the reason this error count happened is because we actually got way more users than we thought. So, we had to go back to our teams and say provision us more uh capacity. The other thing you can see here is that uh for instance, this is a bug that we caught
[13:55] at the last second is we noticed as our usage started growing, you can see this blue line. Actually, I'll zoom in here. Oop.
[14:07] You can see this blue line. This is when the usage started picking up. This is Sunday when everybody started to arrive. Um there is nothing green here. The green is the number of cash tokens. All right, so this is kind of a very big gotcha in the inference world. Um I When I
[14:19] originally looked at this graph, I thought, "Oh, we must, you know, there must be a bug. Of course, we're caching. There's no way that it's not caching anything." Um and it, you know, maybe it's just not rendering. So, I kind of ignored it. Uh it turned out that it wasn't true. And
[14:32] it turned out that what we were doing is we were embedding a custom parameter into our prompts very early on. And uh if you're familiar, what that means is that uh every time that prompt uh runs
[14:45] against the inference endpoint, the values are different. So, you don't get to leverage any cash. This is very stupid rookie mistake. And um I'll actually show you where that's logged. So, in our experiments, we actually trace
[14:58] uh everything that we see. I'm actually I'll have to use the filtered version because I don't want to show everybody's queries live. Um so, these are my queries just in the past. So, you can see I called how Brick Bot works, how many sessions are in between
[15:10] one and three. I come in here. I can see the entire history of this. Um I can see the token counts. I can see the caches. I can see that it's part of a session. So, if I click on the session, you can see that it was two turns. And like I
[15:23] said, I was mentioning the prompts and in the prompts I can come over here and you'll notice that we have three prompts registered here directly. Right? So, none of our prompts live in the application itself. Right? So, like
[15:35] basically everything like the applications that you saw, the interface is just a thin shell. Right? It all it does is call Data Bricks. So, you could like swap out the endpoints and then, you know, it would be the UI for some
[15:47] other chat app. Um and so, the prompts are registered directly also into Data Bricks. So, you can see here for instance our off prompt goes through a lot of iterations. You can see as soon as June 11th, June 14th, June 15th, the
[16:01] conference begins and we're still updating it in real time as we discover that, you know, maybe something's showing up wrong, maybe the model's getting confused and we need to hint and prompt it in a different direction. And essentially, what we were doing is at
[16:14] some point our timestamp for the current time was literally like the second line. Right? So, we were like just hammering our endpoints for no reason. So, we came in here and then we actually just split it up and we said, "Hey, put that at the
[16:27] very end of the prompt so that everybody can actually leverage every time that this is called, we can see back in the AI gateway." I'm just going to show you one more time. This is me coming back. I'm selecting
[16:39] the Haiku model that powers everything. Go to metrics. Do a little bit back, last 2 days. And oh, let's do I'm just going to scroll down and you
[16:52] can see that window where we suddenly went from zero cash tokens and a lot of input and then finally our cash tokens picked up and now we can uh first of all, it's like 10 times cheaper and it's also much faster
[17:04] and we weren't running into more rate limits. So, our internal teams are very thankful that we patched this really dumb mistake. Um so, that's the AI gateway Um, um, that
[17:17] basically monitors everything. The experiment page, which looks at the traces. You can see we have a local experiment, so that's when we're iterating locally. We blog everything to local. Um, in production, we send everything here.
[17:29] Um, and that's where this appears. And, um, essentially, I mean, that's all I really have time for. If you have questions, please after the talk, I'm happy to talk about anything or show you this more live if you if you want to see. Um, in terms of what we used to
[17:42] develop it, um, we used Claude mostly to develop it. Um, we could have used Genie code, but unfortunately that last little layer, which is that app, right? The right now Databricks apps is extremely governed. So, you can't just spin up an
[17:55] app and give it to the world, right? Otherwise, it would be a leak of, you know, we can't control what anyone's going to write in that app. So, it's unfortunately I can't host it on a Databricks app directly. So, that last thin layer lives on Vercel.
[18:07] Um, and, um, and that's that's about it. So, thank you guys for joining. I hope this was kind of a a good brief interlude for whatever you're doing next. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.