Bridging Transactional Speed and Analytical Depth with AI/BI, Genies Lakebase & Databricks Apps

Summary

Databricks' own GTM team rebuilt their analytics platform—GTM Hub 2.0—on the Databricks Data and AI platform, evolving from a basic Gradio prototype to an enterprise-grade React/FastAPI application serving over 6,000 monthly active users projected to exceed one million sessions.
The architecture combines AI/BI dashboards, a supervisor agent built with MLflow that routes natural language queries across specialized Genie Spaces, and Lakebase for managing user preferences, session history, and real-time personalization.
The platform delivers sub-3-second load times for 10,000-plus concurrent users with role-based access control, multi-persona views, and seamless cross-origin embedding.

Bridging Transactional Speed and Analytical Depth with AI/BI, Genies Lakebase & Databricks Apps

Discover how Databricks transformed a basic Gradio app into GTM Hub 2.0, an enterprise-grade platform serving 10,000+ global users (Sales, Marketing, XDR) with personalized insights. Core Architecture & Achievements: Databricks Apps (Enterprise Scale): Built a React/FastAPI application serving 10k+ concurrent users with less than 3s load times. Features include RBAC, multi-persona views, and seamless cross-origin embedding. AI/BI Dashboards: Embedded dynamic dashboards with global filters, bookmarkable states, and usage tracking for optimized analytics. Genie (Conversational AI): Integrated multi-agent support. A supervisor agent (via MLFlow) routes natural language queries across specialized spaces, eliminating the need for SQL. Lakebase (State Management): Leveraged PostgreSQL on Databricks to handle user preferences, session history, and caching for real-time personalization. The Result: A unified, modular analytics experience with robust telemetry and a future-proof architecture.
Talk By: Abhinav Bhatnagar, Sr. Manager, GTM Analytics Engineering, Databricks ; Wenwen Xie, GTM Staff Software Engineer, Databricks ;

Chapters

FAQs

How did Databricks build GTM Hub 2.0?

The team evolved from a Gradio prototype to a production React/FastAPI application built on the Databricks Data and AI platform, incorporating AI/BI dashboards, a multi-agent Genie architecture, and Lakebase for state management. The result serves over 6,000 monthly active users with sub-3-second load times.

What role does Lakebase play in the GTM Hub architecture?

Lakebase, a managed PostgreSQL service on Databricks, serves as the state management layer for user preferences, session history, and caching that powers real-time personalization. This eliminates the need for a separate operational database outside the lakehouse.

How does the supervisor agent work in GTM Hub?

A supervisor agent built with MLflow routes natural language queries from users to specialized Genie Spaces based on the intent of the question. This eliminates the need for users to write SQL while ensuring queries go to the most relevant data domain.

What business problem did the GTM analytics team solve?

Internal surveys consistently showed that employees were wasting significant time on data tasks, described as a productivity crisis. GTM Hub 2.0 unified fragmented dashboards and tools into a single, modular analytics experience with self-service AI-powered querying.

Full transcript

[00:09] Hi and welcome everyone. Um this is Avanov and Vivian and today they're going to be talking about how they've built a really cool application that is powering go to market at data bricks. >> Right.
[00:31] All right. Um last year a few of you saw us here and were we were here to talk about a vision a few AIB dashboards a couple of genies and a bet and this year what we're here
[00:46] to talk about is what does that look like in production and how is it catering to 6,000 monthly active users and is projected to have over a million sessions. S
[01:00] that's what a year of hard work, a solid data foundation and an amazing product can do. Datab bricks.
[01:14] So today I'm going to walk you through a journey of how we became customer zero at data bricks and how we leverage lakebase
[01:27] AIB genie and wrapped it in a blanket of databicks apps and served it to GTM and helped it to transform into something much bigger.
[01:40] I am a Batnaga and I am joined here by uh Vivian. Uh I look after go to market analytics and Vivven why don't you introduce
[01:52] yourself? >> Yeah. Hi everyone. My name is Vivian. I'm GTM staff software engineer. So I'm the main engineer behind the GTM application and I'll show you later a deep dive into the technical stack
[02:06] behind the architecture. Awesome. Right. Like our founder Ali, I'm also a big fan of chapters. So, uh today we are going to be talking more about previous slide
[02:20] please. Yeah, four chapters. The first chapter that's where we'll be talking more about where we started. The second chapter it's going to be much more of deep dive, the evolution, how we
[02:34] became what we are. The third one is going to be more about some numbers, some reliability at scale. What does it look like? How are you going to be managing all of it? And last one is going to be a more like a fast forward
[02:46] to today. What we are looking at, what we are delivering, right? With that in mind, let's move to chapter one where we started. So those who joined us last year,
[02:59] well, you'll have to live with this. uh for those who didn't let me give you a distilled version of the problem statement that we were running into
[03:12] for over the years the cultural surveys every time we did them one theme dominated across we are wasting so much time and when an L7 essay four-year tenure database
[03:26] employees which is like top 1% of our talent says that that's not a complaint, that's a productivity crisis and we needed to fix that.
[03:38] So we started looking more into what's actually causing that. Next slide please. And when we look deeper, we figured there are different different systems
[03:50] and there are different teams responsible for managing those. There are consumption forecasts, you know, we are trying to focus how we're going to be doing business. There is uh t target
[04:02] targets and quotas being set by another team. There is segmentation. There's new spenders. Finance had their own forecast. Data science had another. And let's not forget the comets because commits are important, right?
[04:16] So then we also started looking more into you know what what's the underlying uh work that has been done and data bricks
[04:29] has incredible work culture and builder culture and everybody can build because in go to market everybody's technical enough to build a product for themselves and that's what happened over 1450 agents
[04:43] were built and over 70,000 dashboards that were existing in the system on top of different tools, different solutions that we have. Let me repeat that number.
[04:56] 20,000 dashboards. That's a lot. So the learning here is is simple but painful. Next, the complexity is free and it's
[05:11] our job with our job is which is basically to deliver authenticity, authenticity, clarity and simplicity.
[05:27] When we tried to look more at the state of the GTM, how it exists, where we are, we focus more on three major pillars. We found the data was too fragmented. There were signals disconnected with each
[05:40] other. We had disconnected workflows, different systems working in parallel, not talking to each other. And lastly, the limited AI productivity. It was impossible to
[05:53] deploy an AI on top of the legacy SAS system, right? So we started thinking more from our
[06:05] first principles and we thought more about what if if we can use our own data and AI platform to solve this.
[06:18] If you are thinking about this at this point of time I'm going to give you the answer of it in the next chapter which is the evolution.
[06:30] In the evolution, we're going to be talking more about what we came towards, which is after looking at the problems, we figured we need to think more about in terms of solutions. So, we came up to three northstar metrics. The first one
[06:43] was we need to increase the field productivity. We need to give the sellers more time with our customers. Whatever they are spending, you know, uh more internally that we need to go go
[06:57] in. We need to set we on second uh statement that we had is like we need to scale this for all the teams whether it's our sales whether it's our uh marketing whether it's our field
[07:09] engineering whether it's partners and we need to scale it for all of them and the third one was that we needed to bring a single source of truth to consolidate our dashboards and you know
[07:23] AI into one place and make our data sets AI ready so that our sales sellers they don't need to swivel across different different tools to run their business.
[07:44] So the modern GTM stack runs on data and AI platform our own data and AI platform to unify data analytics and AI and in the later slides I will talk to you more about how we are making it happen.
[07:58] So we came up to the vision statement like what do we want to do right what do we want to do the first one was we needed to unify our data governance we needed to build single source of truth
[08:10] that is answering to all the questions on the business the second vision that we had that we are going to build an AI that understands your business not generic
[08:23] one you're going to be building that understands your consumption your pipeline, your use cases and your opportunities. And the third vision that we had is what we are going to build, it's going to be
[08:37] agentic so that we can automate the repetitive tasks that our sellers are putting their time into. And when we looked at all the vision and we saw our data and AI platform was the right choice.
[08:50] Next please. So while we talk about that those phases of our vision right the first phase was unified governance and we realized that
[09:02] we need to invest and prioritize in our solid data foundation. So we built a a typical medallion architecture lakehouse which worked for
[09:14] us but we also built some reporting views which were optimized for AIBI speeds so AIB dashboards can load faster than ever. We also invested in semantic
[09:29] layer or metric views that became the point of entry for our genies. they had a lot more of business context in it like synonyms, metadata, lineage, everything so that AI can understand
[09:43] what it's looking for. Now let's look at what phase one helped us in delivering. We ended up with 400 key tables across
[09:55] our bronze, silver and gold layer which sort of paved the way for our enterprise lakehouse. We had every column, every lineage track. We looked at who's looking at
[10:08] what, what metrics being powered by what. And we are track, we were tracking almost everything. And I cannot overstate this enough. Whatever data foundation that you are building, you
[10:22] need to make sure that it's trusted because if it's not trusted, it's not going to get adopted.
[10:35] what our phase one did that actually paved the way for our phase two which is you know uh we started looking at what we should be doing next with our dashboards. So we started focusing on
[10:47] getting other dashboarding tooling out of our ecosystem. We started looking at AIBI dashboards as the centerpiece and we started migrating 1,300 dashboards to
[11:00] AIBI and then there was a lot of duplicity in there as well. So what we ended up with is 38 key dashboards catering to different different gotom market
[11:12] personas and apps consolidated in one view and that's what gave birth to GTM hub.
[11:24] The screenshot that you see on your right, right, that's that's the first screenshot that we had of of you know a technical architecture. What does it look like? It was simple gradu app which is sort of connecting with unity catalog and just
[11:38] trying to display a configdriven dashboards and this this this simple experience this simple app sort of ended up giving 74% faster time to insights and yielded
[11:52] 80% in users you know higher user satisfaction and that seems to be you know itself is powerful right because you are catering to all the spec specific personas and you are getting the similar response
[12:05] from everybody that this is what they needed. This is keeping them focused. This is getting them what they need at the end of the day and they can pivot to their customers because that's where
[12:17] they should be investing more time. Next please. As part of the third phase where we are sort of you know flourishing more upon
[12:29] at this point of time agentic workflows we want to increase our AI foothold right so that's where we started investing a lot more in genies which is you know AIBI domains uh genie spaces we
[12:43] also built a lot of custom agents as well sales AI which is sort of an intelligent workspace for account teams they come in together and get pull lot of information there and run agents on
[12:57] your behalf to get information. The third one where we invested is a lot more of custom agents uh on top of agent bricks where we built a bunch of routing agents, supervisor agents and knowledge
[13:09] assistants. And last pillar where we invested heavily in is the GTM alerting because we want to make sure that the our sellers are getting notified just in time. So we're building the intra to
[13:22] support all the sellers to get information about whatever they want.
[13:34] And let me put it this way, the future of GTM is built on data bricks. And this is where I'll pass the ball to Vivian to walk us through the technical nitty-g gritties and you know how we built this
[13:46] amazing cool magic product. Right over to you Vivian. All right. Um up to this point we have discussed the um business visions and
[13:59] the transformation and now let me actually show you what an actual experience looks like for an end user. So this is the homepage of our GTM hub application. Uh you can think of it as
[14:12] the entry point to our entire GTM analytics ecosystem. Um, so one of the biggest goals we want to achieve was reducing the the friction
[14:25] user experience when they're looking for information. Um, so instead of, you know, remember uh a dozens of links and navigate multiple systems and spend time figuring out
[14:39] where information lived, everything now starts from here. At the center is the universal search experience where users can discover dashboards and tools and
[14:52] applications and AI experiences and on the left side you can see the navigation is organized into business workflows instead of technical system. You can see
[15:05] like customer 360. You can open it. You can expand it and then forecast cadence and consumption pipeline review cola reporting. And these are the uh reports or
[15:19] dashboard that are being used by our sales leaders day by day. And now you can also see that it gives you a now user don't need to know where data lives
[15:32] or who create the dashboard. They simply navigate based on the questions they're trying to answer. You can also see this page gives you the personalized content such as saved
[15:46] dashboards, used links, you know, like views and announcements and uh frequently used resources. And each experience is uniquely catered to the
[16:02] user based on their role while still running on the same govern data foundation. >>
[16:16] like. Now, I'm going I'm navigating to forecast cadence and boom, I get my dashboard here. And this is just an example. This is not real. This is all mock data. Um, this is one of the operational dashboards used by our sales
[16:30] leader to drive day-to-day decision making. And there are couple of important things to notice here. So first we create a global filter that
[16:42] provides consistent business context to the end users. So what you can see here if a sales leader selects a business region or they select a quarter you know
[16:55] these contacts follow them everywhere. So even if they open a dashboard that is under maybe another category, it's about consumption, it's about pipeline review,
[17:07] they still operate in the same business context. And the second is all of our dashboards are uh built from our governed semantic layer. That means each user is accessing
[17:22] the same definition of business definition and source of truth. And uh finally you can see the power of combining the analytical workflows and
[17:37] application experience. The user right now can explore data drill into details save views and the launch genie experience interacted with the platform
[17:49] without leaving the workflow. And of course this is all wonderful but we want to uh create we want to have this analytics creation not just limit
[18:02] to the one central team and here we introduce community. We want to be able to channel the builder culture to allow our users to
[18:14] also build the solutions that cater to their unique business requirements. And this is probably one of my favorite parts about this platform because instead of each team building uh the
[18:28] same solution over and over again, we give them like a reusable assets that become the building block across the organization. So now each team can come
[18:40] here and then provide the solutions they have built and then other team can discover, adopt and reuse the solutions. That's how that's how to help us to
[18:52] drive innovation quickly while maintaining the governance because everything is running on data bricks unity catalog. So sharing these assets
[19:05] doesn't come at the cost of security or trust. >> Absolutely. >> All right, let's done with the demo and then go back to the presentation. So let me bring that up real quick.
[19:19] right here. Bring this back. Oh, I don't I need to navigate better with my mouse. Right here. >> It's a big screen >> right >> here because we don't have a mouse. So, I just drag it over here.
[19:32] >> All right, perfect. >> All right, that's it. Um, >> so we have seen this demo. So, uh, now that we have seen what the actual experience looks like,
[19:44] >> you want you want to go there? >> Yeah. >> Okay. Yeah. Perfect. The next natural question is how did we actually build it? Let's go under the hood and then see the architecture behind the GTM hub.
[19:58] What you're looking at right now is the GTM hub platform architecture. One of the important design decisions we followed was simplicity. Instead of
[20:10] having user remember dozens of dashboard links and tools and agent experience, we provide them one single destination. And you can see that this architecture
[20:22] is organized into five layers. Presentation, application spine, intelligence, data and state and governance. Every request flows through these layers and then returns back
[20:36] following the same governed path. That means there is no separate AI application platform. There is no separate governance platform. There is no application platform. Everything runs
[20:49] on data bricks. All right. Now let's take a look at the top of the stack which is the experience that user interacts with every day. So GTM hub serve as the front door for
[21:03] the entire GTM organization. the front end experience. We intentionally keep it lightweight. It's a React application that embeds AIB
[21:16] dashboards, tools, and agent into one single interface. A few design decisions prove especially valuable. For example, the first one is
[21:28] embedded dashboards allow teams to deploy analytics into the hub without building custom front-end experiences. Second, the global filters create a consistent business context across
[21:40] dashboards. So if a sales leader select a region or business unit once, that context follows them everywhere. Third, the personalization ensures that each
[21:52] user only access to the content that's relevant to their role. That means they don't access to the data they don't have permission to. So for each user this feels like one
[22:06] application but actually behind the scenes it's dozens of application dozens of dashboards and services are running together. All right the front end gives
[22:18] user a unified experience but behind that sits the orchestration layers that makes everything work. This is layer is what we call the
[22:30] nervous system for the GTM hub application. Every single request enters through the single fast API service and the fast API service handles authentication, routing,
[22:45] agent orchestration and user session management. One important architectural decision that we had was adopting asynchronized job execution because you know many AI
[22:58] workloads can take more than a few seconds to complete. So instead of block the users while they wait uh we turn their requests into jobs that
[23:10] can be monitored uh retried and scaled independently. We also expose capabilities through MCP servers allowing agent to interact with the platform services
[23:24] in a single interface. So we also um so the result is the uh backend service that connects the front end gen model serving lakebased and SQL
[23:37] warehouses. Of course application needs more than analytics. They also need a fast transactional state and this is where lickbase enters the picture. One of the challenges we encountered was um
[23:52] analytical system and transactional system have very different requirements. Analytical workloads are optimized for uh large scale reporting and aggregation
[24:04] while application needs low latency read and write. >> That's right. >> That's where lakebase fills the gap. Yeah. So we use lickbase to store user sessions, preferences, bookmarks, chat
[24:18] history, agent memory and operational telemetry in in other words everything that makes the application responsive and personalized.
[24:30] So meanwhile Delta continue to handling the analytical workload. Rather than forcing one system to do both jobs, we let each system do what it does best.
[24:42] Lakebase provides the transactional speed. Delta provides analytical depth. Together, they create a complete architecture for the application.
[24:54] Now we've seen the application foundation. Let's talk about the intelligence layer that build on top of it. This is where Genie and agent bricks come together. When the user asks a question, we don't
[25:08] just send this request to a generic agent, right? This request first goes through a supervised agent. And this agent is able to understand its intent and decides which specialized genie to
[25:22] route to. And we have 10 specialized genie. Each special list is optimized for certain domains such as accounts forecasting
[25:34] consumption pipeline analytics. Once this request routed, Genie translates the natural language into govern SQL against our semantic layer
[25:46] and the SQL is then executed through SQL warehouse. The results are returned and cached in lake base. Finally, the response appears back in the application. So the key lesson we
[26:00] learned here is that specialization matters. We found that the routing accuracy actually had a bigger impact than simply using a bigger model. Okay, let's walk
[26:14] through a uh request. What happens when a user clicks submit? So a user ask a question in the GTM hub from the top from the top down the
[26:28] request reaches fast API and the fast API authenticated user and launches asynchronous job as we mentioned the supervisor agent takes the request and
[26:40] decides which genie specialist should answer and the specialist the genie translate the question into SQL and execute it against deltas through SQL warehouse house. The result is then
[26:54] returned and cached into lakebase and then finally the response appears back in the application. That means what I really love about this architecture is there actually very few parts there are
[27:09] every layer sits on data bricks that means fewer integration to manage fewer uh security boundaries to cross and then fewer operational headaches.
[27:22] If we zoom out, you you can see like all the components combined into this single integrated technology technology stack. And to summarize it from the from the top to
[27:36] the bottom, you have a user submit a request to GTM hub and React deliver the user experience and then fast API routes the request.
[27:48] uh Genie and the uh supervisor agent provides intelligence. Lakebase manages the transaction state the delta manages the analytical workload and everything
[28:00] is governed by unity catalog. Notice what isn't there. There is no external customer identity pro uh customer identity platform, no separate
[28:13] database cluster, no external AI occupation platform, no external like governance layer. Everything runs on data bricks. Um
[28:26] all right. So the architecture provides a foundation. The next step we want to do is evolving the user experience on top of the existing foundation. We didn't build the platform at once. The
[28:40] platform involved do multiple generations. Version one we focuses on access. This is where user could access dashboards, find insights, you know, navigate
[28:53] contents. And version two we focus on introducing the intelligent routing building master agent which takes a user request and then route to the specialist genie. Uh version three we added agent a
[29:08] response generation. So instead of returning just raw data the product start generating like summaries recommendations and executive ready answers.
[29:20] So you can see the the key architectural principle here was additive evolution. We didn't rebuild the platform in every quarter. We layered the intelligence on
[29:32] top of the existing foundation that allows us to innovate quickly without breaking things or without introducing regression or instability. All right, we've talked about building
[29:46] the intelligent application and that is only half the challenge. The other half is ensuring uh it's reliable, ensuring it's reliable
[29:58] at scale. So now I'm going to pass it back to Abina to talk about reliability at scale. >> Thank you, Viv. Um let's look at the
[30:10] numbers because that's why we are here. At this point of time we are catering to seven different personas go to market personas we are catering to starting
[30:22] from our sellers to middle level management to senior leadership and up to CRO. So all of them are coming onto this one-stop shop and getting their
[30:34] data needs fulfilled. Majority of the sessions at this point of time are coming from our IC's and which are like sellers and solution architects who are sort of that's were
[30:46] the key targeted personas because their time is most valuable because they are spending time with you guys.
[30:59] GTM hub itself is growing and it's operational. Within last quarter, we delivered over 166K sessions with 6,000 monthly active users and our Genie is
[31:11] really breaking it out there, right? We are going for 18,000 queries last quarter and I I'll talk more about the projected numbers. But I think the key
[31:24] metric that I want you to focus on is that 89% returning users because that's what it tells about stickiness. How many users are coming back on recurring basis and that's where the traffic is coming from. People are loving hub and
[31:39] using it on day-to-day basis. Next please. And whatever we have built, whatever Vivian and I have built right here, it's it's built for scaling for next couple of
[31:51] years for data bricks. And if you look at from FI26 to FI27, which is like know financial years are somewhat different, right? So if you look at the projected numbers, we are
[32:03] set to have over 1 million sessions. We are set designed to have over three to four million of AIBI views and we are projecting our you know that's where we have wider window of confidence interval
[32:18] or somewhere around 200k genie queries and all of it is going to be powered by the same tool now if I look at it that's like a big scale right somewhere on 4x
[32:30] you see big number 4x 5x and in genie like somewhere on 30x growth that looks like A lot of infrastructure factor that we'll have to do right >> yeah I don't think so this is going to
[32:43] be nothing because datab bricks apps lakebase aibbi genies uni catalog all of it scales
[32:55] nearly automatically so we don't even have to spend anything on the infrastructure refactoring how do we scale x or y's and z's all this infra so no more devops overhead for us.
[33:07] >> Right. >> Right. Right. U so how do we keep it up fast and trusted? And I'm going to be
[33:19] emphasizing upon trusted because remember if it's not trusted it's not going to be adopted. So we have always on access we have database oath and SSO powering from day
[33:32] one. You have unity catalog role level security rolled out making sure our sellers are only focusing on their data and nothing else on the business. We have the speed at scale out there which
[33:44] is you know we have our uh loading time as a metric that we track and we are trying to keep it up below 3 seconds regular basis and we are actively
[33:56] monitoring that we are and how we are monitoring that is basically by observing by default we have access to system tables we have built our own telemetry pipelines to make sure we
[34:08] track every bit of GTM hub We are tracking user behavior, how people are coming in and going out, what they're looking at, what's working, what's not working. And this telemetry and
[34:20] observability allows us to iterate faster than before, right? Uh the last slide was the last one was more about the governance end to end and unity
[34:33] catalog as it says it it's powering everything for us. So let's come to the fourth and final chapter like fast forward to today. How's it looking like as of now for us?
[34:47] Next please. So like V said we delivered and we delivered fast and we delivered in every quarter sort of and here we have sort of V1, V2 and V3 sort of versions of it. We
[35:02] trying to focus more on like what exactly worked and what didn't work. The V1 was more about like giving a common interface where people can come in and interact with it. the adoption was really really high but you know people are sort of trying to look at tables and
[35:16] numbers but they were not able to derive insights from this right so that's where we sort of launched the V2 version of it where we introduced master a you know supervisor agent and genies around with it and as we worked across with our
[35:29] routing models and accuracies we iterated a lot on our genies to keep them focused we tracked what queries are basically getting answered what's not getting answered and We sort of iterated more on that and we improved our genie
[35:43] scopes. And lastly which we sh recently shipped is more with help of lakebase which became our data base layer of our agents and we were able to save all the memories and snapshots of what the users
[35:57] have been trying to do and we were able to build that context for the users and deliver on top of it. So, three quarters, three releases. Well, numbers speak for themselves. I
[36:10] don't want to repeat them. I just want to say this. If datab bricks can host the platform that runs the database go to market,
[36:22] it can certainly host yours. Thank you. That's all from our bet.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.