Ticketmaster's self-service analytics: Building AI-powered tools with Databricks Genie

Summary

Ticketmaster built a conversational analytics tool called the Wizard, powered by Databricks Genie, that lets non-technical operations teams ask plain-language questions and receive instant answers during on-sale events without writing SQL.
The team embedded the Wizard into Slack and internal dashboards, injected business context into Genie, and built suggested questions to guide users, making adoption easier and answers more trustworthy.
By creating a continuous improvement feedback loop to maintain answer quality, Ticketmaster freed engineers from interrupt-driven data requests and reduced time-to-insight during events that sell out in 15 minutes or less.

Ticketmaster's self-service analytics: Building AI-powered tools with Databricks Genie

At Ticketmaster, operations teams are flooded with ad hoc data requests during fast-paced on-sale events. When every minute counts and events sell out in 15 minutes, manually pulling data costs engineering time and loses critical business insight. Ticketmaster built the Wizard, a conversational analytics tool powered by Databricks Genie, to enable non-technical users to ask plain language questions and get instant answers without writing SQL or waiting for data owners.
Learn how to design AI-powered internal tools that drive adoption through trust, transparency, and embedding in existing workflows. this video covers Ticketmaster's approach to injecting business context into Genie, building suggested questions to guide users, creating a continuous improvement loop to maintain quality, and the strategic decision to ship to Slack and internal dashboards. Discover why conversational AI became essential to democratizing data access and freeing engineers from interrupt-driven work.
🤝

Chapters

FAQs

What is the Wizard and how does Ticketmaster use it for analytics?

The Wizard is a conversational analytics tool built by Ticketmaster on Databricks Genie that lets non-technical users ask plain-language questions and receive instant answers. It supports operations teams monitoring major on-sale events, where questions need answers in real time because events can sell out in 15 minutes or less.

How did Ticketmaster integrate Databricks Genie into existing team workflows?

Ticketmaster shipped the Wizard through channels where operations teams already worked—specifically Slack and internal dashboards—rather than requiring users to adopt a new tool. This embedding strategy was a key driver of adoption and reduced the friction of moving to self-service analytics.

Why was self-service analytics important for Ticketmaster's operations teams?

During high-stakes on-sale events, operations teams need answers immediately to make decisions about inventory and fan experience, but manually pulling data or finding data owners takes too long. Ticketmaster was also losing insight from questions that teams never asked because the operational effort to get an answer was known to be too high.

How does Ticketmaster maintain quality and accuracy in the Wizard's answers?

Ticketmaster built a continuous improvement loop inspired by AutoResearch-style feedback to monitor answer quality and update the business context injected into Genie over time. Suggested questions also guide users toward queries the system handles well, helping maintain trust as the team expands coverage.

Full transcript

[00:07] All right. How are we doing, guys? You guys feeling good? I'm feeling all right. I'm feeling all right. I'm going to explore the space, if that's all right with you guys. Um So, when a major event goes on sale, uh you know, at at Ticketmaster, this is how a lot of you guys experience our products. When when a major artist goes
[00:23] on sale, it is, you know, a mad rush to get tickets. And our operations teams our clients who are promoters and artists and venues are all watching and reacting in real time to make sure that fan experience is good, right? Making sure that we are
[00:39] redirecting people to pages with tickets, if we have them. Um And there's a lot of operational questions that come up. And um I'm going to show you guys how I built into my product, which is a
[00:54] business analytics live observability platform, a way for our users to get those answers in real time. Where previously, it would have taken quite a while to do manual data pulls or find the the data owners and and ask those questions.
[01:14] So, all of these questions that come up during on sales, you know, what I had anxiety about as a product manager was when I would hear a question that we couldn't answer right away. Or um what I really feared was the questions that weren't being asked because the people who work are are are
[01:30] very smart and talented people that work internally just knew the operational toil of getting that answer, right? And and there's a for a lot of questions, there's a logical decay of the impact of that answer as time goes on, right? Some questions, the answer you need now, or else you
[01:46] don't need it at all. And so, we were losing opportunity for insight, for better decision-making because the time to insight was too long. So, you know, the on sales, they last minutes, right? A lot of times our on
[02:01] sales are selling out in 15 minutes or less, right? And um that gives our operations teams only a short amount of time to react and and make improvements. Talk to the talk to the client, see if we have more inventory to release, and things like that. Um
[02:17] and we don't have we didn't always have the bandwidth to have people standing by to to you know, write SQL on the fly and pull questions out. I run an observability platform and we put metrics in it, obviously, like we put the the KPIs in there in real time so that they have the answers, but for many
[02:35] of you probably know, like there is the obvious things that you should have in a dashboard, and then there's everything else, right? The other questions, the the questions that come out of having those insights. And so, you know, what it came down to was sometimes we had PMs or and software
[02:51] engineers or data scientists who would be like tasked with pulling this data out because they were the data owner, the product owner, um managing the the source topic or something like that, or they had stored it somewhere for their own operational use case, and just had access to it. So,
[03:08] um you know, it wasn't oh, it wasn't the latency of getting the question out of the database. It was the latency of finding a person to get that answer for them. So, you know, the what I say is, you know, the ratio of people who know SQL to the people who
[03:23] need answers from our data is infinite, right? That that just you're not going to solve that problem. You're not going to give everyone SQL 101 lessons and and make them query the database directly, nor do you want to. Um the way I phrase it is these people who need insights shouldn't need to know
[03:39] SQL, right? They shouldn't need to have to pull this data out manually. We can either service it in the dashboard, we can give them the answers that they want that we know, or we can um find a way for them to self-service. So, um,
[03:54] when I set out to build what I'm going to what I'm going to get to continue talking about, I pulled engineering teams. I was like, "How much time do you spend each quarter on, um, direct data pulls, right? Ad hoc requests." And the answer I got was, you know, these oftentimes software engineers were
[04:11] getting pulled out of their work to pull this data for for operational use cases and stuff like that. And, um, I knew that was a problem, right? If if you're a PM or a software engineer, you know that the best use of time for
[04:26] software engineers is doing software engineering, right? Not not writing ad hoc SQL queries that get thrown away when they're done, or um, you know, scrambling to get an answer quickly that they thought they had somewhere, looking for a dashboard that you knew you built once and can't find the bookmark for.
[04:42] Um, so we, you know, we we knew we had a problem. Um, and I I I I feel like it's a common problem. It's been around in a lot of the different departments that I've worked at, and we knew we had this gap between the time
[04:58] to insight and the ability to retrieve that that data. So, this is kind of the before and after that I'm going to walk through. You know, we had hours or days after the moment we could get the request, right?
[05:14] Someone asked a question in a in a call, a lot of times operationally we will have a, you know, a squat call that's watching our high-demand on sales, and teams are all watching from different stakeholder positions, and and they are talking in real time. And as a product manager that owns an
[05:30] observability platform, I would sit there and I would listen, right? They'd say, "It would be cool if we could see X, Y, and Z." Or, "Do we know how well this one did last time?" Or, "Similar artists to this last time in an on sale." Um, and you know, I'm scribbling those down. I'm like, "Oh, we should add that to the
[05:46] road map. We should get these things done." But really it is that is the endless backlog of ad hoc requests that from a product manager's standpoint I'm not going to build every single insight into my tool, right? That we might need once a month.
[06:01] Um and the variability of our business is that, you know, you're you might be busy on a Tuesday and then Wednesday is quiet, right? Like we are we are selling tickets um and and getting tickets in the hands of fans. And we don't own the seats, right? We
[06:18] don't own the venues. We don't We don't control which artists are popular. So, you know, one one day it might be a an a extremely popular artist, the next day it might be, you know, a minor league baseball team, which I go to a lot of minor league baseball games, so no no shade, but you know, the demand
[06:34] is not the same as a tier one artist. So, um that from a product manager's standpoint, I was I wasn't going to just continue adding these metrics to the dashboard. So, I knew I had to I knew I had to do something. I had to figure it out. Had to find a way to self-service.
[06:52] And um this is what the before and after looked like, right? We We We unlocked the ability for our users to get insights in the flow of their work day, right? When they had the question, they could ask it and get the answer.
[07:14] So, what we built, you know, we we needed to design something around the problem, which was you know, everyone that needed an insight didn't know SQL, didn't have to shouldn't have to shouldn't have to ask someone to pull something, certainly shouldn't be asking software engineers to do ad hoc requests. Um
[07:30] And so, we stopped designing around that and we removed the the premise. So, we built the wizard. The wizard is a text to SQL chat interface powered by Databricks Genie that allows our users to ask plain
[07:45] lingual plain language questions that it translates into SQL um and and pulls the insight out. Um what I did as a product manager was um insert the business context into our
[08:02] Genie workspace so that the wizard knew what the user was talking about when they were asking things, right? Your database rows and columns, your tables are not always labeled exactly how someone refers to something in the real world. So um
[08:18] props to the uh to the Genie team because they built a great interface that allowed me to add instructions label my um label my rows and columns in a way that added like semantic interpretation to
[08:35] the Genie so that when someone asks something like "How are we doing today?" right? The Genie knows to answer in a business context, right? Because like if you've experienced LLMs, if you're asking an open-ended question like that, you're going to get an open-ended answer unless it has context and instructions.
[08:52] So um when someone comes in and says like, "What's it look like today? How are we doing? How was the morning? How were our morning on sales?" When they're asking about that, I gave the Genie I gave the wizard the instructions to say, "All right, they want to know about tickets sold. They want to know about how many people we redirected. They probably want
[09:08] it broken down by region because we're selling tickets globally every single day." Um and so this is what we built, you know, we I gave them the ability to ask the questions that they had that they had in real time. Um
[09:24] and from a product management standpoint, it really gave me the ability to draw a line in the sand and say, "These are metrics that we will give you right in front of your face in the dashboard and you can and cuz we know you need these almost every day, almost every
[09:40] hour, almost every minute. Um but for anything else, ask the wizard, right? And strategically from like from my position, it was liberating, right? We didn't have that endless backlog of like oh, some someone in leadership asked
[09:55] this question twice. We better put it in the dashboard so that the third time it's there. Um we we were I was able to redirect, and that was super empowering.
[10:12] So, as I was saying, we give the genie, in our case the wizard, um and for I'm going to continue calling it genie and wizard, very confusing, sorry. It's even it's hard for me, but in our UI, it's the wizard. In uh obviously in Databricks on the Databricks side, it's the genie, but now
[10:28] genie represents a lot of things and not just the text to SQL, so um I will try and refer to it as wizard so you guys know I'm talking to the about the text to SQL interface in our in our UI. Um but I gave it the the domain knowledge. Um
[10:44] this is probably what took the longest, which is not I I hope you guys see that as not a complaint. That is the good part, right? All I had to do was give the LLM business knowledge to say, "Here are our metrics. Here's
[10:59] what they mean. Here's how to interpret them, right?" For those of you that have used Genie workspaces, you know, you can validate SQL queries. So, you ask a question, it gets it right, you're like, "Yes, that's how you use it, right? Every time someone asked this, use that." And swap out the variable. If they want to ask about a different
[11:14] artist, same SQL, different artist name. And artist name tracks to the performance name column in our in our database. So, I gave it domain knowledge. Gave it the language and terminology. Validated questions, real user examples.
[11:30] Um and the moment I knew it had teeth was the moment I was going to the wizard for questions I was getting directly from from my users. They're saying, "Hey, it'd be cool if we had this." Or "Do you know the answer to this?" Um I
[11:47] was like, "Give me a second." You know, historically it was me like trying to figure out a trying to find a Splunk dashboard I'd built 6 months ago for the same question or um just trying to to to dig it up or, you know, pretending I wasn't asking a different person for the answer and then
[12:03] answering back to an executive, "Yep, got it. No problem. Here's your answer." But usually, you know, 45 minutes later, right? And and like I said before, the fear I don't want to call it fear. The anxiety of knowing that people weren't
[12:19] asking questions because they knew the toil to get that answer out, that's real, right? That like the questions we weren't asking because we were like, "Eh, screw it, right? It's not by the time they get it back to me, I'll have moved on to the next on sale, right? The new fire that's going
[12:34] on is that I have to put out." So, that was that was huge. Um and then, like I said, the moment that it transitioned from "I'm working on this Genie workspace. I'm seeing if it's going to work for me." to "User has a question.
[12:49] I ask the wizard." This is before it was ever released as a product, before it was in the UI. I ask the question, I get the answer back, I send it to him. I was like, "Okay, now the fun part. Let's get this out to users." So, um once we had the domain knowledge, the Genie space was set up, um
[13:07] I I knew we had to to to at the very least get this out in a beta testing way to our users to say, "What else can we learn from from releasing this?" So, um I like to call this out for all the PMs
[13:22] or software engineers or data scientists in the room, I was able to take this. This is another props to the day for Xteam. I was able to take the product 90% of the way, right? I built the Genie workspace out. I had it working. And honestly, one of our constraints was
[13:37] I didn't need to bring all of my internal users into Databricks to ask the question, right? They were already in my tool getting business analytics in in live context. I wasn't going to say, "Hey, you know, close out of my product, go over to
[13:54] Databricks and get these answers." It was imperative to me that I bring the surface that I was using in Databricks into our UI. And so, what I did is I built out this whole space. I knew it was working. I had a couple power users of the tool that I gave
[14:09] direct access to to the workspace in Databricks. They used it. They're like, "That just went in my bookmark bar. I'm using this every day now. Thank you. When can we get this out to more people?" And this is when I I started talking to my my software engineers, my development team, because the best thing I can do as
[14:26] a product manager is free my engineers up to do what they're great at and try and get a product as far possible forward as possible like prove it out so that we're not putting something in a lower environment. I'm testing it. We're maybe
[14:42] sending some feature flags to some real users and being like, "Hey, what do you think? Let's screen share and and explore this." And then find out that it doesn't have legs and we've wasted hours engineering hours trying to build something. So, sandboxing myself, building it, using it, knowing that it had impact, and then
[14:58] basically going to my engineering team and saying, "All we have to do is hook up to this API and build a UI, right?" Front-end work and hooking up to the API. Got a service principal all hooked up to it. And
[15:13] we got it out there. And and it was from a from a lift to impact perspective for my engineers, it was amazing because all they had was, you know, couple story points in the sprint for hooking up the API,
[15:29] building the UI, and getting it into the hands of our users. as with all LLMs, there's a trust factor, right? You're going to ask it a question and you're
[15:44] not really sure if it knows the answer. Um like I said before, open-ended questions give open-ended answers, right? You can ask it to give you a 20-page paper on telephone poles, and it's going to, right? Um and who knows what's in those 20 pages.
[16:00] Um so, we know that we needed to build trust. My users, internally, smartest people I know. Some of my Some of my coworkers are here. Pretty smart people. And uh
[16:17] I knew that if I put something out there and they asked a question and it didn't pass the gut check, we weren't going to get anywhere. I knew I needed to give them something that was like, "It can answer these questions 100% of the time." It's an LLM, 99% of the time. We'll say 99. I don't want to be too definitive,
[16:33] but um so, like I said before, we we we put it right next to where their eyes were already in my product. We uh in- internal to my team, we call this putting the the the dial next to the button, right? They're asking
[16:48] questions they have tooling in my product that they're able to use to pull some levers and push some buttons during an on sale to affect fan experience and manage expectations and stuff like that. Um stuff like queue messaging, right? We can change the message that fans see
[17:04] when they're waiting in line for tickets. Um so, by putting the insights directly in the tool that they were using to enact change, we met them where they were living already. Like I said, I wasn't going to direct them to Databricks, pull them out of the
[17:20] moment. Um a lot of them are screen sharing, right? They're they're talking with stakeholders, they're sharing my product on a screen, and they're talking through the operational decisions that they're making. So, we put it there. Eventually, we released it in Slack. I'll show a timeline later that shows
[17:36] we had it in the UI for a while, and then I realized, let me take it a step further. Contrary to what I was saying about not wanting to encourage users to leave my product, I did eventually put the wizard in Slack because it's the other surface they live in all day, right? For those
[17:51] of you that use Slack, it's probably the application that you use the most every day, right? You've got your other dashboards, you've got your other tools, but Slack is always there. So, um we built an agent out in Slack that connected it's identical to the to the API surface um in our UI.
[18:07] So, we we put them in both spaces, met the users in the tool that they already had open. Suggested questions. This one surprised me. Few, you know, when ChatGPT launched, it was a website with a chatbox, right? Um
[18:25] I don't remember if they had suggested questions or not, but for this mental exercise, we're going to say they didn't, right? It's just an op- an open-ended um an open-ended text box. Um I realized very quickly that we needed suggested questions because suggested
[18:41] questions do more than just give a user something uh specifically something to ask. It defi- if you give suggested questions, it defines for them what can be asked, right? So, I So, I built these questions out in a way that
[18:57] touched on the tables that we had access to. One of the questions was, "Show me the average per month of this certain metric over the last 6 months." Now, the users knows that we have at least 6 months of
[19:13] data that they can ask questions about. So, the suggested questions really allowed me to inject user knowledge right into the interface so that they can see what questions they can ask, onboard with zero
[19:28] documentation, right? They get a text box, they don't know exactly what they're going to ask, they see the suggested questions. A lot of them probably just clicked it to start cuz you could click them and it would send it through and like, let me just see how this works. Let me see what it looks and feels like. Um and then the polling response.
[19:44] This, like many of you, you know, we are building things with AI and for AI and with LLMs and agents. And it's not something that we did 5 years ago, right? Not something we have years of experience building. So, for me, when the the first time I moved
[20:01] it from UI to I mean from the Databricks UI to my product UI, I asked a question and I was like, "What's it doing? Feels off." So, immediately, thankfully, it was in it was in the API and the the the the
[20:17] Genie API that you could poll and get your responses. And if for those of you that have used Wizard used Genie in Databricks, um you know that it says like, "Gathering context, thinking, writing SQL, validating the results." These things were were essential. It was it
[20:34] was something that, again, I was learning on the fly. Like, a lot of things, you know, good design is is the stuff that you don't recognize but you feel when you're using it. And we put that in there and it changed, you know, it changed the way it felt. This is just one of those things that like I
[20:50] like to call out cuz it was something I learned on the way. It was like, "Yeah, we need this polling." Another fun thing we did is, since we call it the Genie, I mean, I'm going to keep messing that up. Since we call it the Wizard, we gave it wizard-themed polling responses. So, it says like, "Casting
[21:05] spells" and "Mixing the cauldron" and stuff like that. Um which just brings a little bit of joy to the product, which we try and do every once in a while. We should have fun at work, right? All right. What we shipped and what we deferred. So, um
[21:21] the first iteration of this plain language chat inside the apps that the teams already use, Slack and uh the the on-call management tool that I manage. Um and then beta first. I knew the contingent of my users who
[21:37] were the most eager to get access to it. I wouldn't even call it that, they didn't know about it. The most that would have the most impact when it launched, right? Truthfully, these are the people who were asking me questions every day on Slack and I was like, "Stop bothering
[21:52] me. Take this. We can chat after you get your answer." Um and so we launched to them first, right? Gave them access, stress tested it. Water in the pipes, we call it, right? Let's get things moving. Let's see how it feels. See what we do. And then um
[22:09] Service level access gates. This was This was just, you know, what we needed to launch with. We knew we needed it permission-based so that everyone didn't have it right away. We didn't want to overwhelm the entire user base. Um I wanted to have a have an iteration loop first before we we rolled it out to
[22:25] everybody. Um and then clear guardrails on scope. This comes back to the the um the the suggested questions, right? We had user documentation that they could go and follow, but the suggested questions really gave them an idea of the scope that we had built into
[22:41] the product. Um what did we defer? We didn't have real-time data to start. It was just asking questions of historical data. Truthfully, it was because our ETLs on the database side they were, you know, they were running every hour, running every 6 hours, or running overnight. So, some of the data
[22:57] points that we had you couldn't ask a real-time question about. Um one of the first iteration loops was my users being like, "How fresh is this data? What can I ask questions about, right? What what can I What do I have
[23:13] available to me in here?" And again, I like I tried my best to define this with the suggested questions, but there's still more follow-up questions that come. So, we didn't have real-time data at first. Was not a deal breaker, right? The the the tool that we put the the the
[23:28] UI in, it already had real-time data. So, they could self-service that. That was They were looking right at it. The real value right away was historic. They could go back 6 months and say, "You know, what is the average ticket sales at this venue? How did we do last time we saw this artist come through
[23:46] this city?" Or something like that. Um Again, from from the the the perspective of our users, they're looking at real-time stats. They want context around how it's going and what it what it how we're doing. So, real-time data wasn't imperative. But eventually we got there. Um and that
[24:02] was a couple questions to our data bricks team about how fast we can do this. And uh and then we got real-time data in there, which has been a a nice bump, but it wasn't it wasn't essential for MVP. So, we deferred it. Um Slack surface, again, I didn't put it in there at first. Some of it was just
[24:18] prioritization, right? API into our UI, clean, simple, let's get it to people, see if they like it, see if it has enough adoption that we can roll it out to another surface. Um then like I said, pilot group and then everyone else. Um and then data parity.
[24:35] We kind of walked backwards into this one because the data set they were asking questions about were snapshots of the data in in our UI. But that created alignment. That created trust, right? They could see a number and then ask about that number and get
[24:50] the same response out of our data set. So, um that created just that alignment that I think was super for adoption. All right, making it work at scale. Um had to get people to use it, right? Here's this timeline I was talking about
[25:06] earlier. Um we shipped this in December to users. I would say I probably spent a month or two tinkering, building, testing in the Genie UI before we hooked it up, but um
[25:24] you know, similar um I didn't plan this, but you know, it happened right around the time everyone went on break and started using Cloud Code and getting a little bit more comfortable over the winter break with uh with LLM. So, we actually had people who came back after um
[25:40] you know, holiday break at the end of the year and had more familiar more familiarity with AI than they did before. For those of you guys that are building like AI can be scary, especially to users who are not familiar with it, not using it every day. I mean, I use it every day
[25:56] and it is still scary, right? It's still like, whoa, okay. What can I do with this? What can't I do with this? Sometimes you just have to ask. So, getting it out early allowed for us to kind of have a slow curve up of adoption. We weren't expecting everyone to hit at once. Um and the usage picked up in January. Um
[26:13] we had some hardening to do, right? As soon as we got water in the pipes, we found out where the leaks were and we uh we built it out a little bit more and um the learnings there were were were interesting. Some of it was polling more often, right? I don't think our polling
[26:28] intervals at the first were as fast as they are now um in terms of checking back on on when things change status, right? Going from crafting a spell to stirring the cauldron and getting the results back. Um and then the second service, like I
[26:44] said, in April we we rolled it out to Slack. It was um the how should I frame this? The dashboard that they use daily that we put it in first is not 100% mobile
[26:59] compatible. Um Slack is, right? Everyone Everyone who's on Slack probably has Slack on their phone. So, by bringing it to uh the Slack surface, we gave people access to the wizard wherever they were, no matter what. And
[27:15] this was huge, right? I had people immediately be like, "You know, I already liked the wizard. I already loved using this tool, and now I can do it from my phone, right?" Client bugging me at some odd hour, I don't have to get out of bed and go over to my laptop and boot it up and, you know, MFA and
[27:33] VPN and then finally get the answer. Um I can ask the question from my phone, get the answer. Um you can even, you know, if your if your client's on Slack, you could forward the the answer straight out to them. So,
[27:48] um that was an easy win, honestly. We had already learned what it took to put this in the UI. I had already learned everything that I needed to know to to add this to a surface. And, truth be told, I
[28:04] with Cloud Code and Codex built out the Slack bot. This was not a planned part of my presentation. I'm just realizing that I I the engineers on my team did not touch the Slack integration at all. We built I I essentially vibe coded this into our
[28:19] product. Um the tale of the tape there was the part that took the longest was getting my scopes approved by our Slack admin. So, um you know, cuz AI can't do that for me yet. Um so, we built this in. Um
[28:35] Slack has a really neat feature where you can classify something as an agent and then you get a side panel experience. You can at mention it in threads and stuff like that. Um And, you know, another interesting learning from this that I'll call out is allowing people to use this
[28:53] in an open space was really it was it was a vector for knowledge share in that allowing someone to be having a conversation on Slack and have those questions that we weren't asking before and they at mention the wizard,
[29:09] bring that answer into their thread incredible. And I've always said that the most I'll ever learn about doing my job is watching someone else do it. This falls right into that and that is that watching someone interact with an LLM, watching someone ask a question to the wizard means that I just had
[29:27] 500 people on Slack who can who can view that thread see a real-time use case which was huge, right? I would I can't prove it, but I guarantee you that I have adoption from people seeing like, "Oh it can answer that question. I'm going to use it, right? I'm going to ask that question tomorrow." So, Slack
[29:44] was a really interesting one for me. Where we are today? 83 users, 28 teams. This is just for the internal tool. Um we are working on a version of this for for external-facing use cases. Um and then a continuing continuous improvement loop that I'll get to here in a minute.
[30:00] Um and and where we're headed, continuing to improve, expanding the data coverage. One of the cooler things about this um is that as more teams get their data onboarded into Databricks and have clean tables that are ready to leverage from a from a business use case
[30:17] I can bring them into the to the Genie space, right? I just add the table. I give it more labels. I validate some new some new questions. And we're just improving the quality and the the reach that the the wizard has. So, um those are things that from a product
[30:33] perspective I can't get enough of, right? Get it done once and then the impact continues to expand as other people are doing things around the business, right? Getting tables ready for business use cases, I could be like, "Sweet. How far back does that data go?" Great. I'll add
[30:49] it to the documentation. I'll add a new suggested question. I'll announce a new feature release, right? Make We'll pretend like we worked really hard on it from a product perspective and be like, "All this data is now available." It was some other team landing their data in Databricks. Now we have access to it,
[31:04] plug it in, run with it. So, that's been awesome. Continuous improvement loop. I want to emphasize that I had no idea how I was going to maintain this thing when I released it, right? I had built
[31:20] it once, you know, for a lot of you guys who are vibe coding, who are building, you are taking taking things zero to one is kind of the easy part, right? You like I've found LLMs and and and agentic coding is good
[31:35] for the blank page problem, right? You can get from nowhere to somewhere very quickly. Going from version one to version two, way harder, right? It's That's That's where you say, "Well, I'm not a software engineer, so I'm going to have to try and hand this off to someone who is." And now it's your problem to deal with.
[31:51] Um I didn't do that here. I I built I vibe coded a continuous improvement loop. Um this is loosely modeled after um auto research, which is Andrej Karpathy's method for continuous improvement of ML
[32:06] models. And it's basically if you can validate, record, and allow an LLM to loop around those those those changes and those validations, it can continuously improve and it can throw out things that didn't
[32:22] make improvements and adopt things that did, and it can loop around the problem. So, I built this where using um Databricks MCP or probably Databricks CLI, um we went and saw the the Genie
[32:37] conversation logs. We Claude went and and saw the conversation logs, brought them down, analyzed them for basically was someone This took a little bit of training, but identifying issues, right? Finding someone who said, "That answer's
[32:54] not quite right." or "That doesn't look right." right? Responding to the LLM and like asking a follow-up question that indicates that the first answer wasn't the best answer. Um brought this question down, identified those questions, duplicates the Genie workspace, so it replicates the production workspace, it clones it,
[33:10] benchmarks before, makes a change whether it's a new validated question, add a line to the instructions that says, "Always use this When someone asks X, always use Y." Something like that. Benchmarks on the other side of it, and then it records it for me to come in and
[33:25] say, "Yeah, cool. This improved it. Promote that to production." Then we discard the clone, and we move on. And I run this weekly now. Um I have yet to schedule it, but it's on my list. Um but I just come in and run it. It picks
[33:40] up the questions from the last week. It knows when when the last It's just maintaining a log in a markdown file that's saying like, "Here's the last time I ran it. Here's the dates that I ran it for." So, um and then another markdown file to say, "Here's the improvements that we made." So, this improvement loop
[33:55] was the part that I didn't plan, but is honestly one of the parts that I'm most proud of because once again, you'll notice I did not mention my soft my software engineers in this. This is just me maintaining the workspace. At this point, there's a lot of
[34:11] I know the most about this, right? This is kind of the pitfall of taking this as far as I did and then just having the engineering team hook up the API. But building this out in the way I did, having myself maintain it, I needed to build a way to do to maintain it long term in a way that that worked for me.
[34:27] So, the continuous improvement loop um it allowed me to continue to like someone reports a problem now in our Slack channel for it. I just run the loop. Say, "Cool, I found the problem." Sometimes I have to steer it to the right exact question or exact answer, but um
[34:44] this gets a lot of the work done that I genuinely would not have done without this AI loop, right? I'm not going to comb through these conversation logs manually and look at everyone and find the the question that looked wrong. I would have had to rely on user feedback. And for those of you that manage
[35:00] anything that requires user feedback, it's not a fun way to live. You don't want to sit there waiting for them to thumb up or thumb this down something or you know, actually report an issue. So, this was a big one for me. Simple stack. This was the point. Unity catalog, Genie, Lambda, GraphQL, React,
[35:16] right? We did this in a way that kept it simple. Um you know, it it continues to surprise me how much impact we are having with this product um in a way that that really was not
[35:33] I don't want to downplay it, but it wasn't that much work. Um I was kind of coy about this when we released it. You know, we've been really we've been working on this for months. That's true. We were. But it wasn't that much collective work, right? The hard part was giving them the business context, which
[35:49] myself as you know, as part of the company I had and I had I had some help from our users, but the rest of it was not bad, I'll call it. All right. So, did it work? I'm going not going to lie, it worked really well.
[36:05] Our users love it. Um we've had 83 unique users. For an example, we have around 400 monthly active users in the tool. Again, this is an internal tool. Um uh in the in in collectively about 400. So, about 10% of
[36:20] people are have used it in the in the last 30 days. Um this is good because every single one of these every one of those 766, maybe not everyone, cuz some people are probably just testing it out, but many of those 760 questions would have gone to what I was talking about the beginning, gone to a software
[36:36] engineer, gone to a data scientist. Um and a lot of them are like for those of you that do know SQL, it's like count the distinct rows in this, you know, which is like you don't want to get pulled out of the box to do that question. You're working on something bigger, probably. So, that I'm really proud of, you know, it's easy for me to
[36:52] be like, "Yeah, the users are really happy." More importantly is the hypothetical time that we've saved by not pulling people out of their work to query a table that they may or may not be responsible for. Um like I said at the beginning, I pulled our engineering teams and they were like, "Yeah, about 2 weeks a
[37:08] quarter we were spending on ad hoc data requests." And I've I've followed up with them and they're like, "Yeah, we just don't get this anymore." And if we do, we direct them towards the wizard because it's right there. So, you know, even if you don't own a product service, you can build this into DataBricks and direct people to that, right? You can be like, "Hey, we've got a wizard for this.
[37:24] Put it in Slack." Right? You can do that and direct people to those questions. We've had Since I built this and I did some internal calls of like how we built the wizard and brown bag lunches and stuff like that, other people have added it to their Slack channel to be like, "Yeah, we built this wizard out. We built a genie out as well." And and now
[37:39] don't bother me with your questions cuz you can ask the AI and uh it's been super useful for us. couple more lessons here at the end. Um a flywheel, not a launch date, right?
[37:55] Get it out. I do this a lot with products. Um this is move fast and break things. This is throw at the wall and see what sticks, right? Um this was I I knew from testing that it was it was going to be useful, but I did not know how useful. I didn't know how many people are going to use it. I didn't know if people are
[38:11] going to be like, "I don't trust this thing." cuz it got one question wrong. Um the nice thing about the improvement loop is when it got a question wrong, I could be like, "Hey, try again in 15 minutes." and it would be fixed. And they'd be like, "Sweet, thank you. Cool, moving on." Trust uh re-achieved, right? Um
[38:27] and then guide, don't leave it blank, right? This was the suggested questions. Putting the suggested questions in there truly surprised me about how much impact it has, because you can define so much about what the product does just by putting a couple of suggested questions
[38:42] in there. It's These suggested questions should not be what our users most likely to ask. They'll They will ask those questions if you give them the guidance that this stuff is available to them to ask. So, that's what the suggested questions did for me. And then two surfaces, one
[38:58] brain, right? I built the workspace out. We might be three surfaces in the next couple months, right? Might find another place to stick it. We might connect something to the API in a way that um gives users more access. We might stick this in another product, right? We've We've built it already. The
[39:14] hard part is done. Connecting to an API trivial, right? Get another service principle. Hook it up. Move on. Um so, once we had the brain built the rest is just giving people access.
[39:30] And then why conversational AI was the right fit, right? We were Like I said before it was that line in the sand between how many metrics can I put in this tool in the UI versus like what can I steer someone to on a
[39:46] separate surface, right? By giving them conversational AI, we've allowed myself as a product manager to be liberated to say, "We will never build that. The wizard has that answer. We're not putting that in the tool." Which is maybe a harsh response, but very
[40:02] liberating for me as a PM to be like, "Those questions live for those questions, go to the wizard." Um Freedom to pull the thread. This one was awesome. It's not the first answer. It's usually the last answer that people are after. And sometimes their first question is not the right question. They ask a
[40:18] question, they get their answer, and they're like, "Ah, actually there's more there's more to it, right? I'm going to ask another question that adds more business context to the answer." And, you know, it's how many tickets did we sell that day? Oh, wait, was that a matinee or was that
[40:34] an evening show, right? Like getting asking that follow-up question to be like, "How Why were ticket sales so low on a certain day? Was it the day of the week, right?" Oh, you went on sale on Christmas. Guess what? People weren't on their computer. So, things like that that allow them to pull that thread and
[40:50] dig up the answers to things. Like democratizing the access to the data was huge. Um domain experts, not SQL writers. I talked about this at the top. The people who have the questions, they shouldn't need to know SQL to get the answers. So, putting the the tool in the hand to get that answer was huge. Um and
[41:07] then flexibility. Um the one thing that I do is I do also check the logs to see which questions are being asked most most frequently. Because there's a chance that we promote a question from the wizard up into the UI, right? Someone's asking something enough, probably should put it in the UI, right? There's that line in the
[41:23] sand, but it can be crossed in that like, "Sure, this is just a direct pipe to things that people want out of the data." Very, very useful. So, insight isn't a single answer. It's the conversation, right? Pulling the thread is what matters. Each question sharpens
[41:39] the next. And each answer feeds the one after it. The self-service aspect of it let our teams have it while the moment was still live. Super important for us. We run a fast-paced business, you know, everyone says that in their job descriptions, but the operating live on sales, fast-paced,
[41:57] 100%. Things can be done in 10 minutes. And then we're and then we're moving on to the next hour of on sales. So, putting that in their hands while it was live, while they while they the answer had impact, huge.
[42:13] That's it. Thanks guys.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.