Sponsored By: Domo | From 20 Hours to Two — Databricks and Domo: Billion-Row Insights

Summary

Live Nation manages analytics for 121 million fans and was running billion-row fan behavior datasets through a fragmented legacy pipeline architecture that produced results in more than 20 hours per cycle.
By migrating to Databricks serverless SQL warehouses and integrating with Domo Cloud Amplifier, Live Nation cut analytics cycles from 20-plus hours to 1–2 hours, enabling near-real-time iteration on audience planning at a scale of 100-plus billion-row datasets.
This video covers Domo's role as a last-mile analytics delivery layer on top of Databricks—eliminating Excel extract workflows, surfacing AI-generated audience planning guides, and scaling governed data products to customers through Domo Publish and Domo Everywhere.

Chapters

FAQs

What is Domo and how does it complement Databricks?

Domo is an AI and data products platform that operationalizes and delivers insights from governed Databricks data to business users on phones, tablets, and customer portals. This video explains that while Databricks focuses on governed data engineering and intelligence, Domo handles last-mile delivery—including AI agents, dynamic dashboards, and embedded analytics—for users who do not work inside Databricks directly.

What was Live Nation's analytics challenge before adopting Databricks and Domo?

Live Nation had fragmented data pipelines and a legacy architecture that required more than 20 hours to complete analytics cycles across billion-row fan behavior datasets. The team managed hundreds of concurrent jobs and needed a solution that could scale to 100-plus billion-row datasets while still delivering results fast enough for actionable audience planning.

How did Domo Cloud Amplifier and Databricks serverless solve Live Nation's problem?

Combining Databricks serverless SQL warehouses with Domo Cloud Amplifier reduced Live Nation's analytics cycle time from more than 20 hours to just 1–2 hours. This video attributes the breakthrough to the elimination of the legacy pipeline architecture and the use of serverless compute that scales on demand without manual infrastructure management.

What is 'last-mile ETL' and why does it matter for analytics delivery?

Last-mile ETL refers to the final data transformations needed to make governed analytical data consumable by business users, often previously handled through manual Excel extracts and untracked spreadsheet workflows. Domo eliminates this by providing governed, self-service transformations and delivery so that business users get current, accurate data without creating shadow data systems.

Full transcript

[00:08] I'd like to welcome you to uh our session from 20 hours to two. Uh what we'll be covering is data bricks and Domo integration like different touch points and whatnot of of how you can leverage DOMO with data bricks and then
[00:20] look at how our customer live nation went ahead and leveraged those touch points to bring big data insights to all their customers.
[00:32] Uh Dan Gra, a solution consultant with DOMO. uh and my colleague over here is Max Eley with Live Nation. And so what we wanted to do first is let's introduce Domo, right? I'm sure some may not know
[00:46] exactly what Domo is. Um and so when we think about it, we position ourselves as an AI and data products platform. So we are a complimentary solution to
[01:00] data bricks whereas datab bricks focuses on govern data and intelligence and building towards the key insights we're looking at operationalizing and delivering those insights to the
[01:12] business how they need it. So some examples of how we would do that is looking at smart AI agents, right? We know we can do AI and data bricks, right? I'm sure that's going to be core
[01:24] topic of the week. Um, but what we can do is not everybody is in data bricks. A lot of my customers are business users. They don't go into data bricks. They want to be on their phones. They want to be on their tablets and maybe their
[01:37] laptops. And so what we can do is we can leverage the AI and the work being done in data bricks and deliver it through agentic solutions in Domo so it lands where the user needs it whether again
[01:50] it's a phone a tablet or even a customer portal where our customers can get to it. When we talk about seamless data integration, kind of a contentious topic if you're a
[02:03] data engineer, but really what we're looking at is complementing and being assistive to what you're doing. So we know data engineering is a complex task often focused on the most critical business data, but we find that there's
[02:17] usually a long tale of data sources that are being asked for to go into warehouse. And so we bring a number of connectors, one of the largest connector
[02:30] frameworks in the industry to help the business bring their data in and land it directly into data bricks. And then of course dynamic business intelligence.
[02:42] News flash, it's not just dashboards anymore, or so I've been told by every single customer. It's about designing a consumer experience. And what I mean by that is I don't want to see charts. I
[02:55] don't want to see lines and pies and bar charts. Instead, I need to see information that's going to help me do my job and maybe even give me a button to do my job. Which brings me to the
[03:07] last piece, which is the proverbial button to click. What happens when I do have that aha moment that all the work we've done in data bricks, the way we've surfaced those insights inside of Domo, all of a
[03:19] sudden I had that aha moment. and I just need to click that button. And that's really where the workflows and automation come into play is giving the button the push. And we want to do this all in a secure and governed fashion
[03:31] building often times on top of the data bricks governance in place. So if we're to take a 30,000 foot view of how this all looks, I'm going to give you a generic data journey. Right? Left hand side it's your data sources.
[03:45] Right hand side it's your experience that the consumer wants. In the middle is where the magic happens. And so this just gives us a highle view of where we start connecting between
[03:58] data bricks and Domo. Again, we see the connector framework which can be a way to complement what you're doing already and enable the business to do more. We see that we can give you some
[04:10] transformation capabilities. So we can enable business analysts to help bring some of their data together and then we can plug into your AI. So we can leverage the generative AI or ML if
[04:23] you're still doing ML. So when we do have the data experiences on the right, it's almost a team sport between data engineering and the business.
[04:38] If we take a step down, this shows more of a traditional pattern that we see. And so again the generic data journey on the right hand or lefth hand side I should say is going to be your data sources and up top on the connector side
[04:51] you see what you're doing in data bricks you know lake flow spark maybe building your own connectors to bring data into datab bricks but right down below it
[05:03] handinand you can also leverage our connector framework and again what we're trying to achieve here is solving that long tale of data sources where it feels never ending that
[05:17] someone's always asking for a new data source. So when marketing comes and they're like, hey, you know, I got this really nice social ad use case I want to do and I need to bring in all my Facebook and my LinkedIn.
[05:31] Now we can give them tools they can bring in themselves and they can start prototyping and experiment and get that incremental value and then we can institutionalize it later when we have time and we're finished with some of the more complex data engineering tasks.
[05:49] As we move into the data bricks world the big red box in the middle what we're finding is that oftent times business uses are asking for what we call last mile ETL. I'm sure you've heard the term. It's I just need to bring the data together otherwise I got to dump it out.
[06:02] I gotta put in Excel. I got to do a bunch of VLOOKUPs and then now I have a Excel that I'm working in. And what this does is it eliminates that and preserves the governance because now we're doing that inside of Domo
[06:16] transforming. And the beautiful part here with our integration is that when I hit the run button, it'll push down to data bricks and the compute will execute there and the data will materialize there. So whether they're building
[06:28] towards a silver or a gold in a medallion architecture or they're building their own business views that you've carved out for them in their own schema now you give a secure and govern place for them to do that.
[06:41] And of course the data experience, right? Agentic workflows, automation, designed consumer experiences, no longer just dashboards, and most importantly,
[06:55] that bottom box, what we call the embedded analytics, getting these brilliant data products outside of the four walls and into the hands of your customers and partners in a secure, governed manner. Something that Max will
[07:08] talk about in a little bit. So then just looking at some terminology again, let's take one more step down.
[07:20] What we're going to find is that Domo has three main areas that we play in in the Domo data bricks world up top. Down below that's data bricks, right? You're the data foundation.
[07:32] We'll go ahead and we'll use what we call in that yellow box. I'm not sure how bright that is for you, but we call that cloud amplifier where we can set up a federated read type of integration. We can set up a write integration where our
[07:45] connectors can write into datab bricks and we can set up a native transform. So when I run that last mile ETL, it runs on data bricks and materializes the table there creating that governed world
[07:57] that the business plays in alongside of it. And up top that's your models, right? If you're Mosaic and you want to use that framework and pull those different models like maybe LM or GPT
[08:10] and then use those as your Gen AI, we can go ahead and use those through our model management or our adapter layer and essentially be a pass through.
[08:22] But ultimately those three boxes in the middle is where the capability lies. This is where we bring a lot of power to the business and enable the collaboration with the data world.
[08:36] We talked about on the integrate side what we call connectors, right? Our connector framework. They're wizards, configurable, easy. We talked about data flows. We'll often refer to it as magic ETL, a drag and
[08:49] drop wizzywig type of solution that allows you to architect your flow of transformations and again just run it on data bricks. Couple key areas of the extend. This is
[09:01] where you design a consumer experience. It's no longer developing a dashboard to requirements. We're designing something people actually want to use and it helps them do their job. Not just gives them
[09:13] information, but helps them actually do their job. And so that's where we get app studio or we get dashboards. We can even get into coded applications. So react style apps that are very rich
[09:28] and intuitive and very interactive. And again I like to highlight the Domo everywhere because this is how we deliver it outside of the four walls of your organization. And then finally the automation.
[09:42] We provide alerting on your data. So business users can set business alerts. They can set their thresholds to say if a KPI goes above or below a certain number, I want to be texted or emailed
[09:54] or maybe a little slide out on the app that they have. So instead of Uber telling you your car is arriving, Domo's telling you you're going to have a bad day because your threshold was exceeded.
[10:10] So, just to give you an idea of some some Domo and practice before I hand it over to Max,
[10:23] me to be part of their hackathon since they started it. And what's cool about it is it's takeover week at one of their offices and tackle two things. What can we come up with for known problems? How
[10:35] do we solve them? We know what we want to solve this year. How do we do it? And then the other one is what can we come up with to spark new ideas. Let's go just blue sky come up with something really slick.
[10:47] And so it provides for a fun week, right? Because you're solving problems and you're also ideulating. And so here are a few things that we came up with.
[10:59] hey, we got a bunch of ML models inside of data bricks, right? We got all the data. We build and train the models, you know, create the notebook, uh, you know, do the ML flow, uh, to move it down through into the, uh, models and then
[11:11] into serving the endpoints. How can we get the value out of those? Better yet, how can we give the value of those to a non-data scientist? And so using the integration points with
[11:25] DOMO and data bricks, the first part is all right, everything was built and trained inside of data bricks. the data lives there. So let's take that right the top screen there and let's move to
[11:37] what we call the model adapter or the model management layer of DOMO and connect to it right we'll authenticate to it we'll go ahead and provide our access token and then we'll pull that model in or basically build an endpoint
[11:51] for it and then we can start pushing that throughout the Domo platform. So in a last mile ETL world, what's amazing is that business users can start mocking up their data. They can start prototyping
[12:03] and then they can drag in a model inference tile that ties back to that model. And so now we're able to say, "All right, well cool. This one we just prototyped to be a uh predict audience and we were able to push that through an
[12:17] ETL and then build a pretty slick dashboard on top of it that showed not only historical but also the predictive side of it and what's the attendance going to be the rest of the year.
[12:31] Another pretty slick thing we did understand the band show of hands and you can't leave me hanging because I hate when that happens. Ever hear of the band Dive? Shoeg Gaze Band.
[12:44] Okay, we got one. But I never heard of them until I went to their hackathon and we went to a concert that they sponsored called Dive. So, it's a band. Never heard of them. But it gave me that aha moment of, hey, this would be a pretty slick thing. What happens if I'm decide
[12:57] to throw away my career at Domo and start being a manager for a band? What do I do? Right? So, I have no idea where to book these guys. I have no idea the size audience that they would command. And so this was my idea. Let's just use
[13:11] Gen AI to see if it could help me figure some of this out. And so let's build a pretty slick interactive app on top of it. And I just have to provide some basic stuff. The band name, how many
[13:23] shows, average ticket price, and what states we want to be in. So I could be like, "Hey, all right. We got this band dive and it's going to be uh nine shows across California. Where
[13:36] should I book these guys? No clue. But with the prompting that goes behind the scenes pulling the AI models, I was able to then kick out some pretty cool stuff. And that's where we got the
[13:50] first screen there. Review the AI generated audience planning guide that tells you, hey, this is who dive is. This is the size audience they typically command. The next one was here's where we think
[14:03] they would probably fit in terms of actual venues. You know, here are the different cities. Here are the different uh clubs or uh they're probably not a coliseum band, but and then what's also
[14:16] cool is here's what the price would be. And then the best part about it is here's bands that they're kind of like. So hopefully it'll ring a bell that you know, hey, maybe I've heard of some of these other bands out there.
[14:29] like the Ringo Death Stars, if you've heard of those. Um, but that was just a Bluebird idea that started sparking, hey, we can start using this to come up with solutions that don't have to be
[14:41] dashboards. It can give me information, help me do things like what happens if I want a lightweight yet powerful experience to start almost like a scratch pad, right? I got some ideas,
[14:54] I just want to bang it out. I don't want to have to go to an app. I don't have to launch a bunch of stuff. I don't have to load data. I just want to go into a spreadsheet, bang
[15:08] leveraging that same kind of pattern of, hey, we got all of our data in Unity catalog, we built and trained a really slick predictive model in there. Huh. What if I just allowed someone to drag that spreadsheet in,
[15:22] go to that model, infer a few things like total audience size for the entire tour, projected revenue for the entire tour, and then each tile in that bottom would
[15:36] represent an individual venue capacity, what it would actually fill to on a given month night. So now I have
[15:49] this tool that just lets me bang out some ideas of, hey, I never heard of this band before. The other tool told me it's kind of like this. So I'm going to I think I can get these clubs to to bring them on. And then ultimately I'll go ahead and see if I'm right.
[16:04] So just some ideas of of where we can start using DOMO to deliver the experience and operationalize what's being done in data bricks today. So, if you want to see some of this, you can come down to the booth 205. We have
[16:17] a bunch of demos up. I actually have those demos as well if you want to see those in action. Uh, but definitely come down if you want. And what I'd like to do now is I'd like to hand it over to Max and have him talk about his journey
[16:31] using the Domo Data Bricks integrations to deliver those big data insights to all their customers. >> Cool. Thanks, Dan. Um
[16:43] all right. Yeah. So I am Max Eley, the director of BI um and uh analytics at Live Nation media and sponsorship division. Uh Live Nation is branched out
[16:55] as kind of sort of like a hub and spoke model. We have a bunch of different organizations. So everything that I'll be talking about today is specifically, you know, geared toward media and sponsorship. Um so just a kind of a quick intro um
[17:10] into who we are and what we do. So at media and sponsorship uh you know we basically the whole goal of our division is to connect brands with fans through the power of live music and culture. So
[17:22] if you're familiar with live nation ticket master at all um you kind of know how that drives with with our goal there. Um, you know, we bring together a a bunch of different teams, multiple teams, including strategy, creative, media, experiential design, um, you
[17:36] know, obviously artist and brand management. And then we have our team, the data and analytics team. Um, our scale across the organization, you know, we scale across 44,000 different concerts. I mean, even more than that, I
[17:49] think now, um, 121 million unique fans, hundreds of venues, and hundreds of festivals. Um, and obviously, you know, millions and millions of tickets sold annually. So, as a team, as a data and analytics,
[18:03] we call ourselves the DAPs team. I'm looking over there at my colleagues. Shout out. Um, you know, we create data and insights uh to drive these brand partnerships that we have um to help them solve, you know, real business
[18:16] challenges that they have um help them come up with different campaigns, you know, so that they can drive their business forward. Um, we also work on building fan first experiences across concerts, festivals, whatever it may be.
[18:30] Um, you know, when you go to venues, uh, you know, you see those promotions up on the boards and, you know, you go to festivals, you see the large activations. That's the stuff that we work on. Um, so going on here, um, to kind of lay the
[18:45] groundwork for what I'm going to talk about, this is lots of kind of theory crafting, um, kind of stuff. So a big challenge that we had been running into for a couple years um you know a couple years since I had come on board um was
[18:59] that we had a customer-f facing product that was in front of um you know these enterprise brand partners that would come to us and pay us lots of money and we weren't able to iterate and or add
[19:12] new features or kind of innovate on those things fast enough for those clients. So it wasn't creating like a a very typical or a very good customer experience. We we weren't really running into a problem of finding answers in our
[19:26] data. We could do that just fine, but it was really like the analytics loop and the analytics life cycle that was really trailing behind and causing a problem for us. So we have, you know, obviously
[19:38] just at our division billions of rows being processed daily through data bricks, through Domo, you know, through other platforms. Um and because of all of that complexity, you know, we were running into these cycles that if we
[19:50] wanted to make a simple change in uh a pipeline or if we wanted to make a simple change, you know, like adding a new field to a table, it would take just these vast amounts of time. Um so you
[20:02] know the the the core shift that we wanted to make was we need to get rid of this large reporting queue that we have you know of these requests coming from internal stakeholders coming from
[20:14] external stakeholders you know these people that we're building these products for and turn that more into like a governed data loop that we could actually try and automate and and uh make faster.
[20:39] the changes basically to our underlying data that we're then going to pipe into our BI layer or the user experience layer. Um, we would then use Domo's uh native connectors or cloud connectors to
[20:52] connect back to our cloud warehouse to data bricks, bring that into Domo, land it in Domo, um, and also materialize that in Domo. And I emphasize that because it'll become important in just a second. Um and then you know obviously
[21:05] we would use those materialized tables in DOMO itself to build reporting for internal stakeholders and then you know all the apps and custom things that we would build for our customers as well.
[21:24] So that old architecture uh kind of had two major blockers that were really slowing us down and becoming a problem. As I had mentioned earlier, you know, a single change could easily become, you know, a a one to two to sometimes even
[21:37] like a almost bleeding into a three-day life cycle.
[21:50] were constantly running and running a bunch of, you know, hundreds of other jobs. And then we would, like I said, land that data in Domo. Um, and that would take, you know, the data bricks update itself would take 10 plus hours
[22:04] on some of these giant data sets, some of these giant tables that we're working with. And then when that was done, we would ingest it into Domo and it would take another 10 plus hours. You know, sometimes we would even run into these
[22:16] timeouts. No fault to Domo, but it's just like these tables were, you know, billions and billions and billions of rows and it was just too much to ingest and materialize. So, you know, when somebody internally would come to us and
[22:29] say, you know, I want to start analyzing XYZ year-over-year, we would say, okay, we're going to have to, you know, actually figure out the granularity of this data, bring in these dates, whatever, whatever. It would turn into a
[22:41] two, three day process for us to even get it in to be able to QA it and then, you know, maybe a couple days later to actually get it into them. And same process with our external stakeholders as well.
[22:56] So again, you know, the the copying of the data became the real bottleneck here. Um, you know, the the fact that we were trying to you, you know, this is kind of the old way of serving data. You know, grabbing it from your warehouse,
[23:08] landing it wherever your your user experience layer is, and then reporting off of that. And that's what was really slowing us down. So the architecture pivot that we made
[23:20] was, hey, you know, we're just going to keep all of our data in data bricks. This is our cloud warehouse. We don't need it to act as really anything else but that. So we're going to keep everything in here. We're going to model
[23:32] our data. We're going to transform our data. We're going to make sure that everything is clean, ready to go in data bricks. And then instead of using Domo's connectors to then grab that data and land it and materialize it in DOMO, we
[23:47] are going to utilize you Domo's cloud amplifier which uh Dan had mentioned which allows us to then just create like a federated tunnel to that data. So the difference here is that instead of
[23:59] having a table and you know using Domo as kind of a secondary warehouse, we can then just have instant access to our data for whatever reporting needs that we have. So you know if somebody comes
[24:11] to us and says you know I need to make this change or you know I want to see XYZ instead of having it take 15 16 plus hours to ingest into Domo we now have
[24:23] instant federated access to that data that lives in data bricks touched the mic sorry um and you know another big change here and obviously I highlighted the cloud amplifier but another big change that we kind of
[24:35] shifted to is we all started using serverless SQL warehouses they talked a lot about, you know, these new warehouses coming out. So, this is like outdated as of an hour ago because I think that they're going to come out with some really they did come out with some really cool stuff. But, you know,
[24:48] we made the shift from using allpurpose to serverless and that really sped up um our life cycle quite a bit as well.
[25:00] Uh so, yeah, this is just kind of an overview of the new architecture that we're now following. So, you know, all of our jobs and data bricks are now running off of serverless SQL compute. they are no longer using all-purpose clusters. So you not not not only is it just you know hundreds of times faster
[25:15] also it's cheaper obviously but we don't have to wait for those clusters to spin up and you know and do their thing. It's like serverless is always at the ready ready to go so we can iterate very quickly. Um and then again instead of
[25:28] landing the data in our experience layer we're keeping everything in data bricks and we're just federating access that way.
[25:44] So yeah, the results um you know the the big impact came from changing our whole architecture really not just trying to fine-tune any one step in the process. You know before again we were spending sometimes two to three days uh trying to
[25:56] iterate trying to make changes um that would come in from our stakeholders. you know now in data bricks if we make an update to a notebook to some sort of uh process that we have there it's you know
[26:08] under an hour most times that we we can actually run these jobs um and then when it's time to bring it into Domo utilizing their cloud amplifier uh product we're able to pull that data in
[26:20] almost instantaneously I mean we're you know some of our data sets are 10 billion rows and we're able to federate those and read those you know within a matter of two three four seconds sometimes. Um,
[26:35] oh, and I guess yeah, I did also want to highlight that, you know, a a big blocker for this. It's not just that it took a lot of time, but behind the fact is that, you know, people would often come to us and say, I need this insight.
[26:49] I need this analysis done. And it would take us so long to get back to them that they had kind of moved on from that at that point. So, it was no longer really like at the top of their mind for them. And I think that you know the shift that
[27:01] we're all kind of seeing in the industry is especially with like the commercialization of LLMs and AI in people's hands you know stakeholders really want insights like now and quick and in their face. Um and so you know
[27:14] especially with companies like us who are using these tools that have been you know developed over you know 10 12 more years ago. we have to really try to shift our whole architecture into, you
[27:27] know, being able to provide those instant real-time insights. Um, so yeah, our infrastructure gains became customer gains. You know, uh, for our internal teams, obviously that meant that they had fresher dashboards to look
[27:40] at. Um, you know, it meant that they could come to us and ask us questions, uh, you know, and say, "Hey, I'm looking for, uh, you know, XYZ about this brand." And we were able to deliver that to them in a very timely manner. Um for
[27:53] our sponsors and our clients, you know, our paid customers, uh we were able to uh deliver a much better user experience to them. You know, their data was much fresher. Um they were able to come to us
[28:06] and ask follow-up questions after they got into their apps that we built them, and we were able to iterate and change things for them if they needed it. And then, you know, for our team, especially, you know, the analytics team, there's just less waiting in
[28:18] general. You know, we're able to be super innovative. were able to say, "Hey, like I have this really cool idea of something that I want to try and let's just do it." It's it's only going to take a couple of hours as opposed to
[28:30] like I I don't really want to like go through with that because it's going to take three four days to even like see the the you know the the progress there.
[28:44] Um so yeah the blueprint here and you know they they mentioned it like a lot of times on the main stage today um is that you know keep your keep the heavy data and the compute very close to data bricks right keep it close to the cloud warehouse you know avoid copying these
[28:59] giant tables as much as you can I think again copying was kind of like the old architecture like the way to get stuff done because I think that we just didn't know any better. Um, but you know, now
[29:12] that federated access is becoming kind of like the gold standard, um, you know, the blueprint here is keep your data in your warehouse, federate the access to your user experience layer. Um, so yeah, you know, treat DOMO as the experience
[29:25] and activation layer, right? Uh, Domo or any other BI tool doesn't necessarily have to be your warehouse layer anymore. And I think that you know the the these BI tools are also realizing that and starting to change their architecture to
[29:38] become more of like an experience layer than kind of like a storage layer. Um and
[29:51] you're probably realizing like a very foundational lesson here that we're talking about. you know, there's no like heavy AI that we're including or doing in any of this. And that's because, you know, our team realized long ago that,
[30:04] you know, we can't safely but also accurately and confidently deploy um, you know, any sort of AI architecture unless we are fully governed and unless we have like a very strong foundation in
[30:17] our data first.
[30:29] data foundation that we've been laying out you know our team is exploring nextgen ways to uh you know accelerate our our AI use in the company. Um we're coming up with some really cool use cases. um you know uh you know we have
[30:45] lots of uh fastm moving parts in the business especially when it comes to like a new product is launched or a new show is launching and you know these real-time insights become very important and so we're exploring ways that we'll
[30:57] be able to do that but you know again we're able to do that now because we spent so much time working you know on the foundation here making sure that our iteration loops are very short but also
[31:10] very accurate. accurate, you know, um, very compact. Um, you know, worked a lot with Domo to make sure that our user experience layer is up to speed and where it needs to be to be able to do this kind of stuff.
[31:29] Uh, and that's the end. So, we will, uh, pause for Q&A if anybody has anything. But that's it. >> Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.