How NBCUniversal Migrated Peacock from BigQuery to Databricks and Saved 30%

Summary

NBCUniversal migrated Peacock's data platform from BigQuery to Databricks to achieve better cost efficiency, performance, and data interoperability, executing a zero-downtime migration of 5,600 active users while maintaining full operational continuity.
The migration strategy minimized data duplication using Apache Iceberg for interoperability and external tables, enabling phased user adoption without requiring parallel copies of all datasets during the transition.
The migration delivered immediate results: a 30% cost reduction and improved on-time delivery for data products, along with native performance optimizations that better handle the massive traffic spikes that occur during major live events.

How NBCUniversal Migrated Peacock from BigQuery to Databricks and Saved 30%

When Peacock's data volumes and computational needs outgrew BigQuery, NBCUniversal faced a critical decision: migrate to a platform that would deliver better cost efficiency, performance, and data interoperability. This talk outlines how the team executed a zero-downtime migration of 5,600 active users from BigQuery to Databricks while maintaining full operational continuity.
Learn the migration strategy that minimized data duplication, leveraged Apache Iceberg for interoperability, and delivered immediate results: 30% cost reduction, improved on-time delivery, and the ability to handle massive spiky events. Discover how external tables, phased user adoption, and Databricks' native performance optimizations transformed their analytics stack.
🤝

Chapters

FAQs

Why did NBCUniversal migrate Peacock from BigQuery to Databricks?

As Peacock's data volumes and computational needs grew, BigQuery struggled to deliver the cost efficiency, performance at scale, and data interoperability required. Databricks offered better native performance optimizations, lower cost, and the ability to share data across the organization through Apache Iceberg.

How did NBCUniversal achieve zero downtime during the migration?

The team used a phased approach leveraging external tables in Databricks that pointed to data still stored in BigQuery's underlying storage, eliminating the need to duplicate data during migration. Users were transitioned in phases while both systems ran in parallel, ensuring continuity until the full cutover was complete.

What role did Apache Iceberg play in the Peacock migration?

Apache Iceberg provided an open table format enabling interoperability between BigQuery and Databricks during and after the migration. Using Iceberg as the storage layer allowed both systems to read the same tables during the transition period and avoided vendor lock-in going forward.

What results did Peacock achieve after migrating to Databricks?

The migration delivered a 30% cost reduction and improved on-time delivery for data products. The platform also gained the ability to handle massive spiky events—such as major sports broadcasts—more effectively, since Databricks' compute architecture scales elastically to meet demand without the performance degradation Peacock experienced on BigQuery.

Full transcript

[00:07] Okay, we'll go ahead and get started. Um, I'll spend a couple minutes there, a minute or two just introducing myself. You may have read the bio. Um, my background is actually mathematician. Uh, you may have heard the conversation here. So, uh, I got a bachelor's and masters in in mathematics and then I was
[00:24] like, I don't know if I want to get a PhD in math. I kind of want to do something in the real world, right? So, so I switched to industrial engineering, but I was still studying the same stuff. Um, it served me really well in my career moving into data engineering. So, you know, if you have kids and they're
[00:39] trying to decide what they should do, like a math major is actually pretty solid if you know what to do after it. Um, so I've been uh with NBC for five years. When I joined it was I joined Peacock. It it wasn't actually NBC. So, um, how many people know about Peacock
[00:55] already? That's good. there's a there's a time where not a lot of people would have raised their hand. So, it's really nice uh to see that people know who we are. Let me talk about who we are. So, it's kind of a cool story um for those of you that don't know and I'm going to talk a
[01:11] little bit about um how Disney falls into this. So, for those of you that recall, Disney bought Fox and all of its, you know, assets three, four years, five, five, six years ago, seven years ago, something like that. when they did that they also were trying to acquire Sky which is UK based uh telecom company
[01:30] and for various reasons like I was in the room I don't know didn't happen so that got left on the table Comcast bought it up so this this was I think in around 2018 2019 and Sky already had an over-the-top
[01:46] streaming service that existed they already had a platform so not surprising we can use that to build Peacock And that's exactly what happened. So using the same tech stack that uh Sky had built all their streaming services on it got obviously updated adjusted for the
[02:03] United States and we launched Peacock in 2020. So uh this was the midst of the pandemic and everybody was at home figuring out what they wanted to do. And one of the things that I think was really interesting about as we entered
[02:18] the marketplace was how many people like free stuff? Come on, raise your hands. Everybody likes free stuff. We said, make it free, right? We're going to put ads in it, but it's free. You can sign up. And instead of having like ad ad supported existed, Avod was a thing. We're not innovative
[02:35] there, I don't think. The thing that we did that was different, we said like we're going to put our premium content on a free Avod service. That didn't happen. Netflix, Disney, all at the time, if if you bought, right, you were getting your subscription, no ads,
[02:51] right? if you wanted ads, you were going to watch, you know, whatever was on FUBU or, you know, stuff like that. So, this was a big difference in the marketplace and it definitely helped us establish ourselves and and get a nice foundation.
[03:06] So, going back in time, it's 2020, it's 2022, it's 2023, we're growing. This is great. Um so around 2023 2024 it was in that time window uh where we're starting to get good growth and
[03:23] you have to start looking at you know what does our platform look like you know and and how are we supporting all these users and what are we doing so let's talk a little bit about that um this is our roughly what our ecosystem looks like for analytics and reporting
[03:39] the going up to bronze is owned by Sky still to this day. So that we're still leveraging that same platform that they use to run Now TV as well as other streaming services. It's all all the same. And then starting at silver is
[03:56] where my team picks up. So we're we're the data engineering team. We're now media group. We were originally Peacock. And so this architecture is probably something that most people in this room are somewhat familiar with, but I'm just going to go over a little bit of what we bring in from a a data standpoint. So
[04:12] the key things you can imagine you you have to have some sort of identity management system right people are signing up their emails it has to be protected um if they want to change their email you have to update their identities things like that sales services we process transactions like we
[04:28] ourselves process transactions we have third parties that we integrate to also process transaction you can for instance you know get your peacock subscription through the Apple at the app store um now you can also get peacock talk through Amazon if you're interested.
[04:45] So uh and then the other one the big one is the analytics tracking like what are people actually doing on the platform? We really need to know that in order to operate the business. So that data gets ingested uh there's a data ingestion factory that Sky has. It does the
[05:01] encryption, dduplication, um, and basically anonymizes data to a point where it's considered analytics data and it's separate from PI lands and bronze. It's what you would expect. It's flattened data that, you know, I classify as engineering ready. Uh, an
[05:19] engineer can very reasonably pick it up, start working with it. It's not something I want to put in front of the business. Anybody that's worked with Adobe or some other implementation tool knows the variable the variable names. It's like post ev22. Like what the hell is postr22? Okay, so that's bronze for
[05:35] us. Bronze is I don't expect users like my analytics users to be going in there, but I expect my engineers to be able to work in that and they need to understand how to read the documentation, things like that. Then we start adding in the business logic. There's, you know, a lot of information that we have. Uh I'm
[05:52] distilling it down to what's the core that you would need to operate a streaming business. I need to know who my users are. I need to know what they're watching. and I need to make sure that I have consistent metadata across what they're watching. If I can do that, you can answer 80% of the questions that you'd be interested in. How many subscribers, what are they
[06:08] watching? Right? So, that's silver. It's very atomic data. Uh and then gold build on top of that is that's the aggregations, right? And it's kind of broken out by functional areas. Marketing, engagement, personalizations are just a few of the examples of the areas that we support. And then when we
[06:25] set this all up, as you can see, it's all running in Google Cloud Platform. We're all running in BigQuery after this point. And when we wanted to do the visualizations and and run our uh reporting suite, we found that the the way that we had architected BigQuery, we
[06:42] had a fixed slot allocation. It didn't have we had performance conflicts. So, you'd be having the batch running and then an executive tries to come in and look at a dashboard and they're like, "This is taking like 10 minutes to load." And like not acceptable. Uh so we set up uh Snowflake as this warehouse
[06:58] and we would just push what minimal data needed to go there to operate our executive dashboards in Tableau and then we had our analysts, our data scientists, like the personas that are like I need to query the data. They were not working in Snowflake. They were working in BigQuery. So we were all
[07:14] Google, all BigQuery. Everything's running pretty darn good. Like we're really happy in 2023. Um and not surprisingly though like uh you can see the stats at the bottom around this time we've got four pabytes of data
[07:30] in our platform. We've got you know billions of data points that are coming in particularly through Adobe uh every day and our user base was to the point where you're 10,000 2,000 20,000 queries running per day. So, it's uh you know,
[07:45] we're starting to look at this and like if we keep growing the way we want to grow, does this system scale the way that it needs to? Um, and so what's happening? Costs are going up. That's not surprising. Everybody costs are going up. Um, we ran into an extra bonus
[08:01] problem that Google decided it was time to restructure contracts and it just so happened that BigQuery was impacted in a big way. Uh, flexibility. we we have a lot of like spiky events. Uh I didn't spend too much time talking about it before, but on that graph where we where
[08:17] I showed the the growth curve up until 2024, there were things like we had an exclusive wild card again event for the NFL. So for those of you like to watch football, that's not a small number of concurrent viewerships, right? You have to be able to scale the platform to
[08:32] process all that data to get the reporting ready at 8 a.m. just like any other day. Um, so when that happens and you're on a fixed slot setup with Google, it's like it was hard. Some of this is organizational. I'm not gonna blame Google and say that their platform can't do that. It's the way that we worked with Sky and had these things set
[08:49] up created some challenges. Um, uh, and then the other one is sharing data. So most of MBCU was not on AWS and most or was on AWS, not on GCP. So we're working
[09:04] in GCP to AWS. It's already kind of a a little bit, but more so BigQuery is it's this proprietary file format. If I want to get that to somewhere else, I basically I'm going to have to make a copy and put it there for somebody else to be able to access it. That didn't really matter the first few years,
[09:20] honestly. Um I don't know if anybody's been around when a major initiative happens at a big enterprise company, but it was basically like we're a startup in the enterprise and it's a uh don't bother them, but they can bother you. like if I need something, I can go to my
[09:36] partners and be like, "Hey, help me out. We're Peacock. We're the most important thing right now." Um, but that was starting to change. We want to integrate into our overall enterprise. The strategy is not forever for Peacock to be this like, you know, spunky startup. At some point, we are a part of NBC. And
[09:53] as that's happening, we're looking at how do we share data. So, we looked at many different options. Snowflake was an obvious place for us to start because we had already had it in our stack. Um but data bricks was also a very good option. So uh the things that were interesting
[10:09] for me uh there's a shift in the way that spend is done. Um in particular we were spending a lot of money on storage in BigQuery. Uh shifting storage from BigQuery into buckets very appealing. So
[10:25] that was one dimension of it. Now obviously you're going to have to pay for the compute. So, who knows if it's going to iron out to where you want it to be, but that at least gave us one dimension that we knew we were going to be able to save some money on. The performance potential was very appealing. Uh, the way that uh the
[10:43] compute is allocated in data bricks and the way that our team wanted to operate, it was going to give us more control so that if something was running behind in the morning, we may even have the potential to go in, turn up the knob, add a little bit more compute. So, okay, this is a good option. And then interoperable data delta was a thing at
[11:00] the time. So we weren't really too concerned. I was a little bit because it felt like that locks you a little bit to data bricks. Tate is our account rep. He's in he's in the room. He was talking about this and I think it was like a month later that the announcement of
[11:15] Apache Iceberg being and I was like okay great. So with Apache Iceberg being a file format that's available being fully utilizable within Snowflake meant I don't really have to worry about interoperability in terms of like the data formats anymore. So that was a big
[11:32] check big check uh and so we did a little test um before we actually like went in we did a little test. We've picked our most expensive uh executive dashboard, the query that took the longest and cost the most to run, which that was an interesting exercise in and
[11:48] of itself because we're operating with a fixed slot capacity. So, how do you isolate the cost of that? So, we had to work with our fin team, make sure that we understood, you know, like what are we actually spending to get this data generated? Uh, and then we did a side-by-side comparison and like it's not going to be surprising. We had to go
[12:04] through like five or six iterations of tweaking the SQL a little bit, tweaking the compute parameters a little bit and getting it to a spot where that query now ran faster in data bricks. So with that amount of time spent with our uh
[12:19] account partners and everything like that, it's like this seems good, seems very possible that we can do this and it's going to save us money. Um so we started. So, how do how do you do a migration when you've got five 600 users that are
[12:36] running queries every day? They've got scheduled jobs in BigQuery. You have to deliver your executive dashboards by 9:00 a.m. Like, what are you going to do? And so, first and foremost, it's like you got to figure out a way to make sure that you're not blocking anybody
[12:52] from accessing any of the data at any point in time. So for us is like we're going to end up having a a big query accessible environment somehow and then eventually we have to shift people into a data bricks environment and at any point in time we don't want anything to
[13:08] go down. So the first thing we did was let's just build up the foundation. Uh if I can get my silver tables matching that gives me very high confidence that the gold tables will match as well. I can also work on this kind of
[13:24] separately in parallel. Um there's a little bit of context here where we had a committed uh slot purchase with Google that went through beyond uh the end of the year. This is around like 2024, early 2024. So, I knew that I was going
[13:41] to have six to eight months that I could spend really kind of focusing on getting our ecosystem up and running and optimized before I was going to run into a place where I have to consider the cost separately of these slots are going
[13:56] away and I have to replace them. So, it gave me a long runway. Uh, so we worked with the team, we get this going and we did the naive things that you would expect you would do when you migrate. So we took all our SQL that runs in BigQuery. Uh we worked with we had a
[14:12] really great partner EXL that uh they've been working with us since day one. We made them a huge part of this migration. We worked with data bricks and we de developed some automated tools to go I need to convert data bricks or uh bigquery SQL to data brick SQL. So we
[14:29] did that. We had to migrate data from BigQuery into iceberg format. So we developed tools for that and we turned it on and it was not great. Like it ran but it wasn't faster and it was not
[14:45] cheaper. So that's not surprising like we had spent four years optimizing our code for one ecosystem and then like how much do you expect it to really just like run faster somewhere else? Uh but we knew it was possible because we had done the little PC before. So this is
[15:01] where we started working really heavily with data bricks and being like what do I need to do in my environment to get this to run faster and cheaper than what I've currently got in BigQuery. Uh lot of experiments where I'll say
[15:19] it's what you would think you would do if you want to run an experiment. So I've got a bunch of pipelines. I got a bunch of configurations. I'm going to run those pipelines with those configurations multiple times and see what kind of timing I get. And that helps me understand where I need to use photon acceleration and where I can get
[15:34] away with not using it and things like that. So we probably spent a good 6 months focused on just that like just get silver running faster and cheaper when we did that and everyone was comfortable. We were doing spot checks along the way, you know, like total row
[15:51] counts, you know, some validation work. But once we have this ready, now we say we need to kind of get rid of the other silver, right? So since we have these Apache iceberg open file format, what we can do is create external tables in
[16:07] BigQuery that reference the data that's coming from UC. Um, and it allows us to now start telling people and ourselves as well, don't use the stuff that's in silver, use the stuff that's in silver external.
[16:22] And naturally, you're going to ask a question like, how much work was this for people? And it was like, add four characters to your table strings and that'll be it. So, we made it as easy as possible for people to do this replacement. So, we left this up and running like this for a while, but again, like I want to minimize the
[16:38] amount of time I've got data duplication. So as soon as you know the deadline hits, we move forward and we go ahead and delete that data. So that's gone. All right. So this brings us now to a spot where users primarily are still working in
[16:56] data brick or in BigQuery, right? Everything is still BigQuery, but at least the silver tables are referencing out of data bricks. And this is kind of like step one to get to get the people to move over. But the other thing you have to do is you you do have to build the rest of the stack for them uh before
[17:13] you can get them using uh data bricks. So the next step was let's start building gold again doing the same the exact same philosophy while the users have access to everything in big query went through the same exercise. This was
[17:28] much faster. This happened much more quickly because we already knew a lot of the things that we needed to do. We learned from silver. So this although the volume of work was a lot bigger, we have a lot more gold tables than we do silver tables. Uh it was it was actually faster. So we have this nice foundation.
[17:46] We've got gold data that is available in data bricks. Uh we were able to bring in the bronze data as well so that users that did need to access that data along with silver gold, they'd be able to do it in data bricks. So we created that connection with a foreign catalog.
[18:01] Then we had to figure out how do we get all of our reporting and all of our users into data bricks. This is the part where uh I probably underestimated the amount of work I was very fortunate that my team did not. So they uh they helped
[18:18] identify like the amount of work in particular that the user base would have to do. Uh, I kind of have this philosophy for better or worse that it's like if you tell somebody how to do something, they should be able to do it. Um, turns out it's not that simple. People have day jobs. Uh, so looking
[18:36] back, I'm like, yeah, we should have been way more proactive in pointing out like, hey everybody, we can use tools to help you migrate instead of saying you're going to have to migrate your own queries. You're going to have to migrate your own data. So we we switched. We did as much service to the customers as we
[18:52] could. It was like if you need your data moved, we'll move it for you. We have tools that we use for our migration, we'll do that. If you need help migrating your code, we can at least move it. We'll help diagnose problems with it. We had office hours with data bricks folks. I think we were doing them twice a week for a while there. Um, so
[19:10] this was basically saying, "All right, everybody start practicing. Get in there. Get using it." And once they did, and once we kind of got over a hump, then we say there's going to be a deadline at some point, we're going to turn off
[19:26] Big Query, and you have to use data bricks. Um, this is the other one where I'll call out, there's probably a lesson learned on my side. We started with, let's give everybody a small warehouse and see how they do. And that's that wasn't the right strategy because the
[19:41] immediate response was like, everything is running slower here. I'm like, well, did you did you adjust your queries yet? Because I really wanted people to adjust their queries before they just rely on compute. That's not really fair for your analytics user base as it is for your engineering user base. I don't mind
[19:57] challenge my my own team in that way, but I think that's lesson learned is you don't necessarily want to start right at the base of what's possible. Give, you know, give people a little bit of a spectrum of options. Um, it'll probably go a little bit smoother. So, we're in this spot. We've operation
[20:15] we've operationalized all our gold tables which means we're able to delete that other huge chunk of data we've got. Um and the other thing that we we're doing at this point is instead of copying data from BigQuery to Snowflake because we're using open file format we're able to just expose the data
[20:31] directly to Snowflake without a copy. So this is what our stack looks like now. Uh we still do expose the tables uh in silver and gold in BigQuery. We have partners at Sky that are still primarily BigQuery users. So, we want to be able
[20:47] to accommodate that. But, uh, I would say the rest of the analytics organization that we support, the data scientists we support, they're all using data bricks now. Um, the portion of that migration, I'd say we spent a couple months prepping for it.
[21:04] The actual getting users switched over from BigQuery to data bricks took about three months, maybe a little less. Um, it was great. So, what has happened since then? Uh,
[21:22] someone had asked this question. So, we were at about 36 million subscribers when this project had started and we've grown now to 46. Um there's been a couple things along the way that have helped. Uh the Super Bowl
[21:38] was Simoc cast this past year. That was a great one. Um right now, if you are a World Cup fan, uh you can watch that on Peacock in Spanish. Uh if you like reality TV, Bravo is to die for. There's
[21:55] so much great stuff on there. So, you know, we're moving in the right direction still. Um, now we're running everything in data bricks and the question that we should all ask ourselves after we've done something like this is did it work and like how do
[22:10] we know that it worked? So now is this is the punch line, right? Did we save any money? Yes, we saved money. Uh we reduced our monthly operating by about 30%. Uh total ownership that's including compute, storage, all that stuff. uh
[22:28] which is for us really substantial creates a lot of growth opportunity for us and again like keep in mind this is with the increased consumption that we have on our platform we're saving money uh we have more on-time delivery this was the other thing that
[22:43] it's just easier to uh have more compute available and give more options to our bash process about 25%. We were still we were generally running on time. Uh our on-time rate I think is was above 90% for the year but you know further
[23:00] reducing 20%. Good for us. The other one is these major events that come up when we need to prep for them the overhead in doing so is much lower. You just change the compute configuration good to go. uh and because of the way things are set
[23:17] up, it really makes it easier for us to do that as opposed to needing to go through multiple teams and things like that. And then finally, uh we did reduce data replication. So this was one of the things that we really wanted to do in doing this project was I don't want to have to copy data to Snowflake. I don't
[23:32] want to have to copy, you know, data from other places. You know, we also went through and did an inventory, which was really nice. So we got to identify data that we didn't use anymore while we were doing the migration, delete things, uh optimize more processes. So that was really good in terms of like as we look
[23:49] ahead. I feel like I should have completely just get rid of this and just write Genie, but it's a little much. So Genie is obviously a part of what we're going to be doing. We've already started trying to understand like how does Genie fit into our platform. We know that it's
[24:04] going to change the way people interact with data. You like dashboards. I don't want to say they're gone, but it's not the same as it was. Uh, so we're looking for those. We have some really good PLC's that we we've been able to execute on that by moving over to data bricks. That's been great. Um, this concept of
[24:21] having a knowledge discovery platform. One of the challenges that that I see still with us. Uh, I don't know if other organizations have this problem is the how where do I find the data? You know, it's like data bricks makes it a lot easier to just go to one place and search like I want to find silver video
[24:38] or silver user whatever the table is, you know, that that runs your business. Um, so we're really invested in trying to understand like how do we make that experience as seamless as possible so that our users can find what they need. Um, and then sharing across uh the MBCU
[24:55] Comcast ecosystem is a huge priority for us. So this ability to operate and share data across to Snowflake, you know, other data bricks instances, even proving that we're able to expose the data to something like BigQuery if we
[25:11] need to. This is all really positive for us and points the way forward that we can do these things in a way that we don't have to replicate the data still once again. So very very good uh you know uh payoffs for a lot of the work that we got to do here. Um, and with
[25:28] that I'm gonna close. If anybody has any questions, we got plenty of time. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.