Fox One GenAI Recommendation System: Solving Cold Start at Scale

Summary

Fox One's recommendation system solves three cold start problems — creating coherent shorts from 213 daily hours of live content without editors, ranking new content without mature engagement signals, and personalizing feeds for new users — using a hybrid architecture that applies GenAI where data is ambiguous and deterministic constraints and traditional ML for production reliability.
Gemini identifies candidate clips, Databricks AI Query generates training labels and embeddings at scale through Lakeflow Jobs, and an XGBoost model distilled from LLM reasoning delivers final recommendations within a 600-millisecond SLA using low-latency Lakebase lookups.
Because sparse early-user data makes A/B testing unreliable at launch, the team evaluates recommendation quality using AI judges — a GenAI-based evaluation approach — as a practical alternative for assessing relevance when behavioral signals are insufficient.

Fox One GenAI Recommendation System: Solving Cold Start at Scale

Fox One's AI-powered recommendation system solves 3 cold start problems: creating coherent shorts from long-form video, ranking content without mature engagement signals, and personalizing feeds for new users. Fox and Databricks built a hybrid architecture that applies GenAI to ambiguity while using deterministic constraints and traditional machine learning for reliable production performance.
See how Gemini identifies clips, Databricks AI Query generates training labels and embeddings at scale, and Lakeflow Jobs distill LLM reasoning into an XGBoost relevance model. The walkthrough also covers PySpark training, Databricks Model Serving, semantic features, cosine similarity, Lakebase lookups, a 600 millisecond personalization SLA, AI judges, and GenAI-based evaluation when A/B testing is unreliable.

Chapters

FAQs

What are the three cold start problems Fox One faced when building their recommendation system?

Fox One had to solve three distinct challenges: creating coherent short clips from long-form live programming before editor capacity existed, ranking new content before it had accumulated mature engagement signals, and personalizing feeds for new users who had no prior behavior history. Traditional ML approaches failed because all three problems share a fundamental data sparsity constraint.

How does Fox use Gemini and Databricks to turn live content into personalized shorts?

Gemini identifies candidate clip moments from Fox's live programming, and deterministic constraints are applied afterward to fix clip boundaries and ensure coherent content. Databricks AI Query runs through Lakeflow Jobs to generate training labels and semantic embeddings at scale from those LLM-identified clips.

How does the Fox One system meet a 600-millisecond personalization SLA?

LLM reasoning is distilled into a lightweight XGBoost relevance model for production scoring, avoiding the latency of calling a large language model at inference time. Lakebase provides low-latency lookups for real-time personalization, with semantic features and cosine similarity used for efficient candidate retrieval before the final ranking step.

How did the Fox One team evaluate recommendation quality without reliable A/B test data?

Because new user data was sparse at launch, traditional A/B testing was unreliable for measuring recommendation quality. The team used AI judges — GenAI-based evaluation — as an alternative to assess relevance and engagement quality when behavioral signals were insufficient to draw statistically significant conclusions.

Full transcript

[00:08] Hi everyone. I know we have the afternoon right after lunch slot, so hopefully we make this interesting enough that you can stay interested in what we're saying. But I want to start with a quick show of hands. How many of you have gone on your phone to check one thing and then found yourselves 30
[00:24] minutes later watching shorts instead? Yeah. I've definitely been there, too. What you actually encountered was a state-of-the-art recommendation system. Today, we're going to be talking about
[00:39] how we built that for the Fox One application, but under some real world constraints like not having that many users to start with to give us data and how we use Genai to solve this entire system.
[00:57] Great. So, it is nice to officially meet you all. My name is Reini Gupta and when I don't get stuck watching shorts on my phone, I am an AI for deployed engineer at data bicks. I've been working on the Fox project specializing in traditional ML and Genai use cases.
[01:13] Hi, I'm Nicole Catalino. I'm a product manager on the emerging technology team at Fox. And I'm Peyton Dairo. I'm a senior AI engineer on the FDE team at Data Bricks. and I've served as a technical lead for many projects across uh the Fox brand.
[01:31] Awesome. So, just to give you all a quick tour of what we're going to be talking about, we are going to start with the opportunity at hand for the Fox One application. Then, we're going to go into how we approached actually implementing this. Next, we're going to talk about how we created content for
[01:48] the app. Then, we're going to be talking about how we personalized it. And finally, we're going to be talking about how we evaluated the whole system, ending with some takeaways that hopefully you can bring to one of your next projects.
[02:08] All right, thanks Reneie. So, at Fox, the heart of our business is live content. And if you've been to the Fox activation at the summit here, you have probably already deciphered that yourself. At Fox, we generate over 213 hours of live original programming every
[02:26] single day across 27 networks. And the value of that content does not end when the live broadcast does. Our shows generate thousands of bite-sized moments with the opportunity to inform, engage,
[02:41] and entertain. And but identifying those moments Sorry, resetting. Um, identifying, clipping, tagging, and distributing those moments would require an army of
[02:57] editors and producers working around the clock to do it manually. And if we want to meet consumers where they are today, the challenge becomes even greater as they increasingly expect content to be personalized and available on demand. So
[03:12] at Fox, we already have the content, we already have the audience, and we already have the trust. Now we need the technology to unlock the full potential of what we create. By applying AI, we can transform that live programming into
[03:28] the moments, experiences, and stories our fans want, when, and where they want them. And that was the vision for Fox 1. A personalized Foxonly endless scroll short form experience designed to create
[03:44] more moments of connection with our content fully powered by an autonomous clipping pipeline. And not to skip ahead, but today that pipeline identifies nearly 3,000 potential shorts and publishes 1,200 of them every single
[03:59] day. If Fox one is the connective tissue across the Fox ecosystem, then shorts become the discovery engine. Most consumers only experience a small fraction of what we produce. A sports, but the power of the shorts experience
[04:16] on Fox one is really to introduce them to everything else across the Fox portfolio. Now, a sports fan who subscribes in September, just in time for the NFL kickoff, might stick around for the unique wit of Greg Gutfeld or Jesse Waters. And a TMZ pop culture
[04:34] enthusiast might learn that their favorite celebrity is competing on the next season of Special Forces. Instead of asking our fans to navigate the portfolio themselves, we're bringing the moments directly to them. In that sense,
[04:51] shorts become the top of the funnel, not the destination itself, but the mechanism by which we can introduce them to the full breadth of Fox content. But turning that vision into a reality presented unique challenges. To start,
[05:08] we needed the engineering capacity and the specialized knowledge to build at the required pace and level. Enter data bricks through the forward deployed program. We, Reinie and Payton, were able to embed directly within our existing
[05:24] technical teams and bring that deep AI knowledge and help us deliver against super aggressive timelines. The next hurdle was data. Unlike Instagram and YouTube, we were not
[05:39] launching with billions of user engagement signals. This was a net new content type on a net new surface with almost zero historical consumption behavior to learn from. That created a data sparity problem that compounded at every layer of the system. First, we had
[05:57] to determine what even makes a great Fox short. With 213 hours of programming every single day, not every minute is engaging and not every segment deserves a short. We had to identify the ones that would most likely resonate with our audience and we had to do so
[06:13] automatically at scale. That meant teaching AI to do something we as humans do intuitively. We all can recognize narrative arcs and generally understand when a conversation
[06:28] has reached its natural conclusion. Teaching machines to make that same judgment is just a little bit harder. Second, we had to rank our content across fundamentally different domains. How do you compare a game-winning touchdown to breaking news? And most of
[06:44] our library is news forward, which means time decay is brutal. One story can captivate attention for hours and then disappear overnight. And then finally, we had to personalize for users we know almost nothing about. And given the
[07:00] nature of our content, user preferences are constantly shifting. Weekday and weekend consumption patterns can look completely different depending on the season and the news cycle. So, we were not just solving a single cold start problem. We were simultaneously solving three.
[07:17] And each one depended on the others being solved correctly. I'll let Reinie explain how we approached that. Thank you so much, Nicole.
[07:33] So, we had a clear idea of what the vision looked like for the Fox one application, but now as the engineers working on this, we had to sit down and think about how we're going to go about doing the implementation. We wanted to start out to see if any simple approaches might just work out for us.
[07:49] That lasted about 5 minutes. All of these single approach ideas quickly broke down. Let's start with our first idea. We wanted to know if maybe we could just wait for organic engagement and just get enough users to generate real signal, but we had business
[08:05] requirements and we had a contract that said we need to ship something and we definitely didn't want Nicole to fire us. So, this was going to be way too slow for us. The next thing we thought about trying was heruristic solutions. One example of
[08:20] this would be to serve the most globally trending videos to all users. But if this was going to be something like Tik Tok or YouTube shorts, this would be pretty boring in comparison for most people and users wouldn't actually end up adopting this.
[08:36] Finally, we could go with the most common solution of 2026 and just throw an LLM at the entire application end to end, but we had cost requirements and latency requirements, and this was simply going to be way too slow and way too expensive for us to use.
[08:53] Like most engineering problems, the first idea didn't work. The second one didn't either, and neither did the third. So, it was back to the drawing board where we really had to think about, well, we know what doesn't work, but is there anything we could take away
[09:09] from some of these approaches and maybe combine them together to create a hybrid solution? And it turns out that was the real answer. We couldn't use Genai as our entire system, but we could use it combined with other approaches as this
[09:26] enabler to bootstrap the three main tenants of our application, content, training, and evaluation. The first thing that we did was we generated structured shorts from the long form video that existed at Fox by
[09:42] using Genai. The next thing that we did was actually create our training data set for our personalization machine learning model when we didn't have real user engagement to create a normal training set by using engagement data and we use Genai to
[09:59] infer these early preference signals. Finally, when we couldn't rely on traditional AB testing, we were able to evaluate the quality of our entire system by using Genai in the meantime so we could start getting directional
[10:14] signal much sooner. So going back to the beginning, we knew we had to build a short form video platform, but one thing that was different about Fox was it wasn't like
[10:30] Tik Tok or Instagram reels where we had users who were uploading the content, but rather we needed to create the content from what we already had. So we needed to think about how we were going to do this in a way that scales up.
[10:48] As Nicole mentioned earlier, one thing Fox uniquely had was this large library of long- form video, ranging from TV shows to sporting events to news broadcasts. But all of this was unstructured. There were no boundaries and there were no labels either. So this wasn't going to be usable in its initial format. Our
[11:06] first idea was to use Genai as the structuring layer where we used Gemini which had the additional multimodal information to find starting and ending points within the long form video that would make for good shorts. We also
[11:22] asked it to label and describe the videos and find these moments that have good narrative coherence and context around it. Now, we had exactly what we needed, structured outputs, and we were ready to rank, personalize, and evaluate,
[11:38] but we didn't want to go too fast. And what we were creating here was going to be visible to end users. So, we needed to include this quality layer where we combined human reviewers with AI judges to make sure that what we were generating was actually any good.
[11:55] But if it all looked good, we were set to publish to our downstream applications. We thought this pipeline was pretty creative and working well until we sat down and started watching some of the videos.
[12:19] It might not be playing. Technical difficulties. Let's see. Okay, video is not there, but I will describe what the video was instead. This is real life, guys. So, it's some
[12:35] cute dogs, but it starts in the middle of a sentence and they're just playing in a park and no one has any idea what's going on. So, kind of spoils the punch line, but basically we were getting a lot of shorts that were starting in the middle of a sentence and we weren't
[12:51] really sure what was going on. So, we weren't getting the context as end users to understand what these shorts were supposed to be about. And we found out that many of the shorts that we were were being generated with the pipeline that I just showed ended up being rejected. So now we had this high
[13:08] rejection rate problem to deal with because we needed to be producing enough shorts compared to the amount of long- form video that we had. As good data scientists, the first thing that we did was to sit down and look at the data and figure out what the real
[13:25] failure mode is. We looked at AI judge rejection data as well as human reviewer data and they both overwhelmingly pointed to the same thing. 70 to 80% of rejections came from poor
[13:40] starting and ending points i.e. clip boundaries. So at least we knew what we needed to fix. Uh we needed to think about why AI was choosing these poor starts and ends. When we started to look at how to fix
[13:56] this, one thing that quickly became clear was that simply making better prompts wasn't going to solve this. Let's go back and look at what we were starting with. In our original implementation, we had this long form video and I mentioned
[14:13] that Gemini could choose any starting and ending timestamp. But what does that mean? It means it can choose timestamps that start in the middle of a sentence or end in the middle of a thought and can create some weird results. I'm sure many of you have worked with Jennai at this point and know that you sometimes
[14:28] get weird unexpected stuff out as a result. But once we started looking at it this way, the problem became much clearer. We were giving Gemini a little bit too much creative freedom with the editing. So instead,
[14:45] we thought about what deterministic data we had. And one thing that we did have was transcription information with timestamps. So instead of letting Gemini choose from any boundary, what if we only told it to choose from valid boundaries that we
[15:00] know are the starts of a sentence, end of a sentence, or a change in topic. Seems simple enough, but this change led to much better results. more natural beginnings, more complete thoughts, and we are creating many more successful shorts.
[15:23] So, what did we actually do here? If I were to summarize it, we combined Genai with determinism. We already know Genai does an amazing job of finding these interesting moments by combing through all of the long form video. But as we know there were issues
[15:39] that came out of just using Genai by itself. So we injected a layer of determinism where we restricted the st search space to be valid boundaries and we even leaned on good oldfashioned coding to do post-processing alignment.
[15:54] The result was a huge win. We had much cleaner boundaries, more coherent clips, and best of all, we were able to continue working at the scale that we needed to to be producing enough content to make the app work.
[16:10] Now, I'm going to hand it over to Payton to talk about how we actually personalize this content. Thanks. All right, so at this point, we have a large library of high quality shorts. Our next problem is trying to figure out
[16:27] how do we get the right short to the right user in real time. So before I go very deep into the details, I want to give a high level overview of a recommendation pipeline. So you can break this down into four main phases. First we fetch the user profile. This is where we pull
[16:43] everything we know about the user, their watch histories, their favorites, their location. Then we generate a candidate pool of 10,00 plausibly relevant shorts that have been created recently in the last few days. Next, we score those videos. We give
[17:00] them a relevant score. In our case, we used a trained XG boost model for this. And finally, we rerank and serve. So, reranking is important because, for example, I am a Cowboys fan, but I don't want every single video on my feed to be
[17:15] about Dak Prescott. Maybe I want some Dallas Mavericks content, maybe Texas barbecue. Very targeted, but not duplicates. So, the interesting thing about personalization isn't the shape of the pipeline. It's how we did phase three,
[17:30] how we created a high-quality model with very little user data. So, classic cold start problem. We started where any data science team would start. We structured our data. We did feature engineering. We threw it out
[17:47] stated there at models and hope for the best. Uh but the core assumption of this playbook is that you have an established user base with deep watch histories and interaction data. When you throw that into state-of-the-art models, you get highly accurate personalized
[18:02] recommendations. But that wasn't the case for us. When we started training the model on our data, it was barely better than random. And we had a hypothesis that it wasn't a model issue. It wasn't a data structure or feature engineering issue. We thought
[18:18] we had a data label issue, but we had to prove it. So, we conducted a blind audit where we took a random sample of 100 user video pairs and we labeled them either zero or one based on whether or not we thought
[18:34] the video was actually relevant to the user. So, these were videos actually served to users on the platform. And then we compared those labels to two other signals. One was early user watches, whether or not the users actually watch those videos. And then the other, we had an LLM do the exact
[18:50] same thing that the human labeler did. Saw the same data. It gave a score between zero and one. We used a float for this, like a confidence score. And then we used a threshold to turn it into binary 01. So the results were very revealing. The user views were 50%
[19:08] aligned with how we labeled. So very similar to that random result we were getting in early model training. But the LLM was 91% aligned with the human reviewer. So our hypothesis at this point is we're at the early stages of a brand new feature.
[19:24] We don't have a mature user base. So these early watches aren't signal. It's just pure noise. The UI is still changing, undergoing a lot of changes at this point. Our content library is rapidly changing. So this early behavior, we just can't really use it.
[19:40] So obvious next question, why didn't you just use an LLM? So when you're building a feature for a consumerf facing application, latency becomes extremely important. So in this case, we had a very strict SLA of 600
[19:58] milliseconds and that covers the entire personalization pipeline I covered. So fetching the profile, generating the candidate pool, scoring, re-ranking, and not to mention many other unseen latency contributors that are shown on the screen. So in our ideal scenario, we
[20:14] have 120 milliseconds to 200 milliseconds maybe to score 10,00 videos. So the reason why we can't use an LLM is a single score prediction from the LLM would take over 1,00 milliseconds. So no
[20:29] reasonable amount of engineering is going to close that gap to get us down to this 200 milliseconds for 10,00 scores. All right. So we had a model that really knew what we needed at this point, but we couldn't use it at serving time, but
[20:45] we thought we could use it at training time. So we built a LakeFlow job. This is a daily pipeline that takes a random sample of user video players from the platform similar to that blind audit uh that we conducted.
[21:01] And again, we take everything we know about the user and the video. We combine that into a single prompt and we feed that to an LM. So the LM had a structured output and the output was two things. One, the score, a float between zero and one, how relevant the video is
[21:18] to the user and then two, a rationale. So this is a natural explanation of why that score was chosen. And that's important because it makes the data set auditable. So if at any point we find there's some kind of systemic biases in our data set, we have something to ground ourselves in to do some
[21:35] additional prompt engineering. And the difficult part of this isn't um the difficult part here is that we're not labeling 100 rows of data. So we're starting with tens of thousands then hundreds of thousands up to
[21:51] millions eventually. So we need a scalable solution and the engineering and orchestration that goes into scaling up LLM inference at this level that takes a lot of work. So there's a really nice function on data bricks called AI query where it just acts like a SQL
[22:08] function. You run it across a spark table and it abstracts all of the batching concurrency rate limits that you would otherwise need to pay attention to. you just don't need to do that. So any engineer working with LLMs and data bricks definitely look at AI query. Um another cool thing is you can
[22:25] generate embeddings as well and we did that in in this project um to generate embeddings for videos and uh elements of our user profiles. Um so yeah that's a great tool. Okay so at this point we have a large
[22:40] high quality training data set. Um now we need to figure out how do we use it. So this is where we use distillation at a really high level. Distillation is where you take a large teacher model, you train it, and then you use the output of that large model to train a smaller, more efficient student model.
[22:56] So in our case, that training data set pipeline I just went through, that's our teacher model. The output is the score. We feed that into our XG boost training pipeline. We also used LakeFlow jobs for this. And we're using Pispark for distributed training. So like I said,
[23:12] we're planning on scaling up to millions of rows to train this model. So we need something that's scalable. So Pispark is super important for that. Once we have a trained model, we take the artifact for that model and we serve that in model serving and pure Python. So we're using PIS spark to scale
[23:29] training. We're using pure Python for fast serving on uh model serving. And the results were really great. Our area under the curve was 0.94. Our F1 score was 089. So for most classification
[23:45] models, this is really great. So this was a huge breakthrough for us. We have the best of both worlds. We have the complex reasoning of an LLM in the small package of an XG boost model. All right, so I want to talk a little bit about how we trained it. Um, you
[24:01] might be surprised we only used eight features. So you can kind of break these down into two categories, temporal or semantic. I have a few examples up here. Uh for temporal features, we use video age. How many days old was the video created? Uh and what was it evergreen or not? So evergreen is like is the
[24:21] relevance time bound. So a cute video of a dog running around is pretty much always enjoyable to watch. Um news about a football player being traded to another team, maybe that is only relevant for a day or a month if it's a huge player. Um, so that's useful for the model. But our two like definitely
[24:38] most important models were uh important features were semantic features. So if you think about favorites and watch history, a user can select dozens of favorites, sports, football, cowboys, Mavericks. And then we limited to the
[24:54] last 30 videos for watch history. In order for a model like XG Boost to use this, you have to convert these strings into embeddings. And so you could have dozens of embeddings for each user if you just left them individual embeddings like
[25:09] this. You're going to have a very complex training pipeline and you're going to have problems at serving time. So we did a cool trick. We took the average embedding for all of our favorites and then we took the average embedding for all of our watches. And we found the cosine similarity between those two average embeddings and the
[25:25] video that was being served. And so cosine similarity, it's a score between zero and one. The closer to one, the more similar it is. So it turns out that's extremely effective for a relevance model uh for XG boost.
[25:40] Okay. So at this point we have a very accurate very fast scoring model but for personalization to work the whole system has to be fast. And so as a reminder we have 600 milliseconds to go through fetching the user profile building the candidate pool scoring ranking.
[25:58] So data retrieval is critical. So if we use just a traditional analytical store to fetch these pre-generated embeddings uh and to fetch user profiles, we're going to blow our budget just with retrieval. So we use lakebase that is data bricks postgress
[26:15] transactional layer and it allows us to do this call in uh milliseconds not seconds. So lakebase was absolutely critical for allowing us to host the personalization engine natively on data bricks. Uh and finally it this is in production. And this isn't a notebook
[26:31] demo. If you go to the Fox1 app, you go to the shorts portal, the videos that pop up on your screen have gone through this process that I just walked through. So, at this point, we have a good scoring model. We have a good personalization pipeline. Now, we need to figure out, is it ready to actually put in front of users? And so, that's
[26:47] what Reine is going to talk through. Thank you so much, Payton. All right. So we knew that we got really great training metrics on our XG boost model and we even met the latency requirement of being under 600
[27:04] milliseconds. But there was one thing we still really didn't know which is whether the videos that we were creating and the feeds we were creating are actually enjoyable to our end users. And this was an important question to figure out.
[27:21] The way that people normally test this kind of thing is through AB testing where you give one set of users a feed that does not include personalization and then another set of users a feed that does include personalization. Then you compare the engagement metrics
[27:36] between the two. Turns out evaluation and AB testing is also kind of a cold start problem and doesn't work that well at launch. At the beginning, we had no stable metrics. Our early engagement data was
[27:52] noisy and really unreliable for us to attribute causality to our model serving the personalized feed. In addition to that, remember we're building a brand new application. So, the product experience was going to be continuously evolving and that added in
[28:09] some more confounding variables. Finally, we had cross functional teams that were working on making this app a success that were all iterating simultaneously. All this to say, we needed some kind of better way than traditional AB testing
[28:26] to get some confidence that our model was working for personalization as intended. What could we do? Well, we could go back and use Genai as a simulator for user preferences.
[28:42] This allowed us to iterate much faster even given the uncertainty that I was talking about earlier and we did this in two different ways. The first was through independent scoring where we had feed A that did not include personalization and we sent that
[28:59] to an LLM along with user information for someone who might receive that feed like their watch history, their location and then we asked the LLM to give that feed a score from 1 to 10. Then we repeated the same process independently but instead using feedb
[29:15] that was personalized for that user. Now we had numerical information which was great. We could run statistical tests to see if feed B was actually better than feed A. But we wanted to be extra sure since we were using Gen AI.
[29:31] And so we also incorporated a second approach head-to-head comparison. We still had feed A and feed B, but instead of sending them independently to the LLM, we sent them together. And we just asked the LLM, hey, if you're this
[29:48] user and you've watched these things before, you're located here. Which feed would you actually prefer watching? And between these two methods, we found out that our personalization model was indeed in join in in making the user experience much better.
[30:09] Now I'm going to hand over to Nicole to talk about some takeaways. So what was the result? First, let's rewind a little bit. On February 4th of last year, Fox 1 was announced on an earnings call. Development kicked off the following week. And by August 21st,
[30:26] roughly 6 months later, we were live. That's not a ton of runway for what we were trying to build. And hopefully it's clear by now. What made that possible was the use of Gen AI. And if we revisit the three cold star problems we started
[30:41] with, identification, ranking, and personalization, the pattern showed up everywhere. To start, we used AI not only to identify content, but also to evaluate itself. Today, we deploy AI judges against every short that's
[30:56] created. They ultimately determine what's published and help us maintain high editorial standards at scale. Second, we use it to manufacture ranking signals. It estimates engagement potential and predicts content longevity. Together, those scores pair
[31:13] with business logic to provide a common framework for comparing our content, news, business, weather, sports. And third, we used it to map into our existing ontology that allowed us to connect every short to the players,
[31:28] teams, leagues, shows, and personalities that users selected during onboarding. In each case, Genai filled a gap where traditional data did not exist yet. And
[31:46] it's not working in a vacuum. We validated and guided it at every step of the way. In fact, more than 5,000 shorts were reviewed internally across multiple quality dimensions prior to launch for offline training. The real value came from coupling AI speed and scale with
[32:03] humans editorial judgment and expertise. And it's working. Users watch shorts are two times more engaged on the Fox One app than users who do not. Content starts are greater, total minutes watched are higher, and session length
[32:19] is longer. More importantly, we're seeing audiences shift across the portfolio as we hypothesized. Sports fans are engaging with news. News cohorts are discovering entertainment and short form is driving deeper exploration into long form. But perhaps
[32:36] a bigger takeaway beyond just Fox One, as Payton and Rainey have alluded to, building an AI product is one thing. Building an AI product that can operate reliably in production is another thing entirely. And what made that possible was not throwing AI at every pro, but
[32:53] being intentional and deliberate with where and how we applied it. So I'll pass it over to Reinie to close out with some of the principles that guided that approach. Thank you so much, Nicole. So this is kind of the important part where we're getting to the thesis.
[33:09] I've been talking a lot about where we use Genai and where we didn't. But if I were to summarize it in one sentence, it would be that we use Genai for ambiguity but never for ownership. The first thing that we talked about is how we created shorts by using Genai.
[33:27] But if you remember, this only worked because we incorporated those deterministic constraints to make this something that we could actually rely on in production at scale. The next thing we talked about was how we created our training data set by
[33:43] using generative AI to infer those early preference signals. At the end of the day though, what we were serving in production was XG Boost, a traditional machine learning model that was able to work at scale. And finally,
[33:59] we talked about how we evaluated our feeds offline by using Genai, but we still relied on good old statistics to do the number crunching and make sure our results made sense. If you zoom out on our whole system,
[34:14] a lot of it actually isn't Genai. From orchestration to our machine learning ranking to our policy and safety rules ending in our final personalized feed, we only use Genai surgically as an enabler.
[34:30] To be more specific, what we did was, sorry, the slides are not the updated ones, but I'm going to skip ahead. There we go. What we did was essentially play to
[34:45] Genai strengths and instead of using it as the entire system, we used it as this force multiplier that made the system possible. Now, many of you in the audience today might be working on personalization systems, but I'm assuming not everyone
[35:02] is. So, I wanted to genericize these takeaways so they can apply to a variety of different projects that you might be working on. We sat down and thought about what the four most important things we learned from working on this were and here they are.
[35:17] The first is if you're ever in a situation where you're asked to implement some machine learning or Genai use case and you don't actually have the data you need in the beginning and your boss is saying they want you to ship something, you can use Genai to create the data that you wish you had. So you can make some progress even with limited
[35:34] data. The next thing is think about how you can constrain before you optimize. Better prompting can only take you so far, but introducing guardrails and constraining your LLM can actually lead
[35:49] to much more reliable results in a production application. Next, design the feedback loop first. We specifically incorporated this evaluation methodology even before regular AB testing wasn't reliable so
[36:06] that we were able to get confidence about our model before we had a bunch of unhappy users complaining to us. And this is really powerful to help you iterate and put forth the best version of your model earlier on. And finally operationalize don't just
[36:23] prototype. All of this was possible because we were working with this really excellent platform that made all of what we worked on a lot easier and a lot more reliable. Bear with me as I go back to the last slide that was supposed to be the last slide.
[36:40] Pretend you didn't see it. All right, we've had some technical difficulties for my slides today. I'm getting some bad luck, but it's okay. So we have essentially gone from this really cool concept to something that is today working in production and I wanted
[36:57] to highlight some of the specific tools from data bricks that made this possible so that maybe you can use them on one of your next projects. We were able to scale our Genai workloads with our millions of calls to create that training data set by using AI query. Didn't have to worry about
[37:13] rate limiting. We didn't need to worry about scaling up. This just made it extremely seamless for us to do this. The next thing was real-time personalization. The real key to our latency success was by using Lakebase to
[37:28] do these low latency lookups. We definitely would not have been able to meet our ambitious 600 millisecond latency requirement if we were not able to use a system like Lakebase.
[37:45] And finally, we were able to deliver in production by utilizing data bricks model serving. Specifically, we didn't have to worry about downtime when rolling out new models. We were able to incorporate drift detection and really make this something that is a production grade system that we can rely on.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.