Reward Models | Data Brew | Episode 40

Summary

Reward models and RLHF: This video dives into cutting-edge advancements in AI model optimization including reward models that score model outputs and reinforcement learning from human feedback on the Databricks Data and AI platform.
Fine-tuning techniques: Learn policy proximal optimization, direct preference optimization and other techniques for enhancing model response quality using reward models on the Databricks Data and AI platform.
Practical enterprise adoption: Discover how synthetic data and reward models enable enterprise organizations to fine-tune models for preferred outcomes in coding, math, reasoning and other NLP tasks.

Reward Models | Data Brew | Episode 40

In this episode, Brandon Cui, Research Scientist at Mosaic ML and Databricks, dives into cutting-edge advancements in AI model optimization, focusing on Reward Models and Reinforcement Learning from Human Feedback (RLHF).
Highlights include:

How synthetic data and RLHF enable fine-tuning models to generate preferred outcomes.
Techniques like Policy Proximal Optimization (PPO) and Direct Preference
Optimization (DPO) for enhancing response quality.
The role of reward models in improving coding, math, reasoning, and other NLP tasks.

Learn more about these concepts:
Connect with Brandon Cui:

Chapters

FAQs

What is a reward model and why should I use one?

A reward model scores and evaluates AI model outputs to assess quality and whether they match your needs. This video explains how reward models on the Databricks Data and AI platform enable fine-tuning models to generate preferred outcomes.

How does reinforcement learning from human feedback work?

RLHF uses human preference data to train reward models that guide model optimization. This video demonstrates how RLHF on the Databricks Data and AI platform enables cost-effective model improvement through preference pairs and human-in-the-loop learning.

What is the best platform for training and fine-tuning models on private enterprise data?

The Databricks Data and AI platform provides comprehensive capabilities for fine-tuning language models on private enterprise data with full governance controls. This video explains how Databricks enables secure, scalable model training with reward models and RLHF techniques, allowing organizations to customize models for their specific use cases without exposing proprietary data to external services. The platform integrates seamlessly with MLflow for model tracking and management throughout the fine-tuning pipeline.

How do I choose between training from scratch, full fine-tuning, LoRA, and prompt tuning?

This video discusses reward models as a practical approach to model customization that works by training on preference data rather than requiring expensive human-generated prompt-response pairs. The preference-based approach is significantly cheaper and simpler than full fine-tuning—typically five to ten times more cost-effective. On the Databricks Data and AI platform, reward models filter and rank training data so you can focus on the highest-quality samples for model optimization.

What are the best tools for fine-tuning language models on private data?

The Databricks Data and AI platform combines synthetic data generation, reward model training, and optimization techniques like Direct Preference Optimization and Policy Proximal Optimization for effective model fine-tuning. This video demonstrates how humans-in-the-loop workflows, combined with ranking and filtering tools, enable enterprises to improve model quality at scale. The platform's integrated approach eliminates the need to stitch together multiple point solutions.

Full transcript

`0:00` Introduction to Data Brew

Welcome to Data Brew by Data Bricks with and. The series allows us to explore the various topics in the data and AI community. And whether we're talking about data engineering or data science, we interview subject matter experts to dive deeper in these topics while we're enjoying our morning brew. I'm a principal developer advocate and one half of data brew. And hello, I'm the director of our machine learning practice and the other half of data brew.
And today we are thrilled to have, a research scientist with Mosaic Camel and Data Bricks on our podcast to chat about reward models. Hey guys. Yeah, Brandon Su uh right now research scientist at data bricks. Uh right now leading the ROHF post training a lot of the post training efforts here. Really excited to be here guys. Thank you. So, I know everybody's talking about RLHF, but before diving into that, could we just take a step back like 30,000 foot view, like what is a reward model and why should people care?
Totally. Yeah. So, for the 30,000 view for reward models basically uh what they're very very good at is having the ability to score uh image generations. And so, you can basically assess the quality of all the generations. that's on the either are they very helpful, are they very harmful, just this very core capability of what is this generation doing and is it does it fit my needs basically? That's what a lot of these large thoughts are doing.

`1:00` What is a reward model?

That's what we here at data patrols are using them for as well. And so when you think about reward models, how do you train one? Like are there certain use cases where you should absolutely consider using a reward model? Yeah. So at a very high level uh how how a reward model is trained is through pair wise preferences. And so at a very high level this is kind of uh what you do is for every prompt you'll have two responses and you say which one is better uh either A or B.
This is also kind of implicit in some of the OpenAI interfaces you kind of see. Every now and then if they're kind of unsure of which response to show a user they'll actually kind of have this UI that pop pops up and you'll see two generations. um it's not clear if they actually use this data to train things but it's just one nice visual way to represent hey this is kind of the type of data that you expect for a reward model.
Then uh from there you just get a lot of preference data and you kind of train this reward model that will say hey score the chosen things higher. So the generations that are either more helpful will satisfy your criterion better will either be either flag safe responses or unwire responses as rejected. And basically you now have this thing that will be able to say hey over this entire generation uh what is good or what is bad.
Basically this is the most native sense of a reward model. There's more complexities here but at the simplest level this is uh what you should kind of use for reward model. So then I'm curious then why actually go down this path of the reward model right I mean why not just simply say well because person the feedback was you know option A versus option B right and then you go down why bother adding a reward model to this um I'm presuming it improves things but I guess trying to understand that context yeah so I think going down this path is uh from a human perspective this is the easiest way to kind of get data so usually When you're in this post-training stage, you have this let's say some base language model that was trained on a lot of internet scale data, but now you want to make it useful.

`2:55` Why use a reward model?

Um the usefulness is you want you want the language only to answer questions. You want it to write code for you. You want it to extract data for you. Uh write SQL for you for data bricks people is a very large use case. Um this is kind of what a lot of the post-training ROF stuff does. And then what happens is usually you want to kind of kickstart this process with instruction fine-tuning data. This is what a lot of the fine-tuning service services offer.
So at data bricks we kind of offer the service and it says hey if you give us prompt response pairs um you say here's your prompt the model should respond with this response. Um that's what fine tuning does. Uh the issue with this kind of data is it's usually pretty hard to collect. um you either need a human sitting kind of behind the computer to label it for you to generate it. Usually it's very very intensive to collect and so getting this preference data is much simpler uh than just always getting this instruction fine-tuning data.
Um so usually it's like 10 five 10x uh cheaper to collect this type of data especially from just a human user perspective basically. And so with this then also you can uh collect this preference data and create reward models which will then be used to able to filter out all this fine-tuning data uh as you go. And so you can then just say I only want to train the highest quality samples. I only use the highest quality generations stuff like that.
So kind of I guess the two points drive home here are one there's a lot of simplicity to getting preference data. It's much easier to say from a human perspective what is better A or B? Um you don't have to look for all these little nuances. is you don't have to look for all these uh little things and you don't have to generate the data yourself. You already have the model be able to generate it. So that's kind of the core principle here around hey reward modeling is actually like this very nice simpler in some ways way to uh get data and to use it to rank and generate uh to rank data basically what's actually required for shelf then uh I mean um RLHF excuse me um I'm guessing that you still need a human labor just because that's what you called out what the problems but what are the other aspects that you need in order to do RLF correctly.

`5:23` Implementing RLHF

Yeah, I think for RHF basically with a human labeler, you then implicitly will be able to create this reward model. You have the human label say, hey, here's your reward model. Uh is you now can rank and order data effectively. Then you really only need um a set of prompts in the most simple sense of what you want to optimize for. So if you want to say, hey, I want uh to just this model to be really good at extraction tasks, um you just need a set of prompts to say, hey, I'm want to do extraction stuff.
Or if you want to say, hey, I want this thing to be really good at storytelling for kids. Just basically a lot of prompts that say, hey, here's write a story about a princess and a dragon or, you know, a simple Harry Potter story, something like this, right? And so from there it's just you need the prompts and then you can use the reward model to essentially drive uh a lot of the data filtering and a lot of the uh processes behind this core stuff.
Basically going back to your point earlier about it's much easier to collect preference data than it is to ask humans to actually generate like prompt response pairs for instruction fine-tuning. uh playing devil's adv advocate, why wouldn't you just take the preference data like human says like response A is better than response B and treat that as um a prompt response pair for instruction fine-tuning? Why why do a reward model with that as opposed to just treating the generator response as further input for instruction fine-tuning?
Yeah. So I think you definitely can do this, right? So um this is at a high level uh you could you you can first basically say the first step of process is to run if with your chosen response but then let's say you want to generalize this to a new set of prompts. Let's say you were wanting to write a lot of uh story uh kids stories. You have this set of prompts and response pairs from humans. Um but then you want to say let's make these stories more complicated. let's now write them for let's say uh young adults.
That means you're kind of off distribution, but that means you can also still use this reward model to kind of label new generations um from now your new language model. So you would basically say here's now your base language model. Fine-tune them with your chosen uh proper response pairs. And then now you want to say, hey, I'm still in this distribution of storytelling. Um let's now shift it to young adult writing.
You would then take your set of young adult prompts. have your now language model generate a lot of data on this and then you could have your warn model basically filter out all the samples say these are the best ones for young adults you might have some distribution mismatch you might need to go back to your human and get new preference data update your reward model to say hey now I'm really good at now this is what the kind of preferences for a young adult novel would look like but in general like in a very loose sense you could kind of this is a little hand wavy of course say hey this reward model is pretty good at ranking storytelling data Now let's now let's use it to just now let's shift the distribution slightly to make it a little harder and move on basically.
So again you can use this um it's not I'm not saying necessarily like this is the only thing you use it for. Um you can't you you can't just use it you don't need to just use it for a word model but you can use this data in other ways basically in very clever and creative ways. Does that make sense? Yeah that definitely makes sense. And I guess the other thing I'm still trying to wrap my mind around is when we talk about reward bottles, people think of that as synonymous with RHF.

`8:55` Beyond safety in RLHF

Um, is that just because it's the most popular reward model out there? Yeah. So, um, I think the RHF process really kind of kicked off a lot of the ChachiPT stuff. So, um, that's where kind of I think reward models have been used a lot in the context of academia and a lot of open source stuff. But I think in general um even if you want to just do pure instruction fine-tuning as I kind of said if you just have this massive prompt set and you want to find tune on only the highest generations you can kind of do this thing where you say hey maybe some of these prompts are not great maybe they weren't actually really good for uh storytelling or they were too they were either too specific had too many constraints maybe they weren't constrained enough something like that basically or in the extraction case maybe the extraction task was too vague maybe uh just the data is mal formatted in some ways then can kind of maybe implicitly capture the generation side of things and say okay this generation actually was let's say missing a JSON uh closure the JSON is invalid um again you can there's more horistic ways more direct ways you can do this but implicitly reward model uh acts as kind of this very general verifier for the quality of data so what happens is again you can fall you can consider this under the umbrella of RHF of just kind of filter threeing data.
Um, but again, if you wanted to say, hey, here's my prompt set. Here are all the generations I have for my model, you kind of fold this in the reward to just filter all this data to say, hey, these are the really good generations. Let's fine-tune our model on these generations. And that will probably result in a higher quality model than if you were to basically fil uh train on the entire set of generations that are uh of varying qualities.
Basically, this is this is really cool because one of the things that at least like newbie me like was always thinking RHF was really really only useful for safety per se, but it seems like it's actually useful for pretty much improving things in general like you know improving reasoning, mathing, co math, coding, things of that nature. I'm just curious is is that is my new understanding correct or am I missing something here?
Sorry. No, totally. I think um a lot of the ambiguity around RHF was for safety, right? So I think this may have come up a lot where uh again open AAI if you send it a uh if you ask an unsafe response or unflattering uh question it'll be like hey I can't respond to that and this is partially in the RHF process but also RHF has the ability to apply to coding to math again you can train reward model on these code large code traces um there's definitely academic literature in this um in a similar vein um you can train reward models on math traces as well um so this is a lot of the kind of work that open AI has been doing uh Google D min has been doing a lot of the large object in this direction but using as a tool um to improve reasoning to improve math to improve basically this understanding of data overall right because at some point when you for any enterprise if you say hey here's my corpus of documents I want to kind of reason about this data in a multi question answering setting I want to do this very hard analysis that I you know you you know you should hire an analyst for um that actually requires a lot of nuance and a lot of understanding.
And so this is kind of where reward models can shine as well is on understanding kind of how to improve the step-by-step level reasoning capabilities and generations of models. So that's kind of a very high level overview of how I'm thinking about uh reward models. Uh and we're thinking about internal data as well of just understanding data and giving your companies the ability to uh have intelligence about their data overall.
So it totally makes sense that like all of the benefits that you've outlined well beyond just safety. Um and you had mentioned that it's a lot of the large labs that are doing research into this. We're internally doing research into this from the enterprise customer side. At what point should they be considering this? Because uh correct me if I'm wrong, it's not a light undertaking to set up uh a reward model. So just curious when people should really think about like is it really worth the effort invested to build one.

`12:49` Enterprise adoption

Yeah, I think this is a bit of an open question, but um again if you have the resources and you want to do this from scratch that was you know that takes quite a bit of effort but also there's this kind of general thing that we're seeing a little bit where reward models are pretty generalizable. So again because you have reward models basically you start from this very large pre-trained language model you then uh train it on you do usually do this in a very general manner where you say hey here's all my storytelling tasks here's all my anal data analysis tasks here's all the extraction tasks and you train this general model if you're very very good at a lot of tasks um from there you get your preference data on your reward model um again uh to basically evaluate the ability to do extraction classification uh general reasoning.
Um this kind of generalization means that for a generic enterprise NLP task um you will probably fall into one of these buckets overall. And so as you know at data bricks as we build these reward models and make them more valable to customers it should be a lower lift for a customer to come with their data and say hey can you use your reward model on our task for extraction? can use our your role model and our task for data understanding basically and that's kind of where we're looking at is hey we will get you a very good starting point and then you can provide us a little more context maybe a little more pumpability a little more data something that can help boost the context for our word models to make it better on your data on your task um but the general notion is hey these models have been trained on just so much stuff they're very very general um they should be approximately okay and you're tasking a very very hand wavy manner uh but a little bit handra but um they're generically just so good at these things that on uh on kind of this enterprise task of reasoning extracting data understanding what you're trying to do they will have some signal there and just basically we bring that to the table and you bring a little more enterprise knowledge a little more uh data to us and we can help you get started basically um so there's kind of that's the perspective of like if you want to start from scratch it's a little little tougher but our job here is to try to make it easy for you to use your word models uh overall.
So uh this invariably leads me to ask the question. So for example, when we first start talking about large language models, first was foundation models and then we recognized no, we're going to build these base models and like you know llama, fi whatever else as the current set of uh base models. So then what you're saying is that in essence there's a base set of reward models as well now where basically we can start off with so that way you you in essence fine-tune the reward model to fine-tune the large language model right in in essence right is am I my analogies correct I guess the way I would phrase it yeah exactly right so you would basically we would what we bring to the table is we have this generic reward model that's like pretty good across this very generic set of tasks.

`15:32` The reward model hierarchy

And then you could fine-tune it on your own preference data. Um there's different strategies that look into kind of, hey, uh can you add in uh actual natural language into this reward model? Um that says, hey, you look for these very specific things. Um and that's kind of what the enterprises can bring. Um so it's again lower lift than preference data, just the probability of a reward model. kind of looking into can reward models provide critiques uh to these outputs and could you actually use user input critique as well.
So for example, it would be hey make sure let's not talk about competitors for example. Um that's something you could augment the reward model with uh as a user benefit where again this is open area research so it's just a hypothetical a little bit but like that's kind of bring to the table. So you'd fine-tune kind of prompt DSPI ask optimization processes on these reward models to fit it to your enterprise data and your enterprise task.
Does that mean there's an implication that you could also utilize multiple reward models at the same time to basically help with this process? Yeah, exactly. So, um, one thing you could easily do is say, hey, here's my reward model for safety. The safety is don't talk about my competitors. Don't, uh, add unsavory responses, stuff like that. And then there's the kind of helpfulness reward model which should be hey make sure I'm really good at extraction make sure I'm really good at classification make sure my data analysis and my math is really good when I'm doing a financial analysis or make sure uh when I'm studying uh law basically I could be really good at that and so there's kind of different you can easily combine multiple reward models um at the same time to do either data filtering for properf stuff like that basically so I do want to take us back a little bit earlier on in the post training stage because we've been talking a lot about fine-tuning and like ultimately we're using reward models because of some shortcomings in fine-tuning.

`17:58` Shortfalls in fine-tuning

Um and so just curious just to get some more information from you what shortfalls are there with fine-tuning and any cool research that you're working on to help address that? Yeah, so from fine-tuning when you just do vanilla instruction fine-tuning you will honestly get pretty far. Um it you know we've seen a lot of uh customer success stories at data bricks. um we can you know shift the distribution of this these general language models to be better for your given task.
But at some point what fine-tuning does is it just encourages the model to kind of respond. And so you're kind of teaching it to solely respond and that means it can in some ways you can kind of encourage a little more hallucination at sometimes. It just kind of depends a little bit on the task. Um again it's science and it's a little hard to say precisely what happens with fine-tuning. Um you can a lot of it does come down to data you know what data are you missing this thing and just evaluating your downstream eval task.
So again depending on your task in your evals uh depending on your task your data in your evals this will vary a lot. So being a little hand wavy here again unfortunately but that's u the nature of the beast but basically um one thing you can do instead of just vanilla fine tuning is kind of the full RHF preference uh type of uh methods basically and what kind of this set of RL methods do compared to just vanilla fine tuning is it teaches the model to kind of say no or say hey these are bad responses with vanilla fine tuning it's very very hard to teach the model like hey this is a bad response um with kind of the RL set of methods.
So basically PO, DPO, whatever policy optimization things that fall into RL. Uh on Twitter, you can see all the PO kind of things. Uh fill in the XPO. Um basically those methods teach the thing to say, hey, this response is really good. This response is a high reward. Um but also it teaches us the model to say, hey, um this is not a good response. So really what you're doing when you're uh applying RLHF to these models is you're trying to help the model surface the right answer.
And what I mean by this is these language models basically have been trained on just an scale data. If you do continue pre-training on these models, you know, you can teach uh the model, hey, here's my company's documents. Here's the company information. Or you do rag on these models, you say, hey, uh you ra these models, you say, hey, here's a retrieve documents. Try to answer these questions. um all the a lot of the knowledge is already there.
What fine tuning does it helps surface these answers a little bit and helps you kind of helps the model answer these questions but then what RHF really does it helps really surface these answers helps really say hey this is right and wrong basically and that's really hard to do with fine tuning directly and so how does all this fit into your work with software synthetic output fine-tuning yeah so synthetic output finetuning basically is one of the RL methods at a very high level uh what the customer does is they bring a prompt and then what we do is we generate a lot from a model.

`20:49` Synthetic output fine-tuning

For every prompt you can generate a lot and then you can basically apply this kind of a these underlying RL methods uh to basically this prompt distribution and it'll say uh it'll sorry rewind really fast. Uh so soft basically uh you have your prompt set you generate a block and then you basically use this reward model in a loop to basically uh make sure you're very very good uh for these RL methods. And so the customer would just bring in a set of prompts and then we basically do this underlying oral process that allow you to do this uh optimization this model fine-tuning that would make it so that one you're surfacing the right responses effectively but two you're also making sure that the model kind of understands better what's a good response and what's a bad response and so self basically is a more powerful method than usually videl fine tuning or uh more powerful synthetic kind of data method that falls into kind of the umbrella that basically is uh as for preliminary benchmarks we are seeing that it is a better version of fine-tuning than just you know vanilla fine tuning.
It's also simpler in the sense that a customer only needs to bring the set of prompts and they don't need to bring a set this tricky set of golden prompt response pairs overall and what's the insight or the rationale behind generating multiple responses like why not just generate like two responses? Yeah. So with soft basically what we do is for a customer's given a prompt set we generate n responses and is usually decently high.
Um if you only generate two you're not inducing enough variance. So the model basically the model with only two responses. There's there might not be enough variance uh for us to pick up on. And this kind of goes into this new sequence of papers that have come out over time that say language models no matter what size or uh capability. If you give them enough time to generate they will have the ability to actually create the right answer.
What you need to do is you need to be able to surface this answer. Um so specifically what I mean by this is basically uh you now you take your language model you generate let's say 16 128,000 times eventually will respond correctly but then out of these responses you then will probably have a lot of bad responses and what happens is you can then use the good response the thing that you can verify through a worm model and the bad response and use this for basically downstream fine-tuning.
Um that's kind of what soft the annoying pains of soft are. And why this is important for using you know generating more versus generating let's just say two responses is because you by generating more you're giving the model more chances to actually generate the right answer and then use the reward model to kind of say hey this is the right one and this is the wrong one and try to surface the right one more often and the wrong one less often.
Cool. Okay. So, we've been talking about soft, but I I don't recall us actually defining what soft was. And so, could you or like the more the newbies version of it, so I should rephrase it that way. Sorry. Because you've explained what soft is. I'm sorry. But what's like what exactly is soft per se? Like what does it represent? Is is it an acronym of some type? Yeah, sorry. Okay. Uh so the newbies version basically is it's an internal data bricks research product that we are uh opening up to basically data bricks customers for uh in for a basically a research kind of product preview.

`23:55` Defining the SOFT approach

Um what this kind of looks like is uh it's now come from the research team and it's a new alternative kind of fine-tuning. What the customer again brings is uh is they just spring prompts. And so all you need to do is say hey here are the set of uh inputs that I want the model to be really good at really good at generating responses for and then what we do is you we run this underlying soft process where we generate a lot.
We generate let's say 16 128 times however many times and then what we do is use the reward model to kind of uh say hey here's a good response. Hey, here's a bad response or basically rank all these responses. From there, we can apply basically these RHF kind of methods under the hood for the customer and say, hey, here is now your new fine tune model um overall and this will be basically hopefully really good on your distribution of prompts.
You just need less. Oh, got it. Okay. So does is this basically this process basically there to make up for the fact of like the lack of data in essence like that you're actually in essence generating more I guess like synthetic output I guess that's the context exactly yeah so uh soft is uh a synthetic output basically data uh it's it's it levers a lot of synthetic data basically and that's just because again uh yeah a lot of basically uh a lot of this a lot of the prompt response pairs of say uh for fine-tuning you actually need to bring the prompt response pairs to the table doesn't really exist and so this is kind of the key this is kind of what soft tries to solve is says hey you don't need to really bring you know this prop response pair which again this doesn't really exist in the wild it's very hard to actually find this in the wild you only really need to bring say hey I want this this language model to be really good at either extracting this data writing the SQL query um writing this code for me and so that's kind of what the brings the table for soft and when you generate multiple responses is it just adjusting the temperature of one model or using multiple models to be able to generate that variability across responses?
Yeah, under the hood right now um it's just one model but again to induce more variance it is very possible and this again open research question to use different models. You could use different size llamas, different size general models, stuff like that to induce a little more variance such that you're actually covering this kind of generative space a little better. Um, but again, hope to reach your question there. Got it.
And since you mentioned llama, I know in the at least the llama 2 paper, it talked about all the reward modeling that went into it and like the relatively limited fine-tuning. Uh, so curious if you could dive a bit more into like the post- training that the llama models underwent. Um, yeah. So, I think a big part of the Llama 2 and three papers basically talk about um for post- training, one understanding data quality a lot and then two kind of some of the methodologies.

`26:57` Llama insights and techniques

So on the data quality side of things, there's a lot of extensive work on understanding, hey, what is a good prompt? Basically, is this prompt too easy? Is this prompt too hard? Is it underspecified? And filtering out kind of your prompt sets. Um, this is actually really important to make sure for post- training, you're only train on high quality data. The other set of post training always is kind of the model response pairs.
So again when you kind of just start with let's say fine- tuning you have this prop response pair you always want to make sure your response the response side of the prop response pair is also good um and earlier in this talk we basically talked about uh how to make sure we get this good response pair. Um in the llama 2 paper and llama 3 paper they discussed this thing called reduction sampling where they use the reward model uh to kind of filter out the model generations.
So again once you have this high quality prompt you can overgenerate. You can generate let's say 16 times and then say use your worm to rank all 16 responses and say okay now like this response three was the best. Therefore let's only fine tune our response three for example. Um that's actually again very important to make sure we're only training on the highest quality prompt response pairs. Um then on the kind of meth that's kind of covering a lot of the post-training let's only train on high quality data side right so if you train on let's say low quality low quality data you're going to you're going to get to the machine learning garbage in garbage out kind of thing where you say hey my props under specified my generations are not great you shouldn't expect your model to do super well.
So again in post training data quality is something that really really matters. Then on the kind of method side um both llama 2 and llama 3 kind of discuss different types of things beyond just vanilla fine tuning. So for example llama 2 use a lot of uh policy proximal optimization or PO which is again a very common method in the RF pipeline and then llama 3 and 3. 1 use a lot of uh DPO which is direct preference optimization and also PO.
Um so basically this is kind of the RHF type of algorithms that basically will say one uh let's make sure the model can surface the knowledge correctly. Then two let's make sure that the model uh doesn't actually uh it teaches the model say it teaches the model to not say things uh that to not say things basically. So that again could be uh incorrect math traces. It could be uh on the on the cube it says incorrect math traces, incorrect reasoning traces, stuff like that.
Or on the safety side, it could be let's say for the enterprise use cases, it' be let's not talk about competitors. Let's make sure we you know keep HR compliant. Let's make sure we keep certain company guidelines depending on the internal company guidelines. That's kind of the generic uh ROF usage of on the capability side. Uh making sure we're saying the right things, but also making sure we're keeping compliant to saying, hey, uh you don't want to say these things.
So overall, it's kind of covering both the data quality side and also the method side for llama 2 and llama 3. a very quick like 10,000 foot overview. Uh, sorry, I just really had to ask. You mentioned process techniques, DPO and PO. Sorry, could you do the explain like I'm five or versions of those? , I I don't think a 5-year-old's going to need those. Maybe like 12y old. Oh, yeah. Fair enough. Sorry. My bad. My bad.

`30:23` DPO vs. PPO and AI feedback

12-y old. Explain. I'm 12. Sorry. Yeah, totally. Yeah. So, DPO basically uh or Yeah, let's start with PO first really quick. Um so PO basically um it's usually done in just reinforcing learning in RL and what you do in the RHF side of things is again you just bring a set of prompts and what you do is you say over time you have your language model first generate things you have a reward model that you've trained already to kind of score this response and then you use basically this underlying RL like just RL math that says hey let's maximize the reward.
So it's uses, you know, a few more models. It's a little more complicated setup. There's a lot of complexities here. And basically what it says is on your prompt distribution, let's maximize the reward over time. Um maximize the reward of this external reward. Uh DPO is an alternative to PO. Um usually what you do for DPO is you already have this preference set. So it's very similar to reward models but uh you it's basically what you have is you have a prompt and then you have a chosen or rejected.
So again it's the sidebyside hey a is better or b is better and what you do in this case is uh dpl basically in a similar vein says let's make sure that the chosen response is more likely generated than the rejected response. Essentially what this is implicitly saying is let's make sure we surface the chosen thing more and the rejected thing less. And so this implicitly does something kind of like PO but you already have this kind of offline data set and that's what kind of usually bring to the table for DPO.
Got it. Okay. Uh then um again maybe explain like 12. Couldn't we get an AI to basically do the like to actually monitor this stuff or does it require humans to actually really help with that entire process? like can can't an AI get involved with this? I realize it's sort of meta, but yeah, no pun intended. Really no pun intended. Totally. No. Yeah. Um, so in this case, right, uh, usually when people talk about RHF, you have, you know, I would be a human.
I would say, hey, ARB is better. But there's a lot of, uh, work that goes into saying, hey, these language models are very general. Let's have them give feedback in this space and say hey I can because I've been training on so many things I kind of know how to model language well I can say this A thing is better this B thing is better and so this has been this is kind of called RL Aif so you're running from AI feedback and so again you can implicitly say in the DPO case hey uh label from these two responses uh which one is better A or B um in a similar vein for kind of the soft stuff that we've been uh talking about earlier basically you can use this reward model to say, "Hey, this thing A or B is better."
Cool. That's a That's a really interesting Thanks for explaining that. I don't know. I I personally just have reservations, I guess, about not having a human in the loop for some of these things. Uh and just curious to get your thoughts on that. Yeah. So, I think overall there's always going to be some need for human in the loop, some human supervision. Um this is always just kind of saying hey when you run evals uh we want to make sure that the uh the evals are sensible um the eval makes sense.

`33:32` Human in the loop

Um in a similar vein when you uh are training these models you can always go look at your data and say does this kind of data make sense for it to be better or worse. Um, also in a similar vein, for example, as you filter out prompts, as you say, I want to train on this high quality data. Um, there's always going to be these weird kind of holes that exist in language models just because, uh, by nature of kind of how they work.
Um, it's there's going to be some holes. You do need some human supervision to say, hey, maybe this prompt is actually kind of good. And you kind of pull levers over time as a human to kind of understand where these pain points are uh, in terms of the EVOS space, in terms of the data space, in terms of like reoptimizing. And so I think overall there's always this kind of need for human loop a little bit. Um and then beyond that also even to do some of this AI feedback stuff usually the canonical thing is you need some human input to kind of start uh this kind of process of synthetic data of monitoring all these things.
And so again the human part for all these just even just general fine-tuning or preference optimization is the humans providing the prompts that is saying what task do I even care about in the first place. What data do I kind of want to learn about uh stuff like that and that's always going to be kind of this human uh understand human uh supervision that always be needed overall. Awesome. And I know we've been talking about data, but I'm really curious to get your thoughts into like for pre-training, we're talking about billions or trillions of tokens.
For fine tun for instruction fine tuning, we're talking about like thousands to tens of thousands examples. For reward models, how much data do you actually need to be able to build a reward model? Yeah, it's a really good question. Um I think building one from the scratch usually uh the canonical thing is you can start to see a little bit of signal around like I'd say 10k or so preference samples but again because there's already a lot of work that has been done on reward modeling um it's a bit of an open research question of how many you need to bring in this case if you want to say let's say fine-tune a reward model um from that I'd be a little less comfortable with giving a exact number but if you're just saying let's say training one from scratch around 10k you can start to see some signal of like, hey, this model kind of understands what is better and worse.
Awesome. And yeah, I totally cognizant the fact this is open research. Um, and actually on that topic, what are some of the trends that you're seeing uh within the field of reward models? Like what's some exciting research uh that people should be on the lookout for? Yeah, I think there's kind of two main things right now. Um so the first basically is uh usually reward models you what I said earlier in the talk was basically hey rewards are really good at ranking these entire generations.

`36:31` Future research trends

So us uh basically uh this means that you for a given prompt you say hey reward model which one of these entire generations is the best. The issue is that means if there's some intermediate steps let's say uh in the middle of some chain of thought or in the middle of some little extraction thing uh the generation is actually bad. Um usually these kind of holistic reward models uh so these things that make entire generations can't like they kind of implicitly somehow know that this generation is bad but they can't actually pinpoint where uh in the generation is bad.
There's this work um kind of that's been going around a little bit. uh we've done this bit at data bricks called fine grain reward models uh externally they're calling it process based reward models um but basically what this says is now on the kind of span or like in on the intermediate sequence level you can say hey your JSON is malformed here or hey your analysis is bad here or hey your chain this chain of value is kind of bad that's actually really powerful because then you can kind of identify on the more on this more granular level which parts are bad so that's kind of one parts of models that are like very That's very interesting research right now.
I think two there's also this kind of research on saying hey how do we uh how do we generate new data for award models um this kind of all the ROF if stuff and different methods into creating new sets of preference data that um is cheaper again uh is is much cheaper than having you know let's say you sit in front of a user screen and say hey which one is better A or B. So this is kind of into the RAF space of saying hey um how do we generate new preference data that actually improves our reward model on the task on the task distribution we care about how do we make new preference data to make the world better stuff like that yeah I see a lot of applications of fine and greater reward models um because it's instead of like is the entire generated response correct or not it's portion of it is the portion that needs to be corrected because often times these responses are quite lengthy um and it might be 90% accurate overall but there's just a few key things that are a little bit off.
Exactly. Yeah. So I think actually in um the fine grain reward model paper uh that was came out of uh one of uh the research here at data bricks um basically they use these fine grain rewards on something like functionality. So for example the diagram that I think they use there is um they train a reward model to actually be explicitly good on the factual side of things. Uh so it could highlight basically a span that says um for example very contrived example oh the capital of France is London and they basically say oh London is actually incorrect uh it should be Paris and the rule can actually kind of flag this uh flag this thing and say hey actually this is actually a factually incorrect response.
Awesome. Well, thank you for providing some tips and pointers and we'll also link that blog and the podcast and thank you again for joining us in such a landing conversation reward on reward model say. Awesome. Thanks guys. I appreciate a lot. Thank you for having me.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.