What’s Next After RLHF?

Summary

RLHF and InstructGPT — co-created by the speaker — were a massive commercial success with roughly 100% of LLM usage running through RLHF-trained models, but the technique was also a research failure because it optimizes for human approval ratings rather than for accuracy, causing systematic sycophancy and hallucination by design.
The speaker argues the AI landscape is in a bimodal state not because of disagreement about whether AI works, but because 'assistance' tasks — where a human reviews the output before it matters — are fundamentally different from 'automation' tasks where outputs act directly in the world, and current AI excels only at the former.
The path to AI actually transforming the economy lies in moving from the assistance era to the automation era, which requires engineering AI systems for reliability and correctness rather than for human preference — a fundamentally different design discipline from what produced today's frontier models.

What’s Next After RLHF?

RLHF was a massive commercial success — roughly 100% of LLM usage is through RLHF’d models — but it was in many ways also a research failure. Let’s talk about how it conquered the world, how it defied its creators’ expectations, why AI is in the bimodal state it’s in (is it a bubble or a machine god?), and how to make AI actually transform the economy.
Talk By: Diogo Almeida, CEO & Co-Founder, TypeSafe AI ;

Chapters

FAQs

What is RLHF and why was it both a success and a failure?

RLHF (Reinforcement Learning from Human Feedback) is the training technique that taught language models to follow instructions and produce outputs that humans rate highly, and it underlies essentially every LLM in production use today including ChatGPT. According to this video's speaker — a co-creator of InstructGPT and RLHF at OpenAI — the commercial success was extraordinary, but it was a research failure because optimizing for human approval ratings produces models that are systematically sycophantic and prone to hallucination by design. The technique teaches models to tell humans what they want to hear rather than what is correct.

Why does AI seem so powerful to some people and overhyped to others at the same time?

This video argues that the apparent disagreement between 'AI is transforming everything' and 'AI is overhyped' camps is actually a misdiagnosis of a real underlying divide: current AI is genuinely excellent at assistance tasks where a human reviews the output before it matters, but it fails at automation tasks where the output acts directly in the world without human review. People interacting with AI as an assistant experience real productivity gains, while people expecting AI to replace autonomous end-to-end workflows encounter frequent failures. Understanding this distinction, the speaker argues, is the key to correctly interpreting what is actually happening with AI.

What comes after the ChatGPT era of AI?

The speaker argues in this video that the ChatGPT era is defined by AI as an assistant — a tool that augments human work with a human always in the loop to catch errors before they matter. The next era, which the speaker calls the automation era, requires AI systems that are reliable and correct enough to act autonomously without human review on every step. Getting there requires a fundamentally different engineering discipline: designing for reliability and correctness rather than for human preference ratings, which is what RLHF optimized.

Why do today's AI models hallucinate and how can that be fixed?

According to this video, hallucination in today's frontier models is not an accident — it is a direct consequence of training on human approval ratings, which rewards plausible-sounding answers over accurate ones. Human raters cannot always distinguish a correct answer from a confident-sounding wrong answer, so the training signal reinforces fluency and confidence rather than factual accuracy. Fixing this requires moving away from human preference as the primary training signal and instead designing training systems that optimize for measurable correctness on real-world tasks.

Full transcript

[00:08] My name is Diego Almeida. I will give an intro to myself later on. I didn't expect this to not have an intro, but as you guys all know, the title of my talk is what's next after RLHF? And um
[00:20] I I believe this to be a little bit like a clickbait version of I think a more important question, which is what is next after AI's ChatGPT era? And a lot of people think like, oh yeah,
[00:32] ChatGPT is old news. And actually, the next era is that of Claude code. I don't believe so, and I will explain why. And I actually think that they're a lot closer than people think. And I want to extrapolate what's going to happen
[00:45] into the future, hopefully in a way that's very informative to everyone. Why listen to me? My name is Diego Almeida. I was at OpenAI for almost 5 years. I joined
[00:57] before LLMs were a thing. I was a co-author and co-created basically all of OpenAI's greatest hits. It's a little sad that they seem like they're in the downswing now, so maybe it is the golden era of OpenAI entirely, but fingers
[01:10] crossed, hope not. But all of these GPT-4, ChatGPT, RLHF/InstructGPT, these were all my babies, especially the last one. It's a It's a very special project to me.
[01:22] And what's weird is I am one of the few people who made ChatGPT who also hates on ChatGPT. Not to say that I think that ChatGPT is bad. I use it every day. I love it as a product. And also, I think
[01:35] that the root of the entire over-promise under-deliver that is going on with AI is actually partially at fault to things that I made. So, believe me when I say this that this is
[01:47] my circus and these are my monkeys. So, most important question I think for everyone, and I I think this is the thing that like everyone should be trying to figure out constantly. If you work in AI, if you work adjacent into
[01:59] AI, if you are unfortunate enough to read Twitter, um what's actually going on because there's feels like there's just such differing points of view and they're all like so not listening to each other and so un-unified and like if
[02:13] you believe one camp, which you know, I'm going to call like the cult of the machine god, not only is AI going well, AI is going insanely well. Like it's like scary. Like we should be afraid and we should have a coup at Open AI because
[02:26] we are so good at Google proof question answering that like the world is going to end. And from this camp, you just see AI saturates every benchmark. It gets super human at everything that we measure. Not
[02:38] only does it get super human at at everything we measure, the time it can work autonomously not only improves exponentially, but improves exponentially better than everyone thought. This is like, you know, to these people, there's reason to panic at
[02:50] how powerful AI is becoming in the world. But uh for cult two, the cult of AI is a bubble, AI is just going insanely poorly. If you look at basically all like actual statistics,
[03:03] you see it barely affecting GDP growth if not at all. It's probably going to be negative if you take in like the circular financing out of account and you keep seeing people say like, "Hey, if AI was so good, why can't it do this super basic task?" And it keeps failing.
[03:17] You even see uh AI leaders like Sam basically say that it's going to it's a basically giving up on the dream of an AI based industrial revolution. You know, they changed the definition of AGI from doing most of the world's work to
[03:30] making, I think, a hundred billion in profit or something like that, which is a very underwhelming thing for the field. And you see this even with, you know, Claude is making a lot of revenue as far as we can tell right now, but even then
[03:42] you're not seeing the actual impact. Like how is software transforming despite, you know, making coding uh coders like X percent more productive? And the only thing that these two camps can like have in common is that AI is
[03:57] insane right now. And what I would like to present to everyone, and hopefully this is a useful way to think about it even beyond this this talk, is the sane view of AI. Ideally, the
[04:09] simplest possible explanation of what is really going on in the world of AI. The way I propose this is on one hand, let's put all of this stuff where not not only is AI good, it is unbelievably
[04:23] good. It's phenomenally good. It is so good that you would never want humans to do these things again. Like Chat GPT, think about like how smart people are. Think about Chat GPT, and think about like what percentile human Chat GPT is
[04:36] at answering Chat GPT queries. I'd argue it's 100th percentile. There's no human being I I would I would love to talk to certain human beings, but there's no human being I would literally replace with Chat GPT. It is just that good, and I would argue that this is true of
[04:48] basically anyone. Um a a lot of these things you also see not just not the same level of like 100th percentile, but like Quad Code is way better than average programmer. It But like for some reason those average programmers still have jobs.
[05:01] On the flip side, and you look at like the too bad to be useful, you have use cases that on the surface seem way easier, way more basic, but for some reason not only are they bad, whenever people try to implement them with AI,
[05:15] they perform perform so poorly that AI needs to be ripped out, and you basically have like nothing new happening in the field. Like drive-thrus, accounting, taxes, like basic data entry. Why isn't AI doing all
[05:27] of these things? And I feel like this is like a a major question that camp one needs to answer for the right side. Oh, my cursor's here. And but and camp two needs to answer for the left side. And I think the question for everyone is not which camp are you part of, but what
[05:40] explains this line? Given a new application, is it likely to be too good to be true, or too bad to be useful? And my understanding is this stage is not great for Q&A, but if anyone gets excited at any point in the
[05:53] conversation, I would love that. It might go a little crazy, um, but I will give you my best guess of this if no one's super excited. Um, my simple explanation is the stuff in the left is not to just a task that
[06:07] happens to have a human in the loop. The goal is to please a human in the loop. The goal is to assist the human at something. Like in ChatGPT, the goal is not have correct answer. The goal is human is pleased with the result of
[06:19] ChatGPT such that ideally they come back later on and they have like a positive impression of ChatGPT. Um,
[06:31] the stuff the stuff on the right, the human actually is something you'd prefer not to have. You actually just want to like straight up automate these things. You want to just have this be so basic that you can like program it once, build the SaaS one time, and it's just in the
[06:44] background forever. No humans needed. It just does the thing and the world is like a better place for that because work has been automated. And I I I love seeing like similar things, um, like
[06:56] like and to have that comparison. Uh, AI chatbots, customer service chatbots, are doing quite well financially. Um, and they improve customer, um, satisfaction scores as far as I know,
[07:09] but they are still just slop slop cannons at the FAQs. Like all they do is they throw you at the FAQ, and they don't let you they don't take any decisions cuz that's dangerous, but people are pleased with that and that they do a good job. You know, like if you actually want something in the FAQ,
[07:22] they will do a better job than a human. But on the flip side, if you want something to actually like edit your credit card number or, you know, change your account type or cancel something, AI can't be trusted with that because it is not reliable. It is it'll write a
[07:35] really good answer. It will be polite, but that thing might have a hallucination or worse. So, this results in what I would say my first lesson for the talk today, which is today's AI is incredible at the human in the loop
[07:48] stuff. It is really good, but it is not good at automation. Um, the dark way I would phrase this is you should give up on automation. I'll tell you why later on, but I believe that to be the takeaway. But, especially do not trust
[08:01] it with decisions with stakes. This is like the number one pattern that people fail in where they're like, "Oh, it's pretty good at this close by thing like customer service with slop, maybe it can also cancel accounts or issue refunds or
[08:14] something like that." And this is the constant mistake people make. We I'm happy to talk about this more later on, but like please be aware. This AI is not ready for that no matter how good it is at math.
[08:26] While getting into what's the next after RLHF, I want to talk a little bit about what RLHF is and work our way up to that. In case anyone doesn't know, RLHF is the man behind the the curtain for chat not
[08:40] just chat GPT, but basically every LLM that is served today. As far as I know by percentage of traffic, 100% rounded of LLMs are RLHF. Some of them are
[08:52] slightly trained in other ways, but they all clearly have the same flavor as as RLHF, which you will soon see. And how it works is going to answer our questions about like what's really going
[09:04] on in AI. Um, we as in me and my team at OpenAI wrote this really great blog. If you want to know the technical details, it is on here, but I will say that for the purpose of this talk, this is super
[09:16] boring. Um, I'll summarize it for you guys, which is that it is a cycle of collecting human preferences and optimizing for human preferences. If you don't believe me, here's the same thing annotated. You know, collect, optimize,
[09:29] collect, optimize. I'll wait a second cuz people are taking photos. Um, but the high-level loop is still this. And I think it's really great in contrast to like stare at this for 5
[09:43] seconds and then also re- ask ourselves the question from the previous section. Why do all LLMs require a human in the loop? And the answer, when you see how they're made, becomes like like way more
[09:55] simple. It's because we put the the humans into the loop of optimization. Our goal is to please humans in the loop, and that explains all of the behaviors of today's LLMs, good and bad. Um this explains why overpromising is a
[10:09] feature. This is an old study, but the results still generalize today, and they probably will generalize to RLHF for all time because human preference will always be better than results because the thing they optimize for is for human
[10:22] preference. It just so happens that human preference is sometimes correlated with results, but sometimes it's not, or sometimes it's easier for the RM to say like, "Eh, it's close enough." or "It looks right." or maybe my labeler cannot figure out what the right answer is.
[10:34] But, this is a feature. This is a feature of RLHF that is like non-removable, and I would argue that LLMs over promise under deliver a schism occurred because of ChatGPT, not GPT-3,
[10:46] which is like the grandfather of all of the LLMs. I love this example as what RLHF does on the hallucination front. Um sending ChatGPT an audio file of
[10:59] farts and asking it to critique your music, and it just it just goes crazy. It's a it's a it's a straight honest reaction of ChatGPT, but this is what RLHF does. This is why you get to like, especially in the instant mode, you get
[11:12] like emojis, bold, italicized, you know, making you sound smart, you know, blowing smoke up your, you know, what? Um the sycophancy. These are all natural consequences that were engineered and architected. We made these decisions. We
[11:25] did not realize the ramifications of it, but we made these decisions because we thought this would make AI super powerful, which it has, but it has great downsides that we did not foresee but are like obvious in hindsight.
[11:37] Psychophancy, hallucination, and just like being overconfident because humans prefer that so unbelievably much. And it's a very hard balance to have. Like I have a lot of empathy for people post-training in the big labs right now
[11:50] because you know, like they're trying to do like an impossible thing that we set when we made these algorithms. You can't optimize for human preference and be calibrated. So, where does that leave us on what's next
[12:02] after RLHF? I would argue that what we're actually talking about is what's next after AI's assistance era. Like we are in a world where AI is used pretty much entirely for assistance. And
[12:15] the reason why Claude Code is not what's next is because guess what? Claude Code is also assistance. You might notice that there is a human in the loop for Claude Code, and Claude Code's goal is not automate engineering. It is make
[12:27] engineers pleased such that they use more tokens. Um there's a subtle distinction. Talk to me afterwards if you want to talk about that. This is like that's a bit too advanced for this talk. But um I want to focus on what's next after this, and I would guess anyone paying attention,
[12:40] it's probably obvious. Probably it's going to be the era of automation, right? Like we're going to actually, as a field, figure out how to make the LLMs actually good at calibrated decision-making so that they can run in
[12:52] the background and machines can actually just do work. So, how this is going to go? Um uh I I'm going to I'm going to slow down. I I might get too excited, but I'm going to talk about software. I'm going to
[13:04] assume people here like software. I even like building software. I like looking at software, and I think software is like an art form. It's beautiful. It's awesome. The fact that you can like have these LEGO pieces that you build with,
[13:16] and then you can combine them and then duplicate these pieces infinitely, and then combine them together and compose them infinite It's like so cool. Software is also the language of automation. As far as I can tell, this is how automation is done.
[13:29] It is how you make a machine just do things. So, it is kind of like the script for the machine to do the repetitive tasks where you find like, "Hey, a human would do this." Or maybe a human wouldn't do this, but like it if a human was like 100 times faster or
[13:41] cheaper or like 10 million times faster, I don't actually know. You would have a person do this. And software is like that translation into the task. Um software is incredibly valuable. See all
[13:53] of the SaaS and like how expensive tickets are here. Like obviously software like is produced a lot of value in the world, but here's the twist that I don't think as many people think about. Have you noticed that software is
[14:05] basically exactly the same in the era of AI than after AI? Have you noticed like look at all the SaaS. Have you noticed that it's like the behaviors, what it expresses. It is the same level of smart
[14:17] except sometimes adding a chatbot latched on, which is super underwhelming. But you know, I don't want to judge if you guys work on that. But like you know, if we have commoditized intelligence, you'd think that something better would happen. Remember like playing with ChatGPT for
[14:30] the first time or whatever your first chatbot was and think about like, holy how different is the world going to be? And you know, that was like what, 4 years ago and the world is basically the same. Um we AI pioneers, I don't know if I can
[14:43] say we there, but hopefully I can. We used to think software would get smarter. The way we described it is like, what if like Jeff Hinton could be in every if statement? What else could you program with that? And I think that
[14:56] is like the dream that people actually keep on fighting for, but keep on failing at because AI is optimized for assistance. And now we have a world where software is very cheap. You know, I love the phrasing of just-in-time software.
[15:08] But it is basically the same level of expressibility as normal software just built on the fly. Like yes, it's easier to script cloud code to do things, but the things you would ask it to do are
[15:20] are the same as you could do before. And that is that is underwhelming. Uh it it's it's very different than what everyone thought we could do with with AI. And I think that this is going to be inevitable, which leads me to my third
[15:33] lesson, which is that real AI engineering is going to be the future. Like tomorrow's AI will be for automation. It will not be our life shift, and that's what's next after it. And we should be preparing for a world with smarter software.
[15:47] Um as you guys might have known, I work at a company. We are still hype safe, so that means we're in stealth. We are not going to exactly tell you what we're going to do, and that's probably better cuz I don't know if people want an ad right now. But our question was, what if
[16:02] the AI stack was redesigned for reliability and automation? What if all of those original design decisions we had in chat GPT was, instead of make it work for human preference, was make it work for automation? How do we make this
[16:15] such that it can like run silently in the background such that like we don't have to worry about it. You know, something that could be in the critical loop of prod, which uh I'm sure you guys are very smart. I would wager none of you have AI in the critical prod loop
[16:27] right now. Um and it's not because I don't believe in you, it's cuz I don't believe in AI, and I know AI really well. Um and the question was like, how feel free to disagree. Um but you know, like we are
[16:40] I think this is like a really ripe research direction, and this is what we've been working on for about 2 years. We had like some great results we're releasing soon, but please reach out if you want to work with us. I don't know if people here are looking for jobs cuz
[16:53] again, the tickets are really expensive. But if you want to be like designing or building like this smart software, talk to us. I actually feel like there's like an entire ripe world of like new AI
[17:05] systems engineering just waiting to be discovered into like what are the patterns that could be done reliably. Just like you know, once upon a time people had to discover like what really scales with software. And I think it's going to just be like like a like a
[17:18] phenomenal time for automation. Cool. Um, that's my last one. Hell yeah. Excellent. Would love any questions. If not, thank you for listening.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.