Beyond Frontier Models: Building Closed-Loop LLM Systems That Learn from Outcomes

Summary

Revamp AI builds a custom 'digital twin' model for every individual customer of the companies it works with, training and owning those models rather than wrapping a frontier model API — which means no rate limits, no overnight price changes, and no model swaps that alter product behavior.
The closed-loop system follows a train-act-observe-retrain cycle where live business outcomes — such as whether a message sent to a customer produced the desired result — become the training signal for the next version of the model, compounding improvement over time.
In benchmarks against top frontier models, Revamp AI's open-source closed-loop approach achieved a 40% performance lead, and the gap compounds over time as the model accumulates proprietary outcome data that a general-purpose frontier model can never replicate.

Beyond Frontier Models: Building Closed-Loop LLM Systems That Learn from Outcomes

Frontier models are powerful generalists, but the future of LLM applications will be won by systems that are specialized, measurable and continuously improving. Xinchi Qi, co-founder and CTO of Revamp AI, shares how Revamp uses open-source models, custom training, tight feedback loops, online evaluation and runtime guardrails to outperform frontier models on domain-specific production tasks. He covers how live business outcomes become training signals, how smaller models can become faster, cheaper, and more reliable than general-purpose APIs, and why the next generation of LLM applications will be closed-loop, governed, and outcome-driven from day one.
Talk By: Xinchi Qi, Co-Founder & President, Revamp AI ;

Chapters

FAQs

What is a closed-loop LLM system and how is it different from using a frontier model?

A closed-loop LLM system, as described in this video, is one where live business outcomes — what actually happened after the model's output was acted upon — feed back into the training pipeline to continuously improve the model for the specific domain it operates in. This differs fundamentally from using a frontier model API, where the model is trained on broad human preference data and updated on the provider's schedule without any knowledge of your specific business outcomes. The result is a model that becomes progressively more accurate for your use case over time in a way that a general-purpose frontier model cannot replicate.

Why would a company choose a smaller custom model over a powerful frontier model?

According to this video, smaller custom models trained on domain-specific outcome data can outperform frontier models on production tasks because they are optimized for the specific decisions that matter in that domain rather than for broad benchmark performance. A frontier model trained on human preference ratings learns to produce outputs humans rate highly in a general context, but it has no knowledge of whether its output for your specific use case led to a positive or negative business outcome. The speaker also highlights practical concerns: a frontier model provider can rate-limit access, triple the price overnight, or change model behavior in a way that breaks a product built on top of it.

How does the train-act-observe-retrain cycle work at Revamp AI?

Revamp AI's system trains a customer twin model, deploys it to act in production by generating messages or recommendations, observes the real downstream outcome of those actions, and then uses those outcomes as training signals to retrain an improved version of the model — typically on a weekly cadence according to this video. Because the system trains and runs its own models rather than calling an external API, every business outcome it observes stays proprietary and compounds into a data advantage that grows with each iteration. This continuous feedback loop is what allows the closed-loop approach to achieve and widen its performance lead over static frontier models over time.

What performance results has Revamp AI seen compared to frontier models?

This video reports that Revamp AI's closed-loop, open-source model approach achieved a 40% performance lead over the top available frontier models in benchmarks relevant to their production tasks. The speaker attributes this to the fundamental mismatch between frontier model training objectives — optimizing for human approval ratings — and what actually matters in domain-specific production applications — optimizing for real business outcomes. The gap is described as compounding over time because each production cycle generates more proprietary outcome data that further improves the custom model while the frontier model baseline stays fixed.

Full transcript

[00:08] So, hey everyone here. Uh, I'm Shin, uh, co-founder of Revamp AI, and I'm joined by my co-founder in the back, uh, Stephen right there in the back. Um, so before I jump in, uh, let me do a
[00:20] quick show of hands. Who here has actually gotten to try Fable Five by Anthropic? Okay. and keep your hands up if you still have access to it. Okay.
[00:33] Right. So, for most of the room is no. So, hold on to that because I'm gonna I'm gonna come back to it in a few seconds. Um I as this today because you know it's this is today everything is all about frontier models and it's the
[00:46] ones that everyone is racing to build on. Uh and I'm here to talk about whether they're they're really the right thing to build uh a real product on at scale. So let's get into it.
[00:59] So yeah, so I'm here today to talk about why Frontier model is not the way to go uh in both application at scale, cost and outcomes. Uh well, now there's a new one too, right? Right, which is
[01:12] accessibility. We all saw what happened with anthropic and fable 5. And so why am I a good person to talk about this? Uh I've done early day AI research at MA in Montreal under Joshua
[01:24] Yosua Banjo's lab. uh and have repeatedly applied AI at scale for the past eight years. Uh the reason many of you haven't heard of me yet before today is because AI research and finding product market fit are two very
[01:36] different things. And so speaking of product market fit uh I want to take you know it's very important for me to talk about what we're doing today as at revamp we have
[01:48] found our early traction product market fit. Um so at revamp our product is we build a customer twin for every customer of the company that we work with.
[02:00] So for every customer a company talks to um these companies that we work work uh work with um some of you know uh for example cognition is a customer of ours
[02:13] so it's enthropic um and and the team building the frontier itself are using us for this and the reason we can deliver that for them is because we're building the twins oursel we train them we run them and we
[02:26] own them uh they're not just like a rap or an API that we're renting about a token they're actually ours and we don't send the customer context to a frontier model and and hope it gives us the right
[02:39] answer, right? Uh so we train the systems um and we control how it learns. So nobody can rate limit us uh or triple our price overnight and no one can really swap the model underneath us and
[02:51] change how our product behaves. you know, not throwing a any if you have any uh you know, uh incidental anecdotal experience with a model just like not
[03:03] behaving the the same way the next day or the same the next hour. Uh that's that's what I'm talking about. So what is a customer twin? Uh for us, it's a model built around one actual customer
[03:16] uh rather than a audience segment. And over time it develops a understanding of how that person that person responds to the company. Say someone that usually opens uh email late on a Sunday night.
[03:30] Discounts really work on them, but they respond when there's a real reason to care. So you could take that history and pass it to a frontier model. Um and the model will probably make a better
[03:43] decision than most human would uh because of it. But that's still, you know, context and the model is reading the history you gave it and ma making another guess. Really is that's really what's happening. Um the twin goes
[03:56] further because it learns from what happens next. So every time it sends a message, we see the real outcome. Now how do we train this the these customer twins um is the part that
[04:08] actually matters. It's the loop and agent harness uh that we built in house and we're I'm going to
[04:20] talk about it in three steps. So the steps one is the spec is to spe is to specialize um we take an open o a open weight model uh and train it on uh on on one specific
[04:32] customer's behavior. So it knows them not the average of a million people. Uh the step two is to measure. The measuring part is that every message that it sends gets scored live in the
[04:45] real world and the score isn't just like okay was it just a nice message right um it's if the the real KPI is if they actually converted yes or no um and the
[04:58] step three is how the model improves over time so every one of these outcomes becomes actual the best training data you can possibly have for the next version of the same twin. Uh then it
[05:11] runs again. You know, it's going to send, learn, and send a little smarter next time. And all of this happens uh under the very important piece which is guardrails. So the twin can adapt, but
[05:26] it doesn't have unlimited freedom. Um the harness we develop in house keeps it within within the compy rules. Uh and our eval catch the bad outputs before they ever reach a customer. So if the
[05:39] system is uncertain and steps outside of the the boundaries that the customer has decided uh it falls back on um on this output instead of just improvising. Um so that's what makes it possible to
[05:54] put you know specialized model in front of real customer at scale. Um it can learn and adapt but does not make up the rules. So if you think about it, what are the evals, right? The evals comes directly from the our customer
[06:07] themselves, which is a first s firstand source of data in the best. Um yes.
[06:22] So the point of this you know um the harness and loop is that uh every message that the twin send is an experiment and the conversion that they buy or that they click um that's the label. So that's the answer to the
[06:35] experiment and every me every single message the twin sends it learns something real and true from it. Now if you hold that next to a frontier API right you make a call uh it gives you a decent answer or a great answer and then
[06:48] it forgets you completely. So you still have to build your harness in house and probably like you know your uh identity resolution or customer graph and so for us these signals gets
[07:00] embedded into the model weights themselves. Now let me go to the next page.
[07:16] So, you know, you may ask, okay, you're, you know, we're running open source model. Uh, does this does any of this actually work or is it just a nice theory, right? So, I put I I went away and put some some real numbers. Um,
[07:29] so we put our customer twins head-to-head against human team running email marketing, right? Real marketers who are really good at their jobs. And the twins get about 40% more conversions uh than human-run like agencies that
[07:42] are, you know, email marketers. Um, and I want to be clear, it's not 40% better open rate or anything. It's 40% more people actually buying more from our customers. Then we made it harder.
[07:54] So we actually tested against the our twins or or on open model uh against top frontier models. Um, and the twin still wins by about 15 to 20%.
[08:07] And the part I love the most is that it does all of that for a small fraction of the cost. Uh, there's no So, if if tomorrow revamp we switch to a Frontier model, we'd go probably bankrupt in in
[08:19] two weeks. So, you know, the thing about those number is they're not fixing a gap. They're they're not a fixed gap. Um, a human team is is about as good as today as it's going to be the next quarters.
[08:31] the twin is getting better um every single week. So that lead won't stay at 40%. Um the longer it runs, the further ahead it's going to get.
[08:49] And what's cool about this is that this is not some new breakthrough stuff there. There there's nothing like new breakthrough to build this. Most companies that are AI native today already have this this data. um they already have this the model running in production and they they usually know
[09:02] what happened after the model acted. Uh the problem is that those piece usually do not talk to each other. Um the model makes a decision and the business see the results then the learning stops really. So closing the loops uh meaning
[09:16] taking that result and feeding it back to the system. So that's the shift and you you stop you kind of like stop treating the model uh like a tool you call and start treating it like a system that improves uh from its own work. So
[09:29] if you're already running on data bricks uh a lot of the infrastructure already here as shown in this uh slide and so the hard part is not buying another tool uh it's wiring the system that every outcomes um so so makes so so the
[09:42] outcome makes the the next decision better.
[09:54] So let me pull back a second and tell you where I think all of this is going. uh because I don't think it's just us at revamp. I believe in a few years from now um every serious AI product is going to look like this and it's not going to
[10:06] be like oh we're going to call you know the frontier API and we hope it's going to to work. It's going to be a closed loop, something you can actually measure, something you can govern, and something you can, you know, that you
[10:18] can point to a real business outcome from the very first day. And that system will actually compound uh instead of a just a feature that sits there. And as like the gap gets wide, the team that builds this way uh that own their loop
[10:32] are, you know, just going to pull away from the teams that are still quote unquote renting intelligence, um today. So, you know, it's a it's a bet on how this whole thing plays out.
[10:52] so here's a choice that I want you to leave with. So, on one side, you can rent the smartest generalist uh on the planet by the token. Um, and it's like generally brilliant. I'm not going to pretend otherwise, but you can't always
[11:04] get it. Uh, and you can't always afford it. So, and and it just might change on you next Tuesday for whatever reason. On the other side, you can build something that you actually own, something that gets smarter and cheaper over time, but
[11:18] also safer. Um, so one of those that you're renting and the other one is yours. And you can wait the next month or the next year waiting for the next big model to drop and hope it fixes everything. Or
[11:31] you can ship something today that's already better than it was this morning and will be better again tomorrow. So at revamp we decided to just stop waiting and I think the teams in this room that decide the same thing are going to be you know really glad that
[11:44] they did. Um so you know that's it from me. Uh and if you're building with AI and you keep hitting that wall uh the scale the cost the outcomes
[11:56] um and now the access as well that's exactly the problem we spend our days on. Um we give every our customer like we give every company that we work with we work with and our customer a twin. So
[12:09] if any of this landed for you uh please come find me or my co-founder after. Uh generally you love to talk to and respond to our questions. So yeah I'm Sheni. You can grab me here and or email me at my email. Uh thank you so much for
[12:22] listening to me.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.