Composing High-Accuracy AI Systems With SLMs and Mini-Agents

Summary

• Mini-agents accuracy: Small language models and mini-agents composed together achieve high accuracy by reducing hallucinations through specialized reasoning.
• Memory and tuning: Memory RAG and memory tuning techniques inject determinism into probabilistic AI systems, improving reliability for enterprise use cases.
• Agentic workflows: Real-world examples including text-to-SQL, factual reasoning, and code analysis show how to compose mini-agents into larger AI systems.

Composing High-Accuracy AI Systems With SLMs and Mini-Agents

Talk by Sharon Zhou, CEO & Cofounder, Lamini
For most companies, building compound AI systems remains aspirational. LLMs are powerful, but imperfect, and their non-deterministic nature makes steering them to high accuracy a challenge. In this video, we’ll demonstrate how to build compound AI systems using SLMs and highly accurate mini-agents that can be integrated into agentic workflows. You'll learn about breakthrough techniques, including: memory RAG, an embedding algorithm that reduces hallucinations using embed-time compute to generate contextual embeddings, improving indexing and retrieval, and memory tuning, a finetuning algorithm that reduces hallucinations using a Mixture of Memory Experts (MoME) to specialize models with proprietary data. We’ll also share real-world examples (text-to-SQL, factual reasoning, function calling, code analysis and more) across various industries. With these building blocks, we’ll demonstrate how to create high accuracy mini-agents that can be composed into larger AI systems.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and agenda
• 2:36 Challenges of AI accuracy
• 6:31 Keys to reliable learning
• 12:32 Building agentic systems
• 17:58 System design for agents
• 21:09 Value of agentic AI
• 23:26 Scaling and compute laws
• 28:16 The open AI stack

FAQs

Why do large language models hallucinate and how can I reduce it?

LLMs generate non-deterministic outputs because they work probabilistically. Memory RAG and memory tuning techniques add compute at embedding and fine-tuning time to ground models in factual data, significantly reducing hallucinations.

What are mini-agents and how do they improve accuracy?

Mini-agents are specialized small language models built for specific tasks using fine-tuning and reinforcement learning. When composed together in agentic workflows, mini-agents provide high accuracy on complex tasks like text-to-SQL and code analysis.

What is agent orchestration in enterprise AI systems?

This video explains that composing multiple mini-agents requires careful orchestration to prevent error compounding across AI calls. By building mini-agents using small language models trained on domain-specific data, then connecting them in agentic workflows with memory RAG and memory tuning techniques, enterprises can create accurate, deterministic systems for complex reasoning tasks like text-to-SQL and code analysis.

What are the latest agent architecture models for building agentic systems?

This video presents breakthrough techniques: memory RAG reduces hallucinations using embed-time compute for contextual embeddings, improving indexing and retrieval. Memory tuning uses fine-tuning and reinforcement learning with Mixture of Memory Experts to specialize models on proprietary data. These techniques build foundation for mini-agents that compose into larger, high-accuracy agentic systems.

How do I build high-accuracy AI systems for enterprise?

This video demonstrates breakthrough techniques including memory RAG (which improves indexing and retrieval), memory tuning (which specializes models with proprietary data), and agentic workflows that compose mini-agents for complex reasoning.

Full transcript

0:00 Introduction and agenda

Hello everyone. Sorry I'm a little bit late. Hope you're having a great day at data bricks. Um so I'm uh and I'll be talking about composing high accuracy AI systems using small language models, SLMs and mini agents. So devising these smaller agents from either open- source models or um a set of different model calls. Um let's see. So a little bit about me. So Really good to meet you all. Um I'm the founder and CEO of a startup called Lami that builds infrastructure post-trading platform that includes fine-tuning and reinforcement learning to improve uh often open models, SLMs, and building out mini agents.
Uh my background before this was I was um computer science faculty at leading a research group in generative AI uh where I also did my PhD in generative AI with. Um I teach get the great pleasure of teaching u millions of people online in generative AI with deep learning AI and Corsera. Um and uh before this uh I was the product manager at Google kind of working on Google cloud uh with FE um and it's not announced yet but actually I am working on something new now um and I'll be sharing a little bit more after you know talking about composing AI accuracy about some thoughts I've been having um from that like what how does you know composing AI agents actually help us with AI progress overall and how does an open ecosystem help us with that?
So, I'll touch on that at the end. Um, if you want to stick around and hear some thoughts or I I'd love also to hear your thoughts. Um, so here's the agenda. I'm going to first talk about high accuracy AI agents, how to compose them, kind of the learnings um I've had working with different enterprises about that. Um, why factual accuracy is really hard. You know, getting these models to not hallucinate really really difficult.
Um, so how do we get them to be a little bit deterministic? How do we get them to be, you know, a little bit deterministic, inject a little bit of that determinism into a deeply a highly probabilistic process? Um, and then of course talking about how do we scale these agents and what is necessary in this world for us to be able to scale these further um both for each of our own applications but also AI at large. All right, so starting with the first part, what you guys came for.
Um so according to Gartner you know the stat of a third of enterprise apps will be powered by generative AI and agents. Um I actually think this is way too low now but uh here is the stat and of course there's so many different systems out there still. So of course um maybe this is this is the right target. Um people often ask me if we just step back why do these models hallucinate? Why do they make stuff up? Why do they get so much right and sound so human and yet mess up very simple things like the accurate date that climbed the Golden Gate Bridge which is clearly in its training data because it's on the public Wikipedia page.

2:36 Challenges of AI accuracy

Um and the reason is actually fairly simple. It's how it's trained. Um it's uh it's not just the data you feed it but how it's being trained when you do feed that data to it. And so when I ask a general model like what year did climb the Golden Gate Bridge and if it starts to output starts to decode, he climbed it in and it needs to then sample the next token. Um a model that hasn't been trained will sample a date looking thing just as likely as sampling the token for a cat.
Okay, that's not great, but it hasn't been trained. It hasn't seen anything yet. So after you've trained it, maybe it'll sample things that look like dates, right? more likely than cat and sample maybe a little bit less likely but still more likely than cat. Something that looks like a number. Okay. Um and so it's relying on similarities when it's operating on that date. And similarities are really powerful. Okay. So like even though it's get it might get the date wrong here.
The correct date is 1981 but it might sample 1970 because it kind of looks about the same. Those that similarity is really detrimental in this case. But overall that similarity is really valuable. actually the magic behind um these models at all. And just to give you a really simple example, it's kind of like, you know, back in the day with uh a little bit more like lookup tablely stuff, um I would say hi to maybe that bot um and it could interpret hi.
But if I said howdy, it would just break, right? So it's like those things are similar, but it doesn't know that similarity. For these models, they actually understand that similarity. Hi, hello, howdy. They can respond and understand how you are engaging with it. That's a really powerful thing that it can handle in that very simple case. And of course, that can scale to very difficult complex cases. Um, but in this case, I want it to be super precise.
I don't want it to be sampling from that bell curve. I really, really don't. I really want it to actually be sampling from something um that looks more like a duck delta distribution on 1981. So, only going to sample 1981 in this context and nothing else. like 1970 be sampled as likely as the word cat. Okay, that's what I want it to do. Um, and this might be obvious to some of you, but why having that factual accuracy is important, um, is because it opens up a new set of use cases.
And what I found just working with a bunch of different enterprise customers talking to thousands of them is that it's not just about this one use case where you know suddenly they can either do or make it more precise that they were already doing. It's actually just the opportunity of opening up a completely new set of use cases that they then have the agency the ability to then go push forward and make it um more precise on their own.
Um, and precision is everything for a lot of different agentic use cases. I'm sure many of you have tons of agents running around over a unified agent. Um, and if you know one of them has an error and it passes it on to another agent, that error compounds. So having that precision early on in your pipeline is extremely valuable. Okay. Um, so that is that is ultimately the value is opening up new use cases. um in order to get these um like factual learning to actually work though like when you give it a data point of hey climbed the Golden Gate Bridge in 1981 if you really want the model to remember that there are a lot of things you actually have to consider to get it to but it can it really can um so the first thing is around data and we're at data bricks summit right and I spend a every time I talk to the folks at data bricks the co-founders at data bricks we talk about how important the data is that we give to the model garbage in garbage out.

6:31 Keys to reliable learning

Um, it really is about what kind of data you give it. And so if you give it climbed the Golden Gate Bridge in 1981 a million times, you overwhelm the model with it, it's going to know that sentence. Um, but it might not be able to handle that similarity super well anymore, right? So if you ask the question in a slightly different way, it might not really work. Um, so that data is super important. So giving it that appropriate data, being able to see 1981 appropriately and being able to then have the right algorithm to back prop that into the model so it remembers that uh is really important as well.
So getting your data right is super essential to whatever algorithm you give to the model. And by data I mean it could be data in your prompt, right? It could be just adding that line a few times in your prompt, far more times than your prompt. It could be fine-tuning the model. It could be adding it to your uh reward model, right? If you're doing reinforcement learning. So that data is going to show up in a lot of different places in a lot of different methods as you scale complexity as you try to get the model to improve with more and more complex methods.
Um if the simple methods don't work out of the box. Okay. Um but getting the facts right is important. So if you can't get the facts super right, it is going to learn the wrong facts that you want it to learn. Um the next topic I'm sure you've heard about because I think people are starting to talk about this a lot is around evaluation. Um, and I think if we take evaluation to the limit, um, it basically says that, you know, AI where it's going to shine is where it's going to be easily verifiable.
So, easily evaluated, right? Like that's why we're starting to get good at kind of math problems. Um, that's why coding is so popular with AI. It's because we can actually improve the models quite a bit when we can verify the output. And we can verify it in so many different ways. one for um I actually find this equivalence really interesting was talking to some other researchers about it but essentially in fine-tuning when you're doing like SFT supervised fine-tuning um you're doing something actually very similar to the data in order to get uh you know in order to filter it out just like you do for evaluation and what you're doing is you're doing rejection sampling so for all the data that you could possibly have you want it to get you want to filter that data to only the high quality set and give the model those positive examples in a similar way, but in almost a softer way, not a a hard rejection sampling.
Um, for a reward model or when you're looking at reinforcement learning and you give a positive negative reward on an example, you also keep those negative examples and that also is another way um to fil basically provide that signal back to the model. Um, and I think all of this is connected to evaluation. Uh, and evaluation is really key. Um, I think evaluation the hardest thing that I find when it comes to emails very practically is just people think it needs to be perfect.
Um, and they spend too much time on it or rather they think it takes too much time as they try to build it up and then they realize, oh no, I don't even want to bother doing this. Um, but actually a really good best practice which um, I know a lot of machine learning researchers share is actually just start with a really bad eval first. just start with one that you think is kind of bad. Um, just so you can get started to see how the model is generally doing and keep it really small like something you can handle and just look at and know it whe what the model is doing.
Um, so like 20 examples, right? It doesn't have to be anything complex and something you just put on a page so you can get started because iterating on something is easier than uh starting from a blank page. So eval is really critical to improving the model and I know that um foundation model companies spend a lot of time crafting their evals uh to improve at every single stage of EV uh every single stage of training from pre-training um which is actually easier because you're usually just looking at loss to post-training which is you know making sure the model um is constitutionally is being safe is acting safely um is not being sickantic.
I'm sure some of you saw the blog post from open AI around that. Um but yeah, that's super essential. I think another thing um just to kind of instill that's really important is around fast iteration cycles. Something I find really interesting that I've observed is in some ways machine learning um and AI and working with AI is a highly iterative process and therefore closer to product design than it is to software engineering.
Um and I found that really interesting because that's almost counterintuitive. you feel like, you know, software engineers, machine learning engineers, we all write code. Um, but actually being able to iterate really fast and have a lot of experiments or what a designer would call prototypes would is actually extremely beneficial. And it's okay if it it's wrong first time, the second time, the thousandth time. Um, GP4 was rumored to be, you know, 20,000 training runs.
Uh, so just a lot of different runs to see how something is doing. And it's highly experimental. We can't always predict emerging capabilities from these models. Um so that makes it harder to you know predict and work with but we can what we can do is reach for n experiments. We can say we're going to do a hundred experiments within this time frame. Um and that's usually the way we kind of think through how to how to measure progress I would say across a team.
Um so all of these need to be in place for any kind of learning algorithm to be effective because your learning algorithm which is you know whether it's fine-tuning reinforcement learning um or even you just iterating on your prompt is going to be very good at optimization and optimizing the model for whatever you give it. Okay. Um so that's all fine and good. Maybe you have a model that's really precise and able to do some of those things.
Maybe you got your evals in the right place. It's doing really well on eval getting that 90% accuracy. Um, but how do you put this together into an agentic system? Um so this is one such system um that uh you know we've worked with a with a big customer and essentially it's contract review right they have uh tens of thousands of contracts uh every couple of weeks um and it's just really overwhelming to be able to handle all of that all of those contracts is bottlenecked with um in-person team um looking at very precise language so how do they get agents to help with each step of the process um and you know I I know people probably are really familiar with agents.

12:32 Building agentic systems

Um, I'll just define it real quick because I think there's like a million definitions, but essentially I think of an agent. I really like this analogy from one of our customers. It's it's kind of like a robot a robot, a program, but it has this brain inside of it, and it can you can specialize that brain in different ways through tuning the model um and adjusting its weights um to achieve those specialized tasks better. um and it can you know take those actions whether it's across the web or using tools locally on your computer um to be able to accomplish that task um and for it to be reliable on facts you have to make sure that the errors don't compound and again that's what I was talking about a little bit earlier which is you know how do we make these models precise so those errors don't actually compound okay um so we talked about putting together things into a multi- aent system we talked about how important data is um at and I think that's super important at this conference but not even just at this conference I think for AI it really is about the data and that's what you can control the most and we'll talk about compute later um but essentially uh I think one really big hack that um people don't always realize it's almost like a cheat code is synthetic data and I think people are starting to use this more and more but I think to the limit what people don't realize is how valuable it really is if you look at a mature AI application machine translation.
If you go talk to that team at Google, um they will tell you that they actually don't use humanlabeled data anymore. It's actually worse than machine generated data, right? AI generated data, synthetic data um as rated by professional human translators. So, and it improves the model significantly more. Um, so in a weird way, it's actually this created this flywheel effect where the model is able to improve itself better than any human could.
Um, and it it's a really powerful force. So I think that's where things will go as things mature. Right now this is not the case for every application of course and a lot of human intervention and adding your taste into it. Adding what you know as knowledge as facts as important institutional uh expertise um is is important to improving these models. Um but yeah so what does the process look like? Uh one is you can start with a very small set of examples.
Um so to get started you don't need to feel like oh I have to go you know gather thousands or millions of examples. You can start with just a few. Um and you'll see that's clearly a theme like just get started don't be blocked. Um and then you can use um agents right internally to actually generate a lot of that data um from those small examples. Um and then you can use agents to filter that data and that criteria. You can again reuse those filtering validating uh agents for your uh valuation.
But essentially that's what you can do that generation and uh filtering that validation. Um and then you can also combine it with deterministic validation. For example uh one really common use case that I had been spending a lot of time on was Texas SQL. So SQL is really special because you know you can see if it is um valid or not, right? You can you can test a bunch of things. Is it optimized or not? There's so many different features you can actually evaluate um from generated or non-generated SQL statement.
Um and that's really valuable because then you can actually give a lot of feedback there. Um and uh and then you can get that high quality data set. Um, but what's really cool is I actually think um, so you can get that data set and you could spend like a god- aful amount of time sifting through that data set figuring out, you know, is this really the data set to improve my model or do I need to add more? What do I have to add more?
You know, traditional machine learning headaches. Um instead of that um you can actually start to uh give what I call vibes based feedback kind of like from vibe coding but I I believe in a future where we can like vibe tune these models so that they can actually improve from our very casual natural language feedback. But the idea is you can give vibes based feedback because that just turns into changing the prompts of the generator agents and the validator agents to edit its own training data and improve itself.
Um, so I find this flow really compelling in a way where um it it helps me believe in a future where these models will actually truly self-improve um and help themselves. Uh, and I think this flow actually already works today. We've had several folks actually implement something like this and it actually improves the model significantly while reducing a huge amount of developer time. Um, okay. So this uh mentioned a little bit around SQL uh kind of what does this actually look like concretely.
This is again a simple example um but you can look at a schema. So you can look at a pretty complex schema and we worked with some pretty big customers on this um and it actually generalizes which is cool. I find that really cool. It's like this whole system or this whole pipeline generalizes to another really complex schema inside of an enterprise. Um but uh this was with Colgate the toothpaste company. So get ready for AI toothpaste.

17:58 System design for agents

Um but uh but uh essentially starting with the schema and you can just you know kind of lay out the schema looks like and give maybe examples of what the format for each um each column kind of looks like and you have an agent that can actually convert that um to uh very simple SQL. Um, and then you can have a pattern question agent that can actually look at the possible questions that a user might ask the system and identify patterns and then generate more from those patterns.
Um, and of course you can take those questions, decompose them further and then be able to generate more there. Um, and then doing variations etc. and then having a validator agent. Um, this is just here to demonstrate that it can get pretty complex. um when you're generating all of this data and you can keep getting more and more variations and the important thing is to cover breadth so the model has seen um kind of the breadth of what you might give it uh and what I think is really special is if you do decide to venture beyond prompt engineering and beyond rag and actually try to fine-tune or do reinforcement learning some kind of post- trainining technique on your model um you'll get to see that uh you know some kind of interesting generalization whether it's localized or more broader emerge from these models Um I don't know if you guys remember this from a very very early checkpoint of um which is now Chachi PT I guess generating images.
Uh there was an avocado chair. It was able this model early on was able to generate an avocado chair having never seen an avocado chair. This is a very magical moment at least in my life. Um but it had only seen pictures of chairs and it had only seen pictures of avocados and it was able to generate avocado chairs. I know why that's important here is I think like it's really hard for people to somehow take that concept and see it here but let me just spell it out here.
What that means here is that you can have question answer pairs as your data for just you know you know this one column in your column A in your database maybe doing a join with column uh or with another with another table right um but basically you can have that just column A and then you can have another question answer pair with column B and then suddenly and it has the concept of joints and suddenly even though you don't have examples of column A to column B any kind of you know data point doing that it's able to generalize to That's what's really magical.
Um, if you can give it all of that data to learn from. So, um, so that's kind of the magic point. Of course, that generalizes to much more complex things, but you don't therefore need to have, you know, every possible combination in your data set because that would just turn into a lookup table. Okay. Okay. So um so maybe just to summarize like the value of agentic AI. I'm sure you've seen things kind of like this but essentially um there's that model intelligence layer with reasoning and factuality being really important.

21:09 Value of agentic AI

Scalability um and cost effectiveness. So data is really sorry let me just start with this. Data is really really important. That is going to be the big differentiator I think across different applications. Um we're going to have similar optimization methods. candidly everyone's like people are going to converge on very similar methods there um and of course I think many of you know but don't get too invested into one model or framework uh as new models still do come out maybe less frantically as in 2023 um but uh but yeah and future proofing some of those pipelines such as the one you saw with SQL like that being able to generalize to another use case or another um specific customers is really really important um scalability and cost effectiveness is really important, too.
Um, kind of the methods I showed here are for when you need the model to really improve its accuracy and really think through like how do I ek out more accuracy from this model. Um, and typically you're looking at trying to save cost because you're using a smaller model for maybe the same kind of API call that you're doing with uh GPD4 or cloud. Um and then secure data infrastructure, making sure it's private and secure using open models is a great way of achieving that.
Um and finally, uh making sure that you are using things that are compatible, right? So like using things that are portable across, uh whatever layer of the stack that you're operating on, but things that are portable across the closed model APIs and the open ones. I think things have really converged that OpenAI SDK at the highest level and things are converging um all the way down the stack all the way down to the silicon um which I'll share a little bit more later if we have time.
Um so yeah so that's about how to compose these a AI agents uh that can work together that can get to high accuracy um uh and you know that there are real proof points of people doing that with open models. Um, I want to touch a little bit on what scaling these agents look like and kind of what I've been tinkering with and thinking about um, around how an open ecosystem could help with that. Um, and I want to start with the statement of why AI progress I think is computebound.

23:26 Scaling and compute laws

I think it really is limited by compute. I also think it's limited by really good data that where you can steer your application anywhere um, you want to take it. Um but essentially at least in the large foundation model labs more compute has led to um better models and so that is really like the thing they are they are focused on and pouring money into very very openly putting pouring money into um and clearly it hasn't really fully hit a wall and I'll tell you why.
I know there articles saying maybe it's hit a wall but I don't think it's hit a wall based on the original kind of estimate of where things are going. Um so scaling up parameters has worked. Maybe there's different ways of scaling it up like with mixture of experts. Um more training data has helped. Again that can be synthetic data. I think there was you know a little bit of murmurss around are we are we running out of data but I think with synthetic data we're not.
And then of course more compute being able to run more data into it at a time so that we can get a model out sooner. Um and all three u must scale must scale together. Um so bigger models obviously require much more compute to make that happen. Um uh to get performance gains um uh you you need more flops. Um inference also I think with all these multi- aent systems I'm really excited about how much um people are hitting these model checkpoints without needing to train them.
But that also is very compute intensive. Um I think there's excitement around you know using tools through the MCP layer model context protocol layer. um and that some of those tools are CPU based. So there's kind of this excitement of maybe the fabric between GPU and CPU compute will be um much much more closely tied and coupled than before. Um and of course uh so far compute scaling is the hammer that have that has worked.
Um and uh this comes from scaling laws. So I don't know how familiar folks are with scaling laws but it was a very early paper um and set of papers um back in the day before chat GBT. as a fossil, but it has been pretty much holding. And I think what people don't fully realize is that um uh you know, it's not like you put in 10x more compute and you get a 10x better model necessarily. Um it is, you know, correlated directionally that way, but it's not necessarily um linear.
It's actually sublinear. Um so when people are like, oh, are we tapering out? Are we seeing marginal um diminishing are we seeing diminishing returns? Actually uh this is expected. This is what we expected all along um from these models. Um but what is exciting is that if there are ways to optimize how we leverage that compute. So 10x cheaper compute um we can that means you can get 10x larger model for the same cost.
That's really exciting. Or you can get 10x larger model um means you can get to the next order of intelligence. Whether that's only a little bit of improvement here or based on emerging capability, emerging abilities, excuse me. Um, you can get, you know, new capabilities unlocked, uh, in a way that you you weren't expecting, like a more step function improvement. Um, that's that's where, you know, this that's where a scaling loss really kind of clicks in and plays in.
Um, so something I've been thinking about is how do we actually get AI to scale um with uh in a way that still follows scaling laws but makes it more available to more people to be able to kind of access um the next cliff of intelligence. Um so here are some stats um but they're pretty wild. Uh I was you know thinking about at large like not even going through the stats one by one but just at large just thinking about have we poured more you know crazy amounts of uh money into orders of more money into people.
Yes. Yes. These labs have now paid people way way more. But I can tell you for a fact my salary has not gone up 300,000x. I wish. No just kidding. So um well not not really but like the but like this maybe for but uh but this is how much more um uh you know we are spending or you know how much more compute we demand and clearly that's far more than human capital costs um have have scaled right so this is really really where um people are putting their money and they're really really focused on making this happen but it's super not easy actually like it's not easy to get the model to continue running for a month on tens of thousands of GPUs, hundreds of thousands of GPUs and not have like a ton of those GPUs fail.
Um, if you read the Llama 31 paper, it's just like a crazy number of failures uh throughout the whole training process. So, it's so interesting to just understand, okay, well, to get it to scale, we we hit up on these other limits at every layer of the stack. Um, so I really think democratizing AI compute will lead to more AI progress and being able to invite more people to work on this problem will also push AI progress to the limit.

28:16 The open AI stack

Um, and there are many many layers of the stack. I'll go through them a little bit quickly so there's room for questions. Um, but essentially there's agents that we talked about that you're familiar with. There's hosted services which you're probably really familiar with. They're just hitting OpenAI. Um, the models which we know and love. inference engines. I don't know how familiar folks are with that, but I I can touch on that real quick.
Training and development kits like PyTorch. Um, and then GPU programming like Triton or CUDA um or Rockum, of course, the GPU. So, agents, you're probably very familiar. So many great things out there uh today. Um, guess I should call out data bricks mosaic AI agents, but lots of great agentic frameworks there. I'm sure missing a bunch that are um built by many of you in the audience and I hope will you know make it to the top of this slide.
Um hosted services also you're probably familiar this is just hitting your favorite model um or your best friend. I think three people have independently texted me that quad is their best friend which made me realize that the movie her is is happening um at least in the friendship zone. Um, so that's where hosted services are. Obviously, uh, maybe not obviously, but it costs money, right? There's not an open-source kind of alternative because it costs compute costs a lot of money, um, to scale that out and to maintain it across so many GPUs.
Um, the models that we know and love. So uh different kind of statuses of open um but you know there are open uh fully open source kind of methods data model um as well as just open weight models out there um that that we leverage um and I think this is just really a mix out there today and will continue to be um inference engines I find really interesting. So this is um kind of stuff under the model. So, how do you make the uh model move like really efficiently and fast and be low latency for you and leverage your compute that you spent so much money on um that 300,000x on, but you spent so much money on to the max, right?
To the biggest possible model or to the most tokens that you can squeeze out of it. And so, that's what these are for um and what they're focused on. And there's a lot in the open community kind of building these out to be better and writing out um uh different optimizations to improve them. Um and then uh training and development. I'm sure many of you spend some time here. Um I spent my own entire PhD here um with PyTorch.
Uh so spent a lot of time kind of building out the model architecture or even you know doing things kind of at the boundary um to make sure that the model uh runs smoothly or changing up the architecture of the model slightly right changing it toe and adapters etc. Um uh and then finally at the lowest layer um just kind of interesting you know movement in GPU programming. Um I think you know Triton as a library from OpenAI is really interesting.
It compile it can compile down to Rockom and CUDA so it can operate on both types of chips. Um and it's an open source framework in Python. So I can write kernels much more immediately. Uh so that's that's pretty cool as well. Um, so I think things are evolving really quickly. As you can see, different layers of the stack are different levels of open and there are so many layers. Actually, before really diving into the layers were, I didn't even realize how many layers there were and even figuring out the layers with my team took a really really long time for us to kind of sift through which which layers which who is what because there's, you know, overlap across them.
Um but openness is rising and I feel like that's really important for the community to contribute to um uh to improving these um these different layers. Um but also it is uh still you know I guess there's still barriers right so there's still things that you know remain limited and I hope to help work on some of that through um education and teaching um moving forward. So if you want to help or work on that together I'm super happy uh to chat.
Okay. Thank you. Thank you very much.

Learn more about the Databricks Data and AI platform.