Supercharge Your Apps with Llama 4: Essential Tools and Techniques for Developers

Summary

• Llama 4 model capabilities: Meta's latest Llama models deliver next-level performance and accuracy for diverse AI applications, enabling developers to build more capable and efficient apps.
• Developer tools and recipes: Practical tools, techniques, and code recipes simplify integration of Llama models into production applications.
• Performance optimization for deployment: Guidance on inference optimization, quantization, and resource-efficient serving techniques for cost-effective model deployment.

Supercharge Your Apps with Llama 4: Essential Tools and Techniques for Developers

Talk by Anusha Dudi, AI Partner Engineer, Meta (Sponsor Speaker); Rangaprabhu Parthasarathy, Director of Product Management, Meta (Sponsor Speaker)
Dive into the latest Llama 4 models. See for yourself how to unleash the power of Llama models and achieve next level performance with our curated set of practical tools, techniques and recipes. Join us as we dive into the world of Llama models, exploring their capabilities, developer tools, and exciting use cases. Discover how these innovative models are transforming industries and improving performance in real-world applications.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What improvements does Llama 4 deliver over previous models?

Llama 4 offers enhanced accuracy, better reasoning, improved instruction-following, and stronger performance on specialized tasks while maintaining competitive inference costs.

What developer tools are available for Llama integration?

Meta provides SDKs, inference frameworks, fine-tuning recipes, evaluation tools, and integration guides to simplify building applications powered by Llama models.

How can developers optimize Llama model inference costs?

Techniques include model quantization (reducing precision for faster inference), batch inference for throughput, caching strategies, and serving frameworks that reduce compute requirements.

Can Llama models be fine-tuned for specific domains?

Yes, fine-tuning recipes and frameworks enable developers to adapt Llama models to domain-specific tasks using relatively small training datasets while maintaining model quality.

Full transcript

Hi everybody. Uh all of you who've eaten your lunch, everybody. Uh great to see you. Uh I have a very long name but I go by. Uh I am from Meta. I'm a director of product. Uh I work on Llama. We also have Anushia who's a who's a partner engineer at Meta. Um, I am the I'm the warm-up packed for this for this session to be very honest. She's going to tell you all the cool stuff. She's going to tell you the kinds of things that make a difference in your business.
I am the salesy guy. I'm going to tell you some cool stuff, but really you should wait for her, not for me. Okay. Uh, I'll do probably a few minutes of talking about llama just to context set for everybody. I'm pretty sure every one of you here knows what it is. Otherwise, you wouldn't be inside this room. Uh, but she's got actual code to show you. She's got tools to show you. And that's where all the fun stuff is. All right.
How many of you here are engineer engineer types? And how many of or maybe I'll ask this question. How many here are people who do non-engineery stuff? Not bad. Not bad. Okay. You have some people like me, right, who earn the airkeep without actually doing the cool stuff. Great. All right. Uh, as I said, I'm a director of product on Llama. Uh, and I'm going to talk to you about our models. All right. So, agenda for today, super quick overview, super quick run on Llama for a little bit of time on model selection because I think that's important.
And then obviously the meat of all of this which is how do you get the most out of it. Okay, let's talk about Llama. Uh it's been a crazy ride. Uh like you know if you think about the fact that Llama's just over two years old. It boggles my mind. Uh like personally when you think about this geni itself is fresh and new. Here we are at the data bricks AI summit. Uh many many thousand people talking about not just models but we have multiple layers of the stack that has now been built on top of generative AI.
It's kind of crazy. Businesses have been built businesses worth hundreds of billions of dollars. It's just insane. Uh so just from a context standpoint if you think about llama we started with llama 1 uh which was a 7 13 34 and 65b set of models we kind of grew the most popular ones I think we we can all agree is the llama 3 line this is when we started hitting I would say escape velocity this is your 8b and the 70b models which continue to be super popular to date um then we started iterating on those models there's a 3.
1 version of 8 and 70 and then there is the 405b which many many businesses have used uh for a lot of internal reasons then we started doing llama 3. 2 two. This is when we brought vision models into the mix and we also started building the ondevice models both of which address a set of very uh important use cases. Um and then we iterated on the 7dB with uh the llama 3. 3 line and then obviously earlier this year we did llama 4.
We'll talk about llama 4 in a little more detail. It's been kind of crazy. Lots and lots of models. Um, and a lot of people are using Llama. How many people that 1. 2B one 1. 2 billion downloads is already out of date. Uh, llama models get downloaded at a crazy clip. Um, and here's the interesting part, right? Downloads are not just the only way by which people consume llama which kind of tells you how big this thing is.
People use APIs to use Llama. People download the models. Uh and once downloaded, some of you are probably part of enterprises where somebody downloads a copy of Llama, you fine-tune it for some particular use case and then it's just being used internally by many many hundreds if not thousands of people. So the footprint of llama is pretty remarkable in ways that even we don't fully comprehend. And when you think about derivative models and this is these are just the ones that get registered with us.
That's also an insane number again two-year-old thing and I'd argue most of these derivative models are of llama 3 and beyond which means it's many of these derivative models are probably just over a year old. And again this number is is I I'll tell you an inside joke from Meta. Uh I know the number of the derivative models and uh I went to our our u our team and said hey you know we should publish this but then they said oh we need to go through a stat review to confirm that these numbers are right.
So I'm like okay I I will share the last publicly shared number and this is kind of the scale at which not just llama but generative AI more broadly is is printing which is things are just everybody is using this u and the reason why we talk about these numbers so proudly is because llama lends itself to so much flexibility for all of you. I'd probably argue within the people here and within this this very large uh conference so many people use llama in so many different ways I wouldn't even be able to document that and that's the beauty of open models right so you can deploy it on a CSP VPC onrem and these are just some of them uh you customize it with so much fine-tuning I've seen people fine-tune it in so many different ways and I think that's the magic of building open source models some use cases.
Obviously, we're at the data bricks summit, so we had to pull up data bricks. But by the way, data bricks is a great partner for Llama, so would love to give them a shout out. They do some awesome stuff with it. Um, and then these are a couple of other examples. And again, hundreds of thousands of examples of how enterprises and businesses use Llama today. Uh, and again, I can't even wrap my heads around it. All right.
In addition to that all the ways that businesses use there is one business that I'm very intimately familiar with that is using llama extensively that is meta uh meta which is our assistant product that you can access on WhatsApp on Instagram on on many other surfaces on its own app today um is used by more people than any other assistant. Over a billion people use this product every month and growing. Uh people find different use cases to use meta and llama is the foundation.
So when we come to the world and say hey you know what you should think about using llama it is because we know there are a billion people using it every month uh and we we understand the value of the model and thus the product. So let's talk about Llama 4. Um Llama 4 launched earlier this year. Uh we we made two models available. There's a third model that's cooking. Of the two models that are available, there is scout, there's Maverick.
Uh they're similar models in the sense that they both have a similar number of active parameters. Uh but they do have different number of experts which means the intelligence of these two models are significantly different. Llama for Scout. Both of all all of our models at this point are all multimodal models. Um Scout is smaller. You can you can fit it into one H100 GPU. Um offers 10 million context length, great model.
Maverick uh offers all the stuff that Scout does, much more intelligence and uh larger model of course. Let's talk about some use cases. Again, Anusha will go into a lot more detail. Uh long context is one that I'm generally uh fascinated by because you know the more you can squeeze into the context window, it just means you can get more out of your model in near real time. Um so as you think about this you know some of the use cases that we thought of and we heard from a lot of our partners um in in building longer context windows is legal document analysis uh scientific research financial analysis multilingual analysis.
I mean lots of places where having a longer context window is just that much more meaningful. I think I have a video for this. Yeah, let's run this video. So this is a neat video where we use the context window to do something interesting which is actually why don't you watch that pause. Yeah. So basically what you can do is you can now synthesize very large websites, very large repos and we built an app to showcase that and the value of having this large context window is now open for people to fiddle with.
One of the questions that comes up and again I'm happy to talk about this when we are done with is hey is longer context windows replacing rag? This is a question that comes up fairly often. My personal opinion is that it doesn't replace rag as much as it augments it and there's always a value where rag is the solution. But there's also now opportunities where there are some use cases where you had to use rag that can now be done with having a longer context window.
But we can talk more about that later. Um how can you get llama 4? I'm pretty sure all of you have already seen it, but if not, hugging face, llama. com, or pretty much any provider where you get your models from. I want to spend a few minutes talking about building a model that works for you because I think this is is a very important topic. A year ago when when Llama 3 was launched, there were not a lot of open- source models.
As a result, a lot of people jumped on the llama bandwagon. Now, there are choices and you should obviously use the model that works for you. But it's also hard to choose what is the right model. So again, this is an unbiased view of model choice. And I wanted to put this in front of you because I think there is some value in having this conversation. You want to find models that are accurate, right? So accuracy obviously is super super important and depending on the use case accuracy is either tantamount or accuracy is just important.
So this varies not just on your business but on your use case. Second cost. If you are using a few hundred,000 tokens or a few million tokens cost might not add up. But as you build scalable businesses and you're going into the tune of billions of tokens, cost adds up really really fast. The last part is latency. Again, if you're using passive asynchronous applications, latency is not that big a deal. But if I am having a chat interface where I am going back and forth with a model, if the model is going to take forever to respond, that kind of sucks.
So depending on your application, depending on your use case, you are going to figure out how this triangle looks for you and how this prioritization looks for you. And how you think through this prioritization is then going to reflect on the kinds of models that you actually choose for your business. So this is just all of these are charts from artificial analysis. If you're not familiar with artificial analysis, I assume everybody's familiar with artificial analysis.
If not, great great resource. I recommend everybody check out if you're interested in generative AI models. I would always say go to artificial analysis because they give you a good context of all the models that are out there. So if you think about this in terms of the vectors that I talked to you about, like you think about where Llama 4 Maverick is, how fast the models are, cost, and then intelligence versus cost.
So why am I talking about all this? So let's talk about this. So today intelligence I think I I showed you on top of that pyramid intelligence is super important. Second how fast those models are are super important in our case and I'll talk about this when we built Llama for Maverick. We first have to figure out how are we going to address the needs of the billion people who use Meta AI because Meta AI runs on Llama for Maverick.
So for me latency is ridiculously important. I need these models to be faster than the previous generation models. So one of my requirements when I when I started working with the modeling team is hey guys we have to make this model at least as fast as llama 3 which was powering meta if not faster super important cost very important yes we are meta but ultimately when you are serving something for a billion users the costs add up right GPUs are not infinite and we have to balance between serving customers versus training new models.
So for us we have to always be conscious of are these models optimized so they are they are fast and they are cheap and then obviously you want these models to be smart otherwise you can have the fastest models in the world the cheapest models in the world but if they are not smart enough nobody's going to use it. So trying to find how to balance these needs and and getting to that intersection point was the challenge in building llama 4.
People often think hey you know metab model good for me not good for me. I often tell people think of this in context of how and why these models get built. And uh the one other thing I will I will share with this audience is you know reasoning models are all in vogue. So we all talk about the different reasoning models that are out there. Uh I'm a big fan of reasoning models myself. I use reasoning models for some of my use cases.
The challenge with reasoning models especially as you're building businesses is that reasoning models are extremely token hungry. Right? before you know it, you're spending a lot of money in in using these reasoning models. The other thing about reasoning models is it takes time and a lot of smart prompt engineering to get to the answer fast. So while latency is not a factor, getting to the answer actually takes forever long, right?
I use reasoning models all the time and like the most frustrating thing is I'm like, "Yeah, yeah, yeah. I know you're thinking. I know you're thinking. Get to the answer fast." Right? So just figuring out what application needs a thinking model versus what application needs a non-thinking model and factoring cost into that also makes a huge difference. So I hope you got some flavor for all of this. I' I've kind of bundled this as as a parting chart.
You'll notice all the non-reasoning models are cheaper and faster and more importantly test far take far less output tokens and this matters depending on your business because eventually if you are with whoever your favorite API provider is for tokens this cost just multiplies very very fast. So I hope I gave you some flavor for what the landscape is, our motivations for building the kinds of models that we do, why it's valuable, why it's interesting, and kind of where we are headed.
Um, I will now get to the more interesting part of the presentation, which is Anusha, who's going to talk about how you can actually squeeze the most out of your llama models. Here you go. Thank you. Hello everyone. I'm Anushia, partner engineer at Meta. I I help um enterprises and developers like you achieve success with Llama on our partner platforms. So Prabhu so far walked us through the evolution of the Llama models and different models that are available, our latest Llama 4 models and how these models perform across intelligence, cost and latency.
Now let's get into how how to get the most out of these models. So before doing that uh let's take a moment to understand why do enterprises and developers use llama models in the first place. covered it so well but let's take a quick look. Cost, latency, control over customization and deployment and data security. These have been always the top reasons why these models are so important for enterprises. Um, and we have seen enterprises cut down cost by 10 times or 30 times when moving from proprietary models to open source llama models.
If you're thinking, yes, that's great, but my use case is so complex or they don't really work for me. So, let's try and see if any of the problems we'll talk about will um that you identify with and maybe some of the tools and techniques will help you solve for these problems. So, here is the plan. So we'll look into what are the common issues that developers face starting with uh you might have tweaked your prompt tens of times but not sure how is this the right prompt that gets the optimal performance from these models.
So how to systematically approach that and then the model hallucinations incorrect answers not getting the desired style or ability to reason for the task and you want to customize the model and often you don't have the necessary data set to do so. So we're going to look into how to use some of these tools and techniques to address these challenges starting with prompt optimization techniques in drag synthetic data gen and fine-tuning.
So let's dive into it. Starting with our first problem, how to get the most performance with the right prompt. So just to cover basics, I'm sure all of you are already very familiar with prompt engineering. But let let's quickly glance over it to cover the basics. Um so we give the AI a role on a specific task, write clear instructions, split these instruction into steps that helps the model think and reason better. and start with um zeros and then try giving some examples which generally helps the model to output a better response and test and revise the prompt.
I'm sure all of you have done this multiple times but the challenge is it requires experimentation trial and error and what works for one model may not always work for another and migrating between the models is always very hard. So how do we systematically solve this problem? Introducing llama prompt ops tool. This is a Python package recently open sourced by meta. The tool takes in the prompt and a data set as an input.
A data set is nothing but um the examples of what is the input and uh expected output to the model. and the tool um optimizes the prompt and returns um returns a prompt that gives maximum performance for the for your target model. So here is an example on the left you see a pretty good prompt um that is given as an input to the tool. Um the tool is still able to revise the prompt and also add few short examples which seem to be working um well to get to get great performance.
So if you if you're curious how this performs on a real use case. So here I'm showing a hotspot QA data set which is trying to solve for multihop question answering task which is not an easy task but as you can see using the prompt optimization tool we're able to achieve amazing performance gains and as you can see 3. 370B isn't is even outperforming uh the proprietary model. If you're wondering how to use the tool it's pretty simple to set up.
You can also run it on your datab bricks environment in just three steps. Um, llama pip install llama prompt tops configure your config file. This has four sections. Uh, the model is basically your target model and the data set initial prompt that you can input to the tool and the metric. The metric here basically helps the tool understand um how how should it um match the response from the target model with the expected answer in your data set.
Should it um should it match it semantically or should it exactly match or do you have any specific custom criteria? So the metric can help um the tool understand how to optimize uh for the response. So once you have your config file ready, llama prompt ops migrate the command help outputs the file which has the optimize prompt. So hope this tool will help you to systematically optimize and migrate the prompts. Now let's dive into the next set of issues that uh developers generally face where the model information is outdated. you're getting incorrect responses and uh it does not have the knowledge of your domain and the model hallucinates information.
These are set of issues where you expect the model to know something and return a factual answer but the model is not able to do so. So let's call this the knowledge problem. So how do we tackle knowledge problem? So like prau said we can use long context or we use rag. So, how do we fast? Okay. Sorry. You can slow down. Yeah. Um, thank you for that. Yeah, let me slow down. So, yeah, let me um Yeah, let me um cover what drag is just to cover basics if someone is new.
Um so in a typical system we generally give a query to an LLM and expect an answer. The LLM basically uh answers with its own internal train knowledge. In drag in addition to the query we provide the LLM with the necessary context required to uh to answer the question. So how do we provide this context? There could be various ways we can provide the context. using long context. You could input the whole document um to the model or you can use tool calls to input this context or uh in in co a common pattern is also to use the vector base vector database and retrieve the relevant context and use it um and use that uh to give it to LLM.
So how do uh so how does this works? There are um there are basically two phases I would say a setup and storage phase and then a retrieval phase. In the setup and storage phase, you're basically um collecting all the necessary information and documents that are required to complete the task and uh and segmenting these documents into manageable chunks depending on the model that you choose and um passing that to an embedding model and creating the vector representation of these uh smaller chunks and storing that into your vector database.
So that's the storage phase and then in the retrieval phase which is more real time. So whenever a user queries so we um whenever a user queries so a function basically does a similarity search and retrieves in the vector space and retrieves the relevant rows from the vector database and gives these chunks as a context to the model. So this is a typical um rag system. So now how good your model and your system works in rag is basically depends on how good is your context that you provide to the model.
Right? So um rag works really well for a lot of use cases as you all know but it can also get quite complex quite fast. you might run into complexities like this where as you as you're segmenting um the documents into smaller chunks. The context can often get lost within these chunks. Uh it might not carry over the full context that is in the document and the document could include images, charts, graphs, uh things like that.
And it could it could also be just a complex task where you need to provide context from multiple sources or it could include like multiple rule calls. So for um for complexities like this there there exists like solutions. So I'll talk about contextual chunking in the next slide with an example so it's clear. Um if you for complex documents that include images we can use multimodal drag. So this is nothing but a pre-processing step where you're using a multimodal LLM like Maverick and converting the images into the text format and basically using and then using everything as if it's a it's just a text document and with um and if you require context from different multiple sources and tools we can use agentic rag.
Go ahead. Yeah. So in general the model is able to understand the images if you have specific like architectural diagrams because that can vary and the if the model is not doing well we might need to fine-tune the model to work for that specific use case. here can add more context. No I I would love to actually answer your question outside so she can finish it but happy to answer the question. Yes. Um yeah so if you need um if if if the model needs to get context from different sources then we can use agentic rag.
So the agents basically can um h can coordinate and handle retrieving from different tools or different databases to retrieve the necessary context. So for all of this um there are recipes in llama cookbooks and Asian rag is really well supported on datab bricks too. So I highly encourage looking um uh into tools like this. So let's dive into context while chunking now. Uh so I want to highlight this because this is a community contribution from web aapi. ai.
I also encourage you all to make such contributions to help the open source community. Um so looking at the example here so the chunk one basically talks about BMW XY popularity and locations and chunk two talks about BMW XY pricing right so if you look at chunk two um without chunk one would you know what it's actually referring to you may not um so because as we um as we segmenting the documents into smaller chunks chunks and storing and treating these chunks as as if they are independent.
So this could always be a problem at the retrieval phase. To solve for this um what we can do is um preserve preserve the context of the chunk relative to the whole document with few keywords something like this. So we are extracting the context of the chunk relative to the whole document and adding that to the chunk. So for example here with chunk one we are adding BMW X5 locations and with chunk two we are adding BMW X5 pricing.
So now if you look at chunk two by just by itself you would know um what it's actually um referring to. So just with a simple technique like this they're able to achieve a performance gain almost doubling its performance. So yeah I hope techniques like this really help you to sol solve for the knowledge problem. Um so now let's dive into the next set of issues that developers usually face. So this is generally where um you're not getting the right format or the style from the model or you need to improve the model's ability to adapt to your new or unseen information in a specific domain or um or the model is not reasoning well for your specific task.
Right? So all of these are a set of issues where you need to improve the model's ability to um yeah where you want to change the model's um way the behavior or the way it acts right so we can call this the behavior problem. So how do we solve for a behavior problem through fine-tuning. So fine-tuning solves for how the model needs to act. It helps adapt to your domain and especially for your unique data sets and use cases.
So here are some of the use cases where we have seen um lot of success with uh with fine-tuning. For example, um teaching reasoning to the model specific to your task and matching the performance of a larger model with a smaller model. This is like distilling from a like for example if you want to match the performance of a 70B model with an 8B model and um finally adapting to the specific domain task. So this this has been very common across all enterprise use cases and across different uh domains healthcare, finance, retail etc. um if so to fine-tune um there are many platforms which offer um fine-tuning APIs here I'm showing the datab bricks geni SDK so all it takes is the target model and your training data set and helps you to easily fine-tune the models and meta also publishes the recipes for um especially for the newer models if you want to have fine grain control over how you want to fine-tune the models However, the challenge for fine-tuning is not fine-tuning itself but having the high quality data.
So, ideally you need you need data that reflects how the model needs to behave in the real world across different scenarios and but in reality the data is often uh fragmented or it has gaps or it it is skewed. So if that's the case for your use case, I highly encourage using synthetic data generation tools. I'm showing two to two synthetic data generation tools here. One is um open source by meta called synthetic data kit and the other one is datab brick synthetic data API.
Both of these tools work very similarly. Um they both take in context from the user and any documents or examples of how you want the synthetic data to be generated and the and the tools basically output a well curated data set that doesn't have PII that you can readily use for fine-tuning and this is also very cost- effective. So here is a great example of how you can use um synthetic data generation uh and fine-tuning to get amazing performance.
So here we basically we want to improve the model performance on tool calling. So we are taking a tool calling data set and enhancing the data set with chain of thought um reasoning and once we fine-tune the models as you can see we are able to achieve 6 to 13% improvement on the BFCL benchmark. Uh BFCL is one of the popular benchmarks for tool calling. So before we part, I want to leave you with a cheat sheet for how to build with LLM apps.
So step one, build build a proof of concept and end to end prototype. Second, design an eval. This is an critical step that most developers and enterprises don't take seriously, but it's really important to design your test data set so very well and also with the metrics that you care about. Establish a baseline performance with the prototype that you have built. And once you have a baseline, optimize with the prompts tool, prompt ops tool.
So, and to further optimize for performance, first identify the problem. Um, how do we identify the problem? So, take some samples um from the model output and analyze if the issue is because the task is just too complex. Are you asking the model to do too many things? If that's the case, break that down into uh specific task and use task specific agents. And if if the issue is because um you expect the model to know something but it's not returning a factual answer.
So in that case, use rat techniques that we talked about. And if the issue is behavior where you need to change how the model needs to act or how the model behaves, then use uh fine-tuning and leverage synthetic data create data generation to fill for those data gaps. And sometimes you may need a combination of all three of these depending on your problem identification and your analysis. Right? Finally, don't forget to um run the eval at each step of your optimization to ensure the optimization is moving in the right direction.
So, I hope uh some of these will be useful for your use case. Um be please take a moment and fill out our developer survey. Let us know how we can support you better. any specific recipes or tools that you want to see. Um, yeah, and also any problems that we didn't address today, but you want to find solutions for, please let us know. Um, here are the links for the prompt ops tool and also Llama cookbook. Llama cookbook has recipes for a lot of things that we um that we talked about today.
Thank you. Yep. Just parting note. Uh, thanks all for showing up. Uh again, I think the message we want you to take away is that you can do wonders with techniques that are cheap. You don't have to spend a ton of money on proprietary stuff. Open source is the way. Uh and then the second thing is Llama cookbook. We are constantly adding new recipes, new things. If there is something that you are struggling with, would love to see a a recipe in a llama cookbook, just drop it in the developer survey.
We would love to be able to add stuff that can help all of you. So, thank you.

Learn more about the Databricks Data and AI platform.