Skip to main content

The Future of Domain-Specific AI: Building Effective Agents for Specialized Applications

Summary

  • Agent Bricks for production AI: Address the 90% failure rate of GenAI projects with Agent Bricks on the Databricks Data and AI platform—a comprehensive solution for building, evaluating, and scaling domain-specific agents.
  • Data foundation as backbone: Prepare and structure diverse data sources to create knowledge bases that serve as the foundation for powerful, efficient AI agents on the Databricks Data and AI platform.
  • Quality and cost optimization: Implement evaluation frameworks and cost-efficiency techniques that balance agent performance with resource constraints for sustainable production deployments.

The Future of Domain-Specific AI: Building Effective Agents for Specialized Applications

Watch: The Future of Domain-Specific AI: Building Effective Agents for Specialized Applications
Talk by Archika Dogra, Product Manager, Databricks ; Jun Choi, Staff Software Engineer, Databricks
Ever struggled with getting AI to work effectively for your specific domain needs? Join us to discover how establishing the right data foundation transforms the way you build and deploy specialized AI agents. this video demonstrates how to prepare and structure your information assets to enable more powerful, efficient AI applications. Learn proven approaches for extracting value from both structured and unstructured data, creating knowledge bases that serve as the backbone for domain-specific agents, and implement optimization techniques that balance quality and resource constraints. Key takeaways: Preparing diverse data sources for AI consumption and knowledge extraction Building scalable data foundations that naturally facilitate agent development Optimizing domain-specific agents for performance and cost efficiency Implementing practical governance frameworks for AI systems built on your data
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

Why do most GenAI projects fail?

This video identifies two critical factors: lack of confidence in solution quality and poor ROI due to high production costs—both addressed by the Agent Bricks framework.

What is Agent Bricks?

Agent Bricks is a comprehensive solution on the Databricks Data and AI platform for building, evaluating, and scaling domain-specific agents at production volume.

How do we ensure agent quality?

Agent Bricks provides evaluation frameworks, research-backed optimization techniques, and cost efficiency patterns that ensure reliable, trustworthy agent performance in production.

Can we build agents faster with Agent Bricks?

Yes, Agent Bricks accelerates agent development by providing pre-built patterns, evaluation systems, and governance frameworks on the Databricks Data and AI platform.

Full transcript

0:00 Introduction and agenda

Hello everyone um and welcome to today's session on the future of AI here at data bricks agent bricks as I'm sure everyone here saw at the keynote today um and this session in particular is going to focus on how you can scale domain specialized agents at high production volume so the forward-looking statement Make sure that you complete your surveys after every single session. And with that, we'll start off with some quick introductions.
I'm a product manager here at Data Bricks on the AI platform team working on part of our agent bricks solution. Hello, I'm the tech lead for the product AI builder product group. Awesome. So, uh, the quick agenda of what we're going to go through this session on is we're going to start off with the production problem that we're seeing a lot of our customers face as they're trying to productionize high volume use cases at scale.
We'll talk about how agent bricks solves this problem for them. We'll give you a quick demo live of building an agent brick. Um, we'll talk about the research and engineering sauce that's kind of powering why you should build your next use case on an agent brick. and then we'll end it with key takeaways to walk away from um from today's session. Okay, let's talk about the elephant in the room. Raise your hand if you're goal here maybe it's at this conference it's in your job in general is to take AI to production.

1:23 The production challenge

That is pretty much everyone I would hope. Um but the main elephant in the room today is that what we see is 90% of genai product projects never go into production. So what happens is that someone comes up with a great idea. We can like augment some sort of insight in a way using like an LLM or an agent. They run a P on it for a few months and then that P dies and that project never actually sees the light of day. Well the key question behind this is why?
And since everyone here raised their hand saying that they want to take Genai to production, I would actually love to learn from you guys. Why did your last failed PC not make it to production? There's two questions we have here. Raise your hand if it was because you weren't able to be confident enough in the quality of your solution. It was hard to evaluate, the quality wasn't good enough, etc. Okay, few hands here and there make sense.
Raise your hand if the ROI simply did not make sense. um for example it was just too expensive in production. Cool. So what we're seeing is that these two issues are resonating with some people have the same both problems and some people have one or the other problem of quality and then cost. So let's distill this into a real world example. Meet Sally. is a senior data scientist at Telicks which is one of the biggest telecom companies in the world. ran some numbers and she found some pretty uh alarming numbers from the past year.
Churn, customer churn has increased by 7%. And from the customers that currently exist as Telerick's customers, they're not actually upgrading their plan. So the market share that they already have is kind of staying stagnant and they're losing customers while they're at it. Well, the key question behind this is also why? And the answer to this is in a lot of different places, but one hunch that has is that it's sitting behind a lot of unstructured data.
Namely, the customer service calls that are coming in on the order of millions of calls per week that Telicks is getting that basically capture the customer dissatisfaction, asks and issues that their tele users are facing every single day. However, there's a hundred million calls of these transcripts and this is buried deeply under a vast amount of unstructured data. So, it's really hard for to go in and automatically make sense of this.
But Sally needs to know why because she can't go to her leadership team and be like, "Hey, we're not doing very well." And then they're going to be obviously ask, "Well, what's wrong?" And she's not going to have the full story there. So she definitely needs to answer this question to understand what's going wrong over at Tel Bricks. So what she eventually needs to do is turn this mountain of customer call transcripts into aggregated insights solving the why question.
And Sally's heard about this thing called, you know, LLMs and agents. And she heard that they do a pretty good job of reasoning on top of unstructured data and maybe extracting these nuanced insights from that data. So, where does she begin? First, she needs to choose the right toolkit. She has a bunch of different providers, open source, proprietary, different models. She's kind of like, okay, which LLM is actually going to do best for my task.
She's not super sure, but she kind of just choose one that seems good enough. Maybe it's like a leading open AI model or something and moves forward with it. But this off-the-shelf model isn't necessarily going to understand domain. So she needs to teach it her enterprise data and her domain guidelines that she cares about and different things that the LM needs to learn to infer over time. The problem with this is that the best bet that she has today is prompt engineering.
And she keeps adding rules and context and more and more things to that prompt till she gets to a prompt for her LM that's thousands of tokens long. And she can't even tell when adding something regresses one output but fixes another output. It's a super manual process and very very brittle for her. And that gets to the third problem which is evaluating quality. Now that she's kind of chosen the right toolkit, kind of thinks that she's taught it at her domain.
What is she going to do? A lot of customers that we talk to honestly solely rely on manual evaluation. So Sally starts with that, but then she's like, "Okay, maybe I should go to the customer service team and ask them, do you think this is doing this adequately or not? Is my LLM and prompt doing the right thing? Please let me know." They let her know and their answer is no. So she needs to go back and teach the domain again from these ad hoc insights she got from her subject matter experts.
Okay, let's say she's climbed the third hill now. Now she needs to productionize at scale. But the problem is wasn't really thinking about this. She chose the best LLM, which is great. But now that she's thinking about actually running this on hundreds of millions of customer call transcripts, her manager is like, "Wait, this is way too expensive in production. this is going to cost us hundreds of thousands of dollars.
We can't give this a green light. So now is back to square one and she has become the 90% of PC's that failed to make it to production. Or one could say the elephant in the room is very much still in the room. What exactly did struggle with here? First, she had too many knobs. Which model should she choose? Which techniques should she choose? Prompt optimization? Does she even know how to fine-tune if she needs to fine-tune?
There's a lot of open questions there. Then evaluation was really difficult for her. She was going off of vibe checks and she had to maps me feedback back to actually optimizing um her system. And lastly, cost versus quality was a trade-off that she had to make and she didn't even realize she had to make it until it was way too late, blocking her from actually going to production on high volumes of data. is not alone.
We've been talking to customers over the past few months and the unstructured data gold mine is incredibly elusive. We have customers sitting on hundreds of millions of patient notes getting thousands of customer reviews a day or they're lost in hundreds of thousands of contracts from just this year alone and they're trying to make sense of this data using generative AI but really really are are really struggling to do this effectively.
That's why we built Agent Bricks, which is getting you production AI agents optimized on your data and your domain. All you have to do is specify your problem, give us your enterprise data and your guidelines. We'll generate custom evaluations to help you benchmark how well your system is doing and productionize with confidence. And lastly, we'll make sure that you're never blocked from going to production because you no longer have to think about a quality versus cost tradeoff.

8:04 Overview of Agent Bricks

How do we do this? We'll dive deeper into this um later on in the session, but the high level is that we're going to be choosing the best models and techniques for your task, agnostic to the provider, etc. We just want to choose what works best for you. When it comes to evaluating, we're going to autogenerate custom benchmarks and evals for you to get a good sense of how your agent is doing. And lastly, we'll optimize to get you lower cost without sacrificing the quality that you could get on a much bigger model or agent.
So there's a two different um paths of use cases for agent bricks and what you saw in the keynote today was the high quality agents that can reason across data models and tools like the multi- aent orchestrator or the knowledge assistant. For this session we're going to focus on specialized costoptimized agents at scale across two main business use cases. First one is information extraction. A lot of customers are trying to turn unstructured data into structured insights via summarization, classification or extraction.
And then the second one is custom LLM where you can basically specialize an LLM endpoint to your domain specific guidelines on your enterprise data for whatever task that looks like. So here are some use cases for both um agent bricks. For information extraction, this can look like summarizing and extracting key terms from long contracts. Um, and then classifying customer reviews by product and sentiment. And then for custom LM, you can tailor a chat model to adhere to your enterprise policies at all times and or use it for content generation or translation tasks that adhere to your custom guidelines.
Okay. So now let's help. She needs to go to production. What we're going to help her do is specialize our AI, our agent brick endpoint to her domain. Then we're going to help her productionize by taking away the quality versus cost problem that she faces today. And lastly, we'll get her to the finish line and actually help her deliver real insights to her leadership. Okay, let us switch over to a quick demo on how we're going to be doing this.
Can everyone see my workspace? Okay, perfect. So, I'm kind of starting off in the data bricks workspace. Hopefully, this is a very familiar page for everyone here. Um, and where I'm going to go is the new beta agents tab in the left side navbar. So, I'm going to click on agents. And if I zoom out, I can see that I have these four use cases that are basically available to me today. And the one that I care most about right now is information extraction because I want to take information from these call transcripts and uh extract nuanced insights from them.

10:46 Live demo: building an agent

So I'm going to click on information extraction and I'm going to basically point my information extraction to the data that I want it to process and learn from. So it's going to be called call transcripts 100 in Unity catalog in the delta table. And then I'm going to select the column containing my text data which is this transcript column. So what's happening right now and what's loading is we actually will autogenerate a schema for you based off of what we infer is in your unstructured data because we have a lot of customers dealing with long contracts etc.
They're like I don't even know what I'm supposed to extract from here. Um but thankfully I kind of know what I want if I can get to it here. Um I want I have a schema in mind and I am going to basically ask for the customer sentiment from the call. I'll give a a few issue categories that I want to be pulled out from the call. So like what is the issue that the customer is worried about? And then again what I care about is upsell opportunities and churn risk.
So I'm going to see okay was an upsell opportunity present on this call? If so did the agent attempt to like upsell the product? Was churn at high risk on this call? If so, was this effectively like attempted to be prevented by the customer service agent? Um whether the issue was resolved and then the call date and a quick summary of what actually happened in the call. So, let me now I'll just call this test call transcripts create my information extraction agent.
And if I zoom out here, I can create the agent. And so as this is loading, we're basically getting a good understanding of what does your data look like? Um what does the sample output actually map to? Like what are the guidelines or the outcomes that you actually care about? And we're going to help you autogenerate that straight off out of the um gates so that you don't have to go from zero to one yourself. So this is going to take a little bit of time, but as we wait, there's a lot of stuff going on behind the scenes to get you this iterative quality experience.
Awesome. So, it's going to be updating here. And you can already tell on the sidebar that we're already starting to populate some autogenerated descriptions of your schema and what you need to be extracted from that schema. So, we have customer sentiment, the issue categories, upsell opportunity, etc. with the default instructions that we infer from your task and your data of what you're trying to look to do. So, that's the first component, the autogeneration of these um schema descriptions.
The second component are recommendations. So for example, what we detected was that issue categories is actually pretty open-ended. Your enterprise probably maps different issues to different categories in your own domain or in your own way. And so maybe you should specify if they're possible categories or if they're open-ended. Okay, let me also look at the review results and see if this is coming up in the sample outputs that we generate for you out of the box to actually validate how well your agent is doing.
If I review these results, I can kind of see, okay, yeah, this is saying billing error. The actual name of my category should be billing. This is also a billing error. Payment method is not really a category. Issue category that we track at Tel Bricks, etc. , etc. So, I am going to fix this by iterating on my description. Um, if I go back to issues category, and I already kind of have one. I'm going to basically say that I want a standardized array of issue categories that are network connectivity, account management, product inquiry, billing, cancellation request, plan upgrade, or service outage.
So, I make sure that this agent understands my domain and what I'm actually looking for and I can update the agent to see if it improves. So let me update this agent and see will it do a better job of understanding my issue categories and adhering to the list that I gave it. Um and another reason why this takes a little bit of time is that we actually are like deploying this as an endpoint for you out of the box. So every time you update the agent, you can take the a the endpoint that you have today and basically run with it.
Oh, awesome. So this finished and if I go back and look at my issue categories, I now see that it adheres to my guidelines. So, for example, billing is in an accurate category. Cancellation request is an accurate category. Um, I'm no longer seeing like payment method or things that I'm like not really able to detect. Perfect. This is how we use this interface to help you basically input and inject your domain specific knowledge into the guidelines so we better understand what success looks like.
And then we help you validate that success is there by giving you these two to five uh this these five examples um that you have to ensure that your iterative process is actually yielding into the outcomes that you're looking to see. Once you're satisfied with this, you can then optimize to get an even cheaper agent. So now that we deeply understand your data, your guidelines, and the tasks that you're trying to do, we can sweep a variety of research techniques on the back end to basically distill these insights into a much smaller, faster, and cheaper agent endpoint.
So I'm going to click optimize, but optimize takes a little bit because we're trying um a lot of different things as you can see. And so I already optimized an endpoint with this exact same use case. And if I scroll up, you see kind of two highlevel graphs comparing my optimize endpoint, which is much cheaper, to my active model endpoint, which was the original thing that we were teaching in the configure page. So if I look at the overall score, I can already see that relatively my optimize endpoint is much cheaper, but the quality is actually the same as the more expensive endpoint.
And because it's a smaller model, the optimizes endpoint throughput is much better, making it a lot more effective at processing high volumes of data, even though um the quality is retained to be the same as the other endpoint. But you kind of have to like see it to believe it. Like I'm just kind of giving you a graph with two points. You might be like, wait, you could totally just be pulling a fast one over me. So we'll actually validate this with our auto evaluations that we generate um through agent bricks.
So as you can see here, I'm comparing my optimized model to my active model. So this is the smaller, cheaper one. This is the bigger one that I initially used to train or prompt engineer my model. And so what we see here is actually the quality is matching if not improving in some categories with the optimized model. So issue resolved for example, we do 7% better on the optimized model versus the um the base model basically or base agent.
And then if we look at other things like maybe where quality regressed a little bit. This is pretty negligible like minus 4% on a customer summary. I'm totally fine with that. What I really care about is like the customer sentiment, whether the turn prevention was accurate, etc. Um, and I'm seeing that it's matching the quality of this bigger endpoint um, at a much much cheaper cost. So, I think I'm ready to deploy this given that I'm pretty confident in this 90% plus accuracy that I'm getting across the board.
And if you're also like, okay, how do I know that these auto evaluations are actually doing the right thing? We give you full observability into the reasoning for them. So here we'll kind of see like okay the reason why it passed the call summary was because it inquired about the or it captured the key points like the inquiry what the agent did and what the next step was. Um so I get a really deep understanding of like how these auto evaluations are actually happening and then it guides me to basically understanding where the agents doing well versus poorly.
But in this case I'm pretty satisfied. So I can go ahead and deploy my new optimized endpoint. But I also want to see at the end of the day, how can I use this agent in production? And the great thing about data bricks is that everything is always integrated very end to end. We're a platform that do that does a lot of different things. So if I click on start extraction, we actually have function called AI query. Um I have to zoom out here.
I don't know why I'm so zoomed in. But basically, one sec. Oh, there we go. Okay, perfect. Um, what you can see here is that we're calling our information extraction endpoint in AI query. And AI query is basically our way of doing optimized batch processing. So, it's a SQL function that's able to handle very very high volumes of data super effectively because we do autoscaling for you. We parallelize the tasks etc. So that you can get this job done in a timeeffective manner.
Um, so the fact that you can call your endpoint from AI query basically integrates really well into pointing this to a delta table, running it through AI query and getting the results out of it. As we can see here, I tried it on a few rows and I'm getting pretty good results from the key fields that I'm trying to basically understand. Okay, customer sentiment was negative. It's because of billing. Um, the issue wasn't resolved.
Their churn risk is really high. and the churn ultimately was not uh prevented actively during the call. Okay, so these insights are great on like a oneoff level, but what really needs to do is she needs to tell a story to her leadership. She needs to use these nuanced insights that she's extracted and taught our agent to understand and produce and turn that into business insights to improve the churn and the upsell rates of her uh organization.
So, another great thing about data bricks is we've got a way for you to visualize those insights for your leadership team. So, created this dashboard, which I'm going to zoom in back a little bit more, um, which is basically the customer call insights she's gathered in aggregate um, from her extracted fields and classifications. So she took a look at all the calls that were high churn risk and she basically were trying was trying to figure out what was the issue on the call over here.
It's pretty obvious that network connectivity is the main reason why customers are coming to Telicks knocking on the door and saying I don't want to use this product anymore. My network connectivity sucks. Which is something that for some reason they didn't know before this. But as you can tell, this is a pretty big issue that was uncovered from all this unstructured data that processed. If she wants to go even deeper into this, she can understand over time when did network connectivity issues really surge slashgo down and like what is the overall trend of them appearing in these calls as a key issue over the past year.
And what she can also gather from this insight is hey something happened in February where people have a lot of network connectivity issues. it went down, but still there's a bigger product problem at our organization that we need to address. And she can take this insight, package it, and give it to her leadership so that they can go off and fix what they need to. But product issues are one thing. Customer service issues are also another thing.
When customers come complaining or asking for help, are agents actually delivering the right experience to those customers? So, one thing was that there's not a lot of upselling happening that she's seeing. um when there's an opportunity to give a plan upgrade etc that the agents are trying to push for and what she validates with this data is okay upsell is like pretty low only 40% of agents um are actually upselling when they have the opportunity to do so it hasn't really gone up or down but maybe this is something that we need to add to our customer service training for example to incentivize people to upsell because clearly not enough people are getting the memo in fact less than less than half Churn risk is a whole another story.
So customers are basically coming and being like I'm going to leave this you know provider like telecom provider. I'm going to go to a competitor. And what we're seeing here is less and less over time our agents actually trying to stop churn from happening from the insight that she extracted from these call transcripts. And so the issue here that they really need to do is go back and understand what's missing in the training again and do we need to start incentivizing people that hey if we detect that you stop a customer from churning we'll give you x bonus for example to make sure that 7% churn they were seeing over the year which was awfully high goes down because of active agent efforts on call.
Awesome. Cool. So, another key thing here is that I showed you a cheaper agent, but how much cheaper can we really get you on task? An optimized agent brick today is 50 times cheaper than using a flagship proprietary model from a leading provider. Because what we're doing here is we're your insights, your data, and your guidelines, and we're able to effectively tune a much smaller model to do the same task for a lot cheaper. you don't necessarily need that super heavyweight general general uh generalized LLM to work on your enterprise data and that's what we're proving today by taking away that cost and quality trade-off that you have to think about cool thanks that was incredible so let's get back to our previous example and how did agent bricks help be successful firstly had a really hard time creating ideal agent that has a working prompt Simply there were too many prompts and too many models.

23:56 Deep dive into capabilities

For example, prompting techniques like chain of dot reasoning may be efficient in very complex tasks, but it may be an overkill for very simple classification tasks. Things like this should be aware of and she she should understand all the intricacies of this. Also, as updates the prompt using a few examples, those might not be generalizable for other parts of the data. All of this delved using agent bricks because agent bricks we use all the data as well as your highle task description to infer and give a intelligent recommendations of the guidelines that your agent should follow.
Also every step of the way you have evaluation supported inhouse. So previously had a really hard time systematically evaluating her agent. As a matter of fact every prompt optimization may have led to a regression on other parts of the data undetected. This is also resolved with agent bricks because with agent bricks every prompt optimization or every agent update you have an evaluation report using our in-house custom LM judges so that you have to you have the ability to track your agents performance over time.
Lastly, in most cases the agents that created were often too slow and too costly. As Sally processes millions of her data points now the LM cost would pile up and in some cases just to process that data would take weeks. This is also resolved with agent bricks you can optimize your model and then use with AI query at high scale in a costefficient manner. But this is all great but if Artica if my data is in PDF or image format does it work?
Oh no that's a great question June. Let's see. Okay. So, if I go back to my workspace and I'm trying to do information extraction again. Let's see if I can do this and if I want to point it to let's see um some PDFs that I'm going to find here default. And then where do they live? Probably in the volume somewhere here. Ah, here they are. Oh, here's a bunch of PDFs. Interesting. So if I actually want to convert these PDFs into something readable by our agent brick, I can do this super easily.
I just need to click here to automatically kick off a job that uses another SQL function that we have here at data bricks called AI parse document that will parse your PDFs into a delta table of that text for your agent to easily reason over. So here all I have to do is to use my PDFs in agent bricks. I just need to select the folder with the PDFs in it. Select a destination table. So I'm gonna put it in my own schema.
And then I'm gonna call this uh demo PDFs. Cool. And then I'm going to kick off this job and start importing my PDFs into a delta table. And then I can hook that delta table into my information extraction agent. So now I can see that this has started running. Um, and it'll take a little bit of time, but I've done this in the past and it's been successful and ready to use in your agent brick. So, we're integrating both upstream and downstream with AI parse to get your part your PDFs into parsed text and then AI query to run that super effectively on high volumes.
So, the answer to your question, June, is yes indeed you can use this on PDFs today. That's great. So, as you have seen from the demo, it's easy as ever to create and productionize your AI agents with agent bricks. You know, our research organization has worked really hard to get rid of the elephant in the room for you. So, first with agent bricks, you don't need to think about all these integration tools, all these prompting techniques and all the models.

28:26 Research and methodology

We support all of that out of the box. All you need to do is provide us with a highle description that you want to achieve and we'll handle the nitty-gritty details. And each step of the way of agent optimization, we will have an automated evaluation for you so that you can have a quantitative view of how your agent is performing. Underneath the hood, we use the data bricks mosaic AI agent eval platform which has custom LLM judges for you to evaluate your agents quality.
And now third, we finally have a path forward of using your agents from creation to productionization. So using agent bricks you can create an optimized agent that is very costefficient and use AI query with your agent in very high volumes as well. So why agent bricks how is it different from any other agent creation tools out there? First for agent bricks we abstract away all the knobs and we create the most high quality optimized agent on behalf of you.
Under the hood our platform is model agnostic, our platform is technique agnostic. We try out multiple different models. We try out multiple different prompting techniques and then we use the guidelines that you have provided to steer the agent to higher quality. And at every step of the way, there is custom LLM judges to give you quality score that you can proceed with confidence. Second, agent bricks is a product interface for our state-of-the-art research organization.
By using agent bricks, you get to use all the state-of-the-art research that our in-house research team is developing. For example, we're using synthetic data generation. We're using model training with hyperparameter training, hyperparameter search. We're using reinforcement learning with human preference. Underneath the hood, we will try out all these different techniques and we'll only provide the techniques that works best for your task and we'll select the best agent that works best for your tasks.
Lastly, it is super easy to use your agent end to end regardless of what your data format is. We support delta tables. We support text files. We support PDFs. We support images and support for other file formats such as DOCX and Excel is on its way as well. You can easily integrate your tasks and your agents to these um input formats. Also using AI query, you can use your agents in extremely high volume in a costefficient manner.
If you have a use case where you need to use your agent in a streaming fashion where you have a ongoing data inflow where you need to apply your agent you can easily integrate with lakeflow declarative pipelines to create an automatic pipeline to use your agent to extract a data or convert a data. Lastly, once you have a data that is extracted or converted using agent bricks, you have plethora of apps such as data bricks AIBI genie to visualize those apps and make meaningful business decisions.
So now that we've taken a look at how we help you productionize agents using agent bricks, let's do a small deep dive on how we build client on quality. One of the core components that our research organization work really hard is on agent learning and agent stability. So how do we do this? First in order to optimize quality, we iterate on user feedback. And for this user feedback, this user feedback takes in a form of natural language guidelines.
So everything starts with you providing us with a highle description of your task that you want to achieve. Given this, we automatically create a prompt that works best for you. We analyze your highle task description as well as your data. Create a optimized prompt for you. And note that at every step of the way, you will have agent evaluation to evaluate your agent's performance so that you can give guidance and you can give a you can have a quantitative evaluation of your agent as well.
If the very first agent that we have created for you has missing gaps, you can iteratively update your agent by using our recction system. For example, in our custom LLM agent which is specialized to create a agent that follows enterprise specific guidelines. We have a recommendation system where we recommend guidelines that your agent should follow. You can accept or reject those guidelines or you can even give custom user feedback on a subset of agent responses on a information extraction agent which is specialized to create a JSON from unstructured data.
We also give recommendations on descriptions for each of the keys of the JSON that you want to create. So now that have all these guidelines, these guidelines are now fed back into the prompt optimization loop to create a more ideal higher quality agent. And these guidelines are also use components of model optimization and model training as well. For example, we use these guidelines to create custom LLM judges. So these judges are using your guidelines.
They understand your domain specific knowledge and therefore we can evaluate your agent in a more high quality and more thorough fashion. Also these guidelines are used in reinforcement learning with human preference training process so that we can optimize on your quality even further. Now we've talked about how we hill climb on quality. Let's talk about cost. Agent bricks was built so that we can help you productionize your agent.

33:39 Cost and quality analysis

In this creating a costefficient agent is super super significant. Therefore during this process during the training and optimization process we leverage this technique called model distillation where we train a more smaller costefficient model from a bigger more intelligent model such that we have a more costefficient model that actually understands your guidelines that understands your enterprise data. So now this this is also shown in the prito curve here.
This is a per curve that shows a balance between as well as model cost. On the left bottom side, you'll see Llama 3 AP model, which is super efficient, but not as high quality as the other models. On the top right corner, you'll see the current state-of-the-art model, which is really, really high quality, but due to its sheer size, not as costefficient. With agent bricks, you actually get the best of both worlds. You get an agent that is costefficient that is trained on your data that is trained on your enterprise guidelines and because of its specialized nature because of its model distillation technique that we apply actually agent bricks has similar performance of that of the state state-of-the-art model with only 150th of the cost.
Also agent bricks has similar cost of that of llama 8b but it will have over 20% significant boost in the quality as well. So now we talked about how agent bricks is able to give you costefficient high quality model. Let's talk about our how our customers are using it. Awesome. So as you probably saw in our keynote today, we've been working really closely with customers to basically test out these research techniques and how we're optimizing their agents to see if it actually gets them the value that they need.

35:15 Customer success and takeaways

We have Flow Health for example here um who's using Agent Bricks and the techniques that power it to deliver higher quality results at a significantly lower cost. And what they really care about doing is adhering to medical and safety guidelines. So they use custom LLM to basically train this agent to adhere to those guidelines and ensure that their chat model only gives medically safe and accurate results at an even lower cost at the volume they're operating on because they have hundreds of millions of users across the world that could be talking to this chat model every single day.
In addition, we've been working with Astroenica and they found agent bricks super valuable to parse through over almost half a million clinical trial documents and extract structured data points from it. Previously, they had complex unstructured data. They weren't really sure how to actually get real insights from it. And now they have something that's actually usable for downstream analytics, which is exactly the same thing that we saw do.
She turned a mountain of a 100red million customer call transcripts into aggregated insights that she could go to her leadership team and tell a story with. So the TLVR here is that you should all try agent bricks today. We're in beta and it may be available in your workspace already under the agents tab in our left side navbar or if it isn't ask your work workspace admin and see if they can turn on the toggle in the previews portal.
Our regional availability as of today is listed uh as you can see on the slide, but we're going to be coming to more regions very soon. And if there are the key any key takeaways that you should have from this session and from your understanding of specializing agents at scale for production, it's these three. First is that agent bricks extract the knobs away from you that you shouldn't have to worry about. Oftent times they're distraction and they're not actually effectively getting you to higher quality.
We understand your task. We work off of your enterprise data and we use data intelligence to get you to the quality that you need without having to worry about all these underlying components. Second, using our research techniques via agent learning, we will eliminate the quality versus cost trade-off for you. You no longer have to go to production thinking that there's a trade-off between using a better model for a higher price.
We're going to get you that really good model and we're going to get it to you for a much lower price and one that you can actually productionize at scale. And lastly, you have to see it to believe it. So, we're going to generate for you throughout the process, whether it's updating your agent or optimizing your agent, custom evaluations to drive the confidence to actually bring this into production and systematically know that you can do this today.
With that said, thank you guys so much for coming to our session. Um, and we'll be around to take questions. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.