Building Agents on Databricks Natively with Claude-3.7-Sonnet

Summary

• Native Claude Integration: Build AI agents natively on Databricks using Anthropic's Claude-3.7-Sonnet, improving security and performance with no external APIs required.
• Tool-Calling Chatbots: Create customer support chatbots with tool-calling capabilities by registering functions in Unity Catalog for the LLM to invoke.
• Production Deployment: Register your agent in Unity Catalog and deploy through the AI Gateway with built-in evaluation metrics and human-in-the-loop feedback.

Building Agents on Databricks Natively with Claude-3.7-Sonnet

Build and deploy AI agents natively on Databricks using Anthropic's Claude-3.7-Sonnet. Improve security, performance, and scale—no external APIs required.
In this demo, you will build a tool-calling customer support chatbot with Anthropic's Claude-3.7-Sonnet. After building your agent, you will experiment with LLM-as-a-judge evaluation metrics, learn how to add human-in-the-loop feedback with our SME (subject matter expert) review app, and then prepare your chatbot agent for production. This includes registering your agent in Unity catalog and deploying your agent through the AI Gateway.

Chapters

• 0:00 Building the agent
• 2:44 Experimentation and eval
• 6:12 Deployment and feedback
• 8:23 Monitoring and observability

FAQs

How do you register tools for an agent built with Claude on Databricks?

Tools are registered as functions in Unity Catalog. This video demonstrates registering functions like get_customer_history, get_product_info, and search_return_policy, which the Claude model can invoke when responding to user queries.

Can the agent in this video access external data sources like vector databases?

Yes. One of the registered functions is search_return_policy, which queries a vector database. When the LLM calls that function, it retrieves relevant policy information to answer customer questions accurately.

What evaluation methods are shown in this video for improving agent quality?

The video demonstrates LLM-as-a-judge evaluation metrics to assess agent responses. It also shows how to add human-in-the-loop feedback using a subject matter expert review app to continuously improve the agent's quality.

How do you deploy the agent to production on Databricks?

After building and testing your agent, you register it in Unity Catalog and deploy it through the AI Gateway, which provides governance, monitoring, and control over the agent's execution.

Full transcript

0:00 Building the agent

All right, so let's get started on building this agent. The first thing you're going to want to think about is what's my use case? So, we're going to build a customer support chatbot for a company called Client Care. And first step is selecting the LLM that's going to act as the brain behind the agent for figuring out what tools should I be using in order to respond to this user. Because our cloud 3. 7 sonnet model is now available natively in data bricks.
We're going to select that as our LLM and then select a series of tools. I've already register these tools as functions in Unity catalog. So when you're thinking about what skills you want to give the LLM, just make sure they're registered to UC. So the functions I have available here are get customer history, get the product info, and search return policy. The search return policy is a vector database. So when the LLM calls that function, it's going to be querying a vector database.
I've already added a system prompt. So you are a customer support chatbot for the company client care. Just going to keep it real simple. You can make this as verbose as you would like. And then a very common question you're going to get from a customer if you're a support chatbot is, can you help me with my order and my customer ID is 1 2 3 4. So we're going to send this to the agent that we're building. So, if you're the clawed model, you're going to think about what tools I have available.
If you're asking to return an order, you're going to look up that customer with a customer ID or email. You're going to get that customer history. So, here we can see we're getting the customer history. We're going to get the items that customer has purchased and see which item they're most likely wanting to return. Great. So, we do see that it's likely that they're going to want to return the wireless headphones. So, I'm going to go ahead and say yes, I'm looking to funds.
All right. So, the model then pulled up the item ID, looked at the various information from the return policy. So, it pulled the vector database for the returns and said, "All right, so this item was purchased on March 25th. We have a 30-day return policy. You'll need the original packaging, but you're able to return." And I can continue iterating on this process, maybe creating a return label, seeing if they have any other questions.
But I really like how this agent is built. So, I can then export this code to a notebook and continue to iterate on the quality. And then I can even deploy the agent to share it with others. So once I export this file, I'm able to then experiment with evaluations. These evaluations are either LLM as a judge where you have a third party LLM that's labeling these evaluations or human in the loop feedback. I'm going to show you both now.

2:44 Experimentation and eval

So for experiments, if I go into to my experiment runs here, I can see I have two versions of my chatbot. So I'm going to go into my first version where I run a set of evaluation metrics. We can have our overall evaluation and I have correctness here. Tools like safety and then you can add in as many evaluations or custom evals as you would like. So we can see that overall the agent is doing pretty well for the question did my payment for the Eco Dry clothes dryer go through?
We can click into that, see the request, see the response saying yes, I can confirm it went through. And adding in expected facts or given ground truth, we see that we did confirm that the payment went through. We can look at additional details as well. See the detailed trace view to know exactly what was called. So the getting customer history found additional information to confirm that payment did go through. So, that looks great.
And then there seems to be a problem with the is there a recall for the Blend Master 3000. I'm going to click into this query. And we see that overall it failed. And if you recall the initial functions that we gave the agent access to, none of them were specifically for recalled items. So maybe we need to make another function or give our current functions access to a list of recalled items. Here's the request. there's a recall for the Blind Master 3000.
Our expectation is that the LM will know that there was a recall on March 10th, but as we can see, there's no information here about the Blend Master 3000. So, I'm going to iterate on this agent and maybe create a secondary function that has access to what items were likely recalled or when they were recalled. Looking at my set of runs, I can go to my version one. So my next version and in this version of the chatbot we see that the correctness is at 100%.
So all the eval are performing as expected. The queries are getting the respected responses and this is great performance for our agent. Now if we go back to is there a recall for the blend master 3000 we can see that based on the request expected facts we're now seeing an output of yes I can confirm there was a recall. It was recalled on March 10th and I can help you with the next steps for that return. Going into the details again, correctness, safe and the more eval you have, the more details you will get.
If I see the detailed trace view, I can now see that the client care agent is calling recall status, which is a new function that I added to this version of the agent. So just editing these capabilities and making sure that there's continuous updates to the data you give these agents access to and they're accessing the correct data. Now these are for LM as a judge evaluations. What if you want to be able to provide human in the loop feedback for this agent?
And that's going to come from our subject matter expert review app. So once I'm happy with the performance of this agent, I'm going to serve that model. It will be registered in Unity catalog and it will be deployed with a deployment endpoint. I can click into my deployed agent and now I can see the version of the agent and I can choose to either continue to iterate on this on playground. Use this for batch inference or I can open the review app.

6:12 Deployment and feedback

Here in my review app I can select the client care support chatbot and begin a new chat with my customer support chatbot. So, I can ask, can you help me return my let's say the item is broken? Um, and I'll still put in the customer ID. And this agent's going to have access to the same capabilities that we have given access through all through this demo. So, we can see that the agent is pulling up customer history, looking at the output, and looking at the purchases.
We might say, "This is a good response. It's accurate. It's relevant. Additional feedback. I see they're showing both purchase items. uh only show relevant purchases or just show one purchase or the latest purchase. You can provide this feedback. That feedback is saved and those inferences are stored in Unity catalog. You can then even have tasks. So if you have a subject matter expert or someone on your team or a third party labeler, you can send them a request to start annotating or editing responses.
And all that feedback again is stored in Unity catalog and can be used to fine-tune your model. So just a quick recap, what we've done so far is create a tool calling agent using claude 3. 7 sonnet as the LLM. This agent has been registered in Unity catalog. It's been deployed as a review app. So you can collect human in the loop feedback. We've run LLM as a judge evals. Now let's see how this looks in dashboard capabilities where we're running payload logging with AI gateway.
That way we can see all the observability and monitoring through dashboards. So if we look at our dashboard capabilities, we can see all the raw payloads, unpacked payloads, put in LLM fallback. So you might have a Claude sonnet region one, Claude region two, and be able to track if there's ever a status issue or something going on with routing. So regardless what happens, your customers are always going to be able to have a good experience with that customer support chatbot.

8:23 Monitoring and observability

You could track usage for all endpoints using AI gateway or maybe just looking at a single endpoint if you're concerned about admin issues like usage, token count, and cost. as well as looking for PII detection, evaluation, and observability.

Learn more about the Databricks Data and AI platform.