Generating Laughter: Testing and Evaluating the Success of LLMs for Comedy

Summary

• Creative AI evaluation: Testing nondeterministic AI models like LLMs requires new approaches beyond traditional accuracy benchmarks and rigid standards.
• Multi-tiered feedback loops: This video explores methods including chaos testing and exploratory user testing where AI outputs are evaluated by adaptability and resonance.
• Root source of truth: Establishing a reliable dataset or core principle enables consistency and manages creativity while embracing LLM unpredictability.

Generating Laughter: Testing and Evaluating the Success of LLMs for Comedy

Talk by Erin Staples, Sr. Developer Experience Engineer, Galileo
Nondeterministic AI models, like large language models (LLMs), offer immense creative potential but require new approaches to testing and scalability. Drawing from her experience running New York Times-featured Generative AI comedy shows, Erin uncovers how traditional benchmarks may fall short and how embracing unpredictability can lead to innovative, laugh-inducing results. This talk will explore methods like multi-tiered feedback loops, chaos testing and exploratory user testing, where AI outputs are evaluated not by rigid accuracy standards but by their adaptability and resonance across different contexts — from comedy generation to functional applications. Erin will emphasize the importance of establishing a root source of truth — a reliable dataset or core principle — to manage consistency while embracing creativity. Whether you’re looking to generate a few laughs of your own or explore creative uses of Generative AI, this talk will inspire and delight enthusiasts of all levels.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How do you test LLMs for creative applications like comedy?

Traditional benchmarks and accuracy metrics don't work for creative AI where the goal is engaging output rather than precise answers. This video explores methods like multi-tiered feedback loops, chaos testing and exploratory user testing where success is measured by resonance and adaptability across contexts.

What is chaos testing for AI?

Chaos testing intentionally tries to break AI systems with edge cases and unusual inputs. For creative AI applications, this testing reveals how the model adapts and whether outputs remain engaging and relevant even when pushed to extremes.

Why is unpredictability valuable in creative AI?

While most AI applications optimize for consistency, creative AI requires variability and unexpected outputs to be interesting. This video shows how to embrace non-determinism as a feature rather than a bug, optimizing for metrics that reward creativity.

How do you establish ground truth for creative AI?

Instead of rigid accuracy standards, creative AI needs a reliable dataset or core principle to manage consistency. This might be guidelines, core story elements or evaluation principles that allow creativity to flourish while maintaining coherence and avoiding completely off-topic outputs.

Full transcript

All right. Good afternoon, data bricks. Hello. Hello. Can you hear? Perfect. Thank you. Good afternoon. Let's give it up for an amazing conference and everybody here so far and all the speakers that have taken the stage. Can I get a quick round of applause for those? Awesome. Today we're going to go over generating laughter. How the heck do you evaluate something that's very subjective with objective metrics? Let's dive in.
For those of you who don't know me. I am a developer experience engineer by day at Galileo, but by night I get up to some other shenanigans. I'm a New York Times featured comedian. I've produced comedy shows and fine-tuned models to roast humans. I ran a project called Comedy Bites. In this project, we evaluated, is AI really going to take our jobs? I doubled down and said, "Bet. Let's see. Can AI take our jobs when it comes to comedy?"
Now, we're thinking things through. This is data bricks data and AI summit. Give the people what they want. Uh, as the robot said, and apparently it's not coming through on the robots, but as the robot said, or my clone to be exact. You probably are not here to give me a comedy set. You guys all came for a little bit of the data, right? A little bit of the AI. She's like, "No, sign me up for the comedy bit." Let's walk through what we're getting into today.
We're going to talk about what happens when you give a large language model a little too much power and not enough supervision. And then we're going to ask it to be funny together. And we'll evaluate that together. Building creative tools is a fun task and we're going to build through a fun little app that I built that's open source if you want to go play around with it today, the Startup Simulator 3000. And then we're going to walk through metrics.
How did we build those custom metrics? How did we evaluate that success? Let's dive in and explore. Now, what if I told you that non-determinism isn't a bug, it's the bit. Most of us are taught to treat variability as the problem. They want the same output. You've probably, for those of you building AI apps, can I get a show of hands who's built an AI app, even for fun or for serious? Awesome. Thank you. um you're usually optimizing for consistency, reliability.
However, what happens when you don't? What happens when the variance is the funny part? How do you optimize metrics for that? The punch line doesn't always land. It takes context to make it funny. LLM or foundation models are unpredictable by design. running a comedy show in New York City, I need to know how do I get this opportunity to repeat itself? What items do I want to be consistent and what items do I want to be regular?
Well, it requires us to keep one foot on the ground. Yes, that means using ground truth data sets in order to influence your end result. Let's dive into a quick little example. For those of you who wanted the comedy bit, I saw you earlier. We're going to get weird. Welcome to my very exciting app. Uh I'm assuming some of you work at a startup or have a ideas of running a startup or at one point in your life you may be a founder yourself sitting in this room.
I wanted to know can I make a good startup pitch generator but with a twist. my startup sim generator. We're gonna have it I'm gonna have some audience participation here. So, we're going to have it test and see can we build a startup together. Who says we should do silly first or who says we do serious? Silly or serious? Silly. I love it. All right, I'm going to need some help here. Who wants to give me an industry?
What industry are we starting our startup in? What industry? Fintech. Someone's going to be a fintech founder here. So, behind this application, this is an agent application. I've got a couple of things. It's running models. It's going to query the hit TV show Silicon Valley. Um, it's also commenting from Hacker News, snarky comments that I have filtered out. And it's got a data set of satirical YouTube and comedy metrics from comedy materials.
Who's our target audience? What? deeds. Oh, ID. And then someone want to toss out a random word. And we're going to have it generate this startup. So, it's looking at hacker news trends. It's analyzing the startup ecosystem to see what's out there. And then I have it modeling some absurdity. I've plugged in my own satire comedy in here and let's see what it comes up with. We're all This is a live code demo. This is running locally.
It's calling some API. So, we're at the mercy of some conference Wi-Fi, but should be fun here. While we're doing this, we'll talk about how we want to metrics their success. I'm not m I am not naturally looking for the perfect startup. That's what Sirius mode is for. In the serious mode, it'll query common pitch duct pitch deck structures and it also queries tech crunch to see who recently got funding. Let's let it load.
We're going to see a little bit. Again, mercy of conference Wi-Fi here. We'll see how this comes through. If not, we've got some saved ones examples. We're going to see. Keep it going. let's check the logs. Confirm that things are rocking and rolling. I can get started. I'm checking the logs. I promise I'm not cheating here. We'll pop it over here if you guys want to see, but for the sake of time because you guys don't want to watch a loading screen.
I've got our logs here and we can see what it came up with. Introducing space code. of it. We have there. See if it'll pop over now. We had our result, but we see our task right here. We'll not wait for the speed of conference Wi-Fi. We've got it here. Introducing Spacecoin. We decode the crypto clues left behind from the big bang black hole to generate a new brand of cryptographic currency. By leveraging echolleak AI vulnerabilities, our fintech startup navigates uncharted IT territories.
Pretty silly, right? Now, I want to know what likelihood does that have to make someone laugh? If I'm to look at this from a, we can see right here in Galileo, our live report. Again, we're at the speed of conference Wi-Fi. It's pulling in a couple of things. It's got my hacker news tool. We generated this just a few moments ago. It we can see exactly the choices that I made to run a silly startup. It's calling my hacker news tool.
It's looking for some keywords that I have and some popular uh satire things. And then it comes up with a startup simulator tool. So using generative AI, putting that in together to make everything work. If we dive in, I want to know how successful this can be. I can pop in. Let's go to our little beautiful metrics page. And it's going to keep loading, but we'll show you what one looks like on a finished example. If you make sound effects, I've heard it automatically makes your app better.
We can pop over here, look at our metrics, and we can see what it expects to us. So, we're seeing the action advancement, our standard agentic metrics. Um, and it's giving us, it's not sexist, it's not anything, but if we scroll down, we can see the custom metrics I put in this. Scrolling down, we see things like satire, effectiveness, cultural relevance, funny or not, um, consistency, narrative coherence. These are all metrics that I wrote myself.
And your You're probably wondering, well, how the heck did she get those? Those are not metrics we think of in our day-to-day or metrics that we think of when it comes to this. All of this is generated through a little bit of thinking. So, when using LLM as a judge, we can pop in. There's some certain known as a comedian, there's certain knowns that we always know. I went ahead and I created this. So, actually, I ran this simulator at New York Tech Week last week and a couple of other previous events.
I gathered data and I said, "What are the likely industries that were expected to see things?" I pulled those keywords out. I then took all of this data and I started using it to create a prompt. So, it's in a data set. I started playing around with it of what matters most. Then I took a page out of UCB where I went and studied satire writing and comedy and I put in common satire metrics and efficiency tools right in that.
So, we have it understands the common structure of comedic jokes. Now, you're probably wondering, this is just one example, but we've got multiple different metrics here we can kind of dive into for our look. And they're all very similar in structure and similarity here. We scroll down here. I've got a satire and silliness as a service metric and it combines all the subcategory metrics that I put together into one giant metric to determine is this a good joke or not.
We can see now at a glance what would make a good joke rather objectively by determining what worked in a previous example. All of this is ranked and given a score. Now, you're probably wondering, "That's great, but I'm probably not running an AI comedy show in New York City, and I'm probably building something else in AI for business relevant purposes." If you're a comedian, great. That's awesome. But I fully understand not everybody gets a joy of roasting themselves on stage on a regular basis.
And that's why we need to think about the different types of metrics that come with evaluations. Typically, we look at something like accuracy, token usage, latency, and safety. Notice you're not seeing my satire and silliness as a service up on this page. However, as we know from our silly mode, and this is kind of our agent tools that I've given each mode, we're going to need a different set of metrics to determine the success of this.
Now, typically, we see it in kind of a binary. So metrics operate in this dual kind of ven diagram. On one side you'll see our traditional AI evaluation metrics. That's that rouge blue safetiness trustworthiness. Then we'll add in our agentic experience metrics. Tool calling tool selection. Did it pass the right information? Agent completion metrics. But this still doesn't cover my comedy example. Where the heck does my comedy metric fit in this?
And may I pose the question, where will your subject matter expert metrics fit in? I may be a subject matter expert in comedy, but you may be a subject matter expert in healthcare. You might be a subject matter expert in finance, insurance, law. While yes, my examples about comedy, any subject m subject matter expert can apply and that's what really matters. You'll probably never find a set of metrics that is perfect right out of the box for your use case.
And that's okay. When we measure metrics, we need to make sure that we're measuring them against what we want to see. In my case, it's comedy. In your case, it might be a different subject matter expertise. Start thinking about how you can develop it and build your own set of metrics for your needs. Let's go beyond comedy. Let's dive into healthcare. let's dive into legal or maybe heck, maybe who's going to build the first metric for dog training or so on and so forth.
But when it comes down to it, you need to really think about what is important to measuring in your case. So, let's dive into that and wrap things up in a bow. First off, start with key metrics. What is your make or break in this world? What do you need to have over any other type of metric that's out there? Is it safety? Is it PII? Then establish your baselines. What works for you and what do you expect? You can get something with a really, really high latency and have your application spitting out answers rapidly.
However, if it hallucinates or drops a bunch of errors, that's not worth it. Track trends over time. How many people have built something and then there's a new model update and your app completely breaks? Uh, I see it. Yeah, we're we've been there. It's a painful one. Make sure that you're understanding and identifying key dates and trends over time. Are you refreshing your data sets? Are you diving into a new model update?
Are you changing the deployment process? These are all things that can kind of happen and influence. And by having timebased metrics, you can directly see where they might impact your end metrics. And again, as mentioned previously, we want to combine multiple metrics. There is no one right metrics out of the box. And that's okay. At the end of the day, you're looking for the best comp compilation of metrics that fit your needs.
Now, set thresholds. What is successful? My threshold for comedy, I'm okay with it varying a little bit. I want it to be weird and off-the-wall. However, my threshold for say a customer support app, if I'm giving it three different answers and three different ways to do things, you can all imagine that I'm going to be hearing a good phone call from someone or someone's going to be rage quitting that chatbot. So again, here's that beautiful slide if you want to take a screenshot.
And all of this will be available online afterwards. All of this comes together and says, make sure that you're choosing the right metric stack for you. While yes, my app today is dealing comedy. your app today might be a different subject matter experts. The most important part when we're thinking about the metrics for success, whether it's comedy or we're thinking about healthcare or finance, is to determine and make sure that subject matter expert is kept within the loop and you're building alongside them.
Building in silos helps nobody. So, if you liked this demo and you like this example and maybe you want to build your own version of the startup simulator 3000, this will be up soon on the Galileo SDK repo, so feel free to play it. Um, it'll probably be up by this afternoon. Um, we love when we have like version controls and PR reviews. But if there's any other questions, you can reach out to me at agalo. ai. If you enjoyed this talk, um, I'm open to questions if you've got anything.
Thank you.

Learn more about the Databricks Data and AI platform.