How can I prevent runaway LLM spend?

Summary

• Runaway LLM spend comes from calls you can't see or cap: an agent stuck in a loop, retries with no ceiling, or everyone defaulting to the most expensive model. The fix is to route every call through one control point.
• On Databricks that control point is the Unity Gateway. Every request routes through it, so spend becomes visible, cappable, and attributable instead of scattered across direct-to-model calls.
• Rate limits are the fastest prevention: caps on queries per minute and tokens per minute, per endpoint or per user, reject excess traffic and stop a runaway loop before it gets expensive.
• Budgets add spend thresholds that alert and can block further usage once crossed. Enforcement runs on near-real-time estimates, so pair budgets with rate limits and treat the billing system tables as the final record.
• Usage tracking in the system.ai_gateway.usage table and request tags make spend visible and tie it to a team or project, so cost has an owner instead of arriving as a surprise.

How can I prevent runaway LLM spend?

Route every model, agent, and tool call through one control point, then cap it, watch it, and attribute it. Runaway spend happens because direct-to-model calls are invisible and uncapped: no one sees an agent stuck in a retry loop or a job hammering the priciest model until the invoice lands. A gateway sits between your applications and the models they call, so you can set rate limits that reject excess traffic, budgets that alert and block when spend crosses a threshold, and tags that tie usage to a team. On Databricks, that control point is the Unity Gateway.

What causes runaway LLM spend

The pattern is almost always the same: usage no one is watching. An agent calls a model in a loop, and a loop with no limit is an expensive loop. Retries stack up against a slow endpoint. A batch inference job runs over a far bigger table than anyone expected. Developers reach for the largest model on tasks a smaller one would handle. None of it is visible in the moment, because each application calls the model directly and the only aggregate view is the monthly bill. Prevention takes two things: a ceiling on how fast usage can accrue, and early visibility into where spend is going while you can still act on it.

Controls that prevent runaway spend

Control	How it prevents a surprise bill
Rate limits	Cap queries per minute and tokens per minute, per endpoint or per user, and reject excess traffic, so a stuck loop can't flood the model.
Budgets and spend thresholds	Set thresholds that alert as they approach and can block further requests once crossed.
Usage tracking	Log every request with token counts and requester, so consumption is visible in near real time, not just on the invoice.
Cost attribution	Tag requests by team, project, or environment so spend has a clear owner.
Model permissions	Control which principals can call which models, so premium models are used only where they're needed.

How Databricks approaches this with the Unity Gateway

The Unity Gateway is the single control point for model, agent, and tool traffic on Databricks. Requests route through it, and it applies rate limits, budgets, usage logging, and access control centrally rather than per application.
Cap the rate. Rate limits are set in queries per minute and tokens per minute, at the endpoint or per-user level, and excess requests are rejected. This is the fastest way to hard-stop a runaway agent loop before it accrues cost.
Set budgets. Budgets track spend against thresholds. A threshold can send an alert while access continues, or block further requests once it's crossed. Enforcement runs on near-real-time cost estimates, so a small amount of spend can occur past a threshold, and the billing system tables remain the source of truth for actual charges. That's why the durable practice is to run rate limits and budgets together: rate limits stop volume immediately, budgets watch cost.
See it and attribute it. Every request is logged to the system.ai_gateway.usage system table with token counts and requester, and a built-in usage dashboard shows spend by endpoint, model, or user over time. Request and endpoint tags let you attribute usage to a team, project, or environment, so cost has an owner. You can join the gateway usage table with the billing system tables to estimate spend by any dimension you tag.
Right-size model access. Foundation model permissions govern which principals can call which Databricks-hosted models, so routine work stays on appropriately sized models instead of defaulting to the most expensive one.
Beyond cost, the Unity Gateway also applies guardrails and service policies that govern model inputs and outputs and which tools agents may call. These governance capabilities are evolving, so confirm the current lifecycle state of guardrails and service policies with your Databricks account team before you design around them. For the current feature set, see what's new in the Unity Gateway.

FAQs

What's the fastest way to stop an agent from running up a bill?

Set rate limits on the endpoint. Caps on queries and tokens per minute reject excess traffic, so an agent stuck in a loop is throttled before it accrues much cost. Rate limits act as a firm throughput ceiling today.

Can a budget hard-stop spend on its own?

A budget can block further requests once a threshold is crossed, but it's enforced on near-real-time estimates, so spend can slightly overshoot and the billing system tables are the final record. Pair budgets with rate limits for a firm ceiling.

How do I see which application is driving my LLM bill?

Query the system.ai_gateway.usage system table or open the built-in usage dashboard. Both show consumption by endpoint, model, and user, and request tags let you roll spend up to a team or project.

How do I keep costs down without blocking developers?

Use model permissions to keep routine calls on right-sized models, set per-user rate limits rather than blanket denials, and attribute spend with tags so teams self-correct. Developers keep access; the runaway cases get capped.