how to scale llm applications

Summary

• Scaling an LLM application means handling more traffic, more models and more teams while keeping cost, latency and quality under control.
• Treat capacity, traffic management, cost control, long-running work and quality as separate problems, each with its own control.
• On Databricks, Foundation Model APIs offer pay-per-token and provisioned throughput modes, Model Serving endpoints scale with demand, and Unity Gateway adds rate limits, traffic splitting, fallbacks, usage tracking and budgets.
• Agent Bricks hosts agents on Agent Runtime, and DurableAgentServer persists run state and recovers interrupted runs. MLflow tracing and evaluation keep quality measurable as traffic grows.

how to scale llm applications

Scale an LLM application by giving each pressure point its own control: capacity for bursts, rate limits and fallbacks for traffic, usage tracking and budgets for spend, durable execution for long runs, and traces and evaluation for quality. On Databricks, Foundation Model APIs and Model Serving provide capacity, Unity Gateway provides traffic and cost controls, and Agent Bricks and MLflow cover agent hosting and quality.

What does scaling mean for LLM applications?

Scaling usually combines four pressures.
Volume. Production means sustained traffic from many teams, plus bursts. Capacity and fair sharing both matter.
Complexity. Different tasks need different models, and more agents call more tools.
Cost and latency. Token spend grows with request volume, and model choice affects both cost and latency.
Governance and reliability. More teams need access controls, spend visibility and a plan for when a provider fails.

Scaling challenge	What to look for
Bursty or sustained volume	Shared capacity for exploration, dedicated capacity for production, and autoscaling
Fair sharing across teams	Rate limits per service, user and group
Provider or model failure	Fallback destinations and traffic splitting
Model variety	One API across hosted and external models
Spend growth	Usage tracking, cost attribution by team, and budgets with alerts or blocking
Long-running agent work	Persistent run state and recovery after crashes
Quality drift	Traces, evaluation datasets and production monitoring

How Databricks approaches scaling LLM applications

Capacity. Foundation Model APIs come in modes. Pay-per-token is the easiest way to start, with a priority variant for latency-sensitive requests. Provisioned throughput is recommended for all production workloads, on-demand with no term commitment or reserved for a fixed one- or three-month term. AI Functions optimized models are recommended for batch inference. Model Serving scales up and down with demand on serverless compute, and custom-model endpoints scale with traffic and the provisioned concurrency you configure.
Traffic management. Unity Gateway gives agents one API for frontier and open models, including external providers, so you can switch models without changing agent code. Rate limits apply per service, user or group, in requests per minute for model and MCP services and tokens per minute for model services. No limits are configured by default, and a request over a limit gets HTTP 429. Traffic splitting spreads requests across up to five destinations to roll out new models, run A/B tests or spread load across providers. Fallbacks retry a failed request, for example after a 429 or 5xx error, against backup destinations in the order you set.
Cost control. Unity Gateway logs token usage and latency to system.ai_gateway.usage, and billing records attribute cost to model services, target models, principals and service tags. Budgets set shared or per-user thresholds and can send an alert, block usage, or both. Databricks says they use near-real-time estimates and shouldn't be treated as an absolute cap on billed spend. They don't cover provisioned throughput.
Agent reliability. Agent Runtime hosts agents on Databricks Apps. Once deployed, DurableAgentServer keeps run state through worker restarts, and when a run's heartbeats stop, the server starts a replacement attempt through a recovery handler you register. Recovery covers interrupted workers: if your handler raises an exception, the invocation fails and isn't retried. Recovery can run agent code more than once, so make external calls idempotent. The Agent Bricks CLI that scaffolds and deploys these agents is in Beta.
Quality. MLflow tracing records each step an agent takes, evaluation scores outputs against datasets, and production monitoring (Beta) scores a sample of live traces.

Use cases for scaling

• Many teams on shared models. Per-user and per-group rate limits protect capacity, and service tags attribute spend by team.
• Rolling out a new model. Route a share of traffic to the new model, then adjust the split as results come in.
• Surviving a provider problem. Configure fallback destinations so failed requests retry against backups.
• Batch enrichment. Run batch inference with AI Functions on models optimized for it.
• Long-running agent tasks. Use DurableAgentServer so run state survives a worker restart and interrupted runs recover.

FAQs

How do I keep costs from growing out of control?

Combine usage tracking, cost attribution and budgets. Unity Gateway records token usage per request, billing records attribute cost to services, models, principals and tags, and a budget can alert or block usage at a threshold. Treat budgets as near-real-time controls, not guaranteed caps.

What happens if my primary model provider has an outage?

If you configured fallback destinations on the model service, Unity Gateway retries a failed request against the backups in the order you set. Traffic splitting picks the primary destination and isn't re-applied during retries.

How do I choose between pay-per-token and provisioned throughput?

Start with pay-per-token. Databricks recommends provisioned throughput for all production workloads, especially those that need high throughput, performance guarantees, fine-tuned models or additional security requirements. For latency-sensitive pay-per-token traffic, priority mode admits requests ahead of standard best-effort traffic.

Can I switch models without rewriting my application?

Yes. Unity Gateway gives your agent one API for frontier and open models, so you can switch models without changing agent code.