How can we reduce LLM and AI agent inference costs?

Summary

• Databricks reduces large language model (LLM) and AI agent inference costs by helping you right-size models, automate optimization, and choose efficient serving and inference patterns, all on one governed platform.
• Agent Bricks automatically tests across available models, produces a cost-quality tradeoff analysis, and recommends the most cost-effective configuration; it can also compare your active agent against a version optimized for cost.
• Automated prompt optimization and fine-tuning let you deploy smaller, task-specific models instead of always calling frontier models for every request.
• Batch inference on Model Serving applies LLMs to large datasets in one run, directly on governed data via SQL or Python, without duplicating data.
• Unity Gateway centralizes usage tracking, rate limits, and model routing so you can allocate and control spend across every model endpoint.

How can we reduce LLM and AI agent inference costs?

Inference is often the largest ongoing cost of running LLMs and AI agents in production. On Databricks, you lower that cost by matching each task to an appropriately sized model, automating the tuning that makes agents efficient, and picking the right serving and inference pattern, while keeping everything governed and observable in one place. The goal is to pay for the capability each workload actually needs rather than defaulting to the most expensive option for every call.

Why Databricks Agent Bricks for reducing LLM and AI agent inference costs

• Right-size the model automatically. Agent Bricks tests across the available models, produces a cost-quality tradeoff analysis, and recommends the configuration that meets your quality bar at the lowest cost.
• Optimize agents and prompts automatically. Agent Bricks compares multiple optimization strategies, including automated prompt optimization and foundation-model fine-tuning; with a set of example inputs (Databricks recommends at least 100), it returns a comparison of your currently active agent and an agent optimized for cost so you can deploy the more efficient one.
• Use smaller, fine-tuned, task-specific models. Fine-tune models on your own data so a compact, specialized model can handle a task, rather than routing every request to a large frontier model.
• Run batch inference on your data. Model Serving supports batch LLM inference to apply models to large datasets in one run, using SQL or Python in your existing pipelines and operating directly on governed data without data duplication.
• Choose the right serving mode. Foundation models are available with pay-per-token or provisioned-throughput pricing; you can also front external providers through Databricks with centralized governance and logging, or deploy your own custom and fine-tuned models on Model Serving.
• Centralize cost control with Unity Gateway. The Gateway provides usage tracking per endpoint, rate limits, and the ability to route across models, so you can allocate spend, cap runaway usage, and send workloads to the most efficient model.
• Call models from SQL with unified governance. AI Functions such as ai_query, ai_classify, and ai_extract run models directly from SQL without separate orchestration, and the platform keeps one set of traces, rate limits, and permissions across all models to cut operational overhead.

Getting started

• Read the announcement: Agent Bricks at Data + AI Summit.
• Review model serving and inference options: Mosaic AI announcements at Data + AI Summit.
• Explore serving modes and deployable models: Databricks Model Serving.

FAQs

How does Agent Bricks help lower inference cost?

Agent Bricks tests across available models, analyzes the cost-quality tradeoff, and applies optimization strategies such as prompt optimization and fine-tuning, then reports a comparison of your active agent versus an agent optimized for cost so you can deploy the more efficient configuration.

Does batch inference reduce cost?

Batch inference on Model Serving applies an LLM to a large dataset in a single run, directly on governed data via SQL or Python, so you avoid per-request orchestration overhead and unnecessary data movement.

What serving pricing options are available?

Databricks-hosted foundation models offer pay-per-token or provisioned-throughput pricing, and you can also front external models or deploy your own custom and fine-tuned models on Model Serving.

How do I track and control model spend?

Unity Gateway provides usage tracking per endpoint, rate limits, and model routing, giving you centralized visibility and control over inference spend.