How to use prompt caching and model routing to cut token spend

Summary

• Prompt caching stores frequently used context and system prompts so they are not re-tokenized every request, reducing redundant token consumption for applications that process many similar queries or maintain consistent instructions.
• Model routing directs requests to the most cost-effective model for the task: simple queries go to faster, lighter models while complex reasoning queries use larger, more capable models, avoiding unnecessary compute.
• Combining caching and routing can substantially reduce token volume and model compute cost by ensuring each request uses only the model capacity and tokenization needed.
• Databricks enables both techniques through Model Serving and unified endpoints, allowing teams to optimize token spend at scale without managing multiple deployments.

How to use prompt caching and model routing to cut token spend

Large language model inference costs are driven by token consumption and model compute. Two complementary optimization strategies reduce both: prompt caching eliminates redundant tokenization of repeated context, and model routing ensures each request uses only the model capacity it needs. Together, they lower per-request token cost and model overhead without sacrificing quality or latency.

What is prompt caching?

Prompt caching stores tokens from frequently used context, system instructions, or reference materials so they are reused across requests rather than re-tokenized. For example, if every request includes a large product catalog or detailed instruction set, those tokens are cached after the first request. Subsequent requests that reuse that context skip the tokenization step, reducing total token consumption per batch of similar requests.

What is model routing?

Model routing directs requests to different models based on task complexity or characteristics. Simple queries (syntax checking, formatting) go to smaller or faster models. Complex reasoning queries (data analysis, code generation) go to larger, more capable models. A routing layer evaluates requests and picks the most appropriate model, avoiding unnecessary compute overhead.

Implementing prompt caching and model routing

Strategy	How it works
Prompt caching	Store tokenized context (system prompts, reference documents, code libraries) after first tokenization. Reuse cached tokens across requests that share that context.
Static vs. dynamic caching	Static: store fixed context that never changes. Dynamic: cache context stable within a session or batch but varying across sessions.
Model routing by task	Classify task type from request content. Route simple tasks to small models, complex tasks to large models.
Cost-aware routing	Track per-request token count and latency. Use faster models for simple tasks, fall back to capable models when needed.
Batching with caching	Process requests in batches with shared context. Cache reused across entire batch, multiplying savings.

How Databricks approaches token cost optimization

Databricks provides multiple tools to reduce token spend:

• Model Serving endpoints: Deploy multiple models (small, medium, large) as dedicated endpoints. Use a routing layer to classify requests and call the appropriate endpoint. Cache context in Delta tables or embed it in request payloads.
• Foundational model endpoints: Access foundational models via Foundation Model Endpoints. Use routing logic to choose which model and endpoint based on request characteristics.
• Batch processing: For non-real-time workloads, use Lakeflow Pipelines or Lakeflow Jobs to process requests in batches. Batches maximize cache reuse and enable economies of scale.
• Observability and cost tracking: Log token counts, latency, and model choice per request. Use this data to refine routing rules and validate that caching is reducing token consumption.
• Prompt optimization: Experiment with prompt formulations that minimize token count while maintaining output quality. Store optimized prompts in Delta tables and version them.

Use cases

• Code generation: Platforms that generate code for many concurrent users cache system prompts and style guides once. Routing simple completions to smaller models and complex refactoring to larger models minimizes token overhead for each request type.
• Customer support automation: Support systems process tickets with large policy templates and FAQs. Caching templates and routing yes/no questions to small models while escalations go to large models cuts token consumption significantly.
• Data analysis and reporting: BI tools generating SQL and narratives cache schema and business logic. Routing simple metrics to small models and complex analysis to large models optimizes cost without sacrificing insight quality.
• Internal workflow acceleration: AI-powered IDEs and code assistants cache user preferences and code context once per session, then route syntax suggestions to light models and architectural advice to capable models.

FAQs

How much can prompt caching reduce token spend?

That depends on your workload. If you process many requests with large, repeated context (product catalogs, code libraries, policies), caching reduces token consumption on those requests substantially. Model routing multiplies savings by avoiding unnecessary use of larger models. Together, the impact scales with workload size and repetition.

What's the trade-off with latency?

Prompt caching may add small latency overhead on first requests (cache misses) but improves latency on cached requests by skipping tokenization. Model routing may add latency at the routing decision point but usually recovers it by using faster models for simple tasks. Well-tuned routing typically improves both latency and cost.

How do I set up a routing layer?

Write a Python service that evaluates incoming requests against your routing rules (request size, key terms, user tier) and calls the appropriate Model Serving endpoint. Log decisions and outcomes to refine rules over time. Start simple and add sophistication as you gather data.

Can I use model routing with external LLM providers?

Yes. Route requests to different provider endpoints or model families based on your optimization criteria. However, manage external costs and API rate limits separately.

How do I know if my routing strategy is working?

Monitor three metrics: average token count per request, average model inference cost per request, and latency percentiles. If routing is working, token count and cost should decrease while latency stays stable or improves.