Provisioned throughput vs pay-per-token for LLM serving

Summary

• Pay-per-token bills by tokens consumed with no upfront commitment, making it ideal for evaluation, prototypes, and low-volume or bursty workloads.
• Provisioned throughput reserves dedicated inference capacity billed by time, providing consistent latency and guaranteed concurrency for production applications.
• The two models differ in billing structure, performance guarantees, and the degree of isolation from other traffic.
• A common migration path is to start on pay-per-token to profile real traffic, then move to provisioned throughput once load patterns are well-understood and the latency SLA is firm.
• Databricks Foundation Model APIs offer both options under a unified API surface, so switching between them does not require changing application code.

Provisioned throughput vs pay-per-token for LLM serving

Both options let you call large language models through the Databricks Foundation Model APIs with the same request format. The difference is capacity and billing. Pay-per-token charges by consumption with no reservation; provisioned throughput reserves a slice of inference capacity and bills for it by time, regardless of how much you use. Choosing the right model depends on your traffic volume, latency requirements, and cost predictability needs.

What is pay-per-token?

Pay-per-token endpoints run on shared infrastructure. You call the endpoint, send tokens, receive tokens, and pay for the tokens consumed. There is no commitment, no warm-up, and no minimum. This model is straightforward to start with: no sizing exercise, no reservation, and costs scale naturally with usage.
The tradeoff is that shared capacity is subject to rate limits and can exhibit higher or less predictable first-token latency, especially at high concurrency or during traffic spikes. For low-volume or intermittent workloads, this rarely matters. For a user-facing chat interface where every request needs a fast response, it can.

What is provisioned throughput?

Provisioned throughput endpoints reserve dedicated inference capacity for your workload. You specify the amount of capacity in model units, and that capacity is held exclusively for your requests. Billing runs by time rather than by token consumed.
Because capacity is reserved, provisioned endpoints provide more consistent latency and predictable concurrency characteristics. There is no contention with other tenants. This matters for applications with strict latency requirements or sustained high-concurrency demands, such as a product feature with many simultaneous users.
Databricks also offers reserved provisioned throughput with term commitments (monthly or multi-month), which can reduce the per-unit cost relative to on-demand provisioned capacity.

How they compare

Dimension	Pay-per-token	Provisioned throughput
Billing model	By tokens consumed	By time (capacity reserved)
Upfront commitment	None	Capacity reservation required
Latency consistency	Variable; depends on shared load	Consistent; dedicated capacity
Concurrency guarantees	Subject to shared rate limits	Guaranteed up to reserved capacity
Fine-tuned model support	Base models only	Supported (fine-tuned variants)
Best for	Exploration, prototypes, bursty or low-volume traffic	Production apps, sustained high QPS, strict latency SLAs
Cold-start risk	Present: first token may be slower during shared load spikes	Absent: capacity is pre-allocated
Cost predictability	Variable with usage	Fixed for the reservation period

Sources: Databricks Foundation Model APIs documentation (docs.databricks.com/aws/en/machine-learning/foundation-model-apis), Databricks reserved provisioned throughput docs (docs.databricks.com/aws/en/machine-learning/foundation-model-apis/reserved-provisioned-throughput).

When to choose pay-per-token

Pay-per-token is the right starting point when you are evaluating models, running a proof-of-concept, or serving an internal tool where latency variance is tolerable. It is also appropriate for workloads that are genuinely bursty or seasonal: you pay only for what you use without holding idle capacity.

When to choose provisioned throughput

Provisioned throughput is the right choice when:

• Your application has a latency SLA. User-facing features, real-time agents, and embedding search pipelines all need consistent first-token and inter-token latency that shared capacity cannot guarantee.
• You need to serve a fine-tuned model. Provisioned throughput supports custom fine-tuned model variants; pay-per-token endpoints serve base models only.
• Your traffic is sustained and predictable. Provisioned capacity tends to be the better-value billing model at sustained high volume, because you pay for capacity held rather than per token consumed.

How Databricks approaches this in Foundation Model APIs

Databricks Foundation Model APIs expose both options under the same OpenAI-compatible endpoint format. You select the deployment type when creating the endpoint; your application code does not change. This means you can start on pay-per-token and migrate to provisioned throughput by updating the endpoint configuration rather than rearchitecting your integration.
Unity Gateway sits in front of both endpoint types and adds access control, rate limiting, budget monitoring, and audit logging. You can set per-user or per-team spending thresholds that apply regardless of which billing model the endpoint uses.
For sizing a provisioned endpoint, Databricks provides a model-units estimator in the documentation that takes your expected concurrency and token sizes as inputs. The recommended workflow is: profile real traffic on pay-per-token for several weeks, capture token distribution and peak concurrency, run the estimator, then provision accordingly.

FAQs

Can I run fine-tuned models on pay-per-token endpoints?

No. Pay-per-token endpoints on Databricks serve the base model variants listed in the supported models documentation. If you have fine-tuned a model, deploy it on a provisioned throughput endpoint.

What happens if I exceed my provisioned throughput capacity?

Requests that exceed your reserved capacity receive an error response rather than queuing indefinitely. Design your application to handle throttling gracefully, or provision enough headroom for your peak load.

How do I estimate how much provisioned throughput I need?

Capture baseline metrics from your pay-per-token usage: peak concurrent requests, average input and output token counts, and your acceptable first-token latency. Use the Databricks model-units estimator in the documentation to translate those into a capacity reservation.

Does the API format change between pay-per-token and provisioned throughput?

No. Both endpoint types accept the same OpenAI-compatible request format. Switching is a configuration change on the serving endpoint, not an application code change.

Are there term commitments for provisioned throughput?

Yes. Databricks offers on-demand provisioned throughput (billed hourly) and reserved capacity with term commitments. Reserved capacity typically offers a lower per-unit cost in exchange for the commitment.