What are foundation model APIs and how do they work?

Summary

• Foundation model APIs provide programmatic access to large language models, enabling developers to integrate AI capabilities into applications without building models from scratch.
• These APIs work through REST endpoints where you send text prompts and receive generated completions, with pricing based on input and output token consumption.
• Key features to evaluate include model availability, latency, rate limiting, context window size, and fine-tuning or customization options.
• Databricks Foundation Model APIs provide governance, audit logging, and integration with your data through Unity Catalog.
• Common use cases range from customer service automation and content generation to data analysis and code assistance, with security and compliance requirements varying by use case.

What are foundation model APIs and how do they work?

Foundation model APIs provide programmatic access to large language models, allowing developers to add AI capabilities to applications without building or training models themselves. These APIs accept prompts as input and return text completions, enabling everything from customer support automation to content generation to data analysis tasks.
Under the hood, a foundation model API processes your request through several steps. You send text (your prompt) to the API endpoint via HTTPS, specifying which model you want to use. The API routes your request through the provider's infrastructure, which runs the model on the prompt and generates a completion. You receive the response as structured JSON, typically containing the generated text and metadata like token counts. Pricing is calculated based on tokens consumed: input tokens (from your prompt) and output tokens (the model's response), usually with different rates for each.
These APIs differ from downloadable open-source models in a crucial way: they're managed services, so you don't need to handle deployment, scaling, or maintenance. This makes them ideal when you want production-grade reliability without infrastructure overhead.

Key capabilities to evaluate

Capability	Why it matters
Model selection	Different models have different strengths. Some excel at code generation, others at reasoning or creative writing. Providers offer various options at different capability/cost trade-offs.
Context window	The number of tokens (roughly words) an API can accept in a single request. Larger windows let you include more context, which improves accuracy for complex tasks.
Latency	Response time matters for user-facing applications. Real-time use cases need sub-second responses; batch jobs can tolerate longer latency.
Rate limits and quotas	How many requests per second can you send, and are there monthly usage limits? Critical for production scaling.
Cost per token	Token pricing varies significantly between providers and models. High-volume applications should model costs carefully.
Fine-tuning or customization	Some providers let you adapt models to your domain or use case, improving quality for specialized tasks.
Audit and governance	For regulated industries, you need request logging, access controls, and compliance features.
Data residency and privacy	Whether requests can stay within your infrastructure or organization's data boundaries.

How Databricks approaches foundation model APIs

Databricks Foundation Model APIs provide managed access to a range of models through Databricks Marketplace. The APIs are built with enterprise governance in mind: every request is logged in Unity Catalog audit tables, making it easy to track API usage, cost attribution, and compliance. You authenticate via Databricks service principals or personal access tokens, so access control integrates with your existing Unity Catalog permissions.
Databricks routes requests through a rate-limited endpoint, and pricing follows a per-token consumption model. You can query available models and their current pricing via the API itself, and the REST interface accepts requests in the standard OpenAI-compatible format, making migration from other providers straightforward. Integration with Databricks notebooks and jobs means you can call these APIs directly from your data pipelines or analytics workflows, keeping everything within your lakehouse.
For organizations that need stronger data governance or model customization, Databricks lets you bring your own models to Model Serving, creating private endpoints with the same audit and access-control guarantees.

Use cases

Customer service automation: Route incoming support tickets through a foundation model API to classify severity, draft responses, or extract structured data. The model handles language understanding, freeing your team for escalations.
Content generation and summarization: Automatically generate product descriptions, meeting summaries, or email drafts. Foundation models excel at rephrasing and adapting content for different audiences.
Data analysis assistance: Enable analysts to ask natural-language questions about datasets. The API can interpret queries, generate SQL or Python, and explain results back in plain English.
Code generation and refactoring: Assist developers with code completion, bug detection, or refactoring suggestions. The model can understand code context and suggest improvements.
Financial and regulatory reporting: Extract relevant information from documents, classify compliance issues, or generate standardized reports. Audit logging ensures every action is tracked.

FAQs

How do foundation model APIs differ from open-source models?

Open-source models like Llama run on your own infrastructure, giving you full control but requiring you to handle deployment, scaling, and updates. Foundation model APIs are managed services where the provider handles all operations. You trade control for reliability and lower operational burden.

What determines API pricing?

Most providers charge per token consumed, with separate rates for input (your prompt) and output (the model's response). Some offer monthly or annual subscription tiers. Token counts depend on the model; different models tokenize text differently.

Can I use foundation model APIs for real-time applications?

Yes, but latency varies by provider and model. Cloud-based APIs typically respond within 1-5 seconds. For millisecond-critical use cases, deploying a model locally may be better. Check the provider's SLA and test with your workload.

Are foundation model API requests stored or used to train future models?

This varies by provider and terms of service. Databricks and most enterprise-focused providers do not use API requests to train or improve models without explicit consent. Review your provider's privacy policy if this is a concern.

What's the difference between APIs and fine-tuned models?

APIs give you out-of-the-box capabilities without customization. Fine-tuning trains a model on your data to improve performance on specific tasks. Fine-tuning typically costs more upfront but can reduce token consumption on specialized work, making it cost-effective at scale.