enterprise llm platform

Summary

• An enterprise LLM platform is the layer between applications and models. It gives teams governed access to models and tools, a place to run agents in production, and controls for cost, safety, and quality.
• Model choice is a governance question as well as a technical one. Unity Gateway gives agents one API for frontier and open models, including models hosted by Databricks and models from external providers, and the docs say you can switch models without changing agent code.
• Guardrails and cost controls sit in the same control plane. Unity Gateway applies rate limits and usage tracking, sets guardrails through service policies (Beta), and supports budgets with per-user thresholds and hard caps.
• Unity Catalog governs the data, tools, models, and agents that your agents use, with permissions and lineage.
• Quality is measured, not assumed. MLflow Tracing, evaluation scorers, and production monitoring turn agent behavior into evidence.

enterprise llm platform

An enterprise LLM platform is the layer between your applications and the models. A model API answers a prompt. A platform also answers the organizational questions: who may call which model, what an agent may touch, what it costs, and how you know it works. On Databricks, Agent Bricks provides the building blocks: Unity Gateway for model and tool access, Agent Runtime for hosting, Unity Catalog for governance, and MLflow for tracing and evaluation.

What makes a platform enterprise-grade?

Capability	What to check
Model choice	One governed entry point to frontier and open models, and the ability to switch models without rewriting the agent
Agent hosting	Managed compute that scales, survives restarts and failures, and runs agents built with the framework you choose
Data and tool governance	Permissions, policies, and lineage that cover the data, tools, and models an agent uses
Guardrails	Policies that inspect requests and responses, block unsafe content, and pause sensitive actions for approval
Cost controls	Rate limits, budgets, and usage and cost attribution by user, model, and tag
Quality measurement	Tracing, evaluation, and production monitoring that show whether the agent's output is good

How does Databricks approach an enterprise LLM platform?

Agent Bricks is the Databricks agent developer platform, built for developers who write agent code. You bring an agent built with any framework or harness and choose any model.

• Model choice. Unity Gateway gives agents one API for frontier and open models. Databricks serves models such as GPT and Claude natively through Foundation Model APIs, and you can connect external providers such as OpenAI and Anthropic with bring your own key and govern them through the same control point.
• Hosting. Agent Runtime hosts agents built with any framework or harness, stateful or stateless, on Databricks Apps. DurableAgentServer adds durable execution so that runs survive restarts and crashes. Omnigent (Beta) is a meta-harness that composes and coordinates multiple agents, and Databricks provides a fully managed version.
• Context. Managed memory and sessions are stores backed by Lakebase that work with any framework. Agents can also connect to Databricks-managed MCP servers for Genie Agents, AI Search, Unity Catalog functions, and SQL.
• Governance. By default an agent runs as its app's service principal, and tools that use auth = "user" run with the permissions of the user who sent the request. Unity Catalog permissions apply to the data and tools involved.
• Guardrails. Service policies (Beta) are the mechanism for guardrails. A policy returns ALLOW, DENY, or ASK, and on an MCP service ASK pauses a request for user approval before the tool runs. A custom policy is either a SQL user-defined function or an LLM-as-a-judge policy, and Databricks also has built-in guardrails for common risks. For example, a policy can deny a model response that contains personally identifiable information (PII) or block unsafe content.
• Cost and audit. Unity Gateway enforces rate limits on model and MCP services, supports budgets with per-user thresholds and hard caps, and adds traffic splitting and fallbacks. It tracks requests, token usage, and latency in system tables, attributes cost to services, models, principals, and tags, and can log requests and responses to Unity Catalog Delta tables.
• Quality. MLflow Tracing records the inputs, outputs, latency, token usage, and cost of every intermediate step. Evaluation uses built-in LLM judges, custom scorers, and expert feedback. You can align judges with human feedback, and MLflow prompt optimization (Beta) uses the GEPA algorithm to improve prompts from evaluation metrics and training data.
• Code execution. Databricks Sandbox (Beta) lets an agent run the code it writes in an isolated environment with scoped access to governed data.

FAQs

How do I avoid vendor lock-in with an enterprise LLM platform?

Choose a platform that lets you change models and frameworks without rewriting the agent. Databricks documentation says you keep your framework, harness, and model choices, and that you can switch models without changing agent code.

What does governed data access mean for an agent?

The agent has an identity, and the platform checks what that identity may reach. Unity Catalog permissions apply to the data and tools involved. The Genie One MCP server always enforces them, so users and agents can only query data they are allowed to access.

How does production evaluation work?

MLflow scores traces with scorers, which are LLM judges and code-based checks, and attaches each result to its trace as feedback. Production monitoring runs the same scorers automatically on a sample of production traffic.

What happened to Knowledge Assistant and Supervisor Agent?

Databricks documents them as legacy offerings that are no longer recommended for new agents. For new low-code agents, use Genie Agents. For agents written in code, use the Agent Bricks CLI (Beta).