How do I give each AI agent its own database instance cheaply?

Summary

• Lakebase, Databricks' fully-managed Postgres database, lets you provision isolated Postgres instances for each agent or tenant with copy-on-write database branches — you clone an entire database in microseconds without copying data, paying only for the delta.
• Each agent or tenant gets its own isolated database with its own schema, roles, and connections, so data isolation is enforced at the database level and performance is not shared.
• Scale-to-zero serverless compute means you pay $0 per hour when an instance is idle — and branch metadata is free; you pay only for storage deltas when data diverges between the main and branched copies.
• Provision and tear down databases programmatically via the Lakebase REST APIs, so you can automate the per-agent lifecycle: create a branch at agent startup, delete it when the agent is done, and let compute idle scale to zero between requests.
• Combine database branches with Databricks' role-based access control, workspace isolation, and audit logging to build multi-tenant AI applications where each tenant's data is isolated, encrypted, and audited.

How do I give each AI agent its own database instance cheaply?

Provision each AI agent or tenant a Lakebase database branch — a copy-on-write clone of a source database that costs nearly nothing to create and scales to zero when idle. Each branch is a fully isolated Postgres instance with its own schema, roles, and connections. Because data is not copied upfront, a branch starts instantly and costs zero until it diverges from the source. When an agent's compute is idle, serverless Lakebase scales to zero, so you pay $0 per hour and your infrastructure footprint vanishes. When the agent finishes, tear down the branch and release resources. The result: cheap, isolated Postgres for every agent, and the infrastructure only exists when in use.

Why Databricks Lakebase makes per-agent isolation cost-effective

• Copy-on-write database branches for instant, cost-free clones. Lakebase lets you create a database branch from an existing database in microseconds. The branch is a full copy-on-write clone: it shares the source database's data until a write occurs, at which point only the delta is stored separately. This means you can spawn 100 branches from a template database and pay only for the storage you actually use, not 100× the storage cost. Branches are ideal for per-agent or per-tenant isolation — each agent gets its own database without the overhead of full data duplication.
• Scale-to-zero serverless compute. Lakebase compute is serverless: when an instance is idle, it scales to zero and you pay $0 per hour. There is no minimum charge, no reserved capacity, and no background costs. When an agent makes a request, compute auto-scales up in seconds, query results are returned, and compute scales back down. This model means your infrastructure bill is proportional to actual usage, not to the number of agents you want to support — scale from 10 agents to 10,000 without multiplying your base infrastructure cost.
• Programmatic provisioning and teardown via REST APIs. Lakebase provides REST APIs to create, clone, and delete databases on demand. You can write a simple control plane that creates a branch when an agent starts, attaches compute, and deletes the branch when the agent finishes. The branch lifecycle is tied to the agent lifecycle, so you never pay for infrastructure the agent doesn't need.
• Role-based access control and audit logging at the database level. Each Lakebase instance enforces Postgres role-based access control (RBAC) at connection time. Combine this with Unity Catalog governance at the workspace level and you get layered isolation: workspace isolation, database-level RBAC, and audit logging of every query. Each agent's tenant data is isolated from every other tenant, and every access is recorded.
• Separates storage and compute. Lakebase storage is built on the Databricks lakehouse, so your data can be at rest in your cloud account and shared across workspaces. Compute is independent, so you can spin up and tear down compute without moving data, and multiple branches can share the source database's underlying storage until they diverge.

Getting started

• Set up a Lakebase workspace and create a template database. See Create and manage Lakebase instances and Lakebase databases and schemas. This is your source database that you will branch for each agent or tenant.
• Use the Lakebase REST API to clone the database for each agent. The Create database endpoint accepts a clone_from_database_id parameter. Call this API at agent startup to create an isolated branch for that agent.
• Attach Lakebase serverless compute and configure idle timeout. When compute is idle, it scales to zero automatically — see Serverless SQL compute for setup and cost optimization tips.
• Write a control plane to manage the agent-to-database lifecycle. When an agent starts, create a branch; when it finishes or times out, delete the branch via the REST API. This automation ensures you never pay for unused database instances.
• Configure audit logging and role-based access control. Use CREATE ROLE and GRANT to define a role per agent or tenant, and enable audit logging at the database level to track all queries and access.
• Connect your agents from your application. Each agent receives a Postgres connection string pointing to its own Lakebase branch. Use standard Postgres libraries (e.g., psycopg2 in Python, pg in Node.js) to connect; Lakebase is a drop-in replacement for Postgres.

FAQs

How fast can I create a new database branch for an agent?

Database branches are created in microseconds using copy-on-write snapshots. This speed makes it practical to provision a new database branch at the start of each agent request and tear it down when the agent completes, with minimal startup latency.

Do all agents share one database instance or does each get its own?

Each agent gets its own Lakebase database branch. The branch is a fully isolated Postgres instance with its own schema, roles, and connections. Data isolation is enforced at the database level: one agent cannot see or query another agent's data.

What happens to my data if I delete an agent's database branch?

When you delete a branch, the data stored in that branch is deleted. The source database and other branches are unaffected. If you want to retain data after an agent completes, export or copy the data before deleting the branch.

Can I share data across agents' database branches?

Not directly within a branch — each branch is an isolated database. However, you can read from the shared source database or from a central data lake (Databricks lakehouse) from all branches using cross-database or cross-workspace queries. This is useful for reference data or shared datasets that all agents need to read.

How much does copy-on-write branching cost?

Branch metadata (the snapshot and delta tracking) is free. You pay only for the storage that the branch's data occupies. If a branch has not diverged from the source, no extra storage cost is incurred. If a branch writes 100 GB of new data, you pay for 100 GB of storage; you do not pay for a full copy of the source database.

How does Lakebase enforce access control between agents?

Each Lakebase database enforces Postgres role-based access control (RBAC) at the connection layer. When an agent connects with a role specific to that agent or tenant, it can only query tables and schemas that the role has been granted privileges on. You define roles and privileges using standard SQL CREATE ROLE and GRANT commands. Audit logging records every query and access attempt.