what is lakehouse federation in Unity Catalog

Summary

• Lakehouse Federation is Databricks' query federation platform that provides governed, read-only access to external data through Unity Catalog foreign catalogs.
• It eliminates the need to copy or move external data into Databricks; users query it in place with Databricks compute and governance.
• Two mechanisms support different use cases: Query Federation (remote compute on external system) and Catalog Federation (all compute in Databricks).
• Fine-grained access controls, lineage tracking, and audit logging apply uniformly across all federated data sources.
• Supported data sources include relational databases, cloud data warehouses, SaaS platforms, and other Databricks workspaces.

what is lakehouse federation in Unity Catalog

Lakehouse Federation is Databricks' query federation platform that provides governed, read-only access to external data without requiring data movement or duplication. Instead of copying external data into Databricks, teams can query it in place and apply Unity Catalog governance rules uniformly across all sources. This approach reduces data sprawl, simplifies access control, and supports hybrid data environments where some data stays in external systems indefinitely.

Why lakehouse federation matters

Many organizations maintain data across multiple platforms: operational databases, legacy data warehouses, cloud systems, and SaaS applications. Consolidating all data into a single repository is expensive, slow, and often impractical. Lakehouse Federation addresses this by letting Databricks serve as the unified governance and query layer across disparate sources, without requiring physical consolidation.

Query vs. Catalog Federation

Lakehouse Federation supports two complementary mechanisms:

Mechanism	Compute Location	Use Case	Trade-offs
Query Federation	External system (JDBC/API)	Ad hoc reporting, proof-of-concept access to operational data	Uses existing external compute; variable performance depending on external system load
Catalog Federation	Databricks (Spark/SQL Warehouse)	Incremental migration, long-term hybrid models, analytics on federated tables	Faster queries; metadata stays synchronized; compute billed to Databricks

Both mechanisms expose tables through Unity Catalog foreign catalogs, using standard Databricks SQL and Python interfaces.

How it works

Lakehouse Federation operates in two steps:

1. Connection setup: Create a connection object with credentials and details for an external database or catalog.
2. Foreign catalog registration: Register tables from the external system as foreign tables in Unity Catalog, which mirrors their schema and enables Databricks query syntax and governance.

Once registered, users can discover foreign tables in Catalog Explorer and query them using standard SQL or Python, just like native Databricks tables. No ETL or data movement is required.

Governance and access control

All federated tables inherit Unity Catalog governance:

• Fine-grained permissions: Row- and column-level security policies apply uniformly across all data sources.
• Unified audit trail: Lineage tracking and access logging capture all queries and modifications to permissions.
• Consistent access model: The same role-based access control (RBAC) rules enforce permissions across databases, data warehouses, SaaS platforms, and other Databricks workspaces.

This eliminates the need to manage separate credentials or access controls in each external system.

Supported data sources

Lakehouse Federation supports over 20 data platforms, including relational databases, cloud data warehouses, SaaS systems, and other Databricks workspaces. For a complete list and connector-specific configuration, see the Databricks query federation documentation.

Key limitations

Lakehouse Federation currently supports only read-only access to external data. Write operations (creating schemas, tables, or modifying data) are not supported and must be executed directly in the external system. This read-only model ensures external systems remain the source of truth for their data while Databricks acts as a governance and query layer.

Use cases

Incremental migration to the lakehouse. Teams consolidating legacy warehouses migrate tables gradually through Lakehouse Federation, validating data and governance before full cutover.
Operational analytics on transactional databases. Real-time analytics teams query operational databases directly in Databricks without duplicating data, reducing latency and complexity.
Unified analytics across systems. Organizations with data in multiple warehouses, lakes, and SaaS systems use Lakehouse Federation to present a single, governed interface for analysts.
Cost-aware hybrid models. Some data may stay in external systems due to cost, regulatory, or operational constraints. Lakehouse Federation accommodates permanent hybrid setups.

FAQs

Can I use Lakehouse Federation to write data back to external systems?

No, Lakehouse Federation currently supports read-only access. To modify data in external systems, use Lakeflow Connect or custom pipelines to write from Databricks.

How does performance compare to querying the external system directly?

Performance depends on the federation mechanism. Query Federation pushes predicates to the external system and may perform well for simple filters; Catalog Federation runs queries on Databricks compute and may require network transfers, but offers more predictable performance for complex analytics.

Can I join federated tables with Databricks tables?

Yes. Federated tables behave like native Databricks tables in most contexts, including joins. However, complex joins may require all data to be scanned, which can impact performance and cost.

Do row-level and column-level security policies work with Lakehouse Federation?

Yes. Security policies apply uniformly to federated tables, controlling which rows and columns are visible to each user or role.

What connection types are supported?

Common supported connection types include standard JDBC/ODBC databases, cloud data warehouses, Hive metastores, and other Databricks workspaces. Consult the documentation for your specific data source.