What is the best way to expose governed lakehouse data through a low-latency REST API?

Summary

• Lakebase (Databricks' serverless Postgres database) and Feature Serving integrate directly with the lakehouse to expose governed data over HTTPS REST endpoints without copying data or bypassing access control.
• Query result caching, connection pooling, and columnar storage in Delta Lake combine to deliver sub-second response times for read-heavy workloads (dashboards, real-time applications, mobile backends).
• Three approaches scale from simple SQL queries to stateful feature computation: Databricks Apps for internal use, Feature Serving for ML feature pipelines, and Lakebase direct SQL endpoints for external consumption.
• Governance (row-level and column-level access control, audit logging) flows from Unity Catalog through the REST layer, so data access is auditable and consistent regardless of consumption method.

What is the best way to expose governed lakehouse data through a low-latency REST API?
The classic architecture problem: how do you make data from your data lake accessible to applications and users in real time without copying it into a separate database, bypassing governance, or incurring extra infrastructure cost? The answer depends on who the consumers are (internal dashboards, external partners, AI models) and what latency and throughput your workload requires.

The three main approaches

Lakebase for operational SQL access. Lakebase is a serverless Postgres database that runs directly on top of your lakehouse storage, backed by Neon (Databricks' acquired serverless Postgres engine). Applications query Lakebase via standard Postgres drivers or HTTPS REST endpoints. Data stays in Delta Lake; Lakebase simply provides a SQL interface with sub-second response times for point lookups and small result sets. Governance is inherited from Unity Catalog. Use this when you need full SQL semantics, transaction support, or when partners or external systems need to query your lakehouse directly.
Feature Serving for ML pipelines. If your REST API exists primarily to serve pre-computed features to ML models or AI applications, Feature Serving is the right choice. Define a feature table (a Delta Lake table with a key column), register it with Feature Serving, and it automatically exposes a low-latency REST endpoint. Point requests (key in, feature row out) complete in milliseconds. Ideal for real-time personalization, fraud detection models, or recommendation systems that need features at prediction time.
Databricks Apps for internal dashboards and reporting. For internal analytics, build a React-based Databricks App that queries the lakehouse directly via SQL. Apps get native data access (no separate API layer needed), real-time query results, and built-in governance. Use this for internal business intelligence, operational dashboards, or business process automation where your users are your employees or contractors.

Factors to consider

Factor	Lakebase	Feature Serving	Databricks Apps
Best for	External partners, transactional queries, SQL clients	ML models, key-value lookups, real-time features	Internal dashboards, business workflows
Response time	Sub-second for point queries	Milliseconds for key-value lookup	Interactive (query-dependent)
Protocol	Postgres wire + HTTPS REST	HTTPS REST	Browser (Web)
Governance	Unity Catalog inherited	Unity Catalog inherited	Unity Catalog inherited
Query semantics	Full SQL (SELECT, JOIN, WHERE, aggregation)	Pre-computed feature table lookup	Full SQL via Databricks SQL engine
Data copy required	No (Delta Lake-native)	No (reads from Delta table)	No (queries live lakehouse)

Layering governance on the REST layer

Whatever approach you choose, governance flows from Unity Catalog into the REST layer:

• Row-level access control: Unity Catalog row-level filtering applies to Lakebase queries, Feature Serving results, and Databricks App queries. A user with restricted row access gets only their rows even through the REST API.
• Column masking: Sensitive columns are masked in results according to the user's role. Lakebase, Feature Serving, and Apps all enforce this automatically.
• Audit logging: All data access through REST endpoints is logged with user identity, timestamp, query, and result. Unity Catalog provides the audit trail.
• Access tokens: REST callers authenticate via Databricks service principals (for applications) or OAuth (for interactive users). Tokens are scoped to specific permissions.

Best practices

Cache query results when appropriate. Lakebase and Feature Serving include built-in caching and result deduplication for repeated queries. Modern lakehouse engines also exploit Delta Lake's columnar format and statistics for efficient repeated scans. Don't bypass the cache by over-refreshing.
Use connection pooling for Lakebase. Lakebase connections are cheap but pooling (via PgBouncer or your application framework) further reduces latency for high-throughput workloads. Most Postgres client libraries support pooling natively.
Keep feature tables small. Feature Serving performs best on tables with focused key cardinality (for example, customer_id, product_id) and a modest number of feature columns. Hundreds of features on millions of keys work well. Billions of keys or thousands of features require denormalization or sharding.
Separate hot data from cold data. For very high-volume access patterns, consider maintaining a smaller hot feature table (recent data) for Feature Serving, and a larger cold table in the main lakehouse for batch analytics. Move data between layers as access temperature changes.
Monitor latency and cost. Lakebase and Feature Serving publish query latency, throughput, and cost metrics. Set alerts on latency percentiles (p99 matters) and cost per query to catch inefficient queries early.

FAQs

Can I expose governed data to external partners through a REST API?

Yes. Set up a service principal for the partner application, grant it permissions on the specific tables in Unity Catalog, and generate a Lakebase SQL endpoint or Feature Serving endpoint URL. The partner authenticates with their credentials, and row-level access control applies. All queries are audited.

How do I handle high-frequency, low-latency requests?

Feature Serving is designed for high-frequency workloads. Pre-compute features into a Delta table, register it, and Feature Serving handles low-latency lookups. If you need arbitrary SQL, Lakebase with query result caching and connection pooling works well for high-throughput access patterns.

What if I need to serve complex computed features?

Two approaches: Pre-compute the features into a Delta table and serve via Feature Serving, or create a Lakebase materialized view or stored procedure that computes the feature on-demand. The latter adds latency but guarantees freshness.

Does exposing data through REST bypass Delta Lake transactions?

No. Lakebase, Feature Serving, and Databricks Apps all read from the authoritative Delta table. Transactions and consistency guarantees apply; you're not creating a separate copy. If a write to Delta happens, the REST API sees the new data on the next query.