What is the reference architecture for internal data apps

Summary

• A reference architecture for internal data apps combines a serverless presentation layer (UI framework), a governance layer (Unity Catalog for access control), and a data layer (lakehouse or Lakebase for managed data).
• Databricks Apps provides the presentation layer with support for Python frameworks (Streamlit, Dash, Gradio) and Node.js (React, Express), eliminating the need to manage separate app infrastructure.
• Data apps authenticate users via OAuth and inherit fine-grained access control from Unity Catalog, so row-level and column-level permissions automatically enforce in the app.
• The architecture supports deployment from local development to workspace deployment to cross-workspace portability.
• Monitoring, compliance, and audit trails are built in: logs and telemetry are automatically captured in Unity Catalog.

What is the reference architecture for internal data apps

A reference architecture for internal data apps stacks three layers: a presentation layer (UI built with a web framework), a governance layer that enforces access control, and a data layer that serves reliable, current information. Databricks Apps provides an integrated reference architecture by combining serverless compute, Unity Catalog, and the lakehouse or Lakebase. This eliminates separate infrastructure and ensures every access decision is auditable.

The three-layer architecture

Layer	Component	Purpose
Presentation	Databricks Apps + web framework (Streamlit, React, etc.)	User interface; serverless, no infrastructure to operate
Governance	Unity Catalog + OAuth	Access control, audit trails, compliance; permissions inherited by the app
Data	Lakehouse (Delta) or Lakebase (Postgres)	Single source of truth; managed, versioned, governed data

This eliminates common sources of complexity: managing servers, securing data access separately from the data layer, keeping audit trails disconnected from access enforcement, and reconciling permissions across systems.

Presentation layer: Databricks Apps

Databricks Apps is a serverless platform for building web applications on Databricks. Develop locally using Python (Streamlit, Dash, Gradio) or Node.js (React, Express). Deploy to a workspace with a single command; the platform handles compute, scaling, and billing. Common app types include interactive dashboards, RAG chat applications, data entry forms, and operational dashboards.
Applications run in a secure sandbox, authenticated via OAuth, and require no infrastructure management.

Governance layer: Unity Catalog and OAuth

Databricks Apps integrates natively with Unity Catalog. When an app queries a table or Lakebase database, it inherits the permissions of the authenticated user. Row-level filters and column-level masks automatically apply without permission logic in app code.
Benefits:

• No permission logic in code. Unity Catalog decides what data a user sees based on policies.
• Complete audit trails. Every access is logged to system tables with user identity and timestamp.
• Permissions change without redeployment. Grant a user access to a new table; they immediately see it in the app.

Data layer: Lakehouse or Lakebase

The data layer is either a lakehouse (managed Delta tables in object storage) or Lakebase (managed Postgres database integrated with the lakehouse). Both provide reliable, versioned data with centralized governance in Unity Catalog. The lakehouse scales compute and storage independently; Lakebase offers autoscaling and scale-to-zero.

How it works in practice

1. User authenticates via OAuth.
2. App queries the lakehouse or Lakebase.
3. Unity Catalog checks user permissions; results are filtered accordingly.
4. Query and user identity are logged to a system table for audit.
5. App displays results; user can interact, filtering or drilling down.

Deployment patterns

Local development: Develop and test locally with standard Python or Node.js tooling.
Workspace deployment: Deploy to a Databricks workspace; app is accessible via a stable URL.
Cross-workspace: Export and deploy to another workspace (dev to staging to production). Configuration moves with code.
External embedding: Embed a Databricks app in an external website using iframes or the embedding API.

Operational considerations

App logs, traces, and metrics are automatically captured in Unity Catalog. Query system tables to analyze usage, errors, and performance. If your workspace has a compliance security profile, the app automatically enforces the same audit and encryption standards.
Apps are billed per hour of compute time, with autoscaling adjusting resources based on traffic. Apps can call external APIs, trigger jobs, or invoke AI models using MCP tools or SDK calls.

FAQs

Do I need to manage servers for my internal app?

No. Databricks Apps handles provisioning, scaling, and billing. You write code and deploy; the platform runs it.

How do I control who sees what data?

Define fine-grained access policies in Unity Catalog (row filters, column masks) based on user roles. The app automatically inherits these policies.

Can I deploy to multiple environments?

Yes. Export the app and deploy to another workspace. Configuration and code move together.

How do I integrate with external systems?

Use MCP tools or SDK calls to invoke external APIs from the app. Log all calls to Unity Catalog for audit trails.