End-to-end platform security across data, governance, and AI

Summary

• End-to-end security for a data and AI platform spans authentication, networking, encryption, governance, auditing, and compliance, and each layer has its own configuration in Databricks.
• Unity Catalog is the governance layer for data and AI. It enforces access control when you query a table or call a model, tracks lineage, and logs activity for auditing.
• Access starts closed. Users have no access to data in a metastore until someone grants it, and privileges inherit down the catalog, schema, and table hierarchy.
• Row filters, column masks, and attribute-based access control (ABAC) policies narrow access to specific rows and columns.
• The compliance security profile adds a hardened compute image and other controls, but you stay responsible for configuring it before you process regulated data.

End-to-end platform security across data, governance, and AI

End-to-end security means one connected set of controls, from the network edge to the model that answers a request. On Databricks, authentication, networking, encryption, and compliance features protect the workspace, and Unity Catalog governs the data and AI assets inside it. Network controls do not decide who may read a column, and table permissions do not protect traffic.

Which security layers does Databricks document?

Feature names in this table follow the Databricks on AWS documentation.

Layer	What it controls	Documented features
Authentication	Who can sign in and call APIs	Single sign-on, service principals, OAuth, personal access tokens, SCIM provisioning
Network	Which networks reach the workspace, and what compute can reach out	Customer-managed VPC, front-end and back-end PrivateLink, IP access lists, serverless egress control
Encryption	Data at rest and in transit	Customer-managed keys for managed services and workspace storage, encryption of traffic between cluster worker nodes
Governance	Who sees which data and AI assets	Unity Catalog privileges, row filters, column masks, ABAC policies
Audit and lineage	What happened, and where data came from	Audit logs, lineage system tables, inference tables
Compliance	Regulated workloads	Compliance security profile

How does Unity Catalog enforce access for data and AI?

Unity Catalog is "the unified governance layer for data and AI built into Databricks." It governs tables, views, volumes, functions, models, and services in a three-level namespace, catalog.schema.object.
Access starts closed. The privilege documentation says, "Initially, users have no access to data in a metastore." An object owner, a user with the MANAGE privilege, or a metastore admin must grant access. Privileges inherit downward: granting SELECT on a schema grants it on all current and future tables and views in that schema. Users also need USE CATALOG and USE SCHEMA on parent objects. For functions and registered models, EXECUTE lets a user invoke a function or load a registered model for inference.
Fine-grained controls sit on top. A row filter, a SQL user-defined function that evaluates each row at query time, restricts which rows a user can see. A column mask, also a SQL function, controls the values a user sees in a column. ABAC policies attach at a level such as a catalog, schema, or table, and use governed tags to identify which data a policy protects. Databricks Data Classification uses an agent to classify and tag tables in a catalog, and you can create an ABAC policy from its results page.

How do lineage, audit logs, and inference logs support investigation?

Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level. Two system tables expose it: system.access.table_lineage and system.access.column_lineage. When you train a model on a Unity Catalog table and log the input with mlflow.log_input, registering the model saves lineage that appears on the model version page in Catalog Explorer.
Audit logs record user actions and system events at the workspace and account levels. You query them through the audit log system table, system.access.audit, which is in Public Preview. For AI requests, Unity Gateway inference tables log request and response payloads from model services to Unity Catalog Delta tables.

What does the compliance security profile add?

The compliance security profile "enables additional monitoring, enforced instance types for inter-node encryption, a hardened compute image, and other features and controls on Databricks workspaces." Documented controls include a CIS Level 1 hardened image with FIPS 140 validated encryption modules, automatic cluster updates, enhanced security monitoring, enforced AWS Nitro instance types, and TLS 1.2 or higher for egress communication.
It supports these standards: C5, CCCS Medium (Protected B), DoD IL5, FedRAMP High, FedRAMP Moderate, HIPAA, HITRUST, IRAP, ISMAP, K-FSI, PCI-DSS, TISAX, and UK Cyber Essentials Plus. Databricks' SOC 2 Type II report is a separate document that your Databricks account team can provide.

Use cases

Regulated healthcare data (illustrative): A team enables the compliance security profile with the HIPAA standard, limits patient tables to approved groups, and applies a column mask to direct identifiers.
Financial analytics (illustrative): A team uses a row filter so each analyst sees only the rows for their region, and traces a credit risk model to its training tables with model lineage.

FAQs

Does a compliance standard mean my workloads are automatically compliant?

No. The documentation states: "You are solely responsible for ensuring your own compliance with all applicable laws and regulations." Configure the compliance security profile and the appropriate standards before you process regulated data.

What access does a new user have in Unity Catalog?

None to data. An owner, a user with MANAGE, or a metastore admin must grant a privilege. Because privileges inherit downward, a grant on a catalog or schema also covers current and future child objects, so grant at the narrowest level that fits.

Does Databricks provide multi-factor authentication?

Through your identity provider. The documentation says that once SSO is configured, you can enable multi-factor authentication using your identity provider.

How does governance extend to AI models?

Models in Unity Catalog bring centralized access control, auditing, lineage, and model discovery to ML models across workspaces. The EXECUTE privilege controls who can load a registered model for inference, and model lineage links a model to its training tables.