Getting Started with Unity Catalog: A Step-by-Step Databricks Demo

Summary

• Unified Governance Foundation: Unity Catalog provides centralized discovery, classification, and access management for all data and AI assets across the Databricks Data and AI platform.
• Data Democratization: Unified search, lineage tracking, and certification enable users to find trusted data and understand relationships without waiting for manual provisioning.
• Cross-Cloud Governance: Identities, policies, and controls are centralized at the account level, spanning multiple clouds, regions, and workspaces without manual per-workspace setup.

Getting Started with Unity Catalog: A Step-by-Step Databricks Demo

Unlock the full potential of the Databricks Data and AI platform by mastering its governance backbone: Unity Catalog. In this technical walkthrough, we demonstrate how to move beyond fragmented data silos and establish a unified, secure foundation for all your data, analytics, and AI initiatives.
Whether you are accelerating a data warehouse migration or building the next generation of AI Agents, governance is critical. Watch as we take you step-by-step through the Unity Catalog interface, showing you how to manage permissions, ensure compliance, and democratize data access without compromising security.
Ready to build your unified governance layer?

Chapters

• 0:00 Introduction
• 0:52 Discovery
• 1:55 Compute
• 2:58 Catalog
• 3:52 Search
• 4:29 Governance
• 5:03 Privacy
• 5:26 Masking
• 6:13 Service Plans
• 6:48 Quality Signals
• 7:35 Delta Sharing
• 8:20 Audit Logging
• 9:01 Conclusion

FAQs

How does Unity Catalog centralize governance across multiple workspaces and regions?

Instead of managing access policies in each workspace, Unity Catalog allows account admins to define groups and grant permissions centrally. For example, define 'Lumina IoT Engineers' once and grant access to tables across all catalogs and workspaces.

What discovery and lineage features does Unity Catalog provide?

Unity Catalog provides searchable, centralized discovery with visible lineage and data certification. Users can find data, understand relationships, see ownership, and trust data through these governance features.

Can Unity Catalog enforce row-level access controls?

Yes. This video demonstrates row-level access policies (Aback policies) that restrict data visibility by region or other criteria, ensuring users see only authorized data.

Does Unity Catalog support both classic and serverless compute?

Yes. Unity Catalog works with both classic compute (for specialized workloads requiring specific configurations) and serverless compute (for simpler, more managed workloads).

Full transcript

0:00 Introduction

Welcome to Unity Catalog for Databricks, the unified governance solution for data and AI. The rapid adoption of generative AI introduces unprecedented demands for trustworthy governed data. This new era requires a modern approach beginning with the need for robust data classification, management, discovery, and assessment, all enabling unified discovery in one catalog. Next, this foundation must be secured with AI-native governance for all assets, including granular features for data access management, data quality management, and compute management.

Lastly, organizations require a platform that is cross-cloud and open by design, allowing them to innovate without boundaries. In this demo, we will show how Lumina Technologies, a fictional company, leads innovation through governance and builds a culture of security across their organization. When we get discovery right, we move our users from confusion to clarity. They can find their data through a centralized searchable platform.

0:52 Discovery

They can trust their data with visible lineage and certification. They understand their data with tagging and descriptions, and your users have clear visibility into data ownership. All of this works at scale because identities and policies are centralized at the account level. We start here in the account console. This is where our centralized governance begins. Instead of managing access for every new region Lumina expands to, the account admin manages identities globally.
We sync our corporate identity providers and create and manage users, groups, and service principles to be used across all of our workspaces. For example, I can define a group called Lumina IoT engineers once. I can then grant that group access to the device telemetry table in the operation schema of our Lumina Technologies catalog. This ensures our engineering teams are productive immediately without waiting for manual provisioning if they work across multiple catalogs or workspaces.
Now, let's look at the engine powering Lumina's data. In the compute section, we support both classic and serverless options to match the right tool to the right job. First, we have classic compute. This allows our data engineers to provision specific cluster configurations, choosing exact instance types and libraries needed for complex tasks like training our churn prediction ML models. It offers granular control for specialized workloads.

1:55 Compute

Second, and perhaps more exciting for the business, is serverless. Here, we see our serverless SQL warehouses. These power our our BI tools and SQL queries. They spin up instantly and scale automatically based on user load. This means our finance team never has to wait for a cluster to start up to run a report, and we stop paying the second the query is done. It optimizes both user experience and cost. Crucially, both classic and serverless are Unity Catalog enabled.
Whether you are running a Python script on a classic cluster or a SQL query on serverless, the same governance policies are enforced automatically. Let's look at how Lumina organizes its data using the three-level namespace of catalog, schema, and table. Navigating to the data explorer, we encounter the Unity Catalog three-level namespace structured as catalog, schema, and object. This hierarchy is designed to solve the complexity of managing your modern data estates.

2:58 Catalog

The top level, catalog, serves as your primary container, which ensures rigid isolation between sensitive data and sandbox environments. Inside the catalog, schemas organize your assets logically, while the object layer unifies all your data and AI assets, including tables, views, unstructured volumes, and ML models. This structure not only simplifies discovery, but streamlines the promotion of data pipelines across environments without requiring code refactoring.
Combined with the medallion architecture, you can map your bronze, silver, and gold layers to catalogs, schemas, or tables, depending on your organization's needs. Next, we examine the search capabilities within Unity Catalog. This is not merely a keyword match, it is a semantic discovery engine powered by rich metadata. Because Unity Catalog unifies your technical metadata with business context, such as tags, descriptions, and column comments, users can discover assets using natural business language rather than obscure technical object names.

3:52 Search

Crucially, this search experience is security-aware by design. The results are dynamically filtered based on the user's specific access privileges, ensuring that users only discover assets they are authorized to view, thereby reducing noise and maintaining strict compliance standards automatically. Let's look at the customers table. This contains sensitive contact info, so governance is paramount. Here in the UI, I can see exactly who has access.

4:29 Governance

I can grant the support group team select access to look up accounts, but ensure they cannot modify the records. For our data engineers automating our platform, they can manage this via code. This flexibility supports both our UI-based data stewards and our code-first engineering teams. To address the need for granular privacy, Unity Catalog provides native row-level security. Rather than managing dozens of separate views for different user groups, we can attach a dynamic filter policy directly to the table definition.

5:03 Privacy

This policy evaluates the user identity or group membership at query time to filter out unauthorized records before they are ever returned. We can apply a filter on the orders table so that regional managers only see orders from their specific region. Complementing our row-level controls is column-level masking. This feature allows you to protect sensitive attributes, such as PII, financial details, or health information without restricting access to the entire data set.

5:26 Masking

By applying dynamic masking policies to specific columns, you can mask values in real time based on the user's role. This enables a principle of least privilege architecture where analysts can still perform aggregated reporting and derive value from a table while remaining completely blind to the sensitive individual data points required for compliance with regulations like GDPR and CCPA. On this customers table, we can mask PII.
We can set a policy that masks the email and phone number columns for everyone except for the HR team, ensuring we meet privacy regulations. Let's look at the service plans table. Before I write a complex query to join with subscriptions, I can view sample data. I can quickly verify the plan tiers, basic, premium, protect, to ensure this is the reference table I need. To help my team, I can add a description to the table and comments on each one of the columns.

6:13 Service Plans

But instead of manually doing this, I can leverage generative AI to create descriptions for me. I can also tag this table as domain billing. This helps us organize ownership. If there is an issue with billing data, we know exactly which domain team is responsible. Unity Catalog also services quality signals, so teams spot broken pipelines before analysts ever run queries. Now, for one of the most powerful features for Lumina's data reliability, lineage, which Unity Catalog captures automatically in real time as your data pipelines execute.

6:48 Quality Signals

This provides an always up-to-date map of your data flow by eliminating the need for manual documentation. By visualizing both upstream and downstream dependencies, teams can perform instant root cause analysis to trace the origin of data anomalies or conduct precise impact analysis to understand the blast radius of a potential schema change. Furthermore, with column-level granularity, you can trace specific metrics back to their raw source inputs, ensuring total transparency and trust in your analytical reporting.
Delta sharing allows us to securely share live data assets with any external partner, regardless of the platform or cloud they use. Lumina works with a third-party partner for their hardware devices. This partner runs on another cloud and queries the share directly. No replication, no proprietary connector. Delta sharing is an open protocol fully integrated into Unity Catalog, unlike legacy methods that require data replication, complex API development, or proprietary vendor lock-in.

7:35 Delta Sharing

By managing these shares within Unity Catalog, we can maintain a single control plane for governance, allowing us to grant, monitor, and instantly revoke access to live data sets without ever losing custody of the underlying data or creating stale copies. Governance is incomplete without comprehensive observability, which brings us to audit logging. Unity Catalog automatically captures a centralized immutable record of every action taken across your lakehouse, from access control changes to individual data queries.

8:20 Audit Logging

These logs are exposed directly as queryable system tables, enabling your security and compliance teams to monitor usage patterns and detect anomalies using standard SQL. Whether you need to investigate a specific security incident or demonstrate compliance for regulatory audit, you have immediate granular visibility into exactly who accessed what data and when, without needing to export logs to third-party tools. Unity Catalog fundamentally transforms governance from a bureaucratic bottleneck into a strategic accelerator for innovation.

9:01 Conclusion

By unifying data and AI under a single automated control plane, organizations no longer have to choose between agility and compliance. It democratizes access to trusted data, empowering teams to experiment faster and deploy products with confidence, knowing that security, lineage, and auditability are baked into the foundation. Ultimately, Unity Catalog provides the intelligent guardrails that give your organization the freedom to quickly innovate at scale.
To learn more about Unity Catalog, go to databricks. com/product/unity-catalog.

Learn more about the Databricks Data and AI platform.